Image restoration method, device, storage medium and electronic device

Abstract

The present disclosure relates to an image restoration method, apparatus, storage medium, and electronic device. The method includes: obtaining an image to be restored; inputting the image to be restored into a restoration model to obtain a restored image, the restoration model including a target encoding network and a target decoding network, the target decoding network including a plurality of alternately connected target convolutional upsampling modules and target spatial adaptive denormalization modules, and each target spatial adaptive denormalization module being configured to modulate a feature map from a previous module by using the image to be restored. In this way, the target spatial adaptive denormalization module can modulate and constrain the intermediate features obtained by the target convolutional upsampling module located in front of it by using the image to be restored, which can not only greatly reduce the influence of low-quality information such as noise, but also make the restored image more similar to the image to be restored, so as to obtain an image highly consistent with the image to be restored, ensure the effect of image restoration, and make the restored image more realistic and natural.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to an image restoration method, apparatus, storage medium, and electronic device. Background Art

[0002] In the process of image acquisition, processing, transmission, and recording, due to various possible reasons such as imaging systems, imaging methods, transmission media, and recording devices, or other possible reasons such as object movement and noise pollution, image distortion or image quality degradation will inevitably occur. Therefore, in specific application scenarios, these images are generally restored through image restoration processing. Nowadays, image restoration has been widely applied to specific scenarios such as old photo restoration, cultural relic protection, and removal of redundant noise. Due to the inherent blur and complexity of natural images, especially for face images, there may be problems such as blurred restoration details and uneven restored images, and the restoration effects of traditional methods are not good. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides an image restoration method, apparatus, storage medium, and electronic device.

[0004] According to a first aspect of an embodiment of the present disclosure, an image restoration method is provided, including:

[0005] Obtaining an image to be restored;

[0006] Inputting the image to be restored into a restoration model to obtain a restored image, where the restoration model includes a target encoding network and a target decoding network, the target decoding network includes a plurality of alternately connected target convolutional upsampling modules and target spatial adaptive denormalization modules, the target encoding network is configured to perform feature extraction on the image to be restored to obtain a first feature map, each target convolutional upsampling module is configured to modulate the feature map from the previous module by using the first feature map to obtain a second feature map, each target spatial adaptive denormalization module is configured to modulate the feature map from the previous module by using the image to be restored to obtain a third feature map, and the feature map of the module before the first target convolutional upsampling module of the target decoding network is first random noise.

[0007] Optionally, the target spatial adaptive denormalization module includes a target size adjustment module, a first target convolutional layer, a second target convolutional layer, a target modulation module connected in sequence, and a third target convolutional layer connected between the first target convolutional layer and the target modulation module, where one end of the target modulation module is connected to the previous module of the target spatial adaptive denormalization module where it is located, and the other end of the target modulation module is connected to the next module of the target spatial adaptive denormalization module where it is located;

[0008] Among them, the target size adjustment module is used to adjust the to-be-repaired image to a preset size to obtain a first image;

[0009] The first target convolutional layer is used to perform feature extraction on the first image to obtain a fourth feature map;

[0010] The second target convolutional layer is used to generate a variance feature map according to the fourth feature map;

[0011] The third target convolutional layer is used to generate a mean feature map according to the fourth feature map;

[0012] The target modulation module is used to modulate the feature map from the previous module according to the variance feature map and the mean feature map to obtain a third feature map.

[0013] Optionally, the target spatial adaptive denormalization module further includes a target normalization layer connected between the previous module of the target spatial adaptive denormalization module and the target modulation module;

[0014] The target normalization layer is used to perform normalization processing on the feature map from the previous module;

[0015] The target modulation module is used to modulate the normalized feature map from the previous module according to the variance feature map and the mean feature map to obtain a third feature map.

[0016] Optionally, the repair model is trained in the following way:

[0017] Obtain training data, where the training data includes high-definition images and low-quality images corresponding to the high-definition images;

[0018] Perform image repair on the low-quality image through an encoder-decoder model to obtain a second image, where the encoder-decoder model includes an original encoding network and an original decoding network, the original decoding network includes a plurality of alternately connected original convolutional upsampling modules and original spatial adaptive denormalization modules, the original encoding network is used to perform feature extraction on the low-quality image to obtain a fifth feature map, each original convolutional upsampling module is used to modulate the feature map from the previous module by using the fifth feature map to obtain a sixth feature map, each original spatial adaptive denormalization module is used to modulate the feature map from the previous module by using the low-quality image to obtain a seventh feature map, and the feature map of the module before the first original convolutional upsampling module of the original decoding network is second random noise;

[0019] Update the model parameters of the encoding and decoding model based on the high-definition image and the second image;

[0020] In response to not meeting the training cutoff condition, re-execute the step of obtaining training data to the step of updating the model parameters of the encoding and decoding model based on the high-definition image and the second image;

[0021] In response to meeting the training cutoff condition, obtain the restoration model.

[0022] Optionally, the original decoding network is a generative adversarial network including a generator and a discriminator, wherein the generator includes the multiple alternately connected original convolutional upsampling modules and the original spatial adaptive denormalization modules.

[0023] Optionally, the updating the model parameters of the encoding and decoding model based on the high-definition image and the second image includes:

[0024] Determine a first probability that the high-definition image is a restored image through the discriminator, and determine a second probability that the second image is a restored image through the discriminator;

[0025] Update the model parameters of the discriminator according to the first probability and the second probability;

[0026] Update the model parameters of the generator according to the high-definition image, the second image, and the second probability;

[0027] Update the model parameters of the original encoding network according to the high-definition image and the second image.

[0028] Optionally, the updating the model parameters of the generator according to the high-definition image, the second image, and the second probability includes:

[0029] Calculate a first loss of the discriminator according to the second probability;

[0030] Calculate a second loss of the generator according to the mean square error between the high-definition image and the second image;

[0031] Calculate a third loss of the generator according to the high-definition image, the second image, and the computer vision group model;

[0032] Update the model parameters of the generator according to the first loss, the second loss, and the third loss.

[0033] According to a second aspect of the embodiments of the present disclosure, there is provided an image restoration apparatus, including:

[0034] A first acquisition module, configured to acquire an image to be repaired;

[0035] A first repair module, configured to input the image to be repaired acquired by the first acquisition module into a repair model to obtain a repaired image, where the repair model includes a target encoding network and a target decoding network, the target decoding network includes a plurality of alternately connected target convolutional upsampling modules and target spatial adaptive denormalization modules, the target encoding network is used to perform feature extraction on the image to be repaired to obtain a first feature map, each target convolutional upsampling module is used to modulate the feature map from the previous module by using the first feature map to obtain a second feature map, each target spatial adaptive denormalization module is used to modulate the feature map from the previous module by using the image to be repaired to obtain a third feature map, and the feature map of the module previous to the first target convolutional upsampling module of the target decoding network is first random noise.

[0036] According to a third aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of the image repair method provided in the first aspect of the present disclosure are implemented.

[0037] According to a fourth aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0038] A memory, on which a computer program is stored;

[0039] A processor, configured to execute the computer program in the memory to implement the steps of the image repair method provided in the first aspect of the present disclosure.

[0040] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: obtaining an image to be repaired; inputting the image to be repaired into a repair model to obtain a repaired image, where the repair model includes a target encoding network and a target decoding network, the target decoding network includes a plurality of alternately connected target convolutional upsampling modules and target spatial adaptive denormalization modules, the target encoding network is used to extract features from the image to be repaired to obtain a first feature map, each target convolutional upsampling module is used to modulate the feature map from the previous module using the first feature map to obtain a second feature map, each target spatial adaptive denormalization module is used to modulate the feature map from the previous module using the image to be repaired to obtain a third feature map, and the feature map of the module before the first target convolutional upsampling module of the target decoding network is the first random noise. In this way, the target spatial adaptive denormalization module can use the image to be repaired to modulate and constrain the intermediate features obtained by the target convolutional upsampling module in front of it, which can not only greatly reduce the influence of low-quality information such as noise, but also make the repaired image more similar to the image to be repaired, so that an image highly consistent with the image to be repaired can be obtained, ensuring the effect of image repair and making the repaired image more realistic and natural.

[0041] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure and, together with the specification, are used to explain the principles of the present disclosure.

[0043] Figure 1 is a flowchart of an image repair method shown according to an exemplary embodiment.

[0044] Figure 2 is a schematic structural diagram of a repair model shown according to an exemplary embodiment.

[0045] Figure 3 is a schematic structural diagram of a target spatial adaptive denormalization module shown according to an exemplary embodiment.

[0046] Figure 4 is a schematic structural diagram of a target spatial adaptive denormalization module shown according to another exemplary embodiment.

[0047] Figure 5 is a flowchart of a repair model training method shown according to an exemplary embodiment.

[0048] Figure 6 is a block diagram of an image repair device shown according to an exemplary embodiment.

[0049] Figure 7 is a block diagram of an electronic device shown according to an exemplary embodiment.

[0050] Figure 8 is a block diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0051] Here, the exemplary embodiments will be described in detail, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0052] It should be noted that all actions of obtaining signals, information, or data in this application are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where it is located and obtaining authorization from the owner of the corresponding device.

[0053] Figure 1 is a flowchart of an image restoration method shown according to an exemplary embodiment. Among them, the image restoration method can be applied to a server or terminals such as a smart phone, a tablet computer, and a smart wearable device. The present disclosure does not make specific limitations. As Figure 1 shown, the above image restoration method may include the following S101 and S102.

[0054] In S101, obtain an image to be restored.

[0055] In the present disclosure, the image to be restored may be, for example, a face image, a cartoon animal image, a landscape image, etc. The present disclosure does not make specific limitations.

[0056] In S102, input the image to be restored into a restoration model to obtain a restored image.

[0057] In the present disclosure, as Figure 2 shown, the restoration model may include a target encoding network and a target decoding network. Among them, the target encoding network may include a plurality of cascaded target convolutional downsampling modules (not shown in the figure). The target decoding network may include a plurality of alternately connected target convolutional upsampling modules and target spatially-adaptive denormalization modules (Spatially-Adaptive Denormalization, SPADE), that is, target SPADE. Among them, Figure 2Taking the target decoding network including M alternately connected target convolutional upsampling modules and target spatial adaptive denormalization modules as an example for illustration, that is, the target decoding network includes target convolutional upsampling module A1, target spatial adaptive denormalization module B1, target convolutional upsampling module A2, target spatial adaptive denormalization module B2,... target convolutional upsampling module A M , target spatial adaptive denormalization module B M , where M ≥ 2.

[0058] Specifically, the target encoding network is used to extract features from the image to be repaired to obtain a first feature map. Among them, the target encoding network can extract image features such as color, texture, shape, and spatial relationship from the image to be repaired to obtain the first feature map.

[0059] Each target convolutional upsampling module is used to modulate the feature map from the previous module using the first feature map to obtain a second feature map. Specifically, the feature map of the previous module of the first target convolutional upsampling module of the target decoding network (that is, the first module among the multiple alternately connected target convolutional upsampling modules and target spatial adaptive denormalization modules in the target decoding network, for example, Figure 2 the target convolutional upsampling module A1 shown in ) is the first random noise, where there is no previous module for the first target convolutional sampling module. Among them, the first random noise is a random noise with a preset dimension or preset pixel size, which is usually a random variable subject to a Gaussian distribution and is one of the input quantities of the repair model.

[0060] Each target spatial adaptive denormalization module is used to modulate the feature map from the previous module using the image to be repaired to obtain a third feature map.

[0061] Exemplarily, as Figure 2 shown, the target convolutional upsampling module A1 is used to modulate the first random noise using the first feature map to obtain a second feature map T1; the target spatial adaptive denormalization module B1 is used to modulate the feature map (that is, the second feature map T1) from the previous module (that is, the target convolutional upsampling module A1) using the image to be repaired to obtain a third feature map R1; the target convolutional upsampling module A2 is used to modulate the feature map (that is, the third feature map R1) from the previous module (that is, the target spatial adaptive denormalization module B1) using the first feature map to obtain a second feature map T2; the target spatial adaptive denormalization module B2 is used to modulate the feature map (that is, the second feature map T2) from the previous module (that is, the target convolutional upsampling module A2) using the image to be repaired to obtain a third feature map R2;...; the target convolutional upsampling module A M is used to modulate the feature map from the previous module (that is, the target spatial adaptive denormalization module BM-1 )'s feature map (i.e., the third feature map R M-1 ) is modulated to obtain the second feature map T M ; The target spatial adaptive de-normalization module B M is used to utilize the image to be restored to modulate the feature map (i.e., the second feature map T M ) from the previous module (i.e., the target convolutional upsampling module A M ) to obtain the third feature map R M , that is, the restored image.

[0062] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: obtaining an image to be restored; inputting the image to be restored into a restoration model to obtain a restored image, where the restoration model includes a target encoding network and a target decoding network, the target decoding network includes a plurality of alternately connected target convolutional upsampling modules and target spatial adaptive de-normalization modules, the target encoding network is used to extract features from the image to be restored to obtain a first feature map, each target convolutional upsampling module is used to modulate the feature map from the previous module by using the first feature map to obtain a second feature map, each target spatial adaptive de-normalization module is used to modulate the feature map from the previous module by using the image to be restored to obtain a third feature map, and the feature map of the previous module of the first target convolutional upsampling module of the target decoding network is the first random noise. In this way, the target spatial adaptive de-normalization module can modulate and constrain the intermediate features obtained by the target convolutional upsampling module in front of it by using the image to be restored, which can not only greatly reduce the influence of low-quality information such as noise, but also make the restored image more similar to the image to be restored, so that an image highly consistent with the image to be restored can be obtained, ensuring the effect of image restoration and making the restored image more realistic and natural.

[0063] The specific structure of the above target spatial adaptive de-normalization module will be described in detail below. Specifically, as Figure 3 shown, the target spatial adaptive de-normalization module may include a target size adjustment module, a first target convolutional layer, a second target convolutional layer, a target modulation module connected in sequence, and a third target convolutional layer connected between the first target convolutional layer and the target modulation module. Among them, one end of the target modulation module is connected to the previous module of the target spatial adaptive de-normalization module where it is located, and the other end of the target modulation module is connected to the next module of the target spatial adaptive de-normalization module where it is located. Exemplarily, one end of the target modulation module in the target spatial adaptive de-normalization module B1 is connected to the previous module (i.e., the target convolutional upsampling module A1) of the target spatial adaptive de-normalization module B1, and the other end is connected to the next module (i.e., the target convolutional upsampling module A2) of the target spatial adaptive de-normalization module B1 (the connection relationship is not shown in the figure).

[0064] Specifically, the target size adjustment module is used to adjust the image to be repaired to a preset size to obtain a first image. Among them, the sizes of the second feature maps obtained by the above-mentioned target convolutional upsampling module are also the preset size. That is, the target size adjustment module is used to adjust the size of the image to be repaired to be the same as the sizes of the second feature maps output by the target convolutional upsampling module.

[0065] The first target convolutional layer is used to perform feature extraction on the first image to obtain a fourth feature map.

[0066] The second target convolutional layer is used to generate a variance feature map (i.e., multiplicative coefficient) according to the fourth feature map.

[0067] The third target convolutional layer is used to generate a mean feature map (i.e., additive coefficient) according to the fourth feature map.

[0068] The target modulation module is used to modulate the feature map from the previous module according to the variance feature map and the mean feature map to obtain a third feature map. Specifically, the target modulation module can multiply the feature map from the previous module by the variance feature map to perform variance denormalization on the feature map from the previous module to obtain an eighth feature map; then, the mean feature map can be added to the eighth feature map to perform mean denormalization on the feature map from the previous module to obtain a third feature map.

[0069] Extract features from the image to be repaired through multiple convolutions, and use the extracted features as multiplicative coefficients and additive coefficients to adjust the intermediate feature distribution generated by the target convolutional upsampling module. That is, use the high-dimensional features of the image to be repaired to constrain the intermediate features generated by the target convolutional upsampling module, which can not only greatly reduce the influence of low-quality information such as noise, but also make the repaired image more similar to the image to be repaired, so that an image highly consistent with the image to be repaired can be obtained, ensuring the effect of image repair and making the repaired image more realistic and natural.

[0070] In addition, in order to make the repair model easier to learn the corresponding mapping rules to further improve the image repair effect, before the above-mentioned target modulation module modulates the feature map from the previous module according to the variance feature map and the mean feature map, the feature map from the previous module can be normalized first to constrain the data distribution of the feature map from the previous module. Then, the above-mentioned target modulation module modulates the normalized feature map from the previous module according to the variance feature map and the mean feature map. Specifically, as Figure 4 shown, the above-mentioned target spatial adaptive denormalization module can also include a target normalization layer connected between the previous module of the target spatial adaptive denormalization module and the target modulation module.

[0071] Among them, the target normalization layer is used to normalize the feature map from the previous module; the above-mentioned target modulation module is used to modulate the feature map obtained after normalization from the previous module according to the variance feature map and the mean feature map to obtain the third feature map.

[0072] The following will elaborate on the specific training method of the above-mentioned restoration model. Specifically, it can be obtained through Figure 5 S501 - S505 shown in

[0073] In S501, training data is obtained. The training data includes high-definition images and low-quality images corresponding to the high-definition images.

[0074] In the present disclosure, when the above-mentioned restoration model is used for the restoration of face images, high-definition images can be obtained from a high-definition face dataset (Flickr Faces High-Quality, FFHQ). Additionally, the low-quality images corresponding to the high-definition images can be obtained by degrading the high-definition images.

[0075] In S502, the low-quality image is restored through an encoder-decoder model to obtain a second image.

[0076] In the present disclosure, the encoder-decoder model includes an original encoding network and an original decoding network. Among them, the original encoding network may include multiple cascaded original convolutional downsampling modules. The original decoding network includes multiple alternately connected original convolutional upsampling modules and original spatial adaptive denormalization modules.

[0077] The original encoding network is used to extract features from the low-quality image to obtain a fifth feature map.

[0078] Each original convolutional upsampling module is used to modulate the feature map from the previous module using the fifth feature map to obtain a sixth feature map. Specifically, the feature map of the previous module of the first original convolutional upsampling module of the original decoding network (i.e., the first module among the multiple alternately connected original convolutional upsampling modules and original spatial adaptive denormalization modules in the original decoding network) is the second random noise. Among them, there is no previous module for the first original convolutional sampling module. The second random noise is a random noise with a preset dimension or preset pixel size, which is usually a random variable subject to a Gaussian distribution and is one of the input quantities of the encoder-decoder model.

[0079] Each original spatial adaptive denormalization module is used to modulate the feature map from the previous module using the low-quality image to obtain a seventh feature map.

[0080] In S503, based on the high-definition image and the second image, the model parameters of the encoder-decoder model are updated.

[0081] In S504, it is determined whether the training cutoff condition is satisfied.

[0082] In the present disclosure, the training cutoff condition may be that the number of training times reaches a preset number threshold, or the loss of the encoding and decoding model is less than a preset loss threshold.

[0083] If the training cutoff condition is not satisfied, return to the above S501 and continue to execute until the training cutoff condition is satisfied. If the training cutoff condition is satisfied, execute the following S505.

[0084] In S505, a repair model is obtained.

[0085] It should be noted that the training data obtained by returning to the above S501 each time is new training data, that is, a new high-definition image and a low-quality image corresponding to the new high-definition image.

[0086] The following will detail the specific implementation manner of degrading the high-definition image to obtain a low-quality image corresponding to the high-definition image. Specifically, the high-definition image can be degraded in various ways to obtain the corresponding low-quality image.

[0087] In one implementation manner, a downsampling interpolation strategy (i.e., a downsampling and then interpolation and magnification strategy) can be adopted to degrade the high-definition image to obtain the corresponding low-quality image. Among them, the downsampling interpolation strategy mainly refers to first downsampling the high-definition image, and the degree of downsampling (for example, the number of downsampling times) can be randomly selected within the downsampling range of [2, 5]. Then, after downsampling, the image can be magnified to the original size. Among them, in this downsampling interpolation strategy, the magnification method can be randomly selected from the bilinear interpolation method, the bicubic interpolation method, and the nearest neighbor interpolation method.

[0088] In another implementation manner, a Gaussian noise strategy can be adopted to degrade the high-definition image to obtain the corresponding low-quality image. Among them, the Gaussian noise strategy means adding Gaussian noise to the high-definition image, and the noise mean of the added Gaussian noise can be randomly selected within the noise range of [1, 20].

[0089] In yet another implementation manner, in order to simulate a more realistic low-quality image, the high-definition image can be degraded in the manner shown in the following equation (1) to obtain the corresponding low-quality image:

[0090]

[0091] Among them, I h represents the high-definition image; I ddenotes the low-quality image corresponding to the high-definition image; gray denotes grayscale conversion of the high-definition image with a probability of 0.3; k denotes a blur convolution kernel, and with a probability of 0.5, the grayscale-converted image is blurred, and the blurring method is randomly selected from Gaussian blur, mean blur, median blur, and motion blur; ↓r denotes downsampling the blurred image with a sampling rate of r, where the value range of r is [2, 8], then adding Gaussian-distributed noise n with a probability of 0.2, and then performing JEPG compression on the image after adding noise n with a probability of 0.7, and the compression level is randomly selected within the range of [20, 80]; ↑r denotes upsampling the compressed image to the same size as the above high-definition image at a sampling rate of r, thereby obtaining the low-quality image corresponding to the above high-definition image.

[0092] In addition, in one implementation, the above original decoding network can be a generative adversarial network including a generator and a discriminator. Among them, the generator includes a plurality of alternately connected original convolutional upsampling modules and original spatial adaptive denormalization modules. Exemplarily, the above generative adversarial network can be Stylegan2.

[0093] Next, the specific structure of the above original spatial adaptive denormalization module will be described in detail. Specifically, the original spatial adaptive denormalization module can include an original size adjustment module, a first original convolutional layer, a second original convolutional layer, an original modulation module, and a third original convolutional layer connected between the first original convolutional layer and the original modulation module, which are connected in sequence. Among them, one end of the original modulation module is connected to the previous module of the original spatial adaptive denormalization module where it is located, and the other end of the original modulation module is connected to the next module of the original spatial adaptive denormalization module where it is located.

[0094] Specifically, the original size adjustment module is used to adjust the low-quality image to a preset size to obtain a third image. Among them, the sizes of the second feature maps obtained by the above original convolutional upsampling modules are also the preset size, that is, the original size adjustment module is used to adjust the size of the low-quality image to be the same as the sizes of the second feature maps output by the original convolutional upsampling modules.

[0095] The first original convolutional layer is used to extract features from the third image to obtain a ninth feature map.

[0096] The second original convolutional layer is used to generate an original variance feature map according to the ninth feature map.

[0097] The third original convolutional layer is used to generate an original mean feature map according to the ninth feature map.

[0098] The original modulation module is used to modulate the feature map from the previous module according to the original variance feature map and the original mean feature map to obtain the tenth feature map. Specifically, the original modulation module can multiply the feature map from the previous module by the original variance feature map to perform variance denormalization on the feature map from the previous module, obtaining the eleventh feature map; then, the original mean feature map can be added to the eleventh feature map to perform mean denormalization on the feature map from the previous module, obtaining the seventh feature map.

[0099] In addition, to make the encoding and decoding model easier to learn the corresponding mapping rules and further improve the image restoration effect, before the above original modulation module modulates the feature map from the previous module according to the original variance feature map and the original mean feature map, the feature map from the previous module can be normalized first to constrain the data distribution of the feature map from the previous module. Then, the above original modulation module modulates the normalized feature map from the previous module according to the original variance feature map and the original mean feature map. Specifically, the above original spatial adaptive denormalization module may further include an original normalization layer connected between the previous module of the original spatial adaptive denormalization module and the original modulation module.

[0100] Among them, the original normalization layer is used to perform normalization processing on the feature map from the previous module; the above original modulation module is used to modulate the normalized feature map from the previous module according to the original variance feature map and the original mean feature map to obtain the seventh feature map.

[0101] In the case where the above original decoding network is a generative adversarial network including a generator and a discriminator, S503 may include the following steps (1) to (4).

[0102] (1) Determine the first probability that the high-definition image is a restored image through the discriminator, and determine the second probability that the second image is a restored image through the discriminator.

[0103] (2) Update the model parameters of the discriminator according to the first probability and the second probability.

[0104] (3) Update the model parameters of the generator according to the high-definition image, the second image, and the second probability.

[0105] (4) Update the model parameters of the original encoding network according to the high-definition image and the second image.

[0106] The following details the specific implementation of updating the model parameters of the discriminator according to the first probability and the second probability in the above step (2). Specifically, it can be implemented through the following steps (21) and (22):

[0107] Calculate the fourth loss of the discriminator according to the first probability and the second probability.

[0108] Exemplarily, the fourth loss of the discriminator can be calculated according to the first probability and the second probability through the following equation (2):

[0109]

[0110] where L d represents the fourth loss; I h1 represents the high-definition image; I r1 represents the second image; D(I h1 ) is the first probability; D(I r1 ) is the second probability.

[0111] (22) Update the model parameters of the discriminator according to the fourth loss.

[0112] Specifically, the model parameters of the discriminator can be updated according to the fourth loss through the backpropagation algorithm.

[0113] The following details the specific implementation of updating the model parameters of the generator according to the high-definition image, the second image, and the second probability in step (3) above. Specifically, it can be implemented through the following steps (31) to (34):

[0114] (31) Calculate the first loss of the discriminator according to the second probability.

[0115] Exemplarily, the first loss of the discriminator can be calculated according to the second probability through the following equation (3):

[0116]

[0117] where L gan represents the first loss.

[0118] (32) Calculate the second loss of the generator according to the mean square error between the high-definition image and the second image.

[0119] Exemplarily, the second loss of the generator can be calculated according to the mean square error between the high-definition image and the second image through the following equation (4):

[0120] L2 = ||I h1 - I r1 || 2 (4)

[0121] where L2 represents the second loss.

[0122] (33) Calculate the third loss of the generator based on the high-definition image, the second image, and the computer vision group model.

[0123] Specifically, the high-definition image can be input into a Visual Geometry Group (VGG) model. The VGG model includes multiple convolutional layers. After that, extract the first features of the target convolutional layer of VGG, where the target convolutional layer is part or all of the convolutional layers in the VGG model; input the second image into the VGG model, and then extract the second features of the target convolutional layer (for example, the first convolutional layer, the third convolutional layer, and the fifth convolutional layer) of VGG; then, calculate the third loss of the generator according to the first features of the target convolutional layer and the second features of the target convolutional layer.

[0124] Exemplarily, the third loss of the generator can be calculated by the following equation (5) according to the first features of the target convolutional layer and the second features of the target convolutional layer:

[0125]

[0126] where L content represents the third loss; VGG(I h1 ) i represents the first features of the i-th target convolutional layer of the VGG model, i = 1, 2, …, K, and K represents the total number of target convolutional layers included in the VGG model; VGG(I r1 ) i is the second features of the i-th target convolutional layer of the VGG model; w i is the weight corresponding to the absolute value error between the first features VGG(I h1 ) i of the i-th target convolutional layer of the VGG model and the second features VGG(I r1 ) i of the i-th target convolutional layer of the VGG model, and it can be a value preset by the user.

[0127] (34) Update the model parameters of the generator according to the first loss, the second loss, and the third loss.

[0128] Specifically, the weighted sum of the first loss, the second loss, and the third loss can be used as the total loss of the generator, and according to this total loss, the model parameters of the generator are updated through the backpropagation algorithm.

[0129] In the above-described embodiment, when updating the parameters of the generator model, not only the first loss of the discriminator and the loss at the image level of the generator (i.e., the second loss) are referred to, but also the loss at the feature level of the generator (i.e., the third loss) is referred to, so that the difference between the second image generated by the generator and the high-definition image can be fully analyzed, thereby improving the accuracy of the total loss of the generator, and further improving the accuracy and training efficiency of the model.

[0130] The following will detail the specific implementation manner of updating the model parameters of the original encoding network according to the high-definition image and the second image in step (4) above. Specifically, it can be implemented through the following steps (41) and (42).

[0131] (41) Calculate the fifth loss of the original encoding network according to the high-definition image, the second image, and the computer vision group model.

[0132] In the present disclosure, the fifth loss of the original encoding network is equal to the third loss of the above-mentioned generator.

[0133] (42) Update the model parameters of the original encoding network according to the fifth loss of the original encoding network.

[0134] Specifically, the model parameters of the original encoding network can be updated by the backpropagation algorithm according to the fifth loss of the original encoding network.

[0135] Figure 6 is a block diagram of an image inpainting device shown according to an exemplary embodiment. As Figure 6 shown, the device 600 includes:

[0136] A first acquisition module 601, configured to acquire an image to be inpainted;

[0137] A first inpainting module 602, configured to input the image to be inpainted acquired by the first acquisition module 602 into an inpainting model to obtain an inpainted image, where the inpainting model includes a target encoding network and a target decoding network, the target decoding network includes a plurality of alternately connected target convolutional upsampling modules and target spatial adaptive denormalization modules, the target encoding network is configured to perform feature extraction on the image to be inpainted to obtain a first feature map, each target convolutional upsampling module is configured to modulate the feature map from the previous module by using the first feature map to obtain a second feature map, each target spatial adaptive denormalization module is configured to modulate the feature map from the previous module by using the image to be inpainted to obtain a third feature map, and the feature map of the module before the first target convolutional upsampling module of the target decoding network is first random noise.

[0138] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: obtaining an image to be repaired; inputting the image to be repaired into a repair model to obtain a repaired image, where the repair model includes a target encoding network and a target decoding network, the target decoding network includes a plurality of alternately connected target convolutional upsampling modules and target spatial adaptive denormalization modules, the target encoding network is used to extract features from the image to be repaired to obtain a first feature map, each target convolutional upsampling module is used to modulate the feature map from the previous module by using the first feature map to obtain a second feature map, each target spatial adaptive denormalization module is used to modulate the feature map from the previous module by using the image to be repaired to obtain a third feature map, and the feature map of the previous module of the first target convolutional upsampling module in the target decoding network is the first random noise. In this way, the target spatial adaptive denormalization module can use the image to be repaired to modulate and constrain the intermediate features obtained by the target convolutional upsampling module in front of it, which can not only greatly reduce the influence of low-quality information such as noise, but also make the repaired image more similar to the image to be repaired, so that an image highly consistent with the image to be repaired can be obtained, ensuring the effect of image repair and making the repaired image more realistic and natural.

[0139] Optionally, the target spatial adaptive denormalization module includes a target size adjustment module, a first target convolutional layer, a second target convolutional layer, a target modulation module connected in sequence, and a third target convolutional layer connected between the first target convolutional layer and the target modulation module, where one end of the target modulation module is connected to the previous module of the target spatial adaptive denormalization module where it is located, and the other end of the target modulation module is connected to the next module of the target spatial adaptive denormalization module where it is located;

[0140] Among them, the target size adjustment module is used to adjust the image to be repaired to a preset size to obtain a first image;

[0141] The first target convolutional layer is used to extract features from the first image to obtain a fourth feature map;

[0142] The second target convolutional layer is used to generate a variance feature map according to the fourth feature map;

[0143] The third target convolutional layer is used to generate a mean feature map according to the fourth feature map;

[0144] The target modulation module is used to modulate the feature map from the previous module according to the variance feature map and the mean feature map to obtain a third feature map.

[0145] Optionally, the target spatial adaptive denormalization module further includes a target normalization layer connected between the previous module of the target spatial adaptive denormalization module and the target modulation module;

[0146] The target normalization layer is configured to perform normalization processing on the feature map from the previous module;

[0147] The target modulation module is configured to modulate the feature map obtained after normalization from the previous module according to the variance feature map and the mean feature map to obtain a third feature map.

[0148] Optionally, the inpainting model is trained by an inpainting model training device, where the inpainting model training device may include:

[0149] A second acquisition module configured to acquire training data, where the training data includes a high-definition image and a low-quality image corresponding to the high-definition image;

[0150] A second inpainting module configured to perform image inpainting on the low-quality image through an encoder-decoder model to obtain a second image, where the encoder-decoder model includes an original encoding network and an original decoding network, the original decoding network includes a plurality of alternately connected original convolutional upsampling modules and original spatial adaptive denormalization modules, the original encoding network is configured to perform feature extraction on the low-quality image to obtain a fifth feature map, each original convolutional upsampling module is configured to modulate the feature map from the previous module using the fifth feature map to obtain a sixth feature map, each original spatial adaptive denormalization module is configured to modulate the feature map from the previous module using the low-quality image to obtain a seventh feature map, and the feature map of the module previous to the first original convolutional upsampling module of the original decoding network is second random noise;

[0151] An update module configured to update the model parameters of the encoder-decoder model based on the high-definition image and the second image;

[0152] A trigger module configured to:

[0153] In response to not satisfying the training cut-off condition, trigger the second acquisition module to acquire training data;

[0154] In response to satisfying the training cut-off condition, obtain the inpainting model.

[0155] Optionally, the original decoding network is a generative adversarial network including a generator and a discriminator, where the generator includes the plurality of alternately connected original convolutional upsampling modules and original spatial adaptive denormalization modules.

[0156] Optionally, the update module includes:

[0157] A determination sub-module, configured to determine a first probability that the high-definition image is a repaired image through the discriminator, and determine a second probability that the second image is a repaired image through the discriminator;

[0158] A first update sub-module, configured to update the model parameters of the discriminator according to the first probability and the second probability;

[0159] A second update sub-module, configured to update the model parameters of the generator according to the high-definition image, the second image, and the second probability;

[0160] A third update sub-module, configured to update the model parameters of the original encoding network according to the high-definition image and the second image.

[0161] Optionally, the second update sub-module includes:

[0162] A first calculation sub-module, configured to calculate a first loss of the discriminator according to the second probability;

[0163] A second calculation sub-module, configured to calculate a second loss of the generator according to the mean square error between the high-definition image and the second image;

[0164] A third calculation sub-module, configured to calculate a third loss of the generator according to the high-definition image, the second image, and the computer vision group model;

[0165] A fourth update sub-module, configured to update the model parameters of the generator according to the first loss, the second loss, and the third loss.

[0166] In addition, it should be noted that the above-mentioned repaired model training device may be independent of the above-mentioned image repair device 600, or may be integrated in the above-mentioned image repair device 600. The present disclosure does not make specific limitations.

[0167] Regarding the device in the above-mentioned embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0168] The present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of the above-mentioned image repair method provided by the present disclosure are implemented.

[0169] Figure 7FIG. 0 is a block diagram of an electronic device 800 shown in accordance with an exemplary embodiment. For example, device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0170] Referring Figure 7 , device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0171] The processing component 802 generally controls the overall operation of device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-described image restoration method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0172] The memory 804 is configured to store various types of data to support the operation of device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0173] The power component 806 provides power to the various components of device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for device 800.

[0174] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of a touch or swipe action but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0175] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0176] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0177] The sensor component 814 includes one or more sensors for providing an assessment of the various aspects of the state of the device 800. For example, the sensor component 814 can detect the on / off state of the device 800, the relative positioning of components, such as the display and the keypad of the device 800. The sensor component 814 can also detect a change in the position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor component 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 814 can also include a light sensor, such as a CMOS or a CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0178] The communication component 816 is configured to facilitate communication, either wired or wirelessly, between the device 800 and other devices. The device 800 may access a wireless network based on communication standards, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0179] In an exemplary embodiment, the device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above-described image restoration method.

[0180] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as the memory 804 including instructions, and the above instructions can be executed by the processor 820 of the device 800 to complete the above-described image restoration method. For example, the non-transitory computer-readable storage medium may be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0181] In another exemplary embodiment, a computer program product is also provided, which includes a computer program capable of being executed by a programmable device, and the computer program has a code portion for performing the above-described image restoration method when executed by the programmable device.

[0182] Figure 8 is a block diagram of an electronic device 1900 shown according to an exemplary embodiment. For example, the device 1900 may be provided as a server. Referring to Figure 8 , the device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by the memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above-described image restoration method.

[0183] The apparatus 1900 may also include a power supply component 1926 configured to perform power management of the apparatus 1900, a wired or wireless network interface 1950 configured to connect the apparatus 1900 to a network, and an input / output (I / O) interface 1958. The apparatus 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0184] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and examples are only illustrative, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0185] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. An image restoration method, characterized in that, Including: Obtain the image to be repaired; Input the image to be repaired into a repair model to obtain a repaired image, where the repair model includes a target encoding network and a target decoding network, and the target decoding network includes a plurality of alternately connected target convolutional upsampling modules and target spatial adaptive denormalization modules. The target encoding network is used to extract features from the image to be repaired to obtain a first feature map. Each target convolutional upsampling module is used to modulate the feature map from the previous module using the first feature map to obtain a second feature map. Each target spatial adaptive denormalization module is used to modulate the feature map from the previous module using the image to be repaired to obtain a third feature map. The feature map of the module previous to the first target convolutional upsampling module of the target decoding network is the first random noise.

2. The method according to claim 1, characterized in that, The target spatial adaptive denormalization module includes a target size adjustment module, a first target convolutional layer, a second target convolutional layer, a target modulation module connected in sequence, and a third target convolutional layer connected between the first target convolutional layer and the target modulation module. One end of the target modulation module is connected to the previous module of the target spatial adaptive denormalization module where it is located, and the other end of the target modulation module is connected to the next module of the target spatial adaptive denormalization module where it is located; Among them, the target size adjustment module is used to adjust the image to be repaired to a preset size to obtain a first image; The first target convolutional layer is used to extract features from the first image to obtain a fourth feature map; The second target convolutional layer is used to generate a variance feature map according to the fourth feature map; The third target convolutional layer is used to generate a mean feature map according to the fourth feature map; The target modulation module is used to modulate the feature map from the previous module according to the variance feature map and the mean feature map to obtain a third feature map.

3. The method according to claim 2, wherein The target spatial adaptive denormalization module further includes a target normalization layer connected between the previous module of the target spatial adaptive denormalization module and the target modulation module; The target normalization layer is used to perform normalization processing on the feature map from the previous module; The target modulation module is used to modulate the normalized feature map from the previous module according to the variance feature map and the mean feature map to obtain a third feature map.

4. The method according to claim 1, wherein The repair model is trained in the following way: Obtain training data, where the training data includes high-definition images and the corresponding low-quality images; The low-quality image is restored through an encoding and decoding model to obtain a second image. The encoding and decoding model includes an original encoding network and an original decoding network. The original decoding network includes a plurality of alternately connected original convolutional upsampling modules and original spatial adaptive denormalization modules. The original encoding network is used to extract features from the low-quality image to obtain a fifth feature map. Each original convolutional upsampling module is used to modulate the feature map from the previous module using the fifth feature map to obtain a sixth feature map. Each original spatial adaptive denormalization module is used to modulate the feature map from the previous module using the low-quality image to obtain a seventh feature map. The feature map of the module previous to the first original convolutional upsampling module of the original decoding network is the second random noise; Based on the high-definition image and the second image, update the model parameters of the encoding and decoding model; In response to not meeting the training cutoff condition, re-execute the steps of obtaining training data to the step of updating the model parameters of the encoding and decoding model based on the high-definition image and the second image; In response to meeting the training cutoff condition, obtain the restoration model.

5. The method according to claim 4, wherein The original decoding network is a generative adversarial network including a generator and a discriminator. The generator includes the plurality of alternately connected original convolutional upsampling modules and original spatial adaptive denormalization modules.

6. The method according to claim 5, characterized in that The updating of the model parameters of the encoding and decoding model based on the high-definition image and the second image includes: Determine the first probability that the high-definition image is a restored image through the discriminator, and determine the second probability that the second image is a restored image through the discriminator; Update the model parameters of the discriminator according to the first probability and the second probability; Update the model parameters of the generator according to the high-definition image, the second image, and the second probability; Update the model parameters of the original encoding network according to the high-definition image and the second image.

7. The method according to claim 6, wherein The updating of the model parameters of the generator according to the high-definition image, the second image, and the second probability includes: Calculate the first loss of the discriminator according to the second probability; Calculate the second loss of the generator according to the mean square error between the high-definition image and the second image; Calculate the third loss of the generator according to the high-definition image, the second image, and the computer vision group model; Update the model parameters of the generator according to the first loss, the second loss, and the third loss.

8. An image restoration device, characterized in that, Including: A first acquisition module configured to acquire an image to be restored; The first repair module is configured to input the image to be repaired acquired by the first acquisition module into a repair model to obtain a repaired image, wherein the repair model includes a target encoding network and a target decoding network, the target decoding network includes a plurality of alternately connected target convolutional upsampling modules and target spatial adaptive denormalization modules, the target encoding network is used to perform feature extraction on the image to be repaired to obtain a first feature map, each target convolutional upsampling module is used to modulate the feature map from the previous module by using the first feature map to obtain a second feature map, each target spatial adaptive denormalization module is used to modulate the feature map from the previous module by using the image to be repaired to obtain a third feature map, and the feature map of the module previous to the first target convolutional upsampling module of the target decoding network is first random noise.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instruction is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, Comprising: A memory having a computer program stored thereon; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.

Images