Image restoration method and server

Abstract

The invention discloses an image restoration method and a server. According to the image restoration method, the definition and the quality of a restored image are remarkably improved by optimizing the configuration of multiple convolutional layers and multiple transposed convolutional layers in a generation sub-model, including adjustment of the number of layers and the kernel size; and secondly, the structural design of the sub-model is generated, especially the connection relationship between layers, so that the detail expressive force and the overall visual effect of image restoration are further enhanced. Moreover, the discrimination sub-model improves the evaluation accuracy of the authenticity of the repaired image through the elaborate parameter setting of the multi-layer convolution layer and the logic regression layer, ensures that the repaired image is highly consistent with the original image in visual sense, and shows excellent generalization ability and robustness.

Description

Technical Field

[0001] The invention relates to an image repair method and also to a server for implementing the image repair method, belonging to the technical field of digital image processing. Background Art

[0002] Image restoration technology has been widely used in many fields. However, traditional image restoration methods have significant limitations. First, these methods are highly dependent on manual operation and expertise, making the restoration process time-consuming and subjective. Manual restoration relies on the experience and skills of the restorer, while computer-aided restoration automatically or semi-automatically completes the restoration work through specific algorithms and techniques. This manual dependence not only limits the efficiency of restoration, but also makes the restoration results lack consistency and repeatability to a certain extent. In addition, traditional image restoration methods have limited capabilities in handling complex textures and structural restoration, especially when the damaged area is large or the texture information is rich, the restored image often has obvious visual incoherence.

[0003] On the other hand, traditional image restoration methods lack generalization capabilities. They are usually designed for specific types of damage, so they perform poorly when dealing with new types of damage that have not been seen. This lack of flexibility limits the effectiveness of traditional image restoration methods in changing application scenarios. At the same time, cumbersome operations are also a challenge. Many traditional image restoration methods require users to manually mark damaged areas or provide additional guidance information, which not only increases the burden on users but also reduces processing efficiency. Lack of robustness is also a problem. Traditional image restoration methods have high requirements on the quality and damage type of the input image. Faced with different types and degrees of image damage, it may be necessary to adjust parameters or even replace different algorithms, which increases the complexity and inconvenience of use.

[0004] To overcome these limitations, modern image restoration methods, such as deep learning-driven methods, are constantly evolving. For example, in a recent study, Tseng et al. proposed a lightweight conditional generative adversarial network (published in British Machine Vision Conference 2017) for image restoration. It introduces conditional variables in the generation process. These conditions can be labels, partial data, noise vectors, or any other information that can guide the generation process. The generated output depends not only on random noise, but also on the given conditions, making the generation process more controllable and diverse. This method is particularly suitable for handling various types of image damage, including text, lines, graffiti, and random damage. However, although Tseng et al.'s image restoration model has made significant progress in many aspects, it still faces some challenges. For example, the performance of the model depends heavily on the quality and diversity of the training data. If the training data is biased or insufficient to cover all possible conditions, it cannot generalize well to new or unseen conditions. In addition, mode collapse is also a common problem, that is, the generator starts to generate very similar or repeated images instead of diverse outputs. Summary of the invention

[0005] The primary technical problem to be solved by the present invention is to provide an image restoration method.

[0006] Another technical problem to be solved by the present invention is to provide a server for implementing the image restoration method.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] According to a first aspect of an embodiment of the present invention, there is provided an image restoration method, comprising the following steps:

[0009] S1: Obtain the image to be repaired;

[0010] S2: Generate a restored image according to the image to be restored and the model parameters of the generated sub-model;

[0011] Among them, the generation sub-model includes multi-layer convolutional layers and multi-layer transposed convolutional layers, the model parameters of the generation sub-model are each kernel function corresponding to each layer of the multi-layer convolutional layers and each kernel function corresponding to each layer of the multi-layer transposed convolutional layers, the number of layers of the multi-layer convolutional layers and each kernel function corresponding to each layer of the multi-layer convolutional layers, and the number of layers of the multi-layer transposed convolutional layers and each kernel function corresponding to each layer of the multi-layer transposed convolutional layers are determined according to a preset image restoration clarity.

[0012] Preferably, in the generating sub-model, the multi-layer convolutional layer is n layers of convolutional layers connected in sequence, and the multi-layer transposed convolutional layer is n layers of transposed convolutional layers connected in sequence; wherein the last convolutional layer of the multi-layer convolutional layer is connected to the first transposed convolutional layer of the multi-layer transposed convolutional layer, and n is a positive integer.

[0013] Preferably, the image restoration method further comprises the following steps:

[0014] Get the complete image;

[0015] According to the complete image, the restored image and the discriminant sub-model, it is determined whether the complete image is a restored image, and the image to be restored is restored accordingly according to the determination result.

[0016] Preferably, the discriminant sub-model includes n convolutional layers and one logistic regression layer; wherein the n convolutional layers are connected in sequence, and the last convolutional layer is connected to one logistic regression layer.

[0017] Preferably, the training steps of the generating sub-model and the discriminating sub-model are as follows:

[0018] (1) Initialize the parameters of the generative sub-model and the discriminative sub-model;

[0019] (2) Input the damaged image into the generative sub-model, output the repaired image, and iteratively update the parameters of the generative sub-model according to the first objective function until the first objective function converges; wherein the first objective function is |IM-G(IM part )|, IM represents the complete image, IM part represents the damaged image obtained after cropping of IM, G() represents the repaired image obtained after generating the sub-model, and the Euclidean distance between the pixels of the complete image and the repaired image is calculated;

[0020] (3) Input the damaged image into the generative sub-model to obtain a repaired image, input the repaired image and the complete image into the discriminative sub-model, and update the discriminative sub-model parameters according to the second objective function; wherein the second objective function is:

[0021] C=0.999*|IM-G(IM part )|+0.001*E[log(D(IM))+log(1-D(G(IM) part ))]

[0022] The second objective function consists of two parts: one is the first objective function, which is used to measure the similarity between the restored image and the target image; the other is the adversarial loss function, which is used to measure the authenticity of the restored image. partrepresents the damaged image obtained after cropping of IM, G() represents the repaired image obtained after the generative sub-model, D() represents the probability returned by the discriminative sub-model, and E[] represents the mathematical expectation;

[0023] (4) inputting the damaged image into the generative sub-model to obtain a repaired image, inputting the repaired image and the complete image into the discriminative sub-model, and updating the parameters of the generative sub-model according to the second objective function;

[0024] (5) Repeat steps (3) to (4) until the second objective function converges.

[0025] Preferably, the damaged image is obtained by the following steps:

[0026] (1) Screen the sizes of images in the training data and retain images whose sizes meet the threshold requirements;

[0027] (2) Screening the clarity of images in the training data and retaining images whose clarity meets the threshold requirements;

[0028] (3) Adjust the size of the retained image to meet the unified standard;

[0029] (4) Operating the image of standard size to generate the damaged image.

[0030] Preferably, the calculation formula of the image clarity is:

[0031]

[0032] Among them, θ(IM) represents the image clarity, IM grey Represents the grayscale image of the image, L represents the Laplace operator, Represents the convolution operation, and var() represents the variance value.

[0033] Preferably, the step S2 includes the following sub-steps:

[0034] (1) Select the repair type according to the type of image to be repaired and the type of damage;

[0035] (2) setting the pixel values ​​of the damaged area of ​​the image to be repaired to black;

[0036] (3) Adjust the size of the image to be repaired to a standard size;

[0037] (4) Inputting the image to be repaired into the generation sub-model to obtain a repaired image.

[0038] According to a second aspect of an embodiment of the present invention, a server for implementing the above-mentioned image restoration method is provided, comprising an acquisition module and a generation module; wherein:

[0039] The acquisition module is used to acquire the image to be restored;

[0040] The generating module is used to generate a restored image according to the image to be restored and the model parameters of the generating sub-model; wherein the generating sub-model includes multi-layer convolutional layers and multi-layer transposed convolutional layers, the model parameters of the generating sub-model are each kernel function corresponding to each layer of the multi-layer convolutional layers and each kernel function corresponding to each layer of the multi-layer transposed convolutional layers, the number of layers of the multi-layer convolutional layers and each kernel function corresponding to each layer of the multi-layer convolutional layers, and the number of layers of the multi-layer transposed convolutional layers and each kernel function corresponding to each layer of the multi-layer transposed convolutional layers are determined according to a preset image restoration clarity.

[0041] Compared with the prior art, the image restoration method provided by the embodiment of the present invention achieves significant improvement over the traditional image restoration technology through deep neural network technology, especially generative adversarial network (GAN). The image restoration method can efficiently handle a variety of image damage problems including large-scale damage, while maintaining the consistency of the repaired area with the original image in texture and structure. Through the automated end-to-end restoration technology, the present invention greatly reduces human intervention, reduces labor costs, and improves the efficiency of the restoration process. The optimized configuration of the multi-layer convolutional layer and the multi-layer transposed convolutional layer in the generative sub-model ensures that the restoration image is significantly improved in clarity and quality. In addition, the discriminator in the adversarial training mechanism further improves the authenticity of the restored image, making the restoration effect difficult to distinguish from the original image. The image restoration method provided by the embodiment of the present invention also performs well in generalization ability, can adapt to different types and degrees of image damage, and shows high robustness. Even in the face of more noise or more serious damage, it can maintain a stable restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a process flow of an image restoration method provided by a first embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the structure of generating a sub-model in the first embodiment of the present invention;

[0044] Figure 3 It is a schematic diagram of the structure of the discriminant sub-model in the first embodiment of the present invention;

[0045] Figure 4 This is a training flow chart of a GAN model in the first embodiment of the present invention;

[0046] Figure 5 This is a diagram showing the restoration effect of a face image in the first embodiment of the present invention;

[0047] Figure 6 A schematic diagram of the structure of an image restoration server provided by a second embodiment of the present invention;

[0048] Figure 7 A schematic diagram of the physical structure of an electronic device provided in the third embodiment of the present invention. DETAILED DESCRIPTION

[0049] The technical content of the present invention is described in detail below in conjunction with the drawings and specific embodiments. Here, the exemplary embodiments will be described in detail, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the attached invention content.

[0050] First embodiment

[0051] like Figure 1 As shown, the first embodiment of the present invention first provides an image restoration method, which at least includes the following steps:

[0052] S1: Obtain the image to be repaired.

[0053] Specifically, after the server receives the image that needs to be repaired, it can use the Generative Adversarial Networks (GAN) model for processing. In this process, the GAN model is deployed on the server and is responsible for performing the image repair task. Specifically, the GAN model contains two key components: the generator and the discriminator. The task of the generator is to create realistic images, while the discriminator evaluates the generated images and the actual images and tries to distinguish them. Through this adversarial training, the generator learns how to produce high-quality images that are difficult for the discriminator to recognize. This means that the image to be repaired can be processed by the GAN model to generate an image that fills the missing part, thereby completing the repair work. This GAN model deployed in the server can efficiently handle a large number of image repair requests and provide fast and efficient image repair services.

[0054] S2: Generate a restored image according to the image to be restored and the model parameters of the generated sub-model; wherein the generated sub-model includes multi-layer convolutional layers and multi-layer transposed convolutional layers, the model parameters of the generated sub-model are each kernel function corresponding to each layer of the multi-layer convolutional layers and each kernel function corresponding to each layer of the multi-layer transposed convolutional layers, the number of multi-layer convolutional layers and each kernel function corresponding to each layer of the multi-layer convolutional layers, and the number of multi-layer transposed convolutional layers and each kernel function corresponding to each layer of the multi-layer transposed convolutional layers are determined according to a preset image restoration clarity.

[0055] Specifically, in the GAN model deployed on the server, the generation sub-model is responsible for generating the restored image based on the image to be restored. The structure of this generation sub-model consists of multiple convolutional layers and multiple transposed convolutional layers, which are the core parts of the GAN model and are used to learn the complex features of the image and perform effective image reconstruction.

[0056] Each convolutional layer of the generative sub-model is equipped with a set of kernel functions, and similarly, each transposed convolutional layer also corresponds to a set of kernel functions. These kernel functions are part of the GAN model parameters, and they determine how the model processes and transforms image data. The number of layers of the generative sub-model, including the number of convolutional layers and transposed convolutional layers, as well as the specific parameters of the kernel functions corresponding to each convolutional layer and each transposed convolutional layer, are determined based on the preset image restoration clarity requirements.

[0057] During the image restoration process, these parameters enable the generative sub-model to capture the subtle features of the image and generate high-quality, high-resolution image content in the missing or damaged areas. In this way, the server can effectively process the image to be restored and generate a restored image that meets the expected clarity.

[0058] Figure 2 The schematic diagram of the structure of the sub-model in the embodiment of the present invention is shown in Table 1. The damaged image corresponds to the image to be repaired. The first n (n is a positive integer, the same below) layers of the sub-model are multi-layer convolution layers, and the last n layers are multi-layer transposed convolution layers. The corresponding parameters are shown in Table 1:

[0059] Table 1

[0060]

[0061]

[0062] In one embodiment of the present invention, the number of multi-layer convolutional layers can be determined to be six and the number of multi-layer transposed convolutional layers can be determined to be six according to the preset image restoration clarity. Each kernel function corresponding to each convolutional layer and each kernel function corresponding to each multi-layer transposed convolutional layer can refer to Table 1. Therefore, the multi-layer convolutional layers in the server are six layers of convolutional layers connected in sequence, and the multi-layer transposed convolutional layers are six layers of transposed convolutional layers connected in sequence; wherein, the last layer (i.e., the sixth layer) of the multi-layer convolutional layer is connected to the first transposed convolutional layer of the multi-layer transposed convolutional layer. Taking the first convolutional layer as an example, the kernel size of the kernel function is 4×4, the number of kernels is 64, and the other convolutional layers and transposed convolutional layers can refer to Table 1, which will not be repeated here.

[0063] In one embodiment of the present invention, the GAN model may further include a discriminant sub-model. The discriminant sub-model is used to:

[0064] Obtain a complete image. Determine whether the complete image is a restored image based on the complete image, the restored image, and the discriminant sub-model, and perform corresponding restoration on the image to be restored based on the determination result.

[0065] The discriminant sub-model includes n convolutional layers and one logistic regression layer, and the model parameters of the discriminant sub-model are each kernel function corresponding to each layer of the n convolutional layers and the kernel function corresponding to one logistic regression layer.

[0066] Performing corresponding repairs on the image to be repaired according to the determination result may include: if the determination result is an image that has been repaired, repeatedly repairing the image to be repaired using the repaired image; if the determination result is an image that has not been repaired, repeatedly repairing the image to be repaired using the unrepaired image.

[0067] Figure 3 Schematic diagram of the structure of the discriminant model in an embodiment of the present invention. The undamaged original image corresponds to the complete image. The discriminant model includes n convolutional layers and a logistic regression layer; wherein the n convolutional layers are connected in sequence, and the last convolutional layer is connected to a logistic regression layer. In one embodiment of the present invention, the n convolutional layers are six convolutional layers connected in sequence, and the last (i.e., the sixth) convolutional layer of the n convolutional layers is connected to a logistic regression layer. The corresponding parameters are shown in Table 2:

[0068] Table 2

[0069]

[0070] Taking the first convolution layer as an example, the kernel size of the kernel function is 4×4 and the number of kernels of the kernel function is 64; the weight of the kernel function corresponding to one logistic regression layer is 1. The other convolution layers in the six convolution layers can refer to Table 2 and will not be repeated here.

[0071] In one embodiment of the present invention, the image restoration process can be implemented by the following steps: First, the training data is preprocessed to ensure that it is suitable for training the GAN model. Then, the GAN model is trained using the processed training data to learn the intrinsic structure and features of the image. Then, the trained GAN model is applied to the actual image restoration task.

[0072] Specifically, the GAN model consists of two sub-models: a generative sub-model and a discriminative sub-model. The generative sub-model is responsible for creating realistic images, while the discriminative sub-model evaluates the generated images and tries to distinguish them from real images. The training data should cover images of multiple categories, such as faces, landscapes, text, and digital images, to improve the generalization ability of the GAN model. If the training data comes from the same category, it can be used to train a single GAN model; if multiple categories are involved, it may be necessary to train multiple models separately to adapt to the characteristics of different categories.

[0073] It is important that the image data used for training must be complete and undamaged digital images, which can ensure that the GAN model learns the correct image features and generates high-quality results in the restoration task. In this way, the present invention can effectively repair various categories of damaged images and restore their original appearance and details.

[0074] In one embodiment of the present invention, the training data processing process (also the damaged image generation process) may include the following steps:

[0075] (1) Screen the sizes of images in the training data and retain images whose sizes meet the threshold requirements;

[0076] (2) Screening the clarity of images in the training data and retaining images whose clarity meets the threshold requirements;

[0077] (3) resizing the retained images to conform to a unified standard (i.e., maintaining the same size);

[0078] (4) Operate on the image of standard size to generate a damaged image (used as a reference for repairing the image to be repaired).

[0079] Specifically, in order to create damaged images for training and testing image restoration algorithms, the following steps can be followed: First, select a series of complete images of standard sizes as original materials. Next, perform cropping operations on these complete images by simulating the process of image damage. Specifically, certain areas on the image are randomly selected or masked at fixed positions according to a preset pattern. These masked areas are represented as black pixels on the image, that is, the RGB color values ​​are all [0,0,0]. In this way, we get a set of damaged images, each of which has its corresponding original complete image. This correspondence allows us to evaluate the performance of the algorithm during the restoration process, that is, to measure the restoration quality by comparing the difference between the restored image and the original complete image. This process not only provides training data for the image restoration task, but also provides a benchmark for evaluating the restoration effect.

[0080] In one embodiment of the present invention, for the image to be restored IM, the calculation formula of its image clarity is:

[0081]

[0082] Among them, θ(IM) represents the image clarity, IM grey Represents the grayscale image of the image, L represents the Laplace operator, Represents the convolution operation, var() represents the variance value, that is, the grayscale image IM grey The variance of the elements in the matrix obtained by convolution operation with the Laplace operator L is the clarity of the image.

[0083] Figure 4 The training flow chart of the GAN model in the embodiment of the present invention is shown in FIG. The training flow describes in detail the training process of two key sub-models in the generative adversarial network (GAN): the generator sub-model and the discriminator sub-model. The core function of the generator sub-model is to perform the image restoration task, which restores the damaged image to an image that looks intact. At the same time, the role of the discriminator sub-model is to evaluate whether the input image is a restored image, so as to improve its discrimination ability.

[0084] The two sub-models are trained using adversarial training. In this method, the generative sub-model and the discriminative sub-model update their parameters alternately. Through this alternating training, the repair ability of the generative sub-model is strengthened, while the discrimination ability of the discriminative sub-model is also improved. The training goal is to enable the generative sub-model to repair the damaged image in a realistic and natural way, so that the discriminative sub-model has difficulty distinguishing the difference between the repaired image and the original complete image.

[0085] The data used to train the GAN model consists of two parts: one is full images of standard size, which are used as raw materials in the data processing stage; the other is damaged images obtained by cropping these full images. The cropping operation simulates the process of image damage and generates damaged image samples for training. Such a training dataset provides rich information for the GAN model, enabling it to learn how to perform effective image restoration under various damage conditions.

[0086] In order to solve the problem of unstable training of the GAN model, a method of pre-training the generative sub-model and then adjusting the training of the generative sub-model and the discriminative sub-model can be used. In one embodiment of the present invention, the specific training steps of the GAN model are as follows:

[0087] (1) Initialize the parameters of the generative sub-model and the discriminative sub-model.

[0088] (2) Generative sub-model pre-training: The damaged image is input into the generative sub-model, and the repaired image is output. The parameters of the generative sub-model are iteratively updated according to the first objective function until the first objective function converges. The first objective function can be |IM-G(IM part )|.

[0089] In the above formula, IM represents the complete image, IM part Represents the damaged image after cropping of IM, G() represents the repaired image obtained by generating the sub-model, and calculates the Euclidean distance between the pixels of the two images (the complete image and the repaired image).

[0090] (3) Inputting the damaged image into the generating sub-model to obtain a repaired image, inputting the repaired image and the complete image into the discriminating sub-model, and updating the discriminating sub-model parameters according to the second objective function; wherein the second objective function may be:

[0091] C=0.999*|IM-G(IM part )|+0.001*E[log(D(IM))+log(1-D(G(IM) part ))]

[0092] The second objective function consists of two parts: one is the first objective function, which measures the similarity between the restored image and the target image; the other is the adversarial loss function, which measures the authenticity of the restored image. The adversarial loss function is the expected cross entropy value of the logistic regression classification of the discriminant submodel. The second objective function is the weighted average of these two parts. Usually, the weight of the adversarial loss function is relatively low (0.001). In the above formula, IM partrepresents the damaged image obtained after cropping of IM, G() represents the repaired image obtained by the generative sub-model, D() represents the probability returned by the discriminative sub-model, E[] represents the mathematical expectation, the first term calculates the Euclidean distance between pixels, and the second term is the adversarial loss function.

[0093] (4) The damaged image is input into the generative sub-model to obtain a repaired image, the repaired image and the complete image are input into the discriminative sub-model, and the generative sub-model parameters are updated according to the second objective function.

[0094] (5) Repeat steps (3) to (4) until the second objective function converges.

[0095] In one embodiment of the present invention, the generated sub-model obtained by GAN model training is used to perform image restoration, and the specific steps are as follows:

[0096] (1) Select the repair type according to the type of image to be repaired and the type of damage;

[0097] (2) setting the pixel values ​​of the damaged area of ​​the image to be repaired to black;

[0098] (3) Adjust the size of the image to be repaired to a standard size;

[0099] (4) The image to be repaired is input into the generated sub-model to obtain the repaired image.

[0100] In the above process, steps (1) and (2) require human participation, while steps (3) and (4) are automatically completed by the server.

[0101] Figure 5 The following are pictures of the restoration effect of a face image in an embodiment of the present invention. The first one (left side) is a complete image, the second one (middle) is a damaged image (the eyes are completely blocked), and the third one (right side) is a restored image. It can be seen that the restored image can restore the original complete image more clearly.

[0102] The image restoration method provided by the embodiment of the present invention achieves a significant improvement in technical effects by adopting a GAN model based on deep learning. First, the image restoration method significantly improves the clarity and quality of the restored image by optimizing the configuration of the multi-layer convolutional layer and the multi-layer transposed convolutional layer in the generation sub-model, including adjusting the number of layers and the kernel size. Secondly, the structural design of the generation sub-model, especially the connection relationship between layers, further enhances the detail expression and overall visual effect of the image restoration. In addition, the discriminant sub-model in the GAN model improves the accuracy of the evaluation of the authenticity of the restored image through the careful parameter setting of its multi-layer convolutional layer and logistic regression layer, ensuring a high degree of consistency between the restoration result and the original image. Combining these technical measures, the image restoration method provided by the embodiment of the present invention has not only achieved a breakthrough in clarity, but also achieved important breakthroughs in the naturalness and accuracy of the restoration effect.

[0103] It should be noted that in the design of the generative sub-model, by carefully configuring multi-layer convolutional layers and multi-layer transposed convolutional layers, including flexibly adjusting the number of layers and the size of the convolution kernel, the clarity and overall quality of the repaired image can be significantly improved. In addition, the structural design of the generative sub-model, especially the connection method between the layers, is also crucial to optimizing the image restoration effect. In the GAN model, the discriminant sub-model plays an evaluation role. It is responsible for analyzing and ensuring that the repaired image is visually indistinguishable from the original image, thereby verifying the effectiveness of the restoration work. In order to further improve the accuracy of the discriminant sub-model, the parameters of the multi-layer convolutional layer and the logistic regression layer can be further adjusted. The above measures work together to enable the image restoration method provided in the embodiment of the present invention to not only improve the clarity, but also reach a higher standard in the accuracy of judging the restoration effect.

[0104] Second embodiment

[0105] Based on the image restoration method provided in the first embodiment, the second embodiment of the present invention further provides a server for implementing the image restoration method (referred to as an image restoration server for short). Figure 6 This is a schematic diagram of the structure of an image restoration server provided by the second embodiment of the present invention. Figure 6 As shown, the image restoration server at least includes an acquisition module 1 and a generation module 2, wherein:

[0106] An acquisition module 1 is used to acquire an image to be restored;

[0107] A generation module 2 is used to generate a restored image according to the image to be restored and the model parameters of the generation sub-model; wherein the generation sub-model includes multi-layer convolutional layers and multi-layer transposed convolutional layers, the model parameters of the generation sub-model are each kernel function corresponding to each layer of the multi-layer convolutional layers and each kernel function corresponding to each layer of the multi-layer transposed convolutional layers, the number of layers of the multi-layer convolutional layers and each kernel function corresponding to each layer of the multi-layer convolutional layers, and the number of layers of the multi-layer transposed convolutional layers and each kernel function corresponding to each layer of the multi-layer transposed convolutional layers are determined according to a preset image restoration clarity.

[0108] In one embodiment of the present invention, the multi-layer convolution layer is six layers of convolution layers connected in sequence, and the multi-layer transposed convolution layer is also six layers of transposed convolution layers connected in sequence, wherein the sixth layer of the multi-layer convolution layer is connected to the first layer of the multi-layer transposed convolution layer. Specifically, the kernel size of the kernel function of the six layers of convolution layers is 4×4, and the number of kernels is 64, 64, 128, 256, 512 and 4000, respectively. The kernel size of the kernel function of the six layers of transposed convolution layers is also 4×4, and the number of kernels is 512, 256, 128, 64, 64 and 3, respectively.

[0109] In a further embodiment, the image restoration server further includes a discriminant module (not shown in the figure) with a discriminant sub-model built in. On this basis, the image restoration server is also used to obtain a complete image, determine whether the complete image has been restored based on the complete image, the restored image and the discriminant sub-model, and perform corresponding restoration on the image to be restored accordingly. Among them, the discriminant sub-model includes n convolutional layers and one logistic regression layer, wherein the kernel function corresponding to each layer of the n convolutional layers and the kernel function of the logistic regression layer are GAN model parameters. In one embodiment of the present invention, the n convolutional layers are six convolutional layers connected in sequence, and the sixth convolutional layer is connected to the logistic regression layer. The image restoration server provided by the embodiment of the present invention can improve the clarity of image restoration and the accuracy of judging whether the complete image has been restored by setting parameters such as the number of layers, kernel function size, number of kernels, and their connection relationship in the generation sub-model and the discriminant sub-model.

[0110] Third embodiment

[0111] Based on the image restoration method provided in the first embodiment, the third embodiment of the present invention further provides an electronic device for implementing the image restoration method. Figure 7 FIG. 1 is a schematic diagram of the physical structure of an electronic device provided in the third embodiment of the present invention. Figure 7 As shown, the electronic device includes: a processor (processor) 701, a memory (memory) 702 and a bus 703; wherein the processor 701 and the memory 702 communicate with each other through the bus 703;

[0112] The processor 701 is used to call the program instructions in the memory 702 to implement the image restoration method provided by the above-mentioned embodiments, for example, including: obtaining the image to be restored; generating a restored image according to the image to be restored and the model parameters of the generated sub-model; wherein the generated sub-model includes a multi-layer convolution layer and a multi-layer transposed convolution layer, and the model parameters of the generated sub-model are each kernel function corresponding to each layer of the multi-layer convolution layer and each kernel function corresponding to each layer of the multi-layer transposed convolution layer, the number of multi-layer convolution layers and each kernel function corresponding to each layer of the multi-layer convolution layer, and the number of multi-layer transposed convolution layers and each kernel function corresponding to each layer of the multi-layer transposed convolution layer are determined according to a preset image restoration clarity.

[0113] The processor 701 is used to control the overall operation of the electronic device to complete all or part of the steps of the above-mentioned image restoration method. The processor 701 can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processing (DSP) chip, etc. The memory 702 is used to store various types of data to support the operation of the electronic device, and these data may include instructions for any application or method used to operate on the electronic device, as well as application-related data. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, etc.

[0114] In an exemplary embodiment, the electronic device can be implemented by a computer chip or entity, or by a product with a certain function, for executing the above-mentioned image restoration method and achieving the same technical effect as the above-mentioned method. A typical embodiment is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0115] In another exemplary embodiment, the present invention further provides a computer-readable storage medium including program instructions, which, when executed by a processor, implements the steps in the image restoration method in any of the above embodiments. For example, the computer-readable storage medium may be the above-mentioned memory including program instructions, and the above-mentioned program instructions may be executed by a processor of a system to complete the above-mentioned image restoration method and achieve the same technical effect as the above-mentioned image restoration method.

[0116] Compared with the prior art, the image restoration method provided by the embodiment of the present invention achieves significant improvement over the traditional image restoration technology through deep neural network technology, especially generative adversarial network (GAN). The image restoration method can efficiently handle a variety of image damage problems including large-scale damage, while maintaining the consistency of the repaired area with the original image in texture and structure. Through the automated end-to-end restoration technology, the present invention greatly reduces human intervention, reduces labor costs, and improves the efficiency of the restoration process. The optimized configuration of the multi-layer convolutional layer and the multi-layer transposed convolutional layer in the generative sub-model ensures that the restoration image is significantly improved in clarity and quality. In addition, the discriminator in the adversarial training mechanism further improves the authenticity of the restored image, making the restoration effect difficult to distinguish from the original image. The image restoration method provided by the embodiment of the present invention also performs well in generalization ability, can adapt to different types and degrees of image damage, and shows high robustness. Even in the face of more noise or more serious damage, it can maintain a stable restoration effect.

[0117] The above is a detailed description of the image repair method and server provided by the present invention. For those skilled in the art, any obvious changes made to the present invention without departing from the essence of the present invention will constitute an infringement of the patent right of the present invention and will bear corresponding legal responsibilities.

Claims

1. An image restoration method, characterized in that The steps include: S1: Obtain the image to be repaired; S2: Generate a restored image according to the image to be restored and the model parameters of the generated sub-model; Among them, the generation sub-model includes multi-layer convolutional layers and multi-layer transposed convolutional layers, the model parameters of the generation sub-model are each kernel function corresponding to each layer of the multi-layer convolutional layers and each kernel function corresponding to each layer of the multi-layer transposed convolutional layers, the number of layers of the multi-layer convolutional layers and each kernel function corresponding to each layer of the multi-layer convolutional layers, and the number of layers of the multi-layer transposed convolutional layers and each kernel function corresponding to each layer of the multi-layer transposed convolutional layers are determined according to a preset image restoration clarity.

2. The image restoration method according to claim 1, wherein: In the generating sub-model, the multi-layer convolutional layer is n layers of convolutional layers connected in sequence, and the multi-layer transposed convolutional layer is n layers of transposed convolutional layers connected in sequence; wherein the last convolutional layer of the multi-layer convolutional layer is connected to the first transposed convolutional layer of the multi-layer transposed convolutional layer, and n is a positive integer.

3. The image restoration method according to claim 1, characterized in that The following steps are also included: Get the complete image; According to the complete image, the restored image and the discriminant sub-model, it is determined whether the complete image is a restored image, and the image to be restored is restored accordingly according to the determination result.

4. The image restoration method according to claim 3, wherein: The discriminant sub-model includes n convolutional layers and one logistic regression layer; wherein the n convolutional layers are connected in sequence, and the last convolutional layer is connected to one logistic regression layer.

5. The image restoration method according to claim 3, characterized in that The training steps of the generative sub-model and the discriminative sub-model are as follows: (1) Initialize the parameters of the generative sub-model and the discriminative sub-model; (2) Input the damaged image into the generative sub-model, output the repaired image, and iteratively update the parameters of the generative sub-model according to the first objective function until the first objective function converges; wherein the first objective function is |IM-G(IM part )|, IM represents the complete image, IM part represents the damaged image obtained after cropping of IM, G() represents the repaired image obtained after generating the sub-model, and the Euclidean distance between the pixels of the complete image and the repaired image is calculated; (3) Input the damaged image into the generative sub-model to obtain a repaired image, input the repaired image and the complete image into the discriminative sub-model, and update the discriminative sub-model parameters according to the second objective function; wherein the second objective function is: C=0.999*|IM-G(IM part )|+0.001*E[log(D(IM))+log(1-D(G(IM part ))] The second objective function consists of two parts: one is the first objective function, which is used to measure the similarity between the restored image and the target image; the other is the adversarial loss function, which is used to measure the authenticity of the restored image. part represents the damaged image obtained after cropping of IM, G() represents the repaired image obtained after the generative sub-model, D() represents the probability returned by the discriminative sub-model, and E[] represents the mathematical expectation; (4) inputting the damaged image into the generative sub-model to obtain a repaired image, inputting the repaired image and the complete image into the discriminative sub-model, and updating the parameters of the generative sub-model according to the second objective function; (5) Repeat steps (3) to (4) until the second objective function converges.

6. The image restoration method according to claim 5, characterized in that The damaged image is obtained by the following steps: (1) Screen the sizes of images in the training data and retain images whose sizes meet the threshold requirements; (2) Screening the clarity of images in the training data and retaining images whose clarity meets the threshold requirements; (3) Adjust the size of the retained image to meet the unified standard; (4) Operating the image of standard size to generate the damaged image.

7. The image restoration method according to claim 6, characterized in that The calculation formula of the image clarity is: Among them, θ(IM) represents the image clarity, IM grey Represents the grayscale image of the image, L represents the Laplace operator, Represents the convolution operation, and var() represents the variance value.

8. The image restoration method according to claim 1, characterized in that The step S2 includes the following sub-steps: (1) Select the repair type according to the type of image to be repaired and the type of damage; (2) setting the pixel values ​​of the damaged area of ​​the image to be repaired to black; (3) Adjust the size of the image to be repaired to a standard size; (4) Inputting the image to be repaired into the generation sub-model to obtain a repaired image.

9. A server for implementing the image restoration method according to any one of claims 1 to 8, characterized in that It includes an acquisition module and a generation module; among which, The acquisition module is used to acquire the image to be restored; The generating module is used to generate a restored image according to the image to be restored and the model parameters of the generating sub-model; wherein the generating sub-model includes multi-layer convolutional layers and multi-layer transposed convolutional layers, the model parameters of the generating sub-model are each kernel function corresponding to each layer of the multi-layer convolutional layers and each kernel function corresponding to each layer of the multi-layer transposed convolutional layers, the number of layers of the multi-layer convolutional layers and each kernel function corresponding to each layer of the multi-layer convolutional layers, and the number of layers of the multi-layer transposed convolutional layers and each kernel function corresponding to each layer of the multi-layer transposed convolutional layers are determined according to a preset image restoration clarity.

10. An electronic device, characterized in that including a processor, a memory and a bus; wherein, The processor and the memory communicate with each other via the bus; The processor is used to call the program instructions in the memory to implement the image restoration method according to any one of claims 1 to 8.

Images