Defect Image Restoration Method, Device and Electronic Equipment

Abstract

The present invention discloses a method, apparatus and electronic device for restoring a defective image. Among them, the method includes: obtaining a target defective image; inputting the target defective image into an encoding module in an image restoration generator to obtain a target feature map corresponding to the target defective image, wherein the encoding module includes an encoding convolutional layer and an encoding dense network block, the encoding convolutional layer is used to extract a feature map, and the encoding dense network block is used to widen the number of channels of the feature map; inputting the target feature map into a decoding module in the image restoration generator to restore the target defective image and obtain a target complete image, wherein the decoding module includes a decoding dense network block and a decoding transposed convolutional layer, the encoding dense network block is used to reduce the number of channels of the feature map, and the decoding transposed convolutional layer is used to restore the feature map. The present invention solves the technical problem in the related art that it is difficult to restore any pixel of a defective image with an arbitrary shape when restoring a defective image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular, to a method, device and electronic device for restoring defective images. Background Technique

[0002] Image restoration is a technology for filling in damaged or missing image areas. It inserts the predicted content into the missing areas, and image restoration provides technical support for object removal. Although many image restoration methods have been proposed, image restoration remains a challenging problem. When restoring defective images in related technologies, it is still difficult to restore any pixel of a defective image with an arbitrary shape.

[0003] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of the present invention provide a method, device and electronic device for restoring defective images, so as to at least solve the technical problem that it is difficult to restore any pixel of a defective image with an arbitrary shape when restoring a defective image in related technologies.

[0005] According to an aspect of an embodiment of the present invention, a method for restoring a defective image is provided, including: obtaining a target defective image; inputting the target defective image into an encoding module in an image restoration generator to obtain a target feature map corresponding to the target defective image, where the encoding module includes an encoding convolutional layer and an encoding dense network block, the encoding convolutional layer is used to extract a feature map, and the encoding dense network block is used to widen the number of channels of the feature map; inputting the target feature map into a decoding module in the image restoration generator to restore the target defective image to obtain a target complete image, where the decoding module includes a decoding dense network block and a decoding transposed convolutional layer, the encoding dense network block is used to reduce the number of channels of the feature map, and the decoding transposed convolutional layer is used to restore the feature map.

[0006] Optionally, the step of inputting the target defective image into an encoding module in an image restoration generator to obtain a target feature map corresponding to the target defective image includes: when the encoding convolutional layer includes a first convolutional layer, a second convolutional layer, and a third convolutional layer, and the encoding dense network block includes a first dense network block and a second dense network block, inputting the target defective image into the encoding module in the image restoration generator, and the target defective image sequentially passes through the first convolutional layer, the first dense network block, the second dense network block, the second convolutional layer, and the third convolutional layer that are connected in sequence to obtain a target feature map corresponding to the target defective image, where the convolution kernels and strides of the convolutional layers included in the encoding convolutional layer are determined according to the pixel values of the target defective image.

[0007] Optionally, inputting the target feature map into a decoding module in the image inpainting generator to restore the target defective image to obtain a target complete image includes: when the decoding deconvolution layer includes a first deconvolution layer, a second deconvolution layer, and a third deconvolution layer, and the decoding dense network block includes a third dense network block, a fourth dense network block, and a fifth dense network block, inputting the target feature map into the decoding module in the image inpainting generator, and the target feature map sequentially passes through the sequentially connected third dense network block, the first deconvolution layer, the fourth dense network block, the second deconvolution layer, the fifth dense network block, and the third deconvolution layer to restore the target defective image to obtain the target complete image, where the convolution kernel and stride of the deconvolution layer included in the encoding deconvolution layer are determined according to the pixel values of the target feature map.

[0008] Optionally, before inputting the target defective image into an encoding module in the image inpainting generator to obtain a target feature map corresponding to the target defective image, it further includes: obtaining an initial image inpainting generator; training the initial image inpainting generator with multiple groups of sample data to obtain the image inpainting generator, where the multiple groups of sample data include sample defective images, sample complete images, and real complete images.

[0009] Optionally, training the initial image inpainting generator with multiple groups of sample data to obtain the image inpainting generator includes: constructing a loss function for training, where the loss function includes a first loss function term, a second loss function term, and a third loss function term. The first loss function term is used to represent the feature similarity distance between the sample defective image and the sample complete image, the second loss function term is used to represent the feature Euclidean distance between the sample defective image and the sample complete image, and the third loss function term is used to represent the binary cross-entropy loss between the discrimination value and a predetermined value, and the discrimination value is obtained according to the discrimination result of whether the sample complete image belongs to the real complete image; based on the loss function, training the initial image inpainting generator with multiple groups of sample data to obtain the image inpainting generator.

[0010] Optionally, the method further includes: constructing the first loss function term in the following manner: obtaining a first weight value corresponding to the feature similarity distance, and the feature similarity distance between the sample defective image and the sample complete image; constructing the first loss function term based on the first weight value and the feature similarity distance between the sample defective image and the sample complete image; constructing the second loss function term in the following manner: obtaining a second weight value corresponding to the feature Euclidean distance, and the feature Euclidean distance between the sample defective image and the sample complete image; constructing the second loss function term based on the second weight value and the feature Euclidean distance between the sample defective image and the sample complete image; constructing the third loss function term in the following manner: obtaining a third weight value corresponding to the binary cross-entropy loss, and the binary cross-entropy loss function of the discrimination value and the predetermined value; constructing the third loss function term based on the third weight value and the binary cross-entropy loss function.

[0011] Optionally, after inputting the target feature map into the decoding module in the image restoration generator to restore the target defective image to obtain a target complete image, the method further includes: determining a structural similarity index between the target defective image and the target complete image, and a peak signal-to-noise ratio metric value between the target defective image and the target complete image, where the structural similarity index is obtained based on a structure comparison value, a contrast value comparison value, and a brightness comparison value; determining an image quality index of the target complete image based on the structural similarity index and the peak signal-to-noise ratio metric value.

[0012] According to one aspect of the embodiments of the present invention, there is provided a defective image restoration device, including: an acquisition module, configured to acquire a target defective image; a generation module, configured to input the target defective image into an encoding module in an image restoration generator to obtain a target feature map corresponding to the target defective image, where the encoding module includes an encoding convolutional layer and an encoding dense network block, the encoding convolutional layer is configured to extract a feature map, and the encoding dense network block is configured to widen the number of channels of the feature map; a restoration module, configured to input the target feature map into a decoding module in the image restoration generator to restore the target defective image to obtain a target complete image, where the decoding module includes a decoding dense network block and a decoding transposed convolutional layer, the decoding dense network block is configured to reduce the number of channels of the feature map, and the decoding transposed convolutional layer is configured to restore the feature map.

[0013] According to one aspect of the embodiments of the present invention, there is provided an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the defective image restoration method described in any one of the above.

[0014] According to one aspect of the embodiments of the present invention, there is provided a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the defective image restoration method described in any one of the above.

[0015] In the embodiments of the present invention, a target defective image is obtained, and the target defective image is input into an encoding module in an image restoration generator to obtain a target feature map corresponding to the target defective image, that is, the features corresponding to the target defective image are extracted. Then, the target feature map is input into a decoding module in the image restoration generator, achieving the purpose of restoring the target defective image to obtain a target complete image. Since the encoding module includes an encoding convolutional layer and an encoding dense network block, and the decoding module includes a decoding dense network block and a decoding transposed convolutional layer. The encoding convolutional layer is used to extract the feature map, the encoding dense network block is used to widen the number of channels of the feature map, the encoding dense network block is used to reduce the number of channels of the feature map, and the decoding transposed convolutional layer is used to restore the feature map. Therefore, it is not limited to the pixels of the defective image and can perform fine restoration processing on images of all pixels, thereby solving the technical problem in the related art that it is difficult to restore any pixel of a defective image with an arbitrary shape when performing the restoration of the defective image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention, and constitute a part of this application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0017] Figure 1 is a flowchart of the defective image restoration method according to the embodiments of the present invention;

[0018] Figure 2 is a flowchart of the image restoration method provided by an alternative embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of the encoding and decoding module of the generation network part in the image restoration method provided by an alternative embodiment of the present invention;

[0020] Figure 4 is a schematic diagram of the discriminant network part in the image restoration method provided by an alternative embodiment of the present invention;

[0021] Figure 5 is a structural block diagram of the defective image restoration device according to the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0024] The following introduces the terms that appear in this application:

[0025] Normalized BN: Batch Normalization, the BN layer, can accelerate the speed of network training and convergence, control gradient explosion, prevent gradient disappearance, and prevent overfitting.

[0026] ReLU: Rectified Linear Units, a linear rectification unit activation function, also known as the relu activation function. It uses the principle of biomimetics, more efficient gradient descent and backpropagation: avoids the problems of gradient explosion and gradient disappearance, simplifies the calculation process, eliminates the influence of other complex activation functions such as exponential functions, and at the same time the dispersion of activity makes the overall calculation cost of the neural network decrease.

[0027] Sigmoid: The Sigmoid function is a common S-shaped function in biology, also known as the S-shaped growth curve. In information science, due to its properties such as monotonic increase and monotonic increase of the inverse function, the Sigmoid function is often used as the activation function of neural networks, mapping variables to between 0 and 1, and it is smooth and easy to differentiate.

[0028] SSIM Structural Similarity Index: SSIM (Structural Similarity), structural similarity, is an index for measuring the similarity between two images.

[0029] PSNR Peak Signal-to-Noise Ratio Metric: PSNR is the abbreviation of "Peak Signal to Noise Ratio", which is the peak signal-to-noise ratio. It is an objective standard for evaluating images. It has limitations and is generally used in an engineering project between the maximum signal and background noise. PSNR is the most common and widely used objective measurement method for evaluating image quality.

[0030] Adam: Adam optimizer, which combines the advantages of AdaGrad and RMSProp optimization algorithms. It comprehensively considers the first moment estimation of the gradient (First Moment Estimation, that is, the mean of the gradient) and the second moment estimation (Second Moment Estimation, that is, the uncentered variance of the gradient), and calculates the update step size. It mainly has the following remarkable advantages: simple to implement, computationally efficient, low memory requirement; the update of parameters is not affected by the scaling transformation of the gradient; the hyperparameters have good interpretability and usually do not need to be adjusted or only need very little fine-tuning; the update step size can be restricted within a roughly range; it can naturally implement the step size annealing process (automatically adjust the learning rate); it is very suitable for applications in scenarios with large-scale data and parameters; it is applicable to unstable objective functions; it is applicable to problems with sparse gradients or large noise in gradients.

[0031] BCEloss: Binary CrossEntropy Loss, the binary cross-entropy loss function.

[0032] Embodiment 1

[0033] According to an embodiment of the present invention, an embodiment of a method for restoring a defective image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And, although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0034] Figure 1 is a flowchart of the method for restoring a defective image according to an embodiment of the present invention, as Figure 1 shown, the method includes the following steps:

[0035] Step S102, obtain a target defective image;

[0036] In step S102 provided in the present application, the obtained target defective image may have defects in some regions of the image. The pixels of the target defective image can be any pixels, and image repair processing can be performed on them.

[0037] It should be noted that in the case where the pixels of the target defective image exceed the expected range, the target defective image can be scaled or cut into the expected size before processing.

[0038] Step S104: Input the target defective image into the encoding module in the image restoration generator to obtain a target feature map corresponding to the target defective image. The encoding module includes an encoding convolutional layer and an encoding dense network block. The encoding convolutional layer is used to extract the feature map, and the encoding dense network block is used to widen the number of channels of the feature map.

[0039] In step S104 provided in this application, by directly inputting the target defective image into the encoding module in the image restoration generator, the target feature map that can reflect the image features can be finally obtained through the encoding convolutional layer and the encoding dense network block in the encoding module, so as to restore the defective part of the target defective image.

[0040] Step S106: Input the target feature map into the decoding module in the image restoration generator to restore the target defective image and obtain a target complete image. The decoding module includes a decoding dense network block and a decoding transposed convolutional layer. The decoding dense network block is used to reduce the number of channels of the feature map, and the decoding transposed convolutional layer is used to restore the feature map.

[0041] In step S106 provided in this application, by inputting the image feature map into the decoding module of the image restoration generator, the target defective image is restored through the decoding dense network block and the decoding transposed convolutional layer, and a complete target complete image is obtained, achieving the purpose of image restoration.

[0042] Through the above steps S102 - S106, the target defective image is obtained. The target defective image is input into the encoding module in the image restoration generator to obtain a target feature map corresponding to the target defective image, that is, the features corresponding to the target defective image are extracted. Then, the target feature map is input into the decoding module in the image restoration generator, achieving the purpose of restoring the target defective image and obtaining the target complete image. Since the encoding module includes an encoding convolutional layer and an encoding dense network block, and the decoding module includes a decoding dense network block and a decoding transposed convolutional layer, the encoding convolutional layer is used to extract the feature map, the encoding dense network block is used to widen the number of channels of the feature map, the decoding dense network block is used to reduce the number of channels of the feature map, and the decoding transposed convolutional layer is used to restore the feature map. Therefore, it is not limited to the pixels of the defective image, and fine restoration processing can be performed on images with all pixels, thereby solving the technical problem in the related art that it is difficult to restore any pixel of a defective image with an arbitrary shape during the restoration of a defective image.

[0043] As an alternative embodiment, inputting the target defective image into the encoding module in the image inpainting generator to obtain a target feature map corresponding to the target defective image, including: when the encoding convolutional layer includes a first convolutional layer, a second convolutional layer, and a third convolutional layer, and the encoding dense network block includes a first dense network block and a second dense network block, inputting the target defective image into the encoding module in the image inpainting generator, and the target defective image sequentially passes through the sequentially connected first convolutional layer, first dense network block, second dense network block, second convolutional layer, and third convolutional layer to obtain a target feature map corresponding to the target defective image, where the convolutional kernels and strides of the convolutional layers included in the encoding convolutional layer are determined according to the pixel values of the target defective image.

[0044] In this embodiment, the target defective image sequentially passes through the sequentially connected first convolutional layer, first dense network block, second dense network block, second convolutional layer, and third convolutional layer. When passing through each layer, since the convolutional kernels and strides of the convolutional layers included in the encoding convolutional layer are determined according to the pixel values of the target defective image, the target defective image can be processed preferably, and feature maps with different pixels are obtained correspondingly. Finally, a target feature map corresponding to the target defective image is obtained, realizing the extraction of features.

[0045] As an alternative embodiment, inputting the target feature map into the decoding module in the image inpainting generator to restore the target defective image and obtain a target complete image, including: when the decoding transposed convolutional layer includes a first transposed convolutional layer, a second transposed convolutional layer, and a third transposed convolutional layer, and the decoding dense network block includes a third dense network block, a fourth dense network block, and a fifth dense network block, inputting the target feature map into the decoding module in the image inpainting generator, and the target feature map sequentially passes through the sequentially connected third dense network block, first transposed convolutional layer, fourth dense network block, second transposed convolutional layer, fifth dense network block, and third transposed convolutional layer to restore the target defective image and obtain a target complete image, where the convolutional kernels and strides of the transposed convolutional layers included in the encoding transposed convolutional layer are determined according to the pixel values of the target feature map.

[0046] In this embodiment, the target feature map sequentially passes through the sequentially connected third dense network block, first transposed convolutional layer, fourth dense network block, second transposed convolutional layer, fifth dense network block, and third transposed convolutional layer. When passing through each layer, since the convolutional kernels and strides of the transposed convolutional layers included in the decoding transposed convolutional layer are determined according to the pixel values of the target feature map, the target feature map can be processed preferably, and feature maps with different pixels are obtained correspondingly. Finally, the purpose of restoring the target defective image and obtaining a target complete image is achieved.

[0047] As an alternative embodiment, before inputting the target defective image into the encoding module in the image inpainting generator to obtain the target feature map corresponding to the target defective image, it further includes: obtaining an initial image inpainting generator; training the initial image inpainting generator with multiple groups of sample data to obtain an image inpainting generator, where the multiple groups of sample data include sample defective images, sample complete images, and real complete images.

[0048] In this embodiment, the image inpainting generator is obtained by training the initial image inpainting generator. Therefore, it can better implement the functions of feature extraction and image restoration. Moreover, the training samples include sample defective images, sample complete images, and real complete images, which can not only train the functions in terms of feature extraction and image restoration, but also determine the accuracy, so that the trained image inpainting generator is more accurate.

[0049] As an alternative embodiment, training the initial image inpainting generator with multiple groups of sample data to obtain an image inpainting generator includes: constructing a loss function for training, where the loss function includes a first loss function term, a second loss function term, and a third loss function term. The first loss function term is used to represent the feature similarity distance between the sample defective image and the sample complete image, the second loss function term is used to represent the feature Euclidean distance between the sample defective image and the sample complete image, and the third loss function term is used to represent the binary cross-entropy loss between the discrimination value and a predetermined value. The discrimination value is obtained based on the discrimination result of whether the sample complete image belongs to the real complete image; based on the loss function, training the initial image inpainting generator with multiple groups of sample data to obtain an image inpainting generator.

[0050] In this embodiment, different aspects are considered, and a loss function is constructed for training, making the image restoration effect more realistic in details and the training easier to balance. The clarity of the restored image is improved by introducing the feature similarity loss. It can also distinguish between real images and restored images, view the entire image to evaluate whether its overall is coherent, and view a small area located at the center of the completed missing part to ensure the local consistency of the generated restoration result. Therefore, the image inpainting generator can achieve better effects when applied.

[0051] As an alternative embodiment, the method further includes: constructing a first loss function term in the following manner: obtaining a first weight value corresponding to the feature similarity distance, and the feature similarity distance between the sample defective image and the sample complete image; constructing the first loss function term based on the first weight value and the feature similarity distance between the sample defective image and the sample complete image; constructing a second loss function term in the following manner: obtaining a second weight value corresponding to the feature Euclidean distance, and the feature Euclidean distance between the sample defective image and the sample complete image; constructing the second loss function term based on the second weight value and the feature Euclidean distance between the sample defective image and the sample complete image; constructing a third loss function term in the following manner: obtaining a third weight value corresponding to the binary cross-entropy loss, and the binary cross-entropy loss function of the discriminant value and the predetermined value; constructing the third loss function term based on the third weight value and the binary cross-entropy loss function.

[0052] In this embodiment, the weight values corresponding to different aspects are determined. Therefore, it is possible to balance the loss function terms considering different aspects through the weight values. The weight values can be adaptively set according to the actual application and scenario. Therefore, an appropriate loss function can be determined.

[0053] As an alternative embodiment, after inputting the target feature map into the decoding module in the image inpainting generator to restore the target defective image and obtain the target complete image, it further includes: determining the structural similarity index between the target defective image and the target complete image, and the peak signal-to-noise ratio metric value between the target defective image and the target complete image, where the structural similarity index is obtained based on the structure comparison value, the contrast value comparison value, and the brightness comparison value; determining the image quality index of the target complete image based on the structural similarity index and the peak signal-to-noise ratio metric value.

[0054] In this embodiment, by judging the finally generated target complete image and determining the structural similarity index between the target defective image and the target complete image, the image quality index of the target complete image can be determined, and further, it can be judged how the image quality of the target complete image is, which is beneficial to the learning and determination of the image inpainting generator.

[0055] Based on the above embodiments and alternative embodiments, an alternative implementation manner is provided, which is specifically described below.

[0056] An image inpainting method based on denseness is provided in an alternative implementation manner of the present invention. Figure 2 is a flowchart of the image inpainting method provided in an alternative implementation manner of the present invention, as Figure 2As shown, it can perform feature compression and extraction on the target defective image based on the encoding network (the same as the above encoding module) of the Dense Densenet architecture, and then send the extracted features to the decoding module (the same as the above decoding module) based on the Densenet architecture for image restoration, and make a true or false judgment with the real image in the discriminant network. Specifically, it includes the following steps:

[0057] S1. Obtain the target defective image i m×m , where the pixels of the target defective image are 128 * 128;

[0058] S2. Input the target defective image into the encoding module in the image restoration generator. The target defective image sequentially passes through the first convolutional layer, the first dense network block, the second dense network block, the second convolutional layer, and the third convolutional layer, and a target feature map corresponding to the target defective image is obtained;

[0059] S2.1 Output the target defective image i m×m to the first convolutional layer. The convolutional kernel is a * a, a = 5, use the relu activation function, and the stride d = 1 to obtain i m / 2×m / 2 .

[0060] S2.2 Input i m / 2×m / 2 into the first dense network block. The dense network block includes 16 layers of dense network layers. The structure of each dense network layer is BN normalization, ReLU correction, and convolution in sequence. Among them, the convolutional kernel is 1, the stride is 1, BN, ReLU, and convolution. Among them, the convolutional kernel is 3, the stride is 1, and the size of the picture remains unchanged after each dense processing. Repeating 16 times of the dense network layer is a dense network block; the result after processing by a dense network block is i m / 4×m / 4 .

[0061] S2.3 Input i m / 4×m / 4 into the second dense network block. The structure of the second dense network block is the same as that of the first dense network block, and the result after processing is i m / 8×m / 8 .

[0062] S2.4 Output i m / 8×m / 8 to the second convolutional layer. The convolutional kernel is a * a, a = 3, use the relu activation function, and the stride is d = 2 to obtain i m / 16×m / 16 ;

[0063] S2.5 Output i m / 16×m / 16 to the third convolutional layer. The convolutional kernel is a * a, a = 3, use the relu activation function, and the stride is d = 2 to obtain i m / 32×m / 32 .

[0064] S3. The target feature map i m / 32×m / 32Input into the decoding module of the image inpainting generator, the target feature map passes through the third dense network block, the first transposed convolutional layer, the fourth dense network block, the second transposed convolutional layer, the fifth dense network block, and the third transposed convolutional layer connected in sequence to restore the target defective image and obtain the target complete image.

[0065] Similarly, for i m / 32×m / 32 Enter the decoder part of the generative network. Among them, the dense network block has three layers, the transposed convolutional layer has three layers, the transposed convolutional kernel is 4*4, the stride is 2, and the relu activation function is used except for the last layer, and the sigmoid activation function is used for the last layer. The calculation result of the third dense network block is i m / 16×m / 16 , and the calculation result of the first transposed convolutional layer is i m / 8×m / 8 , and the calculation result of the fourth dense network block is i m / 4×m / 4 , and the calculation result of the second transposed convolutional layer is i m / 2×m / 2 , and the calculation result of the fifth dense network block is i m / 2×m / 2 , and the calculation result of the third transposed convolutional layer is i m×m , and finally the target complete image is obtained.

[0066] S4. Determine the SSIM (Structural Similarity Index) between the target defective image and the target complete image, and the PSNR (Peak Signal-to-Noise Ratio) metric value between the target defective image and the target complete image;

[0067] S5. Determine the image quality index of the target complete image according to the SSIM and the PSNR metric value.

[0068] It should be noted that before performing step S2, it is also necessary to construct an initial image inpainting generator, that is, construct the encoding module and the decoding module of the image inpainting generator, and train the constructed initial image inpainting generator to obtain the image inpainting generator. The following describes this process:

[0069] (I) Pre-acquisition and construction steps:

[0070] (1) Obtain the training set;

[0071] Obtain the training set and preprocess the training set. If the image pixels exceed the expected range, scale or cut them into the expected size (such as 128*128), and normalize the training images and convert them into tensors to obtain a number of training samples;

[0072] (2) Construct a generative network including an encoding module and a decoding module based on Densenet;

[0073] 1) Construct the encoding module:

[0074] Figure 3It is a schematic diagram of the encoding and decoding module in the generation network part of the image restoration method provided by an alternative embodiment of the present invention. As Figure 3 shown, the encoding module is provided with a first convolutional layer, a first dense network block, a second dense network block, a second convolutional layer, and a third convolutional layer connected in sequence.

[0075] Through the above encoding module, the target defective image is input into the encoding module in the image restoration generator. The target defective image sequentially passes through the first convolutional layer, the first dense network block, the second dense network block, the second convolutional layer, and the third convolutional layer connected in sequence to obtain a target feature map corresponding to the target defective image.

[0076] It should be noted that the convolutional kernels and strides of the convolutional layers included in the encoding convolutional layer are determined according to the pixel values of the target defective image.

[0077] For example, the encoding module in the generation network is provided with three convolutional layers and two dense network blocks. The convolutional kernel of the first layer of the first two layers is a*a (if the pixel of the input target defective image is 128*128, a = 5), and the relu activation function is used, and the stride is d (if the pixel of the input target defective image is 128*128, here d = 1). Followed by two dense network blocks to double the number of image channels; the convolutional kernels used in the convolutional layers of the last two layers are (a / 2 + 1 a / 2 + 1), and the relu activation function is also used, and the stride is d*2. (If the pixel of the input target defective image is 128*128, a = 3 d = 2, where a and d are both positive integers.

[0078] 2) Construct the decoding module:

[0079] Figure 3 It is a schematic diagram of the encoding and decoding module in the generation network part of the image restoration method provided by an alternative embodiment of the present invention. As Figure 3 shown, the decoding module is provided with a third dense network block, a first transposed convolutional layer, a fourth dense network block, a second transposed convolutional layer, a fifth dense network block, and a third transposed convolutional layer connected in sequence;

[0080] Through the above decoding module, the target feature map is input into the decoding module in the image restoration generator. The target feature map sequentially passes through the third dense network block, the first transposed convolutional layer, the fourth dense network block, the second transposed convolutional layer, the fifth dense network block, and the third transposed convolutional layer to restore the target defective image and obtain a target complete image.

[0081] It should be noted that the convolutional kernels and strides of the transposed convolutional layers included in the encoding transposed convolutional layer are determined according to the pixel values of the target feature map.

[0082] For example, in the decoder part of the generation network, two transposed convolution layers, two dense network blocks, and three convolution layers are set. The transposed convolution kernel is (a / 2 + 2) * (a / 2 + 2), and the stride is d * 2. The first four convolution layers and transposed convolution layers use the relu activation function, and the last convolution layer uses the sigmoid activation function. The output of the convolution is the output of the generator, which is the target complete image.

[0083] (3) Construct a discriminant network based on Densenet;

[0084] Figure 4 It is a schematic diagram of the discriminant network part in the image inpainting method provided by an alternative embodiment of the present invention. As Figure 4 shown, it can determine whether the target complete image is a real image.

[0085] Two sub-networks are set in the discriminant network. The global discriminant network has 6 layers. The first 5 layers are convolution layers. The convolution kernel is a * a. The first 4 layers use the relu activation function, and the stride is d (if the pixel of the input target defective image is 128 * 128, here a = 5d = 2); the fifth layer is a fully connected layer without using an activation function. The picture of the central missing area to be discriminated is input into the global discriminant network to output a feature vector representing the global discriminant result. The local discriminant network has 5 layers. The first 4 layers are convolution layers. The convolution kernel is a * a. The first 4 layers use the relu activation function, and the stride is d (if the pixel of the input target defective image is 128 * 128, here a = 5d = 2). The fifth layer is a fully connected layer without using an activation function. The picture of the central missing area to be discriminated is input into the global discriminant network, and an output feature vector represents the global discriminant result (if the pixel of the input target defective image is 128 * 128, here the dimension of the feature vector is 1 * 1024). Finally, the discriminant network connects the outputs of the two sub-networks and inputs them into a fully connected layer, which is mapped to a value representing the adversarial discriminant result.

[0086] It should be noted that during the application of this part, the image to be discriminated is j 64×64 , and the global discriminant network has 6 layers. The first 5 layers are convolution layers, and the fifth layer is a fully connected layer. Taking the 128×128 image input as an example: the convolution kernels of the first 4 layers are 5 * 5, the stride is 2, and the relu activation function is used; the fifth layer is a fully connected layer without using an activation function, and the stride is 1. The picture of the central missing area to be discriminated is input into the global discriminant network, and a feature vector (1 * 1024) is output to represent the global discriminant result.

[0087] The local discriminant network has 5 layers. The first 4 layers are convolutional layers, and the fifth layer is a fully connected layer. Taking the 128*128 image input as an example: the convolutional kernels of the first 4 layers are 5*5, and the stride is 2; the fifth layer is a fully connected layer with a stride of 1 and no activation function. The image of the central missing area to be discriminated is input into the global discriminant network, and a feature vector (1*1024) is output to represent the local discriminant result.

[0088] Finally, the discriminant network connects the outputs of the two subnets and inputs them into a fully connected layer, which is mapped to a numerical value, that is, the output of the discriminant network, to discriminate whether the image is a generated image or a sample defective image.

[0089] (2) Training steps:

[0090] After preparing the above items (1), (2), and (3), the training steps can be carried out. The following is an explanation of the training steps:

[0091] S1. Construct a loss function for training. Among them, the loss function includes a first loss function term, a second loss function term, and a third loss function term. The first loss function term is used to represent the feature similarity distance between the sample defective image and the sample complete image. The second loss function term is used to represent the feature Euclidean distance between the sample defective image and the sample complete image. And the third loss function term is used to represent the binary cross-entropy loss between the discriminant value and the predetermined value. The discriminant value is obtained based on the discriminant result of whether the sample complete image belongs to the real complete image;

[0092] S1.1. Construct the first loss function term in the following way: obtain the first weight value corresponding to the feature similarity distance, and the feature similarity distance between the sample defective image and the sample complete image; construct the first loss function term based on the first weight value and the feature similarity distance between the sample defective image and the sample complete image.

[0093] That is, calculate the feature similarity distance between the sample defective image x and the sample complete image G(x0), which is the feature loss function: L CX (x, G(x0), l) = -log(CX(φ l (x), φ l (G(x0)))); φ l (·) represents the feature extractor.

[0094] S1.2. Construct the second loss function term in the following way: obtain the second weight value corresponding to the feature Euclidean distance, and the feature Euclidean distance between the sample defective image and the sample complete image; construct the second loss function term based on the second weight value and the feature Euclidean distance between the sample defective image and the sample complete image.

[0095] That is, calculate the characteristic Euclidean distance between the sample defective image x and the sample complete image G(x0), which is the reconstruction loss function:

[0096] S1.3. Construct the third loss function term in the following way: Obtain the third weight value corresponding to the binary cross-entropy loss, and the binary cross-entropy loss function of the discriminant value and the predetermined value; construct the third loss function term based on the third weight value and the binary cross-entropy loss function. Among them, it includes the part of pre-training the generation network and the part of pre-training the discriminant network:

[0097] For the generation network: Feed the sample complete image x0 into the generator G to generate the sample complete image G(x0), train the generator according to the BCEloss between the output G(x0) and the real complete image x, and use Adam optimization to minimize the loss.

[0098] For the discriminant network: Feed the sample complete image x0 into the pre-trained generator G to generate the sample complete image G(x0), send G(x0) into the discriminator network D to get the result D(G(x)), and send the median corresponding to x into the discriminator network D to judge its relationship with the sample missing image x to get the result And send the vector corresponding to x into the discriminator network D to judge its relationship with the image xp to get the result Calculate the BCELoss between this discriminant result and 0, and minimize this loss. The loss function of the entire discriminant network is: By default, λ is taken as 10. Use Adam optimization to minimize the loss.

[0099] Through the above steps S1.1 - S1.3, the loss function of the entire network can be obtained as the weighted sum of the reconstruction loss and the adversarial loss: L = λrec Lrec + λadvL adv + λCX L CX , where λrec = 0.80, λadv = 0.1, λCX = 0.1.

[0100] S2. Based on the loss function, use multiple groups of sample data to train the initial image inpainting generator to obtain the image inpainting generator.

[0101] Use the Adam optimization algorithm to minimize the loss function. The discriminant network and the generative network are trained simultaneously, and then train the two sequential generative networks once again. Iterate the above process repeatedly. When the probability that the discriminant network judges correctly approaches 0.5, the training is completed.

[0102] It should also be noted that in step S5, the following operations are specifically performed:

[0103] Send the test image into the trained generator network to generate the repaired image, and calculate the PSNR (Peak Signal-to-Noise Ratio) metric and SSIM (Structural Similarity Index). Here, X represents the sample defective image, Y represents the sample complete image, and μ X , μ Y represent the means of images X and Y, σ X 2, σ Y 2 represent the variances of images X and Y, σ XY represents the covariance of images X and Y, C represents the number of channels, H represents the corresponding channel length, W represents the corresponding channel width, m represents all pixel points, and in the following, α = β = r = 1.

[0104]

[0105] K1 = 0.01, K2 = 0.03, L = 255, C1 = (K1 * L) 2 , C2 = (K2 * L) 2 , C3 = C2 / 2.

[0106] Structural comparison:

[0107]

[0108] Contrast comparison:

[0109]

[0110] Brightness comparison:

[0111]

[0112] α = β = 1, and SSIM is defined as:

[0113] SSIM(X,Y) = l(X,Y) α *c(X,Y) β *s(X,Y) γ

[0114] The PSNR calculation method is:

[0115] y i and represent the value of a pixel point;

[0116]

[0117] MAX I = 2 B -1, B represents how many bits are used to encode a pixel.

[0118] Through the above optional implementation manners, at least the following beneficial effects can be achieved:

[0119] 1) By integrating the advantages of the convolutional structure and the dense network block structure, the phenomenon of gradient disappearance can be alleviated, the transmission and utilization of image features can be strengthened, the parameters can be reduced, and the image features can be accurately extracted without increasing the network depth, thus accelerating the speed of image processing. The generative discriminative network improved by the joint training of the reconstruction loss and the adversarial loss can improve the image inpainting accuracy.

[0120] 2) Make the target defective image semantically coherent locally and globally. Use the training model based on the dense network to complete the image inpainting of any pixel in the missing area image of any shape.

[0121] 3) To train the context consistency of the image inpainting area, the trained global and local context discriminative networks are used to distinguish between real images and inpainted images. The global discriminative network views the entire image to evaluate whether its overall is coherent, while the local discriminative network only views a small area located at the center of the completed missing part to ensure the local consistency of the generated inpainting result. Then train the image completion network to deceive these two context discriminative networks.

[0122] 4) It can ensure the local consistency of the generated inpainting result, making the generated image indistinguishable from the real image in terms of overall consistency and details.

[0123] 5) Introduce the dense network to construct the basic network, making the image inpainting effect more realistic in details and easier to balance in training.

[0124] 6) Improve the clarity of the inpainted image by introducing the feature similarity loss, which converts the image into a set of variables, without the need for the images to be completely aligned in size and position, and constraints the local features of the image.

[0125] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0126] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of the various embodiments of the present invention.

[0127] Embodiment 2

[0128] According to an embodiment of the present invention, there is also provided an apparatus for implementing the above-mentioned defective image restoration method. Figure 5 It is a structural block diagram of a defective image restoration apparatus according to an embodiment of the present invention, as Figure 5 shown. The apparatus includes: an acquisition module 502, a generation module 504, and a restoration module 506. The following is a detailed description of the apparatus.

[0129] The acquisition module 502 is configured to acquire a target defective image; the generation module 504 is connected to the above-mentioned acquisition module 502 and is configured to input the target defective image into an encoding module in an image restoration generator to obtain a target feature map corresponding to the target defective image. Among them, the encoding module includes an encoding convolutional layer and an encoding dense network block. The encoding convolutional layer is used to extract a feature map, and the encoding dense network block is used to widen the number of channels of the feature map; the restoration module 506 is connected to the above-mentioned generation module 504 and is configured to input the target feature map into a decoding module in the image restoration generator to restore the target defective image to obtain a target complete image. Among them, the decoding module includes a decoding dense network block and a decoding transposed convolutional layer. The encoding dense network block is used to reduce the number of channels of the feature map, and the decoding transposed convolutional layer is used to restore the feature map.

[0130] It should be noted here that the above-mentioned acquisition module 502, generation module 504, and restoration module 506 correspond to steps S102 to S106 in the implementation of the defective image restoration method. The examples and application scenarios implemented by multiple modules and corresponding steps are the same, but are not limited to the content disclosed in the above-mentioned Embodiment 1.

[0131] Embodiment 3

[0132] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, including: a processor; a memory for storing processor-executable instructions, wherein the processor is configured to execute the instructions to implement the defective image restoration method according to any one of the above.

[0133] Embodiment 4

[0134] According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the defective image restoration method according to any one of the above.

[0135] The serial numbers of the above embodiments of the present invention are only for description and do not represent the merits of the embodiments.

[0136] In the above embodiments of the present invention, the descriptions of the respective embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0137] In the several embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the units or modules can be in electrical or other forms.

[0138] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0140] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0141] The foregoing are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for restoring a defective image, characterized in that, Including: Obtain a target defective image; Input the target defective image into an encoding module in an image restoration generator to obtain a target feature map corresponding to the target defective image, where the encoding module includes an encoding convolutional layer and an encoding dense network block, the encoding convolutional layer is used to extract a feature map, and the encoding dense network block is used to widen the number of channels of the feature map; Input the target feature map into a decoding module in the image restoration generator to restore the target defective image and obtain a target complete image, where the decoding module includes a decoding dense network block and a decoding transposed convolutional layer, the encoding dense network block is used to reduce the number of channels of the feature map, and the decoding transposed convolutional layer is used to restore the feature map; Among them, the step of inputting the target defective image into an encoding module in an image restoration generator to obtain a target feature map corresponding to the target defective image includes: when the encoding convolutional layer includes a first convolutional layer, a second convolutional layer, and a third convolutional layer, and the encoding dense network block includes a first dense network block and a second dense network block, input the target defective image into the encoding module in the image restoration generator, and the target defective image sequentially passes through the sequentially connected first convolutional layer, the first dense network block, the second dense network block, the second convolutional layer, and the third convolutional layer to obtain a target feature map corresponding to the target defective image, where the convolutional kernels and strides of the convolutional layers included in the encoding convolutional layer are determined according to the pixel values of the target defective image; Among them, before inputting the target defective image into an encoding module in an image restoration generator to obtain a target feature map corresponding to the target defective image, it further includes: obtaining an initial image restoration generator; training the initial image restoration generator with multiple groups of sample data to obtain the image restoration generator, where the multiple groups of sample data include sample defective images, sample complete images, and real complete images; Among them, the step of training the initial image restoration generator with multiple groups of sample data to obtain the image restoration generator includes: constructing a loss function for training, where the loss function includes a first loss function term, a second loss function term, and a third loss function term, the first loss function term is used to represent the feature similarity distance between the sample defective image and the sample complete image, the second loss function term is used to represent the feature Euclidean distance between the sample defective image and the sample complete image, and the third loss function term is used to represent the binary cross-entropy loss between a discrimination value and a predetermined value, and the discrimination value is obtained according to the discrimination result of whether the sample complete image belongs to the real complete image; training the initial image restoration generator with multiple groups of sample data based on the loss function to obtain the image restoration generator.

2. The method according to claim 1, wherein The step of inputting the target feature map into a decoding module in the image restoration generator to restore the target defective image and obtain a target complete image includes: In the case that the decoding deconvolution layer includes a first deconvolution layer, a second deconvolution layer, and a third deconvolution layer, and the decoding dense network block includes a third dense network block, a fourth dense network block, and a fifth dense network block, input the target feature map into the decoding module in the image inpainting generator. The target feature map sequentially passes through the third dense network block, the first deconvolution layer, the fourth dense network block, the second deconvolution layer, the fifth dense network block, and the third deconvolution layer which are connected in sequence to restore the target defective image and obtain the target complete image. Wherein, the convolution kernel and stride of the deconvolution layer included in the decoding deconvolution layer are determined according to the pixel values of the target feature map.

3. The method according to claim 1, wherein The method further includes: Construct the first loss function term in the following manner: obtain a first weight value corresponding to the feature similarity distance, and the feature similarity distance between the sample defective image and the sample complete image; construct the first loss function term according to the first weight value and the feature similarity distance between the sample defective image and the sample complete image; Construct the second loss function term in the following manner: obtain a second weight value corresponding to the feature Euclidean distance, and the feature Euclidean distance between the sample defective image and the sample complete image; construct the second loss function term according to the second weight value and the feature Euclidean distance between the sample defective image and the sample complete image; Construct the third loss function term in the following manner: obtain a third weight value corresponding to the binary cross-entropy loss, and the binary cross-entropy loss function of the discrimination value and the predetermined value; construct the third loss function term according to the third weight value and the binary cross-entropy loss function.

4. The method according to any one of claims 1 to 3, characterized in that After inputting the target feature map into the decoding module in the image inpainting generator to restore the target defective image and obtain the target complete image, it further includes: Determine the structural similarity index between the target defective image and the target complete image, and the peak signal-to-noise ratio metric value between the target defective image and the target complete image. Wherein, the structural similarity index is obtained according to the structure comparison value, the comparison value comparison value, and the brightness comparison value; Determine the image quality index of the target complete image according to the structural similarity index and the peak signal-to-noise ratio metric value.

5. A defective image restoration device, characterized in that, It includes: An acquisition module for acquiring a target defective image; A generation module for inputting the target defective image into the encoding module in the image inpainting generator to obtain a target feature map corresponding to the target defective image. Wherein, the encoding module includes an encoding convolution layer and an encoding dense network block. The encoding convolution layer is used to extract the feature map, and the encoding dense network block is used to widen the number of channels of the feature map; A restoration module, configured to input the target feature map into a decoding module in the image restoration generator to restore the target defective image and obtain a target complete image, where the decoding module includes a decoding dense network block and a decoding transposed convolutional layer, the encoding dense network block is used to reduce the number of channels of the feature map, and the decoding transposed convolutional layer is used to restore the feature map; Wherein, when the encoding convolutional layer includes a first convolutional layer, a second convolutional layer, and a third convolutional layer, and the encoding dense network block includes a first dense network block and a second dense network block, the generating module is further configured to input the target defective image into the encoding module in the image restoration generator, and the target defective image sequentially passes through the sequentially connected first convolutional layer, the first dense network block, the second dense network block, the second convolutional layer, and the third convolutional layer to obtain a target feature map corresponding to the target defective image, where the convolutional kernel and stride of the convolutional layer included in the encoding convolutional layer are determined according to the pixel values of the target defective image; Wherein, the generating module is further configured to obtain an initial image restoration generator; and train the initial image restoration generator with multiple groups of sample data to obtain the image restoration generator, where the multiple groups of sample data include sample defective images, sample complete images, and real complete images; Wherein, the generating module is further configured to construct a loss function for training, where the loss function includes a first loss function term, a second loss function term, and a third loss function term, the first loss function term is used to represent the feature similarity distance between the sample defective image and the sample complete image, the second loss function term is used to represent the feature Euclidean distance between the sample defective image and the sample complete image, and the third loss function term is used to represent the binary cross-entropy loss between the discrimination value and a predetermined value, and the discrimination value is obtained according to the discrimination result of whether the sample complete image belongs to the real complete image; based on the loss function, train the initial image restoration generator with multiple groups of sample data to obtain the image restoration generator.

6. An electronic device, characterized in that, Comprising: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the defective image restoration method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the defective image restoration method according to any one of claims 1 to 4.

Images