Image inpainting method, device and computer readable storage medium

Abstract

The application discloses an image repairing method and device and a computer readable storage medium. The method comprises the following steps: determining a contrast image corresponding to a damaged image based on image similarity; determining a damaged area in the damaged image, and acquiring image features corresponding to the damaged area in the contrast image; performing image repairing on the damaged area of the damaged image based on the image features, and generating a repaired image; when a peak signal-to-noise ratio index and a structural similarity index between the repaired image and the damaged image do not satisfy a preset condition, taking the repaired image as the damaged image, and returning to perform the step of determining the contrast image corresponding to the damaged image based on the image similarity. By repairing the damaged area by using the image features corresponding to the damaged area in the contrast image, the problem that when the area of the damaged area is large, effective image features cannot be acquired from the undamaged area to repair the image, and the repairing effect is low, is solved.

Description

Technical Field

[0001] This invention relates to the field of image restoration processing technology, and more particularly to image restoration methods, apparatus, and computer-readable storage media. Background Technology

[0002] Image inpainting is a technique aimed at restoring damaged pixel features in an incomplete image, then reconstructing and generating a high-quality image with a deep semantic approximation to the original. Current image inpainting methods typically involve obtaining pixels surrounding the damaged area and filling it with pixels based on these surrounding pixels; that is, filling the damaged area with pixels from known regions of the image. This method achieves good results in processing images with regular structures, small-area inpainting, and low-resolution image inpainting. However, in cases of large-area damage, the damaged area is too large, while the known area is relatively small. This results in insufficient effective information from the surrounding image to repair the damaged area, leading to poor repair results.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this invention is to provide an image restoration method, apparatus, and computer-readable storage medium, which aims to solve the problem that when the damaged area is large, it is impossible to obtain effective image features from the small undamaged area of ​​the image to be restored, resulting in poor restoration effect.

[0005] To achieve the above objectives, the present invention provides an image restoration method, which includes the following steps:

[0006] Determine the corresponding comparison image for the damaged image based on image similarity;

[0007] Determine the damaged region in the damaged image, and obtain the image features corresponding to the damaged region in the comparison image;

[0008] Based on the image features, the damaged area of ​​the damaged image is repaired to generate a repaired image;

[0009] When the peak signal-to-noise ratio and structural similarity index between the repaired image and the damaged image do not meet the preset conditions, the repaired image is regarded as the damaged image, and the process returns to the step of determining the comparison image corresponding to the damaged image based on image similarity.

[0010] When the peak signal-to-noise ratio and structural similarity index between the repaired image and the damaged image meet preset conditions, the image repair process ends and the repaired image is output.

[0011] Optionally, the step of determining the comparison image corresponding to the damaged image based on image similarity includes:

[0012] Obtain the first image features of the damaged image;

[0013] The first image feature is compared with the second image features of each preset image in the dataset to determine the feature distance between the damaged image and the preset image;

[0014] Based on the feature distance, a preset image whose feature distance is less than a preset feature distance is used as the comparison image, and the feature distance is used to characterize the similarity between the damaged image and the preset image.

[0015] Optionally, the step of obtaining the first image feature of the damaged image includes:

[0016] The damaged image is input into the feature extraction module to obtain the first image features;

[0017] The feature extraction module includes at least two Inception convolutional modules, each including at least two dilated convolutional layers with different kernel sizes, a max pooling layer, and a feature merging layer.

[0018] Optionally, the step of obtaining the image features corresponding to the damaged region in the comparison image includes:

[0019] The first edge map of the damaged image and the second edge map of the comparison image are obtained based on a preset edge detection algorithm.

[0020] The fusion features are obtained based on the first edge map and the second edge map;

[0021] The fused features are used as image features corresponding to the damaged region in the comparison image, and the image features include structural information in the comparison image corresponding to the damaged region.

[0022] Optionally, the step of obtaining the first edge map of the damaged image and the second edge map of the comparison image based on a preset edge detection algorithm includes:

[0023] The first sub-edge map and the second sub-edge map of the damaged image are obtained based on a preset edge detection algorithm, and the first edge map is determined based on the first sub-edge map and the second sub-edge map.

[0024] The third sub-edge map and the fourth sub-edge map of the comparison image are obtained based on a preset edge detection algorithm, and the second edge map is determined based on the third sub-edge map and the fourth sub-edge map.

[0025] The preset edge detection algorithms include the Marr-Hildreth edge detection algorithm and the Canny edge detection algorithm.

[0026] Optionally, the step of performing image inpainting on the damaged area of ​​the damaged image based on the image features to generate a repaired image includes:

[0027] Based on the image features, obtain the structural information in the comparison image corresponding to the damaged area;

[0028] The structural information is filled into the damaged area of ​​the damaged image to generate a coarse repair image;

[0029] Obtain other image features of the undamaged region of the damaged image, including texture features;

[0030] Based on the other image features, the damaged areas in the coarse repair image are repaired to generate the repaired image.

[0031] Optionally, the step of determining the comparison image corresponding to the damaged image based on image similarity includes:

[0032] Based on the input instructions for the damaged image, the damaged area and the undamaged area of ​​the damaged image are generated;

[0033] Based on other image features of the undamaged image, perform the step of determining the corresponding comparison image for the damaged image based on image similarity.

[0034] To achieve the above objectives, the present invention provides an image restoration apparatus, the image restoration apparatus comprising:

[0035] A query network is used to determine the corresponding comparison image for a damaged image based on image similarity;

[0036] An image restoration model is used to determine the damaged region in the damaged image, obtain the image features corresponding to the damaged region in the comparison image, and perform image restoration on the damaged region of the damaged image based on the image features to generate a restored image;

[0037] The repair result discrimination module is used to determine whether the peak signal-to-noise ratio (PSNR) and structural similarity index between the repaired image and the damaged image meet preset conditions. When the PSNR and structural similarity index between the repaired image and the damaged image do not meet the preset conditions, the repaired image is regarded as the damaged image.

[0038] Optionally, the image restoration apparatus further includes: a memory, a processor, and an image restoration program stored in the memory and executable on the processor, wherein the image restoration program, when executed by the processor, implements the steps of the image restoration method as described above.

[0039] To achieve the above objectives, the present invention provides a computer-readable storage medium storing an image restoration program, which, when executed by a processor, implements the steps of the image restoration method described above.

[0040] This invention provides an image restoration method, apparatus, and computer-readable storage medium. The method includes: determining a comparison image corresponding to a damaged image based on image similarity; determining a damaged region in the damaged image and obtaining image features corresponding to the damaged region in the comparison image; performing image restoration on the damaged region of the damaged image based on the image features to generate a restored image; when the peak signal-to-noise ratio (PSNR) and structural similarity index between the restored image and the damaged image do not meet preset conditions, using the restored image as the damaged image and returning to the step of determining the comparison image corresponding to the damaged image based on image similarity; when the PSNR and structural similarity index between the restored image and the damaged image meet preset conditions, ending the image restoration process and outputting the restored image. This invention improves the compatibility between image features and damaged areas by obtaining comparison images similar to the damaged image based on image similarity, acquiring image features in the comparison images corresponding to the damaged areas of the damaged image, and repairing the damaged image based on these image features. This approach prevents repair failures due to the acquired image features not matching the damaged area, and also solves the problem of poor repair results when the undamaged area of ​​the damaged image is small, making it impossible to obtain features matching the damaged area from the undamaged area image. This ultimately improves the accuracy of image restoration. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention;

[0042] Figure 2 This is a flowchart illustrating the first embodiment of the image restoration method of the present invention;

[0043] Figure 3 This is a diagram of the architecture of an image restoration device;

[0044] Figure 4 To query the network architecture diagram;

[0045] Figure 5 This is a detailed flowchart of step S10 in the first embodiment of the image restoration method of the present invention;

[0046] Figure 6 This is a detailed flowchart of step S30 in the first embodiment of the image restoration method of the present invention;

[0047] Figure 7 This is a schematic diagram of image restoration according to the first embodiment of the image restoration method of the present invention;

[0048] Figure 8 This is a detailed flowchart of step S20 in the first embodiment of the image restoration method of the present invention;

[0049] Figure 9 This is a detailed flowchart of step S21 in the first embodiment of the image restoration method of the present invention.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0052] The main solution of this invention is as follows: First, determine the comparison image corresponding to the damaged image based on image similarity; second, determine the damaged region in the damaged image and obtain the image features corresponding to the damaged region in the comparison image; third, perform image restoration on the damaged region of the damaged image based on the image features to generate a restored image; fourth, when the peak signal-to-noise ratio (PSNR) and structural similarity index between the restored image and the damaged image do not meet preset conditions, use the restored image as the damaged image and return to the step of determining the comparison image corresponding to the damaged image based on image similarity; fifth, when the PSNR and structural similarity index between the restored image and the damaged image meet preset conditions, end the image restoration process and output the restored image.

[0053] like Figure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.

[0054] In this embodiment of the invention, the terminal can be a PC, or a smartphone, tablet computer, portable computer, or other terminal device.

[0055] like Figure 1As shown, the terminal may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0056] Optionally, the terminal may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, Wi-Fi module, etc. Sensors may include light sensors, motion sensors, and other sensors. Specifically, light sensors may include ambient light sensors and proximity sensors. The ambient light sensor can adjust the display brightness according to the ambient light level, while the proximity sensor can turn off the display and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, a gravity accelerometer can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity, and can be used for applications that identify the mobile terminal's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition functions (such as pedometers, taps), etc. Of course, the mobile terminal may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, which will not be elaborated here.

[0057] Those skilled in the art will understand that Figure 1 The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0058] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an image restoration program.

[0059] exist Figure 1In the terminal shown, network interface 1004 is mainly used to connect to the backend server and communicate data with it; user interface 1003 is mainly used to connect to the client (user terminal) and communicate data with it; while processor 1001 can be used to call the image restoration program stored in memory 1005 and perform the following operations:

[0060] Determine the corresponding comparison image for the damaged image based on image similarity;

[0061] Determine the damaged region in the damaged image, and obtain the image features corresponding to the damaged region in the comparison image;

[0062] Based on the image features, the damaged area of ​​the damaged image is repaired to generate a repaired image;

[0063] When the peak signal-to-noise ratio and structural similarity index between the repaired image and the damaged image do not meet the preset conditions, the repaired image is regarded as the damaged image, and the process returns to the step of determining the comparison image corresponding to the damaged image based on image similarity.

[0064] When the peak signal-to-noise ratio and structural similarity index between the repaired image and the damaged image meet preset conditions, the image repair process ends and the repaired image is output.

[0065] Furthermore, the processor 1001 can call the image restoration program stored in the memory 1005 and also perform the following operations:

[0066] Obtain the first image features of the damaged image;

[0067] The first image feature is compared with the second image features of each preset image in the dataset to determine the feature distance between the damaged image and the preset image;

[0068] Based on the feature distance, a preset image whose feature distance is less than a preset feature distance is used as the comparison image, and the feature distance is used to characterize the similarity between the damaged image and the preset image.

[0069] Furthermore, the processor 1001 can call the image restoration program stored in the memory 1005 and also perform the following operations:

[0070] The damaged image is input into the feature extraction module to obtain the first image features;

[0071] The feature extraction module includes at least two Inception convolutional modules, each including at least two dilated convolutional layers with different kernel sizes, a max pooling layer, and a feature merging layer.

[0072] Furthermore, the processor 1001 can call the image restoration program stored in the memory 1005 and also perform the following operations:

[0073] The first edge map of the damaged image and the second edge map of the comparison image are obtained based on a preset edge detection algorithm.

[0074] The fusion features are obtained based on the first edge map and the second edge map;

[0075] The fused features are used as image features corresponding to the damaged region in the comparison image, and the image features include structural information in the comparison image corresponding to the damaged region.

[0076] Furthermore, the processor 1001 can call the image restoration program stored in the memory 1005 and also perform the following operations:

[0077] The first sub-edge map and the second sub-edge map of the damaged image are obtained based on a preset edge detection algorithm, and the first edge map is determined based on the first sub-edge map and the second sub-edge map.

[0078] The third and fourth sub-edge maps of the comparison image are obtained based on a preset edge detection algorithm, and the second edge map is determined based on the third and fourth sub-edge maps.

[0079] The preset edge detection algorithms include the Marr-Hildreth edge detection algorithm and the Canny edge detection algorithm.

[0080] Furthermore, the processor 1001 can call the image restoration program stored in the memory 1005 and also perform the following operations:

[0081] Based on the image features, obtain the structural information in the comparison image corresponding to the damaged area;

[0082] The structural information is filled into the damaged area of ​​the damaged image to generate a coarse repair image;

[0083] Obtain other image features of the undamaged region of the damaged image;

[0084] Based on the other image features, the damaged areas in the coarse repair image are repaired to generate the repaired image.

[0085] Furthermore, the processor 1001 can call the image restoration program stored in the memory 1005 and also perform the following operations:

[0086] Based on the input instructions for the damaged image, the damaged area and the undamaged area of ​​the damaged image are generated;

[0087] Based on other image features of the undamaged image, perform the step of determining the corresponding comparison image for the damaged image based on image similarity.

[0088] First Embodiment

[0089] Reference Figure 2 The first embodiment of the present invention provides an image restoration method, the method comprising:

[0090] Step S10: Determine the comparison image corresponding to the damaged image based on image similarity;

[0091] Step S20: Determine the damaged region in the damaged image and obtain the image features corresponding to the damaged region in the comparison image;

[0092] Step S30: Based on the image features, perform image repair on the damaged area of ​​the damaged image to generate a repaired image;

[0093] Step S40: When the peak signal-to-noise ratio and structural similarity index between the repaired image and the damaged image do not meet the preset conditions, the repaired image is taken as the damaged image, and the process returns to the step of determining the comparison image corresponding to the damaged image based on image similarity.

[0094] Step S50: When the peak signal-to-noise ratio and structural similarity index between the repaired image and the damaged image meet preset conditions, the image repair process ends and the repaired image is output.

[0095] In this embodiment, it is applied to an image restoration device, referring to... Figure 3 , Figure 3 A device architecture diagram of an image restoration apparatus is shown, the image restoration apparatus comprising:

[0096] A query network is used to determine the corresponding comparison image for a damaged image based on image similarity;

[0097] An image restoration model is used to determine the damaged region in the damaged image, obtain the image features corresponding to the damaged region in the comparison image, and perform image restoration on the damaged region of the damaged image based on the image features to generate a restored image;

[0098] The repair result discrimination model (not shown in the figure) is used to determine the peak signal-to-noise ratio (PSNR) and structural similarity index between the repaired image and the damaged image. When the PSNR and structural similarity index between the repaired image and the damaged image do not meet the preset conditions, the repaired image is regarded as the damaged image.

[0099] Optionally, the query network is used to determine the comparison image corresponding to the damaged image based on image similarity. Optionally, the query network is used to determine the similarity between each preset image in the dataset and the damaged image, and to take the preset image with a similarity higher than the preset similarity as the comparison image. The preset similarity is the maximum similarity corresponding to other images in the dataset other than the comparison image.

[0100] Optionally, refer to Figure 4 , Figure 4 The diagram illustrates the architecture of a query network, which includes a feature extractor and a classifier. The feature extractor is used to extract first image features from the damaged image. The first image features include high-level semantic features and some local features of the damaged image. The feature extraction module includes at least two Inception convolutional modules. Each Inception convolutional module includes at least two dilated convolutional layers with different kernel sizes, a max pooling layer, and a feature merging layer. The feature extraction module is formed by stacking and residually connecting the Inception convolutional modules.

[0101] Optionally, the Inception convolutional module of the feature extraction module has five layers, and the kernel size of the dilated convolutional layer in each Inception convolutional module includes 7*7 and 3*3. In actual operation, the damaged image is input to the feature extractor, and convolutional operations of different sizes are performed on the damaged image by convolutional kernels in the dilated convolutional layers of the first Inception convolutional module, obtaining convolutional results corresponding to different kernel sizes. Based on using convolutional kernels of different sizes to convolve the damaged image, different receptive fields are obtained; larger kernels have larger receptive fields, allowing for the acquisition of more local information when performing convolutional operations on the damaged image. After completing the convolutional operation, each convolutional result is input to the max pooling layer, which performs max pooling operations on each convolutional result to reduce the amount of operation. After completing the max pooling operation, the max pooling results corresponding to each convolutional result are input to the... The feature merging layer merges the max pooling results and uses the merged max pooling results as the first layer image features. The first layer image features are then input into the second Inception convolutional module to generate the second layer image features. The third layer image features are then input into the second Inception convolutional module to generate the third layer image features, and so on, until all Inception convolutional modules have extracted features from the damaged image. The final feature is used as the first image feature of the damaged image. The first image feature includes multiple image features, denoted as Feature1, Feature2, Feature3, ..., FeatureN-1 and FeatureN.

[0102] Optionally, the classifier in the query network is a pre-trained classification network. The classifier is used to compare the first image feature of the damaged image with the second image feature of each preset image in the dataset to determine the feature distance between the damaged image and the preset image. The feature distance is the feature distance between the first image feature and the second image feature. After determining the feature distance, the preset image with a feature distance less than a preset feature distance is used as the comparison image. The feature distance is used to characterize the similarity between the damaged image and the preset image. The smaller the feature distance, the higher the similarity. The larger the feature distance, the lower the similarity. The preset feature distance is the minimum feature distance among the other preset images in the dataset besides the comparison image. The comparison image is the image most similar to the damaged image in semantic structure and texture details. Optionally, after obtaining multiple first image features (Feature1, Feature2, Feature3, ..., FeatureN-1 and FeatureN) of the damaged image based on the feature extractor, the multiple first image features (Feature1, Feature2, Feature3, ..., FeatureN-1 and FeatureN) are input to the classifier. After receiving the first image features, the classifier calculates the feature distance between the first image features and the second image features of each preset image, compares the feature distances, and obtains the preset image with the smallest feature distance. The preset image with the smallest feature distance is then used as the comparison image.

[0103] Optionally, before inputting the damaged image into the query network, the damaged image is preprocessed. The preprocessing involves generating a mask in the damaged region of the damaged image to cover the damaged region, generating the mask-covered damaged image, and inputting the mask-covered damaged image into the query network so that the query network determines the damaged and undamaged regions based on the mask-covered damaged image, and then obtains the first image feature based on the undamaged region image.

[0104] Optionally, the image restoration device further includes an image restoration model, which is used to determine the damaged area in the damaged image, obtain the image features corresponding to the damaged area in the comparison image, and perform image restoration on the damaged area of ​​the damaged image based on the image features to generate a restored image.

[0105] Optionally, the image restoration model is trained using training data. The method for training the image restoration model includes obtaining training data, inputting the training data into a pre-constructed image restoration network for training, and obtaining the image restoration model.

[0106] Optionally, the training data includes a training set, a validation set, and a test set. The training set includes a mask image set and a complete image set. The mask image set includes multiple mask images, which are generated by covering the original image with a mask. The mask images are used to simulate damaged images. There are no identical images between the complete image and the original image corresponding to the mask image. The similarity between the complete image and the original image is higher than a preset similarity threshold. The mask images corresponding to the complete images and the original images that are similar to each other are used as images in the training set. The mask image is a binary image corresponding to the damaged image. The mask is composed of 0 and 1. The 0-value area (displayed as black) covered by the mask is the damaged area, and the 1-value area (displayed as white) is not processed, corresponding to the undamaged image. The validation set includes the original image corresponding to the mask image.

[0107] Optionally, after acquiring the training data, the image restoration model is obtained by inputting the data to be trained into a pre-constructed image restoration network. The method includes first acquiring edge maps of each image in the training set based on a preset edge detection algorithm, inputting the edge maps into the FFC feature fusion module in the image restoration network, generating fused features corresponding to the edge maps by the FFC feature fusion module, and then inputting the fused features into the encoder module in the image restoration network for training. The encoder module performs restoration training on the mask image. This training is iterated until the image restoration network meets the training requirements, thereby completing the training of the image restoration network. The trained image restoration network is then used as the image restoration model.

[0108] Optionally, the edge detection algorithm includes the Marr-Hildreth edge detection algorithm and the Canny edge detection algorithm, respectively obtaining two edge maps corresponding to each image using different edge detection algorithms. It is understood that during training, different weight values ​​are assigned to different edge detection algorithms. After obtaining two edge maps of the same image based on different edge detection algorithms, the edge maps are fused based on the weight values ​​corresponding to each edge detection algorithm to generate the final edge map of the image. Thus, by extracting different edge maps of the image using different edge detection algorithms and fusing these edges to generate the final edge map, the accuracy of edge detection is improved.

[0109] Optionally, the FFC feature fusion module generates the fusion features corresponding to the edge map by first downsampling each input, then using real fast Fourier transform to extract its respective features, then performing convolution and inverse fast Fourier transform, and finally merging each output to obtain the fusion features. The fusion features include the structural information of the images in the training set, especially the lines in the mask region of the mask image in the comparison image similar to the mask image, which can also be understood as contours. The fusion features also include other edge information besides the structural information.

[0110] Optionally, after obtaining the fusion features, the mask image is repaired based on the fusion features in the mask region of the mask image in the comparison image by the encoder module. Optionally, the encoder module includes a first codec and a second codec. The first codec includes a first encoder and a first decoder, and the second codec includes a second encoder and a second decoder. The first encoder is represented by Encoder0, the first decoder by decoder0, the second encoder by Encoder1, and the second decoder by decoder1. The first encoder, the first decoder, the second encoder, and the second decoder have the same structure and are generated based on a pix2pix model. Optionally, the first codec and the second codec repair the image in different ways. Based on the different repair methods, the training weights corresponding to the first codec and the second codec are different when training them. The repair method of the first codec is to fill the damaged area in the mask image based on the structural information and edge information in the fusion features of the comparison image to generate a coarsely repaired image. Optionally, during the training of the first codec, it is determined whether the edge information of the filled mask region image is consistent with the edge information of other regions in the mask image besides the mask region. If they are consistent, it is determined that the coarse repair requirement is met. If they are inconsistent, the pix2pix model corresponding to the first codec is iteratively trained based on the fusion features until the coarsely repaired image repaired by the first codec meets the coarse repair requirement. Optionally, the second decoder is used to perform fine repair on the mask region using the image features of other regions in the mask image besides the mask region, generating a repaired image. Optionally, the peak signal-to-noise ratio and structural similarity index images between the finely repaired mask region image and other region images meet preset conditions. At this time, the finely repaired mask region image has the same style and harmonious picture as other region images, and has the same image content and structure.

[0111] Optionally, after constructing the image inpainting network, the loss function for training the image inpainting model can be determined using the receptive field loss, adversarial loss, and R1 loss as references.

[0112] Optionally, the loss function is used to calculate the receptive field loss L. HRPL , countering losses L ADV And the joint loss of R1 loss. Calculate the receptive field loss L. HRPL The formula is the first preset formula, which is as follows:

[0113]

[0114] Where x is the repaired image, The mask image corresponding to the repaired image. Let M represent the average operation, and Ф be the loss of the receptive field. HRF This represents a pre-trained VGG-16 model based on a high receptive field obtained through Fast Fourier Transform.

[0115] Optionally, the perceptual field loss is used to evaluate the degree of inconsistency between the input and output of the image inpainting model, so that the high-level information (content and global structure) of the output image is close to that of the real image.

[0116] Optionally, the adversarial loss L ADV Including generator loss and discriminator loss, the adversarial loss L ADV Used to make the generated image distribution approximate the image distribution of the target domain, and used to calculate the adversarial loss L. ADV The formula is the third preset formula, and the second preset formula is the following formula:

[0117]

[0118]

[0119]

[0120] Among them, D ξ For local discrimination scores, E x E x，m L represents expectation. G The purpose is to make the generated images more realistic. D Its function is to identify the differences between the restored image and the original image; optionally, in L G Minimum, L D When the maximum value is reached, the image restoration model is considered to converge.

[0121] Optionally, the formula used to calculate the R1 loss is a third preset formula, which is as follows:

[0122]

[0123] in, D represents ξ The gradient of (x) is used, and R1 is used for gradient descent to backpropagate and obtain the optimal parameters for repair.

[0124] Optionally, based on the perceptual field loss L HRPL , countering losses L ADV The joint loss is calculated by combining the R1 loss and the formula for calculating the joint loss is the fourth preset formula, which is as follows:

[0125]

[0126] Among them, k=10, α=40, β=0.01.

[0127] Optionally, after calculating the joint loss based on the first, second, third, and fourth preset formulas, the parameters of the image restoration network are adjusted based on the joint loss. The image restoration network is then trained using the adjusted parameters until it meets the convergence condition. The trained image restoration network is then used as the image restoration model. Meeting the convergence condition includes the loss function decreasing to a minimum (or decreasing to a point where it is essentially constant; a decrease within a preset range is considered essentially constant). The trained model, i.e., the image restoration model, is then obtained.

[0128] Optionally, the image restoration model further includes a restoration result judgment module. This module determines whether the restored image meets the restoration requirements. If the requirements are met, the image restoration process ends, and the restored image is output. If the requirements are not met, the query network is notified to continue searching for similar comparison images based on the restored image. Image restoration is then performed on the damaged image based on these comparison images until the restored image meets the restoration requirements. Optionally, the method for determining whether the restored image meets the restoration requirements includes:

[0129] The number of repairs is greater than or equal to the preset number of repairs;

[0130] The peak signal-to-noise ratio and structural similarity index between the repaired image and the damaged image meet preset conditions.

[0131] Optionally, the image restoration method proposed in this application is applied to the aforementioned image restoration apparatus. After receiving a damaged image, a comparison image is obtained based on the damaged image. The comparison image is the image most similar to the damaged image in terms of semantic results and texture details, and the damaged image is the image that needs to be restored. This application does not limit the circumstances under which an image needs to be restored. In one possible implementation, the circumstances under which an image needs to be restored include, but are not limited to: the image containing missing parts, the image containing occlusions, and the image containing noise.

[0132] Optionally, refer to Figure 5 Step S10 includes:

[0133] Step S11: Obtain the first image features of the damaged image;

[0134] Step S12: Compare the first image feature with the second image feature of each preset image in the dataset to determine the feature distance between the damaged image and the preset image;

[0135] Step S12: Based on the feature distance, a preset image with a feature distance less than a preset feature distance is selected as the comparison image. The feature distance is used to characterize the similarity between the damaged image and the preset image.

[0136] Optionally, the step of determining the comparison image corresponding to the damaged image based on image similarity includes:

[0137] Based on the input instructions for the damaged image, the damaged area and the undamaged area of ​​the damaged image are generated;

[0138] Based on the undamaged region, perform the step of determining the corresponding comparison image for the damaged image based on image similarity.

[0139] Optionally, before determining the comparison image, an input instruction for the damaged image is obtained. The input instruction is used to determine the damaged image in the damaged image. The triggering method of the input instruction may be to overlay an adjustment box on the damaged image, determine the target area of ​​the adjustment box based on the stretching and shrinking operation of the adjustment box, and take the target area as the damaged area. Optionally, after determining the damaged area, other areas in the damaged image other than the damaged area are taken as undamaged images.

[0140] Optionally, after determining the damaged area and the undamaged area, a step of determining the corresponding comparison image of the damaged image based on image similarity is performed according to other image features of the undamaged image.

[0141] Optionally, after acquiring the damaged image, the damaged image is input into a query network in the image restoration device so that the query network can find a similar comparison image from the dataset. Optionally, the method of acquiring the first image feature of the damaged image includes inputting the damaged image into a feature extraction module to acquire the first image feature. The feature extraction module includes at least two Inception convolutional modules. The Inception convolutional module includes at least two dilated convolutional layers with different kernel sizes, a max pooling layer, and a feature merging layer. The first image feature is the image feature corresponding to the undamaged region.

[0142] Optionally, after extracting the first image features of the damaged image through the feature extraction module, the classifier in the query network compares the first image features with the second image features of each preset image in the dataset to determine the feature distance between the first image features and the second image features of each preset image. Optionally, after determining the feature distance, the preset image with a feature distance less than a preset feature distance is used as the comparison image. The feature distance is used to characterize the similarity between the damaged image and the preset image. The smaller the feature distance, the higher the similarity; the larger the feature distance, the lower the similarity. The preset feature distance is the minimum feature distance among the feature distances between the damaged image and other preset images in the dataset besides the comparison image, so that the preset image with the smallest feature distance is used as the comparison image.

[0143] Optionally, after determining the comparison image, the damaged region in the damaged image is determined, and the image features corresponding to the damaged region in the comparison image are obtained based on the damaged region. The image features include structural information in the comparison image corresponding to the damaged region. The structural information is used to indicate the missing structural information of the damaged region in the damaged image. The missing structural information specifically refers to the original lines in the damaged region, which can also be understood as contours.

[0144] Optionally, after determining the image features, image inpainting is performed on the damaged area of ​​the damaged image based on the image features to generate a repaired image. Optionally, refer to... Figure 6 Step S30 includes:

[0145] Step S31: Obtain the structural information corresponding to the damaged area in the comparison image based on the image features;

[0146] Step S32: Fill the damaged area in the damaged image with the structural information to generate a coarse repair image;

[0147] Step S33: Obtain other image features of the undamaged area of ​​the damaged image;

[0148] Step S34: Based on the other image features, perform image repair on the damaged areas in the coarse repair image to generate the repaired image.

[0149] Optionally, after determining the image features, the image features include structural information in the comparison image corresponding to the damaged region, particularly the lines in the comparison image corresponding to the damaged region. Optionally, after obtaining the structural information, the structural information is filled into the damaged region to generate a coarse restoration image, which includes an undamaged region image and a filled damaged region image. Thus, by filling the damaged region with the structural information from the comparison image, the missing structural information of the damaged region is added. Optionally, the coarse restoration image is a damaged image with auxiliary structural information, and optionally, the coarse restoration image is generated by a first codec in the image restoration model.

[0150] Optionally, after generating the coarse-repaired image, other image features of the undamaged region of the damaged image are obtained. These other image features are image features of the undamaged region image within the damaged image, including but not limited to texture and color features in the undamaged region image. After obtaining these other image features, fine-repair is performed on the damaged region in the coarse-repaired image using these features to generate a fine-repaired image. This fine-repaired image is used as the repaired image, which is composed of the finely repaired damaged region image and the undamaged region image. Optionally, the repaired image is generated by the second codec in the image inpainting model.

[0151] Optionally, after generating the repaired image, the peak signal-to-noise ratio (PSNR) and structural similarity index between the repaired image and the damaged image are obtained. Based on the PSNR and structural similarity indices, it is determined whether the repaired image meets the repair requirements. If the PSNR and structural similarity indices meet preset conditions, the image repair process is terminated, and the repaired image is output. If the PSNR and structural similarity indices do not meet preset conditions, the repaired image is determined not to meet the repair requirements. The repaired image is then updated to the damaged image, and the process returns to determine the comparison image corresponding to the damaged image based on image similarity. The process continues until the peak signal-to-noise ratio (PSNR) and structural similarity index between the last generated repaired image and the damaged image meet preset conditions. These preset conditions characterize the repair requirements when repairing the image. The method for determining whether the repaired image meets the repair requirements can also utilize other feasible indicators besides the PSNR and structural similarity indices, such as the L1 loss index and subjective evaluation index. The subjective evaluation index is used to determine whether the semantic structure of the repaired image and the damaged image is consistent, whether they have the same content and structure, and also to determine whether the style of the repaired damaged area is consistent with the undamaged area and whether the image is harmonious.

[0152] Optionally, based on iteratively repairing the damaged image, each repaired image has richer image features than the previous repaired image. Based on the richer image features, images more similar to the damaged image can be found. When the damaged area is filled based on the image features of the more similar images, the filling effect of the damaged image tends to match the effect of the undamaged area, thereby improving the image repair effect.

[0153] Optionally, refer to Figure 7 , Figure 7 The image shown is a restored image of the damaged image. The image on the left is the damaged image, the image in the middle is the edge map of the damaged image, and the image on the right is the restored edge map.

[0154] In this embodiment, when the damaged area is large, directly training a neural network using undamaged images of the damaged area often fails to achieve the desired repair effect. Furthermore, when the damaged area is large, it is impossible to obtain image features corresponding to the damaged area from surrounding images. Therefore, this embodiment uses a query network to query similar comparison images based on the first image features of the damaged image. It then uses high-level semantic information to reasonably push damaged content corresponding to the damaged area from the comparison images. Finally, it obtains image features corresponding to the damaged area from the damaged content in the comparison images, and fills the damaged area based on these image features. The image features include… In the comparison image, the structural information corresponding to the damaged area is filled into the damaged area to generate a coarse repair image. Then, the damaged area image in the coarse repair image is finely repaired using other image features of the undamaged area image of the damaged image to generate a fine repair image. The fine repair image is used as the repair image. After generating the repair image, the damaged image is iteratively repaired based on the judgment result of whether the repair image meets the repair requirements until the peak signal-to-noise ratio and structural similarity index between the last generated repair image and the damaged image meet the preset conditions. By querying similar images to obtain image features, the problem of not being able to obtain image features from undamaged images is solved, thereby improving the repair effect.

[0155] Second Embodiment

[0156] Based on the first embodiment, referring to Figure 8 Step S20 includes:

[0157] Step S21: Obtain the first edge map of the damaged image and the second edge map of the comparison image based on a preset edge detection algorithm;

[0158] Step S22: Obtain fusion features based on the first edge map and the second edge map;

[0159] Step S23: Use the fused feature as the image feature corresponding to the damaged area in the comparison image, wherein the image feature includes structural information in the comparison image corresponding to the damaged area.

[0160] Optionally, after determining the damaged area, in order to obtain the image features corresponding to the damaged area in the comparison image, this embodiment of the application obtains a first edge map of the damaged image and a second edge map of the comparison image by obtaining a preset edge detection algorithm.

[0161] Optionally, refer to Figure 9The step of obtaining the first edge map of the damaged image and the second edge map of the comparison image based on a preset edge detection algorithm includes:

[0162] Step S211: Obtain a first sub-edge map and a second sub-edge map of the damaged image based on a preset edge detection algorithm, and determine the first edge map based on the first sub-edge map and the second sub-edge map;

[0163] Step S212: Obtain the third sub-edge map and the fourth sub-edge map of the comparison image based on the preset edge detection algorithm, and determine the second edge map based on the third sub-edge map and the fourth sub-edge map;

[0164] The preset edge detection algorithms include the Marr-Hildreth edge detection algorithm and the Canny edge detection algorithm.

[0165] Optionally, determining the first edge map based on the first sub-edge map and the second sub-edge map includes obtaining the fusion weights corresponding to the preset edge detection algorithms, where different preset edge detection algorithms correspond to different fusion weights; and performing a weighted summation of the first sub-edge map and the second sub-edge map based on the fusion weights to fuse and generate the first edge map. For example, as shown... Figure 7 As shown, Figure 7 The intermediate image is the first edge map of the damaged image.

[0166] Optionally, the method of determining the second edge map based on the third sub-edge map and the fourth sub-edge map includes obtaining the fusion weights corresponding to the preset edge detection algorithms respectively, wherein different preset edge detection algorithms correspond to different fusion weights, and performing a weighted summation of the third sub-edge map and the fourth sub-edge map based on the fusion weights to fuse and generate the first edge map.

[0167] Optionally, after determining the first edge map and the second edge map, the method for obtaining the fusion feature based on the first edge map and the second edge map includes inputting the first edge map and the second edge map into the FFC feature fusion module, and having the FFC feature fusion module determine the fusion feature. The method for the FFC feature fusion module to determine the fusion feature includes first downsampling based on the first edge map and the second edge map, then using Fast Fourier Transform to extract features from each edge map, convolving the features of the edge maps to generate a convolution result, performing an Inverse Fast Fourier Transform based on the convolution result, merging the results of each Inverse Fast Fourier Transform to generate the fusion feature, and using the fusion feature as the image feature. The image feature includes structural information in the comparison image corresponding to the damaged region.

[0168] In this embodiment, structural information corresponding to the damaged area in the comparison image is extracted using the first edge map of the damaged image and the second edge map of the comparison image. Then, the damaged image is repaired by filling the damaged area of ​​the damaged image with the structural information. By using the structural information of the comparison image with similar structural information to fill the damaged area, the repaired damaged area image tends to be consistent with the original image, thereby improving the repair effect.

[0169] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing an image restoration program, which, when executed by a processor, implements the steps of the various embodiments described above.

[0170] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0171] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0173] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An image restoration method, characterized in that, The image restoration method includes the following steps: Determining a comparison image corresponding to a damaged image based on image similarity includes: generating a damaged region and an undamaged region of the damaged image according to an input instruction for the damaged image; obtaining a first image feature of the damaged image based on the undamaged region; comparing the first image feature with the second image features of each preset image in the dataset to determine the feature distance between the damaged image and the preset images; and selecting preset images whose feature distance is less than a preset feature distance as comparison images, wherein the feature distance is used to characterize the similarity between the damaged image and the preset images. The damaged region in the damaged image is determined, and a first edge map of the damaged image and a second edge map of the comparison image are obtained based on a preset edge detection algorithm; The fusion features are obtained based on the first edge map and the second edge map; The fusion feature is used as the image feature corresponding to the damaged region in the comparison image, and the image feature includes structural information in the comparison image corresponding to the damaged region; Based on the image features, the damaged area of ​​the damaged image is repaired to generate a repaired image; When the peak signal-to-noise ratio and structural similarity index between the repaired image and the damaged image do not meet the preset conditions, the repaired image is regarded as the damaged image, and the process returns to the step of determining the comparison image corresponding to the damaged image based on image similarity. When the peak signal-to-noise ratio and structural similarity index between the repaired image and the damaged image meet preset conditions, the image repair process ends and the repaired image is output.

2. The image restoration method as described in claim 1, characterized in that, The step of obtaining the first image feature of the damaged image includes: The damaged image is input into the feature extraction module to obtain the first image features; The feature extraction module includes at least two Inception convolutional modules, each including at least two dilated convolutional layers with different kernel sizes, a max pooling layer, and a feature merging layer.

3. The image restoration method as described in claim 1, characterized in that, The steps of obtaining the first edge map of the damaged image and the second edge map of the comparison image based on a preset edge detection algorithm include: The first sub-edge map and the second sub-edge map of the damaged image are obtained based on a preset edge detection algorithm, and the first edge map is determined based on the first sub-edge map and the second sub-edge map. The third and fourth sub-edge maps of the comparison image are obtained based on a preset edge detection algorithm, and the second edge map is determined based on the third and fourth sub-edge maps. The preset edge detection algorithms include the Marr-Hildreth edge detection algorithm and the Canny edge detection algorithm.

4. The image restoration method as described in claim 3, characterized in that, The step of image restoration based on the image features to generate a restored image includes: Based on the image features, obtain the structural information in the comparison image corresponding to the damaged area; The structural information is filled into the damaged area of ​​the damaged image to generate a coarse repair image; Obtain other image features of the undamaged region of the damaged image; Based on the other image features, the damaged areas in the coarse repair image are repaired to generate the repaired image.

5. An image restoration device, characterized in that, The image restoration device includes: A query network is used to determine the corresponding comparison image for a damaged image based on image similarity. This includes generating a damaged region and an undamaged region of the damaged image according to an input instruction for the damaged image; obtaining a first image feature of the damaged image based on the undamaged region; comparing the first image feature with second image features of each preset image in the dataset to determine the feature distance between the damaged image and the preset images; and selecting preset images whose feature distance is less than a preset feature distance as comparison images, where the feature distance characterizes the similarity between the damaged image and the preset images. An image restoration model is used to determine the damaged region in the damaged image and obtain a first edge map of the damaged image and a second edge map of the comparison image based on a preset edge detection algorithm; obtain fusion features based on the first edge map and the second edge map; use the fusion features as the image features corresponding to the damaged region in the comparison image, the image features including structural information in the comparison image corresponding to the damaged region; and perform image restoration on the damaged region of the damaged image based on the image features to generate a restored image. The repair result discrimination module is used to, when the peak signal-to-noise ratio (PSNR) and structural similarity index between the repaired image and the damaged image do not meet preset conditions, identify the repaired image as the damaged image and return to the step of determining the comparison image corresponding to the damaged image based on image similarity; and when the PSNR and structural similarity index between the repaired image and the damaged image meet preset conditions, terminate the image repair process and output the repaired image.

6. An image restoration device, characterized in that, The image restoration apparatus further includes: a memory, a processor, and an image restoration program stored in the memory and executable on the processor, wherein the image restoration program, when executed by the processor, implements the steps of the image restoration method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image restoration program, which, when executed by a processor, implements the steps of the image restoration method as described in any one of claims 1 to 4.

Images