An image inpainting method and system combining edge information and appearance flow operation

Abstract

The disclosure provides an image repairing method and system combining edge information and appearance flow operation, the method comprising: preprocessing an obtained image to be repaired to obtain a smooth image; performing edge detection and prediction to obtain a target edge map; performing smooth structure reconstruction according to the smooth image and the target edge map to obtain reconstructed smooth structure; extracting the correlation between each region of the image to be repaired according to the reconstructed smooth structure, sampling features from the relevant regions based on the appearance flow operation to generate lively texture, and filling the features containing lively texture information to the damaged region of the image to be repaired to obtain a repaired complete image. The disclosure realizes the repair of the damaged image through the repair process from the edge to the smooth structure and then to the whole, combines the edge information and the appearance flow operation to perform three-stage image repair, can generate reasonable structure and real lively texture, and thus achieves good repair effect.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to an image restoration method and system that combines edge information and appearance flow operations. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Image inpainting refers to using information from known areas of an image and applying certain restoration criteria to fill in missing areas, such as ensuring that the known and damaged areas have the same geometric structure or statistical characteristics, resulting in a restored image with the same geometric structure or statistical characteristics and an ideal visual effect. Image inpainting is crucial for many image editing tasks, such as filling in blank areas left after deleting unwanted objects from an image.

[0004] Currently, image inpainting algorithms are mainly divided into two categories: traditional methods and learning-based methods. Traditional image inpainting methods can be further divided into diffusion-based methods and patch-based methods. Diffusion-based methods propagate information from the background to the missing region by solving partial differential equations, but this method is only suitable for small missing regions. Patch-based methods copy the best-matching background patch into the hole starting from the boundary. These methods show good performance, especially in background inpainting tasks, because there are always similar patches in the background. However, these methods are no longer suitable when the missing region contains complex structures or non-repeating textures.

[0005] Learning-based image inpainting models can use the entire image as input to repair missing areas. These methods typically employ an encoder-decoder structure, taking not only the color and texture features of the image to be repaired but also its semantic features, using these to synthesize visually realistic low-level pixels. However, the inventors have found that most existing learning-based methods fail to reconstruct reasonable content for an image, especially when important parts of the image are missing. Furthermore, while some existing methods can reconstruct a continuous and reasonable structure between holes and undamaged areas, they produce blurry textures inconsistent with the surrounding areas when restoring image texture. Summary of the Invention

[0006] To address the aforementioned issues, this disclosure proposes an image inpainting method and system that combines edge information and appearance flow operations. The method achieves image restoration through a three-stage process from edge to smooth structure and then to the whole image. Based on generative adversarial networks, the method combines edge information and appearance flow operations to perform image inpainting, which can generate reasonable structures and realistic and vivid textures, thereby achieving better restoration results.

[0007] To achieve the above objectives, the present disclosure adopts the following technical solution:

[0008] One or more embodiments provide an image inpainting method that combines edge information and appearance flow operations, including the following steps:

[0009] The acquired image to be repaired is preprocessed to obtain a smoothed image;

[0010] The edge map of the smooth image is obtained by the edge detection algorithm. Then, the edge map and the corresponding mask image are used to predict the edge to obtain the target edge map.

[0011] Smooth structure reconstruction is performed based on the smooth image and the target edge map to obtain the reconstructed smooth structure;

[0012] Based on the reconstructed smooth structure, the correlation between regions of the image to be repaired is extracted. Based on the appearance flow operation, features are sampled from the relevant regions to generate vivid textures. Features containing vivid texture information are then filled into the damaged areas of the image to be repaired, and finally, a fully repaired image is obtained.

[0013] One or more embodiments provide an image inpainting system that combines edge information and appearance flow operations, including:

[0014] Preprocessing module: configured to preprocess the acquired image to be repaired to obtain a smoothed image;

[0015] Edge prediction module: It is configured to use an edge detection algorithm to obtain a smooth image edge map, and perform edge prediction based on the smooth image edge map and the corresponding mask image to obtain the target edge map;

[0016] Reconstruction module: Configured to perform smooth structure reconstruction based on the smooth image and target edge map to obtain the reconstructed smooth structure;

[0017] Image inpainting module: It is configured to extract the correlation between regions of the image to be inpainted based on the reconstructed smooth structure, perform feature sampling from the relevant regions based on appearance flow operation to generate vivid textures, and fill the damaged areas of the image to be inpainted with features containing vivid texture information, so as to obtain the fully restored image.

[0018] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps described in the above method.

[0019] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps described in the above method.

[0020] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0021] This disclosure provides a three-stage image restoration method, including target edge prediction, smooth structure reconstruction, and texture generation. Based on edge information and smooth structure information, the structure of the damaged area is made more continuous with the structure of the undamaged area, and the structurally related regions in the image have higher similarity. Furthermore, the appearance flow operation is adopted to make the restored texture more realistic and closer to the real image, which can reconstruct reasonable content for the image. Even when important parts of the image are missing, the appearance flow operation can generate reasonable structures and realistic and vivid textures, while avoiding the generation of blurry textures inconsistent with the surrounding areas, thereby achieving a satisfactory restoration effect.

[0022] The advantages of this disclosure, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0023] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute a limitation thereof.

[0024] Figure 1 This is a network structure diagram of the three-stage image restoration network model of Embodiment 1 of this disclosure;

[0025] Figure 2 This is a flowchart of the image restoration method according to Embodiment 1 of this disclosure;

[0026] Figure 3 This is an example diagram of image restoration performed by the image restoration method of Embodiment 1 of this disclosure;

[0027] Figure 4 This is a comparative example of the image restoration method of Embodiment 1 of this disclosure and existing methods. Detailed Implementation

[0028] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0030] It should be noted that the terminology used herein is for descriptive purposes only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0031] Example 1

[0032] In one or more of the technical solutions disclosed in the embodiments, such as Figures 1 to 4 As shown, an image inpainting method combining edge information and appearance flow operations includes the following steps:

[0033] Step 1 involves preprocessing the acquired image to be repaired to obtain a smoothed image;

[0034] Step 2 uses an edge detection algorithm to obtain a smooth image edge map. Based on the smooth image edge map and the corresponding mask image, edge prediction is performed to obtain the target edge map.

[0035] Step 3: Reconstruct the smooth structure based on the smoothed image and the target edge map to obtain the reconstructed smooth structure;

[0036] Step 4: Extract the correlation between regions of the image to be repaired based on the reconstructed smooth structure, perform feature sampling from the relevant regions based on appearance flow operation to generate vivid textures, and fill the damaged areas of the image to be repaired with features containing vivid texture information to finally obtain the repaired complete image.

[0037] This embodiment provides a three-stage image restoration method, including target edge prediction, smooth structure reconstruction, and texture generation. Based on edge information and smooth structure information, the structure of the damaged area is made more continuous with the structure of the undamaged area, and the structure-related regions in the image have higher similarity. Furthermore, the appearance flow operation is adopted to make the restored texture more realistic and closer to the real image, which can reconstruct reasonable content for the image. Even when important parts of the image are missing, the appearance flow operation can generate reasonable structures and realistic and vivid textures, while avoiding the generation of blurry textures inconsistent with the surrounding areas, thereby achieving a satisfactory restoration effect.

[0038] In this embodiment, edge detection refers to using an edge detection algorithm to detect significant edges in an image to obtain the image's edge map, while edge prediction involves inputting the broken edge map into an edge prediction network to generate a predicted edge map.

[0039] Furthermore, in step 1, the method for preprocessing the acquired image to be repaired to obtain a smooth image can be to use an edge-preserving smoothing method to smooth the image to be repaired and obtain a smooth image.

[0040] Specifically, edge-preserving smoothing methods can employ RTV (relative total variation) smoothing, which can remove high-frequency textures while preserving sharp edges and low-frequency structures, resulting in a representation of the global structure.

[0041] In some embodiments, the above steps are implemented by constructing a three-stage image inpainting network model, including: edge prediction of the target edge map in step 2, smooth structure reconstruction in step 3, and texture generation in step 4; the three-stage image inpainting network model includes an edge prediction network, a smooth structure reconstruction network, and a texture generation network, each of which adopts a generative adversarial network, and each generative adversarial network contains a generator and a discriminator connected in sequence.

[0042] Optionally, the generator in each network of the three-stage image inpainting network model includes a cascaded encoder and decoder, and the discriminator can adopt a 70*70 PatchGAN structure. The discriminator is used to determine whether the input image patch is real.

[0043] Specifically, the edge map of the smoothed image and the corresponding mask image are used as inputs to the first-stage edge prediction network to obtain the target edge map; then the target edge map and the smoothed image are used as inputs to the second-stage smooth structure reconstruction network to obtain the reconstructed smooth structure; in the third stage, the image to be repaired and the reconstructed smooth structure image are input to the texture generation network, and the appearance flow operation is used to make the features containing vivid texture information "flow" to the damaged area, thereby making the texture of the damaged area more realistic.

[0044] In this embodiment, the image to be repaired and its corresponding smoothed image are respectively represented as I. gt and S gt The edge map and grayscale image of the smoothed image are represented by E, respectively. gt and S gray M is a mask in the irregular mask dataset, used to represent the damaged areas of the image.

[0045] In step 2, the method of performing edge prediction based on the smoothed image edge map and the corresponding mask image to obtain the target edge map involves converting the grayscale image of the smoothed image with damaged areas into a single image. Smooth image edge map The image mask M is used as input to the trained edge prediction network to obtain the target edge map E. pred Specifically, it includes the following steps:

[0046] Step 2.1 Perform edge detection on the smoothed image to obtain the edge map of the smoothed image;

[0047] Optionally, the Canny edge detector can be used to perform edge detection on the smoothed image to obtain the edge map of the smoothed image;

[0048] Step 2.2 Based on the edge map of the smoothed image and the corresponding mask image, perform edge prediction to obtain the target edge map;

[0049] The mask image is used to represent the damaged area. Each image corresponds to a mask image. The damaged image can be obtained by performing a Haderma product operation between the mask image and the original image. In this embodiment, the mask images used are obtained from the public dataset provided by Partial Convolution.

[0050] In step 3, a method for reconstructing a smooth structure based on the smoothed image and the target edge map to obtain the reconstructed smooth structure is specifically to reconstruct the smoothed structure from the edge map E of the smoothed image. gt and target edge map E pred The composite edge map E is obtained by combining the edges in the image. comp Then synthesize the edge map E comp Smooth structural images The image mask M is used as input, and the trained smooth structure reconstruction network is input to obtain the complete structure image through the second-stage smooth structure reconstruction network.

[0051] Specifically, the method for reconstructing a smooth structure based on a smoothed image and a target edge map to obtain a reconstructed smooth structure includes the following steps:

[0052] Step 3.1 Smooth the edge map E of the image gt Undamaged area and target edge map E pred The edges in the graph are combined to obtain the synthesized edge map E. comp In this embodiment, the damaged area is the area to be repaired.

[0053] Step 3.2 Based on the smoothed structure image and the synthesized edge graph E comp The generator generates a complete structural image S. pred This is the reconstructed smooth structure image.

[0054] In step 4, the third stage of image inpainting, the reconstructed structural image and the image to be inpainted are used as inputs to a trained texture generation network. The reconstructed structure is used to extract the correlation between regions with similar structures. The appearance flow is used to sample features from the undamaged region, thereby giving the inpainted region a vivid texture and improving the inpainting effect.

[0055] The appearance flow operation is used to sample features from relevant regions to generate vivid textures, and then fill the damaged areas of the image to be repaired with features containing vivid texture information. Specifically, an appearance flow is used to generate a matrix representing the correlation between different regions of the image, and Gaussian sampling is used to sample textures from correlated regions onto the damaged areas using this matrix. Here, the appearance flow is a neural network that extracts the correlation between image regions.

[0056] Furthermore, before using the three-stage image inpainting network model for image inpainting, the process includes training the three-stage image inpainting network model, as follows.

[0057] Step S1: Obtain the image dataset and mask dataset, and smooth the images in the image dataset.

[0058] Optionally, the public dataset Places2 can be used as the training set, and the images in the dataset can be smoothed to construct an image restoration dataset.

[0059] You can download the public dataset Places2 and the irregular mask dataset from the internet, and then use an edge-preserving smoothing method to smooth the images in Places2 to obtain a smoothed image dataset. Smoothed images can represent the global structure of real images.

[0060] Step S2 is the first stage of image inpainting, in which edge prediction is performed to obtain the target edge image.

[0061] Specifically, an edge detection algorithm is used to obtain a smooth image edge map. Based on the smooth image edge map and the corresponding mask image, edge prediction is performed to obtain the target edge map, including the following steps:

[0062] Step S2.1 Perform edge detection on the smoothed image to obtain the edge map of the smoothed image;

[0063] Step S2.2 Based on the edge map of the smoothed image and the corresponding mask image, perform edge prediction to obtain the target edge map E. pred The method employs a multi-layer convolutional neural network to downsample the edge map of a smooth image to extract image features and generate a feature vector. The generated features are then upsampled to finally generate the edge map of the missing part, which is the target edge map.

[0064] Specifically, a grayscale image of a smoothed image with damaged areas. Its edge map The image mask M is used as input to the edge prediction network. In the encoder stage, the edge map of the smoothed image corresponding to the image to be repaired is downsampled three times by a multi-layer convolutional neural network to extract image features and generate a feature vector. Optionally, each downsampling step is half the size of the previous sample. In the decoder stage, the encoded features are upsampled to finally obtain the target edge map of the missing part.

[0065] Optionally, the loss function for the edge prediction network includes adversarial loss and feature matching loss.

[0066] The losses from the confrontation are as follows:

[0067]

[0068] Among them, D e () refers to the output of the discriminator in the edge prediction network.

[0069] The feature matching loss is:

[0070]

[0071] Where, N i This represents the number of elements in the i-th activation layer, and ||*|1 represents the L1 norm. This represents the output obtained by passing the given input through the output of the i-th layer of the discriminator and then through the activation function.

[0072] In this embodiment, the weight of the feature matching loss is set to 10, which is a predefined value. The total loss function for this stage is:

[0073] Step S3, the second stage of image inpainting, involves smoothing the structure reconstruction based on the obtained target edge map; the synthesized edge map E... comp Smooth structural images The image mask M is used as input to the smoothing structure reconstruction network, and the complete structure image is obtained through the second-stage smoothing structure reconstruction network.

[0074] Specifically, smooth structure reconstruction is performed based on the smoothed image and the target edge map to obtain the reconstructed smooth structure, including the following steps:

[0075] Step S3.1 Smooth the edge map E of the image gt Undamaged area and target edge map E pred The edges in the graph are combined to obtain the synthesized edge map E. compThe combined algorithm formula is: E comp =E gt ⊙M+E pred ⊙(1-M);

[0076] Step S3.2 Based on the smoothed structure image and the synthesized edge graph E comp The generator generates a complete structural image S. pred The reconstructed smooth structure image can be represented as:

[0077] The loss function of the smooth structure reconstruction network is a weighted sum of L1 loss, adversarial loss, perceptual loss, and style loss, where:

[0078] The l1 loss is: the reconstructed smooth structure image S pred Smoothed image S of the image to be repaired gt The L1 norm of the pixel difference is given by the following formula:

[0079]

[0080] The losses from the confrontation are:

[0081]

[0082] Among them, D s () refers to the output of the discriminator in the structural reconstruction network;

[0083] The perceived loss is:

[0084]

[0085] Among them, F i (S * ) refers to the input S in the i-th layer of the pre-trained network. * The activation graph, and N i =C i H i W i It refers to S gt The number of elements in the activation of the i-th layer, C i H is the number of channels in the activation map described above. i and W i These are the width and height of the activation map, respectively.

[0086] The style loss is:

[0087]

[0088] Among them, K j It is a normalization factor.

[0089] In summary, the overall loss function of the smooth structure reconstruction network is:

[0090]

[0091] Step S4 is the third stage of image restoration. Based on the reconstructed smooth image structure, the correlation between the regions of the image to be restored is extracted to obtain features with vivid texture information. Based on the appearance flow operation, the relevant regions are sampled to generate vivid textures, and the features containing vivid texture information are filled into the damaged areas of the image to be restored, finally obtaining the restored complete image.

[0092] The third stage uses the reconstructed structural image and the image to be repaired as input. The reconstructed structure is used to extract the correlation between regions with similar structures. The appearance flow is used to sample features from the undamaged region, so that the repaired region has vivid texture and improves the repair effect.

[0093] Specifically, images with damaged areas to be repaired and the reconstructed structural image S pred As input, the image is fed into a texture generation network, which generates features containing vivid texture information through appearance flow operations. These features are then filled into the damaged area, giving the repaired area a vivid and realistic texture. The final result is a fully repaired image.

[0094] Regions with similar structures in an image generally have a high correlation, and the reconstructed structural image S can be used. pred The correlations between these regions are extracted to generate vivid textures. The appearance flow generates a matrix representing the correlations between regions of the image, and uses Gaussian sampling to sample textures from correlated regions onto the damaged areas using this matrix. Here, the appearance flow is a neural network for extracting the correlations between image regions.

[0095] The overall loss function of the texture generation network is a weighted sum of the L1 loss, adversarial loss, and sampling loss, as shown in the following formula:

[0096]

[0097] The L1 loss is:

[0098] The losses from the confrontation are:

[0099]

[0100] Among them, D t () is the output of the discriminator of the texture generation network;

[0101] The sampling loss is: Where cos(*) represents cosine similarity and F represents sampling features.

[0102] Step S5: Train the three generative adversarial networks for each stage separately. After training, remove the discriminators of the first and second stages, connect the generators of the three stages, and train them end-to-end until the model converges to obtain the trained three-stage image inpainting network model.

[0103] Test the restoration effect on the test set and evaluate the restored images, such as... Figure 3 and Figure 4 As shown. Compare the visual effects of the restored images. Figure 3 The image shown is a restoration result of the method in this embodiment, indicating that the image restored by the method in this embodiment has good visual effect; Figure 4 This is a comparison image of the method in this embodiment and other methods in a qualitative comparison of scene images. The comparison shows that the repair result of the method in this embodiment makes the image structure more continuous and reasonable, and the visual effect looks closer to the real image.

[0104] Table 1 shows the quantitative comparison results of the image restoration method of Embodiment 1 of this disclosure and existing methods on the Places2 dataset.

[0105] Table 1

[0106]

[0107] In the table, MASK refers to the mask, CA, GC, and GMCNN are existing algorithms, and Ours refers to the method of this embodiment. The table shows that comparing the peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and L1 error, the bold text indicates the best results, demonstrating that it outperforms other methods in most cases.

[0108] Example 2

[0109] Based on Embodiment 1, this embodiment provides an image inpainting system that combines edge information and appearance flow operations, including:

[0110] Preprocessing module: configured to preprocess the acquired image to be repaired to obtain a smoothed image;

[0111] Edge prediction module: It is configured to use an edge detection algorithm to obtain a smooth image edge map, and perform edge prediction based on the smooth image edge map and the corresponding mask image to obtain the target edge map;

[0112] Reconstruction module: Configured to perform smooth structure reconstruction based on the smooth image and target edge map to obtain the reconstructed smooth structure;

[0113] Image inpainting module: It is configured to extract the correlation between regions of the image to be inpainted based on the reconstructed smooth structure, perform feature sampling from the relevant regions based on appearance flow operation to generate vivid textures, and fill the damaged areas of the image to be inpainted with features containing vivid texture information, so as to obtain the fully restored image.

[0114] Example 3

[0115] This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it performs the steps described in the method of Embodiment 1.

[0116] Example 4

[0117] This embodiment provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps described in the method of Embodiment 1.

[0118] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0119] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. An image inpainting method combining edge information and appearance flow operations, characterized in that, Includes the following steps: The acquired image to be repaired is preprocessed to obtain a smoothed image; An edge detection algorithm is used to obtain a smooth image edge map. Based on the smooth image edge map and the corresponding mask image, edge prediction is performed to obtain the target edge map. Smooth structure reconstruction is performed based on the smooth image and the target edge map to obtain the reconstructed smooth structure; Based on the reconstructed smooth structure, the correlation between regions of the image to be repaired is extracted. Based on the appearance flow operation, features are sampled from the relevant regions to generate vivid textures. Features containing vivid texture information are then filled into the damaged areas of the image to be repaired, and finally, a fully repaired image is obtained. A three-stage image inpainting network model is constructed to achieve edge prediction, smooth structure reconstruction, and texture generation of the target edge map. The three-stage image inpainting network model includes an edge prediction network, a smooth structure reconstruction network, and a texture generation network. Each network adopts a generative adversarial network, and each generative adversarial network contains a generator and a discriminator connected in sequence. It also includes the step of training a three-stage image inpainting network model, including the following: Obtain the image dataset and mask dataset, and smooth the images in the image dataset; The smoothed image, the edge map of the smoothed image, and the corresponding mask image are input into the edge prediction network to perform edge prediction and obtain the target edge map. The synthesized edge map, smooth structure image, and image mask are input into the smooth structure reconstruction network to obtain the reconstructed smooth structure; the synthesized edge map is obtained by combining the undamaged regions of the edge map of the smooth image and the edges of the target edge map. The image to be repaired with damaged areas and the reconstructed structural image are taken as input and fed into the texture generation network. Features containing vivid texture information are generated through appearance flow operation, and the features containing vivid texture information are filled into the damaged areas of the image to be repaired. Three generative adversarial networks (GANs) are trained separately for each stage. After training, the discriminators of the first and second stages are removed, and the generators of the three stages are connected for end-to-end training until the model converges, resulting in the trained three-stage image inpainting network model.

2. The image inpainting method combining edge information and appearance flow operations as described in claim 1, characterized in that: An edge-preserving smoothing method is used to smooth the image to be repaired, resulting in a smoothed image. Alternatively, the edge-preserving smoothing method is the RTV method.

3. The image inpainting method combining edge information and appearance flow operations as described in claim 1, characterized in that: The generator in each network of the three-stage image inpainting network model includes a cascaded encoder and decoder, and a discriminator is used to determine whether the input image patch is real.

4. The image inpainting method combining edge information and appearance flow operations as described in claim 1, characterized in that: The loss functions of edge prediction networks include adversarial loss and feature matching loss; The loss function of the smooth structure reconstruction network is The weighted sum of loss, adversarial loss, perceptual loss, and style loss; The overall loss function of the texture generation network is The weighted sum of loss, adversarial loss, and sampling loss.

5. The image inpainting method combining edge information and appearance flow operations as described in claim 1, characterized in that: Smooth structure reconstruction is performed based on the smoothed image and the target edge map to obtain the reconstructed smooth structure, including the following steps: The undamaged regions of the edge map of the smooth image are combined with the edges in the target edge map to obtain the synthesized edge map; Based on the smooth structure image and the synthesized edge map, a generator is used to generate the reconstructed complete smooth structure image.

6. The image inpainting method combining edge information and appearance flow operations as described in claim 1, characterized in that: Based on the appearance flow operation, feature sampling is performed from relevant regions to generate vivid textures, and features containing vivid texture information are filled into the damaged areas of the image to be repaired. Specifically, a matrix representing the correlation between different regions of the image is generated by the appearance flow, and Gaussian sampling is used to sample textures from the correlated regions into the damaged areas using this matrix.

7. An image inpainting system combining edge information and appearance flow operations, based on the image inpainting method combining edge information and appearance flow operations as described in claim 1, characterized in that, include: Preprocessing module: configured to preprocess the acquired image to be repaired to obtain a smoothed image; Edge prediction module: It is configured to use an edge detection algorithm to obtain a smooth image edge map, and perform edge prediction based on the smooth image edge map and the corresponding mask image to obtain the target edge map; Reconstruction module: Configured to perform smooth structure reconstruction based on the smooth image and target edge map to obtain the reconstructed smooth structure; Image inpainting module: It is configured to extract the correlation between regions of the image to be inpainted based on the reconstructed smooth structure, perform feature sampling from the relevant regions based on appearance flow operation to generate vivid textures, and fill the damaged areas of the image to be inpainted with features containing vivid texture information, so as to obtain the fully restored image.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the steps of any one of claims 1-6.

Images