An Image Inpainting Method and System Based on Multi-Scale Attention Mechanism Edge Prediction

Abstract

This invention provides an image inpainting method and system based on edge prediction using a multi-scale attention mechanism. The method includes: an edge inpainting network taking the damaged image to be repaired and its corresponding mask as input; using a multi-scale attention mechanism to repair the edge contours of the missing areas in the image, combining local detail information and global structural information to accurately and comprehensively repair the image edges; after obtaining the complete edge map, a color filling network taking the obtained edge map, the damaged image, and the mask as input, and running the process to obtain the repaired complete image. In this process, the edge map serves as auxiliary information to further enhance the color filling effect. This invention considers the line texture and detail information of the image more comprehensively and has been successfully applied to the inpainting of large-area missing images with clear line contours. The repaired image not only retains details excellently but also maintains the overall coherence of the image, resulting in a visual effect closer to the real image.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to an image inpainting method and system based on edge prediction using a multi-scale attention mechanism. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Image inpainting is a fundamental task that fills in damaged areas of an image, using certain criteria to fill in the missing regions. Image inpainting methods are mainly divided into traditional inpainting methods and deep learning-based inpainting methods. Traditional inpainting methods use mathematical reasoning to find usable information in the undamaged area to fill the hole. This method is suitable for small-scale background loss. As the damaged area expands or the diffusion distance of pixel information around the hole increases, the effective information that can be obtained gradually decreases. When dealing with images with complex textures, traditional inpainting often results in artifacts and blurred filled areas because it cannot understand the details and textures of the image. Deep learning-based image inpainting mainly utilizes an encoder-decoder structure. The encoder first encodes the original image into an intermediate representation that is easy for machine learning to use. Then, machine learning completes the image inpainting work, and the decoder decodes this intermediate representation into the output. In this process, the machine learns the low-level features of the image, making the inpainted image closer to the real image.

[0004] However, the inventors discovered that most existing image inpainting techniques that use deep networks experience a decline in resolution and detail perception as the network deepens, resulting in visual loss and image textures that are inconsistent with the details of the surrounding areas. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides an image inpainting method and system based on edge prediction using a multi-scale attention mechanism. The scheme employs deep learning technology, based on a generative adversarial network framework, and adopts the approach of first repairing the image's lines and then filling in the colors: firstly, the edge information of the image is repaired, providing crucial image contour references for subsequent color information repair, which helps the color filling network achieve more accurate repair. This invention uses a multi-scale attention mechanism, combining local detail information and global structural information to accurately and comprehensively repair the image. Both edge repair and color repair in this invention are performed using convolutional neural networks, achieving excellent repair results in both image edge and color restoration; especially when repairing large areas of missing image content with clear lines, the repair result not only preserves details well but also maintains the overall coherence of the image.

[0006] The present invention achieves the above objectives through the following technical solutions.

[0007] In a first aspect, the present invention provides an image inpainting method based on edge prediction using a multi-scale attention mechanism, comprising:

[0008] The damaged image to be repaired and its corresponding mask image are used as input to a pre-trained edge repair network to obtain the repaired edge map. The edge repair network specifically performs the following processing steps: First, the damaged image is extracted using an edge detection algorithm. Then, the generator performs two downsampling operations on the damaged edge map to obtain shallow features of the edge image. Next, a multi-scale attention mechanism is introduced to obtain a refined edge feature map. Then, after processing by a residual module, an optimized refined edge map is obtained. Subsequently, it undergoes two upsampling operations by sequentially connected deconvolution modules to obtain the complete repaired edge map. Finally, a discriminator assigns true and false labels to the generated edge map and the real edge map, continuously optimizing its discrimination ability by calculating a loss function. After multiple training iterations, a trained edge repair network is obtained. Based on the obtained edge repair network, a generated edge map is obtained.

[0009] The restored edge map, damaged image, and mask image are used together as input to a deep learning-based color filling network to guide image color restoration. Based on the generated edge map obtained in the previous step, the generator of the color filling network predicts the restored image. Finally, a discriminator is used to evaluate the quality of the generated image, and the color filling network is continuously optimized by calculating the loss function. Finally, a near-realistic restored image is obtained.

[0010] Furthermore, the introduction of a multi-scale attention mechanism to obtain a fine edge feature map involves: first, inputting the obtained shallow features into several branch networks to obtain several feature maps of different scales; then, combining the feature maps of different scales to obtain a multi-scale feature map; then, performing multi-channel compression on the obtained multi-scale feature map to obtain the weight vector corresponding to each channel; finally, performing channel weighting operation based on the obtained multi-scale feature map and the weight vector corresponding to each channel to obtain a fine edge feature map.

[0011] Furthermore, the branch networks are composed of convolution operations with different dilation rates.

[0012] Furthermore, the method of combining several feature maps of different scales to obtain a multi-scale feature map is specifically achieved by cascading feature maps of different scales.

[0013] Furthermore, the residual module's processing specifically involves establishing direct connections between layers, enabling gradients to be directly propagated back to shallower layers, thereby mitigating the gradient vanishing or gradient exploding problem.

[0014] Furthermore, the edge repair network includes a generator and a discriminator, wherein the generator includes a sequentially connected convolutional module that performs downsampling, a multi-scale attention module, a residual module, and a sequentially connected deconvolutional module that performs upsampling; the discriminator includes a sequentially connected convolutional module with different parameter settings than the generator.

[0015] Furthermore, the deep learning-based color filling network includes: a sequentially connected convolutional module for downsampling, a residual module, and a sequentially connected deconvolutional module for upsampling; wherein, the convolutional module is used to encode the input image into a feature representation; the residual module is used to reduce the occurrence of gradient vanishing or gradient exploding problems and to realize feature mapping and transformation; the deconvolutional module is used to decode the mapped features into an output with the same size as the input image.

[0016] Furthermore, edge extraction is performed on the damaged image to obtain a damaged edge map, specifically using the Canny operator for edge extraction.

[0017] Secondly, the present invention provides an image inpainting system based on edge prediction using a multi-scale attention mechanism, comprising:

[0018] An edge restoration unit is used to take the damaged image to be restored and its corresponding mask image as input to a pre-trained edge restoration network to obtain the restored edge map. The edge restoration network specifically performs the following processing steps: First, the damaged image is extracted using an edge detection algorithm. Then, the generator performs two downsampling operations on the damaged edge map to obtain shallow features of the edge image. Next, a multi-scale attention mechanism is introduced to obtain a refined edge feature map. Then, after processing by a residual module, an optimized refined edge map is obtained. Subsequently, it undergoes two upsampling operations by sequentially connected deconvolution modules to obtain the restored complete edge map. Finally, a discriminator assigns true and false labels to the generated edge map and the real edge map, continuously optimizing its discrimination ability by calculating a loss function. After multiple training iterations, a trained edge restoration network is obtained. Based on the obtained edge restoration network, a generated edge map is obtained.

[0019] The color restoration unit takes the restored generated edge map, the damaged image, and the mask image as input to a deep learning-based color filling network to guide the restoration of image colors. Based on the generated edge map obtained in the previous step, the generator of the color filling network predicts the restored image. Finally, a discriminator is used to evaluate the quality of the generated image, and the color filling network is continuously optimized by calculating a loss function. In the end, a near-realistic restored image is obtained.

[0020] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the memory, wherein the processor executes the program to implement the image inpainting method based on multi-scale attention mechanism for edge prediction.

[0021] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned image inpainting method based on multi-scale attention mechanism for edge prediction.

[0022] Compared with the prior art, the beneficial effects of the present invention are:

[0023] (1) This invention provides an image restoration method and system based on edge prediction using a multi-scale attention mechanism. The scheme adopts the idea of ​​first restoring lines and then filling in colors. First, the edge information of the image is restored, which provides the image contour for subsequent color information restoration. In the edge restoration stage, a multi-scale attention mechanism is used to fully consider the combination of image structure and details, which effectively ensures the accuracy of the restoration result.

[0024] (2) The method has simple design steps and is highly practical and operable.

[0025] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0027] Figure 1 This is a flowchart of an image inpainting method based on multi-scale attention mechanism according to an embodiment of the present invention.

[0028] Figure 2 This is a schematic diagram of the overall framework of an image inpainting method based on multi-scale attention mechanism according to an embodiment of the present invention;

[0029] Figure 3 This is a diagram illustrating the multi-scale attention mechanism structure of an image inpainting method based on multi-scale attention mechanism for edge prediction, as described in an embodiment of the present invention.

[0030] Figure 4 The image restoration effect diagram is shown in the embodiment of the present invention, which describes an image restoration method based on multi-scale attention mechanism edge prediction.

[0031] Figure 5This is a comparison diagram of an image inpainting method based on multi-scale attention mechanism described in this embodiment of the invention and existing methods in the case of missing parts in a small area of ​​the image.

[0032] Figure 6 This is a comparison diagram of an image inpainting method based on multi-scale attention mechanism based on edge prediction described in this embodiment of the invention and existing methods in the case of large-area image loss. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0035] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0036] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0037] Example 1:

[0038] The purpose of this embodiment is to provide an image inpainting method based on edge prediction using a multi-scale attention mechanism.

[0039] like Figure 1 As shown, an image inpainting method based on edge prediction using a multi-scale attention mechanism includes:

[0040] The damaged image to be repaired and its corresponding mask image are used as input to a pre-trained edge repair network to obtain the repaired edge map. The edge repair network specifically performs the following processing steps: First, the damaged image is extracted using an edge detection algorithm. Then, the generator performs two downsampling operations on the damaged edge map to obtain shallow features of the edge image. Next, a multi-scale attention mechanism is introduced to obtain a refined edge feature map. Then, after processing by a residual module, an optimized refined edge map is obtained. Subsequently, it undergoes two upsampling operations by sequentially connected deconvolution modules to obtain the complete repaired edge map. Finally, a discriminator assigns true and false labels to the generated edge map and the real edge map, continuously optimizing its discrimination ability by calculating a loss function. After multiple training iterations, a trained edge repair network is obtained. Based on the obtained edge repair network, a generated edge map is obtained.

[0041] The restored edge map, damaged image, and mask image are used together as input to a deep learning-based color filling network to guide image color restoration. Based on the generated edge map obtained in the previous step, the generator of the color filling network predicts the restored image. Finally, a discriminator is used to evaluate the quality of the generated image, and the color filling network is continuously optimized by calculating the loss function. Finally, a near-realistic restored image is obtained.

[0042] In specific implementation, the introduction of a multi-scale attention mechanism to obtain a refined edge feature map involves: first, inputting the obtained shallow features into several branch networks to obtain several feature maps of different scales; then, combining the feature maps of different scales to obtain a multi-scale feature map; next, performing multi-channel compression on the obtained multi-scale feature map to obtain the weight vector corresponding to each channel; finally, performing channel weighting operation based on the obtained multi-scale feature map and the weight vector corresponding to each channel to obtain a refined edge feature map.

[0043] In practice, the branch networks are composed of convolution operations with different dilation rates.

[0044] In specific implementation, the method of combining several feature maps of different scales to obtain a multi-scale feature map is achieved by cascading feature maps of different scales.

[0045] In specific implementation, the residual module performs the processing, which involves establishing direct connections between layers so that gradients can be directly propagated back to shallower layers, thereby alleviating the gradient vanishing or gradient exploding problem.

[0046] In a specific implementation, the edge repair network includes a generator and a discriminator. The generator of the edge repair network includes a sequentially connected convolutional module that performs downsampling, a multi-scale attention module, a residual module, and a sequentially connected deconvolutional module that performs upsampling. The discriminator of the edge repair network includes a sequentially connected convolutional module with different parameter settings than the generator.

[0047] In a specific implementation, the deep learning-based color filling network includes: a sequentially connected convolutional module for downsampling, a residual module, and a sequentially connected deconvolutional module for upsampling; wherein, the convolutional module is used to encode the input image into a feature representation; the residual module is used to reduce the occurrence of gradient vanishing or gradient exploding problems and to realize feature mapping and transformation; the deconvolutional module is used to decode the mapped features into an output with the same size as the input image.

[0048] In practice, edge extraction is performed on the damaged image to obtain a damaged edge map, specifically using the Canny operator for edge extraction.

[0049] For ease of understanding, the solution described in this embodiment will be explained in detail below with reference to the accompanying drawings:

[0050] This embodiment describes an image inpainting method based on a multi-scale attention mechanism for edge prediction, which specifically includes the following two processing stages: the first stage is edge inpainting; the second stage is color filling. In the first stage, the edge inpainting network will inpaint the damaged image... The first stage uses the corresponding mask image M as input to the repair network, whose main function is to extract and repair the edge image; the second stage color filling network then processes the repaired edge image C. out Compared to previous damaged images The mask image M is also used as input, and the complete edge image C obtained in the previous stage is used in this process. out It can better help the network understand image information, thereby generating images that are closer to reality.

[0051] Specifically, an image inpainting method based on edge prediction using a multi-scale attention mechanism, such as... Figure 2 As shown, the process includes the following two stages.

[0052] Phase 1: Extracting and repairing edge maps of damaged images based on edge insulation networks.

[0053] Specifically, the following processing steps are included:

[0054] Step 1: Extract edges from the damaged image, and obtain the shallow features F of the edge map after two downsampling operations. sh ;

[0055] Step Two: As Figure 3As shown, a multi-scale attention mechanism is introduced, including:

[0056] (1) After extracting shallow features, four parallel branches are set up. Each branch is a convolution operation with different dilation rates to extract feature maps of different scales. Where i represents the branch number, with values ​​1, 2, 3, 4; j represents the expansion rate of the i-th branch, j = 1, 2, 4, 8; and k is the size of the convolution kernel. This indicates the use of dilated convolution with dilation rate j and kernel size k×k;

[0057] (2) Combining multi-scale feature maps And recalibrate the combined result F br weight vector w i (i = 1, 2, 3, 4; The four branches in step two (1) are recombined through cascading to obtain the feature map) The fused feature maps are then subjected to multi-channel compression operations, sequentially passing through pooling, fully connected (FC), ReLU, FC again, and the sigmoid function. Each channel's feature map is then compressed into a single value, which is the resulting weight value. The edge repair network then uses the Softmax function to distribute the weights. The new weight vector obtained after recalibration is: (w i (where w is the weight vector corresponding to the i-th feature map), and w i The weights in the formula are limited to between 0 and 1;

[0058] (3) The newly calibrated weight vector w i Corresponding multi-scale channel feature maps The final result obtained by performing channel weighting operation. ( (This represents a channel-weighted operation) which is a fine-grained edge feature map capable of capturing more complex details;

[0059] Step 3: Fine edge feature map F out After passing through the residual module, the processed feature map is obtained. The residual module mitigates the gradient explosion and vanishing problems that may occur during network training by backpropagating the network's gradients. Finally, two upsampling operations are performed to generate the repaired complete edge map C. out ;

[0060] Step 4: During training, the discriminator D1 receives real samples from the real dataset and samples generated by the generator, and labels them as real or fake. Then, the discriminator's predictions are measured by using a loss function, and the model parameters of the edge repair network are adjusted by an optimization algorithm until the model converges, thus obtaining an accurate and detailed generated edge map.

[0061] The loss function of discriminator D1 includes adversarial loss and feature matching loss;

[0062] Adversarial loss is used to evaluate the realism of the generated edge map, thereby ensuring that the generator can produce realistic repair results. The adversarial loss function is:

[0063]

[0064] Where E[.] represents the mathematical expectation; C gt This represents the true edge map extracted from a real image.

[0065] Feature matching loss measures the difference between the restored edge map and the ground truth edge map at the feature level, thereby ensuring that the restored image retains the structural and texture features of the original image. The feature matching loss function is defined as:

[0066]

[0067] Where ||·||1 represents the L1 norm, and T is the total number of layers. N represents the features extracted from the input image at the i-th layer of the discriminator. i This represents the number of elements in the i-th level;

[0068] The overall loss function of discriminator D1 in the edge repair network is:

[0069]

[0070] Where α and β are weights, the repair effect is best when α = 1 and β = 10;

[0071] The first-stage discriminator D1 is a 5-layer convolutional neural network that, through convolution and activation functions, discriminates the generated edge map C. out and the true edge map C gt Provide true and false labels.

[0072] Specifically, using I gtLet M represent the real image and M represent the mask image. In practical applications, the damaged image is converted to grayscale, and then the damaged areas are binarized to white (255) and the undamaged areas are binarized to black (0) according to a set threshold, thus obtaining the corresponding mask image M. The damaged image is represented as... Where ⊙ represents pixel-by-pixel multiplication.

[0073] Specifically, the first-stage edge repair network will repair the damaged image. Taking the mask image M as input, the corresponding damaged edge map of the damaged image is first extracted using the Canny operator; then, the generator (composed of an encoder, a multi-scale attention module, a residual module, and a decoder) downsamples the damaged edge map twice through the encoder to extract shallow features of the damaged edge map; subsequently, the multi-scale attention (MSA) module is used to realize multi-scale processing and combination of image features, and then... Figure 2 The ResBlock module in the model is used to reduce the occurrence of vanishing and exploding gradients. Then, the decoder performs two upsampling operations through deconvolution layers to decode the mapped features into generated edge maps, resulting in the repaired complete generated edge map. Where G1 represents the generator; finally, the discriminator converts the generated edge map C... out Assigning true and false labels to real edge maps: For each input edge map, an output probability value between (0, 1) is generated. When the probability value exceeds a preset threshold (set according to the actual situation), the discriminator classifies the input edge map as a real sample; if the probability value is lower than the threshold, it is classified as a generated sample. The discriminator continuously optimizes its discrimination ability by calculating the loss function. After multiple training sessions, the performance of this model tends to stabilize, and it can obtain generated edge maps with accurate edge repair and rich details.

[0074] The second stage involves restoring the image colors of the repaired edge map based on the constructed color filling network.

[0075] Specifically, the processing steps include the following:

[0076] Step 5: Input from the first stage With output C out The combination of these elements serves as input to the second-stage color inpainting network generator G2. G2 comprises an encoder, residual modules, and a decoder: the encoder encodes the input image into feature representations through convolutional layers; then, the features are processed through eight residual blocks to enhance their representational power; the decoder decodes the mapped features through deconvolutional layers to obtain the restored image. This generates a complete color inpainting result.

[0077] Step Six: During training, the discriminator D2 receives real samples from the real dataset and samples generated by the generator, and labels them to indicate whether they are real or fake. Then, a loss function is used to measure the difference between the discriminator's predictions and the actual labels, and the model parameters of the color-filling network are adjusted accordingly to improve the model's performance and the quality of the generated samples.

[0078] In discriminator D2, the image inpainting loss function is a weighted sum of adversarial loss, perceptual loss, and style loss;

[0079] The adversarial loss is used to enhance the discriminative ability of the discriminator, prompting the generator to produce more realistic images. The adversarial loss function is:

[0080]

[0081] Where E[.] represents the mathematical expectation; I gt Represents a real image;

[0082] Perceptual loss ensures that the generated image is perceptually similar to the real image, thus better preserving the semantic information of the image. The perceptual loss function is defined as:

[0083]

[0084] Among them, C j H j W j C represents the size of the feature map of the j-th layer. j H represents the number of channels in the feature map. j W j These represent the height and width of the feature map, respectively. Denotes the activation function of the j-th layer;

[0085] Style loss is used to constrain the style of the generated image, making the generated image have similar style features to the real image. The style loss function is:

[0086]

[0087] in, The Gram matrix is ​​used to calculate the difference between the predicted results and the true values, thereby constructing the style loss.

[0088] The overall loss function of discriminator D2 in the second stage is:

[0089]

[0090] Where α, β, and γ are weights, the repair effect is optimal when α = 0.1, β = 0.1, and γ = 250.

[0091] The discriminator D2 in this stage is a 5-layer convolutional neural network. Through convolution and activation functions, it discriminates the generated image I. out and real image I gt Assign it a label that indicates it is real or fake.

[0092] The restored complete edge map is used as new input to guide the color restoration of the image. Based on this finer edge map, the restored image can be closer to the real image in both color and texture. The discriminator receives the restored image and the real image, and for each image, it gives a value in the range (0, 1), which is compared with a set threshold to indicate whether the input sample is real or fake, that is, to give them a real or fake label.

[0093] Through edge repair in the first stage and color filling in the second stage, this method can restore damaged images to a near-realistic effect.

[0094] To verify the effectiveness of the solution described in this embodiment, the following experiments were conducted.

[0095] This embodiment is validated by comparing the repair results with those of several other methods. The quantitative evaluation of this embodiment on the Places2 public dataset is shown in Tables 1 to 4.

[0096] Table 1 compares the L1loss evaluation metrics with other methods.

[0097]

[0098] Table 2 Comparison with other methods in terms of PSNR evaluation index

[0099]

[0100] Table 3 Comparison with other methods on SSIM evaluation metrics

[0101]

[0102] Table 4 Comparison with other methods in terms of FID evaluation index

[0103]

[0104]

[0105] The scheme described in this embodiment uses standard evaluation metrics—including L1 error, peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and Fréchet distance (FID)—to compare the repair results of the method disclosed herein with those of existing superior methods. In the table, Mask represents the mask coverage rate (the ratio of the occluded area to the entire image area), GConv, EdgeConnect, and MAPM are all existing superior methods, Ours is the method disclosed herein, and the bold part indicates the optimal result after comparison.

[0106] The repair result obtained by the solution described in this embodiment is as follows: Figure 4 As shown, by comparing the images before and after restoration, it is clear that the damaged parts of the image have been effectively repaired. The restored image presents a clearer and more coherent visual effect, eliminating the original flaws and defects. Details in the damaged areas have been restored, and the overall image quality has been significantly improved.

[0107] Visual comparison results of repairs on smaller defect areas (10%–30%) are as follows: Figure 5 As shown in the figure. Experimental results show that when the defect area is small, the repair effects of each repair algorithm are similar, with only slight differences in the processing of image details.

[0108] Visual comparison results of repairs in larger defect areas (30%–60%) are as follows: Figure 6 As shown, when the missing area accounts for more than 30% of the overall image, the GConv algorithm exhibits obvious restoration artifacts, disrupting the original image structure. The content of the restored area is inconsistent with or unrelated to the surrounding area. In contrast, the EdgeConnect algorithm can produce clearer and smoother restoration results, but it is not accurate in restoring lines and fails to produce satisfactory prediction results when restoring images with sharp edges. The restored image often shows artifacts and blurred areas in the corresponding locations. The method disclosed in this paper can restore large areas of missing images with clear line content and can handle details while maintaining the overall consistency of the image, producing a more appropriate and vivid restoration effect.

[0109] The above experiments demonstrate the restoration effect of the method described in this embodiment and its comparison with other superior methods in image restoration. In summary, the image obtained using the solution described in this embodiment contains clearer line textures and details than other superior methods, and the visual effect is closer to the real image.

[0110] Example 2:

[0111] The purpose of this embodiment is to provide an image inpainting system based on edge prediction using a multi-scale attention mechanism.

[0112] An image inpainting system based on edge prediction using a multi-scale attention mechanism, comprising:

[0113] An edge restoration unit is used to take the damaged image to be restored and its corresponding mask image as input to a pre-trained edge restoration network to obtain the restored edge map. The edge restoration network specifically performs the following processing steps: First, the damaged image is extracted using an edge detection algorithm. Then, the generator performs two downsampling operations on the damaged edge map to obtain shallow features of the edge image. Next, a multi-scale attention mechanism is introduced to obtain a refined edge feature map. Then, after processing by a residual module, an optimized refined edge map is obtained. Subsequently, it undergoes two upsampling operations by sequentially connected deconvolution modules to obtain the restored complete edge map. Finally, a discriminator assigns true and false labels to the generated edge map and the real edge map, continuously optimizing its discrimination ability by calculating a loss function. After multiple training iterations, a trained edge restoration network is obtained. Based on the obtained edge restoration network, a generated edge map is obtained.

[0114] The color restoration unit takes the restored generated edge map, the damaged image, and the mask image as input to a deep learning-based color filling network to guide the restoration of image colors. Based on the generated edge map obtained in the previous step, the generator of the color filling network predicts the restored image. Finally, a discriminator is used to evaluate the quality of the generated image, and the color filling network is continuously optimized by calculating a loss function. In the end, a near-realistic restored image is obtained.

[0115] Furthermore, the system described in this embodiment corresponds to the method described in Embodiment 1, and its technical details have been described in detail in Embodiment 1, so they will not be repeated here.

[0116] In further embodiments, the following is also provided:

[0117] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When executed by the processor, the computer instructions perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0118] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0119] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0120] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0121] The method in Embodiment 1 can be directly implemented by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0122] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0123] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image inpainting method based on edge prediction using a multi-scale attention mechanism, characterized in that, include: The damaged image to be repaired and its corresponding mask image are used as input to a pre-trained edge repair network to obtain the repaired edge map. The edge repair network specifically performs the following processing steps: First, the damaged image is extracted using an edge detection algorithm. Then, the generator performs two downsampling operations on the damaged edge map to obtain shallow features of the edge image. Next, a multi-scale attention mechanism is introduced to obtain a refined edge feature map. Then, after processing by a residual module, an optimized refined edge map is obtained. Subsequently, it undergoes two upsampling operations by sequentially connected deconvolution modules to obtain the complete repaired edge map. Finally, a discriminator assigns true and false labels to the generated edge map and the real edge map, continuously optimizing its discrimination ability by calculating a loss function. After multiple training iterations, a trained edge repair network is obtained. Based on the obtained edge repair network, a generated edge map is obtained. The restored edge map, damaged image, and mask image are used together as input to a deep learning-based color filling network to guide image color restoration. Based on the generated edge map obtained in the previous step, the generator of the color filling network predicts the restored image. Finally, a discriminator is used to evaluate the quality of the generated image, and the color filling network is continuously optimized by calculating the loss function. Finally, a near-realistic restored image is obtained.

2. The image inpainting method based on edge prediction using a multi-scale attention mechanism as described in claim 1, characterized in that, The introduction of a multi-scale attention mechanism to obtain a fine edge feature map involves the following steps: First, the obtained shallow features are input into several branch networks to obtain feature maps of different scales; then, the feature maps of different scales are combined to obtain a multi-scale feature map; next, the obtained multi-scale feature map is subjected to multi-channel compression to obtain the weight vector corresponding to each channel; finally, a channel weighting operation is performed based on the obtained multi-scale feature map and the weight vector corresponding to each channel to obtain a fine edge feature map; wherein, the several branch networks are composed of convolution operations with different dilation rates.

3. The image inpainting method based on edge prediction using a multi-scale attention mechanism as described in claim 2, characterized in that, The method of combining several feature maps of different scales to obtain a multi-scale feature map is specifically achieved by cascading feature maps of different scales.

4. The image inpainting method based on edge prediction using a multi-scale attention mechanism as described in claim 1, characterized in that, The residual module's processing is as follows: the residual module establishes direct connections between layers, enabling gradients to be directly propagated back to shallower layers, thereby alleviating the gradient vanishing or gradient exploding problem.

5. The image inpainting method based on edge prediction using a multi-scale attention mechanism as described in claim 1, characterized in that, The edge repair network includes a generator and a discriminator. The generator includes a sequentially connected convolutional module for downsampling, a multi-scale attention module, a residual module, and a sequentially connected deconvolutional module for upsampling. The discriminator includes a sequentially connected convolutional module with different parameter settings than the generator.

6. The image inpainting method based on edge prediction using a multi-scale attention mechanism as described in claim 1, characterized in that, The deep learning-based color filling network includes: a sequentially connected convolutional module for downsampling, a residual module, and a sequentially connected deconvolutional module for upsampling; wherein, the convolutional module is used to encode the input image into a feature representation; the residual module is used to reduce the occurrence of gradient vanishing or gradient exploding problems and to realize feature mapping and transformation; the deconvolutional module is used to decode the mapped features into an output with the same size as the input image.

7. The image inpainting method based on edge prediction using a multi-scale attention mechanism as described in claim 1, characterized in that, The damaged image is processed to obtain a damaged edge map, specifically using the Canny operator for edge extraction.

8. An image inpainting system based on edge prediction using a multi-scale attention mechanism, characterized in that, include: An edge restoration unit is used to take the damaged image to be restored and its corresponding mask image as input to a pre-trained edge restoration network to obtain the restored edge map. The edge restoration network specifically performs the following processing steps: First, the damaged image is extracted using an edge detection algorithm. Then, the generator performs two downsampling operations on the damaged edge map to obtain shallow features of the edge image. Next, a multi-scale attention mechanism is introduced to obtain a refined edge feature map. Then, after processing by a residual module, an optimized refined edge map is obtained. Subsequently, it undergoes two upsampling operations by sequentially connected deconvolution modules to obtain the restored complete edge map. Finally, a discriminator assigns true and false labels to the generated edge map and the real edge map, continuously optimizing its discrimination ability by calculating a loss function. After multiple training iterations, a trained edge restoration network is obtained. Based on the obtained edge restoration network, a generated edge map is obtained. The color restoration unit takes the restored generated edge map, the damaged image, and the mask image as input to a deep learning-based color filling network to guide the restoration of image colors. Based on the generated edge map obtained in the previous step, the generator of the color filling network predicts the restored image. Finally, a discriminator is used to evaluate the quality of the generated image, and the color filling network is continuously optimized by calculating a loss function. In the end, a near-realistic restored image is obtained.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and running thereon, characterized in that, When the processor executes the program, it implements an image inpainting method based on edge prediction as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements an image inpainting method based on edge prediction as described in any one of claims 1-7.

Images