Image erasing method, electronic equipment, storage medium and computer program product

Abstract

The invention provides an image erasing method, electronic equipment, a storage medium and a computer program product. The method comprises the following steps: down-sampling a first image to obtain a second image; filling the selected area in the first image by using the second image to obtain a third image; the selected area in the third image is subjected to resolution restoration, a fourth image is obtained, and the resolution of the fourth image is the same as that of the first image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to an image erasing method, electronic device, storage medium, and computer program product. Background Technology

[0002] In related technologies, image erasing algorithms based on generative adversarial models are prone to problems such as image corruption and artifacts when erasing high-resolution images. Summary of the Invention

[0003] To address the related technical problems, embodiments of this application provide an image erasing method, an electronic device, a storage medium, and a computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an image erasing method, the method comprising: The first image is downsampled to obtain the second image; The selected area in the first image is filled in using the second image to obtain the third image; The selected area in the third image is subjected to resolution restoration to obtain a fourth image, wherein the resolution of the fourth image is the same as that of the first image.

[0005] In the above scheme, the step of filling the selected area in the first image with the second image to obtain the third image includes: The second image with the selected area added is input into the first network to obtain the fifth image; the first network is used to fill in the selected area of ​​the input image. The image content within the selected area of ​​the fifth image is overlaid onto the selected area of ​​the first image to obtain the third image.

[0006] In the above scheme, the first network includes an encoder, a patch matching layer, and a decoder. The step of inputting the second image with the selected region added into the first network to obtain the fifth image includes: The second image with the selected area added is input into the encoder to obtain the first feature map with the selected area added; The first feature map with the selected area added is input into the patch matching layer to obtain a second feature map with the selected area added; the patch matching layer is used to match and replace the selected area in the input image. The second feature map with the selected area added is input into the decoder to obtain the fifth image with the selected area added.

[0007] In the above scheme, the step of inputting the first feature map with the selected region added into the patch matching layer to obtain the second feature map includes: For each first local image block in one or more first local image blocks within the selected area of ​​the first feature map, the first local image block is replaced with a corresponding first image block to obtain the second feature map; wherein, the first image block represents the second local image block with the highest similarity to the first local image block among all the second local image blocks in the first feature map; the second local image block represents the local image block in the first feature map other than the first local image block.

[0008] In the above scheme, the step of performing resolution restoration on the selected area in the third image to obtain the fourth image includes: Gaussian noise is superimposed on the selected area in the third image to obtain the sixth image; The sixth image is input into the first model to obtain the fourth image; the first model is used to improve the resolution of some or all regions of the input image.

[0009] In the above scheme, the first model includes a second network and a third network. The second network is used for image denoising, and the third network is used to constrain and control the image denoising process of the second network.

[0010] In the above scheme, the second network includes a stable diffusion model, and the third network is the ControlNet control network.

[0011] This application also provides an electronic device, including a processor and a memory for storing a computer program capable of running on the processor. When the processor runs the computer program, it executes the steps of any of the above methods.

[0012] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0013] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0014] In the image erasing method, electronic device, storage medium, and computer program product provided in this application embodiment, a first image is downsampled to obtain a second image; a selected area in the first image is filled using the second image to obtain a third image; and the selected area in the third image is resolution-restored to obtain a fourth image, wherein the resolution of the fourth image is the same as that of the first image. This scheme reduces the image resolution through downsampling, erases the original image based on the reduced-resolution image, and then restores the resolution of the erased area. This image erasing method can be applied to high-resolution images, and by erasing using a low-resolution image, it avoids problems such as image corruption and artifacts that are prone to occur during high-resolution image processing. Attached Figure Description

[0015] Figure 1 This is a schematic flowchart of an image erasing method provided in an embodiment of this application; Figure 2 A first network input / output example diagram provided for an embodiment of this application; Figure 3 This is a schematic diagram of a patchmatch algorithm provided in an embodiment of this application; Figure 4 An example diagram illustrating the execution effect of a patch matching layer provided in this application embodiment; Figure 5 A schematic diagram of a first model structure provided in an embodiment of this application; Figure 6 This is a schematic diagram of an image erasing device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation

[0016] Current image erasure algorithms mainly include the following three types: Image erasing algorithms based on Generative Adversarial Networks (GANs) train the image erasing function using the GAN framework and learn to fill in the erasing area using the generative adversarial model. Generally, this algorithm is fast and efficient, but it is prone to problems such as artifacts and other issues. The higher the image resolution, the more severe these problems become. Therefore, current GAN-based solutions can only erase low-resolution images. If used to erase high-resolution images, there will be obvious flaws in the erased parts, and there is also the problem of insufficient resolution and texture.

[0017] Based on the traditional patch matching algorithm, this algorithm replaces the erased block by searching and matching patches. That is, it uses an image patch information matching algorithm to globally search and match the most suitable block to fill the erased area. Although the image resolution will not be significantly different after repair by this algorithm, it can only find image blocks in the image to fill the erased part. It does not have the ability to repair the damaged object structure, and often pastes seemingly reasonable but actually discordant image blocks onto the erased part.

[0018] Image erasure algorithms based on stable diffusion models utilize the encoding and decoding capabilities of large models, as well as their understanding of semantics and textures, to achieve image erasure. Although large models have strong creativity, their creativity is uncontrollable and can easily generate unwanted objects to achieve erasure.

[0019] Based on this, in various embodiments of this application, a first image is downsampled to obtain a second image; the second image is used to fill in selected areas in the first image to obtain a third image; the selected areas in the third image are then resolution-restored to obtain a fourth image, wherein the resolution of the fourth image is the same as that of the first image. This scheme reduces image resolution through downsampling, erases the original image based on the reduced-resolution image, and then restores the resolution of the erased area. This image erasing method can be applied to high-resolution images, and by erasing using a low-resolution image, it avoids problems such as image corruption and artifacts that are prone to occur during high-resolution image processing.

[0020] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0021] This application provides an image erasing method applied to electronic devices. For example... Figure 1 As shown, the method includes: Step 101: Downsample the first image to obtain the second image.

[0022] Here, the first image is the original image. Downsampling the first image yields the second image. A mask can be used to add selection areas to both the first and second images. When the mask is applied to the first and / or second images, its size is adjusted to match the size of the image to be covered. The mask is used to identify the area or object to be erased. The selection area is the operable region within the mask of the original image; it can be understood as the area to be erased, such as the white area of ​​the mask. A mask is also called a mask, a texture mask, or a texture mask. The selection area can be manually selected by the user, for example, the area the user paints is the selection area; or it can be automatically identified by the electronic device, for example, automatically identifying a person in an image, and the area occupied by the person is the selection area. Downsampling the first image reduces the image resolution, meaning the second image has a lower resolution than the first image. The first image is a high-resolution image, such as a 4K image, capable of displaying detailed images and text with strong detail; the second image is a low-resolution image, with a blurred image and potentially jagged text, such as a 32×32 pixel image.

[0023] Step 102: Use the second image to fill in the selected area in the first image to obtain the third image.

[0024] Here, given the second image, the selected area in the first image is filled in using the second image. Specifically, suitable image blocks can be selected from the second image and filled into the selected area in the first image; for example, image blocks in the second image with colors similar to the surrounding area of ​​the selected area can be selected; another example is using image recognition technology to select image blocks in the second image that contain objects that are the same as or similar to the surrounding objects of the selected area; yet another example is selecting image blocks in the second image with high similarity to the selected area image blocks by calculating similarity. The filled content in the selected area in the resulting third image is obtained by filling in the second image, and the areas in the third image other than the selected area remain unchanged, that is, the areas other than the selected area are the original content of the first image.

[0025] Step 103: Perform resolution restoration on the selected area in the third image to obtain the fourth image.

[0026] The resolution of the fourth image is the same as that of the first image.

[0027] Here, given the third image, since the resolution of the selected area in the third image is lower than that of other areas in the third image, resolution restoration is needed for the selected area. This means increasing the resolution of the selected area so that it matches the resolution of other areas, achieving high-resolution restoration. For example, pixel filling can be used to improve the resolution of the selected area in the third image; another example is improving the resolution by extracting image edge information to recover high-frequency details; yet another example is using a trained deep learning network to improve the resolution of the selected area in the third image.

[0028] To accurately determine suitable image blocks from the second image to fill in selected areas in the first image, thereby improving both the filling and erasing effects, in one embodiment, the process of using the second image to fill in selected areas in the first image to obtain a third image includes: The second image with the selected area added is input into the first network to obtain the fifth image; the first network is used to fill in the selected area of ​​the input image. The image content within the selected area of ​​the fifth image is overlaid onto the selected area of ​​the first image to obtain the third image.

[0029] Here, given the second image, the selected area of ​​the second image is input into the first network to obtain the fifth image. The fifth image can be understood as the second image after the selection area is filled in; that is, the selected area in the second image is filled in using the second image to obtain the fifth image. The second image with the selected area can be a second image with a mask added. Inputting the second image with the selected area into the first network can be done by first covering the second image with a mask and then inputting the masked second image into the first network, or by inputting the mask and the second image into the first network simultaneously. The first network is used to fill in the selected area of ​​the input image using the input image; the first network can be a Generative Adversarial Network (GAN), for example, patchGAN; the second image with the selected area can be understood as a second image with a mask added.

[0030] For example, such as Figure 2As shown, the top-left image is the second image obtained by downsampling the first image. The person in the second image needs to be erased. The bottom-left image is a mask, where the white area occupied by the person is the opaque area (the operable area), and the black area is the transparent area. The mask is then applied to the second image, resulting in a second image with a selection added. The white area of ​​the mask overlaps and covers the person in the second image, representing the area to be erased, while the black area of ​​the mask is invisible on the second image. The second image with the selection added is then input into the first network to obtain the fifth image. Figure 2 The image to the right of the first network in the fifth image shows the selected area now filled. The resolution of the fifth image is usually the same as that of the second image. The fifth image can be understood as the filled second image.

[0031] Given the fifth image, the image content within the selected area of ​​the fifth image is overlaid onto the selected area of ​​the first image to obtain the third image. The fifth image is obtained based on the second image obtained by downsampling the first image, that is, the resolution of the fifth image is lower than that of the first image. When overlaying the image content within the selected area of ​​the fifth image onto the selected area of ​​the first image, the selected area in the fifth image needs to be enlarged to the same size as the selected area in the first image, and the enlarged selected area in the fifth image is overlaid onto the selected area of ​​the first image.

[0032] The training data for the first network includes image pairs, one of which contains an image with erased objects and the other contains an image without erased objects. The other image can be obtained by taking a picture with a camera at the same angle, removing only the erased objects within the shooting range; or the erased objects can be manually removed from the image containing erased objects to obtain the other image in the image pair; for example, by removing erased objects using Photoshop.

[0033] In this embodiment, using a low-resolution second image as input to the first network enables rapid processing of the erased portion of the low-resolution image. The first network, employing the patchGAN structure, can perform lightweight low-resolution erasure on the input image. When erasing low-resolution images, the patchGAN network is less prone to artifacts and other problems.

[0034] To make the patching more natural and reasonable, in one embodiment, the first network includes an encoder, a patch matching layer, and a decoder. The step of inputting the second image with the selected region added into the first network to obtain the fifth image includes: The second image with the selected area added is input into the encoder to obtain the first feature map with the selected area added; The first feature map with the selected area added is input into the patch matching layer to obtain a second feature map with the selected area added; the patch matching layer is used to match and replace the selected area in the input image. The second feature map with the selected area added is input into the decoder to obtain the fifth image with the selected area added.

[0035] Here, the first network may include an encoder, a patch matching layer, and a decoder. The patch matching layer is located between the encoder and the decoder and is used to perform image patch matching and replacement on selected regions in the feature map output by the last layer of the encoder. The patch matching layer is also called the patch match layer. The second feature map can be understood as the padded feature map. The padded feature map is input into the decoder to obtain the fifth image.

[0036] The patch matching layer is based on the traditional patchmatch algorithm, which includes three steps: initialization, propagation, and search. Figure 3 As shown; where initialization includes initializing the matching relationship between each image patch in the source image and the image patches in the target image. Figure 3 Image A is the source image, and image B is the target image. Taking three image patches represented by solid-line boxes, dashed-line boxes, and dotted boxes as an example, the currently processed image patch is the one corresponding to the dotted box. The arrows reflect the matching relationships of the three image patches in the source and target images, respectively. Propagation involves identifying the image patch with the highest similarity among the three image patches in the target image and the one corresponding to the dotted box in the source image. Search involves selecting an image patch centered on the image patch with the highest similarity identified in the propagation step, within an exponentially decaying radius, and matching it with the image patch corresponding to the dotted box in the source image to determine the image patch that best matches the one corresponding to the dotted box in the source image.

[0037] The encoder is used to extract key features from the input image, and the decoder is used to generate new data based on the input feature map. For example, it generates new image textures for filling content within a selected area. The patch matching layer is deployed between the encoder and decoder of the first network, meaning that the matching and filling operation is only performed on the feature layer. This allows the first network to have the matching ability and fast convergence effect of traditional patch matching while retaining the ability of GAN networks to generate new textures. Furthermore, the second image processed by the first network is a low-resolution image, avoiding the artifacts and other problems that the first network is prone to when processing high-resolution images. This makes the capabilities of the first network not limited by image pixels.

[0038] To maintain image consistency and spatial structure rationality during image erasure, in one embodiment, the step of inputting the first feature map with the selected region added to it into the patch matching layer to obtain a second feature map includes: For each first local image block in one or more first local image blocks within the selected area of ​​the first feature map, the first local image block is replaced with a corresponding first image block to obtain the second feature map; wherein, the first image block represents the second local image block with the highest similarity to the first local image block among all the second local image blocks in the first feature map; the second local image block represents the local image block in the first feature map other than the first local image block.

[0039] Here, the first feature map includes multiple local image blocks. From the multiple second local image blocks in the first feature map, the first image block with the highest similarity to each first local image block in the selected area of ​​the first feature map is determined. Then, each first local image block in the selected area of ​​the first feature map is replaced with the corresponding first image block with the highest similarity to obtain the second feature map. The similarity can be cosine similarity. Each local image block in the selected area carries effective pixel blocks around the selected area.

[0040] For example, such as Figure 4 As shown, taking the local image patch corresponding to the black box within the selected area (white region) of the first feature map as an example, the local image patch with the highest similarity to the first local image patch corresponding to the black box in the first feature map is the first image patch corresponding to the white box. Replacing the first local image patch corresponding to the black box with the first image patch corresponding to the white box yields the following result. Figure 4 The rightmost image shows the effect after replacement.

[0041] To activate high-frequency information within the selected area, avoid loss of texture information during resolution enhancement, and improve the rationality of the erased area, in one embodiment, the resolution restoration of the selected area in the third image to obtain a fourth image includes: Gaussian noise is superimposed on the selected area in the third image to obtain the sixth image; The sixth image is input into the first model to obtain the fourth image; the first model is used to improve the resolution of some or all regions of the input image.

[0042] Here, the filled content within the selected area in the third image has low resolution and blurry texture. Therefore, resolution restoration is performed on the selected area in the third image, i.e., the resolution of the selected area is increased. A small amount of Gaussian noise can be superimposed on the selected area in the third image. For example, the standard deviation of the superimposed Gaussian noise should be between 0.1 and 0.5, with the specific value determined based on the actual application. From a visualization perspective, the texture of the selected area in the sixth image obtained after superimposing a small amount of Gaussian noise should be barely visible. The sixth image is then input into the first model, which automatically increases the resolution of the selected area in the sixth image, making the resolution of all regions in the image consistent, resulting in the fourth image. The fourth image is a high-resolution image, thus achieving high-resolution image restoration.

[0043] To maintain structural consistency between the high-resolution local image generated within the selected area and the non-erased area, in one embodiment, the first model includes a second network and a third network, wherein the second network is used for image denoising, and the third network is used to constrain and control the image denoising process of the second network.

[0044] The first model includes a second network and a third network. The sixth image is used as input to both the second and third networks simultaneously; that is, the sixth image is input into the second and third networks respectively, resulting in the fourth image output by the second network. The number of sampling and denoising iterations in the second network can be determined based on the actual situation; for example, it can be set to 20 iterations.

[0045] The first model and the first network can be trained using the same training set, that is, they are trained using the same set of image pairs.

[0046] In one embodiment, the second network includes a stable diffusion model, and the third network is a ControlNet control network.

[0047] Here, the second network can be a Stable Diffusion (SD) model, and the third network can be a ControlNet. That is, the first model can use a network architecture combining an SD model and a ControlNet. For example, the network architecture of the first model is as follows: Figure 5As shown, the left half is the second network, which includes 12 encoder blocks (SDEncoder Block), 12 decoder blocks (SD Decoder Block), and one middle block (SD Middle Block). The weights of the second network are fixed. The right half is the third network, which includes 12 encoder blocks (trainable copies) and middle blocks (trainable copies) copied from the second network, as well as multiple zero convolutional layers. The weights of the third network are adjustable. Because the weights of the second network are locked, the first model does not require gradient calculation training of the encoder of the second network, which improves the training speed of the model. Furthermore, the stable diffusion model with fixed weights will not damage the high-resolution image. Therefore, by using the first model, which includes the second and third networks, to perform resolution restoration on the image, a high-definition image with texture restoration can be obtained.

[0048] The following section provides a more detailed description of this application with reference to application examples.

[0049] The image erasure method is applied to electronic devices and includes the following steps: Step 1: Downsample the first image to obtain the second image.

[0050] Here, the first image is a high-resolution image to be erased, containing the object to be erased. The first image is downsampled and reduced in resolution, enabling the GAN network to effectively erase the object to be erased in the second image, thus preventing artifacts and other problems that easily occur when the GAN network directly processes high-resolution images.

[0051] Step 2: Input the second image with the selected area added into the first network to obtain the fifth image.

[0052] Here, the second image with the added selection area can be understood as a second image with a mask, that is, the mask marking the object to be erased is overlaid on the second image to obtain the second image with the added selection area; alternatively, the mask marking the object to be erased and the second image can be directly input into the first network simultaneously. The first network uses a patchGAN structure. Unlike the traditional patchGAN structure, the first network in this embodiment also includes a patch matching layer. The patch matching layer is deployed between the encoder and decoder. That is, the patch matching layer acts on the feature map output by the last layer of the encoder to perform local matching on the feature map, find the image patch with the highest cosine similarity to one or more local image patches in the selection area, and replace one or more local image patches in the selection area with the corresponding image patch with the highest cosine similarity. By adding the patch matching layer to the patchGAN structure, the first network not only has the good matching ability of the traditional patch matching algorithm, but also has the ability of a GAN network to generate new textures because the matching and filling is only performed at the feature layer. The output result of the patchGAN network is a matrix rather than real numbers.

[0053] Step 3: Overlay the image content within the selected area of ​​the fifth image onto the selected area of ​​the first image to obtain the third image.

[0054] Here, the fifth image is a low-resolution image after filling, that is, the holes in the selected area of ​​the fifth image have been filled, and the selected area in the fifth image is enlarged to the size of the selected area in the first image; specifically, only the selected area in the fifth image can be enlarged, or the entire fifth image can be enlarged to the size of the first image, and then the selected area of ​​the enlarged fifth image is extracted; when the selected area in the enlarged fifth image is obtained to the same size as the selected area in the first image, the selected area in the enlarged fifth image is overlaid on the selected area of ​​the first image to obtain the third image; the filled content of the selected area in the third image has low resolution and blurred texture.

[0055] Step 4: Add Gaussian noise to the selected area in the third image to obtain the sixth image.

[0056] Here, a small amount of Gaussian noise is superimposed on the selected area in the third image, so that the texture of the selected area in the third image is barely visible and has Gaussian noise, thus obtaining the sixth image.

[0057] Step 5: Input the sixth image into the first model to obtain the fourth image.

[0058] Here, the first model includes a second network and a third network. The second network is a stable diffusion model, and the third network is a ControlNet control network. The sixth image is input into the first model, that is, the sixth image is simultaneously input into the second and third networks. Using the sampling of the second network, multiple sampling denoising is performed (e.g., 20-times sampling denoising), resulting in the fourth image output by the second network. The fourth image is a high-resolution image with clear texture, such as a 4K image. Since high-resolution images are not damaged in the stable diffusion model, the stable diffusion model can be used directly for high-resolution image restoration.

[0059] In this application embodiment, the image to be erased is erased in two stages. The first stage is lightweight low-resolution image erasure (steps 1-3), and the second stage is two-stage high-resolution image restoration based on a control network (steps 4-5). This solves the problems of poor erasure effect and uncontrollable restoration of high-resolution images in current image erasure methods.

[0060] To implement the method of the embodiments of this application, the embodiments of this application also provide an image erasing device, such as... Figure 6 As shown, the device includes: The sampling unit 601 is used to downsample the first image to obtain the second image; The filling unit 602 is used to fill in the selected area in the first image using the second image to obtain a third image; The repair unit 603 is used to perform resolution repair on the selected area in the third image to obtain a fourth image, wherein the resolution of the fourth image is the same as the resolution of the first image.

[0061] In one embodiment, the filling unit 602 is specifically used to input the second image with the selected area added into the first network to obtain the fifth image; the first network is used to fill the selected area of ​​the input image; The image content within the selected area of ​​the fifth image is overlaid onto the selected area of ​​the first image to obtain the third image.

[0062] In one embodiment, the first network includes an encoder, a patch matching layer, and a decoder. The patching unit 602 is specifically used to input the second image with the selected region added into the encoder to obtain a first feature map with the selected region added. The first feature map with the selected area added is input into the patch matching layer to obtain a second feature map with the selected area added; the patch matching layer is used to match and replace the selected area in the input image. The second feature map with the selected area added is input into the decoder to obtain the fifth image with the selected area added.

[0063] In one embodiment, the filling unit 602 is specifically used to replace each first local image block in one or more first local image blocks within the selected area of ​​the first feature map with a corresponding first image block to obtain the second feature map; wherein, the first image block represents the second local image block with the highest similarity among all second local image blocks in the first feature map; the second local image block represents a local image block in the first feature map other than the first local image block.

[0064] In one embodiment, the repair unit 603 is specifically used to superimpose Gaussian noise onto the selected area in the third image to obtain a sixth image; The sixth image is input into the first model to obtain the fourth image; the first model is used to improve the resolution of some or all regions of the input image.

[0065] In one embodiment, the first model includes a second network and a third network, wherein the second network is used for image denoising, and the third network is used for constraining and controlling the image denoising process of the second network.

[0066] In one embodiment, the second network includes a stable diffusion model, and the third network is a ControlNet control network.

[0067] In practical applications, the sampling unit 601, the filling unit 602, and the repair unit 603 can be implemented by a processor in the image erasing device.

[0068] It should be noted that the image erasing device provided in the above embodiments is only illustrated by the division of the above-described program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the image erasing device and the image erasing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0069] Based on the hardware implementation of the above program modules, embodiments of this application also provide an electronic device, such as... Figure 7 As shown, the electronic device 700 includes: The communication interface 701 allows for information exchange with other devices or network nodes.

[0070] The processor 702 is connected to the communication interface 701 to enable information interaction with other devices or network nodes, and to execute the methods provided by one or more technical solutions in the above embodiments when running computer programs.

[0071] Memory 703 is used to store computer programs that can run on processor 702.

[0072] Specifically, the processor 702 is used to downsample the first image to obtain the second image; The selected area in the first image is filled in using the second image to obtain the third image; The selected area in the third image is subjected to resolution restoration to obtain a fourth image, wherein the resolution of the fourth image is the same as that of the first image.

[0073] In one embodiment, the processor 702 is specifically used to input the second image with the selected area added into the first network to obtain a fifth image; the first network is used to fill the selected area of ​​the input image; and the image content within the selected area of ​​the fifth image is overlaid onto the selected area of ​​the first image to obtain the third image.

[0074] In one embodiment, the first network includes an encoder, a patch matching layer, and a decoder. The processor 702 is specifically used to input the second image with the selected region added into the encoder to obtain a first feature map with the selected region added. The first feature map with the selected area added is input into the patch matching layer to obtain a second feature map with the selected area added; the patch matching layer is used to match and replace the selected area in the input image. The second feature map with the selected area added is input into the decoder to obtain the fifth image with the selected area added.

[0075] In one embodiment, the processor 702 is specifically configured to replace each first local image block in one or more first local image blocks within the selected area of ​​the first feature map with a corresponding first image block to obtain the second feature map; wherein, the first image block represents the second local image block with the highest similarity among all second local image blocks in the first feature map; and the second local image block represents a local image block in the first feature map other than the first local image block.

[0076] In one embodiment, the processor 702 is specifically configured to superimpose Gaussian noise onto the selected area in the third image to obtain a sixth image; input the sixth image into a first model to obtain the fourth image; the first model is used to improve the resolution of some or all areas of the input image.

[0077] In one embodiment, the first model includes a second network and a third network, wherein the second network is used for image denoising, and the third network is used for constraining and controlling the image denoising process of the second network.

[0078] In one embodiment, the second network includes a stable diffusion model, and the third network is a ControlNet control network.

[0079] It should be noted that the specific processing procedure of processor 702 can be understood by referring to the above method.

[0080] Of course, in practical applications, the various components in electronic device 700 are coupled together through bus system 704. It can be understood that bus system 704 is used to realize the connection and communication between these components. In addition to a data bus, bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 7 The general designated all buses as Bus System 704.

[0081] The memory 703 in this embodiment is used to store various types of data to support the operation of the electronic device 700. Examples of such data include any computer program used to operate on the electronic device 700.

[0082] The methods disclosed in the embodiments of this application can be applied to, or implemented by, the processor 702. The processor 702 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware or by instructions in software form within the processor 702. The processor 702 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 702 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically memory 703. The processor 702 reads information from memory 703 and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0083] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0084] It is understood that the memory (memory 703) in this embodiment of the application can be volatile memory or non-volatile memory, or it can include both volatile memory and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0085] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 703 storing a computer program, which can be executed by the processor 702 of the electronic device 700 to complete the steps described in any of the foregoing methods. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0086] For example, embodiments of this application also provide a computer program product, including a computer program that can be executed by a processor 702 of an electronic device 700 to perform the steps described in any of the foregoing methods.

[0087] It should be noted that terms such as "first" and "second" are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. The term "and / or" in this document merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the technical solutions described in the embodiments of this application can be arbitrarily combined without conflict. The above descriptions are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application.

Claims

1. An image erasing method, characterized in that, The method includes: The first image is downsampled to obtain the second image; The selected area in the first image is filled in using the second image to obtain the third image; The selected area in the third image is subjected to resolution restoration to obtain a fourth image, wherein the resolution of the fourth image is the same as that of the first image.

2. The method according to claim 1, characterized in that, The step of filling in the selected area in the first image with the second image to obtain the third image includes: The second image with the selected area added is input into the first network to obtain the fifth image; the first network is used to fill in the selected area of ​​the input image. The image content within the selected area of ​​the fifth image is overlaid onto the selected area of ​​the first image to obtain the third image.

3. The method according to claim 2, characterized in that, The first network includes an encoder, a patch matching layer, and a decoder. The step of inputting the second image with the selected region added into the first network to obtain the fifth image includes: The second image with the selected area added is input into the encoder to obtain the first feature map with the selected area added; The first feature map with the selected area added is input into the patch matching layer to obtain a second feature map with the selected area added; the patch matching layer is used to match and replace the selected area in the input image. The second feature map with the selected area added is input into the decoder to obtain the fifth image with the selected area added.

4. The method according to claim 3, characterized in that, The step of inputting the first feature map with the selected region added into the patch matching layer to obtain the second feature map includes: For each first local image block in one or more first local image blocks within the selected area of ​​the first feature map, the first local image block is replaced with a corresponding first image block to obtain the second feature map; wherein, the first image block represents the second local image block with the highest similarity to the first local image block among all the second local image blocks in the first feature map; the second local image block represents the local image block in the first feature map other than the first local image block.

5. The method according to any one of claims 1, characterized in that, The step of performing resolution restoration on the selected area in the third image to obtain the fourth image includes: Gaussian noise is superimposed on the selected area in the third image to obtain the sixth image; The sixth image is input into the first model to obtain the fourth image; the first model is used to improve the resolution of some or all regions of the input image.

6. The method according to claim 5, characterized in that, The first model includes a second network and a third network, wherein the second network is used for image denoising, and the third network is used for constraining and controlling the image denoising process of the second network.

7. The method according to claim 6, characterized in that, The second network includes a stable diffusion model, and the third network is the ControlNet control network.

8. An electronic device, characterized in that, This includes a processor and memory for storing computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images