Image inpainting method, image inpainting device, electronic device, and storage medium

Abstract

The application discloses an image repairing method, an image repairing device, an electronic device and a storage medium. The method comprises the following steps: obtaining an erasing mask image of an erasing area of a to-be-erased image; obtaining an erasing image according to the to-be-erased image and the erasing mask image; inputting the erasing image and a prompt word into an image repairing model to obtain a repaired image, wherein the prompt word represents foreground objects and / or background objects of a neighboring area, the neighboring area refers to an area adjacent to the to-be-erased image area in the to-be-erased image, and the image repairing model is used for completing and repairing the erasing area of the erasing image according to the prompt word. The method realizes image completion of the erasing area of the erasing image according to the erasing image and the prompt word based on the image repairing model, so that the image repairing model completes the image according to the prompt word, and the completed image is more integrated with the foreground and / or background objects of the neighboring area in the erasing area of the erasing image, thereby improving the image completion effect of the repaired image and improving the image repairing quality.

Description

[0001] This application claims priority to Chinese Patent Application No. 202410855375.X, filed on June 27, 2024, entitled "Image Restoration Method, Image Restoration Apparatus, Electronic Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and in particular to an image restoration method, an image restoration device, an electronic device, and a storage medium. Background Technology

[0003] Image erasure is a technique that removes image content from a certain area of ​​the original image and then fills in the area with appropriate image content.

[0004] Currently, image erasure is mainly performed on the original image using the Stable Diffusion (SD) model. During the training of the SD model, the original image is input into the model. Random noise is added to the original image, and the model learns from the noise-added image to estimate the added noise value. The original image is then subtracted from the estimated noise value to obtain the restored image.

[0005] However, the SD model focuses on completing image content during training. When erasing image content, the SD model generates image content that is unrelated to the original image, resulting in low restoration quality. Summary of the Invention

[0006] In view of the above, embodiments of this application provide an image restoration method, an image restoration apparatus, an electronic device, and a storage medium to overcome the problems of the prior art.

[0007] In a first aspect, embodiments of this application provide an image restoration method, which includes: obtaining an erasure mask image of the area to be erased in an image to be erased; obtaining an erasure image based on the image to be erased and the erasure mask image; inputting the erasure image and a prompt word into an image restoration model to obtain a restored image, wherein the prompt word represents a foreground object and / or a background object in an adjacent region, and the adjacent region refers to a region in the image to be erased that is adjacent to the area to be erased, and the image restoration model is used to complete and restore the erasure area of ​​the erasure image based on the prompt word.

[0008] The solution provided in this application involves an image restoration model that completes the erased area of ​​an erased image based on the erased image and prompt words. The prompt words indicate foreground and / or background objects in adjacent areas. This allows the image restoration model to complete the erased area of ​​the erased image in a more integrated manner with the foreground and / or background objects in adjacent areas, avoiding random generation during the image restoration process, improving the image restoration effect, and enhancing the image restoration quality.

[0009] In some optional embodiments, the image restoration model includes a text encoding model and a stable diffusion model. The text encoding model is used to encode the prompt words to obtain prompt word features, and the stable diffusion model is used to complete and restore the erased image based on the prompt word features to obtain a restored image.

[0010] The solution provided in this embodiment, based on a stable diffusion model, completes the erased area of ​​the erased image by incorporating foreground and / or background objects in adjacent regions, resulting in a more integrated image. This avoids the random generation phenomenon that occurs when the stable diffusion model performs image completion and repair on the erased area based on user-inputted prompts unrelated to foreground and / or background objects in adjacent regions. This improves the image completion effect and enhances the image repair quality.

[0011] In some optional embodiments, inputting the erased image and prompt words into the image restoration model to obtain the restored image includes: adding noise to the erased image to obtain a noisy image; inputting the noisy image and prompt words into the image restoration model to obtain the restored image.

[0012] The solution provided in this embodiment obtains a noisy image by adding noise to the erased image, thereby increasing the reference information contained in the noisy image. Then, based on a stable diffusion model, the reference information of the noisy image is used for denoising, which further suppresses the random generation phenomenon in the image restoration process and further improves the image restoration quality.

[0013] In some optional embodiments, the noisy image includes a first noisy image. Before the erased image is denoised to obtain the noisy image, the image restoration method further includes: determining whether the area to be erased contains a human image; denoising the erased image to obtain the noisy image includes: when it is determined that the area to be erased contains a human image, denoising the image to be erased according to a first rule to obtain the first noisy image.

[0014] The solution provided in this embodiment, in the scenario of erasing a human image, adds noise to the erased image based on a first rule to obtain a first noisy image, thereby increasing the reference information contained in the first noisy image. When using the reference information of the first noisy image for denoising based on the stable diffusion model, it is beneficial to further suppress the random generation phenomenon of the stable diffusion model and further improve the image restoration quality.

[0015] In some optional embodiments, when it is determined that the area to be erased contains a human image, the image to be erased is subjected to noise processing according to a first rule to obtain a first noise-added image, including: when it is determined that the area to be erased contains a human image, the image to be erased is subjected to noise processing for a first number of noise-added steps according to the first rule to obtain a first noise-added image.

[0016] In some optional embodiments, the first rule is used to characterize the rule that the noise intensity after each noise addition process satisfies the first noise formula; the first noise formula is: Where x0 is the noise intensity of the erased image, K is the number of noise additions, and α K Let α be the noise figure for the Kth noise addition. K ∈(0,1), α K The noise is uniformly distributed and is Gaussian noise.

[0017] In some optional embodiments, the noisy image further includes a second noisy image, and the image restoration method further includes: when it is determined that the area to be erased does not contain a human image, performing noise processing on the image to be erased according to a second rule to obtain a second noisy image, wherein the first noise step number corresponding to the first rule is greater than the second noise step number corresponding to the second rule.

[0018] The solution provided in this embodiment, in the case of erasing non-human images, adds noise to the erased image based on the second rule to obtain a second noisy image, thereby increasing the reference information contained in the second noisy image. When using the reference information of the second noisy image for denoising based on the stable diffusion model, it is beneficial to further suppress the random generation phenomenon of the stable diffusion model and further improve the image restoration quality.

[0019] In some optional embodiments, when it is determined that the area to be erased does not contain a human image, the image to be erased is subjected to noise processing according to the second rule to obtain a second noise-added image, including: when it is determined that the area to be erased does not contain a human image, the image to be erased is subjected to noise processing for a second number of noise-added steps according to the second rule to obtain a second noise-added image.

[0020] In some optional embodiments, the second rule is used to characterize the rule that the noise intensity after each noise addition process satisfies the second noise formula; the second noise formula is: Where x0 is the noise intensity of the erased image, Tα represents the number of times noise is added. T For the first T The noise figure of the second noise addition, α T ∈(0,1), α T The noise is uniformly distributed and is Gaussian noise.

[0021] In some optional embodiments, determining whether the area to be erased contains a human image includes: determining the erase mask category of the erase mask image; and determining whether the area to be erased contains a human image based on the erase mask category.

[0022] The solution provided in this embodiment determines whether the area to be erased contains a human image based on the type of erasure mask in the erasure mask image, thereby improving the accuracy of human image determination in the area to be erased.

[0023] In some optional embodiments, before obtaining the erasure image based on the image to be erased and the erasure mask image, the image restoration method further includes: inputting the image to be erased into a panoramic segmentation model to obtain multiple initial mask information, each mask information including an initial mask image corresponding to an instance object in the image to be erased and an initial mask category; and determining a prompt word based on the erasure mask image and the multiple initial mask information.

[0024] The solution provided in this embodiment obtains the initial mask category of each instance object based on the panoramic segmentation model, and determines the prompt words based on the erased mask image and multiple initial mask information, thereby improving the accuracy of prompt word determination.

[0025] In some optional embodiments, determining prompt words based on the erased mask image and multiple initial mask information includes: determining the first target initial mask category of adjacent instance objects in the adjacent region based on the erased mask image and multiple initial mask information; and generating prompt words based on the first target initial mask category.

[0026] The solution provided in this embodiment generates prompt words based on the initial mask information of the first target of the adjacent instance objects in the adjacent region. This helps to stabilize the diffusion model and complete the image in the erased region based on the prompt words, making it more integrated with the adjacent instance objects and thus improving the image restoration quality.

[0027] In some optional embodiments, determining the erase mask category of the erase mask image includes: obtaining the erase pixel position of the erase mask image; and determining the second target initial mask category corresponding to the target instance object in the image to be erased that matches the erase pixel position as the erase mask category of the erase mask image.

[0028] The solution provided in this embodiment obtains the initial mask category of the second target of the erasure image based on the image to be erased and the erasure mask image, and determines the erasure mask category, thereby improving the accuracy of erasure mask category determination.

[0029] In some optional embodiments, determining whether the area to be erased contains a human image based on the erase mask category includes: when the erase mask category includes a human mask, determining that the area to be erased contains a human image; when the erase mask category does not include a human mask, determining that the area to be erased does not contain a human image.

[0030] In some optional embodiments, the process of inputting the erased image and prompt words into the image restoration model to obtain the restored image includes: an encoding module based on a variational autoencoder model encoding the erased image to obtain a latent space encoded image; inputting the latent space encoded image and prompt words into the image restoration model to obtain a latent space restored image; and a decoding module based on a variational autoencoder model decoding the latent space restored image to obtain the restored image.

[0031] The solution provided in this embodiment repairs erased images based on latent space, which reduces the amount of data computation during the repair process and improves the repair efficiency.

[0032] In some optional embodiments, inputting the latent space encoded image and the prompt word into the image inpainting model to obtain the latent space inpainted image includes: adding noise to the latent space encoded image to obtain a latent space noisy image; and inputting the latent space noisy image and the prompt word into the image inpainting model to obtain the latent space inpainted image.

[0033] The solution provided in this embodiment compresses the erased image through encoding and then denoises it using reference information from the noisy latent space image based on an image inpainting model. This reduces the amount of data computation during the denoising process of the image inpainting model and improves its denoising efficiency for noisy latent space images. Furthermore, by decoding the latent space inpainted image, the resulting inpainted image has the same dimensions as the image to be erased, thus improving the image inpainting quality.

[0034] In some optional embodiments, obtaining an erasure mask image of the area to be erased in the image to be erased includes: receiving a user's selection operation of the area to be erased in the image to be erased; and generating an erasure mask image in response to the selection operation.

[0035] In some optional embodiments, obtaining the erasure image based on the image to be erased and the erasure mask image includes: inputting the image to be erased and the erasure mask image to the image erasure model, so that the image erasure model outputs the erasure image based on the image to be erased and the erasure mask image.

[0036] The solution provided in this embodiment performs image erasure on the image to be erased based on an image erasure model, thereby improving the erasure efficiency of the image to be erased.

[0037] In some optional embodiments, the image erasure model includes the LaMa model.

[0038] The solution provided in this embodiment is based on the LaMa model to erase the image. Since the LaMa model has the advantage of not generating random images during the erasure process, the erased image does not contain randomly generated image information, thus improving the image erasure quality.

[0039] Secondly, embodiments of this application provide an image restoration apparatus, which includes a first acquisition module, a second acquisition module, and a restoration input module. The first acquisition module is used to acquire an erasure mask image of the area to be erased in the image to be erased; the second acquisition module is used to acquire an erasure image based on the image to be erased and the erasure mask image; the restoration input module is used to input the erasure image and a prompt word into an image restoration model to obtain a restored image. The prompt word indicates foreground objects and / or background objects in a neighboring region. A neighboring region refers to a region in the erasure image adjacent to the area to be erased. The image restoration model is used to complete and restore the erasure area of ​​the erasure image based on the prompt word.

[0040] Thirdly, embodiments of this application provide an electronic device, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and one or more processors call the computer instructions to cause the electronic device to perform the image restoration method provided in the first aspect above.

[0041] Fourthly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the image restoration method provided in the first aspect above.

[0042] In some alternative embodiments, the chip system further includes a memory connected to one or more processors via circuits or wires.

[0043] In some alternative embodiments, the chip system also includes a communication interface.

[0044] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the image restoration method provided in the first aspect above.

[0045] In a sixth aspect, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform the image restoration method provided in the first aspect above.

[0046] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This illustration shows a scenario diagram of an image erasure operation provided in an embodiment of this application.

[0049] Figure 2 This illustration shows another scenario of the image erasure operation provided in the embodiments of this application.

[0050] Figure 3 A schematic flowchart of an image restoration method provided in an embodiment of this application is shown.

[0051] Figure 4 This illustration shows a scenario of selecting the area to be erased in the image restoration method provided in this application embodiment.

[0052] Figure 5 This illustration shows a scenario of erasing a mask image in the image restoration method provided in this application embodiment.

[0053] Figure 6 This illustration shows a scenario of image erasure based on the LaMa model in the image restoration method provided in this application.

[0054] Figure 7 This paper illustrates a structural diagram of the Inpainting model in the image restoration method provided in this application embodiment.

[0055] Figure 8 This illustration shows another flowchart of the image restoration method provided in an embodiment of this application.

[0056] Figure 9 This illustration shows another schematic flowchart of the image restoration method provided in the embodiments of this application.

[0057] Figure 10 This illustration shows another schematic flowchart of the image restoration method provided in the embodiments of this application.

[0058] Figure 11 This illustration shows another schematic flowchart of the image restoration method provided in the embodiments of this application.

[0059] Figure 12 This illustration shows a scene diagram of panoramic segmentation processing in the image restoration method provided in this application embodiment.

[0060] Figure 13 This illustration shows a scenario diagram of the SD model in the image restoration method provided in this application embodiment.

[0061] Figure 14 This illustration shows a scenario of restored images generated by different iterations of denoising in the image restoration method provided in this application embodiment.

[0062] Figure 15 The illustration shows a flowchart of the image restoration method provided in this application embodiment in an application scenario.

[0063] Figure 16 A structural block diagram of an image restoration apparatus provided in an embodiment of this application is shown.

[0064] Figure 17 A schematic diagram of a hardware structure of an electronic device provided in an embodiment of this application is shown.

[0065] Figure 18 A functional block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0066] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0067] The following disclosure provides many different implementations or examples for carrying out different structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or reference letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0068] Image erasure is a technique that removes image content from a certain area of ​​the original image and then fills in the area with appropriate image content.

[0069] Currently, image erasure is mainly performed on the original image using the Stable Diffusion (SD) model. During the training of the SD model, the original image is input into the model. Random noise is added to the original image, and the model learns from the noise-added image to estimate the added noise value. The original image is then subtracted from the estimated noise value to obtain the restored image.

[0070] However, the SD model focuses on completing image content during training. When erasing image content, the SD model generates image content that is unrelated to the original image, resulting in low restoration quality.

[0071] To address the aforementioned issues, the image restoration method, image restoration apparatus, electronic device, and storage medium provided in this application obtain an erasure mask image of the area to be erased in the image to be erased, obtain an erasure image based on the image to be erased and the erasure mask image, and input the erasure image and prompt words into the image restoration model to obtain a restored image. The prompt words indicate foreground and / or background objects in adjacent regions. Adjacent regions refer to areas in the image to be erased that are adjacent to the area to be erased. The image restoration model is used to complete and restore the erasure area of ​​the erasure image based on the prompt words. This achieves image completion of the erasure area of ​​the erasure image based on the image restoration model and the erasure image and prompt words. The prompt words indicate foreground and / or background objects in adjacent regions, enabling the image restoration model to complete the erase area of ​​the erasure image based on the foreground and / or background objects in adjacent regions, resulting in a more integrated completed image that blends better with the foreground and / or background objects in the adjacent regions. This avoids random generation during the image restoration process, improves the image completion effect of the restored image, and enhances the image restoration quality.

[0072] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0073] The image restoration method provided in this application can be applied to electronic devices. Electronic devices may include various terminal devices, which may also be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc.

[0074] Terminal devices can include mobile phones, robot vacuum cleaners, drones, smart TVs, wearable devices, personal digital assistants (PDAs), computers with wireless transceiver capabilities, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, and wireless terminals in smart homes. The type of terminal device is not limited here; it can be configured according to actual needs.

[0075] Image erasure, as a common image processing technique, is widely used in various scenarios. For example, electronic devices can erase unwanted human figures from images, or they can erase items worn by a target object in an image. The following section combines... Figure 1 and Figure 2 Explain how electronic devices erase images.

[0076] In response to the user clicking on the gallery application 101 in the display interface 10, such as Figure 1 As shown in (a), the electronic device switches from display interface 10 to display interface 11, as follows: Figure 1 As shown in (b) above. The display interface 11 can display thumbnails of various images already stored in the gallery. At this time, if the user clicks on one of the thumbnails, for example, as shown below... Figure 1 The thumbnail 111 in the first row and first column shown in (b) indicates that the electronic device switches from display interface 11 to display interface 12, as shown in the image. Figure 1 As shown in (c) in the diagram. The display interface 12 shows the image 121 corresponding to the thumbnail 111 in the first row and first column, as well as a share control 122, a favorite control 123, an edit control 124, a delete control 125, and more controls 126. If the user clicks the edit control 124, as shown in (c) in the diagram. Figure 1 As shown in (d), in response to the user clicking the edit control 124, the display interface 12 switches to the display interface 13, as shown in (d). Figure 1As shown in (e) in the diagram. The display interface 13 includes an image 131 corresponding to the thumbnail 111 in the first row and first column, a doodle control 132, a text control 133, an Artificial Intelligence (AI) removal control 134, a mosaic control 135, and a watermark control 136. When navigating to the display interface 13, the electronic device can also display the sub-controls corresponding to the AI ​​removal control 134 by default, such as... Figure 1 The intelligent selection sub-control 1341 and the manual smearing sub-control 1342 are shown in (e) in the diagram. It should be noted that when navigating to display interface 13, the electronic device may also default to displaying the sub-controls corresponding to the graffiti control 132, text control 133, mosaic control 135, or watermark control 136; this embodiment does not impose such limitations. If the electronic device defaults to displaying the sub-control corresponding to the mosaic control 135 upon navigating to display interface 13, the user can also click the AI ​​removal control 134 to display the sub-control corresponding to the AI ​​removal control 134.

[0077] The image 131 displayed in display interface 13 and the image 121 displayed in display interface 12 are both images corresponding to the thumbnail 111 in display interface 11. That is to say, the display content of images 131 and 121 is the display content of thumbnail 111. The size of image 131 and the size of image 121 can be the same or different, and this embodiment does not limit this. The size of image 131 can be a preset size, or it can be a size obtained by adaptively adjusting the preset size based on other information displayed on display interface 13.

[0078] In the display interface 13, if the user clicks the manual smear sub-control 1342, such as Figure 2 As shown in (a) of the image. In response to the user's click on the manual erasure sub-control 1342, the user can erase the target object selected by the user in the image 131. The target object can be any extra person in the image 131, or accessories worn by a person in the image 131, etc., and this embodiment does not impose any limitations on this. As the user's finger moves in the image 131, the user's finger movement trajectory 137 is displayed on the image 131, such as... Figure 2 As shown in (b) above. When the user stops moving their finger, the displayed movement trajectory 138 is as follows. Figure 2 As shown in (c) above. It is understood that the movement trajectory 138 can be a closed curve or an open curve, and this embodiment of the application does not limit this.

[0079] If the movement trajectory 138 is a closed curve (not shown in the figure), the electronic device can directly generate the region to be eliminated 139 based on the movement trajectory 138, such as... Figure 2 As shown in (d) in the figure.

[0080] If the movement trajectory 138 is an open region (not shown in the figure), the start and end points of the movement trajectory 138 can be connected to form a new closed curve. Then, the region to be eliminated 139 can be generated based on the new closed region, such as... Figure 2 As shown in (d) in the figure.

[0081] Then, the electronic device can erase the target person selected in the area to be eliminated 139, and display the image 1312 after the target person has been erased, such as... Figure 2 As shown in (e) in the diagram.

[0082] In one possible scenario, the target object selected by movement trajectory 138 is an item worn by the person in image 131. For example, as... Figure 2 In (f), the target object selected by the user is the brooch worn by the person in image 131. The movement trajectory 138 formed by the user's finger movement can be as follows: Figure 2 As shown in (f) in the figure. The electronic device generates the region to be eliminated 1310 based on the movement trajectory 138, as shown in the figure. Figure 2 As shown in (g) in the image. The electronic device can then erase the target object (brooch) in the area to be erased 1310, obtaining image 1313 after the brooch has been erased, as shown in the image. Figure 2 As shown in (h).

[0083] Please see Figure 3 The document illustrates a flowchart of an image restoration method provided in one embodiment of this application. In a specific embodiment, the image restoration method can be applied to an electronic device. The following example uses an electronic device to illustrate this method. Figure 3 The process shown is described in detail. The image restoration method may include the following steps 201 to 204.

[0084] Step 201: In response to the user's selection of a target pre-stored image on the electronic device, determine that the target pre-stored image is the image to be erased.

[0085] In this embodiment of the application, when a user needs to perform image repair, the user can select a target pre-stored image of the electronic device. The electronic device receives and responds to the user's selection operation and determines that the target pre-stored image is the image to be erased.

[0086] The electronic device is equipped with a gallery containing at least one pre-stored image, and the user's selection of a target pre-stored image on the electronic device can be a click operation on the target pre-stored image in the gallery.

[0087] In some implementations, in response to a user's selection of a target pre-stored image, the electronic device, after determining that the target pre-stored image is the image to be erased, can preview the image to be erased so that the user can perform the erasure operation on the image.

[0088] Step 202: In response to the user's selection of the area to be erased in the image to be erased, generate an erasure mask image of the area to be erased.

[0089] In this embodiment of the application, after the electronic device previews the image to be erased, the user can select the area to be erased in the image. In response to the user's selection of the area to be erased in the image, the electronic device generates an erasure mask image of the area to be erased.

[0090] The area to be erased can be the area selected by the intelligent selection sub-control of the user's touch electronic device through an intelligent selection operation on the image to be erased. For example, the selected area is like... Figure 2 The area enclosed by the movement trajectory shown in (d) can be the area to be erased, or it can be the area obtained by manually applying ink to the image to be erased using a manual smearing sub-control on a user-touch electronic device. For example, the manually applied area could be... Figure 4 The shaded area shown is not limited here.

[0091] The selection operation can be either an intelligent selection operation of the image to be erased based on an electronic device's intelligent selection sub-control, or a manual smearing operation of the image to be erased based on an electronic device's manual smearing sub-control, etc., and there is no limitation here.

[0092] The size of the erase mask image is the same as the size of the image to be erased. The erase mask image can be used to occlude the non-erasable areas of the image to be erased. The values ​​in the erase mask image include 0 and non-zero. The values ​​in the erase mask image corresponding to the area to be erased are non-zero, and the values ​​in the erase mask image corresponding to the non-erasable areas are 0.

[0093] In some implementations, after the electronic device previews the image to be erased, the user can touch the smart selection sub-control of the electronic device to perform a smart selection operation on the image to be erased. The electronic device receives and responds to the user's smart selection operation, determines the selected area as the area to be erased, and generates an erasure mask image based on the area to be erased.

[0094] In some implementations, after the electronic device previews the image to be erased, the user can touch the manual smear sub-control of the electronic device to perform a manual smearing operation on the image to be erased. The electronic device receives and responds to the user's manual smearing operation, determines the smeared area as the area to be erased, and generates an erasure mask image based on the area to be erased.

[0095] In one application scenario, an electronic device can preview an image to be erased. The user can use a smart selection control on the device to make a smart selection of the image. The device receives and responds to the user's smart selection, defining the area enclosed by the user's finger movement as the area to be erased, and generating an erasure mask image based on this area. For example, the erasure mask image could be as follows: Figure 5 As shown.

[0096] Step 203: Obtain the erasure image based on the image to be erased and the erasure mask image.

[0097] In this embodiment of the application, after the electronic device generates an erasure mask image of the area to be erased in response to the user's selection operation of the area to be erased in the image to be erased, it can obtain the erasure image based on the image to be erased and the erasure mask image.

[0098] In some implementations, after the electronic device generates an erasure mask image of the area to be erased in response to the user's selection operation of the area to be erased in the image to be erased, it can input the image to be erased and the erasure mask image into the image erasure model. The image erasure model receives and responds to the image to be erased and the erasure mask image, erases the instance objects in the area to be erased in the image to be erased according to the erasure mask image, and completes the erased instance objects in the erased area with the instance objects in the adjacent areas of the erased area to obtain the erased image. This realizes the image erasure of the image to be erased based on the image erasure model, and improves the erasure efficiency of the image to be erased.

[0099] The image erasure model can include any one of the following: the LaMa model, the Erased Stable Diffusion (ESD) model, or the Concept Semi-Permeable Membrane (SPM) model. The type of image erasure model is not limited here and can be set according to actual needs.

[0100] As one implementation method, the image erasure model can be a LaMa model. The electronic device can input the image to be erased and the erasure mask image into the LaMa model. The LaMa model receives and responds to the image to be erased and the erasure mask image. Based on the erasure mask image, it erases the instance objects in the erasure area of ​​the image to be erased, and completes the erasure area of ​​the erased instance objects with the adjacent instance objects of the adjacent adjacent areas of the erasure area to obtain the erased image. The image erasure is performed on the image to be erased based on the LaMa model. Since the LaMa model has the advantage of not generating random image information during the image erasure process, the erased image will not contain randomly generated image information, thus improving the image erasure quality of the erased image.

[0101] The LaMa model can be used to overlay an image to be erased with an erasure mask image to form a composite image, and to erase instance objects in the area to be erased in the composite image, as well as to perform image completion processing on the erased area of ​​the erased instance objects.

[0102] The LaMa model is a network structure model for image restoration. It uses fast Fourier convolution, has a wide receptive domain, a high receptive field perceptual loss, and a large training mask. The LaMa model can generalize well to images with higher resolutions than during training.

[0103] The LaMa model has a small number of parameters. It does not include an attention mechanism. Furthermore, during training, the LaMa model is trained using a mask image and the original image, without any text information. As a result, the features learned by the LaMa model are relatively weak. The completed image of the erased region in the LaMa model is relatively blurry, and the tonal balance between the completed image and the surrounding image is uneven, leading to poor image completion results for the erased image.

[0104] In one application scenario, such as Figure 6 As shown, the image to be erased contains cars and street scenes. The image to be erased is a close-up image of vehicles. Users can perform intelligent selection operations on the close-up vehicle images in the image to be erased using the intelligent selection sub-control of the electronic device, or manually smear operations on the close-up vehicle images in the image to be erased using the manual smearing sub-control of the electronic device. The electronic device receives and responds to the user's intelligent selection operation or manual smearing operation, determines the selected area containing the close-up vehicle image as the area to be erased, or determines the smeared area containing the close-up vehicle image as the area to be erased, and generates an erasure mask image associated with the close-up vehicle image based on the area to be erased. The image to be erased and the erasure mask image are then input into the LaMa model to obtain the erased image (preliminary completion image).

[0105] In one implementation, the image erasure model can be an ESD model. The electronic device can input the image to be erased and the erasure mask image into the ESD model. The ESD model receives and responds to the image to be erased and the erasure mask image. Based on the erasure mask image, it erases the instance objects in the erasure area of ​​the image to be erased, and completes the erasure area of ​​the erased instance objects with the adjacent instance objects of the adjacent adjacent areas of the erasure area to obtain the erased image.

[0106] As one implementation method, the image erasure model can be an SPM model. The electronic device can input the image to be erased and the erasure mask image into the SPM model. The SPM model receives and responds to the image to be erased and the erasure mask image, erases the instance objects in the erasure area of ​​the image to be erased according to the erasure mask image, and completes the erasure area of ​​the erased instance objects with the adjacent instance objects of the adjacent adjacent areas of the erasure area to obtain the erased image.

[0107] In some implementations, after the electronic device generates an erasure mask image of the area to be erased in response to the user's selection operation of the area to be erased in the image to be erased, it can generate an erasure prompt message and receive the user's manual erasure operation of the instance objects in the area to be erased in the image to be erased according to the erasure prompt message, thereby obtaining the erasure image.

[0108] The erase prompt message can be used to prompt the user to manually erase the instance objects in the area to be erased of the image. The manual erase operation can be a cut operation or an overwrite operation on the instance objects, etc., which is not limited here.

[0109] Step 204: Input the erased image and prompt words into the image restoration model to obtain the restored image.

[0110] In this embodiment, after the electronic device obtains the erased image based on the image to be erased and the erased mask image, it can input the erased image and prompt words into the image inpainting model. The inpainting model receives and responds to the erased image and prompt words, and performs inpainting and repair on the erased image according to the prompt words to obtain the repaired image. This realizes the image inpainting of the erased area of ​​the erased image based on the erased image and prompt words. The prompt words indicate the foreground objects and / or background objects in the adjacent area, so that the inpainting model can fill in the erased area of ​​the erased image with the foreground and / or background objects in the adjacent area in a more integrated way. This avoids the random generation phenomenon in the image repair process, improves the image inpainting effect of the repaired image, and improves the image repair quality.

[0111] The cue words can represent foreground and / or background objects in the adjacent region. The adjacent region can refer to the area in the image to be erased that is adjacent to the area to be erased. The cue words can include negative cue words to represent foreground objects and positive cue words to represent background objects. For example, negative cue words can include people, vehicles, and tables, while positive cue words can include the ground, sky, and streetlights, etc., without limitation. The inpainting model can be used to complete and repair the erased area of ​​the erased image based on the cue words.

[0112] In some implementations, the inpainting model may include a text encoding (TE) model and a stable diffusion (SD) model. The TE model can be used to encode prompt words, obtaining prompt embeddings (PEs). The SD model can be used to complete and repair the erased image based on the PEs, resulting in a repaired image.

[0113] After obtaining the erased image based on the image to be erased and the erased mask image, the electronic device can input prompts into the TE model and the erased image into the SD model. The TE model receives and responds to the prompts, encodes them, and outputs the PE to the SD model. The SD model receives and responds to the erased image and the PE, and performs image completion on the erased area of ​​the erased image based on the PE to obtain the repaired image. This achieves image completion based on the SD model using foreground and / or background objects in adjacent areas, resulting in a more integrated image with the foreground and / or background objects in the erased area. This avoids the random generation phenomenon that occurs when the SD model performs image completion and repair on the erased area based on prompts input by the user that are unrelated to the foreground and / or background objects in adjacent areas, thus improving the image completion effect and image repair quality of the repaired image.

[0114] In the process of image completion of the erased area, the SD model uses the PE (Presence Object) that is strongly correlated with the foreground and / or background objects in the adjacent area to denoise the added random Gaussian noise based on the PE, and generates a completed image in the erased area that is more integrated with the foreground and / or background objects in the adjacent area.

[0115] The SD model has a Unet network structure. During the image restoration process, the input of the SD model consists of three parts: initial noise, denoising time step (Timestep) k, and PE.

[0116] The SD model includes a forward diffusion model process and a backward diffusion model process. During training, the SD model executes the forward diffusion model process, taking the first image noise corresponding to a certain state as the first noise, and iteratively adding noise to the first noise multiple times until the second noise is obtained. During inference, the SD model executes the backward diffusion model process, iteratively denoising the second noise multiple times until the first noise is obtained.

[0117] The inpainting model can be obtained by fusing the SD model and the TE model. The PE output module of the TE model is connected to the PE input module of the SD model.

[0118] Regarding the process of the above-mentioned electronic device inputting the erased image and prompt words into the Inpainting model to obtain the repaired image, in some embodiments, the electronic device can input the prompt words into the TE model, and input the erased image, erased mask image, and background image into the SD model. The TE model receives and responds to the prompt words, encodes the prompt words, obtains and outputs the PE to the SD model. The SD model receives and responds to the erased image, erased mask image, background image, and PE, and performs image completion on the erased area of ​​the erased image based on the erased mask image, background image, and PE to obtain the repaired image. The SD model performs completion and repair on the erased image based on the erased mask image, background image, and prompt words, which can accurately complete the image in the erased area and make it more integrated with the foreground objects and / or background objects in the adjacent area, further improving the image repair quality.

[0119] The electronic device can overlay the erased image and the erased mask image to obtain an overlaid image, and remove the region image corresponding to the erased area in the overlaid image to obtain a background image.

[0120] In some implementations, the inpainting model may also include a variational autoencoder (VAE) model, which can be obtained by fusing the SD model, TE model, and VAE model. The VAE model includes an encoder module and a decoder module, with the SD model connecting the encoder and decoder modules, such as... Figure 7 As shown.

[0121] In some implementations, the Inpainting model can also be an Aggregated Contextual Transformations Generative Adversarial Network (AOT-GAN) model or a Mask-Interactive Generative Adversarial Network (MI-GAN) model, etc. The type of Inpainting model is not limited here, and can be set according to actual needs.

[0122] The solution provided in this application involves an electronic device responding to a user's selection of a target pre-stored image, determining the target pre-stored image as the image to be erased, and responding to the user's selection of a region to be erased within the image to be erased, generating an erasure mask image for the region to be erased, obtaining the erased image based on the image to be erased and the erasure mask image, and inputting the erased image and a prompt word into an Inpainting model to obtain a repaired image. The prompt word indicates foreground and / or background objects in adjacent regions. Adjacent regions refer to areas in the image to be erased that are adjacent to the region to be erased. The Inpainting model... This tool is used to complete and repair erased areas of an erased image based on prompts. It implements an Inpainting model to complete the erased areas of an erased image based on the erased image and prompts. The prompts indicate foreground and / or background objects in the adjacent areas. This allows the Inpainting model to complete the erased areas of the erased image more seamlessly with the foreground and / or background objects in the adjacent areas, avoiding random generation during the image restoration process, improving the image completion effect and enhancing the image restoration quality.

[0123] Please see Figure 8 This document illustrates a flowchart of an image restoration method provided in another embodiment of this application. In a specific embodiment, the image restoration method can be applied to an electronic device. The following example uses an electronic device to illustrate this method. Figure 8 The process shown is described in detail. The image restoration method may include the following steps 301 to 305.

[0124] Step 301: In response to the user's selection operation of the target pre-stored image of the electronic device, determine that the target pre-stored image is the image to be erased.

[0125] Step 302: In response to the user's selection of the area to be erased in the image to be erased, generate an erasure mask image of the area to be erased.

[0126] Step 303: Obtain the erasure image based on the image to be erased and the erasure mask image.

[0127] In this embodiment, steps 301, 302 and 303 can be referred to the corresponding steps in the previous embodiments, and will not be repeated here.

[0128] Step 304: Add noise to the erased image to obtain a noisy image.

[0129] In this embodiment, after the electronic device obtains the erasure image based on the image to be erased and the erasure mask image, it can perform noise addition processing on the erasure image to obtain a noisy image. Based on the noise addition processing of the erasure image, the noisy image contains more reference information.

[0130] Step 305: Input the noisy image and prompt words into the Inpainting model to obtain the repaired image.

[0131] In this embodiment, the electronic device adds noise to the erased image to obtain a noisy image. After obtaining the noisy image, the noisy image and prompt words can be input into the Inpainting model. The Inpainting model receives and responds to the noisy image and prompt words, and repairs the noisy image according to the prompt words to obtain a repaired image. The noisy image after noise processing contains more reference information than the erased image. Based on the Inpainting model, the reference information of the noisy image is used to repair the noisy image, which further suppresses the random generation phenomenon in the image repair process and further improves the image repair quality.

[0132] In some implementations, the electronic device can input a prompt word to the TE model, and input a noisy image and a denoising time step k to the SD model. The TE model receives and responds to the prompt word, encodes the prompt word, and outputs the PE to the SD model. The SD model receives and responds to the noisy image, the denoising time step k, and the PE. Using the noisy image as the initial noise, the initial noise is denoised multiple times according to the denoising time step k and the PE to obtain the repaired image.

[0133] The denoising time step k corresponds to the number of noise addition steps in the noise addition process. For example, when the number of noise addition steps is 1000, the denoising time step k is 40; when the number of noise addition steps is 749, the denoising time step k is 30. The number of noise addition steps and the denoising time step k are not limited here and can be set according to actual needs.

[0134] The solution provided in this embodiment involves an electronic device responding to a user's selection of a target pre-stored image, determining the target pre-stored image as the image to be erased, and responding to the user's selection of the area to be erased within the image to be erased, generating an erasure mask image for the area to be erased, obtaining an erasure image based on the image to be erased and the erasure mask image, adding noise to the erasure image to obtain a noisy image, and inputting the noisy image and prompt words into an Inpainting model to obtain a repaired image. This achieves the goal of obtaining a noisy image by adding noise to the erasure image, increasing the reference information contained in the noisy image, and using the reference information of the noisy image based on the Inpainting model to repair the noisy image, further suppressing the random generation phenomenon in the image repair process and further improving the image repair quality.

[0135] Please see Figure 9 This document illustrates a flowchart of an image restoration method provided in another embodiment of this application. In a specific embodiment, the image restoration method can be applied to an electronic device. The following example uses an electronic device to illustrate this method. Figure 9 The process shown is described in detail. The image restoration method may include the following steps 401 to 407.

[0136] Step 401: In response to the user's selection operation of the target pre-stored image of the electronic device, determine that the target pre-stored image is the image to be erased.

[0137] Step 402: In response to the user's selection of the area to be erased in the image to be erased, generate an erasure mask image of the area to be erased.

[0138] Step 403: Obtain the erasure image based on the image to be erased and the erasure mask image.

[0139] In this embodiment, steps 401, 402, and 403 can be found in the corresponding steps of the aforementioned embodiments, and will not be repeated here.

[0140] Step 404: The encoding module based on the variational autoencoder model encodes the erased image to obtain a latent space encoded image.

[0141] In this embodiment, after the electronic device obtains the erasure image based on the image to be erased and the erasure mask image, it can encode the erasure image based on the Encoder module of the VAE model to obtain the latent space encoded image. Encoding the erasure image to compress it helps to reduce the amount of data computation in the process of completing and repairing based on the latent space encoded image.

[0142] The data size of a latent space coded image is smaller than that of a erased image.

[0143] Latent space is a space for compressing data representation. The purpose of latent space is to learn data features and simplify data representation in order to find patterns.

[0144] Data compression refers to encoding information using fewer bits than the original representation. For example, compressing 19-dimensional data into 9-dimensional data. The purpose of data compression is to learn the more important information in the data.

[0145] Step 405: Add noise to the latent space encoded image to obtain a latent space denoised image.

[0146] In this embodiment, the electronic device encodes the erased image using the Encoder module based on the VAE model to obtain a latent space encoded image. Then, the latent space encoded image can be denoised to obtain a latent space denoised image.

[0147] The process of adding noise to the latent space coded image by the electronic device is similar to the process of adding noise to the erased image by the electronic device. For details, please refer to the content under step 304, which will not be repeated here.

[0148] Step 406: Input the noisy latent space image and the prompt words into the Inpainting model to obtain the latent space repaired image.

[0149] In this embodiment, the electronic device adds noise to the latent space encoded image to obtain a noisy latent space image. Then, it can input the noisy latent space image and a prompt word into the Inpainting model. The Inpainting model receives and responds to the noisy latent space image and the prompt word, and completes and repairs the noisy latent space image according to the prompt word to obtain a repaired latent space image. The Inpainting model uses the reference information of the noisy latent space image for denoising, which reduces the amount of data computation in the denoising process of the Inpainting model and improves the denoising efficiency of the Inpainting model for noisy latent space images.

[0150] Step 407: The decoding module based on the variational autoencoder model decodes the latent space-restored image to obtain the restored image.

[0151] In this embodiment, the electronic device inputs a noisy latent space image and a prompt word into the Inpainting model to obtain a latent space repaired image. Then, the latent space repaired image can be decoded based on the VAE model's Decoder module to obtain a repaired image. By decoding the latent space repaired image, the obtained repaired image has the same dimension as the image to be erased, thus improving the image repair quality.

[0152] The solution provided in this embodiment involves an electronic device responding to a user's selection of a target pre-stored image, determining the target pre-stored image as the image to be erased, and responding to the user's selection of the area to be erased within the image to be erased, generating an erasure mask image for the area to be erased, obtaining an erasure image based on the image to be erased and the erasure mask image, encoding the erasure image using an encoding module based on a variational autoencoder model to obtain a latent space encoded image, adding noise to the latent space encoded image to obtain a latent space noisy image, inputting the latent space noisy image and prompt words into an Inpainting model to obtain a latent space repair image, and decoding the latent space repair image using a decoding module based on a variational autoencoder model to obtain a repair image. Encoding the erasure image compresses it, and the Inpainting model utilizes reference information from the latent space noisy image for denoising, reducing the data computation load during the denoising process of the Inpainting model and improving the denoising efficiency of the Inpainting model for latent space noisy images. Furthermore, by decoding the latent space-restored image, the resulting restored image has the same dimensions as the image to be erased, thus improving the image restoration quality.

[0153] Please see Figure 10 This document illustrates a flowchart of an image restoration method provided in another embodiment of this application. In a specific embodiment, the image restoration method can be applied to an electronic device. The following example uses an electronic device to illustrate this method. Figure 10 The process shown is described in detail. The image restoration method may include the following steps 501 to 506.

[0154] Step 501: In response to the user's selection operation of the target pre-stored image of the electronic device, determine that the target pre-stored image is the image to be erased.

[0155] Step 502: In response to the user's selection of the area to be erased in the image to be erased, generate an erasure mask image of the area to be erased.

[0156] Step 503: Obtain the erasure image based on the image to be erased and the erasure mask image.

[0157] Step 504: The encoding module based on the variational autoencoder model encodes the erased image to obtain a latent space encoded image.

[0158] In this embodiment, steps 501, 502, 503, and 5040 can be referred to the corresponding steps in the foregoing embodiments, and will not be repeated here.

[0159] Step 505: Input the latent space encoded image and the prompt word into the Inpainting model to obtain the latent space inpainted image.

[0160] In this embodiment, the electronic device encodes the erased image using the Encoder module of the VAE model to obtain a latent space encoded image. This latent space encoded image can then be input into the Inpainting model. The Inpainting model receives and responds to the latent space encoded image and prompts, and completes and repairs the latent space encoded image based on the prompts to obtain a latent space repaired image. The Inpainting model utilizes the reference information of the latent space encoded image to complete and repair it, reducing the amount of data computation during the completion and repair process and improving the repair rate.

[0161] Step 506: The decoding module based on the variational autoencoder model decodes the latent space inpainted image to obtain the inpainted image.

[0162] In this embodiment, the electronic device inputs the latent space encoded image and prompt words into the Inpainting model to obtain the latent space repaired image. Then, the latent space repaired image can be decoded based on the Decoder module of the VAE model to obtain the repaired image. By decoding the latent space repaired image, the obtained repaired image has the same dimension as the image to be erased, thus improving the image repair quality.

[0163] The solution provided in this embodiment involves an electronic device responding to a user's selection of a target pre-stored image, determining the target pre-stored image as the image to be erased, and responding to the user's selection of the area to be erased within the image to be erased, generating an erasure mask image for the area to be erased, obtaining the erased image based on the image to be erased and the erasure mask image, encoding the erased image using an encoding module based on a variational autoencoder model to obtain a latent space encoded image, inputting the latent space encoded image and a prompt word into an Inpainting model to obtain a latent space repaired image, and decoding the latent space repaired image using a decoding module based on a variational autoencoder model to obtain a repaired image. This achieves latent space-based image repair processing, reduces the amount of data computation during the image repair process, and improves the repair rate of the erased image.

[0164] Please see Figure 11 This document illustrates a flowchart of an image restoration method provided in another embodiment of this application. In a specific embodiment, the image restoration method can be applied to an electronic device. The following example uses an electronic device to illustrate this method. Figure 11 The process shown is described in detail. The image restoration method may include the following steps 601 to 608.

[0165] Step 601: In response to the user's selection operation of the target pre-stored image of the electronic device, determine that the target pre-stored image is the image to be erased.

[0166] In this embodiment, step 601 can be referred to the corresponding steps in the previous embodiments, and will not be repeated here.

[0167] Step 602: Perform panoramic segmentation on the image to be erased to obtain multiple initial mask information.

[0168] In this embodiment, in response to the user's selection operation of the target pre-stored image of the electronic device, after determining that the target pre-stored image is the image to be erased, the electronic device can perform panoramic segmentation processing on the image to be erased to obtain multiple initial mask information.

[0169] The image to be erased includes multiple instance objects, such as people, vehicles, ground, sky, and streetlights. Each initial mask information corresponds to one instance object in the image to be erased. Each initial mask information can include an initial mask image corresponding to an instance object and an initial mask category. The initial mask category can include a person mask, vehicle mask, ground mask, sky mask, and streetlight mask, etc.

[0170] Specifically, in response to a user's selection of a target pre-stored image, the electronic device, after determining that the target pre-stored image is the image to be erased, can input the image to be erased into the SegmentAnything Model (SAM). The SAM receives and responds to the image to be erased, performs panoramic segmentation on the image to be erased, and obtains multiple initial mask information, such as... Figure 12 As shown, obtaining the initial mask category for each instance object based on panoramic segmentation SAM is beneficial to improving the accuracy of determining the erasure mask category of the erasure mask image based on the initial mask category of each instance object.

[0171] In some implementations, after the electronic device inputs the image to be erased into the panoramic segmentation SAM model and obtains multiple initial mask information, it can determine prompt words based on the erased mask image and the multiple initial mask information. Based on the panoramic segmentation SAM model, the initial mask category of each instance object is obtained, and the prompt words are determined based on the erased mask image and the multiple initial mask information, thereby improving the accuracy of prompt word determination.

[0172] Specifically, the electronic device inputs the image to be erased into the panoramic segmentation SAM model. After obtaining multiple initial mask information, it can determine the first target initial mask category of the adjacent instance objects in the adjacent region based on the erased mask image and the multiple initial mask information. Based on the first target initial mask category, prompt words are generated. Based on the first target initial mask information of the adjacent instance objects in the adjacent region, prompt words are generated. This helps the stable diffusion model to complete the image in the erased region based on the prompt words, making it more integrated with the adjacent instance objects, which helps to improve the image restoration quality.

[0173] Step 603: In response to the user's selection of the area to be erased in the image to be erased, generate an erasure mask image of the area to be erased.

[0174] Step 604: Input the image to be erased and the erasure mask image into the LaMa model, so that the LaMa model outputs the erasure image based on the image to be erased and the erasure mask image.

[0175] In this embodiment, steps 603 and 604 can be found in the corresponding steps in the foregoing embodiments, and will not be repeated here.

[0176] In some implementations, the electronic device inputs the image to be erased and the erasure mask image into the LaMa model, so that after the LaMa model outputs the erasure image based on the image to be erased and the erasure mask image, it can determine whether the area to be erased contains a human figure.

[0177] Specifically, the electronic device inputs the image to be erased and the erasure mask image into the LaMa model. After the LaMa model outputs the erased image based on the image to be erased and the erasure mask image, it can determine the erasure mask category of the erasure mask image based on the initial pixel position of each initial mask image and the erasure pixel position of the erasure mask image. Based on the erasure mask category, it determines whether the area to be erased contains a human image. The judgment on whether the area to be erased contains a human image is improved based on the erasure mask category of the erasure mask image.

[0178] When the erase mask category includes a human mask, it is determined that the area to be erased contains a human image; when the erase mask category does not include a human mask, it is determined that the area to be erased does not contain a human image.

[0179] The electronic device can obtain the erase pixel position of the erase mask image and determine the second target initial mask category corresponding to the target instance object in the image to be erased that matches the erase pixel position as the erase mask category of the erase mask image. Based on the erase pixel position and the second target initial mask category of the target instance object in the image to be erased, the erase mask category is determined, which improves the accuracy of the erase mask category determination.

[0180] Because the LaMa model does not perform well in scenes where people are being erased, stronger noise needs to be added when the LaMa model erases people. That is, the noise intensity added during the noise addition process when the image to be erased contains people should be stronger than the noise intensity added during the noise addition process when the image to be erased does not contain people.

[0181] In some implementations, the electronic device can add noise to the erased image based on the result of determining whether the area to be erased contains a human image, thus obtaining a noisy image.

[0182] The noise-added image may include a first noise-added image and a second noise-added image. The first noise-added image is an image obtained by adding noise to the erased image when the area to be erased contains a human image, and the second noise-added image is an image obtained by adding noise to the erased image when the area to be erased does not contain a human image.

[0183] Step 605: When it is determined that the area to be erased contains a human image, noise is added to the image to be erased according to the first rule to obtain the first noisy image.

[0184] In this embodiment, when the electronic device determines that the area to be erased contains a human image, it can perform noise processing on the image to be erased for a first number of noise-adding steps according to the first rule to obtain a first noisy image. Since the LaMa model does not perform well in the scene of erasing human images, the noise-adding process on the erased image increases the reference information contained in the noisy image. Based on the stable diffusion model, the reference information of the noisy image is used for denoising, which further suppresses the random generation phenomenon of the stable diffusion model and further improves the image restoration quality.

[0185] The first rule can be used to characterize the rule that the noise intensity after each noise addition process satisfies the first noise formula. The first noise formula is: x0 represents the noise intensity of the erased image, K represents the number of noise additions, and α K Let α be the noise figure for the Kth noise addition. K ∈(0,1), α K The noise is uniformly distributed and is Gaussian noise.

[0186] The first noise-adding step count can be 1000 steps, 500 steps, or 1500 steps. There is no limit to the value of the first noise-adding step count; it can be set according to actual needs.

[0187] Since the erased image obtained by the LaMa model contains a lot of poorly presented information, it is necessary to add a large amount of noise to the erased image after erasing the image. This helps to suppress the SD model from generating random images due to too much poorly presented information during the denoising process, and thus improves the quality of image restoration.

[0188] In one application scenario, when an electronic device determines that the area to be erased contains a human image, it can perform noise processing on the erasing image for 1000 steps according to the first rule to obtain the first noisy image.

[0189] Wherein, the first rule corresponds to the first noise-adding step number of 1000 steps, α0=0.9991, α 999 =0.0049. The noise intensity of the first noisy image obtained after the 999th step of noise addition processing is:

[0190] Step 606: Input the first noisy image and the prompt word into the Inpainting model to obtain the first repaired image.

[0191] In this embodiment, the repaired image may include a first repaired image corresponding to the first noisy image. When the electronic device determines that the area to be erased contains a human image, it performs noisy processing on the image to be erased according to the first rule to obtain the first noisy image. After obtaining the first noisy image, a prompt word can be input to the TE model and the first noisy image can be input to the SD model. The TE model receives and responds to the prompt word, encodes the prompt word, obtains and outputs the PE to the SD model. The SD model receives and responds to the first noisy image and the PE, and performs image completion on the erased area of ​​the first noisy image according to the PE to obtain the first repaired image. This realizes the achievement of obtaining a noisy image by adding noise to the erased image, which increases the reference information of the noisy image. Furthermore, the SD model uses the reference information of the noisy image to perform denoising, further suppressing the random generation phenomenon of the SD model and further improving the image repair quality.

[0192] In one application scenario, when an electronic device determines that the area to be erased contains a human image, it performs 1000 steps of noise reduction processing on the erasing image according to the first rule to obtain the first noise-reduced image. After obtaining the first noise-reduced image, a prompt word can be input to the TE model and the first noise-reduced image can be input to the SD model. The TE model receives and responds to the prompt word, encodes the prompt word, obtains and outputs the PE to the SD model. The SD model receives and responds to the first noise-reduced image and the PE, and performs iterative denoising on the first noise-reduced image 40 times according to the PE to obtain the first repaired image.

[0193] In the SD model, when repairing erased images of erased portraits, it needs to iterate the erased image 40 times to obtain the final completed image. The SD model uses the pseudo-numerical methods for Diffusion Models (PNDM) sampler to gradually denoise the noise obtained from step 999 until the noise obtained from step 0 is completely denoised.

[0194] Step 607: When it is determined that the area to be erased does not contain a human image, the erased image is subjected to noise processing according to the second rule to obtain a second noise-added image.

[0195] In this embodiment, when the electronic device determines that the area to be erased does not contain a human image, it can perform noise processing on the erased image according to the second rule for a second noise-adding step to obtain a second noise-adding image.

[0196] The second rule can be used to characterize the rule that the noise intensity after each noise addition process satisfies the second noise formula. The second noise formula is: T α represents the number of times noise is added. T For the first T The noise figure of the second noise addition, α T ∈(0,1), α T It is uniformly distributed.

[0197] Since the erased image obtained by the LaMa model after erasing non-human images contains less information that is not good, a small amount of noise can be added to the erased image after erasing non-human images. This helps to avoid the phenomenon of the SD model generating random images due to excessive noise during the denoising process of the erased image, and helps to improve the image restoration quality.

[0198] Therefore, the first noise-adding step number corresponding to the first rule is greater than the second noise-adding step number corresponding to the second rule.

[0199] In one application scenario, when the electronic device determines that the area to be erased does not contain a human image, it performs noise processing on the erased image for 749 steps according to the second rule to obtain a second noisy image.

[0200] In this case, the second rule corresponds to 749 noise-adding steps, α0 = 0.9991, α 749 =0.2545. The noise intensity of the second noisy image obtained after step 749 of the noise addition process is:

[0201] Step 608: Input the second noisy image and the prompt word into the Inpainting model to obtain the second repaired image.

[0202] In this embodiment, the repaired image may include a second repaired image corresponding to the second noisy image. When the electronic device determines that the area to be erased does not contain a human image, it adds noise to the erased image according to the second rule to obtain the second noisy image. Then, it can input a prompt word to the TE model and the second noisy image to the SD model. The TE model receives and responds to the prompt word, encodes the prompt word, and outputs the PE to the SD model. The SD model receives and responds to the second noisy image and the PE, and performs image completion on the erased area of ​​the second noisy image according to the PE to obtain the second repaired image. Since the noise intensity added to the erased image that does not contain a human image is relatively small, when iterating the denoising of the second noisy image, the SD model performs fewer iterations of denoising on the second noisy image than it performs fewer iterations of denoising on the first noisy image, thus improving the repair efficiency of the erased image after erasing the non-human image.

[0203] In one application scenario, when the electronic device determines that the area to be erased does not contain a human image, it performs 749 steps of noise reduction processing on the erasing image according to the second rule to obtain a second noisy image. After obtaining the second noisy image, a prompt word can be input to the TE model and the second noisy image can be input to the SD model. The TE model receives and responds to the prompt word, encodes the prompt word, obtains and outputs the PE to the SD model. The SD model receives and responds to the second noisy image and the PE, and iteratively denoises the second noisy image 30 times according to the PE to obtain the second repaired image. Since the noise intensity added to the erasing image that does not contain a human image is relatively small, the first 10 denoising processes of the SD model are directly skipped when iterating the denoising of the second noisy image, which improves the repair efficiency of image restoration of the erasing image.

[0204] The SD model uses PNDM to gradually denoise the noise obtained from the noise addition in step 749 until the noise addition in step 0 is completely denoised.

[0205] It should be noted that steps 605 to 606 and steps 607 to 608 are mutually exclusive steps. The electronic device may execute steps 605 to 606 but not steps 607 to 608, or execute steps 607 to 608 but not steps 605 to 606.

[0206] In one application scenario, such as Figure 13 As shown, when the initial noise input to the SD model is x T Or x K At that time, the SD model performs the reverse process of the diffusion model, iteratively denoising the initial noise multiple times, and finally obtains a noise image with a noise intensity of x'0.

[0207] Where x'1 is the noise intensity of the noise image obtained from the penultimate denoising step in the diffusion model direction process.

[0208] In one application scenario, when it is necessary to erase a person's image from an image to be erased, the image to be erased is as follows: Figure 14 As shown in (a), the image to be erased is subjected to a portrait erasing step followed by a 1000-step noise-adding process according to the first rule, resulting in a first noisy image. The first noisy image is then denoised using the SD model. When the first noisy image is iterated and denoised 30 times using the SD model, the first image is obtained, as shown in (a). Figure 14 As shown in (b); when the first noisy image is iterated and denoised 40 times based on the SD model, the second image is obtained, as shown in (b). Figure 14 As shown in (c) in the figure.

[0209] contrast Figure 14 (b) and Figure 14 (c) In the portrait erasure and repair scenario, due to the high intensity of the added noise, when the number of denoising steps is insufficient, the residual noise will cause the SD model to generate random portraits and the erasure will not be thorough.

[0210] When it is necessary to erase non-human figures from an image to be erased, such as... Figure 14 As shown in (d), the image to be erased undergoes a series of steps, including a non-human image erasure step and a 749-step noise reduction process according to the second rule, to obtain a second noisy image. This second noisy image is then denoised using the SD model. When the second noisy image is iterated and denoised 30 times using the SD model, a third image is obtained, as shown in (d). Figure 14 As shown in (e); when the second noisy image is iterated 40 times based on the SD model to obtain the fourth image, as shown in (e). Figure 14 As shown in (f) in the figure.

[0211] contrast Figure 14 (e) and Figure 14 In (f), in non-human portrait erasure and restoration scenarios, due to the low intensity of the added noise, when there are too many denoising steps, the background noise is excessively removed, resulting in the generation of disordered restoration images.

[0212] In one application scenario, Figure 11 Based on the flowchart shown, image inpainting methods can also include latent space encoding and latent space decoding processes, such as... Figure 15 As shown, the image restoration method may include steps 701 to 711.

[0213] Step 701: In response to the user's selection operation of the target pre-stored image of the electronic device, determine that the target pre-stored image is the image to be erased.

[0214] Step 702: Perform panoramic segmentation on the image to be erased to obtain multiple initial mask information.

[0215] Step 703: In response to the user's selection of the area to be erased in the image to be erased, generate an erasure mask image of the area to be erased.

[0216] Step 704: Input the image to be erased and the erasure mask image into the LaMa model, so that the LaMa model outputs the erasure image based on the image to be erased and the erasure mask image.

[0217] Step 705: The Encoder module based on the VAE model encodes the erased image to obtain a latent space encoded image.

[0218] Step 706: When it is determined that the area to be erased contains a human image, the latent space encoded image is denoised according to the first rule to obtain the first latent space denoised image.

[0219] Step 707: Input the noisy image in the first latent space and the prompt word into the Inpainting model to obtain the repaired image in the first latent space.

[0220] Step 708: The Decoder module based on the VAE model decodes the first latent space repair image to obtain the first repair image.

[0221] Step 709: When it is determined that the area to be erased does not contain a human image, the latent space encoded image is denoised according to the second rule to obtain the second latent space denoised image.

[0222] Step S710: Input the noisy image in the second latent space and the prompt word into the Inpainting model to obtain the repaired image in the second latent space.

[0223] Step 711: The Decoder module based on the VAE model decodes the second latent space repair image to obtain the second repair image.

[0224] Steps 706 to 708 and steps 709 to 711 are mutually exclusive steps. The electronic device does not execute steps 709 to 711 when executing steps 706 to 708, or does not execute steps 706 to 708 when executing steps 709 to 711.

[0225] The solution provided in this embodiment involves an electronic device responding to a user's selection of a target pre-stored image, determining the target pre-stored image as the image to be erased, performing panoramic segmentation on the image to be erased to obtain multiple initial mask information, and responding to the user's selection of the region to be erased in the image to be erased to generate an erasure mask image for the region to be erased. The image to be erased and the erasure mask image are then input into a LaMa model, causing the LaMa model to output an erased image based on the image to be erased and the erasure mask image. When it is determined that the region to be erased contains a human image, noise is added to the erased image according to a first rule to obtain a first noisy image, which is then input into the second noisy image. A noisy image and a prompt are fed into the Inpainting model to obtain a first repaired image. When it is determined that the area to be erased does not contain a human image, the erased image is denoised according to a second rule to obtain a second noisy image. The second noisy image and the prompt are then input into the Inpainting model to obtain a second repaired image. Since the noise intensity added to the erased image that does not contain a human image is relatively small, the number of times the SD model performs iterative denoising on the second noisy image is less than the number of times the SD model performs second denoising on the first noisy image when iterating denoising on the second noisy image, thus improving the repair efficiency of image restoration on the erased image after erasing non-human images.

[0226] Please see Figure 16 This illustration shows an image restoration apparatus 800 provided in one embodiment of this application. The image restoration apparatus 800 can be applied to electronic devices. The following example uses an electronic device to illustrate... Figure 16 The image restoration device 800 shown will be described in detail. The image restoration device 800 may include a first acquisition module 801, a second acquisition module 802, and a restoration input module 803.

[0227] The first acquisition module 801 can be used to acquire the erasure mask image of the area to be erased in the image to be erased; the second acquisition module 802 can be used to acquire the erasure image based on the image to be erased and the erasure mask image; the repair input module 803 can be used to input the erasure image and prompt words into the image repair model to obtain the repaired image. The prompt words can represent foreground objects and / or background objects in the adjacent area. The adjacent area can refer to the area in the erasure image that is adjacent to the area to be erased. The image repair model can be used to complete and repair the erasure area of ​​the erasure image based on the prompt words.

[0228] In some implementations, the image restoration model may include a text encoding model and a stable diffusion model. The text encoding model can be used to encode the prompt words to obtain prompt word features, and the stable diffusion model can be used to complete and restore the erased image based on the prompt word features to obtain a restored image.

[0229] In some implementations, the repair input module 803 may include a first noise-adding unit and a first input unit.

[0230] The first noise-adding unit can be used to add noise to the erased image to obtain a noisy image; the first input unit can be used to input the noisy image and prompt words into the image restoration model to obtain a restored image.

[0231] In some implementations, the noisy image may include a first noisy image, and the image restoration device 800 may also include a first determining module.

[0232] The first determining module can be used by the noise-adding unit to add noise to the erased image. Before obtaining the noise-adding image, it determines whether the area to be erased contains a human image.

[0233] In some implementations, the first noise-adding unit may include a first noise-adding subunit.

[0234] The first noise-adding subunit can be used to add noise to the image to be erased according to the first rule when it is determined that the area to be erased contains a human image, so as to obtain the first noise-adding image.

[0235] In some implementations, the first noise-adding subunit may include a noise-adding secondary subunit.

[0236] The noise-adding sub-unit can be used to add noise to the image to be erased according to the first rule for a first number of noise-adding steps when it is determined that the area to be erased contains a human image, so as to obtain the first noise-adding image.

[0237] In some implementations, the first rule can be used to characterize the rule that the noise intensity after each noise addition process satisfies the first noise formula; the first noise formula is: Where x0 is the noise intensity of the erased image, K is the number of noise additions, and α K Let α be the noise figure for the Kth noise addition. K ∈(0,1), α K The noise is uniformly distributed and is Gaussian noise.

[0238] In some embodiments, the noisy image may further include a second noisy image, and the image restoration apparatus 800 may further include a noisy module.

[0239] The noise-adding module can be used to add noise to the image to be erased according to the second rule when it is determined that the area to be erased does not contain a human image, so as to obtain a second noise-adding image. The first noise-adding step number corresponding to the first rule can be greater than the second noise-adding step number corresponding to the second noise-adding rule.

[0240] In some implementations, the noise-adding module may include a second noise-adding unit.

[0241] The second noise-adding unit can be used to add noise to the image to be erased according to the second rule for a second number of noise-adding steps when it is determined that the area to be erased does not contain a human image, so as to obtain a second noise-adding image.

[0242] In some implementations, the second rule can be used to characterize the rule that the noise intensity after each noise addition process satisfies the second noise formula; the second noise formula is: Where x0 is the noise intensity of the erased image, T α represents the number of times noise is added. T For the first T The noise figure of the second noise addition, α T ∈(0,1), α T The noise is uniformly distributed and is Gaussian noise.

[0243] In some implementations, the first determining module may include a first determining unit and a second determining unit.

[0244] The first determining unit can be used to determine the erasure mask category of the erasure mask image; the second determining unit can be used to determine whether the area to be erased contains a human image based on the erasure mask category.

[0245] In some embodiments, the image restoration apparatus 800 may further include a segmentation input module and a second determination module.

[0246] The segmentation input module can be used by the second acquisition module 802 to input the image to be erased into the panoramic segmentation model before acquiring the erased image based on the image to be erased and the erased mask image, and obtain multiple initial mask information. Each mask information can include an initial mask image corresponding to an instance object in the image to be erased and an initial mask category. The second determination module is used to determine the prompt word based on the erased mask image and the multiple initial mask information.

[0247] In some implementations, the second determining module may include a third determining unit and a first generating unit.

[0248] The third determining unit can be used to determine the first target initial mask category of the adjacent instance object in the adjacent region based on the erased mask image and multiple initial mask information; the first generating unit can be used to generate prompt words based on the first target initial mask category.

[0249] In some implementations, the first determining unit may include an acquisition subunit and a first determining subunit.

[0250] The acquisition subunit can be used to acquire the erase pixel position of the erase mask image; the first determination subunit can be used to determine the second initial mask category corresponding to the target instance object in the image to be erased that matches the erase pixel position as the erase mask category of the erase mask image.

[0251] In some implementations, the second determining unit may be a second determining subunit and a third determining subunit.

[0252] The second determining subunit can be used to determine that the area to be erased contains a human image when the erasure mask category includes a human mask; the third determining subunit can be used to determine that the area to be erased does not contain a human image when the erasure mask category does not include a human mask.

[0253] In some implementations, the repair input module 803 may include an encoding unit, a second input unit, and a decoding unit.

[0254] The encoding unit can be used to encode the erased image using an encoding module based on a variational autoencoder model to obtain a latent space encoded image; the second input unit can be used to input the latent space encoded image and prompt words into the image restoration model to obtain a latent space restored image; the decoding unit can be used to decode the latent space restored image using a decoding module based on a variational autoencoder model to obtain a restored image.

[0255] In some implementations, the second input unit may include a second noise-adding subunit and an input subunit.

[0256] The second noise-adding subunit can be used to add noise to the latent space encoded image to obtain a latent space noisy image; the input subunit can be used to input the latent space noisy image and prompt words into the image restoration model to obtain a latent space restored image.

[0257] In some implementations, the first acquisition module 801 may include a receiving unit and a second generation unit.

[0258] The receiving unit can be used to receive the user's selection operation of the area to be erased in the image to be erased; the second generating unit can be used to generate an erasure mask image in response to the selection operation.

[0259] In some implementations, the repair input module 803 may also include a third input unit.

[0260] The third input unit can be used to input the image to be erased and the erasure mask image into the LaMa model, so that the LaMa model outputs the erasure image based on the image to be erased and the erasure mask image.

[0261] The solution provided in this embodiment involves an electronic device responding to a user's selection of a target pre-stored image, determining the target pre-stored image as the image to be erased, and responding to the user's selection of an area to be erased within the image to be erased, generating an erasure mask image for the area to be erased, obtaining the erased image based on the image to be erased and the erasure mask image, and inputting the erased image and a prompt word into an Inpainting model to obtain a repaired image. The prompt word indicates foreground and / or background objects in adjacent areas. Adjacent areas refer to regions in the image to be erased that are adjacent to the area to be erased. This model is used to complete and repair the erased areas of an erased image based on prompts. It realizes image completion based on the Inpainting model, which indicates foreground and / or background objects in the adjacent areas. This allows the Inpainting model to complete the erased areas of the erased image more seamlessly with the foreground and / or background objects in the adjacent areas, avoiding random generation during the image restoration process, improving the image completion effect and enhancing the image restoration quality.

[0262] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For device embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to in the description of the method embodiments. Any processing method described in the method embodiments can be implemented in the device embodiments through corresponding processing modules, and will not be elaborated upon further in the device embodiments.

[0263] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0264] Please see Figure 17 This illustrates a schematic diagram of the hardware structure of an electronic device 900 provided in one embodiment of this application. Figure 17As shown, the electronic device 900 may include a processor 910, an external memory interface 920, an internal memory 921, a Universal Serial Bus (USB) interface 930, a charging management module 940, a power management module 941, a battery 942, an antenna 1, an antenna 2, a mobile communication module 950, a wireless communication module 960, an audio module 970, a speaker 970A, a receiver 970B, a microphone 970C, a headphone jack 970D, a sensor module 980, buttons 990, a motor 991, an indicator 992, a camera 993, a display screen 994, and a Subscriber Identification Module (SIM) card interface 995, etc. The sensor module 980 may include a pressure sensor 980A, a gyroscope sensor 980B, a barometric pressure sensor 980C, a magnetic sensor 980D, an accelerometer sensor 980E, a distance sensor 980F, a proximity light sensor 980G, a fingerprint sensor 980H, a temperature sensor 980J, a touch sensor 980K, an ambient light sensor 980L, a bone conduction sensor 980M, etc.

[0265] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 900. In other embodiments of this application, the electronic device 900 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0266] Please see Figure 18 This illustrates a functional block diagram of an electronic device 1000 according to an embodiment of this application. Figure 18 As shown, the electronic device 1000 includes: one or more processors 1010 ( Figure 18 Only one processor is shown in the diagram) and a memory 1020, which is coupled to one or more processors 1010. The memory 1020 is used to store computer program code 1030, which includes computer instructions. One or more processors 1010 call the computer instructions to cause the electronic device 1000 to perform the steps in any of the above methods.

[0267] Those skilled in the art will understand that Figure 18This is merely an example of electronic device 1000 and does not constitute a limitation on electronic device 1000. In practice, electronic device 1000 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input / output devices, network access devices, etc. Electronic device 1000 may also be the same device as electronic device 900 described in the above embodiments.

[0268] The processor 1010 can be a Central Processing Unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0269] In some embodiments, memory 1020 may be an internal storage unit of electronic device 1000, such as a hard disk or memory of electronic device 1000. In other embodiments, memory 1020 may be an external storage device of electronic device 1000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on electronic device 1000. Optionally, memory 1020 may include both internal and external storage units of electronic device 1000. Memory 1020 is used to store operating system, application programs, bootloaders, data, and other programs, such as program code of computer programs. Memory 1020 may also be used to temporarily store data that has been output or will be output.

[0270] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0271] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0272] This application also provides a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to implement the steps in any of the above methods.

[0273] In some implementations, the chip system also includes a memory connected to one or more processors via circuitry or wiring.

[0274] In some implementations, the chip system also includes a communication interface.

[0275] This application also provides a computer-readable medium including instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the above-described method embodiments.

[0276] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform the aforementioned related steps to implement the methods described in the various method embodiments above.

[0277] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0278] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0279] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0280] In the embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0281] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0282] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0283] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0284] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0285] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0286] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image restoration method, characterized in that, include: Obtain the erasure mask image of the area to be erased in the image to be erased; The erasure image is obtained based on the image to be erased and the erasure mask image; The erased image and prompt words are input into the image restoration model to obtain the restored image. The prompt words represent foreground objects and / or background objects in the adjacent region. The adjacent region refers to the region in the image to be erased that is adjacent to the region to be erased. The image restoration model is used to complete and restore the erased region of the erased image according to the prompt words. The step of inputting the erased image and the prompt word into the image restoration model to obtain the restored image includes: Determine whether the area to be erased contains a human image; When it is determined that the area to be erased contains a human image, the erasing image is subjected to noise processing according to a first rule to obtain a first noisy image. The first rule includes adding noise to the erasing image with a first number of noise-adding steps. Input the first noisy image and the prompt word into the image restoration model to obtain the first restored image; When it is determined that the area to be erased does not contain a human image, the erased image is denoised according to the second rule to obtain a second denoised image. The second rule includes denoising the erased image with a second number of denoising steps, where the first number of denoising steps is greater than the second number of denoising steps. Input the second noisy image and the prompt word into the image restoration model to obtain the second restored image; or, The process of inputting the erased image and prompt words into the image restoration model to obtain the restored image includes: The encoding module based on the variational autoencoder model encodes the erased image to obtain a latent space encoded image. When it is determined that the area to be erased contains a human image, the latent space encoded image is denoised according to the first rule to obtain a first latent space denoised image. The first rule includes denoising the latent space encoded image with the first number of denoising steps. Input the first noisy latent space image and the prompt word into the image restoration model to obtain the first latent space restored image; The decoding module based on the variational autoencoder model decodes the first latent space repair image to obtain the first repair image. When it is determined that the area to be erased does not contain a human image, the latent space encoded image is denoised according to the second rule to obtain a second latent space denoised image. The second rule includes denoising the latent space encoded image with the second denoising step number, where the first denoising step number is greater than the second denoising step number. Input the second latent space noisy image and the prompt word into the image restoration model to obtain the second latent space restored image; The decoding module based on the variational autoencoder model decodes the second latent space repair image to obtain the second repair image.

2. The image restoration method according to claim 1, characterized in that, The image restoration model includes a text encoding model and a stable diffusion model. The text encoding model is used to encode the prompt words to obtain prompt word features. The stable diffusion model is used to complete and restore the erased image based on the prompt word features to obtain the restored image.

3. The image restoration method according to claim 1 or 2, characterized in that, The first rule is used to characterize the rule that the noise intensity after each noise addition process satisfies the first noise formula; The first noise formula is: ; Among them, the x 0 represents the noise intensity of the erased image. K For the number of noise additions, the α K For the first K The noise figure of the second noise addition, the α K ∈ (0, 1), the α K For uniform distribution, the noise It is Gaussian noise.

4. The image restoration method according to claim 1 or 2, characterized in that, The second rule is used to characterize the rule that the noise intensity after each noise addition process satisfies the second noise formula; The second noise formula is: ; Among them, the x 0 represents the noise intensity of the erased image. T For the number of noise additions, the α T For the first T The noise figure of the second noise addition, the α T ∈ (0, 1), the α T For uniform distribution, the noise It is Gaussian noise.

5. The image restoration method according to any one of claims 1 to 4, characterized in that, Determining whether the area to be erased contains a human image includes: Determine the erase mask category of the erase mask image; Based on the erase mask type, determine whether the area to be erased contains a human image.

6. The image restoration method according to claim 5, characterized in that, Before obtaining the erasure image based on the image to be erased and the erasure mask image, the image restoration method further includes: Input the image to be erased into the panoramic segmentation model to obtain multiple initial mask information. Each mask information includes an initial mask image corresponding to an instance object in the image to be erased and an initial mask category. Based on the erased mask image and the multiple initial mask information, a prompt word is determined.

7. The image restoration method according to claim 6, characterized in that, The step of determining prompt words based on the erased mask image and the multiple initial mask information includes: Based on the erased mask image and the multiple initial mask information, determine the first target initial mask category of the adjacent instance objects in the adjacent region; Based on the initial mask category of the first target, generate prompt words.

8. The image restoration method according to claim 6 or 7, characterized in that, Determining the erase mask category of the erase mask image includes: Obtain the erase pixel positions of the erase mask image; The second target initial mask category corresponding to the target instance object in the image to be erased that matches the position of the erased pixel is determined as the erase mask category of the erase mask image.

9. The image restoration method according to any one of claims 5 to 8, characterized in that, The step of determining whether the area to be erased contains a human image based on the erase mask type includes: When the erase mask category includes a human mask, it is determined that the area to be erased contains a human image; When the erase mask category does not include a human mask, it is determined that the area to be erased does not contain a human image.

10. The image restoration method according to any one of claims 1 to 9, characterized in that, The process of obtaining the erasure mask image of the area to be erased in the image to be erased includes: Receive the user's selection operation for the area to be erased in the image to be erased; In response to the selection operation, the erase mask image is generated.

11. The image restoration method according to any one of claims 1 to 10, characterized in that, The step of obtaining the erasure image based on the image to be erased and the erasure mask image includes: The image to be erased and the erasure mask image are input into the image erasure model, so that the image erasure model outputs the erasure image based on the image to be erased and the erasure mask image.

12. The image restoration method according to claim 11, characterized in that, The image erasure model includes the LaMa model.

13. An image restoration device, characterized in that, include: The first acquisition module is used to acquire the erasure mask image of the area to be erased in the image to be erased; The second acquisition module is used to acquire the erasure image based on the image to be erased and the erasure mask image; The repair input module is used to input the erased image and prompt words into the image repair model to obtain a repaired image. The prompt words represent foreground objects and / or background objects in the vicinity. The vicinity refers to the area in the erased image that is adjacent to the area to be erased. The image repair model is used to complete and repair the erased area of ​​the erased image according to the prompt words. Specifically, the repair input module is used for: Determine whether the area to be erased contains a human image; When it is determined that the area to be erased contains a human image, the erasing image is subjected to noise processing according to a first rule to obtain a first noisy image. The first rule includes adding noise to the erasing image with a first number of noise-adding steps. Input the first noisy image and the prompt word into the image restoration model to obtain the first restored image; When it is determined that the area to be erased does not contain a human image, the erased image is denoised according to the second rule to obtain a second denoised image. The second rule includes denoising the erased image with a second number of denoising steps, where the first number of denoising steps is greater than the second number of denoising steps. Input the second noisy image and the prompt word into the image restoration model to obtain the second restored image; Alternatively, the repair input module is specifically used for: The encoding module based on the variational autoencoder model encodes the erased image to obtain a latent space encoded image. When it is determined that the area to be erased contains a human image, the latent space encoded image is denoised according to the first rule to obtain a first latent space denoised image. The first rule includes denoising the latent space encoded image with the first number of denoising steps. Input the first noisy latent space image and the prompt word into the image restoration model to obtain the first latent space restored image; The decoding module based on the variational autoencoder model decodes the first latent space repair image to obtain the first repair image. When it is determined that the area to be erased does not contain a human image, the latent space encoded image is denoised according to the second rule to obtain a second latent space denoised image. The second rule includes denoising the latent space encoded image with the second denoising step number, where the first denoising step number is greater than the second denoising step number. Input the second latent space noisy image and the prompt word into the image restoration model to obtain the second latent space restored image; The decoding module based on the variational autoencoder model decodes the second latent space repair image to obtain the second repair image.

14. An electronic device, characterized in that, The electronic device includes: one or more processors, and a memory; The memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the electronic device to perform the image restoration method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the image restoration method as described in any one of claims 1 to 12.

Images