Image inpainting method and apparatus, and device, medium and program product

Abstract

Provided in the embodiments of the present disclosure are an image inpainting method and apparatus, and a device, a medium and a program product. The image inpainting method comprises: in response to a trigger operation for a target control, displaying an image inpainting interface, wherein the image inpainting interface comprises an image to be inpainted and a reference image loading control; in response to a loading operation for the reference image loading control, displaying a reference image in the image inpainting interface; and in response to an inpainting trigger operation, displaying a target image in the image inpainting interface, wherein the target image is obtained by means of inpainting, on the basis of the reference image, the image to be inpainted. In the embodiments of the present disclosure, a reference image that meets the photographing expectation of a user can be selected to inpaint an image to be inpainted, such that the effect of an inpainted target image meets the photographing expectation, thereby realizing convenient image inpainting.

Description

Image restoration methods, devices, equipment, media and program products

[0001] This application claims priority to Chinese Patent Application No. 202410683637.9, filed on May 29, 2024, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to an image restoration method, apparatus, device, medium, and program product. Background Technology

[0003] With the development of computer technology, the shooting devices in electronic devices can achieve increasingly higher and higher image quality. In everyday scenarios, users can use electronic devices to take photos. For example, they can take group photos and travel photos using mobile devices.

[0004] Since the shooting effect of photos or videos can only be seen after they are taken, some scenes that cannot be replicated may fail to meet the shooting expectations due to the user's failure to manage facial expressions, such as closing their eyes, opening their mouth, poor eye contact, or poor facial expressions. It is also difficult to restore the image by recreating the scene. Summary of the Invention

[0005] This disclosure provides an image restoration method, apparatus, device, medium, and program product that can conveniently restore images taken by users to meet shooting expectations.

[0006] In a first aspect, embodiments of this disclosure provide an image restoration method, including:

[0007] In response to a trigger operation on the target control, an image repair interface is displayed, which includes an image to be repaired and a reference image loading control;

[0008] In response to a loading operation on the reference image loading control, a reference image is displayed on the image restoration interface;

[0009] In response to a repair trigger operation, a target image is displayed on the image repair interface, wherein the target image is obtained by repairing the image to be repaired based on the reference image.

[0010] Secondly, embodiments of this disclosure also provide an image restoration apparatus, the apparatus comprising:

[0011] The interface display module is configured to display an image restoration interface in response to a trigger operation on a target control. The image restoration interface includes a loading control for the image to be restored and a reference image.

[0012] An image display module is configured to display a reference image in the image restoration interface in response to a loading operation of the reference image loading control.

[0013] An image restoration module is configured to display a target image on the image restoration interface in response to a restoration trigger operation, wherein the target image is obtained by restoring the image to be restored based on the reference image.

[0014] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0015] One or more processors;

[0016] Storage device for storing one or more programs.

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the image restoration method as described in any embodiment of this disclosure.

[0018] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the image restoration method as described in any embodiment of this disclosure.

[0019] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the image restoration method as described in any embodiment of this disclosure. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 is a schematic flowchart of an image restoration method provided in an embodiment of this disclosure;

[0022] Figure 2 is a schematic diagram of an image restoration interface provided in an embodiment of this disclosure;

[0023] Figure 3 is a schematic diagram of a reference image loading process provided in an embodiment of this disclosure;

[0024] Figure 4 is a schematic diagram of an image restoration device provided in an embodiment of this disclosure;

[0025] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0029] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0030] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0031] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0032] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0033] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0034] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0035] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0036] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0037] Figure 1 is a schematic flowchart of an image restoration method provided in an embodiment of this disclosure. This embodiment is applicable to image restoration, particularly to images where facial expression management fails in scenarios that cannot be replicated. Examples include images with closed eyes, open mouths, unnatural expressions, or poor eye contact. This method can be executed by an image restoration device, which can be implemented in software and / or hardware, optionally through an electronic device such as a mobile terminal or PC.

[0038] As shown in Figure 1, the image restoration method includes the following steps S110 to S130.

[0039] S110. In response to a trigger operation on the target control, display an image repair interface, the image repair interface including the image to be repaired and a reference image loading control.

[0040] For example, the target control can represent the control that triggers the display of the image restoration interface. The image restoration interface is used for interactive operations related to image restoration. For example, the image restoration interface includes an image to be restored and a reference image loading control. In some embodiments, the image restoration interface may include an image to be restored, a reference image loading control, and candidate restoration items. The reference image loading control is used to trigger an image loading operation. If a trigger operation is entered for the reference image loading control, the user is redirected to an image selection interface to select a reference image. Candidate restoration items include expression restoration, closed eye restoration, eye gaze restoration, and mouth restoration.

[0041] For example, in response to a triggering operation on a target control, displaying an image restoration interface includes: displaying a candidate control collection interface, the candidate control collection interface including the target control; and in response to a click operation on the target control, displaying the image restoration interface.

[0042] For example, the candidate control collection interface includes control information for the candidate controls. After selecting the image to be repaired, if the image is identified to include objects such as portraits or animals, the candidate control collection interface is displayed. Optionally, the candidate control collection interface is displayed in response to input editing operations. For example, the candidate control collection interface includes one-click beautification, makeup, and target controls. If a click operation on a target control in the candidate control collection interface is detected, the image repair interface is displayed.

[0043] Figure 2 is a schematic diagram of an image restoration interface provided in an embodiment of this disclosure. As shown in Figure 2, the candidate control collection interface 200 displays the target control 210. If the target control 210 is clicked, the image restoration interface 220 is displayed. The image restoration interface 220 may include an image display area 230 and an information setting area 240, etc. The image to be restored is displayed in the image display area 230. The information setting area 240 includes a reference image loading control 250 and candidate restoration items 260, etc.

[0044] S120. In response to the loading operation of the reference image loading control, the reference image is displayed on the image restoration interface.

[0045] For example, a reference image can be used to repair the image to be repaired. The reference image can be a local image or an image downloaded from the internet. The reference image can be an image containing objects from the image to be repaired, and the objects have similar angles. For example, the reference image can contain a person from the image to be repaired, and the person's facial angle in the reference image is similar to that in the image to be repaired. It should be noted that the reference image can also be an image containing a different person from the person in the image to be repaired but with similar angles. Alternatively, the reference image can contain an animal from the image to be repaired, and the animal's facial angle in the reference image is similar to that in the image to be repaired. This disclosure does not limit the objects in the reference image and the image to be repaired to be the same object, nor does it limit the facial angle of the object in the reference image to be consistent with the object to be repaired. When the angular deviation between the facial angle of the object in the reference image and the facial angle of the object to be repaired exceeds a set threshold, the error in the generated image caused by the large angular deviation can be overcome by a pre-trained generative model. For example, during the training phase of the generative model, some image samples with large angular deviations can be used to train the generative model, so that the generative model learns from the experience of generating repaired images that meet the expected results from images with large angular deviations.

[0046] It should be noted that if the reference image includes multiple faces, a prompt message is output. In some embodiments, the prompt message is used to suggest selecting the target face for reference. For example, if the reference image includes multiple faces, a prompt message is output to suggest selecting one face as the target face for reference. Alternatively, the prompt message may suggest reselecting the reference image. In other embodiments, the target face among the multiple faces in the reference image with an angle similar to the face of the object to be repaired can be automatically determined; this disclosure does not specifically limit this. In still other embodiments, if the image to be repaired is a group photo, a reference image with multiple faces can be selected, and the correspondence between the faces in the reference image and the object to be repaired in the image to be repaired can be specified. This allows for the repair of different objects in the group photo based on the reference features of different faces in the reference image, achieving diverse repair options and enriching the image display effect.

[0047] For example, in response to a loading operation on a reference image loading control, displaying a reference image on an image restoration interface includes: in response to a triggering operation on a reference image loading control, displaying an image selection interface; and in response to an image selection operation on the image selection interface, displaying a thumbnail of the reference image at the position corresponding to the reference image loading control.

[0048] For example, the image selection interface includes multiple candidate images and selection prompts. The prompts suggest selecting a reference image. For instance, it could suggest selecting an image with an angle similar to the object in the image to be repaired, thus improving repair speed.

[0049] Figure 3 is a schematic diagram of a reference image loading process provided in an embodiment of this disclosure. As shown in Figure 3, clicking the reference image loading control 310 in the image restoration interface 300 displays the image selection interface 320. The image selection interface 320 includes multiple candidate images, etc. Based on the reference image selection operation, a thumbnail of the reference image is displayed at the position corresponding to the reference image loading control 310.

[0050] Optionally, the image restoration method further includes: replacing the reference image in the image restoration interface in response to a replacement trigger operation for the reference image.

[0051] For example, a replacement trigger operation represents an action that triggers the replacement of the reference image. Examples of replacement trigger operations include clicking the image loading control again, or using voice input to change the reference image.

[0052] If a click on the reference image loading control is detected, it is considered a replacement trigger operation for the reference image, and the image selection interface is displayed again. A new reference image is then selected from the image selection interface, and its thumbnail is loaded into the corresponding position of the reference image loading control. By flexibly setting the reference image, the image restoration effects can be enriched.

[0053] S130. In response to the repair trigger operation, a target image is displayed on the image repair interface, wherein the target image is obtained by repairing the image to be repaired based on the reference image.

[0054] For example, a repair trigger operation can be used to trigger the repair of the image to be repaired. For instance, a repair trigger operation may include selecting candidate repair items, initiating repair via voice input, or inputting a repair gesture. This disclosure does not specifically limit the meaning of the repair trigger operation.

[0055] Specifically, after selecting a reference image, candidate restoration items become operable. In response to a selection operation on a candidate restoration item, the selected candidate restoration item can be identified as the target restoration item. The target restoration item represents the area to be restored in the image. For example, the target restoration item can be expression restoration, closed eye restoration, eye gaze restoration, or mouth restoration, etc. Correspondingly, the area to be restored can be the eyes and / or mouth, etc.

[0056] For example, in response to the selection operation for the candidate repair item, a target repair item is determined; based on the reference features in the reference image corresponding to the target repair item, the target features in the image to be repaired corresponding to the target repair item are repaired to obtain a candidate image; in response to the selection operation for the candidate image, a target image is determined and the target image is displayed on the image repair interface.

[0057] Reference features characterize the image features in the reference image that correspond to the target restoration item. For example, reference features may include physiological features of the eyes or mouth. Physiological features of the eyes can characterize the iris, sclera, and other eye details. Physiological features of the mouth can characterize the lips, corners of the mouth, teeth, and gums. Target features characterize the image features in the image to be restored that correspond to the target restoration item. For example, target features may include eye makeup features or makeup features of the mouth area.

[0058] Optionally, based on the reference features in the reference image corresponding to the target restoration item, the target features in the image to be restored corresponding to the target restoration item are restored to obtain a candidate image, including: obtaining the reference features in the reference image corresponding to the target restoration item; obtaining the target features in the image to be restored corresponding to the target restoration item; inputting the reference features and target features into a preset generation model to obtain the candidate image output by the preset generation model.

[0059] For example, a pre-defined generative model converts the target features of the image to be repaired into pure noise through a forward diffusion process, and then, guided by reference features, recovers a generated result similar to the reference features from the pure noise through a reverse diffusion process. The similarity can be calculated using an attention module.

[0060] Reference features and target features are input into a preset generation model. The generation result and corresponding confidence level are determined based on the preset generation model. Multiple generation results are selected based on the confidence level and fused into the image to be repaired to obtain multiple candidate images.

[0061] Taking closed-eye restoration as an example, since the image to be restored is a closed-eye image, it's impossible to obtain eye feature information from a closed-eye image. To solve this problem, a clear image with open eyes is selected to provide the physiological characteristics of the eyes as reference features. Since the purpose of closed-eye restoration is to redraw the eye area of ​​the image to be restored, preserving the makeup information of the eyes in the image to be restored, and generating a naturally open-eye image based on the reference features, the redrawn image has eye makeup consistent with the image to be restored and eyes consistent with the reference image.

[0062] Optionally, obtaining the target features corresponding to the target repair item in the image to be repaired includes: if the image to be repaired includes at least two objects to be repaired, displaying a prompt message, wherein the prompt message is used to prompt the selection of a target repair object from the at least two objects to be repaired; in response to the object selection operation, determining the target repair object among the at least two objects to be repaired; and obtaining the target features corresponding to the target repair object and the target repair item.

[0063] For example, if the image to be repaired includes at least two faces, a prompt is displayed to suggest selecting the target face from the at least two faces. Based on the selection operation, the target face is selected from the at least two faces. Target features corresponding to the target repair item are obtained from the target face. Taking the target repair item as closed-eye repair as an example, the target features represent the physiological information of the target face's eyes. For example, the physiological information of the eyes includes the iris, the sclera, and other detailed information about the eyes.

[0064] If the image to be repaired includes at least two faces, the corresponding regions of each face and the target repair item can be identified to determine the target repair object. The target features corresponding to the target repair object and the target repair item are then obtained. Taking the target repair item as an example of closed-eye repair, if the image to be repaired includes at least two faces, the target face with closed eyes among the at least two faces is identified. The physiological information of the eyes in the target face is obtained as the target features.

[0065] Optionally, the reference features and target features are input into a preset generation model to obtain candidate images output by the preset generation model, including: injecting the reference features and target features into the noise prediction network of the preset generation model through an attention module; and generating candidate images through the noise prediction network.

[0066] For example, the image to be repaired is input into the compression module of a preset generation model. The compression module compresses the image to a low-dimensional latent space, obtaining the features of the image to be repaired. These features are then input into the noise prediction network of the preset generation model. Through forward diffusion, noise is added to the region of the image to be repaired corresponding to the target repair item, resulting in a local noise map. If the target repair item is closed-eye repair, noise is added to the eye region of the image to be repaired, resulting in a local noise map. In the local noise map, only the eye region is pure noise; the remaining regions represent the features of the image to be repaired in the latent space. Reference features and target features are injected into the noise prediction network of the preset generation model through an attention module. Thus, during the backdiffusion process, the noise prediction network recovers a generation result similar to the reference features from the local noise map based on the reference features. For example, during the backdiffusion process, the noise prediction network recovers a naturally open eye from the local noise map based on the physiological features of the eye in the reference image, while other regions retain the image features of the image to be repaired. Finally, the decompression module restores the repaired generated image to the same size as the image to be repaired, obtaining the generated result. Optionally, generated results with a confidence level exceeding a set threshold can be used as candidate images.

[0067] Taking closed-eye restoration as an example, makeup features are extracted from the image to be restored and injected into the noise prediction network through an attention module. Eye physiological features are extracted from the reference image and injected into the noise prediction network through the attention module. Thus, during the noise reduction process, the noise prediction network generates eye features similar to the reference image while preserving the makeup features of the image to be restored.

[0068] The technical solution of this disclosure embodiment triggers the display of an image restoration interface through a target control. The image restoration interface includes an image to be restored and a reference image loading control. A loading operation is input to the reference image loading control to load the reference image into the image restoration interface. Then, based on the restoration trigger operation, the image to be restored is restored using the reference image to obtain the target image, which is then displayed on the image restoration interface. This disclosure embodiment can restore the image to be restored by selecting a reference image that meets the user's shooting expectations, ensuring that the restored target image meets the shooting expectations, thus achieving convenient image restoration. For some scenes that cannot be replicated, this disclosure embodiment restores the target object in the image to be restored based on the reference image, without needing to restore the scene, thus solving the problem of unsatisfactory shooting results and difficult image restoration, improving the user experience.

[0069] Figure 4 is a schematic diagram of an image restoration device provided in an embodiment of this disclosure. The device can be implemented in the form of software and / or hardware, and optionally, it can be implemented by an electronic device, such as a mobile terminal or a PC.

[0070] As shown in Figure 4, the image restoration device includes: an interface display module 410, an image display module 420, and an image restoration module 430.

[0071] The interface display module 410 is configured to display an image repair interface in response to a trigger operation on a target control. The image repair interface includes a loading control for the image to be repaired and a reference image.

[0072] The image display module 420 is configured to display a reference image in the image restoration interface in response to a loading operation of the reference image loading control;

[0073] The image restoration module 430 is configured to display a target image on the image restoration interface in response to a restoration trigger operation, wherein the target image is obtained by restoring the image to be restored based on a reference image.

[0074] Optionally, the interface display module 410 is specifically configured as follows:

[0075] Displays a collection of candidate controls, which includes the target control;

[0076] In response to a click on the target control, the image repair interface is displayed.

[0077] Optionally, the image display module 420 is specifically configured as follows:

[0078] In response to a trigger action on the reference image loading control, display the image selection interface;

[0079] In response to an image selection operation on the image selection interface, a thumbnail of the reference image is displayed at the location corresponding to the reference image loading control.

[0080] Optionally, the image restoration apparatus also includes:

[0081] The replacement module is configured to replace the reference image in the image restoration interface in response to a replacement trigger operation for the reference image.

[0082] Optionally, the image restoration interface also includes candidate restoration items;

[0083] The image restoration module 430 is specifically configured as follows:

[0084] In response to the selection operation for the candidate repair items, a target repair item is determined;

[0085] Based on the reference features in the reference image that correspond to the target restoration item, the target features in the image to be restored that correspond to the target restoration item are restored to obtain a candidate image;

[0086] In response to the selection operation for candidate images, the target image is determined and displayed in the image restoration interface.

[0087] Further, based on the reference features in the reference image corresponding to the target restoration item, the target features in the image to be restored corresponding to the target restoration item are restored to obtain candidate images, including:

[0088] Obtain the reference features corresponding to the reference image and the target inpainting item;

[0089] Obtain the target features in the image to be repaired that correspond to the target repair item;

[0090] Input the reference features and target features into the preset generation model to obtain the candidate images output by the preset generation model.

[0091] Further, the target features corresponding to the target restoration item in the image to be restored are obtained, including:

[0092] If the image to be repaired includes at least two objects to be repaired, a prompt message is displayed, which prompts the user to select the target object to be repaired from the at least two objects to be repaired.

[0093] In response to an object selection operation, identify the target object to be repaired from at least two objects to be repaired.

[0094] Obtain the target features corresponding to the target repair object and the target repair item.

[0095] Furthermore, the reference features and target features are input into a preset generation model to obtain candidate images output by the preset generation model, including:

[0096] The reference features and target features are injected into the noise prediction network of the pre-defined generative model through the attention module;

[0097] Candidate images are generated using a noise prediction network.

[0098] The image restoration apparatus provided in this disclosure can execute the image restoration method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0099] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0100] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Referring now to Figure 5, a schematic diagram of the structure of an electronic device (e.g., the terminal device in Figure 5) 500 suitable for implementing an embodiment of this disclosure is shown. The terminal device in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in Figure 5 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this disclosure.

[0101] As shown in Figure 5, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.

[0102] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 shows an electronic device 500 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0103] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0104] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0105] The electronic device provided in this embodiment and the image restoration method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0106] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the image restoration method provided in the above embodiments.

[0107] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0108] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0109] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0110] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0111] In response to a trigger operation on the target control, an image restoration interface is displayed, which includes a loading control for the image to be restored and a reference image;

[0112] In response to a loading operation on the reference image loading control, the reference image is displayed in the image restoration interface;

[0113] In response to the repair trigger operation, the target image is displayed in the image repair interface, where the target image is obtained by repairing the image to be repaired based on the reference image.

[0114] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0116] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0117] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0118] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0119] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0120] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0121] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image inpainting method, comprising: in response to a trigger operation on a target control, displaying an image inpainting interface, the image inpainting interface comprising a to-be-inpainted image and a reference image loading control; in response to a loading operation on the reference image loading control, displaying a reference image in the image inpainting interface; in response to an inpainting trigger operation, displaying a target image in the image inpainting interface, wherein the target image is obtained by inpainting the to-be-inpainted image based on the reference image.

2. The method of claim 1, wherein, The displaying of the image inpainting interface in response to the trigger operation on the target control comprises: displaying a candidate control set interface, the candidate control set interface comprising the target control; in response to a click operation on the target control, displaying the image inpainting interface.

3. The method according to claim 1 or 2, wherein, The displaying of the reference image in the image inpainting interface in response to the loading operation on the reference image loading control comprises: in response to a trigger operation on the reference image loading control, displaying an image selection interface; in response to an image selection operation on the image selection interface, displaying a thumbnail of the reference image at a position corresponding to the reference image loading control.

4. The method of any one of claims 1-3, further comprising: in response to a replacement trigger operation on the reference image, replacing the reference image in the image inpainting interface.

5. The method according to any one of claims 1 to 4, wherein, The image inpainting interface further comprises a candidate inpainting item; The displaying of the target image in the image inpainting interface in response to the inpainting trigger operation comprises: in response to a selection operation on the candidate inpainting item, determining a target inpainting item; based on a reference feature corresponding to the target inpainting item in the reference image, inpainting a target feature corresponding to the target inpainting item in the to-be-inpainted image to obtain a candidate image; in response to a selection operation on the candidate image, determining a target image and displaying the target image in the image inpainting interface.

6. The method of claim 5, wherein, The inpainting of the target feature corresponding to the target inpainting item in the to-be-inpainted image based on the reference feature corresponding to the target inpainting item in the reference image to obtain the candidate image comprises: obtaining the reference feature corresponding to the target inpainting item in the reference image; obtaining the target feature corresponding to the target inpainting item in the to-be-inpainted image; inputting the reference feature and the target feature into a preset generation model to obtain a candidate image output by the preset generation model.

7. The method of claim 6, wherein, The obtaining of the target feature corresponding to the target inpainting item in the to-be-inpainted image comprises: if the to-be-inpainted image comprises at least two to-be-inpainted objects, displaying prompt information, wherein the prompt information is used to prompt selection of a target inpainting object from the at least two to-be-inpainted objects; in response to an object selection operation, determining a target inpainting object from the at least two to-be-inpainted objects; obtaining a target feature of the target inpainting object corresponding to the target inpainting item.

8. The method of claim 6 or 7, wherein, The inputting of the reference feature and the target feature into the preset generation model to obtain the candidate image output by the preset generation model comprises: injecting the reference feature and the target feature into a noise prediction network of the preset generation model through an attention module; generating, by the noise prediction network, the candidate image. 9.An image inpainting apparatus, comprising: an interface display module configured to display an image inpainting interface in response to a trigger operation on a target control, the image inpainting interface comprising a to-be-inpainted image and a reference image loading control; an image display module configured to display a reference image in the image inpainting interface in response to a loading operation on the reference image loading control; an image inpainting module configured to display a target image in the image inpainting interface in response to an inpainting trigger operation, wherein the target image is obtained by inpainting the to-be-inpainted image based on the reference image. 10.An electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the image inpainting method according to any one of claims 1-8.

11. A storage medium containing computer-executable instructions, wherein, The computer executable instructions when executed by a computer processor are for performing the image inpainting method according to any one of claims 1-8.

12. A computer program product comprising a computer program, wherein, The computer program when executed by a processor implements the image inpainting method according to any one of claims 1-8.

Images