Image inpainting method and apparatus, and computer device cluster

Abstract

An image inpainting method, comprising: displaying a first image; detecting a first operation, the first operation being an operation of selecting a first area in the first image; determining a reference area in the first image; and inpainting the first area in the first image on the basis of the reference area, wherein the degree of attention on the reference area is increased and / or decreased in the process of inpainting the first area. In this way, in an image inpainting process, attention can be focused on a specific area in an image by increasing and / or decreasing the attention to certain areas, so that features in the specific area can be better used for inpainting, the rationality of image inpainting is improved, artifacts such as ghosting are reduced, and an inpainting result can better conform to the expectations of users, thereby improving the image inpainting effect.

Description

Image inpainting method, device and computing device cluster

[0001] The present application claims priority to the Chinese patent application No. 202410874261.X, filed on July 1, 2024, and entitled "Image inpainting method, device and computing device cluster", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI), and in particular, to an image inpainting method, device and computing device cluster. BACKGROUND

[0003] At present, mobile phones and other computing devices with photographing functions have become indispensable tools in people's daily life. Users can take pictures of scenes such as scenery or objects around them at any time through computing devices with photographing functions, thereby bringing more fun to life. In the pictures taken by users, there are often some elements that users do not want to exist, such as passers-by, shadows, etc. Therefore, users want to be able to eliminate these elements in the image. Generally, the elements that users do not want to exist can be eliminated by image inpainting. However, this method often leaves obvious inpainting traces on the image, such as random generation in the eliminated area or unclear texture generation, etc., and the inpainting effect is poor. SUMMARY

[0004] The present application provides an image inpainting method, device, computing device cluster, computer storage medium and computer product, which can improve the image inpainting quality.

[0005] In a first aspect, the present application provides an image inpainting method, comprising: displaying a first image; detecting a first operation, the first operation being an operation of selecting a first region on the first image; determining a reference region in the first image; and inpainting the first region in the first image based on the reference region, wherein the attention degree of the reference region is increased and / or decreased during the inpainting of the first region.

[0006] In this way, during the image inpainting process, by increasing and / or decreasing the attention to certain regions, the attention can be focused on specific regions in the image, so that the features in the specific regions can be better utilized for inpainting, the rationality of image inpainting is improved, the random generation such as ghosting is reduced, the inpainting result can better meet the user's expectation, and the image inpainting effect is improved.

[0007] In a possible implementation, determining the reference region in the first image comprises: identifying different regions on the first image on the first image; displaying a first prompt, the first prompt being used to prompt selection of a high-attention region; detecting a second operation, the second operation being an operation of selecting at least one region on the first image; and taking the region selected in the second operation as the reference region, where the attention degree of the region selected in the second operation is increased in the repairing of the first region. In this way, the reference region can be determined in a user-specified manner.

[0008] In a possible implementation, determining the reference region in the first image comprises: identifying different regions on the first image on the first image; displaying a second prompt, the second prompt being used to prompt selection of a low-attention region; detecting a third operation, the third operation being an operation of selecting at least one region on the first image; and taking the region selected in the third operation as the reference region, where the attention degree of the region selected in the third operation is decreased in the repairing of the first region. In this way, the reference region can be determined in a user-specified manner.

[0009] In a possible implementation, determining the reference region in the first image comprises: identifying different regions on the first image on the first image; displaying a first prompt, the first prompt being used to prompt selection of a high-attention region; detecting a second operation, the second operation being an operation of selecting at least one region on the first image; displaying a second prompt, the second prompt being used to prompt selection of a low-attention region; detecting a third operation, the third operation being an operation of selecting at least one region on the first image; and taking the regions selected in the second operation and the third operation as the reference region, where the attention degree of the region selected in the second operation is increased in the repairing of the first region, and the attention degree of the region selected in the third operation is decreased in the repairing of the first region. In this way, the reference region can be determined in a user-specified manner.

[0010] In a possible implementation, determining the reference region in the first image comprises: taking the first region as the reference region, where the attention degree of the first region is decreased in the repairing of the first region. In this way, the reference region can be determined.

[0011] In a possible implementation, determining the reference region in the first image comprises: taking the background and / or the foreground of the first image as the reference region, where the attention degree of the background of the first image is increased in the repairing of the first region, and the attention degree of the foreground of the first image is decreased in the repairing of the first region. In this way, the reference region can be determined.

[0012] In a possible implementation, the repairing the first region in the first image based on the reference region comprises: calling an image inpainting model to inpaint the first region with the reference region as a guide image, the guide image being used to guide the image inpainting model to increase and / or decrease attention to the reference region in the process of inpainting the first region. In this way, by using the image itself information as a visual guide, the image inpainting model can be caused to increase and / or decrease attention to the reference region in the process of inpainting the first region, the rationality of image inpainting is improved, and the generation of ghosting and other random artifacts is reduced.

[0013] In a second aspect, the present application provides an image inpainting apparatus, comprising: a display module and a processing module. The display module is configured to display a first image. The processing module is configured to detect a first operation, the first operation being an operation of selecting a first region on the first image; determine a reference region in the first image; and repair the first region in the first image based on the reference region, wherein an attention degree of the reference region is increased and / or decreased in the process of repairing the first region.

[0014] In a possible implementation, when the processing module determines the reference region in the first image, the processing module is specifically configured to: identify different regions on the first image on the first image; display a first prompt, the first prompt being used to prompt selection of a high-attention region; detect a second operation, the second operation being an operation of selecting at least one region on the first image; and take the region selected in the second operation as the reference region, wherein the attention degree of the region selected in the second operation is increased in the process of repairing the first region.

[0015] In a possible implementation, when the processing module determines the reference region in the first image, the processing module is specifically configured to: identify different regions on the first image on the first image; display a second prompt, the second prompt being used to prompt selection of a low-attention region; detect a third operation, the third operation being an operation of selecting at least one region on the first image; and take the region selected in the third operation as the reference region, wherein the attention degree of the region selected in the third operation is decreased in the process of repairing the first region.

[0016] In a possible implementation, when determining the reference region in the first image, the processing module is specifically configured to: identify different regions on the first image; display a first prompt, the first prompt being used to prompt selection of a high-attention region; detect a second operation, the second operation being an operation of selecting at least one region on the first image; display a second prompt, the second prompt being used to prompt selection of a low-attention region; detect a third operation, the third operation being an operation of selecting at least one region on the first image; and take the selected regions in the second operation and the third operation as the reference region, wherein the selected region in the second operation has an increased attention degree in repairing the first region, and the selected region in the third operation has a decreased attention degree in repairing the first region.

[0017] In a possible implementation, when determining the reference region in the first image, the processing module is specifically configured to: take the first region as the reference region, wherein the first region has a decreased attention degree in repairing the first region.

[0018] In a possible implementation, when determining the reference region in the first image, the processing module is specifically configured to: take a background and / or a foreground of the first image as the reference region, wherein the background of the first image has an increased attention degree in repairing the first region, and the foreground of the first image has a decreased attention degree in repairing the first region.

[0019] In a possible implementation, when repairing the first region in the first image based on the reference region, the processing module is specifically configured to: take the reference region as a guide map, and call an image repairing model to repair the first region, the guide map being used to guide the image repairing model to increase and / or decrease attention to the reference region in repairing the first region.

[0020] In a third aspect, the present application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect. Illustratively, the computing device cluster can include one or more computing devices.

[0022] Fifthly, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0023] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0024] Figure 1 is a schematic flowchart of an image restoration method provided in an embodiment of this application;

[0025] Figure 2 is a schematic diagram of interface changes on a computing device during an image restoration process according to an embodiment of this application;

[0026] Figure 3 is a schematic diagram of a step for determining a reference region in a first image according to an embodiment of this application;

[0027] Figure 4 is a schematic diagram of another step for determining a reference region in a first image according to an embodiment of this application;

[0028] Figure 5 is a schematic diagram of interface changes on a computing device during another image restoration process provided in an embodiment of this application;

[0029] Figure 6 is a schematic diagram of another step in determining a reference region in a first image according to an embodiment of this application;

[0030] Figure 7 is a schematic diagram of interface changes on a computing device during another image restoration process provided in an embodiment of this application;

[0031] [Correction 17.02.2025 based on Rule 91] Figure 8 is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application;

[0032] Figure 9 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0033] Figure 10 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0034] Figure 11 is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0035] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0036] The terms "first" and "second" and the like in the description and claims of this application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. For example, the first response message and the second response message are used for distinguishing between different response messages and not necessarily for describing a particular sequential or chronological order.

[0037] In the embodiments of the present application, the words "exemplary" and "for example" are used to mean serving as an example, instance, or illustration, at 99 2 any point in the manufacturing or processing. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being superior to other embodiments or designs. Rather, use of the words "exemplary" or "for example" is intended to present concepts in a concrete manner. In the embodiments of the present application, the term "include" and its variants are intended to mean that the object include the stated element or method step but not exclude the presence of other elements or method steps. In the embodiments of the present application, the term "comprise" and its variants are intended to mean that the object comprise the stated element or method step but not exclude the presence of other elements or method steps.

[0038] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "plurality" is two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0039] Generally, in the image inpainting process, after the user selects the region to be processed on the image, the repaired image can be directly generated. But the repair effect of this way is poor. In order to improve the repair effect, an image inpainting method is provided in the embodiments of the present application. In the method, after the user selects the region to be processed on the image, the region which needs to be increased and / or reduced in attention in the subsequent repair process can be determined from the image first, and then the determined region is used as a guide to guide the image inpainting model to repair the image. In this way, the model can focus attention on a specific region in the image, so that the features in the specific region can be better utilized for repair, the rationality of image inpainting can be improved, the random generation of ghosting and the like can be reduced, and the image inpainting effect can be improved. The image inpainting method provided in the embodiments of the present application is introduced below.

[0040] For example, FIG. 1 shows a flowchart of an image inpainting method provided in the embodiments of the present application. It can be understood that the method can be executed by any device, equipment, platform, device cluster with computing and processing capability. For ease of description, the computing device is taken as the execution subject for introduction. The client for image inpainting can be configured on the computing device. The client can be an applet, a desktop application, a mobile application, a Web application or a Web-based application, etc. As shown in FIG. 1, the image processing method can include:

[0041] S101, display a first image.

[0042] In this embodiment, the user can open a first image which he wants to modify on the computing device using an APP for image inpainting. After the user opens the first image on the computing device, the computing device can display the first image. For example, as shown in FIG. 2, in (A) of FIG. 2, after the user opens the image 21 in the client on the computing device 200, the computing device 200 can display the image 21.

[0043] S102, detecting a first operation, the first operation being an operation of selecting a first region on the first image.

[0044] In this embodiment, the user can select a first region on the first image displayed on the computing device which he wants to eliminate or optimize. After the user completes the selection operation, the computing device can detect a first operation. The first operation is an operation of selecting a first region on the first image. For example, taking the selection of a region to be eliminated as an example, continuing to refer to (A) of FIG. 2, when the computing device 200 displays the image 21, the computing device 200 can also display a control 22 for selecting a region to be eliminated. Then, the user can select the control 22 and select a region on the image 21 which he wants to eliminate. During the selection of the region which the user wants to eliminate, the computing device 200 can mark the region selected by the user to be eliminated. In (A) of FIG. 2, when the user selects the region to be eliminated, the computing device 200 can display an interface as shown in (B) of FIG. 2. In (B) of FIG. 2, the region occupied by the "book" selected by the user can be marked by a line 23. In addition, after the user selects the control 22, the computing device 200 can also display a selection control "whether to complete the selection of the region to be eliminated". In (B) of FIG. 2, after the user completes the selection of the region to be eliminated, the user can select the control 24 to confirm that he has completed the selection of the region to be eliminated. After the user selects the control 24, the computing device can detect the operation of selecting the region to be eliminated (i.e. the region marked by the line 23) on the image 21. In some embodiments, before S102, the computing device can also display a prompt to prompt the user to select the region to be eliminated.

[0045] S103, determining a reference region in the first image.

[0046] In this embodiment, the reference region can be specified by the user, can be determined by the computing device itself, or can be partially specified by the user and partially determined by the computing device itself, and the specific method can be determined according to actual conditions, which is not limited here. The attention of the reference region can be increased or decreased in the image repairing process. When there are multiple reference regions, the attention of a part of the reference regions can be increased, and the attention of another part of the reference regions can be decreased. For example, the “attention of the reference region is increased” can be understood as increasing the attention to the reference region, and the “attention of the reference region is decreased” can be understood as decreasing the attention to the reference region.

[0047] S104, repairing the first region in the first image based on the reference region, wherein the attention of the reference region is increased and / or decreased in the process of repairing the first region.

[0048] In this embodiment, after the reference region is determined, the first region in the first image can be repaired at least by using the reference region, for example, the first region is eliminated and regenerated, or the display effect of the first region is improved, and the like. In some embodiments, the computing device can use the determined reference region as a guide map and call an image repairing model to repair the first region. The guide map can be used to guide the image repairing model to increase and / or decrease the attention to the reference region in the process of repairing the first region. For example, the image repairing model can be a neural network model based on a stable diffusion (SD) architecture, but is not limited to this. It should be understood that the image repairing model can be configured on the computing device or on a server, and the specific method can be determined according to actual conditions, which is not limited here. When the image repairing model is configured on the server, the computing device can interact with the server to repair the first region in the first image.

[0049] In this way, in the image repairing process, by increasing and / or decreasing the attention to some regions, the attention can be focused on specific regions in the image, so that the features in the specific regions can be better used for repairing, the rationality of image repairing can be improved, the random generation of ghosting and the like can be reduced, the repaired result can better meet the user's expectation, and the image repairing effect is improved.

[0050] Next, the three determination methods of the reference region described in S103 of FIG. 1 (i.e., 1, specified by the user, 2, determined by the computing device itself, and 3, partially specified by the user and partially determined by the computing device itself) will be introduced.

[0051] (1) The reference region is specified by the user

[0052] When the reference region is specified by the user, the attention of the reference region selected by the user can be only increased, only decreased, or partially increased and partially decreased. The three cases are described below.

[0053] a) The attention of the reference region selected by the user is only increased

[0054] For example, as shown in FIG. 3, when the attention of the reference region selected by the user is only increased, determining the reference region in the first image can include the following steps:

[0055] S301, identifying different regions on the first image.

[0056] In this embodiment, the computing device can process the first image by using a neural network model that can perform semantic segmentation on images, to obtain the regions contained in the first image. Then, the computing device can present the regions contained in the first image to the user on the first image, that is, display the regions contained in the first image on the first image. For example, continuing to refer to FIG. 2, after the user selects the control 24 in (B) of FIG. 2, the computing device 200 can display an interface as shown in (C) of FIG. 2. In (C) of FIG. 2, the regions contained in the image 21, that is, a, b, c, d, e, f, and g in the figure, are identified on the image 21. Of course, in order to distinguish different regions, different colors can be used to identify different regions, so as to better present different regions to the user and facilitate the user to select.

[0057] S302, displaying a first prompt, the first prompt being used to prompt to select a high-attention region.

[0058] In this embodiment, after identifying the different regions on the first image, the computing device can display a prompt (that is, a first prompt) to prompt the user to select a high-attention region. The high-attention region is a region that needs to increase the attention in the subsequent process. For example, continuing to refer to FIG. 2, in (C) of FIG. 2, a prompt in the region 25 can be displayed to remind the user to select a high-attention region.

[0059] S303, detecting a second operation, the second operation being an operation of selecting at least one region on the first image.

[0060] In this embodiment, the user can select one or more regions from the regions identified by the computing device on the first image. After the user completes the selection operation, the computing device can detect the second operation. The second operation is an operation of selecting at least one region on the first image. For example, continuing to refer to FIG. 2, in (C) of FIG. 2, the user can select the regions a and e, and select the control 26. After the user selects the control 26, the computing device can detect the operation of the user selecting the regions a and e.

[0061] S304, taking the selected region in the second operation as the reference region, wherein the attention of the selected region in the second operation is increased in the repairing of the first region.

[0062] In this embodiment, after detecting the second operation, the computing device can take the selected region in the second operation as the reference region. Wherein the attention of the selected region in the second operation is increased in the repairing of the first region. Then, the computing device can repair the first region in the first image based on the selected region in the second operation. For example, continuing to refer to FIG. 2, in (C) of FIG. 2, after the user selects the regions a and e, the computing device 200 can repair the region 23 in the image 21 based on the regions a and e, and obtain the image 27 shown in (D) of FIG. 2.

[0063] b) the attention of the selected region in the second operation is only decreased

[0064] For example, as shown in FIG. 4, when the attention of the selected region in the second operation is only decreased, determining the reference region in the first image can include the following steps:

[0065] S401, identifying different regions on the first image on the first image.

[0066] S402, displaying a second prompt, the second prompt is used to prompt to select a low attention region. Wherein the low attention region is a region that needs to be decreased in attention in the future.

[0067] S403, detecting a third operation, the third operation is an operation of selecting at least one region on the first image.

[0068] S404, taking the selected region in the third operation as the reference region, wherein the attention of the selected region in the third operation is decreased in the repairing of the first region.

[0069] In FIG. 4, for S401 to S404, please refer to the description in FIG. 3, which will not be repeated here.

[0070] In addition, to facilitate understanding of the image inpainting process corresponding to FIG. 4, the following continues to take the elimination of a region in an image as an example for introduction. Referring to FIG. 5, in (A) of FIG. 5, the user can select control 52 to select the region that needs to be eliminated in image 51. Then, as shown in (B) of FIG. 5, the user can select region 53 as the region that needs to be eliminated, and after selection, select control 54. Then, as shown in (C) of FIG. 5, the computing device can identify different regions contained in image 51, i.e., a, b, c, d, e, f and g in the figure, on image 51. At the same time, the computing device can display a prompt in region 55 to prompt the user to select the region that needs to be reduced in attention in the image inpainting process. Then, in (C) of FIG. 5, the user can select regions b, c and f, and select control 56. Finally, the computing device can inpaint region 53 based on regions b, c and f in image 51, and obtain image 57 shown in (D) of FIG. 5.

[0071] c) The attention of the reference region selected by the user is partially increased and partially reduced

[0072] For example, as shown in FIG. 6, when the attention of the reference region selected by the user is partially increased and partially reduced, determining the reference region in the first image can include the following steps:

[0073] S601, identifying different regions on the first image on the first image;

[0074] S602, displaying a first prompt, the first prompt being used to prompt selection of a high-attention region;

[0075] S603, detecting a second operation, the second operation being an operation of selecting at least one region on the first image;

[0076] S604, displaying a second prompt, the second prompt being used to prompt selection of a low-attention region;

[0077] S605, detecting a third operation, the third operation being an operation of selecting at least one region on the first image;

[0078] S606, taking the regions selected in the second operation and the third operation as the reference region, wherein the attention of the region selected in the second operation is increased in the process of inpainting the first region, and the attention of the region selected in the third operation is reduced in the process of inpainting the first region.

[0079] In FIG. 6, for S601 to S606, please refer to the relevant description in FIG. 3 and FIG. 4, which will not be repeated here. In addition, in FIG. 6, the execution order of S602 and S603 can also be exchanged with S604 and S605, and the execution flow at this time can be changed to: S601, S604, S605, S602, S603, S606.

[0080] In addition, in order to facilitate the understanding of the image repairing process corresponding to FIG. 6, the following will continue to take the elimination of the region in the image as an example for introduction. Please refer to FIG. 7, in (A) of FIG. 7, the user can select control 72 to select the region he needs to eliminate on image 71. Then, as shown in (B) of FIG. 7, the user can select region 73 as the region to be eliminated, and after selecting, select control 74. Then, as shown in (C) of FIG. 7, the computing device can identify different regions contained in image 71 on image 71, that is, a, b, c, d, e, f and g in the figure. At the same time, the computing device can display the prompt in region 75 to prompt the user to select the region that needs to be paid more attention in the image repairing process. Then, in (C) of FIG. 7, the user can select regions a and e, and select control 76. After that, as shown in (D) of FIG. 7, the computing device can display the prompt in region 77 to prompt the user to select the region that needs to be paid less attention in the image repairing process. Then, in (D) of FIG. 7, the user can select regions b, c and f, and select control 78. Finally, the computing device can repair region 73 based on the selected regions a and e in (C) of FIG. 7, and the selected regions b, c and f in (D) of FIG. 7, and obtain image 79 shown in (E) of FIG. 7.

[0081] (2) The reference region is determined by the computing device itself

[0082] When the reference region is determined by the computing device itself, the computing device can take the first region as the reference region. At this time, the attention degree of the first region is reduced in the process of repairing the first region.

[0083] In addition, the computing device can also take the background and / or the foreground of the first image as the reference region. In this case, the attention of the background of the first image is increased during the repairing of the first region, and the attention of the foreground of the first image is decreased during the repairing of the first region. For example, the foreground of the first image can be understood as the main object in the first image. The background of the first image can be understood as the part (such as the environment, the scene, etc.) in the first image other than the foreground. For example, in an image containing a person and a landscape, the person can be considered as the foreground, and the landscape can be considered as the background. In some embodiments, for the foreground and the background of the first image, the first image can be processed by a neural network model, but is not limited to this. For example, the foreground and the background of the first image can be obtained by performing pixel-level classification on the first image through a semantic segmentation network (such as a U-Net network, etc.).

[0084] (3) Part of the reference region is specified by the user, and the other part is determined by the computing device

[0085] When part of the reference region is specified by the user, and the other part is determined by the computing device, the region specified by the user can be determined with reference to the related description of the foregoing “the reference region is specified by the user”, and the region determined by the computing device can be determined with reference to the related description of the foregoing “the reference region is determined by the computing device”, which will not be described here.

[0086] The above is a related introduction to the image repairing method provided by the embodiments of the present application. In the above image repairing method, the image can be repaired by using an image repairing model with a self-attention mechanism (such as the neural network model based on the SD architecture described above). Generally, the calculation method of the attention used by such a model can be as follows:

[0087] wherein Q is a query vector matrix, K is a key vector matrix, and V is a value vector matrix, is a scaling factor. Q, K and V can be obtained by performing convolution operation on the feature map obtained from the first image, but are not limited to this.

[0088] In the embodiments of the present application, in order to enable the image repairing model to focus attention on a specific region in the image, the “Formula 1” is improved. The calculation method of the improved attention is as follows:

[0089] wherein max(QK T ) is the maximum value of the elements in the matrix obtained by matrix cross multiplication of Q and K T , and min(QK T ) is the minimum value of the elements in the matrix obtained by matrix cross multiplication of Q and K TThe minimum value of the elements in the matrix obtained by matrix cross multiplication, λ pos and λ neg are coefficients, Mask pos is a mask matrix obtained based on a region that needs to be increased in attention (hereinafter referred to as a "high attention region"), Mask neg is a mask matrix obtained based on a region that needs to be reduced in attention (hereinafter referred to as a "low attention region"). It should be understood that, in the calculation process, λ pos max(QK T )Mask pos and λ neg min(QK T )Mask neg may be selected alternatively or simultaneously, which can be determined according to actual conditions, and is not limited here.

[0090] In this embodiment, after the high attention region is determined, the pixel values in the other regions of the first image except the high attention region can be set to 0, and the pixel values in the high attention region can be set to 1, so that Mask pos is obtained. After the low attention region is determined, the pixel values in the other regions of the first image except the low attention region can be set to 1, and the pixel values in the low attention region can be set to 0, so that Mask neg is obtained.

[0091] In addition, in order to make the improved attention value as much as possible within the previous range, so as to avoid negative impact on the performance of the model. In this embodiment, the calculation method of attention can be further improved on the basis of "Formula 2". At this time, the improved calculation method of attention is:

[0092] It should be understood that, in the calculation process, λ pos W pos Mask pos and λ neg W neg Mask neg may be selected alternatively or simultaneously, which can be determined according to actual conditions, and is not limited here.

[0093] In this way, through "Formula 2" or "Formula 3", the image restoration model can refer to more features of specific regions during the image restoration process, so as to improve the rationality of image restoration and reduce the generation of ghosting and the like.

[0094] It can be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, the various embodiments described above or the technical features involved in the embodiments can be combined according to the actual situation, and the combined scheme is still within the protection scope of the present application.

[0095] Based on the method in the above embodiments, the embodiments of the present application further provide an image repairing device.

[0096] For example, FIG. 8 shows a structural schematic diagram of an image repairing device provided by the embodiments of the present application. As shown in FIG. 8, the image repairing device 800 can include a display module 810 and a processing module 820. The display module 810 is configured to display a first image. The processing module 820 is configured to detect a first operation, the first operation being an operation of selecting a first region on the first image; determine a reference region in the first image; and repair the first region in the first image based on the reference region, wherein the attention of the reference region is increased and / or decreased in the process of repairing the first region.

[0097] In some embodiments, when the processing module 820 determines the reference region in the first image, it is specifically configured to: identify different regions on the first image on the first image; display a first prompt, the first prompt being used to prompt to select a high-attention region; detect a second operation, the second operation being an operation of selecting at least one region on the first image; and take the region selected in the second operation as the reference region, wherein the attention of the region selected in the second operation is increased in the process of repairing the first region.

[0098] In some embodiments, when the processing module 820 determines the reference region in the first image, it is specifically configured to: identify different regions on the first image on the first image; display a second prompt, the second prompt being used to prompt to select a low-attention region; detect a third operation, the third operation being an operation of selecting at least one region on the first image; and take the region selected in the third operation as the reference region, wherein the attention of the region selected in the third operation is decreased in the process of repairing the first region.

[0099] In some embodiments, when the processing module 820 determines the reference region in the first image, specifically for: identifying different regions on the first image on the first image; displaying a first prompt, the first prompt is used to prompt to select a high attention region; detecting a second operation, the second operation is an operation of selecting at least one region on the first image; displaying a second prompt, the second prompt is used to prompt to select a low attention region; detecting a third operation, the third operation is an operation of selecting at least one region on the first image; the selected region in the second operation and the third operation is taken as the reference region, wherein the attention degree of the selected region in the second operation is increased in the process of repairing the first region, and the attention degree of the selected region in the third operation is reduced in the process of repairing the first region.

[0100] In some embodiments, when the processing module 820 determines the reference region in the first image, specifically for: taking the first region as the reference region, wherein the attention degree of the first region is reduced in the process of repairing the first region.

[0101] In some embodiments, when the processing module 820 determines the reference region in the first image, specifically for: taking the background and / or foreground of the first image as the reference region, wherein the attention degree of the background of the first image is increased in the process of repairing the first region, and the attention degree of the foreground of the first image is reduced in the process of repairing the first region.

[0102] In some embodiments, when the processing module 820 repairs the first region in the first image based on the reference region, specifically for: taking the reference region as a guide map, calling an image repairing model to repair the first region, and the guide map is used to guide the image repairing model to increase and / or reduce the attention to the reference region in the process of repairing the first region.

[0103] In some embodiments, the display module 810 and the processing module 820 shown in FIG. 8 can be implemented by software or can be implemented by hardware. For example, the implementation of the display module 810 is introduced as follows. Similarly, the implementation of the processing module 820 can refer to the implementation of the display module 810.

[0104] As an example of a software functional unit, the display module 810 can include code running on a compute instance. The compute instance can include at least one of a physical host (computing device), a virtual machine, a container. Further, the compute instance can be one or more. For example, the display module 810 can include code running on multiple hosts / virtual machines / containers. It is noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs, each of which includes one data center or multiple data centers in close geographical proximity. Typically, a region can include multiple AZs.

[0105] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Typically, a VPC is set up within a region, and communication between two VPCs in the same region, or between VPCs in different regions, requires a communication gateway in each VPC to enable interconnection between the VPCs.

[0106] As an example of a hardware functional unit, the display module 810 can include at least one computing device, such as a server, etc. Alternatively, the display module 810 can also be a device implemented using an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), etc. The PLD can be implemented as a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0107] The plurality of computing devices included in the display module 810 can be distributed in the same region or in different regions. The plurality of computing devices included in the display module 810 can be distributed in the same AZ or in different AZs. Similarly, the plurality of computing devices included in the display module 810 can be distributed in the same VPC or in multiple VPCs. The plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0108] It should be noted that, in other embodiments, the display module 810 can be configured to perform any of the steps of the image inpainting method described in the above embodiments, and the processing module 820 can be configured to perform any of the steps of the image inpainting method described in the above embodiments. In addition, the display module 810 can be combined with the processing module 820 to be responsible for performing any of the steps of the image inpainting method described in the above embodiments. In addition, the steps implemented by the display module 810 and the processing module 820 can also be specified as needed, and the display module 810 and the processing module 820 respectively implement different steps of the image inpainting method described in the above embodiments to realize the entire function of the image inpainting apparatus 800 shown in FIG. 8.

[0109] The present application also provides a computing device 900. As shown in FIG. 9, the computing device 900 includes a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other through the bus 902. The computing device 900 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 900.

[0110] The bus 902 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 9, but it does not mean that there is only one bus or only one type of bus. The bus 904 can include a path for transmitting information between various components (e.g., the memory 906, the processor 904, the communication interface 908) of the computing device 900.

[0111] The processor 904 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), among other processors.

[0112] The memory 906 can include volatile memory, such as random access memory (RAM), and non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid-state drive (SSD).

[0113] The memory 906 stores executable program code that the processor 904 executes to respectively implement the functions of the display module 810 and the processing module 820 shown in FIG. 8, thereby implementing the image inpainting method described in the above embodiments. That is, the memory 906 stores instructions for executing the image inpainting method described in the above embodiments.

[0114] Alternatively, the memory 906 stores executable program code that the processor 904 executes to respectively implement the functions of the image inpainting apparatus 800 shown in FIG. 8, thereby implementing the image inpainting method described in the above embodiments. That is, the memory 906 stores instructions for executing the image inpainting method described in the above embodiments.

[0115] The communication interface 908 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to enable communication between the computing device 900 and other devices or communication networks.

[0116] The embodiments of the present disclosure also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smartphone.

[0117] As shown in FIG. 10, the computing device cluster includes at least one computing device 900. The memory 906 in one or more computing devices 900 in the computing device cluster can store the same instructions for performing the image inpainting method described in the foregoing embodiments.

[0118] In some possible implementation manners, the memory 906 in one or more computing devices 900 in the computing device cluster can also respectively store partial instructions for performing the image inpainting method described in the foregoing embodiments. In other words, the combination of one or more computing devices 900 can collectively perform the instructions for performing the image inpainting method described in the foregoing embodiments.

[0119] It should be noted that the memory 906 in different computing devices 900 in the computing device cluster can store different instructions, respectively used for performing part of the functions of the image inpainting apparatus 800 shown in FIG. 8. That is, the instructions stored in the memory 906 in different computing devices 900 can implement the functions of one or more of the display module 810 and the processing module 820.

[0120] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network, a local area network, or the like. FIG. 11 shows a possible implementation manner. As shown in FIG. 11, two computing devices 900A and 900B are connected through a network. Specifically, the computing devices are connected to the network through the communication interfaces in the computing devices. In this kind of possible implementation manner, the memory 906 in the computing device 900A stores instructions for performing the functions of the display module 810. Meanwhile, the memory 906 in the computing device 900B stores instructions for performing the functions of the receiving module 1102 and the processing module 820.

[0121] It should be understood that the functions of the computing device 900A shown in FIG. 11 can also be completed by multiple computing devices 900. Similarly, the functions of the computing device 900B can also be completed by multiple computing devices 900.

[0122] The embodiments of the present application further provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manners of the computing device clusters described with reference to FIG. 10 and FIG. 11. The difference is that the memory 906 in one or more computing devices 900 in the computing device cluster can store the same instructions for performing the method in the foregoing embodiments.

[0123] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster can also respectively store partial instructions for performing the foregoing image inpainting method. In other words, the combination of one or more computing devices 900 can collectively execute the instructions for performing the foregoing image inpainting method.

[0124] It should be understood that each step of the foregoing method embodiments can be completed by a logic circuit in the form of hardware in the processor or instructions in the form of software.

[0125] Based on the method in the foregoing embodiments, an embodiment of the present application provides a computer-readable storage medium including computer program instructions, which, when executed by a computing device cluster including at least one computing device, cause the computing device cluster to perform the method in the foregoing embodiments. Illustratively, the computer-readable storage medium can be any available medium or a data storage device such as a data center including one or more available media that a computing device is capable of storing. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), or the like.

[0126] Based on the method in the foregoing embodiments, an embodiment of the present application provides a computer program product including instructions, which, when executed by a computing device cluster including at least one computing device, cause the computing device cluster to perform the method in the foregoing embodiments.

[0127] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor can be a microprocessor, or any conventional processor.

[0128] The method steps in the embodiments of the present application can be implemented by hardware, or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, so that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0129] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as a coaxial cable, an optical fiber, a digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0130] It can be understood that the various numerical numbers involved in the embodiments of the present application are only used for differentiation for convenience of description, and do not limit the scope of the embodiments of the present application.

[0131] Finally, it should be noted that: the above examples are used to illustrate the technical solutions of the present application, but not limited to them; although the present application is described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An image inpainting method characterized by, The method comprises: displaying a first image; detecting a first operation, the first operation being an operation of selecting a first region on the first image; determining a reference region in the first image; repairing the first region in the first image based on the reference region, wherein the attention of the reference region is increased and / or decreased during the repairing of the first region.

2. The method of claim 1, wherein, Determining a reference region in the first image comprises: identifying different regions on the first image on the first image; displaying a first prompt for prompting selection of a high-attention region; detecting a second operation, the second operation being an operation of selecting at least one region on the first image; selecting the region selected in the second operation as the reference region, wherein the attention of the region selected in the second operation is increased during the repairing of the first region.

3. The method of claim 1, wherein, Determining a reference region in the first image comprises: identifying different regions on the first image on the first image; displaying a second prompt for prompting selection of a low-attention region; detecting a third operation, the third operation being an operation of selecting at least one region on the first image; selecting the region selected in the third operation as the reference region, wherein the attention of the region selected in the third operation is decreased during the repairing of the first region.

4. The method of claim 1, wherein, Determining a reference region in the first image comprises: identifying different regions on the first image on the first image; displaying a first prompt for prompting selection of a high-attention region; detecting a second operation, the second operation being an operation of selecting at least one region on the first image; displaying a second prompt for prompting selection of a low-attention region; detecting a third operation, the third operation being an operation of selecting at least one region on the first image; selecting the regions selected in the second operation and the third operation as the reference region, wherein the attention of the region selected in the second operation is increased during the repairing of the first region, and the attention of the region selected in the third operation is decreased during the repairing of the first region.

5. The method according to any of claims 1 to 4, characterized in that, Determining a reference region in the first image comprises: selecting the first region as the reference region, wherein the attention of the first region is decreased during the repairing of the first region.

6. The method according to any one of claims 1 to 5, characterized in that, Determining a reference region in the first image comprises: selecting the background and / or the foreground of the first image as the reference region, wherein the attention of the background of the first image is increased during the repairing of the first region, and the attention of the foreground of the first image is decreased during the repairing of the first region.

7. The method according to any of claims 1 to 6, characterized in that Repairing the first region in the first image based on the reference region comprises: using the reference region as a guide map to call an image repairing model to repair the first region, the guide map being used to guide the image repairing model to increase and / or decrease the attention to the reference region during the repairing of the first region.

8. An image inpainting apparatus characterized by comprising: comprises: a display module configured to display a first image; The processing module is configured to detect a first operation, the first operation being an operation of selecting a first region on the first image; The processing module is further configured to determine a reference region in the first image; The processing module is further configured to repair the first region in the first image based on the reference region, wherein a degree of attention of the reference region is increased and / or decreased during the repairing of the first region.

9. The apparatus of claim 8, wherein, When determining the reference region in the first image, the processing module is specifically configured to: identify different regions on the first image on the first image; display a first prompt, the first prompt being used to prompt selection of a high-attention region; detect a second operation, the second operation being an operation of selecting at least one region on the first image; select the region selected in the second operation as the reference region, wherein a degree of attention of the region selected in the second operation is increased during the repairing of the first region.

10. The apparatus of claim 8, wherein, When determining the reference region in the first image, the processing module is specifically configured to: identify different regions on the first image on the first image; display a second prompt, the second prompt being used to prompt selection of a low-attention region; detect a third operation, the third operation being an operation of selecting at least one region on the first image; select the region selected in the third operation as the reference region, wherein a degree of attention of the region selected in the third operation is decreased during the repairing of the first region.

11. The apparatus of claim 8, wherein, When determining the reference region in the first image, the processing module is specifically configured to: identify different regions on the first image on the first image; display a first prompt, the first prompt being used to prompt selection of a high-attention region; detect a second operation, the second operation being an operation of selecting at least one region on the first image; display a second prompt, the second prompt being used to prompt selection of a low-attention region; detect a third operation, the third operation being an operation of selecting at least one region on the first image; select the regions selected in the second operation and the third operation as the reference region, wherein a degree of attention of the region selected in the second operation is increased during the repairing of the first region, and a degree of attention of the region selected in the third operation is decreased during the repairing of the first region.

12. The apparatus of any of claims 8-11, wherein, When determining the reference region in the first image, the processing module is specifically configured to: select the first region as the reference region, wherein a degree of attention of the first region is decreased during the repairing of the first region.

13. The apparatus of any of claims 8-12, wherein, When determining the reference region in the first image, the processing module is specifically configured to: select a background and / or a foreground of the first image as the reference region, wherein a degree of attention of the background of the first image is increased during the repairing of the first region, and a degree of attention of the foreground of the first image is decreased during the repairing of the first region.

14. The apparatus of any of claims 8-13, wherein, When repairing the first region in the first image based on the reference region, the processing module is specifically configured to: With the reference region as a guide map, an image inpainting model is called to inpaint the first region, and the guide map is used to guide the image inpainting model to increase and / or decrease attention to the reference region in the process of inpainting the first region.

15. A cluster of computing devices, characterized in that, comprise at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, comprise computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any one of claims 1-7, wherein the cluster of computing devices comprises at least one computing device.

17. A computer program product comprising instructions, characterized in that, comprise computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any one of claims 1-7, wherein the cluster of computing devices comprises at least one computing device.

Images