Image inpainting method, device and electronic equipment

Abstract

The disclosure provides a repairing method, device and electronic equipment of an image, a specific embodiment of the method includes: acquiring a first image; the first image is an image obtained after processing a target object in an original image; determining a first region to be repaired in the first image; the first region is at least a partial region after the processing; acquiring a target semantic map corresponding to the first image; based on the target semantic map, repairing the first region to obtain a second image after the repairing. This embodiment can repair the image to be repaired based on semantic information, reduce residual traces of the original image in the image obtained after the repairing, make the boundary of different semantic regions clear, the texture more abundant, and the image more real.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image restoration method, apparatus, and electronic device. Background Technology

[0002] Artificial intelligence (AI) technology is increasingly being used in the field of image processing. It is commonly used to repair damaged original images or remove occlusions from original images to generate new ones. Currently, however, the processed areas in the resulting new images often retain traces of the original image, resulting in poor quality. Therefore, a solution is needed to repair modified areas within images. Summary of the Invention

[0003] This disclosure provides an image restoration method, apparatus, and electronic device.

[0004] According to a first aspect, an image restoration method is provided, the method comprising:

[0005] Obtain the first image; the first image is the image obtained after processing the target object in the original image;

[0006] A first region to be repaired is identified in the first image; the first region is at least a portion of the region that has undergone the aforementioned processing.

[0007] Obtain the target semantic map corresponding to the first image;

[0008] Based on the target semantic map, the first region is repaired to obtain the repaired second image.

[0009] According to a second aspect, an image restoration apparatus is provided, the apparatus comprising:

[0010] The first acquisition module is used to acquire a first image; the first image is an image obtained by processing the target object in the original image;

[0011] A determining module is used to determine a first region to be repaired in the first image; the first region is at least a portion of the region that has undergone the processing.

[0012] The second acquisition module is used to acquire the target semantic map corresponding to the first image;

[0013] The repair module is used to repair the first region based on the target semantic map to obtain a repaired second image.

[0014] According to a third aspect, a computer-readable storage medium is provided, the storage medium storing a computer program that, when executed by a processor, implements the method described in any one of the first aspects above.

[0015] According to a fourth aspect, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of the first aspects.

[0016] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0017] This disclosure provides an image restoration method and apparatus. By using the semantic map corresponding to the image to be restored, at least some modified regions in the image are restored, resulting in an image with better display quality. Because the solution provided in this embodiment considers the semantic map of the image to be restored when restoring modified regions, and the semantic map contains richer semantic information, restoration can be performed based on this richer semantic information. This reduces residual traces of the original image in the restored image, making the boundaries of different semantic regions clearer, the texture richer, and the image more realistic.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating an image restoration scenario according to an exemplary embodiment of the present disclosure;

[0021] Figure 2 This is a flowchart illustrating an image restoration method according to an exemplary embodiment of the present disclosure;

[0022] Figure 3 This is a flowchart illustrating another image restoration method according to an exemplary embodiment of the present disclosure;

[0023] Figure 4 This is a block diagram of an image restoration apparatus according to an exemplary embodiment of the present disclosure;

[0024] Figure 5 This is a schematic block diagram of an electronic device provided in some embodiments of this disclosure;

[0025] Figure 6 This is a schematic block diagram of another electronic device provided in some embodiments of this disclosure;

[0026] Figure 7 This is a schematic diagram of a storage medium provided in some embodiments of this disclosure. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0028] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0029] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0031] Artificial intelligence (AI) technology is increasingly being applied in the field of image processing. It is commonly used to repair damaged original images or remove occlusions from them to generate new images. For example, it can be used to shorten long hair in a portrait image or remove trees or buildings from a landscape image. Currently, however, the processed areas in the resulting new images often retain traces of the original image, resulting in poor quality. For instance, when shortening long hair in an image, the exposed occluded areas often contain residual hair strands, unclear clothing boundaries, and color anomalies. Therefore, a solution is needed to repair modified areas in an image.

[0032] This disclosure provides an image restoration scheme that repairs at least some modified regions of the image by using the semantic map corresponding to the image to be restored, thereby obtaining an image with better display quality. Because the scheme provided in this embodiment considers the semantic map of the image to be restored when repairing the modified regions, and the semantic map of the image to be restored contains richer semantic information, it can restore the image based on richer semantic information. This reduces residual traces of the original image in the restored image, makes the boundaries of different semantic regions clearer, the texture richer, and the image more realistic.

[0033] See Figure 1 This is a schematic diagram illustrating an image restoration scenario according to an exemplary embodiment. References below... Figure 1 The present invention will be illustrated with a complete and specific application example. This application example describes a specific process of image restoration.

[0034] like Figure 1 As shown, the original image A is an image that needs to have occlusions removed or contains missing regions. After modifying the original image A (e.g., removing occlusions or filling in missing regions), image B is obtained. Since the modified region a in image B suffers from significant loss of texture detail and unclear edges, further restoration of image B is needed for region a. Specifically, semantic segmentation can be performed on image B to obtain the corresponding semantic map C, and information about region a can be obtained. Then, a masking operation is performed on image B based on the information of region a, setting the pixel values ​​of region a in image B to 0, resulting in image D. Image D and semantic map C are then input into a pre-trained image restoration network, which performs restoration on region a.

[0035] It should be noted that the semantic map C used here is the semantic map corresponding to image B, which is fundamentally different from the semantic map corresponding to the original image A. Because the information about the region to be modified is severely lacking in the original image A, the semantic map corresponding to the original image A lacks semantic information about the region to be modified.

[0036] In an image inpainting network, image D can first be downsampled using a downsampling module to extract its features. For example, the downsampling module can consist of multiple convolutional layers, which sequentially convolve image D. Simultaneously, semantic correction can be applied to the convolutional results after each convolution, based on the semantic map C. Specifically, two parameters α and β (both vectors) can be learned from the semantic map C through two different convolutional layers, and these parameters α and β can be used to semantically correct the feature map obtained from the convolution. For example, a SPADE spatial adaptation approach can be used for semantic correction based on the semantic map C. After multiple convolutional layers, the feature map to be repaired is obtained, and then the image inpainting module processes this feature map.

[0037] Specifically, based on semantic graph C, the unknown region corresponding to region a in the feature map to be repaired can be divided into multiple unknown sub-regions according to semantics, so that each unknown sub-region corresponds to only one semantic. The known regions in the feature map to be repaired, excluding the unknown regions, are also identified and divided into multiple known sub-regions, each corresponding to only one semantic. For any unknown sub-region, the initial features corresponding to that unknown sub-region in the feature map to be repaired can be determined, and the features of that unknown sub-region can be reconstructed using known sub-regions with the same semantics as that unknown sub-region, resulting in reconstructed features (see [link to documentation] for details). Figure 3 Example). By performing feature fusion between the initial features and the reconstructed features through stacking processing, a repaired feature map can be obtained.

[0038] The repaired feature map is then processed by upsampling, thereby converting it into the repaired target image E. For example, the upsampling module can consist of multiple deconvolution layers, which sequentially perform deconvolution processing on the repaired feature map. Similarly, semantic correction can be performed on the deconvolution result after each deconvolution process, based on the semantic map C.

[0039] It should be noted that during the training phase of the image inpainting network, complete real images can be selected as sample images, and the corresponding semantic maps can be obtained. A portion of the sample image (e.g., regions rich in semantic information) is then masked. The semantic map corresponding to the sample image and the masked image are input into the image inpainting network to be trained, and the predicted image output by the network is obtained. The prediction loss is calculated based on the predicted image and the sample image, and the network parameters of the image inpainting network are adjusted according to the prediction loss, thereby training the image inpainting network.

[0040] The present disclosure will now be described in detail with reference to specific embodiments.

[0041] Figure 2 This is a flowchart illustrating an image restoration method according to an exemplary embodiment. The implementer of this method can be any device, platform, server, or device cluster with computing and processing capabilities. The method includes the following steps:

[0042] like Figure 2 As shown, in step 201, a first image is acquired, and a first region to be repaired in the first image is determined.

[0043] In this embodiment, the first image is the image obtained after processing the target object in the original image, and the first region is at least a portion of the processed region. In one scenario, the first image can be the image obtained by removing occlusions from the original image (the target object being the occlusion), and the first region can be at least a portion of the region corresponding to the removed occlusion. For example, to shorten long hair in a person's image, it is necessary to remove some of the hair ends from the image. The image obtained after removing the hair ends is the first image, and the area of ​​the removed hair ends is the first region. Since the region to be repaired in this scenario often contains multiple semantic meanings, and the region to be repaired occupies a large proportion of the image with limited known information available, the repair effect achieved by the solution provided in this embodiment is more significant.

[0044] In another scenario, the first image can also be obtained by repairing and filling in damaged or missing areas of the original image. The first region can be at least a portion of the damaged or missing information area (the target object is the damaged or missing information part). For example, scanning an old photograph with severely damaged areas yields an original image. Repairing the area corresponding to the damaged part in the original image yields the first image, where the repaired area is the first region. It is understood that this solution can also be applied in other scenarios, and this embodiment does not limit the specific application scenario.

[0045] In step 202, the target semantic map corresponding to the first image is obtained, and in step 203, the first region is repaired based on the target semantic map to obtain the repaired second image.

[0046] In this embodiment, the first image can be semantically segmented to obtain a target semantic map corresponding to the first image. Based on the target semantic map, the features corresponding to the first region in the first image can be repaired to obtain new features corresponding to the repaired first region. Then, the repaired second image can be generated based on the new features corresponding to the first region.

[0047] It is important to note that the semantic map used here is the semantic map corresponding to the modified first image, not the semantic map of the unmodified original image. This is because the original image lacks a significant amount of semantic information in the region to be modified, while the modified first image contains much richer semantic information in the region to be repaired.

[0048] In one implementation, features corresponding to the first region in the first image can be repaired based on the target semantic map and the features corresponding to the second region (the region outside the first region in the first image). For example, repair parameters are obtained based on the target semantic map and the features corresponding to the second region, and the repair parameters are used to repair the features corresponding to the first region (such as adding or multiplying the repair parameters with the features corresponding to the first region, or performing preset operations, etc.).

[0049] In another implementation, a first feature map corresponding to the first image can be obtained. Based on the target semantic map, the features corresponding to the first region in the first feature map are used to regenerate the features of the second region, resulting in a second feature map. The second image is then obtained based on the second feature map. For example, for a first region corresponding to a semantic meaning, the features corresponding to the first feature map of the nearest semantically identical second region within a preset range can be used to regenerate the features of that first region.

[0050] Optionally, a first cell corresponding to the first region can be determined, and based on the target semantic map, at least one second cell with the same semantic meaning as the first cell (the second cell corresponds to the second region) can be determined. Then, the features corresponding to the first cell are regenerated based on the features of the second cell. Since this implementation further subdivides the first region to be repaired into first cells, and uses the features of the second cell with the same semantic meaning as the first cell to regenerate the features corresponding to the first cell, the quality of the repaired image is higher, and the semantic boundaries are clearer and more natural.

[0051] This disclosure provides an image restoration method that repairs at least some modified regions of the image by using the semantic map corresponding to the image to be restored, thereby obtaining an image with better display quality. Because the solution provided in this embodiment considers the semantic map of the image to be restored when repairing the modified regions, and the semantic map of the image to be restored contains richer semantic information, it can restore the image based on richer semantic information. This reduces residual traces of the original image in the restored image, makes the boundaries of different semantic regions clearer, the texture richer, and the image more realistic.

[0052] It should be noted that although various methods exist for image restoration in the prior art, the quality of the restored images is generally poor. The restored images often retain traces of the original image, and the boundaries between different semantic regions are blurred and unnatural. Those skilled in the art have failed to recognize this problem because the influence of the semantic information of the restored image on the restoration effect was not considered during the restoration process. There may be various reasons for poor image restoration results, and those skilled in the art would find it difficult to conceive of the above-mentioned reasons without effort. The technical solution of this application considers the influence of the semantic information of the restored image on the restoration effect; therefore, by identifying the problem, it solves the aforementioned technical issues.

[0053] The following two complete application examples illustrate the solution disclosed herein.

[0054] One application scenario is to shorten the long hair of the person in the original image 1 by removing the ends of the long hair, resulting in image 2. However, since the areas in image 2 where the long hair has been removed have significant texture loss, further restoration of image 2 is necessary.

[0055] Specifically, firstly, image 2 can be acquired as the first image, and the modified region f in image 2 can be identified as the first region. Region f can be at least a portion of the area corresponding to the removed hair ends. Region g in image 2, excluding region f (e.g., region g includes clothing, skin, and background around the hair), can be identified as the second region. Then, the semantic map C corresponding to image 2 is acquired as the target semantic map. Based on semantic map C, regions f and g are semantically partitioned, identifying multiple sub-regions f′ in region f corresponding to different semantics, and multiple sub-regions g′ in region g corresponding to different semantics.

[0056] Next, sub-regions g′ with the same semantic meaning are repaired using sub-regions f′. For example, sub-region g1′ corresponding to skin semantics is used to repair sub-region f1′ corresponding to skin semantics; sub-region g2′ corresponding to clothing semantics is used to repair sub-region f2′ corresponding to clothing semantics; sub-region g3′ corresponding to clothing semantics is used to repair sub-region f3′ corresponding to clothing semantics, and so on. Finally, the repaired image 3 can be obtained.

[0057] Another application scenario is to scan a partially damaged old photograph to obtain the original image 4, and then fill in the missing areas in the original image 4 to obtain image 5. However, since the filled missing areas in image 5 have a lot of texture loss and detail, further restoration is required for image 5.

[0058] Specifically, first, image 5 is acquired as the first image, and at least a portion of the region w corresponding to the missing area to be filled in image 5 is identified as the first region. The region v outside region w in image 5 is identified as the second region. Then, the semantic map D corresponding to image 5 is acquired as the target semantic map. Based on semantic map D, regions w and v are semantically segmented, identifying multiple sub-regions w′ in region w corresponding to different semantics, and multiple sub-regions v′ in region v corresponding to different semantics. Next, sub-regions v′ are used to repair sub-regions w′ with the same semantics. Finally, the repaired image 6 is obtained.

[0059] Figure 3 This is a flowchart illustrating another image restoration method according to an exemplary embodiment, which describes the process of restoring a first region, including the following steps:

[0060] like Figure 3 As shown, in step 301, the first feature map corresponding to the first image is obtained.

[0061] In this embodiment, features of the first image can be extracted first to obtain a first feature map. For example, the first image can be directly input into a downsampling module (which may consist of multiple convolutional layers) to obtain the first feature map output by the downsampling module. Alternatively, the first image can be masked using a first region, and the masked image can be processed. Specifically, masking the first image using the first region can involve setting the pixel values ​​of the first region in the first image to 0. Then, the masked image is input into the downsampling module. Optionally, the masked image can be convolved using multiple convolutional layers. After processing by the convolutional layers, the result of the convolution processing can be semantically corrected based on the target semantic map corresponding to the first image to obtain the first feature map.

[0062] For example, semantic correction can be performed using the target semantic map after each convolutional layer. Alternatively, semantic correction can be performed after multiple convolutional layers. It is understood that this embodiment does not limit the specific number of semantic corrections. After multiple convolutional layers, a first feature map corresponding to the first image can be obtained. Because this embodiment uses semantic information to correct the extracted features during feature extraction from the first image, it uses semantics to guide the extraction and generation of subsequent features, resulting in clearer boundaries and richer textures in the repaired image across different semantic regions.

[0063] In step 302, multiple first cells corresponding to the first region and multiple second cells corresponding to the second region are determined. And in step 303, each first cell corresponding to a first feature in the first feature map and each second cell corresponding to a second feature in the first feature map are obtained.

[0064] In this embodiment, each feature point in the first feature map corresponds to a pixel in the first image. Furthermore, if downsampling is performed, the number of feature points in the first feature map is less than the number of pixels in the first image. Therefore, each feature point can find a corresponding pixel in the first image. Semantic labels can be added to each pixel in the first image beforehand based on the target semantic map, and region markers (indicating whether the pixel belongs to a first region or a second region) can be added to each pixel. Therefore, after obtaining the first feature map, each feature point in the first feature map also has the same semantic label and region marker as its corresponding pixel.

[0065] Then, the first feature map can be evenly divided into multiple cells, which can be squares, rectangles, etc., each cell having the same size and containing the same number of feature points. For example, each cell can include m×n feature points. Multiple first cells corresponding to the first region and multiple second cells corresponding to the second region can be determined based on the region labels corresponding to the feature points. For example, for a cell, if it contains feature points corresponding to the first region, then that cell can be determined as a first cell. If the cell does not contain feature points corresponding to the first region (i.e., all included feature points correspond to the second region), then that cell can be determined as a second cell.

[0066] Additionally, the semantics of each cell can be determined based on the semantic labels of the feature points included in each cell. For example, if all the feature points in a cell have the same semantic labels, then the semantics indicated by that semantic label is the semantics corresponding to that cell. If the feature points in a cell have different semantic labels, then the semantics indicated by the most frequent semantic label can be taken as the semantics corresponding to that cell.

[0067] Next, we can obtain the first features (e.g., the feature values ​​of feature points in the first cell) corresponding to each first cell in the first feature map and the second features corresponding to each second cell in the first feature map.

[0068] In step 304, based on the first features corresponding to each first cell and the second features corresponding to each second cell, the features corresponding to each first cell are regenerated to obtain the second feature map.

[0069] Specifically, based on the semantics corresponding to each first cell and each second cell, at least one semantically identical second cell can be determined for each first cell. The features corresponding to any first cell can be regenerated based on the second features of the second cell corresponding to any first cell in the first feature map.

[0070] For example, the first feature map includes cells A1m, A2m, A3n..., B1m, B2m, B3n, B4n, B5m, B6n..., where A represents the first cell, B represents the second cell, and m and n represent two different semantics. Therefore, the second cells with the same semantics as cell A1m include B1m, B2m, and B5m, and the features corresponding to cell A1m can be regenerated using cells B1m, B2m, and B5m. Similarly, the second cells with the same semantics as cell A2m also include B1m, B2m, and B5m, and the features corresponding to cell A2m can be regenerated using cells B1m, B2m, and B5m. Likewise, the second cells with the same semantics as cell A3n include B3n, B4n, and B6n, and the features corresponding to cell A3n can be regenerated using cells B3n, B4n, and B6n.

[0071] Specifically, for any first cell, the features corresponding to that first cell can be regenerated as follows: calculate the similarity between the first cell and each second cell with the same semantic meaning; determine the weight of the second feature corresponding to each second cell based on the similarity; calculate the weighted sum of the second features based on the weights; and regenerate the features corresponding to the first cell using the weighted sum. Optionally, the similarity between the first cell and the second cell can be calculated using an inner product method. It is understood that any method known in the art or that may emerge in the future for calculating image similarity can be applied to this embodiment, and this embodiment does not limit the specific method for calculating image similarity.

[0072] For example, cell A1m has similarities S1, S2, and S3 with semantically similar cells B1m, B2m, and B5m, respectively. S1, S2, and S3 can be normalized to obtain weights w1, w2, and w3. The second features corresponding to cells B1m, B2m, and B5m in the first feature map are V1, V2, and V3, respectively. A weighted sum of the second features can be calculated based on the weights to obtain the reconstructed feature V', where V' = w1V1 + w2V2 + w3V3. Optionally, the first feature V” corresponding to cell A1m in the first feature map can be obtained, and V' and V” can be stacked to obtain the regenerated feature V corresponding to cell A1m.

[0073] Since a higher similarity between the first and second cells with the same semantics indicates a closer similarity in their corresponding features, the weight determined by similarity better reflects the relationship between the first and second cells. This embodiment regenerates the features corresponding to the first cell based on the similarity between the first and second cells, resulting in a more realistic and natural image.

[0074] In step 305, a second image is generated based on the target semantic map and the second feature map.

[0075] In this embodiment, the second feature map can be input into the upsampling module. For example, the upsampling module can consist of multiple deconvolution layers, which can perform deconvolution processing on the second feature map. Optionally, after processing by the deconvolution layers, the result of the deconvolution processing can be semantically corrected based on the target semantic map to obtain the second image. For example, semantic correction can be performed once using the target semantic map after each deconvolution layer processing. Alternatively, semantic correction can be performed once using the target semantic map after multiple deconvolution layer processing. It is understood that this embodiment does not limit the specific number of semantic corrections. Because this embodiment uses semantic information to correct the result of the upsampling process during the upsampling process, it uses semantics to guide the generation of subsequent images, making the boundaries of different semantic regions in the obtained image clearer and the texture richer.

[0076] In this embodiment, when repairing an image, the correlation between known regions (i.e., the second region) and unknown regions (i.e., the first region) in the image is considered. The relationship between the known and unknown regions is determined through semantics. Under the guidance of rich semantics, the features of the known regions are used to regenerate the features of the unknown regions with the same semantics, thereby obtaining the repaired image and further improving the quality of the repaired image.

[0077] It should be noted that although the operations of the methods of this disclosure embodiment are described in a specific order in the above embodiments, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0078] Corresponding to the aforementioned image restoration method embodiments, this disclosure also provides embodiments of an image restoration apparatus.

[0079] like Figure 4 As shown, Figure 4 This is a block diagram of an image restoration apparatus according to an exemplary embodiment of the present disclosure. The apparatus may include: a determining module 401, a first acquiring module 402, and a restoration module 403.

[0080] The determining module 401 is used to acquire a first image and determine a first region to be repaired in the first image. The first image is an image obtained after processing the target object in the original image, and the first region is at least a portion of the processed region.

[0081] The first acquisition module 402 is used to acquire the target semantic map corresponding to the first image.

[0082] Repair module 403 is used to repair the first region based on the target semantic map to obtain the repaired second image.

[0083] In some implementations, the above processing includes removing the target object.

[0084] In other embodiments, the repair module 403 may include: a first acquisition submodule, a repair submodule, and a second acquisition submodule (not shown in the figure).

[0085] The first acquisition submodule is used to acquire the first feature map corresponding to the first image.

[0086] The repair submodule is used to regenerate the features of the first region based on the target semantic map and the features corresponding to the second region in the first feature map to obtain the second feature map. The second region is the region outside the first region in the first image.

[0087] The second acquisition submodule is used to acquire the second image based on the second feature map.

[0088] In other embodiments, the first acquisition submodule may acquire the first feature map corresponding to the first image by: performing mask processing on the first image using the first region, performing downsampling processing on the image after mask processing, and performing semantic correction on the result obtained by downsampling processing based on the target semantic map to obtain the first feature map.

[0089] In other implementations, the repair submodule may include a determination submodule and a generation submodule (not shown in the figure).

[0090] The determination submodule is used to determine the first cell corresponding to the first region, and based on the target semantic graph, to determine at least one second cell with the same semantics as the first cell, wherein the second cell corresponds to the second region.

[0091] The generation submodule is used to regenerate the features of the first cell based on the features corresponding to the second cell in the first feature map.

[0092] In other implementations, the generation submodule is configured to: obtain the first feature corresponding to the first cell in the first feature map and the respective second features corresponding to each second cell in the first feature map, and regenerate the feature corresponding to the first cell based on the first feature and the second feature.

[0093] In other implementations, the second acquisition submodule is configured to generate a second image based on the target semantic map and the second feature map.

[0094] In other embodiments, the second acquisition submodule generates a second image based on the target semantic map and the second feature map in the following manner: the second feature map is upsampled, and the result of the upsampling is semantically corrected based on the target semantic map to obtain the second image.

[0095] In other implementations, the generation submodule regenerates the feature corresponding to the first cell based on the first feature and the second feature as follows: calculates the similarity between the first feature and each of the second features, and regenerates the feature corresponding to the first cell based on the similarity.

[0096] In other implementations, the generation submodule regenerates the features corresponding to the first cell based on the similarity in the following manner: determining the weights of each second feature based on the similarity, calculating the weighted sum of the second features, and stacking the weighted sum with the first features to obtain the features corresponding to the first cell.

[0097] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiments of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0098] Figure 5 This is a schematic block diagram of an electronic device provided for some embodiments of this disclosure. For example... Figure 5 As shown, the electronic device 910 includes a processor 911 and a memory 912, and can be used to implement a client or server. The memory 912 stores computer-executable instructions (e.g., one or more computer program modules) non-transitoryly. The processor 911 executes these computer-executable instructions, which, when run by the processor 911, can perform one or more steps of the image restoration method described above, thereby implementing the image restoration method described above. The memory 912 and the processor 911 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0099] For example, processor 911 can be a central processing unit (CPU), a graphics processing unit (GPU), or other form of processing unit with data processing and / or program execution capabilities. For example, the central processing unit (CPU) can be an x86 or ARM architecture. Processor 911 can be a general-purpose processor or a special-purpose processor, and can control other components in electronic device 910 to perform desired functions.

[0100] For example, memory 912 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and processor 911 may run one or more computer program modules to implement various functions of electronic device 910. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0101] It should be noted that, in the embodiments of this disclosure, the specific functions and technical effects of the electronic device 910 can be referred to the description of the image restoration method above, and will not be repeated here.

[0102] Figure 6 This is a schematic block diagram of another electronic device provided in some embodiments of this disclosure. The electronic device 920 is, for example, suitable for implementing the image restoration method provided in the embodiments of this disclosure. The electronic device 920 can be a terminal device, etc., and can be used to implement a client or server. The electronic device 920 can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. It should be noted that... Figure 6 The illustrated electronic device 920 is merely an example and does not impose any limitation on the functionality and scope of use of the embodiments of this disclosure.

[0103] like Figure 6As shown, the electronic device 920 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 921, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 922 or a program loaded from a storage device 928 into a random access memory (RAM) 923. The RAM 923 also stores various programs and data required for the operation of the electronic device 920. The processing unit 921, ROM 922, and RAM 923 are interconnected via a bus 924. An input / output (I / O) interface 925 is also connected to the bus 924.

[0104] Typically, the following devices can be connected to I / O interface 925: input devices 926 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 927 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 928 including, for example, magnetic tapes, hard disks, etc.; and communication devices 929. Communication device 929 allows electronic device 920 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 6 An electronic device 920 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 920 may alternatively implement or have more or fewer devices.

[0105] For example, according to embodiments of this disclosure, the image restoration method described above can be implemented as a computer software program. For instance, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the image restoration method described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 929, or installed from a storage device 928, or installed from a ROM 922. When the computer program is executed by a processing device 921, the functions defined in the image restoration method provided by embodiments of this disclosure can be implemented.

[0106] Figure 7 This is a schematic diagram of a storage medium provided for some embodiments of this disclosure. For example, such as... Figure 7 As shown, the storage medium 930 can be a non-transitory computer-readable storage medium for storing non-transitory computer-executable instructions 931. When the non-transitory computer-executable instructions 931 are executed by a processor, the image restoration method described in the embodiments of this disclosure can be implemented. For example, when the non-transitory computer-executable instructions 931 are executed by a processor, one or more steps in the image restoration method described above can be performed.

[0107] For example, the storage medium 930 can be used in the aforementioned electronic device, such as the storage medium 930 may include the memory in the electronic device.

[0108] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0109] For example, the description of storage medium 930 can be found in the description of memory in the embodiments of the electronic device, and will not be repeated here. The specific functions and technical effects of storage medium 930 can be found in the description of the image restoration method above, and will not be repeated here.

[0110] It should be noted that, in the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0111] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0112] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for image restoration, the method comprising: Get the first image; The first image is the image obtained after processing the target object in the original image; Identify the first region to be repaired in the first image; The first region is at least a portion of the region that has undergone the aforementioned processing; Obtain the target semantic map corresponding to the first image; Obtain the first feature map corresponding to the first image; Based on the target semantic map, the features of the first region are regenerated using the features of the second region in the first feature map to obtain the second feature map; The second region is the region outside the first region in the first image; Based on the second feature map, a second image is obtained after repairing the first region.

2. The method of claim 1, wherein, The process includes removing the target object.

3. The method of claim 1, wherein, The step of obtaining the first feature map corresponding to the first image includes: The first image is masked using the first region; The image after the masking process is downsampled, and the result of the downsampling process is semantically corrected based on the target semantic map to obtain the first feature map.

4. The method of claim 1, wherein, The step of regenerating the features of the first region based on the target semantic map and utilizing the features of the second region corresponding to the first feature map includes: A first cell corresponding to the first region is determined, and based on the target semantic map, at least one second cell with the same semantics as the first cell is determined; the second cell corresponds to the second region. Based on the features corresponding to the second cell in the first feature map, the features of the first cell are regenerated.

5. The method of claim 4, wherein, The step of regenerating the features of the first cell based on the features corresponding to the second cell in the first feature map includes: Obtain the first feature corresponding to the first cell in the first feature map and the respective second features corresponding to each second cell in the first feature map; The features of the first cell are regenerated based on the first feature and the second feature.

6. The method of claim 1, wherein, The step of obtaining a second image after repairing the first region based on the second feature map includes: generating a second image based on the target semantic map and the second feature map.

7. The method of claim 6, wherein, The step of generating a second image based on the target semantic map and the second feature map includes: The second feature map is upsampled, and the result of the upsampling process is semantically corrected based on the target semantic map to obtain the second image.

8. The method of claim 5, wherein, The step of regenerating the features of the first cell based on the first feature and the second feature includes: Calculate the similarity between the first feature and each of the second features; Based on the similarity, the features of the first cell are regenerated.

9. The method of claim 8, wherein, The step of regenerating the features of the first cell based on the similarity includes: Based on the similarity, the weights corresponding to each of the second features are determined, and the weighted sum of the second features is calculated. Based on the weighted sum, the features of the first cell are regenerated.

10. The method of claim 9, wherein, The step of regenerating the features of the first cell based on the weighted sum includes: The weighted sum and the first feature are stacked to obtain the feature of the first cell.

11. An image restoration apparatus, the apparatus comprising: The first acquisition module is used to acquire the first image; The first image is the image obtained after processing the target object in the original image; The determination module is used to determine the first region to be repaired in the first image; The first region is at least a portion of the region that has undergone the aforementioned processing; The second acquisition module is used to acquire the target semantic map corresponding to the first image; The repair module is used to obtain the first feature map corresponding to the first image; Based on the target semantic map, the features of the first region are regenerated using the features of the second region in the first feature map to obtain the second feature map; the second region is the region outside the first region in the first image. Based on the second feature map, a second image is obtained after repairing the first region.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-10.

13. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-10.

Images