Image inpainting method, device, readable medium and electronic device

Abstract

The application relates to the field of image processing, and discloses an image repairing method, device, readable medium and electronic equipment. The method comprises the following steps: repairing at least one first reference region outside a to-be-repaired region in a to-be-repaired image by using a first repairing method, to obtain a first reference repairing region corresponding to each first reference region; determining whether a repairing result of the to-be-repaired region obtained by repairing the to-be-repaired region in the to-be-repaired image by using the first repairing method meets a repairing requirement based on a first similarity of each first reference region and the corresponding first reference repairing region. Through the image repairing method provided in the application, the electronic equipment can quickly and effectively evaluate whether the repairing result of the to-be-repaired image obtained by using the first repairing method meets the requirement, and in the case that the repairing result of the first repairing method does not meet the requirement, other methods are used to repair the to-be-repaired region.

Description

[0001] This application claims priority to Chinese patent application No. 202122450788.4, filed with the State Intellectual Property Office of China on October 11, 2021, entitled “Image Restoration Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing, and in particular to an image restoration method, apparatus, readable medium, and electronic device. Background Technology

[0003] Image inpainting technology refers to using algorithms to fill in missing areas in an image, making the restored image look very natural and difficult to distinguish from the undamaged image. It can be applied to various image / video editing scenarios such as removing specified objects, removing logos and captions from images. With the rapid development of the computer vision industry, image inpainting technology has become increasingly mature, and people have higher and higher requirements for the quality of the images output by the inpainting algorithms.

[0004] However, since there is no correct reference answer for the missing areas of the image, it is difficult to evaluate the quality of image restoration. Currently, image restoration quality is usually evaluated manually. For example, multiple staff members score the restored image to obtain a Mean Opinion Score (MOS) value, and the image restoration quality is determined based on the MOS value. However, the lack of a unified evaluation standard makes the evaluation results highly subjective, affecting their validity. In addition, the manual evaluation method is inefficient. Summary of the Invention

[0005] In view of this, embodiments of this application provide an image restoration method, apparatus, readable medium, and electronic device. By employing a first restoration method to restore regions outside the area to be restored in the image to be restored and / or at least one reference region in adjacent images of the image to be restored, reference restoration regions are obtained for each reference region. Then, by assessing the similarity between each reference restoration region and its corresponding reference region, it is determined whether the restoration result of the area to be restored obtained by the first restoration method in the image to be restored meets the requirements. This allows for a rapid and effective evaluation of the image restoration quality, enabling the use of other methods to restore the area to be restored if the restoration result of the first restoration method does not meet the requirements.

[0006] In a first aspect, embodiments of this application provide an image restoration method applied to an electronic device. The method includes: using a first restoration method to restore at least one first reference region outside the region to be restored in the image to be restored, thereby obtaining a first reference restoration region corresponding to each first reference region; determining, based on a first similarity between each first reference region and the corresponding first reference restoration region, whether the restoration result of the region to be restored obtained by using the first restoration method to restore the region to be restored meets the restoration requirements.

[0007] In other words, the electronic device repairs a portion of the region (first reference region) in the image to be repaired, other than the region to be repaired, using the first repair method to obtain the repair result of that region (first reference repair region). It then calculates the similarity between the repair result and the corresponding image (first reference region) in the image to be repaired, and determines whether the repair result of the region to be repaired obtained by repairing the region to be repaired in the image to be repaired using the first repair method meets the repair requirements based on the similarity.

[0008] The image restoration method provided in this application embodiment can quickly and effectively evaluate the quality of image restoration, facilitating further operations by the electronic device. For example, after restoring the image to be restored using the first restoration method, the method provided in this application embodiment can be used to determine whether the restoration using the first restoration method meets the restoration requirements, avoiding the output of restoration results that do not meet the requirements and affecting the user's image restoration experience. As another example, before restoring the image to be restored using the first restoration method, the method provided in this application embodiment can be used to determine whether the restoration using the first restoration method meets the restoration requirements, avoiding the continued use of the first restoration method even if the restoration result does not meet the requirements, thus saving the computing resources of the electronic device.

[0009] In one possible implementation of the first aspect above, the repair result of the region to be repaired is determined to meet the repair requirements in the following way: if the average value of each first similarity is greater than the similarity threshold, the repair result of the region to be repaired is determined to meet the repair requirements.

[0010] That is, in this embodiment of the application, the repair result of the above-mentioned region to be repaired can be determined to meet the repair requirements by comparing the average value of the first similarity of each first reference region and the corresponding first reference repair region with the size of the preset similarity threshold.

[0011] In one possible implementation of the first aspect above, the repair result of the region to be repaired is determined to meet the repair requirements by: using a first repair method to repair at least one second reference region in at least one adjacent image of the image to be repaired in the video, to obtain a second reference repair region corresponding to each second reference region; calculating the second similarity between each second reference region and the corresponding second reference repair region, and determining that the repair result of the region to be repaired meets the repair requirements if the average value of each second similarity and / or each first similarity is greater than a preset value.

[0012] In other words, in the embodiments of this application, if the image to be repaired is an image in a video, the second reference repair area can be set in other images in the video (e.g., adjacent frames as described below), thereby improving the accuracy of determining whether the result of repairing the image to be repaired using the first repair method meets the repair requirements.

[0013] In one possible implementation of the first aspect described above, the frame difference between each of the adjacent images and the image to be repaired in the video is less than a preset value.

[0014] In other words, the frame difference between adjacent images and the image to be repaired in the video is less than a preset value. For example, the position of the adjacent images in the video is within 25 frames before and after the image to be repaired. This can ensure the similarity between the adjacent images and the image to be repaired, thereby improving the accuracy of determining whether the result of repairing the image to be repaired using the first repair method meets the repair requirements.

[0015] In one possible implementation of the first aspect described above, each of the second reference regions is a portion of the image that has the highest similarity to the image to be repaired among the adjacent images.

[0016] In other words, in the embodiments of this application, each second reference region is set in the image with the highest similarity to the image to be repaired among the adjacent images of the image to be repaired in the video (i.e., similar frames as described below), so that the repair quality of the second reference region repaired by the first repair method can more accurately reflect the repair quality of the region to be repaired by the first repair method, and improve the accuracy of determining whether the result of repairing the image to be repaired by the first repair method meets the repair requirements.

[0017] In the embodiments of this application, the image with the highest similarity to the image to be repaired among the neighboring images can be determined based on at least one of the following parameters: structural similarity between each neighboring image and the image to be repaired; histogram similarity between each neighboring image and the image to be repaired; and cosine similarity between each neighboring image and the image to be repaired.

[0018] In one possible implementation of the first aspect described above, the positions of each of the second reference regions in the corresponding images are the same as the positions of the regions to be repaired in the images to be repaired.

[0019] In other words, in this embodiment, the position of the second reference region in the corresponding image of each second reference region is set to be the same as the position of the region to be repaired in the image to be repaired. This allows the repair quality of the second reference region repaired using the first repair method to more accurately reflect the repair quality of the region to be repaired using the first repair method, thereby improving the accuracy of determining whether the result of repairing the image to be repaired using the first repair method meets the repair requirements. Furthermore, in some embodiments, the shape and size of each second reference region and the region to be repaired can also be set to be the same. This allows the repair quality of the second reference region repaired using the first repair method to more accurately reflect the repair quality of the region to be repaired using the first repair method, further improving the accuracy of determining whether the result of repairing the image to be repaired using the first repair method meets the repair requirements.

[0020] In one possible implementation of the first aspect described above, the first reference regions and the regions to be repaired have the same shape and size.

[0021] In one possible implementation of the first aspect described above, the distance between each of the first reference regions and the region to be repaired is less than a preset value.

[0022] In one possible implementation of the first aspect above, the method further includes: if it is determined that the repair result of the area to be repaired meets the repair requirements, a first repair method is used to repair the area to be repaired in the image to be repaired; if it is determined that the repair result of the area to be repaired does not meet the repair requirements, a second repair method is used to repair the area to be repaired in the image to be repaired.

[0023] In other words, in this embodiment, the electronic device first determines whether the repair result of the first repair method on the image to be repaired meets the repair requirements, and if it determines that the requirements are met, it uses the first repair method to repair the area to be repaired in the image to be repaired. This avoids the situation where, before using the first repair method to repair the image to be repaired, the electronic device can first determine whether the repair using the first repair method will meet the repair requirements, thus avoiding the use of the first repair method to repair the image to be repaired even if the result does not meet the requirements, saving computational resources.

[0024] Furthermore, if the electronic device determines that the restoration result of the first restoration method does not meet the restoration requirements, it uses a second restoration method to restore the image, thereby improving the quality of image restoration. It is understood that the second restoration method can be completely different from the first restoration method, or it can be a method with adjusted parameters of the first restoration method.

[0025] Secondly, embodiments of this application provide an image restoration apparatus, which includes: an image restoration module, configured to use a first restoration method to restore at least one first reference region outside the region to be restored in the image to be restored, thereby obtaining a first reference restoration region corresponding to each first reference region; and a restoration quality evaluation module, configured to determine, based on a first similarity between each first reference region and the corresponding first reference restoration region, whether the restoration result of the region to be restored obtained by using the first restoration method to restore the region to be restored meets the restoration requirements.

[0026] In other words, the image restoration device repairs a portion of the region (first reference region) in the image to be restored, other than the region to be restored, using a first restoration method (such as the preset image restoration method below), to obtain the restoration result of that region (first reference restoration region). It then calculates the similarity between the restoration result and the corresponding image (first reference region) in the image to be restored, and determines whether the restoration result of the region to be restored obtained by repairing the region to be restored in the image to be restored using the first restoration method meets the restoration requirements based on the similarity.

[0027] The image restoration apparatus provided in this application embodiment can quickly and effectively evaluate the quality of image restoration, facilitating further operations and improving the overall quality of image restoration. For example, after restoring the image to be restored using the first restoration method, the apparatus can determine whether the restoration using the first restoration method meets the restoration requirements, avoiding the output of unsatisfactory restoration results that would negatively impact the user's image restoration experience. Furthermore, before using the first restoration method to restore the image to be restored, the apparatus can determine whether the restoration using the first restoration method meets the restoration requirements, preventing the continued use of the first restoration method even if the restoration result does not meet the requirements, thus saving computational resources.

[0028] It is understandable that the first similarity between each first reference region and its corresponding first reference repair region can be determined by calculating the structural similarity and cosine similarity between each first reference region and its corresponding first reference repair region, and by comparing the similarity of the histograms of each first reference region and its corresponding first reference repair region.

[0029] In one possible implementation of the second aspect above, the repair quality evaluation module determines whether the repair result of the region to be repaired meets the repair requirements in the following way: if the average value of each first similarity is greater than the similarity threshold, it is determined that the repair result of the region to be repaired meets the repair requirements.

[0030] That is, in this embodiment of the application, the repair result of the above-mentioned region to be repaired can be determined to meet the repair requirements by comparing the average value of the first similarity of each first reference region and the corresponding first reference repair region with the size of the preset similarity threshold.

[0031] In one possible implementation of the second aspect above, the repair quality evaluation module determines whether the repair result of the region to be repaired meets the repair requirements in the following way: the image repair module uses a first repair method to repair at least one second reference region in at least one adjacent image of the image to be repaired in the video, to obtain a second reference repair region corresponding to each second reference region; the repair quality evaluation module calculates the second similarity between each second reference region and the corresponding second reference repair region, and determines that the repair result of the region to be repaired meets the repair requirements if the average value of each second similarity and / or each first similarity is greater than a preset value.

[0032] In other words, in the embodiments of this application, if the image to be repaired is an image in a video, the second reference repair area can be set in other images in the video (e.g., adjacent frames as described below), thereby improving the accuracy of determining whether the result of repairing the image to be repaired using the first repair method meets the repair requirements.

[0033] In one possible implementation of the second aspect described above, the frame difference between each of the adjacent images and the image to be repaired in the video is less than a preset value.

[0034] In other words, the frame difference between adjacent images and the image to be repaired in the video is less than a preset value. For example, the position of the adjacent images in the video is within 25 frames before and after the image to be repaired. This can ensure the similarity between the adjacent images and the image to be repaired, thereby improving the accuracy of determining whether the result of repairing the image to be repaired using the first repair method meets the repair requirements.

[0035] In one possible implementation of the second aspect described above, each of the second reference regions is a portion of the image that has the highest similarity to the image to be repaired among the adjacent images.

[0036] In other words, in the embodiments of this application, each second reference region is set in the image with the highest similarity to the image to be repaired among the adjacent images of the image to be repaired in the video (i.e., similar frames as described below), so that the repair quality of the second reference region repaired by the first repair method can more accurately reflect the repair quality of the region to be repaired by the first repair method, and improve the accuracy of determining whether the result of repairing the image to be repaired by the first repair method meets the repair requirements.

[0037] In some embodiments, the above apparatus further includes: a preprocessing module, configured to determine, based on at least one of the following parameters, the image with the highest similarity to the image to be repaired among the neighboring images: structural similarity between the neighboring images and the image to be repaired; histogram similarity between the neighboring images and the image to be repaired; and cosine similarity between the neighboring images and the image to be repaired.

[0038] In one possible implementation of the second aspect described above, the positions of each of the second reference regions in the corresponding images are the same as the positions of the regions to be repaired in the images to be repaired.

[0039] In other words, in this embodiment of the application, the position of the second reference region in the corresponding image of each second reference region is set to be the same as the position of the region to be repaired in the image to be repaired. This makes the repair quality of the second reference region repaired by the first repair method more accurately reflect the repair quality of the region to be repaired by the first repair method, thereby improving the accuracy of determining whether the result of repairing the image to be repaired by the first repair method meets the repair requirements.

[0040] Furthermore, in some embodiments, the shape and size of each second reference region can be set to be the same as the region to be repaired, so that the repair quality of the second reference region repaired by the first repair method can more accurately reflect the repair quality of the region to be repaired by the first repair method, thereby improving the accuracy of determining whether the result of repairing the image to be repaired by the first repair method meets the repair requirements.

[0041] In one possible implementation of the second aspect described above, the first reference regions and the regions to be repaired have the same shape and size.

[0042] In other words, in this embodiment of the application, the shape and size of each first reference region are set to be the same as the region to be repaired, so that the repair quality of the first reference region repaired by the first repair method can more accurately reflect the repair quality of the region to be repaired by the first repair method, thereby improving the accuracy of determining whether the result of repairing the image to be repaired by the first repair method meets the repair requirements.

[0043] In one possible implementation of the second aspect described above, the distance between each of the first reference regions and the region to be repaired is less than a preset value.

[0044] In other words, in this embodiment of the application, the positions of each first reference region are set near the region to be repaired, so that the repair quality of the second reference region repaired by the first repair method can more accurately reflect the repair quality of the region to be repaired by the first repair method, thereby improving the accuracy of determining whether the result of repairing the image to be repaired by the first repair method meets the repair requirements.

[0045] In one possible implementation of the second aspect above, if the restoration quality evaluation module determines that the restoration result of the area to be restored meets the restoration requirements, the image restoration module uses the first restoration method to restore the area to be restored in the image to be restored; if the restoration quality evaluation module determines that the restoration result of the area to be restored does not meet the restoration requirements, the image restoration module uses the second restoration method to restore the area to be restored in the image to be restored.

[0046] In other words, in this embodiment, the image restoration device first determines whether the restoration result of the first restoration method on the image to be restored meets the restoration requirements, and then uses the first restoration method to restore the area to be restored in the image if the requirements are met. This avoids the need to determine whether the first restoration method will meet the restoration requirements before using it, and prevents the first restoration method from being used even if the result does not meet the requirements, thus saving computational resources.

[0047] Furthermore, if the image restoration device determines that the restoration result of the first restoration method does not meet the restoration requirements, it uses a second restoration method to restore the image, thereby improving the quality of image restoration. It is understood that the second restoration method can be a completely different method from the first restoration method, or it can be a method with adjusted parameters of the first restoration method.

[0048] Thirdly, embodiments of this application provide a readable medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the first aspect and various possible implementations of the first aspect to achieve any of the image restoration methods described.

[0049] Fourthly, embodiments of this application provide an electronic device comprising: a memory for storing instructions executed by one or more processors of the electronic device; and a processor, one of the processors of the electronic device, for executing the instructions to control the electronic device to implement any of the image restoration methods described in the first aspect and various possible implementations of the first aspect. Attached Figure Description

[0050] Figure 1A According to some embodiments of this application, a schematic diagram of an image restoration scenario is shown;

[0051] Figures 1B to 1D The results of image restoration using different methods are shown;

[0052] Figure 2 According to some embodiments of this application, a flowchart of an image restoration method is shown;

[0053] Figure 3A According to some embodiments of this application, a schematic diagram of an area to be repaired in an image is shown;

[0054] Figure 3B According to some embodiments of this application, a schematic diagram of another region to be repaired in an image is shown;

[0055] Figure 4A According to some embodiments of this application, a reference region and a schematic diagram of the reference region are shown;

[0056] Figure 4B According to some embodiments of this application, it is shown that Figure 4A A schematic diagram of the reference repair area shown in the diagram;

[0057] Figure 5 According to some embodiments of this application, a flowchart of an image restoration method is shown;

[0058] Figure 6A According to some embodiments of this application, a schematic diagram of an image to be repaired and similar frames of the image to be repaired is shown;

[0059] Figure 6B According to some embodiments of this application, a reference region and a schematic diagram of the reference region are shown;

[0060] Figure 6C According to some embodiments of this application, it is shown that Figure 6B A schematic diagram of the reference repair area shown in the diagram;

[0061] Figure 7 According to some embodiments of this application, a flowchart of an image restoration method is shown;

[0062] Figure 8 According to some embodiments of this application, a schematic diagram of the structure of an image restoration quality evaluation device is shown;

[0063] Figure 9 According to some embodiments of this application, a schematic diagram of the structure of an electronic device 100 is shown. Detailed Implementation

[0064] The technical solutions provided by the embodiments of this application are described below with reference to the accompanying drawings.

[0065] Figure 1A According to some embodiments of this application, a schematic diagram of an image restoration scenario is shown. As shown in Figure 1, image 0 includes subtitle 01. To change the language of the subtitle, obtain an image without subtitles, etc., image 0 can typically be restored by electronic device 100 using a preset image restoration method. For example, an image restoration method based on Generative Adversarial Networks (GAN) can be used to restore the area where subtitle 01 is located in image 0 as the missing area, thereby removing subtitle 01 and obtaining the restored image 0A. After the restored image 0A meets the restoration requirements, subsequent operations can be performed, such as adding subtitles in other languages ​​or sharing images without subtitles. That is to say, when restoring image 0 using the preset image restoration method, electronic device 100 determines the area where subtitle 01 is located as the missing area and fills in the missing area using the preset image restoration method, making the restored image, such as image 0A, look more natural and less prone to showing signs of restoration in image 01A.

[0066] As mentioned earlier, the restored image is usually evaluated manually to determine whether the restored image 0A meets the restoration requirements. However, the evaluation of image restoration quality by manual evaluation is highly subjective and not conducive to objectively evaluating whether the restoration result of image 0 using the above-mentioned GAN-based image restoration method meets the restoration requirements. Furthermore, the evaluation of image restoration quality by manual evaluation is inefficient and not conducive to the electronic device 100 adjusting the parameters of the image restoration method or taking other remedial measures based on the restoration result.

[0067] For example, Figures 1B to 1D The image shows the restored areas containing subtitle 01 in image 0, obtained by using different image restoration methods. Figure 1B The 0B subtitle marks in the image shown are still quite clear, and the restoration result can be considered unsatisfactory during manual evaluation. Figure 1CThe 0C subtitle marks in the image shown are almost invisible, and the restoration result can be considered satisfactory during manual evaluation; however, for Figure 1D The image shown, 0D, has few traces of subtitles, but requires careful observation to discern. During manual evaluation, different people will have varying opinions. Figure 1D The evaluation results regarding whether the repair requirements are met may differ.

[0068] In view of this, embodiments of this application provide an image restoration method. An electronic device can set a reference region in an area outside the region to be restored in the image to be restored, and restore the reference region based on a preset image restoration method to generate a reference restored region (i.e., an image generated by restoring the reference region using the preset image restoration method). The similarity between the reference restored region and the reference region determines whether the restoration of the region to be restored in the image to be restored using the preset image restoration method meets the restoration requirements. The image restoration method provided by this application embodiment can quickly and objectively determine whether the quality of image restoration performed by the electronic device meets the restoration requirements, improving the efficiency and effectiveness of evaluating the image restoration quality during the image restoration process. This facilitates subsequent operations by the electronic device, such as adjusting the parameters of the preset image restoration method and re-restoring the image to be restored; or using an image restoration method different from the preset image restoration method to restore the image to be restored. This improves the accuracy of image restoration by the electronic device and avoids outputting restoration results that do not meet the restoration requirements.

[0069] It is understood that the preset image inpainting method can be a neural network-based image inpainting method, such as a GAN-based image inpainting method or a Convolutional Neural Network (CNN)-based image inpainting method, or an image decomposition-based image inpainting method, such as an image inpainting method based on Singular Value Decomposition (SVD) similarity matrices, or other types of image inpainting methods, which are not limited here.

[0070] The following is combined Figure 1A The scenario shown provides a detailed description of the technical solution of the embodiments of this application.

[0071] Figure 2 According to some embodiments of this application, a flowchart illustrating the process by which an electronic device 100 evaluates the restoration result after restoring an image to be restored using a preset image restoration method is shown. The electronic device 100 is the executing entity of this process. Figure 2 As shown, the process includes the following steps:

[0072] S201: Obtain the region to be repaired in the image to be repaired. Electronic device 100 obtains the region to be repaired in the image to facilitate setting a reference region outside the region to be repaired.

[0073] For example, refer to Figure 3A The electronic device 100 uses the GAN-based image inpainting method to repair the rectangular region 01A in image 0 to obtain image 0A. Then, region 01A is the region to be repaired in image 0.

[0074] For example, in other embodiments, reference is made to... Figure 3B When an electronic device uses a preset image restoration method to restore the image 0 to remove the subtitle 01, it can also restore only the area 01B formed by the edges of each character in the subtitle 01. In this case, the area 01B is the area to be restored in the image 0.

[0075] S202: Set at least one reference region in the image to be repaired, outside the area to be repaired. During the evaluation of the repair results, the electronic device 100 sets at least one reference region in the image to be repaired, outside the area to be repaired, so that a preset image repair method can be used to repair the reference region. The repaired reference region is then compared with the original reference region to determine whether the repair result meets the repair requirements.

[0076] For example, refer to Figure 3A and Figure 4A In image 0, the region to be repaired is a rectangular region 01A. The reference region can be set as a rectangular region 02 with the same size and shape as the rectangular region 01A, located at a certain distance above the rectangular region 01A (for example, 100 pixels above the rectangular region 01A). This allows the rectangular region 02 to be repaired using the aforementioned GAN-based image inpainting method to generate a reference repair region. The reference repair region and the rectangular region 02 are then compared to determine whether the image inpainting result of the rectangular region 01A of image 0 using the GAN-based image inpainting method meets the repair requirements.

[0077] It is understood that setting the shape and size of the reference region 02 to be the same as that of the rectangular region 01A is to more accurately evaluate the restoration effect of the preset image method on the image to be restored. In other embodiments, the reference region 02 may also be other shapes and sizes, such as a region with the same shape as the rectangular region 01A but a different size, or a region of other shapes that are larger or smaller than the rectangular region 01A, etc., which are not limited here.

[0078] It is understood that in some embodiments, the distance between the reference region 02 and the region to be repaired is less than a preset value, for example, the distance between the reference region 02 and the center of the region to be repaired is less than 200 pixels, thereby ensuring the similarity between the reference region 02 and the region to be repaired, and thus more accurately evaluating the repair effect of the preset image method on the image to be repaired.

[0079] S203: The reference regions are repaired using a preset image restoration method to generate a reference restoration region corresponding to each reference region. The electronic device 100 repairs the reference regions set in step S202 using a preset image restoration method to generate a reference restoration region corresponding to each reference region. That is, the electronic device sets each reference region in the image to be repaired as a missing region, and completes each reference region to generate a reference restoration region corresponding to each reference region.

[0080] It is understandable that the preset image inpainting method is the same as the method used to inpaint the image to be inpainted in the inpainting result to be evaluated. For example, referring to the scenario shown in Figure 1, the inpainting result to be evaluated is image 0A generated by inpainting image 0 using a GAN-based image inpainting method. Therefore, the same GAN-based image inpainting method is used to inpaint each reference region, for example, for... Figure 4A The reference area 02 shown is repaired to generate a reference repair area 02A corresponding to the reference area 02.

[0081] S204: Determine whether the restoration result of the image to be restored meets the restoration requirements based on the similarity between each reference region and the corresponding reference restoration region. For example, the electronic device 100 can determine that the restoration result of the image to be restored meets the restoration requirements if the similarity between each reference region and the corresponding reference restoration region is greater than a similarity threshold, such as 0.9.

[0082] Specifically, in some embodiments, the electronic device 100 can calculate the structural similarity (SSIM) value between the reference region 02 and the reference repair region 02A using the following formula (1) to determine the similarity between the reference region 02 and the reference repair region 02A.

[0083]

[0084] Wherein, SSIM(X,Y) represents the SSIM values ​​of reference region 02 and reference repair region 02A; X is reference region 02; Y is reference repair region 02A; μ X The average pixel value of reference region 02; μ Y The average pixel value of the reference repair area 02A; σ XYThe covariance of pixel values ​​in reference region 02 and reference repair region 02A; The variance of pixel values ​​in reference region 02 The variance of pixel values ​​in the reference repair area 02A; c1 = (0.01L) 2 c2 = (0.03L) 2 L represents the dynamic range of pixel values ​​in reference region 02 and reference repair region 02A (e.g., when image 02 and image 02A are 8-bit images, L is 2). 8 -1 = 255).

[0085] It can be understood that the range of SSIM value calculated by formula (1) is [-1, 1]. The larger the value, the higher the similarity of the images. When X and Y are the same, the SSIM value is 1.

[0086] It is understood that in other embodiments, the SSIM of reference region 02 and reference repair region 02A can also be calculated in other ways, which is not limited here.

[0087] It is understood that in other embodiments, the similarity between the reference region 02 and the reference repair region 02A can also be determined in other ways, such as by comparing the similarity between the reference region 02 and the reference repair region 02A and the histogram, or by calculating the cosine similarity between the reference region 02 and the reference repair region 02A, etc., which are not limited here.

[0088] It is understood that in some embodiments, when there are multiple reference regions, the similarity between each reference region and the corresponding reference repair region can be calculated separately, and the average value of each calculated similarity can be compared with the similarity threshold to determine whether the repair result of the image to be repaired meets the repair requirements.

[0089] It is understood that in some embodiments, if the electronic device 100 determines that the repair result of the image to be repaired does not meet the repair requirements, the electronic device 100 may adjust the parameters of the preset image repair method and re-repair the image to be repaired, or it may use an image repair method different from the preset image repair method to repair the image to be repaired. In other embodiments, if the electronic device 100 determines that the repair results of multiple repair methods do not meet the repair requirements, it may also prompt the user that a satisfactory repair result cannot be obtained, so that the user can take other measures.

[0090] It is understood that in some other embodiments, the execution order of steps S201 to S204 may also be other, and no limitation is made here.

[0091] The image restoration method provided in this application can quickly and objectively evaluate the quality of image restoration, improving the efficiency and effectiveness of image restoration quality evaluation. This facilitates subsequent operations by electronic devices, such as adjusting the parameters of the preset image restoration method and restoring the image to be restored; or using an image restoration method different from the preset image restoration method to restore the image to be restored. This can improve the quality of image restoration by electronic devices and prevent electronic devices from outputting restoration results that do not meet the restoration requirements.

[0092] The above embodiments describe the implementation process of a single image restoration method. If the image to be restored is a frame in a video, the aforementioned reference area can also be set in a video frame other than the original frame (hereinafter referred to as the original frame) where the image to be restored is located. This can improve the accuracy of the evaluation results, thereby improving the accuracy of the electronic device in restoring the image and avoiding the output of restoration results that do not meet the restoration requirements by the electronic device.

[0093] The following section will continue to introduce a technical solution for evaluating the restoration quality of images in a video, using the scenario shown in Figure 1 as an example.

[0094] Specifically, Figure 5 According to some embodiments of this application, a flowchart illustrating the process by which an electronic device 100 evaluates the restoration result after restoring an image to be restored in a video using a preset image restoration method is shown. The electronic device 100 is the executing entity of this process. (Refer to...) Figure 5 The process includes the following steps:

[0095] S501: Acquire the region to be repaired in the image to be repaired and multiple adjacent frames of the corresponding video frame in the image to be repaired. Electronic device 100 acquires images of the region to be repaired in the image to be repaired and multiple adjacent frames of the corresponding video frame in order to set a reference region.

[0096] For example, refer to Figure 1A The repair results to be evaluated are Figure 1A The image shown is image 0A generated by image inpainting image 0 using a GAN-based image processing method. That is, the image to be inpainted is image 0, and image 0 is a frame from a video. (Reference) Figure 3A The electronic device 100 can acquire the region 01A to be repaired in image 0; and reference Figure 6A The electronic device 100 can also acquire k frames of images after the video frame corresponding to image 0 as adjacent frames of the original frame, such as adjacent frame 1, adjacent frame 2, ... adjacent frame k, so that the electronic device 100 can set a reference area based on region 01A, image 0 and each adjacent frame.

[0097] It is understood that in some embodiments, multiple adjacent frames of the original frame may include the first m frames and / or the last m frames of the original frame. This application does not impose such limitations.

[0098] It is understood that in some embodiments, the frame difference between multiple adjacent frames of the original frame and the original frame (i.e., the number of video frames between two video frames) does not exceed a preset value, such as no more than 25 frames. That is, multiple adjacent frames of the original frame are within the range of the first 25 frames or the last 25 frames of the original frame, so as to ensure the similarity between adjacent frames and the original frame, thereby improving the accuracy of the evaluation results.

[0099] S502: Set at least one reference region in the region outside the region to be repaired in the image to be repaired and / or in the adjacent frame of the original frame.

[0100] For example, in some embodiments, a reference region with the same shape and size as the region to be repaired can be set in the region above the region to be repaired in the image to be repaired, and in the frame (similar frame) with the highest similarity to the image to be repaired among multiple adjacent frames.

[0101] Specifically, in some embodiments, reference is made to Figure 3A The electronic device 100 can set a reference area 02 100 pixels above area 01A in image 0. (Reference) Figure 6B The electronic device 100 can also first determine the frame (similar frame) with the greatest similarity to the image to be repaired among adjacent frames 1 to adjacent frames k. For example, if it is determined that adjacent frame k has the greatest similarity to the image to be repaired, then adjacent frame k is a similar frame, and a reference region 03 is set in the similar frame at the same position as region 01A in image 0.

[0102] It is understood that in some embodiments, similar frames can be determined based on the SSIM value between the original frame and adjacent frames calculated using the aforementioned formula (1), for example, the frame with the largest SSIM value among the adjacent frames of the original frame can be used as the similar frame. In other embodiments, it can also be determined by the similarity of the histograms between the original frame and each adjacent frame, cosine similarity, etc., which is not limited in the embodiments of this application.

[0103] It is understood that setting the reference region 03 at the same position as region 01A in image 0 within a similar frame is to more accurately determine whether the restoration result of the image to be restored using the preset image restoration method meets the restoration requirements. In other embodiments, the reference region 03 may also be set at other positions within the similar frame, which is not limited here.

[0104] It is understood that in some embodiments, the reference region in the image to be repaired and the reference region in the adjacent frame may be other numbers, or the reference region may be set only in the image to be repaired or the adjacent frame, which is not limited here.

[0105] It is understood that in some embodiments, similar frames can be frames without subtitles. The electronic device 100 can first identify whether there is text in multiple adjacent frames of the original frame, and take the frame with the highest similarity to the image to be repaired among the adjacent frames without text as the similar frame. In this way, it can more accurately determine whether the repair result of repairing the image to be repaired using the preset image repair method meets the repair requirements.

[0106] S503: The preset image restoration method is used to restore each reference region, generating a reference restoration region corresponding to each reference region. The electronic device 100 restores each reference region according to the preset image restoration method, generating a reference restoration region corresponding to each reference region. For details, please refer to step S203, which will not be repeated here.

[0107] For example, referring to the scenario shown in Figure 1, if the restoration result to be evaluated is image 0A generated by restoring image 0 using a GAN-based image inpainting method, then the same GAN-based image inpainting method is used to restore each reference region. For example, using a GAN-based image inpainting method to restore... Figure 4A The reference area 02 shown is repaired and generated as follows. Figure 4B The reference restoration area 02A is shown; for example, an image inpainting method based on GAN is used to restore the image. Figure 6C The reference area 03 shown is repaired to generate a reference repair area 03A.

[0108] S504: Calculate the similarity between each reference region and the corresponding reference restoration region. The electronic device 100 calculates the similarity between each reference region and the corresponding reference restoration region to evaluate whether the restoration result of the image to be restored meets the restoration requirements. For details, please refer to step S204, which will not be elaborated here.

[0109] For example, the SSIM value of reference region 02 and reference repair region 02A can be determined to be 0.97, and the SSIM value of reference region 03 and reference repair region 03A can be determined to be 0.93, using the aforementioned formula (1).

[0110] S505: Determine whether the similarity meets the preset conditions. For example, the electronic device 100 determines whether the repair result of the image to be repaired meets the repair requirements based on whether the average similarity of each reference region is greater than the similarity threshold. If the average similarity is greater than the similarity threshold, it is determined that the repair result of the image to be repaired meets the repair requirements, and the process proceeds to step S506 to output the evaluation result that the repair result meets the repair requirements; otherwise, it is determined that the repair result of the image to be repaired does not meet the repair requirements, and the process proceeds to step S507 to output the evaluation result that the repair result does not meet the repair requirements.

[0111] For example, as mentioned above, the SSIM value of reference region 02 and reference repair region 02A is 0.97, and the SSIM value of reference region 03 and reference repair region 03A is 0.93. The average value is 0.95, which is greater than the similarity threshold of 0.9. Therefore, it can be determined that the repair result of image 0 meets the repair requirements.

[0112] It can be understood that when there is only one reference region, the average similarity is the similarity between the reference region and the reference repair region.

[0113] S506: Determine that the restoration result of the image to be restored meets the restoration requirements. If the average value of the aforementioned similarities is greater than the similarity threshold, the electronic device 100 determines that the restoration result of the image to be restored meets the restoration requirements.

[0114] In some embodiments, when the electronic device 100 determines that the repair result of the image to be repaired meets the repair requirements, it may prompt the user so that the user can share the repair result of the image to be repaired with other electronic devices.

[0115] S507: Determining that the repair result of the image to be repaired does not meet the repair requirements. If the average value of the aforementioned similarities is less than or equal to the similarity threshold, the electronic device 100 determines that the repair result of the image to be repaired does not meet the repair requirements.

[0116] In some embodiments, if the electronic device 100 determines that the restoration result of the image to be restored does not meet the restoration requirements, it can adjust the parameters of the preset image restoration method and restore the image to be restored again. Alternatively, it can use an image restoration method different from the preset method to restore the image to be restored. In other embodiments, if the electronic device 100 determines that the restoration results of multiple restoration methods do not meet the restoration requirements, it can also prompt the user that a satisfactory restoration result cannot be obtained, so that the user can take other measures.

[0117] It is understood that in some other embodiments, the execution order of steps S501 to S507 may also be other, and no limitation is made here.

[0118] The method provided in this application embodiment can quickly and objectively evaluate the restoration quality of images in a video, improving the efficiency and effectiveness of image restoration quality evaluation. Furthermore, since the reference region can be set in the video frame with the highest similarity to the image to be restored, the accuracy of the evaluation can be improved. The electronic device can then perform subsequent operations based on the evaluation results, such as adjusting the parameters of the preset image restoration method and re-restoring the image to be restored; or using an image restoration method different from the preset method to restore the image to be restored, thereby improving the accuracy of image restoration by the electronic device and avoiding the output of restoration results that do not meet the restoration requirements.

[0119] This application also provides an image restoration method, which can first determine whether the preset image restoration method meets the restoration requirements of the image to be restored. If it is determined that the restoration result of the image to be restored by the method meets the restoration requirements, then the method is used to restore the image to be restored, thereby saving the computing resources of electronic devices.

[0120] Specifically, Figure 7 According to some embodiments of this application, a schematic flowchart of an image restoration method is shown. This process is performed by an electronic device 100, such as... Figure 7 As shown, the process includes the following steps.

[0121] S701: Obtain the region to be repaired in the image to be repaired.

[0122] If the image to be repaired is a single image, please refer to step S201 for details, which will not be repeated here.

[0123] When the image to be repaired is a frame in a video, the electronic device 100 can also acquire multiple adjacent frames of the video frame containing the image to be repaired. For details, please refer to step S501, which will not be elaborated here.

[0124] S702: Set at least one reference area in the image to be repaired, outside the area to be repaired.

[0125] If the image to be repaired is a single image, the reference area can be set with reference to step S202, which will not be elaborated here.

[0126] When the image to be repaired is a frame from a video, the reference area can be set with reference to step S502, which will not be elaborated here.

[0127] S703: The preset image inpainting method is used to repair each reference region, generating a reference repair region corresponding to each reference region. The specific process can be found in step S203, and will not be elaborated here.

[0128] S704: Calculate the similarity between each reference region and the corresponding reference repair region. The specific calculation method can be found in step S204, and will not be elaborated here.

[0129] S705: Determine whether the similarity meets the preset conditions. If the electronic device 100 determines that the similarity between the reference area and the corresponding reference repair area meets the preset conditions, it indicates that the preset image repair method can meet the repair requirements, and proceeds to step S706; otherwise, it indicates that the preset image repair method cannot meet the repair requirements, and proceeds to step S707. The specific judgment method can be found in step S505, and will not be elaborated here.

[0130] S706: The image to be repaired is repaired using a preset image repair method. That is, when the electronic device 100 determines that repairing the image to be repaired using the preset image repair method can meet the repair requirements, it uses this method to repair the image to be repaired.

[0131] S707: The image to be repaired is repaired using an image restoration method different from the preset image method. That is, when the electronic device 100 determines that repairing the image using the preset image restoration method cannot meet the repair requirements, it uses an image restoration method different from the preset image method to repair the image. For example, refer to... Figure 1A In the scenario shown, if the restoration result of the GAN-based image inpainting method cannot meet the restoration requirements, CNN-based image inpainting methods, SVD similarity matrix-based image inpainting methods, etc., can be used to restore the image.

[0132] It is understood that in some other embodiments, the execution order of steps S701 to S707 may also be other, and no limitation is made here.

[0133] The image restoration method provided in this application can avoid using the same method to restore the image even when the preset image restoration method fails to achieve the desired restoration result, thus saving the computing resources of electronic devices.

[0134] This application also provides an image restoration device that can quickly and objectively evaluate the restoration effect of an image to be restored using a preset method, thereby improving the objectivity and effectiveness of evaluating the quality of image restoration. For example, the device can adjust the parameters of the preset image restoration method and restore the image to be restored again; or use an image restoration method different from the preset image restoration method to restore the image to be restored, thereby improving the accuracy of image restoration by electronic devices and avoiding the output of restoration results that do not meet the restoration requirements by electronic devices.

[0135] Specifically, Figure 8 According to some embodiments of this application, a schematic diagram of the structure of an image restoration apparatus 800 is shown, such as... Figure 8 As shown, the image restoration device 800 includes: a preprocessing module 801, a reference area acquisition module 802, an image restoration module 803, and a restoration quality evaluation module 804.

[0136] The preprocessing module 801 is used to acquire the region to be repaired in the image to be repaired, so that the reference region acquisition module 802 can set the reference region according to the region to be repaired. Furthermore, when the image to be repaired is a frame from a video, the preprocessing module 801 is also used to acquire adjacent frames of the video frame containing the image to be repaired. See [reference needed] for details. Figure 2 , Figure 5 The embodiments shown in Figure 7 will not be described in detail here.

[0137] In addition, refer to Figure 5 In the embodiment shown, the preprocessing module 801 can also be used to determine similar frames of the video frame containing the image to be repaired.

[0138] The reference region acquisition module 802 is used to set a reference region outside the region to be repaired in the image to be repaired obtained by the preprocessing module 801, so that the image repair module 803 can repair the reference region according to a preset image repair method, generating a reference repair region corresponding to the reference region. For example, reference... Figure 2 In the illustrated embodiment, the reference region acquisition module 802 can set a reference region outside the region to be repaired in the image to be repaired. For example, the reference... Figure 5 In the embodiment shown, the reference region acquisition module 802 can set a reference region in adjacent frames of the video frame containing the image to be repaired, or in similar frames of the video frame containing the image to be repaired.

[0139] The image restoration module 803 is used to perform image restoration on the reference area set by the reference area acquisition module 802 according to the preset image restoration method, and generate a reference restoration area corresponding to each reference area, so as to evaluate the restoration result of the preset image restoration method on the image to be restored.

[0140] In some embodiments, reference Figure 7 In the embodiment shown, if the image restoration module 803 determines that the result of restoring the image to be restored using the preset image restoration method does not meet the preset conditions, the image restoration module 803 may also use other methods to restore the image to be restored.

[0141] For example, referring to the scenario shown in Figure 1 and Figure 7 In the embodiment shown, when the image restoration quality evaluation module 804 determines that the restoration result of the GAN-based image restoration method cannot meet the restoration requirements, the image restoration module 803 can use a CNN-based image restoration method, an SVD similarity matrix-based image restoration method, or adjust the parameters of the GAN-based image restoration method to restore the image to be restored again.

[0142] Repair quality evaluation module 804: It is used to calculate the similarity between each reference region and the corresponding reference repair region, and to determine whether the preset image repair method can meet the repair requirements based on the calculated similarity.

[0143] In some embodiments, reference Figure 2 , Figure 5 and Figure 7 In the embodiment shown, the restoration quality evaluation module 804 can determine the similarity between each reference region and the reference restoration region corresponding to each reference region based on the SSIM value between each reference region and the reference restoration region corresponding to each reference region generated by the image restoration module 803. If the average similarity between each reference region and the reference restoration region corresponding to each reference region is greater than a preset similarity threshold, it is determined that the image restoration method can meet the restoration requirements when used to restore the image to be restored.

[0144] Understandable. Figure 8 The structure of the image evaluation device 800 shown is only an example. In other embodiments, the image evaluation device 800 may include more or fewer modules, and some modules may be merged or split. This application does not limit the embodiments.

[0145] It is understood that the electronic device 100 in the foregoing embodiments can be any electronic device, including but not limited to mobile phones, smart screens, desktop computers, tablet computers, laptop computers, wearable devices, head-mounted displays, mobile email devices, portable game consoles, personal digital assistants (PDAs), virtual reality (VR) or augmented reality (AR) devices, ultra-mobile personal computers (UMPCs), netbooks, and televisions in which one or more processors are embedded or coupled.

[0146] Figure 9 According to some embodiments of this application, a schematic diagram of the structure of an electronic device 100 is shown. For example... Figure 9 As shown, the electronic device 100 may include: one or more processors 101, system memory 102, non-volatile memory (NVM) 103, input / output (I / O) devices 104, communication interface 105, and system control logic 106 for coupling the processor 101, system memory 102, non-volatile memory 103, input / output (I / O) devices 104, and communication interface 105. Wherein:

[0147] Processor 101 may include a Central Processing Unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. In some embodiments, processor 101 may be used to execute instructions to implement the image quality evaluation methods provided in the above embodiments. In other embodiments, the NPU may also repair a reference region based on a preset image inpainting method.

[0148] System memory 102 is volatile memory, such as random-access memory (RAM) or double data rate synchronous dynamic random-access memory (DDR SDRAM). System memory is used for temporary storage of data and / or instructions. For example, in some embodiments, system memory 102 can be used to temporarily store reference regions, corresponding reference regions, and reference restoration regions; it can also be used to store instructions corresponding to preset image restoration methods.

[0149] The non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD), a digital versatile disc (DVD), a solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a secure digital storage card (SD). In some embodiments, the non-volatile memory 103 may be used to store instructions corresponding to the image restoration methods provided in the above embodiments, instructions corresponding to preset image restoration methods, and images to be restored corresponding to restoration results to be evaluated, etc.

[0150] Specifically, system memory 102 and non-volatile memory 103 may each include a temporary copy and a permanent copy of instruction 107. Instruction 107 may include, when executed by at least one of processors 101, causing electronic device 100 to implement the image restoration methods provided in the embodiments of this application.

[0151] Input / output (I / O) device 104 may include a user interface that allows a user to interact with electronic device 100. For example, in some embodiments, input / output (I / O) device 104 may include an output device such as a display, and may also include an input device such as a keyboard, mouse, or touchscreen. The user can interact with electronic device 100 via a display, keyboard, mouse, touchscreen, etc., to identify the area to be repaired in the image to be repaired.

[0152] The communication interface 105 may include a transceiver for providing a wired or wireless communication interface for the electronic device 100, thereby enabling communication with any other suitable device via one or more networks. In some embodiments, the communication interface 105 may be integrated into other components of the electronic device 100, for example, the communication interface 105 may be integrated into the processor 101. In some embodiments, the electronic device 100 may communicate with other devices through the communication interface 105; for example, the electronic device 100 may acquire the image to be repaired from other electronic devices through the communication interface 105, or it may transmit the evaluation results of the image repair quality to other electronic devices through the communication interface 105.

[0153] System control logic 106 may include any suitable interface controller to provide any suitable interface to other modules of electronic device 100. For example, in some embodiments, system control logic 106 may include one or more memory controllers to provide an interface to system memory 102 and non-volatile memory 103.

[0154] In some embodiments, at least one of the processors 101 may be packaged together with the logic of one or more controllers for system control logic 106 to form a system in package (SiP). In other embodiments, at least one of the processors 101 may also be integrated on the same chip with the logic of one or more controllers for system control logic 106 to form a system-on-chip (SoC).

[0155] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0156] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0157] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0158] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0159] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0160] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0161] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0162] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. An image inpainting method applied to an electronic device, comprising: The method comprises the following steps: Repairing at least one first reference region outside the to-be-repaired region in the to-be-repaired image by using a first repairing method to obtain a first reference repairing region corresponding to each of the first reference regions; Determining whether the repairing result of the to-be-repaired region obtained by repairing the to-be-repaired region in the to-be-repaired image by using the first repairing method meets the repairing requirement based on the first similarity of each of the first reference regions and the corresponding first reference repairing region.

2. The method of claim 1, wherein, The repairing result of the to-be-repaired region is determined to meet the repairing requirement in the following manner: In a case where the average value of each of the first similarities is greater than a similarity threshold value, the repairing result of the to-be-repaired region is determined to meet the repairing requirement.

3. The method of claim 2, wherein, The repairing result of the to-be-repaired region is determined to meet the repairing requirement in the following manner: In a case where each of the second similarities and / or the average value of each of the first similarities is greater than a preset value, the repairing result of the to-be-repaired region is determined to meet the repairing requirement, each of the second similarities being a similarity of each of the second reference regions and the corresponding second reference repairing region, each of the second reference repairing regions being a reference repairing region corresponding to each of the second reference regions obtained by repairing at least one second reference region in at least one adjacent image adjacent to the to-be-repaired image in a video by using the first repairing method.

4. The method of claim 3, wherein, Each of the adjacent images has a frame difference with the to-be-repaired image in the video less than a preset value.

5. The method of claim 3, wherein, Each of the second reference regions is a partial region in an image having the greatest similarity with the to-be-repaired image among the adjacent images.

6. The method of claim 3, wherein, The position of each of the second reference regions in the corresponding image is the same as the position of the to-be-repaired region in the to-be-repaired image.

7. The method according to any one of claims 1 to 6, characterized in that, Each of the first reference regions has the same shape and size as the to-be-repaired region.

8. The method of claim 7, wherein, Each of the first reference regions is located at a distance less than a preset value from the to-be-repaired region.

9. The method of claim 8, wherein, The method further comprises the following steps: In a case where it is determined that the repairing result of the to-be-repaired region meets the repairing requirement, repairing the to-be-repaired region in the to-be-repaired image by using the first repairing method; In a case where it is determined that the repairing result of the to-be-repaired region does not meet the repairing requirement, repairing the to-be-repaired region in the to-be-repaired image by using a second repairing method.

10. An image inpainting apparatus characterized by comprising: The method comprises the following steps: An image repairing module is configured to repair at least one first reference region outside a to-be-repaired region in a to-be-repaired image by using a first repairing method to obtain a first reference repairing region corresponding to each of the first reference regions; A repairing quality evaluation module is configured to determine whether a repairing result of the to-be-repaired region obtained by repairing the to-be-repaired region in the to-be-repaired image by using the first repairing method meets a repairing requirement based on a first similarity of each of the first reference regions and the corresponding first reference repairing region.

11. The apparatus of claim 10, wherein, The repairing quality evaluation module determines whether the repairing result of the to-be-repaired region meets the repairing requirement in the following manner: In a case where the average value of each of the first similarities is greater than a similarity threshold value, the repairing result of the to-be-repaired region is determined to meet the repairing requirement.

12. The apparatus of claim 10, wherein, The repair quality evaluation module determines whether the repair result of the to-be-repaired region meets the repair requirement by the following manner: The repair quality evaluation module determines that the repair result of the to-be-repaired region meets the repair requirement in a case where each second similarity and / or an average value of each first similarity is greater than a preset value, each second similarity is a similarity between each second reference region and a corresponding second reference repair region, and each second reference repair region is a corresponding reference repair region of each second reference region obtained by repairing at least one second reference region in at least one adjacent image adjacent to the to-be-repaired image in the video using a first repair method by the image repair module.

13. The apparatus of claim 12, wherein, Each adjacent image has a frame difference less than a preset value with the to-be-repaired image in the video.

14. The apparatus of claim 12, wherein, Each second reference region is a partial region in an image having the greatest similarity with the to-be-repaired image in each adjacent image.

15. The apparatus of claim 12, wherein, Each second reference region has a same position in a corresponding image as a position of the to-be-repaired region in the to-be-repaired image.

16. The apparatus of any one of claims 10 to 15, wherein, Each first reference region has a same shape and size as the to-be-repaired region.

17. The apparatus of claim 16, wherein, Each first reference region has a distance less than a preset value from the to-be-repaired region.

18. The apparatus of claim 17, wherein, The image repair module repairs the to-be-repaired region in the to-be-repaired image using the first repair method in a case where the repair quality evaluation module determines that the repair result of the to-be-repaired region meets the repair requirement, and repairs the to-be-repaired region in the to-be-repaired image using a second repair method in a case where the repair quality evaluation module determines that the repair result of the to-be-repaired region does not meet the repair requirement.

19. A readable medium characterized by The readable medium has instructions stored thereon, and the instructions, when executed on the electronic device, cause the electronic device to implement the image repair method of any one of claims 1 to 9.

20. An electronic device, comprising: comprising: a memory configured to store instructions for execution by one or more processors of the electronic device; and a processor, which is one of the processors of the electronic device, configured to execute the instructions to cause the electronic device to implement the image repair method of any one of claims 1 to 9.

Images