Method, processing system and computer program product for repairing a high resolution image

Abstract

A method, processing system and computer program product for inpainting a high resolution image, comprising: providing a high resolution input image; providing at least one inpainting mask; selecting at least one rectangular sub-region of the input image and at least one aligned rectangular sub-region of the inpainting mask; receiving an inpainted sub-region of the input image and at least one set of synthesized pixels, processing the sub-region of the input image and the corresponding aligned sub-region on the inpainting mask using a machine learning model; generating a high resolution output image, the output image comprising the inpainted sub-region, the inpainted sub-region having at least one set of synthesized pixels, the at least one set of synthesized pixels being semantically and visually adapted to the surrounding part of the input image at the same time.

Description

TECHNICAL FIELD

[0001] The present invention relates to a method for inpainting a high resolution image, in particular applied to mobile devices. The present invention also relates to a processing system able to implement said method and to a computer program product associated with said method. BACKGROUND

[0002] Inpainting is the process of reconstructing missing, lost or deteriorated parts in images and videos. It can be used in editing automatic scenes, retouching photos, restoring old photos and old videos and many other applications.

[0003] There are many inpainting techniques based on classical computer vision methods, such as the one disclosed in Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas S. Huang “Generative Image Inpainting with Contextual Attention”, 2018. There are also many inpainting techniques based on machine learning methods, such as the ones disclosed in Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman “PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing”, 2009 and in Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, Bryan Catanzaro “Image Inpainting for Irregular Holes Using Partial Convolutions”, 2019.

[0004] However, the state of the art has drawbacks. Classical computer vision methods usually produce unrealistic results for complex scenes, as they do not contain any semantic information and are usually based on patch matching and copy-pasting. The results of machine learning methods are usually validated at low resolution (e.g. 256x256 or 512x512) and cannot be applied on mobile platforms with high resolution images (e.g. 4K images).

[0005] Therefore, there is a need to provide a inpainting method that is reliable for complex scenes and suitable for high-resolution images, especially for mobile devices with limited memory and computing power. SUMMARY

[0006] Starting from the described prior art, the object of the present invention is to develop a method for inpainting images that is firstly suitable to ensure the processing of high-resolution images without additional computing power and secondly can be implemented on a mobile phone and guarantees good inpainting quality.

[0007] The object is achieved by a method for inpainting high-resolution images, a processing system comprising an image processing pipeline of the invention and an associated computer program product.

[0008] A first aspect of the invention provides a method for inpainting high-resolution images implemented on a computer. The method comprises providing a high-resolution input image; providing at least one inpainting mask indicating at least one group of pixels to be removed and synthesized; selecting at least one rectangular sub-region of the input image and at least one aligned rectangular sub-region of the inpainting mask such that the rectangular sub-region of the input image contains at least one group of pixels to be removed and synthesized, the at least one sub-region of the input image and the corresponding aligned sub-region on the inpainting mask have the same minimum possible size, the at least one sub-region of the input image and the corresponding aligned sub-region on the inpainting mask do not decrease the information gain computed at a position; processing the sub-region of the input image and the corresponding aligned sub-region on the inpainting mask using a machine learning model in case of receiving an inpainted sub-region of the input image and at least one group of synthesized pixels that are semantically and visually adapted to the surrounding part of the input image at the same time; generating a high-resolution output image, the output image comprising the inpainted sub-region with at least one group of synthesized pixels that are semantically and visually adapted to the surrounding part of the input image at the same time.

[0009] Advantageous developments of the method for inpainting high-resolution images of the invention are detailed in the dependent claims. In particular, the method of the invention can also have one or more of the following features alone or in combination:

[0010] The step of selecting at least one rectangular sub-region of the input image and at least one aligned rectangular sub-region of the inpainting mask comprises obtaining information about the selected mask region within the inpainting mask.

[0011] The step of selecting at least one rectangular sub-region of the input image and at least one aligned rectangular sub-region of the inpainting mask further comprises a step of clustering mask regions.

[0012] The step of providing a inpainting mask comprises automatically proposing an inpainting mask.

[0013] The method further comprises a step of post-processing the provided inpainting mask to meet the requirements of the machine learning model.

[0014] The step of post-processing the inpainting mask further comprises a user-defined manual mask refinement.

[0015] The method further comprises a step of rotating the selected image sub-region and the selected mask sub-region before processing by the machine learning model.

[0016] The method further comprises a step of rotating the repaired sub-region output by the machine learning model.

[0017] The method further comprises a step of downscaling at least one selected image sub-region and at least one selected mask sub-region to adjust the resolution to belong to a predefined set of resolutions supported by the machine learning model.

[0018] The method further comprises a step of upsampling the repaired sub-region output by the machine learning model.

[0019] The step of generating a high resolution output image comprises blending the repaired sub-region into a high resolution input image using the provided inpainting mask.

[0020] A second aspect of the invention provides a processing system implementing the method of the invention.

[0021] A third aspect of the invention provides a mobile phone comprising the processing system according to the invention.

[0022] A fourth aspect of the invention provides a computer program product comprising instructions which, when executed by a processor, cause the processor to perform the method of repairing a high resolution image of the invention.

[0023] The invention allows a user to select parts of an image that should be removed from the image (manually or semi-automatically) and to automatically reconstruct the missing parts based on the "non-missing" areas of the image. The invention can be efficiently applied on mobile devices and can process high resolution images (e.g. 4K resolution) directly on the mobile device.

[0024] Thanks to the step of image / mask cropping selection, it is possible to find the best image / mask crop that allows to process in presence of certain computational power limitations while providing enough information to receive a high quality inpainted image crop at the output of the machine learning model.

[0025] The pipeline presented in this invention allows to implement high quality inpainting features on mobile devices. The inpainting features support the inpainting of high resolution images (up to 4K), achieve competitive visual quality results with powerful machine learning models and run directly on mobile computing platforms (without any external service). BRIEF DESCRIPTION OF DRAWINGS

[0026] Other advantages and features will become apparent from reading the description of the invention and from reviewing the associated drawings, in which:

[0027] Figure 1 The general steps of the method for inpainting images according to the invention are shown;

[0028] Figure 2 A possible effect of the image processing procedure according to the invention is shown;

[0029] Figure 3 A flowchart of the method according to an embodiment of the invention is shown;

[0030] Figures 4A-4C An example of a mask and cropping is shown;

[0031] Figure 5 An exemplary result of the mask region aggregation according to the invention is shown;

[0032] Figure 6 The image pipeline according to the invention is shown;

[0033] Figure 7 An exemplary image viewing application implementing the method according to the invention on a mobile device is shown. DETAILED DESCRIPTION

[0034] Figure 1 A high level view of the method for inpainting images according to the invention, in particular for inpainting high resolution images, is shown, wherein GAN is a generative adversarial network, UNET is a network architecture for image semantic segmentation, and high resolution images are images with a vertical resolution of 720 or more, also called high definition images. A smaller region (i.e. image subregion) with respect to the high resolution input image is selected on the high resolution input image, where the object to be removed is located. A special inpainting mask within the smaller region, i.e. within the image subregion, defines a determined group of pixels to be removed from the input image. The pre-processed smaller image region is then transmitted together with the inpainting mask to a trained neural network for processing. At the machine learning model output, the image subregion with the synthesized group of pixels is received, and the entire output image with the inpainted region is received after post-processing.

[0035] As Figure 2As illustrated, one of the purposes of the method of the application is to remove people from an image. It can be observed that the inpainting method of the application provides reasonable results for users and complex scenes.

[0036] Reference will now be made to Figure 3 The method of inpainting an image of the application is described, Figure 3 A relevant step flowchart is illustrated. The inpainting system provided by the application is composed of several steps.

[0037] The method starts by providing an input image. The input image can be an image recorded by a digital camera, in particular by a digital camera on a mobile device, stored in a memory. Nowadays, digital cameras can take images of different sizes. Digital cameras currently installed on mobile devices usually record images of 2048 pixels x 1536 pixels, 3024 pixels x 4032 pixels, etc. Preferably, the step of providing an input image comprises reading the input image from a memory, retrieving image parameters, in particular the size and resolution of the image; optionally, displaying the input image on a display for the user to observe. Alternatively, the input image and the parameters can be retrieved directly from the camera.

[0038] Then, the method enters the step of providing an inpainting mask. An "mask" is understood as a binary image having the same resolution as the input image, i.e. the mask is composed of pixels of two colors, typically black and white. An "inpainting mask" is understood as a mask indicating which pixels should be removed from the input image and replaced by pixels generated by the inpainting method. For example, when a pixel at a given position in the inpainting mask is black, we need to remove the pixel at this position from the input image. Preferably, the step of providing an inpainting mask comprises manually drawing a user-defined inpainting mask through the user interface (UI) of the mobile device. By way of example, the user can use a touchscreen and use a brush tool to select pixels to be removed.

[0039] In a preferred embodiment, the step of providing an inpainting mask comprises automatically proposing an inpainting mask. The optional step of automatically proposing an inpainting mask comprises a step of analyzing the input image. The input image is analyzed using a semantic image segmentation model to detect objects. One existing image segmentation technique can be a model based on machine learning. The segmentation model is fine-tuned with the aim of generating high-quality inpainting masks typically removing objects such as people, cars, trash, etc. Based on the detected objects and their fragments, the step ends with the automatic proposal of an inpainting mask, i.e. the mask indicates which objects in the input image can be removed in the next steps and later replaced by a reconstructed background.

[0040] In another embodiment, in this sub-step of automatic mask post-processing, the inpainting mask is carefully post-processed to meet the requirements of the inpainting model (including smoothness and coherence enhancement, etc.). The steps can include operations such as morphological erosion and dilation, blurring and thresholding, etc. Here the "inpainting model" refers to the model described in the further machine learning model prediction step, and the "requirements of the inpainting model" refer to the fact that the inpainting mask provided in the method of the present application should be similar in smoothness and coherence to the mask used during the training of the machine learning model. For example, the black pixels within the selected inpainting mask should form smooth, coherent and regular shapes, such as overlapping lines, curves or ellipses, etc. Thanks to the automatic inpainting mask post-processing, noise and artifacts generated by the machine learning model due to the difference between the data used during training and the data used during prediction can be avoided.

[0041] Preferably, this step of mask post-processing also includes manual user-defined mask refinement. For example, after this step of automatic mask post-processing, only certain objects from the inpainting mask can be selected. Once the inpainting mask is selected, it can be refined by the user through the user interface UI. For example, the user can use a brush and eraser tool to adjust the precise layout of the pixels within the inpainting mask.

[0042] Then, the method enters the step of pre-processing the input image and the selected inpainting mask. Knowing the resolution of the input image and at least one final inpainting mask, i.e. at least one selected inpainting mask and optionally refined inpainting mask, it is possible to automatically determine how to extract partial images (i.e. image sub-regions), pre-process them and further pass them to the machine learning model.

[0043] In particular, this step of pre-processing the input image and the selected inpainting mask includes the step of selecting at least one image crop and at least one aligned mask crop, the image crop to be extracted from the input image. "Image crop" is understood here as a rectangular crop of the input image, i.e. a rectangular sub-region of the input image, smaller than the entire image. "Mask crop" is understood here as the corresponding rectangular crop on the inpainting mask (same coordinates as the image crop), i.e. a rectangular sub-region of the inpainting mask, smaller than the entire inpainting mask. Thus, the image crop is always aligned with the corresponding mask crop. In this step, it is decided how many crops should be extracted from the input image (and correspondingly from the inpainting mask) and how these crops should be treated (rotated, rescaled, etc.) in order to effectively obtain the best quality results.

[0044] The step of selecting at least one image crop and at least one aligned mask crop comprises obtaining information about the mask region located within the selected inpainting mask. A "mask region" is to be understood as a set of pixels forming a continuous shape within the inpainting mask (having the same color, e.g. black, indicating pixels to be removed). For example, Figures 4A-4C Inpainting masks with a single mask region within each inpainting mask are shown, while Figure 5 Examples of masks composed of multiple mask regions are shown.

[0045] For a single mask region detected within a selected inpainting mask, the system analyzes the resolution of the input image and the size and shape of the mask region (e.g. computes the minimal bounding box). Based on this information, the size and position of the best image / mask crop is selected, while the same size and position of the best image crop is selected, to provide information for the inpainting model (for reconstruction quality) while minimizing the number of pixels to process (for performance purposes). The image / mask crop size is selected from the smallest possible among the available image / mask crop sizes. The information gain after increasing the crop size can be estimated based on the ratio of black and white pixels within the mask crop and the statistics (e.g. standard deviation, variance, etc.) of the non-removed pixels within the image crop. Based on the estimated information gain, it is decided whether a larger crop size is advantageous (using a parameterized threshold). The system can also consider rotated rectangular crops to find the best fitting crop (based on the minimal bounding box computed for the rotated mask region). If this feature is activated and the estimated information gain is higher for a rotated crop, i.e. for a rectangular crop with sides rotated with respect to the image principal axes, the system can decide to output the rotated crop.

[0046] The size of the available mask / image crops is defined according to the specific inpainting model characteristics and hardware specifications (e.g. memory and central processing unit, CPU, limitations). For example, for a 4K image, the available set of crops can include the following rectangles: 512x512, 512x1024, 1024x1024, 2048x2048, 1024x4096, 512x4096, etc. For example, depending on the hardware specifications, only devices with less than 4 GB of memory will consider the following rectangles: 512x512, 512x1024, 1024x1024.

[0047] Figures 4A-4C Example inpainting masks and example mask crops containing example mask regions are shown. As Figure 4A and Figure 4B shown, a mask region can be a simple and thin shape (in black color), so the associated specified mask crop will be a rectangle (the rectangle marked in red (dark gray in grayscale)). As Figure 4CAs shown, the mask regions can have more complex and wider shapes, then the associated specified mask crop will be a square.

[0048] Preferably, in an embodiment, the step of selecting at least one image crop and at least one mask crop further comprises a step of mask region aggregation. When the inpainting mask is composed of many mask regions, the method enters the step of mask region aggregation. First, each mask region is enclosed by a minimal bounding box (which can be rotated with respect to the image principal axis if needed). Then, the available crop sizes are verified. If possible, a single mask crop is always preferred. If the mask region cannot be enclosed by a single crop, then close mask regions are combined together. The aggregation algorithm minimizes the number of mask crops needed to cover all the mask regions. Thus, multiple mask / image crops can be further passed to the machine learning inpainting model (see example in Figure 5 ).

[0049] Depending on the hardware capabilities, the selected mask / image crops can be further downscaled to reduce the computation, for example, a 2048x2048 crop is downscaled to a 1024x1024 or 512x512 model input (depending on the parameters). The image crops can be downscaled using the Bicubic method, or the mask crops can be downscaled using the nearest neighbor method (which is suitable for binary images). In summary, in the step of selecting at least one image crop and at least one mask crop, it is decided which kind of crop should be selected. In particular, the parameters of the image crops and the mask crops can be adjusted according to the hardware limitations. Preferably, for example, knowing that the hardware is a low-end smartphone with limited CPU / memory, and that an initial high-resolution image of about 20% is selected for removal, we will use a 1024x1024 crop and downscale it to a resolution of 256x256. Knowing that the hardware is a high-end smartphone with a powerful CPU, the same case can be handled with a 1024x1024 crop without downsizing. The result of this step is a list of image crop and mask crop pairs as further machine learning model input.

[0050] The image preprocessing procedure based on the input image and the input mask allows the use of powerful machine learning models in a high-resolution inpainting pipeline on resource-limited devices. The method described in the present invention allows the automatic extraction of smaller parts, i.e., crops of the high-resolution image that can be processed by the machine learning model at lower resolution on mobile devices.

[0051] The method then enters a step of prediction by a machine learning model. At this point, a previously trained inpainting model is used to process the at least one image crop and the at least one mask crop to receive an inpainted image crop. Typically, a Convolutional Neural Network (CNN) is used. Training a CNN typically involves applying an input cycle to the input layer of the CNN and comparing the desired output to the actual output of the output layer of the CNN to compute an error term. The error term is used to adjust the weights and biases in the hidden layers of the CNN so that the next time the output values are closer to the correct values.

[0052] The inpainting machine learning model of the present invention takes as input at least one image crop of the input image and at least one corresponding mask crop of the inpainting mask (indicating which parts of the input should be removed and reconstructed). The machine learning model outputs an inpainted image crop, where the missing / removed parts have been reconstructed. The model is trained to provide high quality inpainting. By high quality inpainting, it is understood that the reconstructed pixels should both semantically and visually fit the surrounding parts of the input image. For example, given a picture of a dog on a grassy field, when we remove the dog from the input image, the reconstructed pixels should represent grass (semantically fit) and they should smoothly fill the removed area in the input image, making it indistinguishable from the surrounding area (visually fit). These objectives can be measured during training and can be used to guide the training process.

[0053] For example, the above objectives are measured in the machine learning process as follows: semantic similarity between the reconstructed area and its neighborhood (based on semantic features extracted from a pre-trained CNN), perceptual and style similarity (based on the LI distance between the input image and the reconstructed image, computed in the semantic feature space), smoothness (based on total variation). These values can be computed relatively easily for the reconstructed image and can be used as measures of the quality of the inpainting (it was observed that high values of these measures correlate with the subjective visual assessment of the inpainting result). The same measures can be used to determine the method used for the inpainting: if the values of two methods are similar on several measures, it is likely that very similar methods were used for the inpainting.

[0054] The inpainting machine learning model is run at a predefined resolution and within a predefined set of input dimensions. Typically the model can support the following inputs: 512x512, 256x1024, 256x768. The one or more image crops and mask crops selected in the previous step need to be adjusted to one of the input shapes supported by the model, for example a 1024x4096 crop needs to be downscaled to 256x1024 resolution before it can be input to the model. In a preferred embodiment, the resolution of the inpainting machine learning model can be adjusted according to hardware limitations. For example, the model can be replaced by a model that supports 256x256, 128x512 and 128x384 inputs to reduce the computational cost. The output of the inpainting machine learning model, i.e. the inpainted image crops, always have the same resolution as the input. For example, given a 512x512 input image crop and a corresponding 512x512 mask crop, the model returns a 512x512 inpainted image crop.

[0055] Preferably, the method of the present invention then enters a step of post-processing of the model output, i.e. post-processing of the inpainted sub-regions of the input image. In this step, the machine learning model output is processed to match the original input image, e.g. to restore scaling, rotation, etc. Regarding restore scaling, if the machine learning model is run at 512x512 resolution and the original image crop was at 1024x1024 resolution, a 2x bicubic upscaling is performed in the step of restore scaling.

[0056] Finally, the method of the present invention enters a step of providing an output inpainted image. This step comprises blending the at least one reconstructed sub-region, i.e. the at least one inpainted image crop, into the original image, i.e. blending the at least one inpainted image crop into the high resolution input image using the original input mask, in order to produce a final output image having the same resolution as the input image. Optionally, the final blending operation incorporates additional post-processing transformations specific to the selected reconstructed regions. For example, if the image / mask crops were scaled down and up during processing, an additional sharpening filter is applied in the step of filtering. For example, a known algorithm such as a de-sharpening mask can be used in the step of filtering.

[0057] In summary, all pre-processing steps and post-processing steps are parameterized and can be adjusted to match the hardware capabilities of the target platform. In particular, it is allowed to process high resolution images on mobile devices. Now referring to Figure 6 The image inpainting pipeline of the present invention will be described. The pipeline can be part of any device capable of storing, reading and displaying images. The device can be a mobile phone or a tablet or any device with a mobile platform. Furthermore, the pipeline can also be implemented in a smart TV.

[0058] The inpainting pipeline comprises an input image acquisition and analysis module 10. This module is responsible for reading an image from memory (not shown) to acquire the parameters of the image after analyzing it; optionally, it sends the image to a display module (not shown). The module outputs the following parameters: image width, image height, image format, etc.

[0059] The inpainting pipeline within the device with processing system of the present invention further comprises a mask provision module 20. This module is used to acquire at least one selected inpainting mask. The inpainting pipeline further comprises an automatic mask proposal module 21, a user-defined inpainting mask proposal module 22 and a user-defined mask refinement module 23. Optionally, the automatic mask proposal module 21 is used to generate mask proposals from the input image to improve user experience. The user-defined inpainting mask proposal module 22 is used to provide a manual mask by the user through a user interface. In practical use, the user-defined inpainting mask proposal module can be a software tool that uses a brush tool or an eraser tool, etc. to manually identify pixel data to be removed in the image. The user-defined mask refinement module 23 is used to manually correct the mask proposed by the automatic mask proposal module. In practical use, the user-defined mask refinement module can be a software tool that uses a brush tool or an eraser tool, etc. to manually modify the pixel data of the proposed mask. Each of the modules outputs a binary inpainting mask and parameters such as mask width and height (usually the same as the input image size); optionally, it also outputs the number of mask regions (this can be output by the mask provision module 20 or by the input image and selected inpainting mask preprocessing module 30 in later processing).

[0060] In particular, the automatic mask proposal module 21, as part of the present invention, allows the user to achieve a more attractive user experience for inpainting on a mobile device. The automatic mask proposal module can be based on existing image segmentation techniques (a number of machine learning-based models have been tested for this purpose). The segmentation models are fine-tuned to generate high-quality inpainting masks for objects that are usually to be removed (e.g. people, cars, garbage, etc.) and carefully post-processed to meet the requirements of the inpainting model (including smoothing and coherence improvement).

[0061] The automatic mask proposal module 21 can enhance the UI experience and together with the user-defined mask refinement module 23 can produce an automatic and attractive inpainting result (see again, for example, Figure 2 ).

[0062] The inpainting pipeline within the apparatus of the present invention further comprises an input image and selected inpainting mask preprocessing module 30. The input image and selected inpainting mask preprocessing module 30 is used to select the best image crop (and corresponding mask crop) and generate the input for the machine learning model. Preferably, the input image and selected inpainting mask preprocessing module 30 comprises a mask region detection module 31. The mask region detection module 31 can use the information about the mask regions passed from the mask provision module 20 (if available) to optimize the computation. The input image and selected inpainting mask preprocessing module 30 comprises an image / mask crop selection module 32. The image / mask crop selection module decides which image / mask crop is the best based on different computations, in particular based on the information gain of the computation. The image / mask crop selection module 32 takes as input the input image, the corresponding inpainting mask and a list of different mask regions. The image / mask crop selection module considers various shapes and sizes of crops, starting from the crop with the smallest possible size and no further preprocessing needed. Preferably, the input image and selected inpainting mask preprocessing module 30 further comprises an image / mask crop rotation module 33 with respect to the possible image / mask crop preprocessing process. The image / mask crop rotation module 33 is activated when the image / mask crop selection module 32 outputs a rotated crop: a non-rotated rectangular crop needs to be provided for the machine learning model. Thus, the image / mask crop rotation module 33 performs a simple rotation operation to obtain a non-rotated rectangular crop as input for the machine learning model. Preferably, in another embodiment, the input image and selected inpainting mask preprocessing module 30 comprises a downscaling module (image / mask crop downscaling module 34) for further adjusting the parameters of the image crop and the mask crop to meet the hardware limitations. The input image and selected inpainting mask preprocessing module 30 outputs a list of image crops and corresponding mask crops, all adjusted to the dimensions and shapes supported by the machine learning model.

[0063] The inpainting pipeline within the apparatus of the present invention further comprises a machine learning model 40. The present invention provides an end-to-end artificial intelligence (AI) approach to solve the inpainting problem. The machine learning model is a deep neural network trained using millions of real photos and millions of randomly generated masks that remove different parts of these photos. During training, the model needs to reproduce the missing parts of the original image. After training, a machine learning model is obtained that can look at any input image with missing parts and try to "imagine" the missing parts based on the non-missing areas.

[0064] The machine learning model described in the present invention is itself run on a predefined set of low resolutions (e.g. 256x256, 512x512). However, it works in a high resolution image inpainting pipeline: a high resolution input image is generated, processed and down-scaled to one of the input sizes supported by the model, and then the output low resolution of the model is processed and up-scaled again to the original high resolution. The parameters of the model (e.g. supported input resolutions) and the parameters of the processing pipeline can be adjusted according to the available hardware.

[0065] As previously mentioned, the image inpainting pipeline within the device described in the present invention also comprises a model output post-processing module 50. The model output post-processing module 50 works in exactly the reverse direction of the input image and selected inpainting mask pre-processing module 30. The model output post-processing module is used to fit / match the output of the machine learning to the original input high resolution image, including up-scaling and rotation (if needed) of the reconstructed output of the machine learning model. Finally, the image inpainting pipeline comprises a repaired high resolution image generation module 60.

[0066] As previously mentioned, the entire high resolution image inpainting pipeline can be deployed on a mobile device (e.g. on a mid-end smartphone) and repair a 4K resolution image in a single second, obtaining visually credible results. The image inpainting pipeline allows to achieve a high quality inpainting functionality on a mobile device. The inpainting functionality supports inpainting on high resolution images (up to 4K), achieves competitive visual quality results with a powerful machine learning model and runs directly on a mobile computing platform (without the need of any external service). Figure 7 An exemplary screenshot of an exemplary inpainting application running on a mobile device is shown.

[0067] Aspects of the present invention can be implemented by a computer having a system or device (or an apparatus such as a central processing unit (CPU) or a micro processing unit (MPU)) that can read and execute a computer program product recorded on a storage device to implement the functions described in the above embodiments, and by a method whose steps are executed by a computer having a system or device by reading and executing a program recorded on a storage device to implement the functions described in the above embodiments. For this purpose, the claimed computer program product is provided to the computer through, for example, a network or various types of recording media (as storage devices). The computer program product described in the present invention also includes a non-transitory machine-readable medium.

[0068] It should be understood that the application is not limited to the embodiments described above. Improvements or changes can be made by those of ordinary skill in the art based on the above description, all of which should fall within the scope of the appended claims of the application.

Claims

1. A method of repairing a high resolution image implemented on a computer, characterized in that, comprising: providing a high resolution input image; providing at least one inpainting mask, the inpainting mask indicating at least one set of pixels to be removed and synthesized; selecting at least one rectangular sub-region of the input image and at least one aligned rectangular sub-region of the inpainting mask, such that the rectangular sub-region of the input image contains at least one set of pixels to be removed and synthesized, the at least one rectangular sub-region of the input image and the corresponding aligned rectangular sub-region of the inpainting mask have the same minimum size, the at least one rectangular sub-region of the input image and the corresponding aligned rectangular sub-region of the inpainting mask do not decrease the information gain computed at a position; processing the rectangular sub-region of the input image and the corresponding aligned rectangular sub-region of the inpainting mask with a machine learning model, in the case that a repaired sub-region of the input image and at least one set of synthesized pixels are received, the at least one set of synthesized pixels are semantically and visually adapted to the surrounding part of the input image at the same time; the providing the inpainting mask comprises automatically proposing an inpainting mask; the step of automatically proposing an inpainting mask comprises a step of analyzing the input image; the input image is analyzed using a semantic image segmentation model to detect objects, for generating a high quality inpainting mask of typical removed objects; post-processing the inpainting mask to meet the requirements of the inpainting model after automatic mask post-processing; the requirements of the inpainting model refer to the inpainting mask should be similar to the masks used during the training of the machine learning model in terms of smoothness and coherence; the automatic mask post-processing comprises morphological erosion and dilation, blurring and thresholding operations; the post-processing of the inpainting mask further comprises user-defined manual mask refinement; generating a high resolution output image, the output image comprising the repaired sub-region, the repaired sub-region having at least one set of synthesized pixels, the at least one set of synthesized pixels being semantically and visually adapted to the surrounding part of the input image at the same time; the method further comprises: downscaling at least one selected image rectangular sub-region and at least one selected mask rectangular sub-region to adjust the resolution to belong to a set of predefined resolutions supported by the machine learning model.

2. The method of claim 1, wherein, the selecting at least one rectangular sub-region of the input image and at least one aligned rectangular sub-region of the inpainting mask comprises: acquiring information about mask regions within the selected inpainting mask.

3. The method of claim 2, wherein, the selecting at least one rectangular sub-region of the input image and at least one aligned rectangular sub-region of the inpainting mask further comprises: aggregating mask regions.

4. The method of claim 1, wherein, the method further comprises: rotating selected image rectangular sub-regions and selected mask rectangular sub-regions before being processed by the machine learning model.

5. The method of claim 4, wherein, the method further comprises: rotating the repaired sub-region output by the machine learning model.

6. The method of claim 5, wherein, the method further comprises: upscaling the inpainted sub-region output by the machine learning model.

7. The method of claim 1, wherein, the generating a high resolution output image comprises: blending the repaired sub-region into a high resolution input image using the provided inpainting mask.

8. A data processing system, characterized by comprising: a processor; and a memory coupled with the processor, the memory having stored instructions that the processor executes to perform operations comprising: providing a high resolution input image; providing at least one inpainting mask, the inpainting mask indicating at least one group of pixels to be removed and synthesized; selecting at least one rectangular sub-region of the input image and at least one aligned rectangular sub-region of the inpainting mask, such that the rectangular sub-region of the input image contains at least one group of pixels to be removed and synthesized, the at least one rectangular sub-region of the input image and the corresponding aligned rectangular sub-region of the inpainting mask have the same minimum size, the at least one rectangular sub-region of the input image and the corresponding aligned rectangular sub-region of the inpainting mask do not decrease the information gain computed at a position; processing the rectangular sub-region of the input image and the corresponding aligned rectangular sub-region of the inpainting mask using a machine learning model, in the case that a repaired sub-region of the input image and at least one group of synthesized pixels are received, the at least one group of synthesized pixels are semantically and visually adapted to the surrounding part of the input image at the same time; the providing the inpainting mask comprises automatically proposing an inpainting mask; the step of automatically proposing an inpainting mask comprises a step of analyzing the input image; the input image is analyzed using a semantic image segmentation model to detect objects, for generating a high quality inpainting mask of typical removed objects; after automatic mask post-processing, the inpainting mask is post-processed to meet the requirements of the inpainting model; the requirements of the inpainting model mean that the inpainting mask should be similar to the masks used during the training of the machine learning model in terms of smoothness and coherence; the automatic mask post-processing comprises morphological erosion and dilation, blurring and thresholding operations; the post-processing of the inpainting mask further comprises user-defined manual mask refinement; generating a high resolution output image, the output image comprising the repaired sub-region, the repaired sub-region having at least one group of synthesized pixels, the at least one group of synthesized pixels are semantically and visually adapted to the surrounding part of the input image at the same time; the operations further comprise: downscaling at least one selected image rectangular sub-region and at least one selected mask rectangular sub-region to adjust the resolution to belong to a predefined set of resolutions supported by the machine learning model.

9. A mobile phone comprising the data processing system of claim 8.

10. A computer program product comprising instructions which, when executed by a processor, perform the method of any one of claims 1-7.

Images