Method and system for video restoration

Abstract

Presented herein are systems and methods for video restoration. In one or more embodiments, using a pre-trained image restoration model, live training data can be generated using optical flow-guided sampling. In one or more embodiments, the sampled patches are used to generate a training dataset, which is used to further train the image restoration model until a stopping condition is reached. The adaptively trained restoration model can be used to generate a corrected video in which a desired object or objects have been deleted and the corresponding deleted portions have been filled (or restored) to maintain the integrity of the image.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for computer learning that can provide improved computer performance, features, and uses. More particularly, the present disclosure relates to systems and methods for video image editing. Background Art

[0002] Video inpainting, also known as video patching, involves attempting to fill in missing regions of a given video sequence. The goal is to fill a portion of the video with spatially and temporally coherent content. It has many real-world applications, such as removing unwanted objects and restoring damaged video files. Removing relatively stationary objects, such as logos and text-based titles, is also an application that can benefit from video inpainting.

[0003] Over the past few years, the amount of video content generated and available has increased dramatically. A significant amount of video is posted online to video publishing sites or via video-sharing applications. Often, the publishing site, the video publisher, or both, will include content such as logos, images, URLs, identifiers, social media identifiers, and more. Removing this content is sometimes desirable, but difficult to achieve.

[0004] Like many other computer vision applications, deep learning-based video restoration methods have pushed their performance limits in recent years. Early models used three-dimensional (3D) convolutions to try to ensure spatiotemporal consistency, but their high memory requirements limited their application, especially for high-resolution videos.

[0005] Recognizing that backgrounds and most objects in video sequences typically have trackable motion, many recent video inpainting methods attempt to exploit the optical flow of the sequence. Some have proposed first synthesizing coherent optical flow across frames so that most regions can be propagated from known regions. Since not all missing regions can be filled in this way, others have introduced non-local flow connections to temporally distant frames to further improve inpainting performance.

[0006] Flow-guided video inpainting methods work well for removing foreground objects when they are in motion relative to the background. However, when it propagates based on optical flow from known regions, it may relocate unwanted patches leading to undesirable visual artifacts when the optical flow estimate is unreliable. This is more of a concern for objects such as logos and text because they are relatively stationary and located near boundaries or edges where the motion is less significant than the central region. For areas that cannot be recovered by propagation, image rendering methods based on generative adversarial networks (GANs) have been attempted to synthesize missing content with high spatial coherence. However, these image inpainting models are typically trained with still images, taking neither the known content of the video nor temporal coherence into account.

[0007] Therefore, better systems and methods are needed to detect, remove, and replace content in videos. Summary of the Invention

[0008] As noted above, there is a need for improved systems and methods for detecting, removing, and replacing content in videos.Presented herein are systems, methods, and computer-readable media comprising instructions for performing the following methods.

[0009] In one or more embodiments, a method includes: dividing an input video into a set of video sequences, the video sequence set including one or more video frames; generating one or more object masks representing one or more regions, the one or more regions including one or more objects to be deleted and repaired in the video sequence; generating optical flows for each video sequence; and for each video sequence in the video sequence set: using at least some of the optical flows, updating a pre-trained repair model using patch samples selected from the video sequence; and using the updated repair model to modify the video sequence to repair at least a portion of the video sequence.

[0010] In one or more embodiments, the step of updating the inpainting model using patch samples selected from the video sequence using at least some of the optical flows includes: defining one or more sampling regions in the video sequence using optical flows associated with one or more objects detected in the video sequence; sampling one or more patches in the one or more sampling regions; for each patch, generating a corresponding masked patch that represents a masked version of the patch; and updating the inpainting model using the masked patches as input to the inpainting model and using the patches as corresponding ground truth references.

[0011] In one or more embodiments, the step of defining one or more sampling regions in a video sequence using optical flow associated with one or more objects detected in the video sequence includes, for an object mask of the video sequence: calculating an average velocity function around the object mask using the optical flow from the sequence; selecting a video frame from the sequence; and estimating the sampling regions in the video frames of the video sequence using the average velocity function.

[0012] In one or more embodiments, the step of generating one or more object masks may include a first method, comprising: given a known template corresponding to an object to be located, applying template matching at different image scales to detect the position and scale of the object, and generating an object mask.

[0013] In one or more embodiments, the step of generating one or more object masks may include a second method, the second method including: applying an object detection model, wherein, for an object, the object detection model detects the object and outputs a bounding box surrounding the detected object; and forming the object mask using the bounding box. In one or more embodiments, the second method may further include: using foreground segmentation on at least a portion of the video frame corresponding to the bounding box to identify one or more foreground parts; and forming the object mask by refining the bounding box using at least a portion of the identified one or more foreground parts to reduce the area of ​​the object mask to more closely match the object.

[0014] In one or more embodiments, the step of generating one or more object masks may include a third method comprising applying a joint detection and segmentation model to generate the object masks, wherein the joint detection and segmentation model has been pre-trained using one or more training datasets.

[0015] In one or more embodiments, the step of generating one or more object masks may include performing one or more of the first, second, and third methods.

[0016] In one or more embodiments, the step of generating one or more object masks further comprises: applying one or more filters to the detected objects to select a subset of the one or more detected objects; and generating an object mask for each detected object in the subset of the one or more detected objects. In one or more embodiments, the one or more filters include one or more of the following: one or more location filters to exclude one or more objects detected in a specific area of ​​the video frame; one or more size filters to exclude objects of one or more specific sizes; one or more text recognition filters to exclude objects associated with specific text; and one or more template matching filters to exclude objects of a specific configuration.

[0017] In one or more embodiments, a final revised version of the input video may be output including a revised video sequence, wherein the video sequence is combined into the final revised version of the video.

[0018] Using the video restoration with adaptive learning disclosed in the present invention, artifacts in images are better avoided.

[0019] Certain features and advantages of embodiments of the present invention have been generally described in this summary; however, additional features, advantages, and embodiments are set forth herein or will be apparent to one of ordinary skill in the art in view of the drawings, description, and claims. Therefore, it should be understood that the scope of the present invention is not limited by the specific embodiments described in this summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Reference will be made to embodiments of the present disclosure, examples of which may be shown in the accompanying drawings. These figures are intended to be illustrative and not limiting. Although the present disclosure has been generally described in the context of these embodiments, it should be understood that they are not intended to limit the scope of the present disclosure to these specific embodiments. Items in the drawings may not be drawn to scale.

[0021] Figure 1 A system of end-to-end systems for video inpainting according to embodiments of the present disclosure is depicted.

[0022] Figure 2 An end-to-end overview process for video inpainting of a video according to an embodiment of the present disclosure is depicted.

[0023] Figure 3 An alternative approach to object removal and inpainting with flow-guided adaptive learning for video scenes is depicted in accordance with an embodiment of the present disclosure.

[0024] Figure 4 Yet another alternative approach to object deletion and repair through flow-guided adaptive learning using templates according to an embodiment of the present disclosure is depicted.

[0025] Figure 5 Various methods are described that may be used to generate object masks according to embodiments of the present disclosure.

[0026] Figure 6 A filtering method according to an embodiment of the present disclosure is described.

[0027] Figure 7 A method for sampling a training patch according to an embodiment of the present disclosure is described.

[0028] Figure 8 Depicted are an original image 805 from an input video and a corresponding inpainted output image 810 , in accordance with an embodiment of the present disclosure.

[0029] Figure 9 Depicted is a simplified block diagram of a computing device / information processing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In the following description, for the purpose of explanation, specific details are set forth to provide an understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these details. Furthermore, those skilled in the art will recognize that the embodiments of the present disclosure described below can be implemented in a variety of ways, such as as a process, apparatus, system, device, or method on a tangible computer-readable medium.

[0031] The components or modules shown in the figures are illustrative of exemplary embodiments of the present disclosure and are intended to avoid obscuring the present disclosure. It should also be understood that throughout this discussion, components may be described as separate functional units, which may include subunits, but those skilled in the art will recognize that various components or portions thereof may be divided into separate components or may be integrated together, for example, included in a single system or component. It should be noted that the functions or operations discussed herein may be implemented as components. Components may be implemented in software, hardware, or a combination thereof.

[0032] In addition, the connections between the components or systems in the figure are not intended to be limited to direct connections. On the contrary, the data between these components can be modified, reformatted or otherwise changed by intermediate components. In addition, additional or fewer connections can be used. It should also be noted that the terms "coupling", "connection", "communication coupling", "interfacing", "interface" or any derivatives thereof should be understood to include direct connections, indirect connections through one or more intermediate devices, and wireless connections. It should also be noted that any communication, such as signals, responses, replies, confirmations, messages, queries, etc., may include one or more information exchanges.

[0033] References in the specification to "one or more embodiments," "preferred embodiment," "one embodiment," "an embodiment," etc., mean that a particular feature, structure, characteristic, or function described in connection with the embodiment is included in at least one embodiment of the present disclosure and may be included in more than one embodiment. Furthermore, appearances of the above phrases in different places in the specification are not necessarily all referring to the same embodiment.

[0034] Certain terms used in various places in the specification are for illustrative purposes and should not be construed as limiting. Services, functions, or resources are not limited to single services, functions, or resources; use of these terms may refer to groups of related services, functions, or resources, which may be distributed or aggregated. The terms "include" and "comprise" should be understood as open terms, and any lists below are examples and are not meant to be limited to the items listed. A "layer" may include one or more operations. The words "optimal," "optimize," "optimize," and the like refer to improvements in results or processes, and do not require that the specified results or processes have reached an "optimal" or peak state. The use of memory, database, repository, data store, table, hardware, cache, and the like may be used herein to refer to system components that can input or otherwise record information.

[0035] Those skilled in the art should recognize that: (1) certain steps may be performed selectively; (2) the steps may not be limited to the specific order described herein; (3) certain steps may be performed in a different order; and (4) certain steps may be performed simultaneously.

[0036] Any headings used herein are for organizational purposes only and should not be used to limit the scope of the specification or claims. Each reference / document mentioned in this patent document is incorporated herein by reference in its entirety.

[0037] It should be noted that any experiments and results provided herein are provided in an illustrative manner and were performed under specific conditions using one or more specific embodiments; therefore, these experiments and their results should not be used to limit the scope of the disclosure of this patent document.

[0038] It should also be noted that although the embodiments described herein may be within the context of removing logos, text, etc. from a video, aspects of the present disclosure are not limited thereto. Therefore, aspects of the present disclosure may be applied or adapted for use in other contexts.

[0039] A. General Introduction

[0040] As mentioned above, flow-guided video inpainting methods work well to remove foreground objects when they are in motion relative to the background, but as they propagate from known regions based on optical flow, they may relocate unwanted patches that lead to undesirable visual artifacts when the optical flow estimate is unreliable. This is more of a concern for objects such as signs and text, as they are relatively stationary and located near boundaries or edges where the motion is less significant than in the central region. For areas that cannot be recovered by propagation, image inpainting methods based on generative adversarial networks (GANs) have been attempted to synthesize missing content with high spatial coherence. However, these imaging inpainting models are typically trained using still images and do not take into account the known content or temporal coherence of the video.

[0041] To address these practical problems, this paper proposes embodiments of an end-to-end solution for object removal (e.g., logos, text, or both) using temporal adaptive learning. Embodiments can be built on top of a pre-trained image inpainting model and can use optical flow-guided sampling to select live training data to update the inpainting model. In one or more embodiments, the sampled patches are used to update the training of the image inpainting model until a stopping condition is reached.

[0042] The embodiments present a number of novel contributions, including but not limited to the following. First, the embodiments present an end-to-end solution to remove objects such as logos and text from video sequences and repopulate them with spatially and temporally coherent content. Second, live training patch sampling can be guided by the optical flow of the video sequence. Third, the embodiments employ adaptive learning using live training samples with a configurable convergence tolerance.

[0043] B. System and Method Embodiments

[0044] 1. General Overview

[0045] Figure 1 A system for an end-to-end system for video inpainting according to an embodiment of the present disclosure is depicted. In one or more embodiments, the system includes a scene detection module 110, an optical flow module 115, an object mask module 120, a patch sampling module 125, an inpainting model 140, and an adaptive learning module 135. In one or more embodiments, the system also includes a block 130 representing a process of applying the inpainting model 140 to image frames of a video after adaptive learning.

[0046] like Figure 1 As shown, the system receives an input image 105, which is processed through the system 100 to receive an output video 145, wherein one or more portions of the video have been inpainted to remove a desired element or elements.

[0047] Figure 1 Each component or process is explained in more detail below.

[0048] about Figure 1 , Figure 2 Depicted is an end-to-end overview process for video restoration according to an embodiment of the present disclosure. It should be noted that Figure 2 The method may be repeated for all video scenes or sequences of the input video.

[0049] In one or more embodiments, the first step is to divide (205) the input video V 105 into individual scenes (also referred to as sequences, clips, or segments) V i , where i belongs to the range of scenes as determined by the scene detection process or module 110. In one or more embodiments, scene detection can be performed by any existing scene detection system or method. An example of a module or tool that can be used is PySceneDetect, which is an open source application and Python library for detecting scene changes in videos and for automatically segmenting videos into separate sequences. It should be noted that other scene detection systems or methods can also be used.

[0050] In one or more embodiments, another step is to generate (210) one or more object masks for each scene, the one or more object masks representing one or more regions of target objects, such as logos and text, to be deleted and refilled. For the mask M of the entire video, it may include the mask M corresponding to the scene V. i The corresponding series of masks M i , used for scene V i Each object mask M i May include one or more masks A collection of scenes where V iOne or more objects o are included that may be targets to be masked. As will be explained in more detail below, the object masking module may employ one or more modules or methods to detect and segment target objects in a video scene to form one or more masks.

[0051] In one or more embodiments, another process involves generating (215) an optical flow F, which may include an optical flow F corresponding to each scene. i A collection of .

[0052] Given a video sequence or scene, for each video scene, the corresponding object mask, the corresponding optical flow, and the pre-trained inpainting model can be used (220) to perform object removal and inpainting using a flow-guided adaptive learning method to output a restored video scene. For example, in one or more embodiments, given these inputs, the embodiment adaptively updates or trains 135 the inpainting model 140 using sample portions from the video using the patch sampling module 125, which can be pre-trained. Figure 1 As shown in , the final output is a video 145 with the restoration to remove the desired object.

[0053] Method 1 describes an example overview method for object removal and restoration using stream-guided adaptive learning according to an embodiment of the present disclosure. It should be noted that this process can be repeated for all scenes of the input video.

[0054]

[0055] Figure 3 An alternative approach to object removal and inpainting with flow-guided adaptive learning for a video scene is depicted in accordance with an embodiment of the present disclosure.

[0056] for Figure 3 The method described in

[0045] assumes that the original video has been divided into scenes / sequences and that for each scene there is a mask with one or more object masks and a corresponding optical flow. In one or more embodiments, for a given scene / sequence, an object mask is selected (305) from the masks. The optical flow associated with the object can be a function of the average velocity in the x and y directions around the object mask area, but other optical flow metrics can be used.

[0057] In one or more embodiments, based on the optical flow associated with an object in the scene, a set of patches P outside the object mask are sampled (310), and corresponding masking patches are generated (315). It represents the masked version of the patch. The patch set P and the corresponding masked patch set This forms a training dataset that can be used for supervised learning to update the pre-trained restoration model I(·).

[0058] In one or more embodiments, a masking patch may be used As input and using the corresponding patch (P) as a ground truth reference, an inpainting model is adaptively trained (320). In one or more embodiments, a learning rate for training the inpainting model can be adjusted (325), and the inpainting model can be updated (330) to inpaint the region corresponding to the object mask.

[0059] In one or more embodiments, one or more stopping conditions may be checked (340) to determine if the stopping conditions have been met. The stopping conditions may include: (1) a set number of iterations have been performed; (2) an amount of processing time has been reached; (3) convergence (e.g., the difference between the repaired region of the video scene from the previous iteration and the corresponding repaired region from the current iteration is less than a threshold - see, for example, equation (5) below); (4) divergence (e.g., performance deteriorates); (5) an acceptable result has been achieved.

[0060] In response to not meeting the stop condition, the process may return (345) to step 310, where the repair model I(·) is further refined.

[0061] In response to the stop condition being met, the process can then check whether the mask has another object mask for this scene. If there is another object mask, the process can be repeated by selecting the next object mask and returning to step 305. If there is no other object mask for this scene, the process ends for this scene (355). It should be noted that this method can be repeated for another scene from the input video and can continue until all scenes of the input video have been processed. Once all scenes have been processed, they can be combined and the final repaired video can be output.

[0062] Figure 4 Another alternative method for object removal and inpainting using stream-guided adaptive learning using templates according to an embodiment of the present disclosure is depicted.In one or more embodiments, if more than one input video needs to be processed, an input video is selected (405). Figure 4 The method involves using a template of one or more objects to be removed. For example, if the object to be removed is a known sign, the sign can be used to generate (410) a template. As explained in more detail in the next section, in one or more embodiments, given a known template of one or more objects (e.g., a template of a sign, text, an item, etc.), a template matching function is applied to different scaled versions of the template relative to a video frame of the scene to generate one or more masks of the scene.

[0063] If not already done, in one or more embodiments, video scenes are detected to segment the selected video into different scenes.The scenes can then be selected (420) for object removal and repair according to the following steps.

[0064] In one or more embodiments, an optical flow F is generated (425) for the selected scene. i Using the flow-guided patch samples, the inpainting model I(·) (which may be pre-trained) is trained (430) until a stopping condition is reached. Embodiments of how to sample flow-guided patches are discussed in more detail below. In one or more embodiments, the adaptively trained inpainting model is used (435) to correct the selected scene, i.e., V i =I(V i , M), where V i is the scene and M stands for mask.

[0065] The process can be repeated for another scene (440).Once all scenes have been processed, they can be combined and the final repaired video can be output (445).

[0066] Note also that if more than one video is to be processed, the method can be repeated (450) for another input video. Otherwise, the process can terminate (455).

[0067] 2. Object Masking Example

[0068] As mentioned above, part of this process is generating an object mask. In one or more embodiments, the object mask can be generated in one or more ways. Figure 5 Various methods are described that may be used to generate object masks according to embodiments of the present disclosure.

[0069] As a preliminary matter, a mask may be generated from one or more images. For example, in one or more embodiments, a single video image may be used to represent a video scene (e.g., a still frame selected from the video scene may be used, or an average image of frames selected from the video scene may be used). Given one or more video images, a mask may be generated (505) by applying one of the following processes:

[0070] In one or more embodiments, given known templates of one or more objects (e.g., templates of a logo, text, item, etc.), a template matching function is applied to differently scaled versions of the template relative to a video image of a scene to generate a mask of the scene. For example, the template of the object can be scaled to different sizes, and the differently scaled templates can be systematically compared (e.g., in a sliding manner) across one or more video frames of the video scene to find a matching position and scale for the template for each frame.

[0071] In one or more embodiments, one or more object detection models (e.g., one or more models of one or more signs, text, and / or items) are applied (515) to the video image. The output of the one or more models includes one or more bounding boxes of the detected objects, which can be used as masks.

[0072] In one or more embodiments, one or more object detection models of step 515 may be used, but additionally, a foreground segmentation process may be applied to at least the bounding box portion of the video frame to refine (520) the mask from the bounding box to the neighborhood of the object boundary. An example segmentation model is DeepLab, which is a semantic segmentation architecture that, once pre-trained / trained, assigns each pixel of an image to foreground or background. It is noted that there are readily applicable existing models that do not require training; therefore, subsequent segmentation can be applied directly to the region within the bounding box. It is also noted that other segmentation models may be employed.

[0073] In one or more embodiments, a (525) joint detection and segmentation model can be used to directly generate a mask that closely approximates the object boundary. For example, in one or more embodiments, a training dataset includes image pairs, where one image in the image pair corresponds to a binary mask of the other image, which can be the original image, which can be used to train such a model. In this case, a pixel with a value of "1" associated with the pixel indicates the presence of a target object, while a pixel with a value of "0" associated with the pixel does not represent a target object in the image. One benefit of using a joint model is that the joint detection and segmentation model can effectively generate a target object mask as a single step. As described above, an example segmentation model is DeepLab, which is a semantic segmentation architecture that associates or assigns each pixel of an image to either foreground or background.

[0074] Such a training dataset may be obtained or may be generated. In one or more embodiments, the training dataset may be manually generated by drawing on the target object. Alternatively or additionally, mask images for the training pairs may be automatically generated prior to filtering by visual inspection. For example, in one or more embodiments, automatic generation of masks may be achieved using template matching, detection with subsequent foreground segmentation, or using a pre-trained joint detection and segmentation model. Alternatively or additionally, mask images for the training pairs may be automatically generated by superimposing a synthetic object (e.g., a logo, text, or other object) on an existing image such that the mask is mapped directly from the synthetic object.

[0075] It should be noted that the object masking module 120 may implement one or more of the methods presented herein for generating masks, including filtering, which is described in the next section.

[0076] 3. Filtration Example

[0077] In one or more embodiments, one or more filters may be applied as part of the mask generation of the previous section or as a subsequent process. Thus, when generating object masks for deletion and refilling / inpainting, the list of detected objects may be further filtered to select a subset of detected objects.

[0078] Figure 6 A filtering method according to an embodiment of the present disclosure is depicted. In one or more embodiments, given a list of detected objects for masking, one or more filters are applied (605) to filter the list of detected objects to select a subset to generate an object mask for further removal and refilling / inpainting. The filters may include one or more of the following (provided as examples and not limitations): one or more location filters to exclude one or more objects detected in a particular area of ​​a video frame; one or more size filters to exclude objects of one or more particular sizes (e.g., excluding objects exceeding a particular size limit); one or more text recognition filters to exclude objects associated with particular text; and one or more template matching filters to exclude certain objects. Once filtered, a final mask may be output (610) for object removal and refilling / inpainting.

[0079] 4. Flow-guided patch sampling example

[0080] An important part of an embodiment of the adaptive learning of the improved inpainting model I(·) is to refill the deleted object area with spatially and temporally coherent content. In one or more embodiments, selecting patches can involve multiple considerations. First, the sampled patches are preferably outside the object mask area. Second, if possible, the sampled patches in one video frame are preferably located in (or at least partially include) the object mask area in another video frame. In one or more embodiments, one method for improving accuracy is to use template matching to find the best match, and optical flow can be used as a guide to reduce the search area. However, such an embodiment can be time consuming and can also be problematic when the motion in the scene is too complex or too little. Therefore, presented below is an embodiment of an approximate and efficient sampling method that has near-optimal results and is more resilient when optical flow is unreliable. It should be noted that the patch sampling module 125 can implement one or more of the methods presented herein.

[0081] Figure 7 A method for sampling a training patch according to an embodiment of the present disclosure is described. In one or more embodiments, based on the optical flow F, for scene i, calculate (705) t∈(t1, t N ) is the average velocity function around the mask area during the duration of time, expressed as vx (t) and v y (t), where t1 is the timestamp of the start of scene i, t N is the timestamp of the end of scene i. In one or more embodiments, FlowNet, its variants, or another optical flow estimator may be used to generate optical flow information. Select (710) a video frame j in scene i, where t j is the timestamp of frame j in scene i. In one or more embodiments, video frame j may be randomly selected.

[0082] In one or more embodiments, a sampling region in the video frame is determined (715). In one or more embodiments, the sampling region is a rectangular region defined by coordinates (x1, x2, y1, y2), which can be determined using the following equation:

[0083]

[0084]

[0085]

[0086]

[0087] In one or more embodiments, given the size of the sampling region defined by the coordinates, if it is smaller than the object mask patch, the sampling region can be grown (720) to fit the object patch. For example, if the grown sampling region overlaps the mask region, the sampling region can be grown uniformly from the center and moved away. One skilled in the art will recognize a variety of ways to grow a sample—including, but not limited to, growing in certain directions, in certain ways, and / or measured from certain features.

[0088] Given a sampling region, a patch may be selected (e.g., randomly) from the sampling region. In one or more embodiments, one or more patches may be selected from the sampling region. Additionally or alternatively, in one or more embodiments, one or more patches may be sampled at a time and the process may loop between steps 710 and 725 until a set of patches is obtained.

[0089] After obtaining a set of patches, the set of patches can be used for adaptive learning of the repair model.

[0090] 5. Adaptive Learning Example

[0091] In one or more embodiments, the inpainting model I(·) can be any learning-based image inpainting model. For example, an inpainting model such as that described in Rui Xu et al., “Deep Flow-guided video inpainting,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3723-3732 (2019) can be used—although other inpainting models can be used.

[0092] In one or more embodiments, the inpainting model is pre-trained with a large dataset. In one or more embodiments, to further train the inpainting model I(·) to fit a certain video sequence V, a set of training patch pairs are sampled within V and masked accordingly (e.g., using masking patches and its corresponding patch (P) as the ground truth reference), as described herein. As shown in the above embodiment, I(·) can be trained with multiple iterations until convergence. It should be noted that the adaptive learning module 135 can implement one or more of the methods presented herein.

[0093] In one or more embodiments, the convergence criterion can be based on a loss function of these sampled patch pairs. In one or more embodiments, the quality of restoration of the refilled object mask region is the ultimate goal, but does not involve computing a loss function. Alternatively, the following practical metric can be calculated to evaluate the convergence of the adaptive training, where I l (·) is the newly trained repair model after iteration 1. When E l When the convergence tolerance ∈ is lower than a predetermined one, the training may be stopped. It should be noted that in one or more embodiments, ∈ may be adjusted to achieve different processing speeds.

[0094] E l =||I l (V, M)-I l-1 (V, M)|| (5)

[0095] In one or more embodiments, a final revised version of the input video including the revised video sequence can be output, wherein the video sequence is combined into the final revised version of the video. It should be noted that the video restoration processing module 130 can use the adaptively trained restoration model to restore the video scene to form the final output video.

[0096] C. Experimental Results

[0097] It should be noted that these experiments and results are provided by way of illustration and were performed under specific conditions using one or more specific embodiments; therefore, these experiments and their results should not be used to limit the scope of the disclosure of this patent document.

[0098] Figure 8 Depicted is an original image 805 from an input video and a corresponding inpainted output image 810, according to an embodiment of the present disclosure. Example 810 illustrates how an embodiment can help improve video quality by finding and removing unwanted objects from a video. Because the source video clips have logos of different sizes, shapes, styles, and / or qualities, they need to be removed and repopulated. Furthermore, once repopulated, the video may be imprinted with the new logo throughout the final video.

[0099] like Figure 8 As shown in , the logo is removed and refilled with natural-looking content. In contrast, the results of refilling using pre-trained I(·) without flow-guided adaptive learning show more obvious artifacts compared to the results with adaptive learning.

[0100] D. Computing System Embodiments

[0101] In one or more embodiments, aspects of this patent document may be directed to, may include, or may be implemented on one or more information processing systems (or computing systems). An information processing system / computing system may include any tool or collection of tools operable to calculate, predict, determine, classify, process, send, receive, retrieve, initiate, route, switch, store, display, communicate, indicate, detect, record, copy, process, or utilize any form of information, intelligence, or data. For example, a computing system may be or may include a personal computer (e.g., a laptop), a tablet computer, a mobile device (e.g., a personal digital assistant (PDA), a smartphone, a tablet phone, a tablet, etc.), a smartwatch, a server (e.g., a blade server or a rack server), a network storage device, a camera, or any other suitable device, and may vary in size, shape, performance, functionality, and price. A computing system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, read-only memory (ROM), and / or other types of memory. Additional components of a computing system may include one or more drives (e.g., a hard drive, a solid-state drive, or both), one or more network ports for communicating with external devices, and various input and output (I / O) devices, such as a keyboard, mouse, stylus, touch screen, and / or video display. A computing system may also include one or more buses operable to transmit communications between the various hardware components.

[0102] Figure 9A simplified block diagram of an information processing system (or computing system) according to an embodiment of the present disclosure is depicted. It will be understood that the functionality shown for system 900 is operable to support various embodiments of the computing system—although it should be understood that the computing system may be configured differently and include different components, including those having Figure 9 Fewer or more components than depicted in .

[0103] like Figure 9 As shown in , the computing system 900 includes one or more central processing units (CPUs) 901 that provide computing resources and control the computer. The CPU 901 can be implemented using a microprocessor or the like, and may also include one or more graphics processing units (GPUs) 902 and / or floating-point coprocessors for mathematical calculations. In one or more embodiments, one or more GPUs 902 may be incorporated into a display controller 909, such as part of one or more graphics cards. The system 900 may also include a system memory 919, which may include RAM, ROM, or both.

[0104] Multiple controllers and peripherals are also available, such as Figure 9As shown in . Input controller 903 represents an interface for various input devices 904, such as a keyboard, mouse, touch screen, and / or stylus. The computing system 900 may also include a storage controller 907 for interfacing with one or more storage devices 908, each storage device 908 including a storage medium such as a tape or disk, or an optical medium that can be used to record instruction programs for operating systems, utilities, and applications, which may include embodiments of programs that implement various aspects of the present disclosure. According to the present disclosure, storage device 908 may also be used to store processed data or data to be processed. The system 900 may also include a display controller 909 for providing an interface to a display device 911, which may be a cathode ray tube (CRT) display, a thin film transistor (TFT) display, an organic light emitting diode, an electroluminescent panel, a plasma panel, or any other type of display. The computing system 900 may also include one or more peripheral controllers or interfaces 905 for one or more peripheral devices 906. Examples of peripheral devices may include one or more printers, scanners, input devices, output devices, sensors, etc. The communication controller 914 can interface with one or more communication devices 915, which enable the system 900 to connect to remote devices via any network including the Internet, cloud resources (e.g., Ethernet cloud, Fibre Channel over Ethernet (FCoE) / Data Center Bridging (DCB) cloud, etc.), a local area network (LAN), a wide area network (WAN), a storage area network (SAN), or via any suitable electromagnetic carrier signal including infrared signals. As shown in the depicted embodiment, the computing system 900 includes one or more fans or fan trays 918 and a cooling subsystem controller or controllers 917 that monitor the thermal temperature of the system 900 (or its components) and operate the fans / fan trays 918 to help regulate the temperature.

[0105] In the illustrated system, all major system components can be connected to bus 916, which can represent more than one physical bus. However, various system components may or may not be physically close to each other. For example, input data and / or output data can be remotely transmitted from one physical location to another. In addition, programs that implement various aspects of the present disclosure can be accessed from a remote location (e.g., a server) over a network. Such data and / or programs can be transmitted via any of a variety of machine-readable media, including, for example: magnetic media such as hard disks, floppy disks, and tapes; optical media such as compact disks (CDs) and holographic devices; magneto-optical media; and hardware devices specifically configured to store or store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, other non-volatile memory (NVM) devices (such as 3D XPoint-based devices), and ROM and RAM devices.

[0106] Aspects of the present disclosure may be encoded on one or more non-transitory computer-readable media having instructions for one or more processors or processing units to cause the steps to be performed. It should be noted that the one or more non-transitory computer-readable media should include volatile and / or non-volatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. The functions of the hardware implementations can be implemented using ASICs, programmable arrays, digital signal processing circuits, etc. Therefore, the "means" term in any claim is intended to cover both software and hardware implementations. Similarly, the term "computer-readable medium or medium" as used herein includes software and / or hardware having an instruction program thereon, or a combination thereof. In view of these implementation alternatives, it should be understood that the drawings and accompanying descriptions provide the functional information required by those skilled in the art to write program code (i.e., software) and / or manufacture circuits (i.e., hardware) to perform the desired processing.

[0107] It should be noted that embodiments of the present disclosure may further relate to computer products having non-transitory, tangible computer-readable media having computer code thereon for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a kind known or available to those skilled in the relevant art. Examples of tangible computer-readable media include, for example: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CDs and holographic devices; magneto-optical media; and hardware devices specially configured to store or store and execute program code, such as ASICs, programmable logic devices (PLDs), flash memory devices, other non-volatile memory (NVM) devices (such as 3D XPoint-based devices), and ROM and RAM devices. Examples of computer code include machine code, such as generated by a compiler, and files containing higher-level code executed by a computer using an interpreter. Embodiments of the present disclosure may be implemented in whole or in part as machine-executable instructions, which may be in program modules executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules may be physically located locally, remotely, or both.

[0108] Those skilled in the art will recognize that no computing system or programming language is critical to the practice of the present disclosure. Those skilled in the art will also recognize that the above-mentioned multiple elements can be physically and / or functionally separated into modules and / or submodules or combined together.

[0109] Those skilled in the art will understand that the foregoing examples and embodiments are illustrative and do not limit the scope of the present disclosure. All permutations, enhancements, equivalents, combinations, and improvements apparent to those skilled in the art upon reading the specification and studying the drawings are intended to be included within the true spirit and scope of the present disclosure. It should also be noted that the elements of any claim may be arranged differently, including in multiple dependencies, configurations, and combinations.

Claims

1. A method for video restoration, comprising: Dividing an input video into a video sequence set, wherein the video sequence set includes one or more video frames; generating one or more object masks, the one or more object masks representing one or more regions including one or more objects to be deleted and inpainted in a video sequence; Generate optical flow for each video sequence; as well as For each video sequence in the video sequence set, the object mask is: computing an average velocity function around an object mask using optical flow from the sequence, selecting a video frame from the sequence, estimating one or more sampling regions in the video frame of the video sequence using the average velocity function, sampling one or more patches in the one or more sampling regions, for each patch, generating a corresponding masked patch representing a masked version of the patch, and updating a pre-trained inpainting model using the masked patches as input to the inpainting model and using the patches as corresponding ground truth references; as well as The video sequence is modified using the updated restoration model to restore at least a portion of the video sequence.

2. The method of claim 1 , wherein generating one or more object masks comprises: Given a known template corresponding to the object to be located, template matching is applied at different image scales to detect the location and scale of the object and generate an object mask.

3. The method of claim 1 , wherein generating one or more object masks comprises: Applying an object detection model, for an object, the object detection model detects the object and outputs a bounding box surrounding the detected object; as well as Use bounding boxes to form object masks.

4. The method of claim 1 , wherein generating one or more object masks comprises: A joint detection and segmentation model is applied to generate object masks, where the joint detection and segmentation model has been pre-trained using one or more training datasets.

5. The method according to claim 3, wherein given a known template corresponding to the object to be located, applying template matching at different image scales to detect the position and scale of the object and generating an object mask comprises: identifying one or more foreground portions using foreground segmentation on at least a portion of the video frame corresponding to the bounding box; as well as An object mask is formed by refining the bounding box using at least a portion of the identified one or more foreground portions to reduce an area of ​​the object mask to more closely match the object.

6. The method of claim 1 , wherein generating one or more object masks comprises: applying one or more filters to the detected objects to select a subset of the one or more detected objects; as well as An object mask is generated for each detected object in a subset of the one or more detected objects.

7. The method of claim 6, wherein the one or more filters include one or more of the following: one or more location filters for excluding one or more objects detected in specific areas of the video frame; One or more size filters to exclude objects of one or more specific sizes; One or more text recognition filters to exclude objects associated with specific text; and One or more template matching filters to exclude objects of a specific configuration.

8. The method according to claim 1, further comprising: The output comprises a final corrected version of the input video of the corrected video sequence.

9. A system for video restoration, comprising: one or more processors; as well as A non-transitory computer readable medium comprising one or more sets of instructions which, when executed by at least one of one or more processors, result in the execution of the steps of the method of any one of claims 1 to 8.

10. A non-transitory computer-readable medium comprising one or more sequences of instructions which, when executed by at least one processor, cause the at least one processor to perform the steps of the method of any one of claims 1 to 8.

11. A computer program product comprising a computer program, which, when executed by a processor, causes the processor to perform the steps of the method according to any one of claims 1 to 8.

Images