Loop guidance for mask-based video mask

Abstract

The invention relates to loop guidance for mask-based video masks. Embodiments of a digital design system for being trained to generate alpha mask frames of a video sequence using loop guidance of previous video frames are disclosed. The method may include receiving a video sequence and an input masked video frame for a first video frame of the video sequence. The disclosed systems and methods also include generating alpha mask frames for the video sequence using the video sequence and the input masked video frames, where the first network generates masked video frames based on stored features of previous frames of the video sequence, and the second network generates an alpha mask frame based on the masked video frame and a feature of a previous frame of the stored video sequence. Using the generated alpha mask frame, an alpha mask video sequence representation of the video sequence may be output.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more particularly to loop booting for mask-based video matting. Background Technology

[0002] Effective video editing is crucial for storytelling, marketing, and content creation. One aspect of video editing is creating alpha masks for objects of interest, which is useful for tasks such as background replacement and color adjustments. Because objects of interest can move continuously through each frame of a video sequence, creating alpha masks for the entire video sequence can be a challenging task. Summary of the Invention

[0003] This article introduces the technology / process that allows digital design systems to generate an alpha mask video sequence representation of an input video sequence given a single masked video frame as input.

[0004] More specifically, in one or more embodiments, the digital design system processes a sequence of video frames of a video sequence through a pipeline of machine learning models. The input to the digital design system is the video sequence and an input mask video frame of the first video frame of the video sequence. The input mask video frame may be a binary mask of the first frame, indicating whether each pixel is a foreground or background pixel. Each video frame of the video sequence is then processed first through a first encoder-decoder network of a video segmentation module, which is trained to generate a mask video frame representation of the video frame. The mask video frame representation of the video frame is then passed through a second encoder-decoder network of a video masking module, which is trained to generate an alpha mask frame representation of the video frame. The features of the alpha mask frames generated for each frame by the second encoder-decoder are stored in one or more memories. The features used to generate the mask video frames and alpha mask frames, respectively, by the first encoder-decoder network and the second encoder-decoder network are then supplemented with the stored features of the alpha mask frames of previous video frames of the same video sequence previously processed by the pipeline. After processing the video frames of a video sequence, the video frames can be combined to generate an alpha mask video sequence representation of the video sequence.

[0005] In one or more embodiments, additional features may be provided to further improve the alpha masking process performed by the video masking module. In such embodiments, multi-layer features generated by one or more layers of the encoder of the video segmentation module may be provided as additional input to the decoder of the video masking module.

[0006] Additional features and advantages of the exemplary embodiments of this disclosure will be set forth in the description which follows, and some features and advantages will be apparent from the description or may be learned by practicing these exemplary embodiments. Attached Figure Description

[0007] The specific implementation method is described with reference to the accompanying drawings, in which:

[0008] Figure 1 A diagram illustrating the process of generating an alpha mask video sequence representation of a video sequence using a machine learning model according to one or more embodiments;

[0009] Figure 2 A diagram illustrating the process of generating a masked video frame representation of a video frame using a video segmentation module according to one or more embodiments;

[0010] Figure 3 A diagram illustrating the process of generating an alpha mask representation of a video frame using a video masking module according to one or more embodiments;

[0011] Figure 4 A comparison of qualitative results generated by a digital design system according to one or more embodiments is shown;

[0012] Figure 5 A diagram illustrating the process of training a machine learning model according to one or more embodiments to generate masked video frames and alpha mask frames for a video sequence;

[0013] Figure 6 A schematic diagram of a digital design system according to one or more embodiments is shown;

[0014] Figure 7 A flowchart illustrating a series of actions in a method for generating an alpha mask video sequence representation of a video sequence using a machine learning model of a digital design system according to one or more embodiments; and

[0015] Figure 8 A block diagram of an exemplary computing device according to one or more embodiments is shown. Detailed Implementation

[0016] One or more embodiments of this disclosure include a digital design system with a neural network trained to generate an alpha mask video sequence representation of an input video sequence given a single masked video frame. Some prior techniques require a ternary graph prior, where the user needs to manually annotate the foreground, background, and unknown regions (e.g., pixels that blend foreground and background). Other techniques may use image segmentation methods to assign pixels as foreground or background, and then manually select pixels in unknown regions. However, both of these prior techniques can be difficult, time-consuming, and resource-intensive because they require the user to repeat the manual annotation process for each frame of the video sequence. These and other existing techniques that attempt to propagate a single ternary graph prior produce unsatisfactory results, which are problematic in terms of temporal coherence (e.g., flickering in the resulting output) and stability. Furthermore, as the length of the video sequence increases, the temporal coherence and stability problems only worsen.

[0017] To address these and other shortcomings in conventional systems, the digital design system of this disclosure includes a neural network trained to generate masked video frames and alpha mask frames representing video frames of a video sequence. This network simultaneously uses a circular memory, updating the circular memory with features of the previous video frames after each previous video frame has been processed through the pipeline. In embodiments, the cyclically guided masking process feeds back features of previously generated alpha masks (at least for the video frames of the video sequence) to a video segmentation module to produce a masked video frame (e.g., one created in the initial input masked video frame) that more accurately captures the current video frame of the object of interest. Similarly, the cyclically guided masking process feeds back features of previously generated alpha masks (at least for the video frames of the video sequence) to a video masking module to produce an alpha mask with greater temporal coherence.

[0018] The digital design system disclosed herein improves the alpha masking of input video sequences, overcoming the limitations of existing solutions. One advantage of the digital design system disclosed herein is the generation of more consistent alpha masks for multiple video frames of a video sequence, where the first frame of the video sequence requires only a single mask video frame as input. The digital design system disclosed herein also results in an enhanced user experience, as the user only needs to generate the initial input mask video frame, thus saving time and computational resources. Furthermore, even when the input video sequence is long (e.g., comprising a large number of video frames), the use of a loop-guided process to feed features from previous video frames when generating the current video frame produces improved masking results.

[0019] Figure 1 A diagram illustrating the process of generating an alpha mask video sequence representation of a video sequence using a machine learning model, according to one or more embodiments. Figure 1As shown, the digital design system 100 receives input 102, as indicated by digit 1. For example, the digital design system 100 receives input 102 from a user or from memory or a storage location via a computing device. In one or more embodiments, input 102 includes at least video sequence 106, a first video frame 107 of video sequence 106, and an input mask video frame 108 version of the first video frame 107. For example, input 102 may be a document or file including the video sequence. In one or more embodiments, input 102 may be provided in a graphical user interface (GUI). For example, a user may indicate a storage location (e.g., on a computing device) or a URL pointing to a location storing video sequence 106, the first video frame 107, and / or the input mask video frame 108.

[0020] Digital design system 100 includes an input analyzer 104 that receives input 102. In some embodiments, the input analyzer 104 is configured to extract video sequence 106, a first video frame 107, and an input mask video frame 108 at digit 2. In one or more embodiments, the first video frame 107 may be a first video frame of video sequence 106 in sequence or an intermediate frame of video sequence 106. In one or more embodiments, the input mask video frame 108 is a binary mask representation of the first video frame 107, wherein each pixel of the input mask video frame 108 indicates whether the corresponding pixel of the first video frame 107 is a foreground pixel or a background pixel. The mask in the input mask video frame 108 specifies the object of interest in video sequence 106. In one or more embodiments, the input includes video sequence 106 and input mask video frame 108, and the first video frame 107 is extracted from video sequence 106 based on information indicating video frames of video sequence 106 associated with input mask video frame 108. In one or more embodiments, the input analyzer 104 may segment video sequence 106 into multiple video frames.

[0021] In one or more embodiments, as shown at numeral 3, a first video frame 107 and an input mask video frame 108 are sent to the video segmentation module 110. In one or more embodiments, the video segmentation module 110 includes an encoder-decoder network 112, or a similar neural network, and a frame feature memory 116. The neural network may include a machine-learning model that can be tuned (e.g., trained) based on training inputs to approximate an unknown function. In particular, the neural network may include a model of interconnected digital neurons that communicate and learn to approximate complex functions, and generate outputs based on multiple inputs provided to the model. For example, the neural network includes one or more machine learning algorithms. In other words, a neural network is an algorithm that implements deep learning techniques, i.e., machine learning that uses a set of algorithms to attempt to model high-level abstractions in data.

[0022] In one or more embodiments, the frame feature memory 116 is configured to store alpha mask frames from previous video frames processed by the video segmentation module 110 and the video masking module 118. In other embodiments, the frame feature memory 116 is also configured to store masked video frames from previous video frames processed by the video segmentation module 110.

[0023] In one or more embodiments, as shown at numeral 4, the video segmentation module 110 is configured to generate a masked video frame 114 by passing a first video frame 107 and an input masked video frame 108 through an encoder-decoder network 112. The masked video frame 114 is a binary masked representation of the first video frame 107. (See reference...) Figure 2 Additional details are described regarding the process of generating masked video frame 114.

[0024] In one or more embodiments, as shown at numeral 5, masked video frame 114, or features of masked video frame 114, may optionally be stored in frame feature memory 116. In one or more embodiments, masked video frame 114, or features of masked video frame 114, may be stored in frame feature memory 116 to aid in the generation of masked video frames for subsequent video frames of video sequence 106. Alternatively, processing may be performed as shown at numeral 6, without storing masked video frame 114 or features of masked video frame 114 in frame feature memory 116.

[0025] As shown at digit 6, after the video segmentation module 110 generates the masked video frame 114, it can be sent to the video masking module 118. As shown at digit 7, the first video frame 107 can also be sent to the video masking module 118. At digit 8, the video masking module 118 is configured to generate an alpha mask frame 122 by passing the first video frame 107 and the masked video frame 114 through the encoder-decoder network 120. (Reference) Figure 3 Additional details describe the process of generating Alpha Mask Frame 122.

[0026] In one or more embodiments, as shown at numeral 9, the features of alpha mask frame 122 or alpha mask frame 122 are sent to frame feature memory 116 and mask memory 124 for storage. In one or more embodiments, the features of alpha mask frame 122 or alpha mask frame 122 may be stored in frame feature memory 116 and mask memory 124 to aid in the generation of alpha mask frames for subsequent video frames of video sequence 106.

[0027] Then, steps 3-9 can be iteratively repeated for each consecutive frame of video sequence 106 to generate the corresponding alpha mask frame. Features stored in frame feature memory 116 and mask memory 124 are updated after each pass through digital design system 100, providing cyclic guidance for processing subsequent video frames of the same video sequence. In one or more embodiments, steps 3-9 are iteratively repeated until all video frames of video sequence 106 or a specified subset of video frames have been processed. For each video frame of video sequence 106 after the first video frame 107, the input to video segmentation module 110 consists only of that video frame.

[0028] After processing the video frames of video sequence 106 to generate corresponding alpha mask frames, the alpha mask frames generated by digital design system 100 can be combined to generate an alpha mask video sequence representation of video sequence 106. In one or more embodiments, as shown at numeral 10, the alpha mask video sequence representation of video sequence 106 can be sent as output 130. In one or more embodiments, following the process described in numerals 1-9 above, output 130 is sent via a communication channel to a user device or computing device that has provided input requesting an alpha mask video sequence representation of video sequence 106, another computing device associated with a user or another user, or another system or application.

[0029] In one or more embodiments, as the image encoder of the encoder-decoder network 112 of the video segmentation module 110 compresses the first video frame 107 through multiple layers to extract features of the first video frame 107, finer details visible at large scale / resolution can subsequently be compressed at small scale / resolution, making them no longer visible. In some embodiments, to preserve these details, these multi-scale features generated by the image encoder of the encoder-decoder network 112 at each layer are extracted, and these multi-scale features are concatenated with the masked video frame 114. These multi-scale features can then be fed as additional features for generating the alpha mask frame 122 into the mask decoder of the encoder-decoder network 120 of the video masking module 118. In such embodiments, the masking process performed by the video masking module 118 can be informed by providing multi-scale features, thereby recovering details that may have been lost during the generation of the masked video frame 114.

[0030] Figure 2A diagram illustrates a process for generating a masked video frame representation of a video frame using a video segmentation module according to one or more embodiments. The video segmentation module 110 may include an encoder-decoder network 112 and a frame feature memory 116. In one or more embodiments, the encoder-decoder network 112 includes an image encoder 204, a feature refinement module 206, and a decoder 208. Figure 2 As shown, the video segmentation module 110 receives video frame 202. In one or more embodiments, video frame 202 can be received from a storage location. In one or more embodiments, video frame 202 is passed to encoder-decoder network 112. Video frame 202 can be a video frame from a plurality of video frames constituting a video sequence. The plurality of video frames can pass through encoder-decoder network 112 sequentially.

[0031] When video frame 202 is the first video frame of the video sequence, the input to the video segmentation module 110 also includes an input mask video frame 203 corresponding to the first video frame. In such an embodiment, the input mask video frame 203 is a mask video frame that indicates whether the corresponding pixel of the first video frame is a foreground pixel or a background pixel. The input mask video frame 203 can be sent to the frame feature memory 116 as initial memory data to guide the mask video frame generation process performed by the video segmentation module 110.

[0032] In one or more embodiments, after providing video frame 202 to image encoder 204, image encoder 204 generates features (e.g., feature vector representation) representing video frame 202. The generated features are then passed to feature refinement module 206. In one or more embodiments, feature refinement module 206 retrieves or receives feature data from one or more previous video frames in the same video sequence as video frame 202 from frame feature memory 116. Note that when video frame 202 is the first video frame of the video sequence, frame feature memory 116 may only include input mask video frames 203 associated with the first video frame of the video sequence. For subsequent video frames 202 after the first video frame, frame feature memory 116 is populated with data from alpha mask video frames generated previously by video segmentation module 110 and through mask video frames (e.g., video mask module 118). For example, if video frame 202 is the fifth video frame in a video sequence, the frame feature memory 116 may include feature data of the alpha mask frames preceding the fifth video frame generated by the video mask module 118.

[0033] In one or more embodiments, the feature data in the frame feature memory 116 may also include feature data of masked video frames generated from video frames previously processed by the video segmentation module 110.

[0034] In one or more embodiments, the feature refinement module 206 combines, concatenates, or otherwise applies stored features from the frame feature memory 116 to features representing video frame 202. In such embodiments, the output of the feature refinement module 206 is an enhanced feature (e.g., enhanced feature vector) representation of video frame 202. The enhanced features are then passed to decoder 208. In one or more embodiments, decoder 208 generates a masked video frame 210. The masked video frame 210 may be a binary mask representation of video frame 202, wherein the value assigned to each pixel indicates whether the corresponding pixel of video frame 202 is a foreground pixel or a background pixel.

[0035] Figure 3 A diagram illustrating the process of generating an alpha mask frame representation of a video frame using a video masking module according to one or more embodiments is shown. The video masking module 118 may include an encoder-decoder network 120, a memory encoder 310, and a mask memory 124. In one or more embodiments, the encoder-decoder network 120 includes a mask encoder 304 and a mask decoder 306. Figure 3 As shown, the video masking module 118 receives video frame 202 and is processed by the video segmentation module 110 (as shown in the figure). Figure 2 The masked video frame 210 of the generated video frame 202 (described) is a representation of the masked video frame 210. In one or more embodiments, when the masked video frame 210 is generated, it can be received from a storage location or directly from the video segmentation module 110. In one or more embodiments, the video frame 202 and the masked video frame 210 are passed to the encoder-decoder network 120. The masked video frame 210 can be a masked video frame among a plurality of masked video frames, wherein the plurality of video frames can pass through the encoder-decoder network 120 sequentially.

[0036] In one or more embodiments, mask encoder 304 generates features (e.g., feature vector representations) representing video frame 202 and masked video frame 210. These features are then passed to mask decoder 306. In one or more embodiments, mask decoder 306 generates an initial alpha mask frame 308. The initial alpha mask frame 308 can then be provided to memory encoder 310. Additionally, video frame 202 is also provided to memory encoder 310.

[0037] In one or more embodiments, when video frame 202 is a video frame following the first video frame, video frame 202 and masked video frame 210 are additionally passed through encoder-decoder network 120 to generate the final alpha mask frame 312 of video frame 202. During the second pass, mask memory 124 provides multi-layer features to mask decoder 306 to improve the output of video mask module 118. Providing multi-layer features can also reduce model redundancy (e.g., avoid features encoded by video segmentation module 110 being re-encoded by video mask module 118) and allow video mask module 118 to focus on encoding mask-specific features. In such embodiments, memory encoder 310 generates feature data (e.g., feature vectors) representing video frame 202, an initial alpha mask frame 308 generated for masked video frame 210, and an initial alpha mask frame 308 and a final alpha mask frame 312 generated for previous video frames in the video sequence. The feature data generated by memory encoder 310 is then provided to mask memory 124. For example, when the mask decoder 306 passes compressed features generated by the mask encoder 304 through layers of the mask decoder 306, the compressed features are upsampled. At one or more layers of the mask decoder 306, the mask memory 124 feeds the stored multi-layer features of the corresponding layer / level to the mask decoder 306 to improve the quality and resolution of the final alpha mask frame 312. In one or more embodiments, multi-layer features from the mask memory 124 may be added to or concatenated to the features upsampled by the mask decoder 306 to eliminate unwanted features in the background region of the video frame and supplement necessary features in the foreground region of the video frame.

[0038] Figure 4 A comparison of qualitative results generated by a digital design system according to one or more embodiments is shown. Figure 4 In this context, video frame 402 is a video frame from the video sequence. Fast Triple Graph Propagation-Video Masking (FTP-VM) uses a single ternary graph prior from the video frame as input, which is propagated to other frames to produce an alpha mask. For example... Figure 4 As shown, when the object of interest moves between frames, the generated FTP-VM output video frame 404 exhibits noticeable artifacts, resulting in a loss of coherence in the alpha mask generated by the FTP-VM. Furthermore, for subsequent video frames of the input video frame 402, the generated FTP-VM output video frame 404 shows an increased loss of coherence in the object of interest. Conversely, as described herein, the output video frame 406 generated by the digital design system using a loop of previous alpha mask frames produces an alpha mask with minimal or even no artifacts or loss of coherence in the object of interest.

[0039] Figure 5A diagram illustrating the process of training a machine learning model to generate masked video frames and alpha mask frames for a video sequence according to one or more embodiments is shown. In one or more embodiments, training system 500 is configured to train neural networks (e.g., encoder-decoder network 112 and encoder-decoder network 120) to generate masked video frames and alpha mask frames for a video sequence. In some embodiments, training system 500 is part of digital design system 100. In other embodiments, training system 500 may be a standalone system or may be part of another system and deployed within digital design system 100. For example, training system 500 may be implemented as a separate system on an electronic device separate from the electronic device implementing digital design system 100. Figure 5 As shown, the training system 500 receives training input 502. For example, the digital design system 100 receives training input 502 from a user via a computing device or from a memory or storage location. Training input 502 may include training video sequence frames 504 and corresponding training mask video frames 506, as well as training alpha mask frames 508.

[0040] As indicated by number 1, training video sequence frames 504 are sent to video segmentation module 110. In one or more embodiments, training video sequence frames 504 are sent to video segmentation module 110 serially or in parallel. (As previously mentioned...) Figure 2 As described, the video segmentation module 110 generates a masked video frame 510 for each video sequence frame of the training video sequence frame 504 at digit 2. Then, as shown at digit 3, the masked video frame 510 of each video sequence frame of the training video sequence frame 504 is sent to the loss function 512. As shown at digit 4, the training masked video frame 506 is also passed to the loss function 512. At digit 5, using the training masked video frame 506 and the masked video frame 510 of each video sequence frame of the training video sequence frame 504, the loss function 512 can calculate the loss. In one or more embodiments, a bootstrapped cross-entropy loss and a dice loss with equal weights are used. Then, as shown at digit 6, the calculated loss can be backpropagated to train the encoder-decoder network 112.

[0041] As shown at number 7, the masked video frame 510 for each video sequence frame of the training video sequence frame 504 generated by the video segmentation module 110 is also sent to the video masking module 118. (As previously mentioned...) Figure 3As described, at digit 8, the video masking module 118 generates an alpha mask frame 514 corresponding to each masked video frame 510. Then, as shown at digit 9, the alpha mask frame 514 for each masked video frame 510 is sent to the loss function 516. As shown at digit 10, the training alpha mask frame 508 is also passed to the loss function 516. At digit 11, using the training alpha mask frame 508 and the alpha mask frame 514 for each masked video frame in the masked video frame 510, the loss function 516 can calculate the loss. In one or more embodiments, the loss function 516 may include a regression loss (e.g., L1 or L2 loss between training alpha mask frames 508 and alpha mask frames 514 for each masked video frame of masked video frame 510), a synthesis loss (e.g., L1 or L2 loss between synthesized images calculated from training alpha mask frames 508 and alpha mask frames 514), and a Laplacian pyramid loss (e.g., calculated between training alpha mask frames 508 and alpha mask frames 514). In one or more embodiments, the synthesized image calculated using training alpha mask frames 508 and alpha mask frames 514 may be represented as follows:

[0042] C = aF + (1-a)B

[0043] C ′ =a ′ F+(1-a ′ B

[0044] Where a and a ′ These are training alpha mask frames 508 and 514, where F is the foreground image and B is the background image. Then, as shown at number 12, the calculated loss can be backpropagated to train the encoder-decoder network 120.

[0045] Figure 6 A schematic diagram of a digital design system (e.g., the "digital design system" described above) according to one or more embodiments is shown. As shown, the digital design system 600 may include, but is not limited to, a user interface manager 602, an input analyzer 604, a video segmentation module 606, a video masking module 608, a neural network manager 610, a training system 612, and a storage device manager 614. The video segmentation module 606 includes an encoder-decoder network 616. The video masking module 608 includes an encoder-decoder network 618 and an encoder 620. The storage device manager 614 includes input data 622, training data 624, a frame feature memory 626, a mask memory 628, and an alpha mask frame memory 630.

[0046] like Figure 6 As shown, the digital design system 600 includes a user interface manager 602. For example, the user interface manager 602 allows a user to provide input data to the digital design system 600. In some embodiments, as discussed above, the user interface manager 602 provides a user interface through which the user can upload a video sequence and an input mask video frame corresponding to the first frame of the video sequence. Alternatively or additionally, the user interface may enable the user to download the video sequence and the input mask video frame from a local or remote storage location (e.g., by providing an address associated with a data source (e.g., a URL or other endpoint)).

[0047] like Figure 6 As further shown, the digital design system 600 also includes an input analyzer 604. The input analyzer 604 analyzes the input received by the digital design system 600 to identify the video sequence. In one or more embodiments, the input analyzer 604 may also segment the video sequence into multiple video frames.

[0048] like Figure 6 As further shown, the digital design system 600 also includes a video segmentation module 606 configured to generate masked video frames for input video frames of a video sequence. In one or more embodiments, the video segmentation module 606 includes an encoder-decoder network 616. The encoder-decoder network 616 can be trained to extract features or feature vectors from the input video frames of the video sequence and generate a binary masked video frame representation of the input video frames. The process of generating the binary masked video frame representation of the input video frames can be further guided by previous alpha masked video frames (optionally, and features of the masked video frames), which are generated for previous video frames of the same video sequence.

[0049] In one or more embodiments, the neural network includes a deep learning architecture for learning representations of audio and / or video. The neural network may include a machine learning model that can be tuned (e.g., trained) based on training inputs to approximate an unknown function. Specifically, the neural network may include a model of interconnected digital neurons that communicate and learn to approximate complex functions, and generate outputs based on multiple inputs provided to the model. For example, the neural network includes one or more machine learning algorithms. In other words, a neural network is an algorithm that implements deep learning techniques, i.e., machine learning that utilizes a set of algorithms to attempt to model high-level abstractions in data.

[0050] like Figure 6As further shown, the digital design system 600 also includes a video masking module 608 configured to generate alpha mask frames for input masked video frames of a video sequence. In one or more embodiments, the video masking module 608 includes an encoder-decoder network 618 and an encoder 620. The encoder-decoder network 618 can be trained to extract features or feature vectors from the video frames and the input masked video frame representations of the video frames generated by the video segmentation module 606, and generate alpha mask frame representations of the video frames. The process of generating alpha mask frame representations of the video frames can be further guided by features of previous alpha masked video frames generated for previous video frames of the same video sequence. In such an embodiment, features of previous alpha masked video frames are provided to a mask memory via the encoder 620. The encoder 620 can generate features by processing the video frames and the initial alpha mask frame, and provide the generated features to the mask memory 628.

[0051] like Figure 6 As shown, the digital design system 600 also includes a neural network manager 610. The neural network manager 610 can host multiple neural networks or other machine learning models, such as encoder-decoder networks 616, encoder-decoder networks 618, and encoders 620. The neural network manager 610 may include an execution environment, libraries, and / or any other data required to execute the machine learning models. In some embodiments, the neural network manager 610 may be associated with dedicated software and / or hardware resources to execute machine learning models. Although in Figure 6 The neural network is depicted as being hosted by a single neural network manager 610, but in various embodiments, the neural network may be hosted in multiple neural network managers and / or as part of different components.

[0052] like Figure 6 As shown, the digital design system 600 also includes a training system 612. The training system 612 can teach, guide, adjust, and / or train one or more neural networks. Specifically, the training system 612 can train the neural networks based on multiple training data sets. More specifically, the training system 612 can access, identify, generate, create, and / or determine training inputs, and use these training inputs to train and fine-tune the neural networks. In particular, the training system 612 can train at least encoder-decoder network 616, encoder-decoder network 618, and encoder 620 based on the training data.

[0053] like Figure 6As shown, the digital design system 600 also includes a storage device manager 614. The storage device manager 614 maintains the data of the digital design system 600. The storage device manager 614 can maintain data of any type, size, or kind as needed to perform the functions of the digital design system 600. For example... Figure 6 As shown, the storage device manager 614 includes input data 622, training data 624, frame feature memory 626, mask memory 628, and alpha mask frame memory 630. Specifically, input data 622 may include a video sequence and input mask video frames of individual frames of the corresponding video sequence received by the digital design system 600. Training data 624 may include multiple training video sequences and corresponding training mask video frames, as well as training alpha mask frames used by the training system 612 to train one or more neural networks to generate alpha mask video sequence representations of the input video sequences.

[0054] In one or more embodiments, frame feature memory 626 may include alpha mask frames generated by video masking module 608 for previous frames of a video sequence. In one or more embodiments, feature data in frame feature memory 626 is used to guide a masking process performed by video segmentation module 606. In some embodiments, frame feature memory 626 may also include masked video frames generated by video segmentation module 606. In one or more embodiments, mask memory 628 may include alpha mask frames generated by video masking module 608 for previous frames of a video sequence. In one or more embodiments, feature data in mask memory 628 is used to guide an alpha masking process performed by video masking module 608.

[0055] In one or more embodiments, the alpha mask frame memory 630 may include alpha mask frames of a video sequence generated sequentially by the digital design system 600. In one or more embodiments, the alpha mask frames of the video sequence may be stored in the alpha mask frame memory 630 until all video frames of the video sequence have been processed and an alpha mask video sequence representation can be generated.

[0056] Each component in components 602-614 of the digital design system 600 and their corresponding elements (such as...) Figure 6 (As shown) can communicate with each other using any suitable communication technology. It will be recognized that, although in Figure 6 The components 602-614 shown and their corresponding elements are separate, but any one of the components 602-614 and their corresponding elements can be combined into fewer components, such as combined into a single facility or module, divided into more components, or configured as different components, as may serve a particular embodiment.

[0057] Components 602-614 and their corresponding elements may include software, hardware, or both. For example, components 602-614 and their corresponding elements may include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices. When executed by one or more processors, the computer-executable instructions of the digital design system 600 may cause client devices and / or server devices to perform the methods described herein. Alternatively, components 602-614 and their corresponding elements may include hardware, such as a dedicated processing device that performs a function or group of functions. Additionally, components 602-614 and their corresponding elements may include a combination of computer-executable instructions and hardware.

[0058] Furthermore, for example, components 602-614 of the digital design system 600 can be implemented as one or more independent applications, one or more modules of an application, one or more plugins, one or more library functions or functions that can be called by other applications, and / or cloud computing models. Therefore, components 602-614 of the digital design system 600 can be implemented as independent applications, such as desktop or mobile applications. Additionally, components 602-614 of the digital design system 600 can be implemented as one or more web-based applications hosted on a remote server. Alternatively or additionally, components of the digital design system 600 can be implemented within a set of mobile device applications or "apps".

[0059] As shown in the figure, the digital design system 600 can be implemented as a single system. In other embodiments, the digital design system 600 can be implemented as a whole or in part across multiple systems. For example, one or more functions of the digital design system 600 can be performed by one or more servers, and one or more functions of the digital design system 600 can be performed by one or more client devices. As described herein, one or more servers and / or one or more client devices can generate, store, receive, and transmit any type of data used by the digital design system 600.

[0060] In one implementation, one or more client devices may include or implement at least a portion of the digital design system 600. In other implementations, one or more servers may include or implement at least a portion of the digital design system 600. For example, the digital design system 600 may include an application running on one or more servers, or a portion of the digital design system 600 may be downloaded from one or more servers. Additionally or alternatively, the digital design system 600 may include a web-hosted application that allows client devices(s) to interact with content hosted on one or more servers.

[0061] For example, in one or more embodiments, when a client device accesses a webpage or other web application hosted on one or more servers, the one or more servers may provide access to one or more files, including video sequences and input mask video frames stored on the one or more servers. Furthermore, the client device may receive a request (i.e., via user input) to generate an alpha mask video sequence representation of the video sequence and provide that request to the one or more servers. Upon receiving the request, the one or more servers may automatically execute the methods and processes described above to generate the alpha mask video sequence representation of the video sequence. The one or more servers may provide the alpha mask sequence representation of the video sequence to the client device for display to the user.

[0062] Multiple servers and / or multiple client devices may communicate using any communication platform and technology suitable for transmitting data and / or communication signals, including any known communication technologies, devices, media, and protocols that support remote data communication, as referenced below. Figure 8 Examples of these communication platforms and technologies are described in more detail below. In some embodiments, (multiple) servers and / or (multiple) client devices communicate via one or more networks. Networks may include a single network or a collection of networks (such as the Internet, corporate intranet, virtual private network (VPN), local area network (LAN), wireless local area network (WLAN), cellular network, wide area network (WAN), metropolitan area network (MAN), or a combination of two or more such networks). The following will discuss... Figure 8 Discuss one or more networks in more detail.

[0063] The (multiple) servers may include one or more hardware servers (e.g., hosts), each with its own computing resources (e.g., processor, memory, disk space, network bandwidth, etc.), which can be securely partitioned among multiple clients (e.g., client devices), each of which can host its own application on the (multiple) servers. The (multiple) client devices may include one or more personal computers, laptops, mobile devices, mobile phones, tablets, dedicated computers, televisions, or other computing devices, including those described below. Figure 8 The computing device described.

[0064] Figure 1-6 The corresponding text and examples provide many different systems and devices that use loop guidance to generate alpha mask video sequence representations of video sequences. In addition to the above, embodiments can also be described using flowcharts, which include actions and steps in methods for achieving specific results. For example, Figure 7A flowchart illustrating an exemplary method according to one or more embodiments is shown. About Figure 7 The described method can be performed with fewer or more steps / actions, or the steps and / or actions can be performed in a different order. Additionally, the steps / actions described herein can be repeated or performed in parallel with each other or with different instances of the same or similar steps / actions.

[0065] Figure 7 A flowchart illustrating a series of actions in a method for generating an alpha mask video sequence representation of a video sequence using a machine learning model of a digital design system, according to one or more embodiments, is shown. In one or more embodiments, method 700 is performed in a digital media environment including a digital design system 600. Method 700 is intended to illustrate one or more methods according to this disclosure and is not intended to limit potential embodiments. Alternative embodiments may include... Figure 7 The steps described herein may include more, fewer, or different steps.

[0066] like Figure 7 As shown, method 700 includes action 702 of receiving a video sequence and an input mask video frame of a first video frame of the video sequence. In one or more embodiments, the video sequence is input to a digital design system, and a user requests the creation of an alpha mask video sequence representation for that input. In one or more embodiments, the input mask video frame is a binary mask that designates each pixel of the first video frame of the video sequence as a foreground pixel or a background pixel. In one or more embodiments, the digital design system receives the video sequence and the input mask video frame from a user (e.g., via a computing device). In one or more embodiments, a user may select or provide the video sequence and the input mask video frame in an application, or a user may submit the video sequence and the input mask video frame to a web service or application configured to receive input. The video sequence may be a selected portion of a longer video sequence. For example, after providing the video sequence to an application, the application may provide an interface that allows the user to select a portion of the video sequence.

[0067] like Figure 7As shown, method 700 includes an action 704 of generating an alpha mask frame for a video sequence using a video sequence and an input masked video frame, wherein a first network generates the masked video frame based on stored features of previous frames of the video sequence, and a second network generates the alpha mask frame based on the masked video frame and stored features of previous frames of the video sequence. In one or more embodiments, a first encoder-decoder network trained to generate the masked video frame receives a first video frame of the video sequence and an input masked video frame of the first video frame of the video sequence. In one or more embodiments, features of the input masked video frame of the first video frame of the video sequence are stored in a frame feature memory. In such an embodiment, when the first encoder-decoder processes the first video frame, the features of the input masked video frame of the first video frame of the video sequence are combined with features of the first video frame generated by the encoder of the first encoder-decoder network. Then, the decoder of the first encoder-decoder network generates the first masked video frame based on the features of the first video frame of the video sequence and the features of the input masked video frame of the first video frame of the video sequence.

[0068] In one or more embodiments, a first masked video frame is then sent to a second encoder-decoder network trained to generate an alpha mask. The second encoder-decoder network generates a first alpha mask frame based on the first masked video frame. In one or more embodiments, the second encoder-decoder network generates an initial alpha mask frame by passing the first video frame and the first masked video frame through the second encoder-decoder network. The initial alpha mask frame is then provided to a memory encoder to generate features for the initial alpha mask frame. The features of the initial alpha mask frame and the features of the first video frame are then combined and stored in a mask memory. The second encoder-decoder network then generates a first alpha mask frame representing a first video frame of a video sequence by passing the first video frame and the first masked video frame through the second encoder-decoder network, wherein one or more features in the mask memory are combined with the features of the first video frame and the first masked video frame in one or more layers of the decoder in the second encoder-decoder network. In one or more embodiments, the frame feature memory and the mask memory are updated using at least the features of the first alpha mask frame. In one or more embodiments, the frame feature memory is further updated using the first masked video frame.

[0069] In one or more embodiments, each additional video frame of the video sequence is subsequently processed sequentially by a digital design system to generate a corresponding alpha mask frame. For example, using the previously described process, a first encoder-decoder uses the stored features of the second video frame and the previously generated first alpha mask frame to generate a second video frame consecutive to the first video frame, or a second mask video frame of the next frame. Similarly, a second encoder-decoder network uses the stored features of the second mask video frame of the second video frame of the video sequence and the previously generated first alpha mask frame to generate a second alpha mask frame representing the second video frame of the video sequence. The frame feature memory and mask memory are then updated using at least the features of the second alpha mask frame. This process is performed iteratively until all video frames of the video sequence or a selected subset of all video frames have been processed.

[0070] like Figure 7 As shown, method 700 includes action 706 of outputting an alpha-mask video sequence representation of a video sequence, the video sequence including the generated alpha-mask frames. After processing all video frames of a selected subset of the video frames of the video sequence, the alpha-mask frames can be combined to generate an alpha-mask video sequence representation of the video sequence.

[0071] As discussed in more detail below, embodiments of this disclosure may include or utilize a dedicated or general-purpose computer, which includes computer hardware such as one or more processors and system memory. Embodiments within the scope of this disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any media content access device described herein). Generally, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., memory, etc.) and executes those instructions to perform one or more processes, including one or more processes described herein.

[0072] Computer-readable media can be any available medium accessible to general-purpose or special-purpose computer systems. A computer-readable medium storing computer-executable instructions is a non-transitory computer-readable storage medium (device). A computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of this disclosure may include at least two distinct types of computer-readable media: a non-transitory computer-readable storage medium (device) and a transmission medium.

[0073] Non-transient computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives (“SSDs”) (e.g., RAM-based), flash memory, phase-change memory (“PCM”), other types of memory, other optical disc storage devices, disk storage devices or other magnetic storage devices, or any other non-transient storage medium that can be used to store desired program code in the form of computer-executable instructions or data structures and is accessible by a general-purpose or special-purpose computer.

[0074] A “network” is defined as one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or other communication connection (hardwired, wireless, or a combination of hardwired and wireless), the computer correctly regards the connection as a transmission medium. Transmission media may include networks and / or data links that can be used to carry desired program code in the form of computer-executable instructions or data structures and are accessible by general-purpose or special-purpose computers. The combination of the above should also be included within the scope of computer-readable media.

[0075] Furthermore, upon arrival at various computer system components, program code in the form of computer-executable instructions or data structures can be automatically transferred from the transmission medium to a non-transitory computer-readable storage medium (device) (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be buffered in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system RAM and / or a less volatile computer storage medium (device) at the computer system. Therefore, it should be understood that non-transitory computer-readable storage media (devices) can be included in computer system components that also (even primarily) utilize the transmission medium.

[0076] For example, computer-executable instructions include instructions and data that, when executed at a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a function or group of functions. In some embodiments, the computer-executable instructions are executed on a general-purpose computer to turn the general-purpose computer into a special-purpose computer that implements the elements of this disclosure. For example, the computer-executable instructions may be binary files, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as exemplary forms for implementing the claims.

[0077] Those skilled in the art will understand that this disclosure can be practiced in networked computing environments with many types of computer system configurations, including personal computers, desktop computers, laptops, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, mobile phones, PDAs, tablets, pagers, routers, switches, etc. This disclosure can also be practiced in distributed system environments, where local and remote computer systems linked via a network (by hardwired data links, wireless data links, or a combination of hardwired and wireless data links) perform tasks. In a distributed system environment, program modules can reside in local and remote memory storage devices.

[0078] The embodiments of this disclosure can also be implemented in a cloud computing environment. In this specification, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be used in marketplaces to provide universal and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly configured via virtualization and released with low management effort or service provider interaction, and then scaled up accordingly.

[0079] Cloud computing models can be composed of various characteristics, such as on-demand self-service, broad network access, resource pooling, rapid elasticity, and measurable services. Cloud computing models can also expose various service models, such as Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). Cloud computing models can also be deployed using different deployment models, such as private clouds, community clouds, public clouds, and hybrid clouds. In this specification and claims, a “cloud computing environment” means an environment employing cloud computing.

[0080] Figure 8 An exemplary computing device 800 is illustrated in block diagram form, which can be configured to perform one or more processes described above. It will be understood that one or more computing devices, such as computing device 800, can implement a digital design system. Figure 8 As shown, the computing device may include a processor 802, a memory 804, one or more communication interfaces 806, a storage device 808, and one or more I / O devices / interfaces 810. In some embodiments, the computing device 800 may include a processor 802, a memory 804, one or more communication interfaces 806, a storage device 808, and one or more I / O devices / interfaces 810. Figure 8 The diagram shows fewer or more components. A more detailed description will follow. Figure 8 The components of the computing device 800 shown.

[0081] In certain embodiments, processor(s) 802 includes hardware for executing instructions, such as those constituting a computer program. By way of example and not limitation, in order to execute instructions, processor(s) 802 may retrieve (or fetch) instructions from internal registers, internal caches, memory 804, or storage device 808, and decode and execute them. In various embodiments, processor(s) 802 may include one or more central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), system-on-a-chip (SoCs), or other processor(s) or combinations of processors.

[0082] Computing device 800 includes memory 804 coupled to processor(s) 802. Memory 804 can be used to store data, metadata, and programs executed by processor(s). Memory 804 may include one or more of volatile and non-volatile memory, such as random access memory (“RAM”), read-only memory (“ROM”), solid-state drive (“SSD”), flash memory, phase-change memory (“PCM”), or other types of data storage devices. Memory 804 may be internal memory or distributed memory.

[0083] The computing device 800 may also include one or more communication interfaces 806. The communication interface 806 may include hardware, software, or both. The communication interface 806 provides one or more interfaces for communication (such as packet-based communication) between the computing device and one or more other computing devices 800 or one or more networks. By way of example and not limitation, the communication interface 806 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired-based networks, or a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks such as Wi-Fi. The computing device 800 may also include a bus 812. The bus 812 may include hardware, software, or both for coupling components of the computing device 800 to each other.

[0084] Computing device 800 includes storage device 808, which includes a storage means for storing data or instructions. By way of example and not limitation, storage device 808 may include the non-transient storage media described above. Storage device 808 may include a hard disk drive (HDD), flash memory, a universal serial bus (USB) drive, or a combination of these or other storage devices. Computing device 800 also includes one or more input or output (“I / O”) devices / interfaces 810 provided to allow a user to provide input (such as user strokes) to computing device 800, receive output from computing device 800, and otherwise transfer data between computing device 800 and computing device 800. These I / O devices / interfaces 810 may include a mouse, keypad or keyboard, touchscreen, camera, optical scanner, network interface, modem, other known I / O devices, or combinations of such I / O devices / interfaces 810. Touchscreens can be activated with a stylus or finger.

[0085] I / O device / interface 810 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O device / interface 810 is configured to provide graphical data to a display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0086] In the foregoing specification, embodiments have been described with reference to specific exemplary embodiments. Various embodiments have been described with reference to the details discussed herein, and various embodiments are illustrated in the accompanying drawings. The foregoing description and drawings are illustrative of one or more embodiments and should not be construed as limiting. Numerous specific details have been described to provide a full understanding of the various embodiments.

[0087] Without departing from its spirit or essential characteristics, embodiments may include other specific forms. The described embodiments are to be considered illustrative in all respects only, and not restrictive. For example, the methods described herein may be performed with fewer or more steps / actions, or with steps / actions performed in a different order. Additionally, the steps / actions described herein may be repeated or performed in parallel with each other or with different instances of the same or similar steps / actions. Therefore, the scope of the invention is indicated by the appended claims, not by the foregoing description. All modifications within the equivalent meaning and scope of the claims should be included within their scope.

[0088] In the various embodiments described above, unless otherwise specifically stated, disjunctive language such as the phrase “at least one of A, B, or C” is intended to be understood as meaning A, B, or C, or any combination thereof (e.g., A, B, and / or C). Therefore, disjunctive language is not intended, nor should it be construed, as implying that a given embodiment requires the presence of at least one of A, at least one of B, or at least one of C.

Claims

1. A method comprising: Receive a video sequence and an input mask video frame for a first video frame of the video sequence; The video sequence and the input masked video frame are used to generate an alpha mask frame for the video sequence, wherein a first network generates the masked video frame based on stored features of previous frames of the video sequence, and a second network generates the alpha mask frame based on the masked video frame and the stored features of the previous frames of the video sequence. as well as The output includes an alpha mask video sequence representation of the video sequence of the generated alpha mask frames.

2. The method of claim 1, wherein generating the alpha mask frame for the video sequence using the video sequence and the input mask video frame further comprises: The first network receives the first video frame of the video sequence and the input mask video frame for the first video frame of the video sequence; The first masked video frame is generated by the first network based on the first video frame of the video sequence and the input masked video frame for the first video frame of the video sequence; The second network generates the first alpha mask frame based on the first masked video frame; as well as At least the features of the first alpha mask frame are used to update the frame feature memory and the mask memory.

3. The method of claim 2, wherein updating the frame feature memory and the mask memory using at least the features of the first alpha mask frame further comprises: The first video frame and the first alpha mask frame of the video sequence are encoded using an encoder to generate the features of the first alpha mask frame. as well as The features of the first alpha mask frame are stored in the mask memory.

4. The method according to claim 2, further comprising: The frame feature memory is updated using a second feature representing the first masked video frame.

5. The method according to claim 2, further comprising: Each additional video frame of the video sequence is processed continuously by the first network and the second network to generate a corresponding alpha mask frame.

6. The method of claim 5, wherein continuously processing each additional video frame comprises: The first network uses the stored features of the previous frame of the video sequence to generate a next masked video frame for the next video frame of the video sequence, wherein the next video frame is consecutive to the previous video frame, and wherein the stored features of the previous frame of the video sequence include at least the first alpha mask frame representing the first video frame of the video sequence. The second network uses the features of the next mask video frame for the next video frame of the video sequence and the stored features of the previous frames of the video sequence to generate the next alpha mask frame representing the next video frame of the video sequence; as well as The frame feature memory and the mask memory are updated using at least the features of the next alpha mask frame.

7. The method of claim 2, wherein generating the first alpha mask frame based on the input mask video frame further comprises: An initial alpha mask frame is generated by passing the first video frame and the first mask video frame through the second network; as well as The first alpha mask frame representing the video sequence is generated by passing the first video frame and the first mask video frame through the second network, wherein one or more features in the mask memory are combined with features of the first mask video frame and the first video frame in one or more layers of the decoder in the second network.

8. The method of claim 1, wherein the input mask video frame designates each pixel of the first video frame of the video sequence as a foreground pixel or a background pixel.

9. A non-transient computer-readable medium storing executable instructions, which, when executed by a processing device, cause the processing device to perform an operation, the operation comprising: Receive a video sequence and an input mask video frame for a first video frame of the video sequence; The video sequence and the input masked video frame are used to generate an alpha mask frame for the video sequence, wherein a first network generates the masked video frame based on stored features of previous frames of the video sequence, and a second network generates the alpha mask frame based on the masked video frame and the stored features of the previous frames of the video sequence. as well as The output includes an alpha mask video sequence representation of the video sequence of the generated alpha mask frames.

10. The non-transient computer-readable medium of claim 9, wherein the instructions for generating the alpha mask frame for the video sequence using the video sequence and the input mask video frame further include: The first network receives the first video frame of the video sequence and the input mask video frame for the first video frame of the video sequence; The first masked video frame is generated by the first network based on the first video frame of the video sequence and the input masked video frame for the first video frame of the video sequence; The second network generates the first alpha mask frame based on the first masked video frame; as well as At least the features of the first alpha mask frame are used to update the frame feature memory and the mask memory.

11. The non-transient computer-readable medium of claim 10, wherein the instructions further comprise: The frame feature memory is updated using a second feature representing the first masked video frame.

12. The non-transient computer-readable medium of claim 10, wherein the instructions further include: Each additional video frame of the video sequence is processed continuously by the first network and the second network to generate a corresponding alpha mask frame.

13. The non-transient computer-readable medium of claim 12, wherein the instructions for continuously processing each additional video frame further include: The first network uses the stored features of the previous frame of the video sequence to generate a next masked video frame for the next video frame of the video sequence, wherein the next video frame is consecutive to the previous video frame, and wherein the stored features of the previous frame of the video sequence include at least the first alpha mask frame representing the first video frame of the video sequence. The second network uses the features of the next mask video frame for the next video frame of the video sequence and the stored features of the previous frames of the video sequence to generate the next alpha mask frame representing the next video frame of the video sequence; as well as The frame feature memory and the mask memory are updated using at least the features of the next alpha mask frame.

14. The non-transient computer-readable medium of claim 10, wherein the instructions for generating the first alpha mask frame based on the input mask video frame further include: An initial alpha mask frame is generated by passing the first video frame and the first mask video frame through the second network; as well as The first alpha mask frame representing the video sequence is generated by passing the first video frame and the first mask video frame through the second network, wherein one or more features in the mask memory are combined with features of the first mask video frame and the first video frame in one or more layers of the decoder in the second network.

15. A system comprising: Memory components; as well as A processing device, coupled to the memory component, performs operations including: Receive a video sequence and an input mask video frame for a first video frame of the video sequence; The video sequence and the input masked video frame are used to generate an alpha mask frame for the video sequence, wherein a first network generates the masked video frame based on stored features of previous frames of the video sequence, and a second network generates the alpha mask frame based on the masked video frame and the stored features of the previous frames of the video sequence; and The output includes an alpha mask video sequence representation of the video sequence of the generated alpha mask frames.

16. The system of claim 15, wherein the operation of generating the alpha mask frame for the video sequence using the video sequence and the input mask video frame further comprises: The first network receives the first video frame of the video sequence and the input mask video frame for the first video frame of the video sequence; The first masked video frame is generated by the first network based on the first video frame of the video sequence and the input masked video frame for the first video frame of the video sequence; The second network generates the first alpha mask frame based on the first masked video frame; as well as At least the features of the first alpha mask frame are used to update the frame feature memory and the mask memory.

17. The system of claim 16, wherein the operation further comprises: The frame feature memory is updated using a second feature representing the first masked video frame.

18. The system of claim 16, wherein the operation further comprises: Each additional video frame of the video sequence is processed continuously by the first network and the second network to generate a corresponding alpha mask frame.

19. The system of claim 18, wherein the operation for continuously processing each additional video frame further comprises: The first network uses the stored features of the previous frame of the video sequence to generate a next masked video frame for the next video frame of the video sequence, wherein the next video frame is consecutive to the previous video frame, and wherein the stored features of the previous frame of the video sequence include at least the first alpha mask frame representing the first video frame of the video sequence. The second network uses the features of the next mask video frame for the next video frame of the video sequence and the stored features of the previous frames of the video sequence to generate the next alpha mask frame representing the next video frame of the video sequence; as well as The frame feature memory and the mask memory are updated using at least the features of the next alpha mask frame.

20. The system of claim 16, wherein the operation of generating the first alpha mask frame based on the input mask video frame further comprises: An initial alpha mask frame is generated by passing the first video frame and the first mask video frame through the second network; as well as The first alpha mask frame representing the video sequence is generated by passing the first video frame and the first mask video frame through the second network, wherein one or more features in the mask memory are combined with features of the first mask video frame and the first video frame in one or more layers of the decoder in the second network.

Images