Unwinding feature transformation for video object segmentation

Abstract

Systems and methods for performing video object segmentation are provided. In an example, video data representing a sequence of image frames and video data representing an object mask can be received at a video object segmentation server. Image features can be generated based on a first image frame of the sequence of image frames, image features can be generated based on a second image frame of the sequence of image frames; and object features can be generated based on the object mask. A transformation matrix can be computed based on the image features of the first image frame and the image features of the second image frame; the transformation matrix can be applied to the object features, resulting in transformed object features. A predicted object mask associated with the second image frame can be obtained by decoding the transformed object features.

Description

Background Technology

[0001] Video object segmentation aims to separate foreground objects from background regions in a video sequence. Video object segmentation is a crucial task in video analysis and editing, often requiring significant manual interaction to achieve satisfactory results. Semi-supervised video object segmentation (Semi-VOS) segments instances of object of interest from a video sequence, provided that a fine-grained mask is provided in the first frame. While Semi-VOS has wide applications in robotics, autonomous driving, and video editing tasks, most Semi-VOS implementations require real-time response, making speed and quality limiting factors. Early Semi-VOS methods relied on the provided mask to fine-tune the network to fit the target object during inference, resulting in slow execution. Therefore, some Semi-VOS methods eliminate the fine-tuning step by extracting information from the provided mask and corresponding image in a feedforward manner to predict the mask for a specific object. However, these methods remain time-consuming, and different object masks require separate processing. For example, the Spatial-Temporal Memory (STM) method proposes a three-step process involving feature encoding, correlation-based feature transformation, and feature decoding. This process requires object masks and related images to be encoded by a heavy encoder network to generate object-specific memory features; furthermore, feature transformations are recomputed for each object. The Fast End-to-End Embedded Learning Approach for Video Object Segmentation (FEELVOS) uses semantic pixel embeddings along with global and local matching mechanisms to transfer information from the first frame and the previous frame of the video to the current frame. However, FEELVOS requires computing high-resolution feature maps to preserve feature details; therefore, FEELVOS may require high computational resources and may execute slower than other encoder-decoder methods.

[0002] Embodiments have been described with respect to these and other general considerations. Although relatively specific problems have been discussed, it should be understood that the examples described herein are not limited to solving the specific problems identified in the background section above. Summary of the Invention

[0003] According to examples in this disclosure, a system and method for an unwrapped feature transformation architecture for feature computation of a single unwrapped object are provided. Specifically, paired features transform propagated information from previous frames into the current frame, which is unknown to any object in the video sequence. Object-specific features are then computed using an efficient, lightweight network. A feature decoding module is also included, and the decoded image features are shared across multiple target objects. In this way, the unwrapped feature transformation architecture requires a single pass of feature encoding and decoding for each image and minimizes the computational cost per object.

[0004] According to at least one example of this disclosure, a method for performing video object segmentation is described. The method may include: receiving video data representing a sequence of image frames; receiving an object mask; generating image features based on a first image frame of the image frame sequence; generating object features based on the object mask; generating image features based on a second image frame of the image frame sequence; calculating a transformation matrix based on the image features of the first and second image frames; applying the transformation matrix to the object features resulting in transformed object features; and obtaining a predicted object mask associated with the second image frame by decoding the transformed object features.

[0005] According to at least one example of this disclosure, a system configured to perform video object segmentation is described. The system may include a processor and a memory including instructions that, when executed by the processor, cause the processor to: receive video data representing a sequence of image frames; receive an object mask; generate image features based on a first image frame of the image frame sequence; generate object features based on the object mask; generate image features based on a second image frame of the image frame sequence; compute a transformation matrix based on the image features of the first and second image frames; apply the transformation matrix to the object features resulting in transformed object features; and obtain a predicted object mask associated with the second image frame by decoding the transformed object features.

[0006] According to at least one example of this disclosure, a computer-readable storage medium is described including instructions for performing video object segmentation. The computer-readable storage medium may include instructions that, when executed by a processor, cause the processor to: receive video data representing a sequence of image frames; receive an object mask; generate image features based on a first image frame of the image frame sequence; generate object features based on the object mask; generate image features based on a second image frame of the image frame sequence; calculate a transformation matrix based on the image features of the first and second image frames; apply the transformation matrix to the object features resulting in transformed object features; and obtain a predicted object mask associated with the second image frame by decoding the transformed object features.

[0007] This summary is provided to introduce a series of concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0008] The following figures illustrate a non-limiting and non-exhaustive example.

[0009] Figure 1 A semi-supervised video object segmentation (semi-VOS) system according to an example of this disclosure is described.

[0010] Figure 2 Details of a video object segmentation server based on an example of this disclosure are described.

[0011] Figure 3 Additional details are depicted for the image processing module, object encoder module, and feature transformation and decoding module according to examples of this disclosure.

[0012] Figure 4 Details of an object encoder based on an example of this disclosure are depicted.

[0013] Figure 5 Details of the feature transformation workflow according to the examples of this disclosure are described.

[0014] Figure 6 Details of an object encoder based on an example of this disclosure are depicted.

[0015] Figure 7 Details of a method for generating a predictive object mask according to an example of this disclosure are described.

[0016] Figure 8 Details of a method for generating multiple prediction object masks according to an example of this disclosure are described.

[0017] Figure 9 It is a block diagram illustrating the physical components (e.g., hardware) of a computing device that can implement aspects of this disclosure.

[0018] Figures 10A-10B A mobile computing device is shown that can be used to practice embodiments of the present disclosure.

[0019] Figure 11 This illustrates one aspect of the architecture of a system used for data processing. Detailed Implementation

[0020] In the following detailed description, reference is made to the accompanying drawings, which form a part of the description, and specific embodiments or examples are illustrated therein. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from this disclosure. Embodiments may be implemented as methods, systems, or devices. Therefore, embodiments may take the form of hardware implementations, entirely software implementations, or implementations combining software and hardware aspects. Consequently, the following detailed description is not to be construed as limiting, and the scope of this disclosure is defined by the appended claims and their equivalents.

[0021] Video object segmentation aims to separate foreground objects from background regions in a video sequence. Semi-supervised video object segmentation (Semi-VOS) segments instances of object of interest from a video sequence by providing a fine-grained mask in the first frame. While Semi-VOS has wide applications in robotics, autonomous driving, and video editing tasks, most Semi-VOS implementations require real-time response, making speed and quality limiting factors. Methods for performing semi-VOS can generally be divided into two categories: online adaptation methods that require fine-tuning the model based on the target object, and matching-based methods that learn a match and propagate the initial mask to subsequent video frames.

[0022] Online adaptation methods typically utilize given base-fact videos as training samples and use feature similarity to tune the neural network to avoid independent processing associated with each object. However, online adaptation methods often require high-resolution feature maps to preserve feature details, which increases the computational cost and slows down the execution of these processes. Matching-based methods typically utilize a given mask in a feedforward manner and are faster than online adaptation methods. For example, they use the mask from the previous frame as input and refine it based on the target object. Some matching-based methods match features of the current frame with features from the first and / or previous frames to predict the object mask; however, most matching-based methods tend to be computationally inefficient and exhibit high computational costs.

[0023] More efficient VOS methods can leverage a global context module to encode temporal features from previous frames for efficient temporal information retrieval. In some examples, reinforcement learning is used to select the matching method and update the object template. Some examples utilize a two-stream design to dynamically focus on regions around a target object. While speed improvements can be achieved, these methods still require significant time to process multiple objects in a video because most computations cannot be shared across different objects. For example, when predicting object masks, image features and / or transformation matrices may not be shared across different objects. Therefore, additional computation may be needed to generate image features and object-specific transformation matrices. For instance, in FEELVOS, different objects can share the same feature maps to compute feature similarities, which are further refined into masks. However, the feature maps are generated at high resolution, which slows down the object segmentation process.

[0024] Based on examples from this disclosure, an unwrapped feature transformation architecture for unwrapping feature computation of a single object is described. Specifically, paired features are transformed from propagated information from previous frames into the current frame of a video sequence where any object is unknown. Object-specific features are then computed using an efficient, lightweight network. A feature decoding module shares the decoded image features across multiple target objects. In this way, the unwrapped feature transformation architecture encodes and decodes features for each image only once, thereby minimizing the computational cost added for each added object.

[0025] As described in this paper, the Unwrapped Feature Transform (DiFT) network divides the processing framework into two main parts to achieve a significant performance improvement in execution speed. First, the image processing backbone generates multi-scale image features and cross-frame feature transforms. Second, the DiFT network can include object-specific parts for merging masks. The processed image features and correlation maps can be reused for feature transforms between different masks, thereby improving the processing speed of multi-object scenes. Furthermore, multi-scale image features can be used to generate object-specific features using a lightweight network to fuse mask features and image features.

[0026] According to the examples in this disclosure, the proposed DiFT network comprises three modules: an image processing backbone, an object encoding module, and a feature transformation and decoding module. The image processing backbone generates global features that are unwrapped with specific objects in the image. Using the global features, cross-frame attention maps can be generated for transforming the features of each individual object at a later stage. For clarity, such attention maps are represented herein as transformation matrices. The object encoding module combines the mask of the specific object with the generated image features. Specifically, features from the image encoder are combined with features of the mask at the photomask encoder to produce mask features. The collected transformation matrices (e.g., attention maps) generated from the image features are applied to the corresponding mask features to produce transformed features. The lightweight object decoder can combine the transformed features with shared features from the image decoder to output a mask prediction. By leveraging global image features, each object can be processed with reduced computational cost. For example, the DiFT network can encode each image, generating features for mask prediction and temporal feature encoding using a single pass. Therefore, the DiFT network can operate at increased speed for both single objects and multiple object instances.

[0027] Figure 1A semi-supervised video object segmentation (semi-VOS) system 100 according to an example of this disclosure is depicted. More specifically, user 102 may acquire or otherwise obtain video clips or image frame sequences 106. Image frame sequences 106 may include one or more objects 108 that will be segmented or otherwise separated from the background of the image. For example, user 102 may wish to segment a first object 108A corresponding to a person and a second object 108B corresponding to a ski from a background 110 in each image frame of image frame sequence 106. In the example, user 102 may use computing device 104 to acquire image frame sequences 106 via network 118 and transmit them to video object segmentation server 122. Computing device 104, while depicted, for example, as a desktop computer, may be any of portable or non-portable computing devices. For example, computing device 104 may be a smartphone, laptop, desktop computer, or server. Image frame sequences 106 may be obtained in any format and may be compressed and / or decompressed. Furthermore, the object mask 112A corresponding to the first object 108A in the first image frame of the image frame sequence 106 can be sent to the video object segmentation server 122. Additionally, the object mask 112B corresponding to the second object 108B in the first image frame of the image frame sequence 106 can be sent to the video object segmentation server 122.

[0028] In the example, user 102 can generate each of object masks 112A and 112B, for example, using a video editing application. Alternatively, or additionally, user 102 can generate each object in object masks 112A and 112B using an image editing application, since each object in object masks 112A and 112B corresponds to an object within a single image frame (e.g., a first image frame) of the video. Object masks 112A and 112B can use a first pixel value (e.g., 0,0,0) to indicate the pixels that cover the object and a second pixel value (e.g., 255,255,255) to indicate the pixels that do not cover the object in the image.

[0029] The video object segmentation server 122 can receive image frame sequence 106 and object masks 112A and / or 112B of the first image frame in image frame sequence 106; the video object segmentation server 122 can generate predicted object masks 120A and / or 120B corresponding to subsequent image frames in image frame sequence 106. For example, the video object segmentation server 122 can generate predicted object masks 120A and / or 120B corresponding to each object in each image frame in image frame sequence 106 that occurs after the first image frame in image frame sequence 106. In the example, the video object segmentation server 122 can generate predicted object masks 120A and / or 120B corresponding to each object in each image frame in image frame sequence 106 that occurs before another image frame in image frame sequence 106. The sequence of object masks 120A and / or 120B can then be provided to the computing device 104 of the requesting user 102.

[0030] Figure 2 Details of a video object segmentation server 202 according to an example of this disclosure are depicted. The video object segmentation server 202 may be the same as or similar to the previously described video object segmentation server 122. The video object segmentation server 202 may include a communication interface 204, a processor 206, and a computer-readable storage device 208. In the example, the communication interface 204 may be coupled to a network and receive image frame sequences 106 and object masks 112A and / or 112B. Figure 1 Image frame sequence 106 can be stored as video frame 216, and object masks 112A and / or 112B can be stored as object mask 218. In some examples, one or more CNN model parameters can also be received at communication interface 204 and stored as CNN model parameters 220. CNN model parameters 220 can include one or more parameters and hyperparameters that define the CNN model. In the example, CNN model parameters 220 can correspond to a specific CNN implementation that will be implemented at video object segmentation server 202. For example, CNN model parameters 220 can refer to a specific model selected by the user (e.g., a specific model trained with a specific training dataset). While video frame 216, object mask 218, and CNN model parameters 220 are depicted as input 212, other information and inputs can be received at communication interface 204 and stored as input 212.

[0031] In this example, one or more applications 210 may be provided by a video object segmentation server 202. One or more applications 210 may include a video acquisition manager 222, an image processing module 224, an object encoder module 226, and a feature transformation and decoding module 228. The image processing module 224 may include an image encoder 230, an image decoder 232, and a transform generator 234. The object encoder module 226 may include an object encoder 236. The feature transformation and decoding module 228 may include a transform application module 238 and an object decoder 240.

[0032] Unlike STM-based methods that use a separate image processing backbone, a single image feature extraction backbone can be used for both mask prediction and temporal feature generation. Therefore, the image processing backbone can be referred to as image processing module 224. In the example, a convolutional neural network (such as ResNet-50) can be used to extract features from image frames, where the image frames correspond to image frame sequence 106. Therefore, an image encoder (such as image encoder 230) can generate two pieces of information corresponding to the extracted features. The first piece of information can correspond to the image features used to generate the transformation matrix, and the second piece of information can correspond to the features provided to image decoder 232. Image decoder 232 can be a skip-connected decoder utilizing a single residual block and multiple refinement modules. Notably, image decoder 232 may not directly output the prediction. Instead, the features generated by image decoder 232 can be used for feature transformation and decoding module 228 to predict the object mask. In this way, the intermediate features of image processing module 224 can be repeatedly used for object-specific computations, such as relative to... Figure 4 The transform generator can generate a transform matrix based on adjacent image frames in an image frame sequence.

[0033] To improve the efficiency of feature encoding for a specific object, the object mask can be encoded using a lightweight neural network (such as object encoder 236) and reuse previously generated image features instead of combining the object mask and the image to extract features. According to examples of this disclosure, the transform application module 238 can be a correlation-based feature transform to utilize temporal information from previous frames. In contrast to previous work that required recompiling transform matrices for different target objects, the transform application module 238 computes the transform matrix only once using the image features, untangling the feature transform with downstream tasks. Therefore, the feature transform can be viewed as a feature alignment function for finding correspondences of features across frames. This correspondence is generally independent of any specific object. Thus, the feature transform can be computed by the transform application module using the image and further applied to features encoded for the specific object. Similar to object encoder module 226, features from image decoder 232 can be used by object decoder 240. Object decoder 240 can generate a predicted object mask based on the transformed features from image decoder 232 and the image decoded features.

[0034] Figure 3 Additional details of the image processing module 302, object encoder module 324, and feature transformation and decoding module 338 according to examples of this disclosure are depicted. Image processing module 302 may be the same as or similar to image processing module 224; object encoder module 324 may be the same as or similar to object encoder module 226; and feature transformation and decoding module 338 may be the same as or similar to feature transformation and decoding module 228. Image processing module 302 may receive a first image frame 304 of image frame sequence 106. Image encoder 312 may encode features of the first image frame 304 and generate image features 314. That is, image encoder 312 may generate image features from the first image frame 304, rather than from the image along with the object mask. Therefore, the extracted features are not specific to any particular object in image frame sequence 106.

[0035] To predict the object mask for each object of interest, object features can be generated from the corresponding object mask. For example, object encoder 326 (a network encoder smaller than image encoder 312) can receive object mask 308 and previously computed image features from image encoder 312. Object encoder 326 can combine the image features from image encoder 312 with object mask 308 to generate object feature 330. Similarly, for another object of interest, object encoder 328 (a network encoder smaller than image encoder 312) can receive a second object mask 310 and previously computed image features from image encoder 312. Object encoder 328 can combine the image features from image encoder 312 with object mask 310 to generate object feature 332. More specifically, features from image encoder 312 can be combined with the corresponding photomask object encoders 326 and 328 using the mask ( and ) feature combinations to generate mask features ( and ).

[0036] When a new image frame, such as image frame 306, is received, the image features of the new frame can be used to generate a similarity transformation matrix with the image features of the previous frame. For example, image encoder 316 can encode the features of image frame 306 and generate image features 318. Similar to image encoder 312, image encoder 316 can generate image features from image frame 306 together with an object mask, rather than from the image together with the object mask. Therefore, the extracted features are independent of any specific object in the image frame sequence 106.

[0037] Transform generator 320 can receive image features 314 and 318 and generate a similarity transformation matrix 322. The similarity transformation matrix 322 can be a weighted matrix indicating the similarity between image features 318 and image features 314. The similarity transformation matrix 322 can then be used to aggregate image features from previous frames and apply the aggregated image features to objects. When multiple objects are segmented, a set of common transformation matrices 344 can be applied to object features that result in transformed features. For example, transform application module 346 can apply a set of transformation matrices 344 to object features 340 to obtain transformed object features 348, where transformed object features 348 can be an aggregation of object features 340 from previous image frames. Similarly, transform application module 350 can apply a set of transformation matrices 344 to object features 342 to obtain transformed object features 352, where transformed object features 352 can be an aggregation of object features 342 from previous image frames. More specifically, the collected transformation matrix ([T) generated from image features... 1→b T 2→b ,....,T b-1→b]) Applied to the corresponding mask features and To generate transformed features and

[0038] Image decoder 336 can be used to process image features from image frame 306 and provide the processed image features to object decoders 354 and 358. Image decoder 336 is not targeted at any specific object; object decoders 354 and 358 are targeted at specific objects, as indicated in the corresponding object masks 308 and 310. Therefore, predicted object mask 356 can be generated by object decoder 354, and predicted object mask 360 can be generated by object decoder 358. More specifically, object decoders 354 and 358 combine the transformed features with shared features from image decoder 336 to output a prediction. and By leveraging global image features, this pipeline efficiently processes each object at a lower computational cost, as each image requires a single encoding, which generates features for both mask prediction and temporal feature encoding. Therefore, the proposed processing pipeline accelerates semi-supervised VOS in both single-object and multi-object scenarios.

[0039] Figure 4 Details of an object encoder module 402 according to an example of this disclosure are depicted. The object encoder module 402 can be used with the previously discussed object encoders 326 and 328 (…). Figure 3 The object encoder module 402 may include five convolutional layers 406, 412, 414, 416, and 418. An object mask 404, represented by a single-channel probability map between 0 and 1, is first downsampled using a lightweight convolution 406 and a pooling layer 408. Then, at residual blocks (e.g., residual 1, residual 2, residual 3), the mask features are concatenated with the features of the last convolutional layer in each stage of the object encoder. As depicted by reference numeral 410, the residual block may include convolutional layers, batch normalization layers, and rectified linear units (ReLU). After each concatenation, the features are downsampled using a strided convolutional layer (e.g., stride = 2) to ensure a fit with the size of the next concatenation. Subsequently, two additional convolutional layers 416 and 418 are used to process the features to generate object features M. a A 3x3 kernel can be used in convolutional layers to output 256 channels and ensure an appropriate receptive field. As the number of channels increases, a 1x1 kernel can be used to improve efficiency. Compared to an STM memory encoder that uses 50 convolutional layers to merge masks, the proposed architecture uses only 5 convolutional layers, ensuring the efficiency of the proposed object-specific feature encoding strategy.

[0040] As previously described, a correlation-based feature transformation is employed to leverage temporal information from previous frames. Unlike previous work that recomputes the transformation matrix for different target objects, the example in this disclosure computes the transformation matrix only once using image features, untangling the feature transformation with downstream tasks. Therefore, the feature transformation can be viewed as a feature alignment function aiming to find correspondences of features across frames. This correspondence is independent of any particular object. Thus, the feature transformation can be computed using images and further applied to features encoded for specific objects. Compared to STM, image features I... a 314 (e.g.) Figure 3 (as shown) and object feature M a These can be used as keys and values, respectively. Formally, the transformed feature can be represented as:

[0041]

[0042] in Let T represent matrix multiplication, and let T be the matrix multiplication result. a→b Let Z represent the feature transformation matrix from frame a to frame b. Z is a normalization factor, formally expressed as... Through image feature I a and I b Formulated by the pixel correlation between them, that is,

[0043] T a→b =f(I b )·f(I a ) T

[0044] Where f is a function that flattens image features along the spatial axes. It is object-agnostic and can be shared between different objects.

[0045] Figure 5 A comparison between the DiFT network and the STM network described in this paper is presented. Figure 5 The DiFT described in the paper divides the processing framework into two main parts to achieve a significant performance improvement in execution speed. First, the image processing backbone generates multi-scale image features and cross-frame feature transformations. Second, the DiFT network can include object-specific parts for merging object masks. DiFT can reuse processed image features and correlation maps for feature transformations between different object masks, thereby improving the processing speed of multi-object scenes. Furthermore, multi-scale image features can be reused to generate object-specific features using a lightweight network to fuse mask features and image features.

[0046] While both feature transformation workflows 504 and 516 utilize a two-stage design including transform generation stages 508 / 520 and transform application stages 512 / 524, the STM's transform generation stage 520 generates different transformation matrices for different objects. That is, to predict the object mask for each object in the STM, an object-specific transformation matrix is ​​generated. Therefore, the STM does not reuse the correlation maps of processed image features and feature transformations between different object masks. Thus, increasing the number of objects segmented by the STM proportionally increases the number of transformation matrices created. In contrast to the STM, the DiFT network's transform generation stage 508 uses a uniform transformation matrix for different objects (e.g., transform matrix 322 is object-independent). Therefore, increasing the number of objects segmented in DiFT does not increase the number of transformation matrices generated.

[0047] Compared to feature transformation workflow 516, this method improves the efficiency of feature transformation workflow 504 in multi-object video segmentation scenarios.

[0048] Figure 6 Details of an object decoder according to an example of this disclosure are depicted. Object decoder module 604 may be the same as or similar to the previously described object decoders 354 / 358, and may include multiple refinement modules and one or more residual networks. Similar to object encoder module 402, features from the image decoder (e.g., Figure 3 336) is used to reduce the network of object decoder module 604, such as Figure 6 As shown. The input to the object decoder module 604 is the transformed features. The receptive field is first effectively amplified by a global convolutional module 606. Following the global convolutional module 606, features are progressively merged with the image-decoded features, for example using a stitching layer 624 and convolutional layers 608, 612, and 616 with 3×3 kernels. The decoded features are upsampled at 610 and 614 according to the size of the features from the image decoder. After merging with the image features at a reduced resolution from the original image, the features are fed into an additional convolutional layer 618, followed by a softmax layer 620, to generate the final prediction 622. By reusing the image-decoded features in the object decoder module 604, the significant computational cost for predicting masks can be reduced for the same video sequence with multiple objects.

[0049] In this example, the Unwrapped Feature Transform (DiFT) network described herein can be trained using a two-step method. First, the DiFT network can be pre-trained on a simulated dataset generated from static images. Then, the DiFT network can be trained on a target video dataset. Since the DiFT network handles multi-object scenes, multi-object videos are used for both pre-training and main training.

[0050] Figure 7 Details of a method 700 for generating a predictive object mask according to an example of this disclosure are depicted. The general sequence of the steps of method 700 is shown in... Figure 7 In general, method 700 begins at 702 and ends at 722. Method 700 may include more or fewer steps, or may be combined with... Figure 7 The steps shown are arranged differently in different order. Method 700 can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer-readable medium. In the example, aspects of method 700 are executed by one or more processing devices, such as a computer or server. Furthermore, method 700 can be executed by gates or circuits associated with a processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SoC), neural processing unit, or other hardware device. The following should be referenced in conjunction with... Figure 1-6 Method 700 is explained by describing the system, components, modules, software, data structures, user interface, etc.

[0051] The method begins at 702, where the stream can proceed to 704. At 704, a sequence of image frames, such as image frame sequence 106, can be received. In the example, this can be achieved, for instance, in video capture manager 222 ( Figure 2 The image frame sequence is received at 706. The method can proceed to 706, where the object mask is received. In the example, the object mask can be the same as or similar to object mask 308. Figure 3 An object mask can conceal objects that will be segmented from a video background or otherwise from a sequence of image frames. In the example, the object mask corresponds to the first image frame in the image frame sequence. In other examples, the object mask corresponds to image frames in the image frame sequence other than the first image frame.

[0052] Method 700 can proceed to 708, where image features are generated based on the first image frame of the image frame sequence. The image features can be generated by an encoder (such as image encoder 312). Figure 3 The method 700 can proceed to 710, where image features of a second image frame based on the image frame sequence are generated. The image features can be generated by an encoder (such as image encoder 316). Figure 3The object encoder 326 generates the object features based on the received object mask and the image features associated with the first image frame. In the example, the first image frame may correspond to an image frame associated with the received object mask. The method can proceed to 712, where object features are generated based on the received object mask and the image features associated with the first image frame. Figure 3 Object features can be generated based on object mask 308 and image features generated by image encoder 312.

[0053] Method 700 can proceed to 714, where a transformation matrix is ​​computed based on image features of a first image frame and image features of a second image frame. In an example, the transformation matrix may be a similarity matrix, indicating the similarity between pixels in the first and second frames and / or between image features associated with the first and second frames. The method can proceed to 716, where transformed object features are generated by applying the transformation matrix to the generated object features. At 720, the object decoder can generate a predicted object mask for an image frame (such as the second image frame) based on the transformed object features and decoded image features associated with the second image frame. For example, object decoder 354 can generate a predicted object mask from image decoder 336 ( Figure 3 ) Receive features and receive transformed object features 348 ( Figure 3 The object decoder 354 can generate an object mask prediction 356 and provide the object mask prediction 356, either alone or as part of a mask prediction sequence, to the requesting user's computing device. Method 700 may end at 722.

[0054] Figure 8 Details of a method 800 for generating multiple prediction object masks according to an example of this disclosure are depicted. The general sequence of the steps of method 800 is shown in... Figure 8 In the middle. Typically, method 800 begins at 802 and ends at 822. Method 800 may include more or fewer steps, or may be combined with... Figure 8 The steps shown are arranged differently in different order. Method 800 can be executed as a set of computer-executable instructions that are executed by a computer system and encoded or stored on a computer-readable medium. In the example, aspects of method 800 are executed by one or more processing devices, such as a computer or server. Furthermore, method 800 can be executed by gates or circuits associated with a processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SoC), neural processing unit, or other hardware device. The following should be referenced in conjunction with... Figure 1-7 Method 700 is explained by describing the system, components, modules, software, data structures, user interface, etc.

[0055] The method begins at 802, where the stream can proceed to 804. At 804, a sequence of image frames, such as image frame sequence 106, can be received. In the example, this can be achieved, for instance, in video capture manager 222 (…). Figure 2 The image frame sequence is received at point 806. The method can proceed to 806, where first and second object masks are received. In the example, the object masks can be the same as or similar to object masks 308 and 310. Figure 3 An object mask can conceal different objects that will be segmented from a video background or otherwise from a sequence of image frames. In the example, the object mask corresponds to the first image frame in the image frame sequence. In other examples, the object mask corresponds to image frames in the image frame sequence other than the first image frame.

[0056] Method 800 can proceed to 808, where image features are generated based on the first image frame of the image frame sequence. The image features can be generated by an encoder (such as image encoder 312). Figure 3 The method 800 can proceed to 810, where image features of a second image frame based on the image frame sequence are generated. The image features can be generated by an encoder (such as image encoder 316). Figure 3 The process involves generating a transformation matrix. In the example, the first image frame may correspond to an image frame associated with a received object mask. The method can proceed to step 812, where a transformation matrix is ​​computed based on image features of the first image frame and image features of the second image frame. In the example, the transformation matrix may be a similarity matrix, indicating the similarity between pixels in the first and second frames and / or between image features associated with the first and second frames.

[0057] Method 800 can be performed as 814A, wherein a first object feature is generated based on a received first object mask and image features associated with a first image frame. For example, object encoder 326 ( Figure 3 The first object feature can be generated based on the first object mask 308 and image features generated by the image encoder 312. Furthermore, the method 800 can generate the second object feature based on the received second object mask and image features associated with the first image frame. For example, the object encoder 328 ( Figure 3 The second object features can be generated based on the second object mask 310 and the image features generated by the image encoder 316.

[0058] Method 800 can proceed to 816A and 816B, wherein transformed object features are generated by applying a transformation matrix to the generated object features. In 816A, a first transformed object feature is generated by applying a transformation matrix to the generated first object feature. In 816B, a second transformed object feature is generated by applying a transformation matrix to the generated second object feature. The method can proceed to 820A and 820B, wherein a predicted object mask for an image frame (such as the second image frame) is generated based on the second transformed object feature and decoded image features associated with the second image frame. For example, in 820A, the object decoder 354 can generate a predicted object mask from the image decoder 336 ( Figure 3 ) Receive features and receive the first transformed object features 348 ( Figure 3 Object decoder 354 can generate a first object mask prediction 356 and provide the first object mask prediction 356 to the requesting user's computing device, either alone or as part of a mask prediction sequence. In 820B, object decoder 358 can obtain the first object mask prediction 356 from image decoder 336. Figure 3 ) Receive features and receive the object features after the second transformation 352 ( Figure 3 The object decoder 358 can generate a second mask prediction 360 and provide the second mask prediction 360 to the requesting user's computing device, either alone or as part of a mask prediction sequence. Therefore, the object encoder and object decoder are specific to the received mask. Method 800 can end at 822. Advantageously, by reusing image decoding features in the object decoder, the computational cost for predicting masks in video sequences with multiple objects is significantly reduced.

[0059] Figure 9-11 The associated description provides a discussion of various operating environments in which aspects of this disclosure may be implemented. However, relative to... Figure 9-1 The devices and systems shown and discussed in 3 are for illustrative purposes and are not intended to limit the wide range of computing device configurations that can be used to practice the aspects of this disclosure described herein.

[0060] Figure 9 This is a block diagram illustrating the physical components (e.g., hardware) of a computing system 900, in which aspects of this disclosure may be implemented. The computing device components described below are applicable to the computing and / or processing devices described above. In a basic configuration, the computing system 900 may include at least one processing unit 902 and a system memory 904. Depending on the configuration and type of the computing device, the system memory 904 may include, but is not limited to, volatile memory (e.g., random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM)), flash memory, or any combination of such memories.

[0061] System memory 904 may include operating system 905 and one or more program modules 906 adapted to run software application 920, such as one or more components supported by the system described herein. As an example, system memory 904 may include video capture manager 921, object encoder module 922, image processing module 923, and feature transformation and decoding module 924. Video capture manager 921 may be the same as or similar to the previously described video capture manager 222. Object encoder module 922 may be the same as or similar to the previously described object encoder module 226. Image processing module 923 may be the same as or similar to the previously described image processing module 224. Feature transformation and decoding module 924 may be the same as or similar to the previously described feature transformation and decoding module 228. For example, operating system 905 may be adapted to control the operation of computing system 900.

[0062] Furthermore, embodiments of this disclosure can be implemented with graphics libraries, other operating systems, or any other application, and are not limited to any particular application or system. Figure 9 In the diagram, the components within the dashed line 908 illustrate this basic configuration. The computing system 900 may have additional features or functions. For example, the computing system 900 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. This additional storage... Figure 9 The image shows a removable storage device 909 and a non-removable storage device 910.

[0063] As described above, numerous program modules and data files can be stored in system memory 904. When executed on processing unit 902, program module 906 (e.g., software application 920) can perform processes including, but not limited to, the aspects described herein. Other program modules that can be used according to aspects of this disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, PowerPoint presentation applications, drawing or computer-aided programs, etc.

[0064] Furthermore, embodiments of this disclosure can be implemented on discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or single chips containing electronic components or microprocessors. For example, embodiments of this disclosure can be implemented via a system-on-a-chip (SoC), wherein... Figure 9Each or more components shown can be integrated onto a single integrated circuit. Such a SoC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all integrated (or “burned in”) as a single integrated circuit onto a chip substrate. When operating via the SoC, the functions described herein can be operated via application-specific logic integrated with other components of the computing system 900 on a single integrated circuit (chip), relative to the client’s ability to switch protocols. Embodiments of this disclosure can also be implemented using other techniques capable of performing logical operations, such as AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. Furthermore, embodiments of this disclosure can be implemented within a general-purpose computer or in any other circuit or system.

[0065] The computing system 900 may also have one or more input devices 912, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. One or more input devices 912 may include an image sensor. It may also include output devices(s) 914, such as a display, speaker, printer, etc. The above devices are examples and other devices may be used. The computing system 900 may include one or more communication connections 916 that allow communication with other computing devices / systems 950. Examples of suitable communication connections 916 include, but are not limited to, radio frequency (RF) transmitters, receivers and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports.

[0066] As used herein, the term computer-readable medium can include computer storage media. Computer storage media can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, or program modules. System memory 904, removable storage device 909, and non-removable storage device 910 are examples of computer storage media (e.g., memory storage devices). Computer storage media can include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage devices, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and is accessible by computing system 900. Any such computer storage medium may be part of computing system 900. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0067] Communication media can be implemented by computer-readable instructions, data structures, program modules, or other data (such as carrier waves or other transmission mechanisms) in modulated data signals, and include any information transmission medium. The term "modulated data signal" can describe a signal having one or more characteristics set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media can include wired media, such as wired networks or direct wired connections, and wireless media, such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0068] Figures 10A-10B A mobile computing device 1000 is illustrated, such as a mobile phone, smartphone, wearable computer (e.g., smartwatch), tablet computer, laptop computer, etc., which can be used to implement examples of this disclosure. In some examples, the mobile computing device 1000 may be the same as or similar to computing device 104. In some examples, the mobile computing device 1000 may be the same as or similar to video object segmentation server 202. In some aspects, the client may be a mobile computing device. Reference Figure 10A This illustration shows one aspect of a mobile computing device 1000 used to implement these aspects. In a basic configuration, the mobile computing device 1000 is a handheld computer with input and output elements. The mobile computing device 1000 typically includes a display 1005 and one or more input buttons 1010 that allow users to input information into the mobile computing device 1000. The display 1005 of the mobile computing device 1000 can also be used as an input device (e.g., a touchscreen display).

[0069] If included, the optional side input element 1015 allows for further user input. The side input element 1015 can be a rotary switch, a button, or any other type of manual input element. Alternatively, the mobile computing device 1000 may incorporate more or fewer input elements. For example, in some embodiments, the display 1005 may not be a touchscreen.

[0070] In yet another alternative embodiment, the mobile computing device 1000 is a portable telephone system, such as a cellular phone. The mobile computing device 1000 may also include an optional keypad 1035. The optional keypad 1035 may be a physical keypad or a “soft” keypad generated on a touchscreen display.

[0071] In various embodiments, output elements include a display 1005 for displaying a graphical user interface (GUI), a visual indicator 1020 (e.g., a light-emitting diode), and / or an audio transducer 1025 (e.g., a speaker). In some aspects, the mobile computing device 1000 incorporates a vibration transducer for providing haptic feedback to a user. In another aspect, the mobile computing device 1000 includes input and / or output ports for sending signals to or receiving signals from external devices, such as an audio input port (e.g., a microphone jack), an audio output port (e.g., a headphone jack), and a video output port (e.g., an HDMI port).

[0072] Figure 10B This is a block diagram illustrating the architecture of one aspect of a mobile computing device. That is, the mobile computing device 1000 may incorporate system (e.g., architecture) 1002 to achieve certain aspects. In one embodiment, system 1002 is implemented as a "smartphone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, system 1002 is integrated as a computing device, such as integrating a personal digital assistant (PDA) and a wireless phone.

[0073] One or more applications 1066 may be loaded into memory 1062 and run on or associated with operating system 1064. Examples of applications include telephone dialers, email programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, internet browser programs, messaging programs, map programs, etc. System 1002 also includes a non-volatile storage area 1068 within memory 1062. The non-volatile storage area 1068 can be used to store persistent information that should not be lost when system 1002 is powered off. Applications 1066 may use and store information, such as emails or other messages used by email applications, in the non-volatile storage area 1068. A synchronization application (not shown) also resides on system 1002 and is programmed to interact with a corresponding synchronization application residing on the host to keep the information stored in the non-volatile storage area 1068 synchronized with the corresponding information stored on the host. It should be understood that other applications can be loaded into memory 1062 and run on the mobile computing device 1000 described herein (e.g., video capture manager 921, object encoder module 922, image processing module 923, and feature transformation and decoding module 924, etc.).

[0074] System 1002 has a power supply 1070, which can be implemented as one or more batteries. The power supply 1070 may also include an external power source, such as an AC adapter or power docking bracket for supplementing or recharging batteries.

[0075] System 1002 may also include a radio interface layer 1072, which performs the functions of transmitting and receiving radio frequency communications. Radio interface layer 1072 facilitates wireless connectivity between system 1002 and the "external world" through a communications operator or service provider. Transmissions to and from radio interface layer 1072 are conducted under the control of operating system 1064. In other words, communications received by radio interface layer 1072 can be propagated to application 1066 via operating system 1064, and vice versa.

[0076] A visual indicator 1020 may be used to provide visual notifications, and / or an audio interface 1074 may be used to generate audible notifications via an audio transducer 1025. In the illustrated embodiment, the visual indicator 1020 is a light-emitting diode (LED), and the audio transducer 1025 is a speaker. These devices may be directly coupled to a power supply 1070 so that when activated, they remain on for the duration specified by the notification mechanism, even if the processor 1060 and other components may be turned off to conserve battery power. The LED may be programmed to remain lit indefinitely until the user takes action to indicate the device's power-on status. The audio interface 1074 is used to provide and receive audible signals to and from the user. For example, in addition to being coupled to the audio transducer 1025, the audio interface 1074 may also be coupled to a microphone to receive sound input, such as for telephone conversations. According to embodiments of this disclosure, the microphone may also be used as an audio sensor to control notifications, as described below. System 1002 may also include a video interface 1076, which enables the onboard camera 1030 to operate to record still images, video streams, etc.

[0077] The mobile computing device 1000 implementing system 1002 may have additional features or functions. For example, the mobile computing device 1000 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. Such additional storage devices in Figure 10B The non-volatile storage region 1068 is shown in the middle.

[0078] As described above, data / information generated or captured by mobile computing device 1000 and stored via system 1002 can be stored locally on mobile computing device 1000, or the data can be stored on any number of storage media accessible by the device via radio interface layer 1072, or accessed via a wired connection between mobile computing device 1000 and an independent computing device associated with mobile computing device 1000 (e.g., a server computer in a distributed computing network, such as the Internet). It should be understood that such data / information can be accessed via mobile computing device 1000 via radio interface layer 1072 or via a distributed computing network. Similarly, such data / information can be easily transferred between computing devices used for storage and use based on well-known data / information transmission and storage devices (including email and collaborative data / information sharing systems).

[0079] Figure 11 This illustrates one aspect of the architecture of a system, as described above, for processing data received at a computing system from a remote source, such as a personal computer 1104, a tablet computing device 1106, or a mobile computing device 1108. The personal computer 1104, tablet computing device 1106, or mobile computing device 1108 may include one or more applications 1120; such applications may include, but are not limited to, video capture managers, object encoder modules, image processing modules, and feature transformation and decoding modules. Content at the server device 1102 may be stored in different communication channels or other storage types. For example, various documents may be stored using directory services, web portals, video restoration services, instant messaging storage, or social networking services.

[0080] As described above, one or more of the previously described program modules 906 or software applications 920 can be used by server device 1102 and / or personal computer 1104, tablet computing device 1106, or mobile computing device 1108. For example, server device 1102 may include a video capture manager 1121, an object encoder module 1122, an image processing module 1123, and a feature transformation and decoding module 1124. Video capture manager 1121 may be the same as or similar to the previously described video capture manager 222. Object encoder module 1122 may be the same as or similar to the previously described object encoder module 226. Image processing module 1123 may be the same as or similar to the previously described image processing module 224. Feature transformation and decoding module 1124 may be the same as or similar to the previously described feature transformation and decoding module 228.

[0081] Server device 1102 can provide data to and from client computing devices (such as personal computer 1104, tablet computing device 1106, and / or mobile computing device 1108 (e.g., smartphone)) via network 1115. For example, the aforementioned computer system can be implemented in personal computer 1104, tablet computing device 1106, and / or mobile computing device 1108 (e.g., smartphone). In addition to receiving content that can be preprocessed at the graphics initiation system or post-processed at the receiving computing system, any of these embodiments of the computing device can also obtain content from storage repository 1116.

[0082] Furthermore, the aspects and functions described herein can operate on distributed systems (e.g., cloud-based computing systems), where application functions, memory, data storage and retrieval, and various processing functions can remotely operate on each other via distributed computing networks (such as the Internet or intranets). Various types of user interfaces and information can be displayed via onboard computing device displays or via remote display units associated with one or more computing devices. For example, various types of user interfaces and information can be displayed on and interacted with on a wall, or projected onto a wall. Interaction with numerous computing systems that can implement embodiments of the present invention includes keystroke input, touchscreen input, voice or other audio input, gesture input, wherein the associated computing device is equipped with detection (e.g., camera) functions for capturing and interpreting user gestures used to control the functions of the computing device, etc.

[0083] This disclosure relates to systems and methods for performing video object segmentation, based on examples provided at least in the following sections:

[0084] (A1) In one aspect, some examples include a method for performing video object segmentation. The method may include: receiving video data representing a sequence of image frames; receiving an object mask; generating image features based on a first image frame of the image frame sequence; generating object features based on the object mask; generating image features based on a second image frame of the image frame sequence; calculating a transformation matrix based on the image features of the first and second image frames; applying the transformation matrix to the object features resulting in transformed object features; and obtaining a predicted object mask associated with the second image frame by decoding the transformed object features.

[0085] (A2) In some examples of A1, the method includes receiving a second object mask, generating second object features based on the second object mask, applying a transformation matrix to the second object features based on the second object mask to generate second transformed object features, and obtaining a second predicted object mask associated with a second image frame by decoding the second transformed object features.

[0086] (A3) In some examples of A1-A2, object features are generated based on an object mask and image features based on a first image frame, and second object features are generated based on a second object mask and image features based on a first image frame.

[0087] (A4) In some examples of A1-A3, the predicted object mask associated with the second image frame is based on the transformed object features and the decoded object features associated with the second image frame, and the second predicted object mask associated with the second image frame is based on the second transformed object features and the decoded object features associated with the second image frame.

[0088] (A5) In some examples of A1-A4, the method includes obtaining a predicted object mask associated with a second image frame by decoding the transformed object features at a decoder different from the decoder that decodes the object features associated with the second image frame.

[0089] (A6) In some examples of A1-A5, the method includes generating image features at an encoder based on a first image frame of an image frame sequence, which is different from an encoder that generates object features based on an object mask.

[0090] (A7) In some examples of A1-6, the transformation matrix includes similarity information indicating the similarity between the first image frame and the second image frame.

[0091] In another aspect, some examples include computing systems that include one or more processors and memory coupled to the one or more processors, the memory storing instructions configured to be executed by the one or more processors, which, when executed by the one or more processors, cause the one or more processors to perform any of the methods described herein (e.g., A1-A7 described above).

[0092] In another aspect, some examples include a non-transient computer-readable storage medium storing one or more programs for execution by one or more processors of a storage device, the one or more programs including instructions for performing any of the methods described herein (e.g., A1-A7 described above).

[0093] For example, various aspects of this disclosure are described above with reference to block diagrams and / or operational schematics of methods, systems, and computer program products according to various aspects of this disclosure. Functions / actions indicated in a block may not occur in the order shown in any flowchart. For example, two blocks shown sequentially may, in fact, be executed substantially simultaneously, or sometimes may be executed in reverse order, depending on the functions / actions involved.

[0094] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the claimed disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to convey the best mode of possession and to enable others to make and use the claimed disclosure. The claimed disclosure should not be construed as limited to any aspect, example, or detail provided in this application. Various features (structural and methodological) are intended to be selectively included or omitted, whether shown and described in combination or separately, to produce embodiments with a particular set of features. After the description and illustration of this application have been provided, those skilled in the art can conceive of variations, modifications, and alternatives within the spirit and scope of the broader aspects of the general inventive concept practiced in this application without departing from the wider scope of the claimed disclosure.

Claims

1. A method for performing video object segmentation, the method comprising: Receive video data representing a sequence of image frames; Receive the first object mask; Image features are generated based on the first image frame of the image frame sequence; Generate first object features based on the first object mask; Image features are generated based on the second image frame of the image frame sequence; The transformation matrix is ​​calculated based on the image features of the first image frame and the image features of the second image frame; The transformation matrix is ​​applied to the first object feature to generate the first transformed object feature; as well as A first predicted object mask associated with the second image frame is obtained by decoding the first transformed object features.

2. The method of claim 1, wherein the first predicted object mask associated with the second image frame is obtained based on the first transformed object features and the decoded image features associated with the second image frame.

3. The method according to claim 1, further comprising: Receive the second object mask; Generate second object features based on the second object mask; The transformation matrix is ​​applied to the second object feature based on the second object mask to generate the second transformed object feature; as well as A second predicted object mask associated with the second image frame is obtained by decoding the second transformed object features.

4. The method of claim 3, wherein the first object feature is generated based on the first object mask and the image features based on the first image frame, and the second object feature is generated based on the second object mask and the image features based on the first image frame.

5. The method of claim 4, wherein the first predicted object mask associated with the second image frame is generated based on the first transformed object features and the decoded image features associated with the second image frame, and wherein the second predicted object mask associated with the second image frame is generated based on the second transformed object features and the decoded image features associated with the second image frame.

6. The method according to claim 5, further comprising: The first predicted object mask associated with the second image frame is obtained by decoding the first transformed object features at a decoder, which is different from the decoder that decodes the first object features associated with the second image frame.

7. The method according to claim 5, further comprising: The image features are generated at the encoder based on the first image frame of the image frame sequence, the encoder being different from the encoder that generates the first object features based on the first object mask.

8. A system configured to perform video object segmentation, the system comprising: processor; as well as The memory includes instructions that, when executed by the processor, cause the processor to: Receive video data representing a sequence of image frames; Receive the first object mask; Image features are generated based on the first image frame of the image frame sequence; Generate first object features based on the first object mask; Image features are generated based on the second image frame of the image frame sequence; The transformation matrix is ​​calculated based on the image features of the first image frame and the image features of the second image frame; The transformation matrix is ​​applied to the first object feature to generate the first transformed object feature; as well as A first predicted object mask associated with the second image frame is obtained by decoding the first transformed object features.

9. The system of claim 8, wherein the instructions, when executed by the processor, cause the processor to: Receive the second object mask; Generate second object features based on the second object mask; The transformation matrix is ​​applied to the second object feature based on the second object mask to generate the second transformed object feature; as well as A second predicted object mask associated with the second image frame is obtained by decoding the second transformed object features.

10. The system of claim 9, wherein the first object feature is generated based on the first object mask and the image features of the first image frame, and the second object feature is generated based on the second object mask and the image features of the first image frame.

11. The system of claim 10, wherein the first predicted object mask associated with the second image frame is generated based on the first transformed object features and the decoded image features associated with the second image frame, and wherein the second predicted object mask associated with the second image frame is generated based on the second transformed object features and the decoded image features associated with the second image frame.

12. The system of claim 11, wherein the instructions, when executed by the processor, cause the processor to: obtain the first predicted object mask associated with the second image frame by decoding the first transformed object features at a decoder, the decoder being different from the decoder that decodes the first object features associated with the second image frame.

13. The system of claim 8, wherein the instructions, when executed by the processor, cause the processor to: generate the image features at an encoder based on the first image frame of the image frame sequence, the encoder being different from an encoder that generates the first object features based on the first object mask.

14. The system of claim 8, wherein the transformation matrix includes similarity information indicating the similarity between the first image frame and the second image frame.

15. A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to: Receive video data representing a sequence of image frames; Receive the first object mask; Image features are generated based on the first image frame of the image frame sequence; Generate first object features based on the first object mask; Image features are generated based on the second image frame of the image frame sequence; The transformation matrix is ​​calculated based on the image features of the first image frame and the image features of the second image frame; The transformation matrix is ​​applied to the first object feature to generate the first transformed object feature; as well as A first predicted object mask associated with the second image frame is obtained by decoding the first transformed object features.

16. The computer-readable storage medium of claim 15, wherein the instructions, when executed by a processor, cause the processor to: Receive the second object mask; Generate second object features based on the second object mask; The transformation matrix is ​​applied to the second object feature based on the second object mask to generate the second transformed object feature; as well as A second predicted object mask associated with the second image frame is obtained by decoding the second transformed object features.

17. The computer-readable storage medium of claim 16, wherein the first object feature is generated based on the first object mask and the image features based on the first image frame, and the second object feature is generated based on the second object mask and the image features based on the first image frame.

18. The computer-readable storage medium of claim 17, wherein the first predicted object mask associated with the second image frame is generated based on the first transformed object features and the decoded image features associated with the second image frame, and wherein the second predicted object mask associated with the second image frame is generated based on the second transformed object features and the decoded image features associated with the second image frame.

19. The computer-readable storage medium of claim 18, wherein the instructions, when executed by a processor, cause the processor to: obtain the first predicted object mask associated with the second image frame by decoding the first transformed object features at a decoder, the decoder being different from the decoder that decodes the first object features associated with the second image frame.

20. The computer-readable storage medium of claim 15, wherein the instructions, when executed by a processor, cause the processor to: generate the image features at an encoder based on the first image frame of the image frame sequence, the encoder being different from an encoder that generates the first object features based on the first object mask.

Images