End-to-end merging for video object segmentation (VOS)

Abstract

A method and computing device are provided for performing end-to-end video object segmentation using semi-supervised learning, tracking corresponding objects from single frame annotations of a reference frame through a sequence of video frames. The reference frame can be annotated using a known deep learning model to provide original labeled locations and masks for each corresponding object. A current frame is processed to determine the current frame object location, and an object score map is defined as the normalized cross-correlation between the encoded object features of the current frame and the encoded object features of the previous frame. A score map can be defined for each of more than one previous frames. Candidate object proposals defined from the reference frame annotations are sorted in response to an intersection over union (IoU) function of the score map to associate the corresponding object with the corresponding location in the current frame. A merge function responsive to the score map can be used to remove pixel-level overlap.

Description

[0001] Cross-references

[0002] For the United States, this application claims Paris Convention priority to U.S. Provisional Application No. 62 / 935,851, filed on November 15, 2019, and respective domestic interests in other territories, the contents of which are incorporated herein by reference where permitted. Technical Field

[0003] The present disclosure relates to image processing using neural networks, and more particularly to video object segmentation (VOS), tracking and segmenting one or more objects through a video clip. Background Art

[0004] Object segmentation typically involves processing an image (e.g., a still image) to determine the presence and location of one or more significant (e.g., common, everyday) objects, outputting a mask or other construct that identifies the pixels in the image where each object is located. Video object segmentation is similar, however a video is a sequence of frames over a period of time, where each frame defines an image, and the location of objects in different frames may be different. For example, a video may start with an initial frame and advance to subsequent frames in an ordered sequence. A video clip may be the entire video or a portion thereof, starting with an initial frame (e.g., a reference frame) and continuing through subsequent frames until the end of the clip. An object that appears in a reference frame may move in location from one frame to another, for example, because the object moves relative to the camera, the camera moves relative to the object, or both the camera and the object move. Tracking and segmenting objects through a video clip is desirable for many reasons. For example, it may be desirable to apply effects to objects in each frame and provide output as an augmented reality, virtual reality, or mixed reality experience. Tracking and segmenting objects may also have other purposes.

[0005] Tracking and segmenting objects through video is a challenge for computing devices. Image processing techniques for identifying and segmenting objects in a single static image are well known, using various network models trained with supervised learning techniques. However, tracking an object from one image to the next in a video is problematic. Known networks are not trained for such tasks. One problem is that appropriate training data for supervised learning is not widely available. It is problematic to associate an object detected in a first image with the same object detected in a second image of a video, thereby tracking the object. Summary of the invention

[0006] A method for semi-supervised video object segmentation is presented, which involves tracking and segmenting one or more objects throughout a video clip, conditioned on a single-frame reference annotation for each object track in a sequence of frames of the video clip. The method exploits prior knowledge about the appearance and shape of salient objects from image-level datasets such as COCO, while also learning an end-to-end merging of per-frame predictions using a VOS dataset such as YouTube-VOS (YouTube is a trademark of GOOGLE LLC). Those skilled in the art will appreciate these and other aspects, and such skilled persons will also appreciate that, although these aspects may be summarized or described separately for convenience, they may be combined and / or used together.

[0007] A method and computing device for performing end-to-end video object segmentation using semi-supervised learning is provided, where corresponding objects are tracked from single frame annotations of a reference frame through a sequence of video frames. The reference frame can be annotated using a known deep learning model to provide a ground truth location and mask for each corresponding object. A current frame is processed to determine the current frame object location, and an object score map (also referred to herein as an attention map) is defined as the normalized cross-correlation between the encoded object features of the current frame and the encoded object features of the previous frame. A score map can be defined for each of more than one previous frames. Candidate object proposals defined from the reference frame annotations are ranked in response to an intersection-over-union (IoU) function of the score map to associate the corresponding object with the corresponding position in the current frame. A merge function responsive to the score map can be used to remove pixel-level overlap. The score map and ranking function find a probability or normalized score distribution representing the likelihood that the corresponding object belongs to the corresponding position in the current frame.

[0008] In one embodiment, a semi-supervised video object segmentation method is provided for tracking and segmenting one or more objects throughout a video clip, the video clip comprising a sequence of frames, the sequence of frames comprising a previous frame and a current frame, the previous frame defining a target, and the current frame defining a source. The method comprises: encoding features of each corresponding object in the source; defining an attention map between features encoded from the source and features of the corresponding object encoded from the target; associating the corresponding object with a corresponding position in the current frame using an intersection over union (IoU) function responsive to the attention map to rank candidate object proposals for the corresponding position, wherein each candidate object proposal is tracked from a single frame reference annotation of a reference frame of the video clip, the reference frame providing a ground truth position of the corresponding object in the reference frame; and defining a video segmentation mask for the corresponding object in the current frame based on the association.

[0009] In one example, a semi-supervised video object segmentation method is provided for tracking and segmenting one or more objects throughout a video clip, the video clip comprising a sequence of frames, the sequence of frames comprising corresponding previous frames and a current frame, each of the corresponding previous frames defining a corresponding target, the current frame defining a source, the method comprising: for each corresponding object: encoding features of the corresponding object in the source; and for each corresponding target, defining a corresponding attention map between features encoded from the source and features of the corresponding object encoded from the corresponding target. The corresponding object is associated with a corresponding position in the current frame using an intersection-over-union (IoU) function responsive to the corresponding attention map to rank candidate object proposals for the corresponding position, wherein each candidate object proposal is tracked from a single frame reference annotation of a reference frame of the video clip, the reference frame providing a ground truth position of the corresponding object in the reference frame; and a video segmentation mask is defined for the corresponding object in the current frame based on the association.

[0010] In one example, a method is provided, the method comprising: training each of an encoder model and a locator model using semi-supervised learning to track and segment one or more corresponding objects in a video clip, the video clip comprising a plurality of frames in a sequence, the training being conditioned on a single frame reference annotation of a reference frame for each corresponding object in the video clip. In one example: the encoder model comprises a neural network model configured to encode object features; encoding object features in a reference frame, a current frame, and a previous frame using the encoder model during training to generate corresponding object features from each frame; and locating the object using the locator model during training, and forcing the encoder model to learn to discriminate object features so as to be able to track the corresponding object from the reference frame to the current frame via the previous frame.

[0011] Computing device and computer program product aspects are also provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Image pairs (frames of a video) from a virtual try-on application with annotations are shown.

[0013] Figure 2 is a block diagram illustrating a training system architecture according to an embodiment.

[0014] Figure 3 is a block diagram illustrating a tracking submodule according to an embodiment.

[0015] Figure 4 is a block diagram illustrating an inference system architecture according to an embodiment.

[0016] Figure 5 is a block diagram of a computer network of computing devices according to an embodiment.

[0017] Figure 6 ,7 , 8A, 8B and 9 are flowcharts of corresponding operations of computing devices according to corresponding embodiments. DETAILED DESCRIPTION

[0018] In one embodiment, the first component of the method involves learning discriminative features for tracking the motion of individual pixels in the video by learning frame-to-frame correspondences of them, and this knowledge can be used in downstream augmented reality (AR) tasks. For example, the correspondences can be used to track the motion of a 2D or 3D nail model in order to stabilize Figure 1 Rendering of textures and lighting in the virtual nail polish try-on application shown.

[0019] Figure 1 An image pair 100 (respectively 100A, 100B) from a virtual try-on application is shown with annotations (e.g., 102) showing correspondences, according to an embodiment. Image 100 shows an example of sparse correspondences (lines connecting points) in consecutive frames 100A, 100B of a first-person nail polish try-on application. For visualization, correspondences 102 are shown sparsely, while the methods herein produce dense correspondences for each nail. Dense correspondences can be used for downstream stable rendering in augmented reality applications, such as Figure 1 That is, in one embodiment, the augmented reality application is configured to track the object from frame to frame, and in response to the tracking of the object, the object is annotated, for example, its color or other appearance characteristics change from the source image in each frame.

[0020] Other applications, such as hair color and makeup try-on, and clothing try-on, will also benefit from the temporal smoothing of presentations achieved by the correspondences generated by our method. In these AR applications, correspondences are needed for temporally consistent tracking, and video object segmentation is needed to remove outliers, e.g., to prevent nail polish from being presented outside the nail region of the video.

[0021] Furthermore, the final video object segmentation output of the systems and methods shown and described herein according to the examples can be directly used in tracking applications, such as player tracking in motion analysis, person tracking in security footage, and car and road obstacle tracking in autonomous vehicle applications.

[0022] The disclosure herein demonstrates a method for video object segmentation that uses prior knowledge learned from more readily available static datasets. According to the examples and techniques herein, a method for ranking object proposals (e.g., an object proposal ranking function) is shown, which utilizes a "soft mIoU" (mean intersection over union) method to rank object proposals of a cross-correlation score map of a given object proposal with a reference object proposal. According to the examples and techniques herein, a method for mask merging is also shown: in the case where the bounding boxes of multiple object tracks overlap, each pixel in the overlapping (object) region is assigned to its true object track.

[0023] Our approach is distinct from previous work in video object segmentation (VOS) and feature matching. Following our examples and techniques, we produce a VOS architecture that unifies object proposal and mask refinement models by predicting frame-to-frame correspondence score maps.

[0024] Method and computing device / system

[0025] Training system architecture

[0026] Figure 2 2 is a block diagram illustrating data and processing flow of a training system architecture 200 (e.g., as implemented in a computing device) according to an embodiment. As proposed herein, the system architecture differs at training and inference time because during training time, an encoder component ("encoder") is trained to extract features that a locator component ("locator") can use to locate an object. At training time, coupling is used between the locator and the encoder to force the encoder to learn discriminative features for tracking.

[0027] Figure 2 A training system architecture 200 is shown, where a video clip 202 (e.g., a sequence of frames) is first processed by a known model (KM) 204 to extract bounding boxes and instance segmentation proposals for corresponding objects 206. Figure 2 The video clip 202 is shown to include 4 frames, but this is merely an illustrative representation. Video clips typically have more than 4 frames. Similarly, although the KM block 204 is shown to output results for 4 corresponding objects, this is merely an illustrative representation. More or fewer corresponding objects may be detected (for each corresponding object detected, a corresponding bounding box and instance segmentation proposal are provided). In implementation, Mask R-CNN (region-based convolutional neural network) is used as the KM, but other models may also be used for object detection.

[0028] The known model 204 may include a deep learning model that is conditioned using one or more image-level datasets that uses supervised learning to identify objects in an image and provides corresponding bounding boxes and segmentations for each identified object (e.g., 206). As further described, candidate proposals for objects in the current frame are defined from corresponding bounding boxes generated for a reference frame or bounding boxes of objects tracked from a reference frame.

[0029] The video clip (eg, its frame) is also input to the tracking submodule 208 (see Figure 3 ), which outputs a set of sampling grids 212 {G(θ t )}, the sampling grid is defined by the locator component 208 (see Figure 3 ) The predicted source-to-target distortion.

[0030] In this example of a training system architecture, three losses, tracking loss, similarity loss, and dissimilarity loss, are applied on the predicted sampling grid to train discriminative features for tracking.

[0031] loss

[0032] In the following description of the system loss function, θ t-1 is θ calculated by the localizer using the previous frame, θ 0 is θ calculated by the localizer using the reference frame, θ t is the ground truth value θ of the current frame, G(θ t ) corresponds to using the transformation parameter θ t The sampling grid used to extract patches from image features, and P(θ t ) corresponds to using θ t Extracted image features.

[0033] Tracking: Tracking loss is defined as,

[0034] L 跟踪 =L 1 (G(θ t-1 ),G(θ t ))+L 1 (G(θ 0 ),G(θ t )) (1)

[0035] Where L 1 It is a smooth L 1 loss.

[0036] Similarity: The predicted patches and the ground truth patches should be similar in the feature space, which is enforced by introducing a similarity loss as shown below.

[0037] L相似性 =- <P(Θ t-1 ),P(Θ t )>- <P(Θ 0 ),P(Θ t )> (2)

[0038] Dissimilarity: A dissimilarity loss is introduced to distinguish the target object track from other object tracks, and to distinguish the target object track from the background. It also pushes the encoder to not only output the same content, but also make the features similar.

[0039] L 相异性 = <P(Θ t-1 ),P(Θ t )>+ <P(Θ 0 ),P(Θ t )> (3)

[0040] Tracking submodule

[0041] In the example, the following description Figure 3 Blocks of the tracking submodule. Except for the localizer, many blocks (e.g., components) of the tracking submodule are reused in training and inference (e.g., at user runtime) because their purpose is only to strengthen the training of discriminative tracking features.

[0042] Encoder

[0043] In the example, the Siamese Network is used as an encoder for the current 302, reference 304, and previous frames 306. In the example, each of the encoders 308, 310, and 312 of the Siamese Network is a residual network ResNet-50 architecture without the fifth residual block. In the implemented example, the input frame is 3×240×240 (indicated by a line with a single vertical line “|” annotation), and the output feature has a shape of 512×30×30 (indicated by a line with a single forward line annotation “ / ”). Figure 3 In , the annotation line with two vertical lines annotation "||" has shape 512×14×14, and the annotation line with one backward slanted line "\" annotation has shape 900×14×14. It should be understood that in the examples herein, where specific dimensions are referenced, the dimensions may be different in various implementations. At inference time, the encoder only needs to be run once per frame, and at time step t, the features extracted from the "previous frame" frame t-1 are reused in the previous time step (when it is the "current frame").

[0044] In the example, two target images are used as input: a reference frame 304 and a previous frame 306. Both the reference frame 304 and the previous frame 306 are used because they serve complementary purposes: as the video sequence progresses, the previous frame is usually the most similar to the current frame, however, a reference frame may still be needed to recover from errors in the prediction of the previous frame, for example, due to "attenuation", "blurring" or occlusion. It should be understood that in the example, the method can be configured to use only previous frames as targets. In fact, any non-empty subset of the power set of all previous frames can be used as targets. When more than one target is used, the sorting results may be fused. Described below are fusion functions (e.g., Equations 21 and 22) that work on any number of targets.

[0045] use The method is described to represent one of the two target images (e.g., frames), with the understanding that the same operations are applied to both target images in parallel. At training and inference time, due to the nature of the single-shot video object segmentation task (e.g., processed by a known model to provide such locations), the location of the object is known in the reference frame 304 (e.g., the reference frame ground truth bounding box 314 from 206). At training time, in the example, the "teacher is forced" by the known object location in the previous frame (e.g., the previous frame ground truth bounding box 316). At inference time, when the same frame is used as the "current frame", the previous frame object location predicted in the previous time step is used. The target image (e.g., reference frame) is defined as The source image (current frame) is defined as

[0046] A feature extractor is a function:

[0047]

[0048] Among them, for the encoder step length stride F , H f =H / stride F and W f =W / stride F .

[0049] Therefore, the source and target features are:

[0050]

[0051] Bbox feature clipping

[0052] In the example, the ground truth bounding box (e.g., 314 or 316) of the object in the target image (i.e., the reference object, or the predicted object in the previous frame) is downsampled to obtain a bbox F , then use bbox FClip feature f to get

[0053] Attention Map (Score Map)

[0054] In the example, the normalized cross-correlation function is defined,

[0055]

[0056] It acts on the source feature f s and the cropped target feature f tc To generate the attention map,

[0057]

[0058] In the example, by first defining the unnormalized cross-correlation score map element To define the elements of equation 8, and then define the elements of s as,

[0059]

[0060] in,‖·‖ F is the Frobenius norm, and is the normalized eigenvector.

[0061] Attention map resizing

[0062] In the example, the attention map s is resized to a fixed size along its source image axes 0 and 1, regardless of the source image aspect ratio. Since the localizer takes a normalized grid in [-1, 1] × [-1, 1] as input and assumes that the corresponding scores stored in the attention map are locally linearly related, bilinear resizing (via bilinear samplers 318 and 320) is an effective transformation of the attention map. The attention map s is resized along its target image axes 2 and 3.

[0063] Geometric transformation estimation

[0064] The localizer (322 or 324 for the corresponding target) G predicts the normalized 2D warp T g , [-1,1]×[-1,1] to the location of the target object in the source image. In other words, the localizer predicts the location of the tracked object in the current frame.

[0065] In the example, the locator is defined as follows:

[0066]

[0067] Where K is the number of degrees of freedom of the geometric transformation. The input of the localizer is the attention map, and the output is the definition of the 2D distortion T gThe transformation parameters are, for example, Corresponds to the localization of the object in the target image.

[0068] The localizer G consists of a 2D CNN that uses the reshaped shape attention map As input, and output the distortion parameter vector The shape is intentionally HW×H'×W': the source feature map pixels are flattened into vectors so that the tensor can be processed by the 2D CNN (which accepts 3D tensors of shape C×H×W, where C represents channels). The corresponding θ outputs 322 and 324 for training are also provided.

[0069] Inferring system architecture

[0070] In the example, Figure 4 4 is a block diagram illustrating a system architecture 400 for tracking an object in a video frame 401 at inference time. The system architecture 400 uses a trained tracking module 402 (e.g., having a trained model) instead of the tracking module 208 having a model to be trained (or being trained). As described below, the trained tracking module 402 outputs prediction data 404. This "runtime" architecture 400 builds on the training system architecture 200 by adding corresponding proposal sorting and mask merging modules (e.g., 406 and 410), which will be described below. The trained tracking module 402 does not have the localizer of the tracking module 208, which is only used to enforce the training of discriminative tracking features. Thus, at inference time, the output 404 of the tracking module 402 is a source-target attention score map, which the proposal sorting and mask merging modules (406 and 410) each utilize to first sort the object proposals (e.g., output 408) and then resolve ambiguity in merging any sets of object proposals that may overlap (e.g., including selected proposals 412 with other object tracks) to produce a final output 414.

[0071] Sorting suggestions

[0072] In the example, two formulas for soft mIoU are proposed as ranking functions, which are metrics (i.e., operations or functions) for ranking candidate object (e.g., candidate bounding box) proposals responsive to the locations of corresponding objects in a reference frame. As described above, when processing a video segment, the corresponding objects defining the candidate object proposals and their locations in the reference frame are determined only once using the known model 204. These corresponding objects are tracked through the frames of the video, and with each processing of a new frame in the video (e.g., the current frame), the operation here associates the corresponding objects with the corresponding locations in the current frame. The operation determines a probability or normalized score distribution representing the likelihood that the corresponding object belongs to the corresponding location in the current frame. The ranking function is selected based on the likelihood.

[0073] As discussed further below with respect to mask merging, for any particular pixel in the current frame, two or more corresponding objects may overlap at that location (e.g., because the video frame is a 2D representation of a 3D space). Only one such object may be in the immediate or front-most foreground. The mask merging function of module 410 assigns a particular pixel to one of the corresponding objects by selecting the maximum score from the probability or normalized score distribution for each corresponding object at the location of the particular pixel.

[0074] Back to the ranking function, for the two soft mIoU formulas, the common input is assumed as follows. The input includes the corresponding score map (e.g., normalized cross-correlation) Among them, the source frame dimension is H s ×W s , the target frame dimension is H t ×W t From the output 206 of the known model 206, there are B candidate bounding box proposals, where Together with the corresponding candidate instance segmentation mask proposals in

[0075] True / False Positive / Negative Formula

[0076] In this example, the first formula of soft mIoU reinterprets the usual formula of mIoU, i.e., true and false positives and negatives, using the correspondence score map s. The attention map is used to define the "soft mIoU" on the object proposal using its predicted mask m, which is assumed to be the instance segmentation mask of the object proposal produced by the known model 206 (e.g., Mask R-CNN).

[0077] First, s is normalized so that max s=1.

[0078] The true positive TP, false positive FP, and false negative FN of the foreground (i.e., object) are defined as follows,

[0079] TP f ≡∑ kl m kl max s kl∷ , (11)

[0080] FP f ≡∑ kl m kl (1-max s kl∷ ), (12)

[0081]

[0082] The definition of true / false positives and negatives for background is similar to:

[0083]

[0084] FN b ≡∑ kl m kl (1-max s kl∷ ). (16)

[0085] The motivation behind defining TP, FP, and FN is that since ∑ ij s ijkl = 1, the positive correspondence prediction between a pixel (k, l) in the source image and the target image is measured by the entropy of the score distribution of all pixels in the source image s kl∷ .

[0086] The definitions of soft true, false positive, and false negative are used to define the soft mean IoU. According to the usual definition of mIoU,

[0087]

[0088] When predicting, choose the solution with the highest soft mIoU.

[0089] Intersection and Union Ratio Formula

[0090] In the second formula, the “intersection over union” formula for soft mIoU directly reinterprets intersection I and union U using the corresponding score map, and then defines soft mIoU as I / U.

[0091] Assume that each target frame pixel corresponds to a source frame pixel, and thus normalize s so that ∑ kl s klij = 1 as shown in Equation 9. After normalization, the intuitive measure of the correspondence between the reference pixel (i, j) and the source solution is Among them, m s is the instance segmentation mask of the current candidate.

[0092] The intersection I is defined as,

[0093]

[0094] And U is,

[0095]

[0096] Among them, soft mIoU is I / U.

[0097] Intuitively, the “soft intersection” defined in Equation 18 is the sum of all corresponding scores belonging to the source proposal and the reference patch, i.e., their intersection in the correspondence score map. The “soft union” given in Equation 19 is all pixels in the reference patch, plus the sum of their corresponding scores belonging to the source proposal, minus the intersection defined in Equation 18. The first and second terms in Equation 19 are at least as large as I, since s is normalized so that ∑ kl s klij =1.

[0098] Mask Merge

[0099] The refined masks extracted from multiple selected object proposals can overlap, in which case the pixel belongs to the object whose proposal has the largest single correspondence score for the slice of the correspondence map at pixel (k, l) in the target image.

[0100]

[0101] Predict the pixels in the video segmentation of the current frame Set to in, belongs to the corresponding to from Equation 22 object trajectory.

[0102] Sort and merge fusion

[0103] The system uses two target frames: a given reference frame and a predicted previous frame to produce a correspondence score map s (r) and soft mIoU score y (r) , where r represents the index of the target frame, and in the current case r∈(0,1) indexes the reference frame and the previous frame. In order to (r) and (r) The final sorting is produced from the collection of , and a fusion function is defined to produce a single proposal sort and mask merge order.

[0104] Fusion of soft mIoU values ​​by calculation

[0105] y=Φ({y (r)}) (twenty one)

[0106] where Φ is a pooling function such as maximum, average, or harmonic mean. A single proposal ranking is produced using the fused soft mIoU y of Equation 21.

[0107] Multiple score maps {s (r) The mask merge of a set of B proposals overlapping at pixel (k, l) on

[0108]

[0109] Among them, the range of r covers all target frames.

[0110] For object proposal generation, Mask R-CNN is used. YouTube-VOS

[17] uses a dataset of 4453 YouTube video clips and 94 object categories, which is the largest video object segmentation dataset at the time of writing. The 2018 DAVIS challenge dataset is used for evaluation, which extends the original DAVIS dataset to 150 sequences including 376 objects.

[0111] The system architecture can be configured to execute on more computing devices. For example, the training architecture can include a developer computing device. The inference architecture can include a typical user computing device. The developer computing device can include a laptop, desktop computer, PC, server, or other computing device, which generally has superior resources to provide enhanced processing and storage capabilities for typical user computing devices (e.g., mobile devices, smart phones, tablets, or standard home laptops, PCs, desktop computers, or other devices that are not enhanced for gaming or other graphics processing). The features and functions, components, modules, sub-modules, etc. described herein can be configured using software (e.g., instructions and / or data) stored in a non-transient storage device (e.g., memory, CD-ROM, DVD-ROM, optical disk, etc.) for execution by a processing unit of a computing device. Therefore, in addition to the computing device aspect, a person of ordinary skill will understand that a computer program product aspect is disclosed.

[0112] Figure 5 5 is a block diagram of an example computer network 500 in which, according to an example, a developer computing device 502 operated by a user / developer 504 communicates via a communication network 506. Also shown are a second user 508 and a target computing device 510 configured to communicate via the communication network 506. The second user 510 may be a consumer. Server computing devices 512, 514, and 6516 communicate with the communication network 506, for example, with one or more of the devices 502 and 510 described further.

[0113] The developer computing device 502 is configured to train the tracking module 208 (e.g., its model) using the system architecture 200. The developer computing device 502 may have higher storage, processing power, and memory than the target device 510. Examples of the developer computing device 502 may include, but are not limited to, computing devices built to maximize graphics or other processing performance. Although described as a laptop computer, other form factors that meet the task for the computing device 502 are included. It should be understood that in some examples, the developer computing device may include multiple devices. In one example, a first computing device instructs a more powerful second computing device to perform a training task (or a component thereof). In some instances, multiple computing devices perform tasks in collaboration.

[0114] The target computing device 510 is configured to execute an inference model (e.g., a training model) such as the training tracking module 402. The target computing device 510 may have reduced storage, processing power, and memory compared to the developer device 502. According to the disclosed embodiments, examples of the target computing device 510 may include, but are not limited to, a personal computer, a laptop computer, a tablet computer, a notebook computer, a handheld computer, a personal digital assistant (PDA), a mobile phone, a smart phone, and any other type of computing device that may be configured to store data and software instructions, execute software instructions to perform operations, and / or display information on an interface module.

[0115] In one example, the developer computing device 502 is configured to train a tracking model using training data to track segmented objects in a video frame (e.g., 202). The training data including a plurality of training data video frames (e.g., 202) may be stored at the server computing device 512, or the training data video frames (e.g., 202) may be stored and used on the developer computing device 502. The developer computing device 502 may store the training model (e.g., such as module 402, as described in the system architecture 400) in the server 512 (or another server, not shown) to be retrieved by the target device 510 for execution. The target device 510 may also store the runtime model (e.g., module 402) on board for execution. In one example (not shown), the system architecture 400 is executed by the server computing device for or on behalf of the target device 510, and the target device does not need to store the system architecture 400.

[0116] In one example, the target computing device 510 is configured to receive a runtime model (e.g., module 402) developed by the developer computing device 400 as part of the architecture 400. The target computing device 510 can receive the runtime model (e.g., module 402) by communicating to the developer computing device 502 or from the server 512 via the communication network 506, or by any other method to digitally transmit data (e.g., downloading from an application store (e.g., an e-commerce service provided by the server computing device 514)). The runtime model is typically incorporated into a runtime application 514A (e.g., downloaded from an application store) that executes the model (e.g., module 402). A user 508 of the target computing device 510 provides a video (e.g., a selfie video) having frames (e.g., as 401) to the runtime application 514A. The provided video (e.g., frame 401) may be stored on the target computing device 510, may be received from a server or other computing device (e.g., via the web (e.g., http) or other communications (e.g., text messaging, instant messaging, short messaging service, proprietary messaging, email, etc.)), may be obtained from an input device (e.g., a camera) of the target computing device 510 or coupled to an input device (e.g., a camera) of the target computing device 610, or may be via other methods capable of providing video.

[0117] In one example, runtime application 514A uses module 402 to track objects between frames and apply effects to the tracked objects in each frame, thereby providing an augmented reality experience. According to the example, the effects are hair effects, makeup effects, nail effects, etc. The runtime application 514A executes runtime model 402 to determine the location of the object (which can be more than one object, such as a set of nails on the hand) in each frame (image) and generates an enhanced image or video with the effect applied at the location tracked by the model. Image processing techniques (e.g., via overlay / change of pixel data) are used to apply the effect at the tracked location. An enhanced video can be provided for display on device 510 or communication by it.

[0118] In one example, the effect to be applied to the image frame of the video is associated with one or both of a product and a service (e.g., a product and / or a service). In one example, the effect is a makeup effect associated with a product. In one example, the effect is a nail treatment effect associated with a service and a product. In one example, the application provides an interface to an e-commerce service (e.g., application 516) available from server computing device 518 to purchase a product for a makeup effect or a service and a product for a skin treatment effect. Figure 5 Payment services and payment networks are not shown.

[0119] In other runtime applications, the tracking module 402 is used to provide hair color, makeup try-on, or clothing try-on. In other applications that do not require augmented reality, the tracking module 402 is used directly for tracking applications, such as player tracking in sports analysis, people tracking in security footage, and car and road obstacle tracking in autonomous vehicle applications. For example, objects can be annotated by highlighting bounding boxes or mask outlines, or visualizing objects in a video in another way (e.g., updating the source video through image processing).

[0120] According to one or more aspects of the present disclosure, the developer computing device 502 may include one or more processors 520, one or more input devices 522, gesture-based I / O devices 524, one or more communication units 526, and one or more output devices 528. The developer computing device 502 also includes one or more storage devices 530 that store one or more modules and / or data of the system architectures 200 and 400.

[0121] Storage device 530 may store additional modules, such as an operating system and other modules (all not shown) including a communication module; photo (image / media) editing; a media player and / or streaming module; a browser module; etc. Herein, a storage device may be referred to as a storage unit.

[0122] One or more processors 520 may implement functions and / or execute instructions within the computing device 502. For example, the processor 520 may be configured to receive instructions and / or data from the storage device 530 to perform functions of the system architecture 200 and / or 400, as well as other functions, including the runtime application 514A. The developer computing device 502 may store data / information to the storage device 530. It should be understood that operations may not fall entirely within a particular module, such that one module may assist the functionality of another module.

[0123] The one or more processors 520 send a request to convert an image or video stored on the board or acquired from the camera 704 from a first domain space to a second domain space according to the effect. The processing unit 702 provides the plurality of unpaired images 102 to the first model module 104 to train the first model. Then, it provides the plurality of unpaired images 102 to the trained first model to generate a plurality of paired images 106, which are provided to train the runtime model using the runtime module 112. The processing unit may request the optional filter 114 to filter the plurality of paired images before providing the trained runtime model.

[0124] Communication channel 738 can couple each of components 520, 522, 524, 526, 528, and 530 with any module 200 and 400 for inter-component communication, whether communicatively, physically, and / or operationally. In some examples, communication channel 532 can include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.

[0125] Computer program code for performing operations may be written in any combination of one or more programming languages, for example, an object-oriented programming language such as Java, Smalltalk, C++, etc., or a traditional procedural programming language such as the "C" programming language or similar programming languages.

[0126] The developer computing device 502 may generate output for display on a screen of the gesture-based I / O device 524, or in some examples, for display by a projector, monitor, or other display device. It will be appreciated that the gesture-based I / O device 524 may be configured using a variety of technologies (e.g., with respect to input capabilities: resistive touch screen, surface acoustic wave touch screen, capacitive touch screen, projected capacitive touch screen, pressure sensitive screen, acoustic pulse recognition touch screen, or another presence sensitive screen technology; and with respect to output capabilities: liquid crystal display (LCD), light emitting diode (LED) display, organic light emitting diode (OLED) display, dot matrix display, e-ink, or similar monochrome or color display).

[0127] In the example described herein, gesture-based I / O devices 524 include touch screen devices that can receive tactile interactions or gestures as inputs from users interacting with the touch screen. Such gestures can include click gestures, drag or swipe gestures, flick gestures, pause gestures (e.g., the same position of the user touching the screen for at least one threshold time period), where the user touches or points to one or more positions of gesture-based I / O devices 524. Gesture-based I / O devices 524 can also include non-click gestures. Gesture-based I / O devices 524 can output or display information to users / developers, such as graphical user interfaces. Gesture-based I / O devices 524 can present various applications, functions, and capabilities of developer computing devices 502, including, for example, image processing applications, messaging applications, and other applications or functions for viewing images, processing images, and displaying new images, etc.

[0128] The one or more communication units 532 can communicate with external devices (e.g., server 512, target computing device 510) by sending and / or receiving network signals over one or more networks, such as via the communication network 506, for example, for the purposes described and / or for other purposes (e.g., printing). The communication unit may include various antennas and / or network interface cards, chips (e.g., global positioning satellite (GPS)), etc. for wireless and / or wired communication.

[0129] Input device 522 and output device 528 may include any of one or more buttons, switches, pointing devices, cameras, keyboards, microphones, one or more sensors (e.g., biometrics, etc.), speakers, bells, one or more lights, haptic (vibration) devices, etc. One or more of them may be coupled via a universal serial bus (USB) or other communication channel (e.g., 532). The camera (input device 522) may be front-facing (i.e., on the same side as it) to allow a user to capture an image using the camera while viewing gesture-based I / O device 524 for “selfies”.

[0130] One or more storage devices 730 can take different forms and / or configurations, such as short-term storage or long-term storage. Storage device 730 can be configured to store information short-term as volatile memory, which does not retain the stored contents when power is removed. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), etc. In some examples, storage device 730 also includes one or more computer-readable storage media, for example, for storing larger amounts of information than volatile memory and / or for storing such information long-term, retaining the information when power is removed. Examples of non-volatile memory include magnetic hard disks, optical disks, floppy disks, flash memory, or electrically programmable memory (EPROM) or electrically erasable and programmable (EEPROM) memory.

[0131] It should be appreciated that in one example, the target computing device 510 can be configured similarly as the developer computing device 502 , albeit with fewer components for developing a model.

[0132] Figure 66 is a flow diagram of operations 600 of a computing device such as device 510, although other devices are contemplated. In one embodiment, operations 600 provide a semi-supervised video object segmentation method for tracking and segmenting one or more objects throughout a video clip, the video clip comprising a sequence of frames, the sequence of frames comprising a previous frame and a current frame, the previous frame defining a target, and the current frame defining a source. At 602, the operation encodes features of each corresponding object in the source. At 604, the operation defines an attention map between the features encoded from the source and the features of the corresponding object encoded from the target. At 606, the operation associates the corresponding object with a corresponding position in the current frame using an intersection over union (IoU) function responsive to the attention map to rank candidate object proposals for the corresponding position, wherein each candidate object proposal is tracked from a single frame reference annotation of a reference frame of the video clip, the reference frame providing a ground truth position of the corresponding object in the reference frame. And, at 608, the operation defines a video segmentation mask for the corresponding object in the current frame based on the association. In one example, for each candidate object proposal, an intersection over union (IoU) function provides an IoU value responsive to an attention map used to rank the candidate object proposals. The ranking selects the highest IoU value of the corresponding object to identify the corresponding object in the current frame, thereby tracking the corresponding object from the reference frame through the previous frame to the current frame.

[0133] In one example, a semi-supervised video object segmentation method is provided for tracking and segmenting one or more objects throughout a video clip, the video clip comprising a sequence of frames, the sequence of frames comprising corresponding previous frames and a current frame, each of the corresponding previous frames defining a corresponding target, and the current frame defining a source. The method comprises: for each corresponding object: encoding features of the corresponding object in the source; and for each of the corresponding targets, defining a corresponding attention map between features encoded from the source and features of the corresponding object encoded from the corresponding target. The corresponding object is associated with a corresponding position in the current frame using an intersection-over-union (IoU) function responsive to the corresponding attention map to rank candidate object proposals for the corresponding position, wherein each candidate object proposal is tracked from a single frame reference annotation of a reference frame of the video clip, the reference frame providing a ground truth position of the corresponding object in the reference frame. A video segmentation mask is defined for each corresponding object in the current frame based on the association.

[0134] In one example, for each candidate object proposal, an intersection over union (IoU) function provides an IoU value for each attention map in response to ranking the candidate object proposals, the ranking selecting the highest IoU value for the corresponding object to identify the corresponding object in the current frame, thereby tracking the corresponding object from the reference frame through the previous frame to the current frame. The IoU value can be a fusion of the corresponding IoU values ​​for the corresponding object from each of the corresponding attention maps.

[0135] In one example, defining the video segmentation mask includes removing pixel-level overlaps from corresponding objects in the current frame.

[0136] In one example, the video segmentation mask is used to apply the effect to the corresponding object for display.

[0137] Figure 7 700 for a computing device such as device 502, although other devices (e.g., server 512) are contemplated. At 602, operations train each of an encoder model and a locator model using semi-supervised learning to track and segment one or more corresponding objects in a video clip, the video clip comprising a plurality of frames in a sequence, the training being conditioned on a single frame reference annotation of a reference frame for each corresponding object in the video clip. In one example, the encoder model comprises a neural network model configured to encode object features; encoding object features in a reference frame, a current frame, and a previous frame using the encoder model during training to generate corresponding object features from each frame; and locating objects using the locator model during training and forcing the encoder model to learn discriminative object features so as to be able to track the corresponding object from the reference frame to the current frame via the previous frame.

[0138] In one example, the encoder model is a CNN-based model. In one example, the encoder model and the localizer model are conditioned using a tracking loss and at least one object feature encoding loss. In one example, one object feature encoding loss is a similarity loss and another object feature encoding loss is a dissimilarity loss. In one example, the tracking loss is responsive to a geometric transformation estimate that predicts a position of a corresponding object in a current frame from a position in a reference frame or a previous frame, respectively.

[0139] In one example, the geometric transformation estimation is performed by a localizer model configured to predict a 2D warp τ defined in response to a corresponding attention map associating object features of a current frame with corresponding object features in one of a reference frame and a previous frame, respectively. g The transformation parameters of .

[0140] In one example, the localizer model is configured to receive each corresponding attention map that associates scores stored in a normalized grid in [-1, 1]x[-1, 1] with the location of the corresponding object in the current frame.

[0141] In one example, the locator model includes a 2D convolutional neural network (CNN).

[0142] In one example, the locator model is conditioned using a tracking loss and at least one object feature encoding loss.

[0143] In an example:t-1 is θ calculated by the localizer model (G) using the previous frame; Θ 0 is θ calculated by the localizer model using the reference frame; Θ t is the ground truth value θ of the current frame; G(θ t ) corresponds to using the transformation parameter Θ t A sampling grid of patches extracted from object features of the corresponding object extracted from the current frame; and P(Θ t ) corresponds to using Θ t The object features of the corresponding object extracted from the current frame; the tracking loss is defined as L 跟踪 =L1(G(θ t-1 ),G(Θ t ))+L1(G(Θ o ),G(Θ t )), where L1 is the smoothed L1 loss; the similarity loss is defined as L 相似性 =- <P(Θ t-1 ),P(Θ t )>- <P(Θ 0 ),P(Θ t )>; and the dissimilarity loss is defined as L 相异性 = <P(Θ t-1 ),P(Θ t )>+ <P(Θ 0 ),P(Θ t )>.

[0144] In one example, the operations at 700 may include generating reference annotations using a neural network model tuned to detect objects in a static image dataset, the detection producing a ground truth location for each respective object.

[0145] In one example, the operations at 700 may include: adjusting an encoder model, which once trained, defines a runtime model to track and segment one or more corresponding objects in a video clip; and defining a runtime application, which uses the trained and adjusted encoder model and applies effects to the corresponding objects to provide a modified video clip for display.

[0146] Fig. 8A800 is a flowchart for a computing device such as device 502 to perform operations according to an embodiment of the method, however other devices (e.g., device 516) are contemplated. At 802, operations detect corresponding objects in a reference frame of a video clip using a neural network model tuned to detect objects in a static image dataset, the detection producing a ground truth location of each corresponding object. At 804, operations encode object features in the reference frame using an encoder component of the neural network model to identify the object. At 806, operations encode object features in a next frame of the video clip using the same encoder component. At 808, operations correlate object features of the reference frame with object features of the next frame to define corresponding attention maps, one for each corresponding object in the reference frame. And at 806, operations associate each corresponding object in the reference frame with a corresponding object in the next frame using an intersection over union (IoU) function responsive to the corresponding ground truth location of each corresponding object and the corresponding attention map, thereby tracking each of the corresponding objects from the reference frame to the next frame.

[0147] Figure 8B 820 is a flowchart for a computing device such as device 502 to perform operation 820 of a method according to an embodiment, however other devices are contemplated (e.g., device 516). At 822, the operation responsive to the step of associating the tracking of the corresponding object to the next frame, defines a ground truth position of the corresponding object in the next frame.

[0148] At 824, the operation encodes the object features in another frame of the video clip using the same encoder component (as at operation 804), wherein the video clip includes the reference frame, the next frame, and another frame in the sequence. For example, the other frame is the current frame, and the next frame is a previous frame of the current frame.

[0149] At 826, an operation correlates the object features of the next frame with the object features of the other frame to define another corresponding attention map, one for each corresponding object in the reference frame. And at 828, an operation associates each corresponding object in the reference frame with the corresponding object in the other frame using an intersection over union (IoU) function responsive to the corresponding ground truth position in the next frame and the other corresponding attention map, thereby tracking each of the corresponding objects from the reference frame to the other frame through the next frame.

[0150] In one example, operations 800 or 820 are configured at 812 and 830, respectively, to remove pixel-level overlap of the tracked respective objects, and assign the respective pixel in the video frame to one of the respective objects in response to the highest score of the pixel in the respective attention map of the video frame.

[0151] In one example, operation 800 is configured (not shown) to perform a fusion function to generate a single proposal ranking from candidate object proposals, fusing values ​​of IoU functions for corresponding objects, wherein the fusion function is a merge function.

[0152] In one example, operations 800 or 820 are configured at 814 and 832, respectively, to modify at least one frame of the video clip at a corresponding location of at least one of the tracked corresponding objects to provide for display.

[0153] In one example, an application effect is modified, the effect comprising one of a virtual reality (VR) effect, an augmented reality (AR) effect, or a mixed reality (MR) effect. In one example, such as a virtual try-on or try-on example, the effect is associated with one or both of the product and service, and operations 800 or 820 or both (e.g., but only shown at 834) include providing an e-commerce interface to purchase one or both of the product and service.

[0154] In one example, at least a portion of operations 800 and / or 820 are performed using a Siamese Network architecture to process frames of a video clip.

[0155] Fig. 9 900 is a flowchart of operation 900 for a computing device such as computing device 510 according to an example. Other devices such as devices 502, 512, 514 and / or 516 can be envisioned. Operation 900 includes operations of providing a virtual try-on application of augmented reality, modifying a video to apply an effect associated with a product such as a hair color product, a nail color / nail effect product, or a makeup product such as a lipstick. At 902, a video, such as a selfie video, is received via a camera (e.g., on a computing device) via a video acquisition interface provided by the virtual try-on application. The video can be displayed (e.g., in real time) when received via a display device (e.g., on a computing device). At 904, an input is received to select a product to try out via a graphical user interface (GUI) and or other interface provided by the virtual try-on application. For example, in a virtual try-on application for nails, a nail polish product or effect is selected. The nail polish selection can include input of a color, finish (matte, shimmer, glitter, stripes, white tip, etc.). In one example, input to select an effect or product can be received before video reception begins. In one example, the try-on interface allows for "dynamic" selection of effects / products via an overlaid (or other) interface during video display.

[0156] At 906, objects in the video frame to which the effect is to be applied are tracked and segmented end-to-end in the video frame according to the techniques, methods, and examples herein. The segmentation defines a corresponding mask for the corresponding object in each frame. At 908, the effect is applied using the corresponding mask, and the video frame is modified for display via a display device. It should be understood that the operations at 902, 904, and 906 overlap, so that some video frames are processed for tracking and segmentation, and the effect is applied when other frames are received to provide an augmented reality experience to the user. At 910, input to an e-commerce service (e.g., provided by a server computing device 516) is received via an e-commerce interface of the virtual try-on application to facilitate the purchase of a product associated with the effect. Although operation 900 is described for a product, in the example, services or both products and services are considered. That is, the effect to be applied is associated with any one of the product and / or service. The e-commerce interface and service can also facilitate the purchase of other (e.g., suggested or related) products or services. The e-commerce service may have a recommendation component to recommend products or services related to effect-related products (e.g., hair products, makeup products, clothing products, footwear products, headwear products, and accessory products, etc.).

[0157] Actual implementations may include any or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, devices, systems, components, program products, and other combinations of features described herein for performing functions. Many embodiments or examples have been described. However, it will be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided from the described processes, or steps may be eliminated, and other components may be added to the described systems, or removed from the described systems. Therefore, other embodiments or examples are within the scope of the appended claims.

[0158] Throughout the description and claims of this specification, the words "comprise" and "comprising" and their variations mean "including but not limited to", and are not intended to (and do not) exclude other components, integers or steps. Throughout the specification, the singular encompasses the plural, unless the context requires otherwise. In particular, where the indefinite article is used, the specification should be understood as contemplating plural as well as singularity, unless the context requires otherwise.

[0159] Features, integer features, compounds, chemical moieties or groups described in conjunction with a particular aspect, embodiment or example of the invention are to be understood to be applicable to any other aspect, embodiment or example unless incompatible therewith. All features disclosed herein (including any accompanying claims, abstract and drawings) and / or all steps of any method or process so disclosed may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The invention is not limited to the details of any foregoing examples or embodiments. The invention extends to any novel one or any novel combination of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one or any novel combination of the steps of any method or process disclosed.

[0160] References

[0161] The following documents are incorporated herein by reference.

[0162] [1]The 2nd large-scale video object segmentation challenge.

[0163] [2] Linchao Bao, Baoyuan Wu, and Wei Liu. CNN in MRF: video objectsegmentation via inference in A cnn-based higher-order spatio-temporal MRF. InComputer Vision and Pattern Recognition (CVPR), 2018.

[0164] [3] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taix′e, Daniel Cremers, and Luc Van Gool. Oneshot video object segmentation. InCVPR, 2017.

[0165] [4]Sergi Caelles,Alberto Montes,Kevis-Kokitsi Maninis,Yuhua Chen,LucVan Gool,Federico Perazzi,and Jordi Pont-Tuset.The 2018davis challenge onvideo object segmentation.arXiv:1803.00557,2018.

[0166] [5]Jingchun Cheng,Yi-Hsuan Tsai,Wei-Chih Hung,Shengjin Wang,and Ming-Hsuan Yang.Fast and accurate online video object segmentation via trackingparts.In CVPR,2018.

[0167] [6]Jingchun Cheng,Yi-Hsuan Tsai,Shengjin Wang,and Ming-HsuanYang.Segflow:Joint learning for video object segmentation and optical flow.InICCV,2017.

[0168] [7]Suyog Dutt Jain,Bo Xiong,and Kristen Grauman.Fusionseg:Learning tocombine motion and appearance for fully automatic segmentation of genericobjects in videos.In CVPR,2017.

[0169] [8]Bumsub Ham,Minsu Cho,Cordelia Schmid,and Jean Ponce.Proposalflow.In Proceedings of the IEEE Conference on Computer Vision and PatternRecognition,2016.

[0170] [9]Kaiming He,Georgia Gkioxari,Piotr Doll′ar,and Ross B.Girshick.Maskr-cnn.2017IEEE International Conference on Computer Vision(ICCV),2017.

[0171]

[10] Ping Hu,Gang Wang,Xiangfei Kong,Jason Kuen,and Yap-PengTan.Motion-guided cascaded refinement network for video objectsegmentation.In CVPR,2018.

[0172]

[11] Yuan-Ting Hu,Jia-Bin Huang,and Alexander Schwing.Maskrnn:Instancelevel video object segmentation.In Advances in Neural Information ProcessingSystems 30,2017.

[0173]

[12] Varun Jampani,Raghudeep Gadde,and Peter V.Gehler.Videopropagation networks.In CVPR,2017.

[0174]

[13] Won-Dong Jang and Chang-Su Kim.Online video object segmentationvia convolutional trident network.In The IEEE Conference on Computer Visionand Pattern Recognition(CVPR),2017.

[0175]

[14] Anna Khoreva,Federico Perazzi,Rodrigo Benenson,Bernt Schiele,andAlexander Sorkine-Hornung.Learning video object segmentation from staticimages.In CVPR,2017.

[0176]

[15] Xiaoxiao Li and Chen Change Loy.Video object segmentation withjoint re-identification and attention-aware mask propagation.In The EuropeanConference on Computer Vision(ECCV),2018.

[0177]

[16] Rosanne Liu,Joel Lehman,Piero Molino,Felipe Petroski Such,EricFrank,Alex Sergeev,and Jason Yosinski.An intriguing failing of convolutionalneural networks and the coordconv solution.In Advances in Neural InformationProcessing Systems,2018.

[0178]

[17] Jonathon Luiten,Paul Voigtlaender,and Bastian Leibe.Premvos:Proposal-generation,refinement and merging for video object segmentation.InAsian Conference on Computer Vision,2018.

[0179]

[18] Kevis-Kokitsi Maninis,Sergi Caelles,Yuhua Chen,Jordi Pont-Tuset,Laura Leal-Taix′e,Daniel Cremers,and Luc Van Gool.Video object segmentationwithout temporal information.2018.

[0180]

[19] Francisco Massa and Ross Girshick.maskrcnn-benchmark:Fast,modularreference implementation of Instance Segmentation and Object Detectionalgorithms in PyTorch.

[0181]

[20] Seoung Wug Oh,Joon-Young Lee,Ning Xu,and Seon Joo Kim.Videoobject segmentation using space-time memory networks.ICCV,2019.

[0182]

[21] Seoung Wug Oh,Joon-Young Lee,Kalyan Sunkavalli,and Seon JooKim.Fast video object segmentation by referenceguided mask propagation.InCVPR,2018.

[0183]

[22] F.Perazzi,J.Pont-Tuset,B.McWilliams,L.Van Gool,M.Gross,andA.Sorkine-Hornung.A benchmark dataset and evaluation methodology for videoobject segmentation.In Computer Vision and Pattern Recognition,2016.

[0184]

[23] I.Rocco,R.Arandjelovi′c,and J.Sivic.Convolutional neural networkarchitecture for geometric matching.In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition,2017.

[0185]

[24] I.Rocco,R.Arandjelovi′c,and J.Sivic.End-to-end weakly-supervisedsemantic alignment.2018.

[0186]

[25] Amaia Salvador,Miriam Bellver,Manel Baradad,Ferran Marques,JordiTorres,and Xavier Giro-i Nieto.Recurrent neural networks for semanticinstance segmentation.In arXiv:1712.00617,2017.

[0187]

[26] Pavel Tokmakov,Karteek Alahari,and Cordelia Schmid.Learning videoobject segmentation with visual memory.In ICCV,2017.

[0188]

[27] Yi-Hsuan Tsai,Ming-Hsuan Yang,and Michael J.Black.Videosegmentation via object flow.In CVPR,2016.

[0189]

[28] Paul Voigtlaender,Yuning Chai,Florian Schroff,Hartwig Adam,Bastian Leibe,and Liang-Chieh Chen.Feelvos:Fast end-to-end embedding learningfor video object segmentation.In CVPR,2019.

[0190]

[29] Paul Voigtlaender and Bastian Leibe.Online adaptation ofconvolutional neural networks for video object segmentation.In BMVC,2017.

[0191]

[30] Paul Voigtlaender,Jonathon Luiten,and Bastian Leibe.BoLTVOS:Box-Level Tracking for Video Object Segmentation.arXiv:1904.04552,2019.

[0192]

[31] Xiaolong Wang,Allan Jabri,and Alexei A.Efros.Learningcorrespondence from the cycle-consistency of time.In CVPR,2019.

[0193]

[32] ZiqinWang,Jun Xu,Li Liu,Fan Zhu,and Ling Shao.Ranet:Rankingattention network for fast video object segmentation.In ICCV,2019.

[0194]

[33] Ning Xu,Linjie Yang,Yuchen Fan,Jianchao Yang,Dingcheng Yue,YuchenLiang,Brian Price,Scott Cohen,and Thomas Huang.Youtube-vos:Sequence-to-sequence video object segmentation.In ECCV,2018.

[0195]

[34] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas S. Huang. YouTubevos: A large-scale video objects segmentation benchmark. In ECCV, 2018.

[0196]

[35] Linjie Yang, Yanran Wang, Xuehan Xiong, Jiangchao Yang, and AggelosK. Katsaggelos. Efficient video object segmentation via network modulation. InCVPR, 2018.

Claims

1. A semi-supervised video object segmentation method for tracking and segmenting one or more objects throughout a video clip, the video clip comprising a sequence of frames, the sequence of frames comprising a previous frame and a current frame, the previous frame defining a target and the current frame defining a source, the method comprising: encoding a characteristic of each corresponding object in the source; defining an attention map between the features encoded from the source and the features of the corresponding object encoded from the target; associating the corresponding object with the corresponding position in the current frame using an intersection over union (IoU) function in response to the attention map to rank candidate object proposals for the corresponding position, wherein each candidate object proposal is tracked according to a single frame reference annotation of a reference frame of the video segment, the reference frame providing a ground truth position of the corresponding object in the reference frame; and defining a video segmentation mask for the corresponding object in the current frame according to the association, wherein, for each candidate object proposal, the intersection over union (IoU) function provides an IoU value of the attention map in response to sorting the candidate object proposals, the sorting selecting the highest IoU value for the corresponding object to identify the corresponding object in the current frame, thereby tracking the corresponding object from the reference frame through the previous frame to the current frame.

2. The method according to claim 1, wherein: Defining a video segmentation mask includes removing pixel-level overlaps from the corresponding objects in the current frame.

3. The method of claim 1 or 2, comprising applying an effect to the corresponding object for display using the video segmentation mask.

4. A semi-supervised video object segmentation method for tracking and segmenting one or more objects throughout a video clip, the video clip comprising a sequence of frames, the sequence of frames comprising corresponding previous frames and a current frame, each of the corresponding previous frames defining a corresponding target, the current frame defining a source, the method comprising, For each corresponding object: encoding features of the corresponding object in the source; for each of the respective targets, defining a respective attention map between the features encoded from the source and features of the respective object encoded from the respective target; associating the corresponding object with the corresponding position in the current frame using an intersection over union (IoU) function in response to the corresponding attention map to rank candidate object proposals for the corresponding position, wherein each candidate object proposal is tracked according to a single frame reference annotation of a reference frame of the video segment, the reference frame providing a ground truth position of the corresponding object in the reference frame; and defining a video segmentation mask for the corresponding object in the current frame according to the association, wherein, for each candidate object proposal, the intersection over union (IoU) function provides an IoU value for each attention map in response to sorting the candidate object proposals, the sorting selecting the highest IoU value for the corresponding object to identify the corresponding object in the current frame, thereby tracking the corresponding object from the reference frame through the previous frame to the current frame.

5. The method according to claim 4, wherein: The IoU value is a fusion of the corresponding IoU values ​​for the corresponding object from each of the corresponding attention maps.

6. The method according to claim 4 or 5, wherein: Defining a video segmentation mask includes removing pixel-level overlaps from the corresponding objects in the current frame.

7. A method according to claim 4 or 5, comprising using the video segmentation mask to apply an effect to the corresponding object for display.

8. The method according to claim 7, wherein: The effect is associated with one or both of a product and a service, and the method includes providing an interface to an e-commerce service for purchasing one or both of the product and the service.

9. A method for tracking and segmenting an object detected in a video clip, comprising: detecting corresponding objects in a reference frame of the video clip using a model conditioned using supervised learning to localize objects in static images, the model generating a ground truth bounding box for each of the corresponding objects detected in the reference frame; processing a current frame of the video segment to track the corresponding object relative to a previous frame of the video segment, the current frame comprising a source, the reference frame and the previous frame comprising a corresponding target, the processing comprising: encoding characteristics of objects in the source; encoding features of the objects in the corresponding target; For each of the corresponding objects detected in the reference frame: cropping the features for the objects in the corresponding targets using a corresponding ground truth bounding box for the corresponding objects in the corresponding targets; computing a corresponding attention map associating features of objects in the source with features of objects in a corresponding target of the crop; ranking candidate object proposals for each corresponding object in the source, wherein each candidate is determined from a ground truth position of the corresponding object in the corresponding target, the ranking step ranking the candidate proposals using an intersection over union (IoU) function responsive to the corresponding attention map, the ranking step selecting one candidate proposal having a highest IoU value from the candidate proposals, thereby tracking the corresponding object to the source; and Determine a video segmentation mask for each corresponding object in the current frame using the corresponding attention map; Wherein, the ground truth position of the corresponding object in the corresponding target is defined by: For the reference frame, conditioning the ground truth bounding box generated by the model using supervised learning; and For the previous frame, a ground truth bounding box of the corresponding object in the previous frame determined by tracking the corresponding object from the reference frame.

10. The method of claim 9, further comprising removing pixel-level overlap from the corresponding video segmentation masks for the corresponding objects.

11. The method according to any one of claims 9 and 10, comprising performing a fusion function to produce a single proposal ranking from the candidate object proposals, fusing the values ​​of the IoU function of the corresponding object determined relative to the reference frame with the values ​​of the IoU function of the same corresponding object determined relative to the previous frame, wherein The fusion function is a merging function.

12. A method comprising: detecting corresponding objects in a reference frame of the video clip using a neural network model tuned to detect objects in a static image dataset, the detection producing a ground truth position of each of the corresponding objects; encoding features of the object in the reference frame using an encoder component of a neural network model tuned to recognize objects; encoding the object features in a next frame of the video segment using the same encoder component; Associating the object features of the reference frame with the object features of the next frame to define corresponding attention maps, one attention map corresponding to each corresponding object in the reference frame; as well as Each of the corresponding objects in the reference frame is associated with the corresponding object in the next frame using an intersection-over-union function responsive to the corresponding ground truth position of each of the corresponding objects and the corresponding attention map, thereby tracking each of the corresponding objects from the reference frame to the next frame.

13. The method according to claim 12, further comprising: In response to the associating step of tracking the corresponding object to the next frame, defining a ground truth position of the corresponding object in the next frame; encoding the object features in another frame of the video segment using the same encoder component, wherein the video segment includes the reference frame, the next frame, and another frame in the sequence; Associating the object features of the next frame with the object features of the other frame to define another corresponding attention map, one attention map corresponding to each corresponding object in the reference frame; as well as Each of the corresponding objects in the reference frame is associated with a corresponding object in the other frame using an intersection-over-union function responsive to the corresponding ground truth position in the next frame and the other corresponding attention map, thereby tracking each of the corresponding objects from the reference frame to the other frame through the next frame.

14. The method of any one of claims 12 and 13, comprising modifying at least one frame of the video segment at the respective position of at least one of the respective tracked objects for display.

15. The method according to claim 14, wherein: The effect includes one of a virtual reality (VR) effect, an augmented reality (AR) effect, or a mixed reality (MR) effect.

16. The method according to any one of claims 12 and 13 includes removing pixel-level overlap of the tracked corresponding objects, and assigning the corresponding pixel in the video frame to one of the corresponding objects in response to the highest score of the pixel in the corresponding attention map of the video frame.

17. The method according to any one of claims 12 and 13, wherein: The method is performed using a Siamese architecture to process frames of the video segment.

18. A computing device comprising a non-transitory storage device and a processor coupled thereto, the storage device storing instructions which, when executed by the processor, configure the computing device to perform a method in accordance with any one of method claims 1 to 17.

19. A device comprising a non-transitory storage device storing instructions which, when executed by a processor of a computing device, configure the computing device to perform the method according to any one of method claims 1 to 17.

Images