Implementing video segmentation

Abstract

This disclosure describes techniques for implementing video segmentation. A video is divided into a plurality of segments. Each segment of the plurality of segments includes a plurality of frames. Axial trajectory attention is applied to each of the plurality of segments by the first sub-model. A segment feature corresponding to each segment of the plurality of segments is generated by the first sub-model. A set of object queries corresponding to each of the plurality of segments is generated by the transformer decoder based on the segment features. Track attention is applied by the second sub-model to refine a set of object queries corresponding to the plurality of segments. A video level segmentation result is generated based on the refined object query.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to U.S. Provisional Patent Application No. 63 / 540,992, filed on September 28, 2023, which is incorporated herein by reference in its entirety. Background Art

[0003] Computer vision is increasingly used in various industries to perform different tasks. The most pioneering work in the field of computer vision mainly deals with image processing. Due to the complexity of video data and high computational cost, the field of video processing has not been explored deeply enough. It is desirable to improve video segmentation techniques. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The following detailed description may be better understood when read in conjunction with the accompanying drawings.For illustrative purposes, example embodiments of various aspects of the disclosure are shown in the drawings; however, the disclosure is not limited to the specific methods and instrumentalities disclosed.

[0005] Figure 1 An example system for implementing video segmentation according to the present disclosure is shown.

[0006] Figure 2 An example first sub-model according to the present disclosure is shown.

[0007] Figure 3 An example visualization of learned axial trajectory attention in accordance with the present disclosure is shown.

[0008] Figure 4 An example second sub-model according to the present disclosure is shown.

[0009] Figure 5 An example process for implementing video segmentation according to the present disclosure is shown.

[0010] Figure 6 An example process for implementing video segmentation according to the present disclosure is shown.

[0011] Figure 7 An example process for implementing video segmentation according to the present disclosure is shown.

[0012] Figure 8 An example process for implementing video segmentation according to the present disclosure is shown.

[0013] Fig. 9 An example process for implementing video segmentation according to the present disclosure is shown.

[0014] Fig.10 An example process for implementing video segmentation according to the present disclosure is shown.

[0015] Fig.11 An example process for implementing video segmentation according to the present disclosure is shown.

[0016] Fig.12 An example process for implementing video segmentation according to the present disclosure is shown.

[0017] Fig.13 Example evaluation results according to the present disclosure are shown.

[0018] Fig.14 An example computing device is shown that can be used to perform any of the techniques disclosed herein. DETAILED DESCRIPTION

[0019] Video panoptic segmentation is a challenging computer vision task that requires temporally consistent pixel-level scene understanding of videos. This temporally consistent pixel-level scene understanding can be obtained by jointly segmenting various classes of objects (e.g., "things" (people, cars, etc.) and "stuff" (sky, grass, etc.)) and associating them (i.e., tracking them) across all frames in the video. Video segmentation can benefit a wide variety of downstream applications such as autonomous driving, robotic visual control, and video editing.

[0020] Existing video segmentation techniques have unified different video segmentation tasks into a simple set prediction task, where the input video is segmented into a set of tubes (obtained by linking segmentation masks along the time axis) to match the ground-truth tube. Specifically, given an input video Where L represents the video length, and H, W represent the frame height and frame width, video segmentation aims to segment it into a collection of pipelines of N labeled categories:

[0021]

[0022] in and represents the predicted pipeline and its corresponding semantic category probabilities. The ground truth set of pipelines containing M labeled categories is similarly represented as The two sets are matched during training via Hungarian Matching (Kuhn 1955) to compute the loss.

[0023] The above video segmentation formulation is theoretically applicable to any video sequence of length L. However, it is not feasible to put the entire complete video into a modern large-scale network backbone during training. Therefore, most works adopt frame-level segmenters or clip-level segmenters (clips are usually short video sequences of two or three frames) to first get a frame-level or clip-level pipeline, and further concatenate them to get the final video-level pipeline. The prior art ignores the potential of improving within-clip tracking and long-term consistent tracking beyond neighboring clips. This paper describes improved video segmentation techniques.

[0024] Figure 1 An example system 100 for implementing video segmentation by using intra-segment tracking submodels and cross-segment tracking submodels is shown. A video can be divided into multiple segments. Each segment of the multiple segments includes multiple frames of the video. Formally, the entire video V can be divided into a set of non-overlapping segments: Where T represents the length of each segment in the time domain dimension (for simplicity, it is assumed that L is divisible by T; if not, the last frame can be duplicated). T can be greater than or equal to two. The system 100 includes a segment level segmenter 109 and a video level segmenter 107.

[0025] The segment level segmenter 109 may include a pixel encoder 101, a first sub-model 102 (e.g., an intra-segment tracking sub-model), a pixel decoder 103, and a transformer decoder 104. The first sub-model 102 may be configured to track objects across frames within the same segment among a plurality of segments. Each of the plurality of segments may be input into the pixel encoder 101. The pixel encoder 101 may generate segment features for each of the plurality of segments. The segment features may be input into the first sub-model 102. The first sub-model 102 may apply axial trajectory attention and multi-scale deformable attention for enhancing local consistency within adjacent frames of the same segment. The pixel decoder 103 may output segment features (e.g., spatiotemporal enhancement features) for each of the plurality of segments. The pixel decoder 103 may generate segment features (e.g., spatiotemporal enhancement features) corresponding to each of the plurality of segments based on the output of the first sub-model 102.

[0026] The transformer decoder 104 may generate a set of object queries corresponding to each of the plurality of segments. The transformer decoder 104 may generate a set of object queries corresponding to each of the plurality of segments based on the segment features generated by the pixel decoder 103. The transformer decoder 104 may be a k-means transformer decoder configured to generate segment-level predictions, wherein the object query (for the k-th segment) is iteratively refined over multiple layers of the transformer decoder 104. The resulting object query can be used to generate a set of pipelines of N class labels.

[0027] In an embodiment, the trajectory attention of the first sub-model 102 learns to model the temporal correspondence between adjacent frames of the same segment by estimating the motion path along the time axis and aggregating information along the trajectory. However, due to the quadratic dependence of the trajectory attention on the input size, it is not easy to directly extend it to the dense prediction task per pixel. To address this challenge, the first sub-model 102 applies axial trajectory attention to effectively calculate the trajectory attention sequentially along the height axis and the width axis.

[0028] In an embodiment, the first sub-model 102 (e.g., the intra-segment tracking sub-model) can be enhanced by incorporating multi-scale deformable attention, which can be iteratively stacked with the axial trajectory attention to ensure that the learned segment features are consistent in both the temporal and spatial domains. Thereafter, the transformer decoder 104 can be applied to obtain segment-level predictions, where object queries are learned to encode objects in each segment (i.e., each query is responsible for predicting the mask and semantic category of the object in the segment). At this stage, the segment-level segmenter enhanced with the first sub-model 102 (e.g., the intra-segment tracking sub-model) is able to capture motion information within each segment, thereby achieving consistent predictions in a nearly online manner.

[0029] The video level segmenter 107 may include a second sub-model 106 (e.g., a cross-segment tracking sub-model). The second sub-model 106 may be configured to track each object across multiple segments. The second sub-model 106 may use trajectory attention and temporal atrous spatial pyramid pooling (temporal ASPP) to improve long-term consistency. The second sub-model 106 may apply trajectory attention to refine a set of object queries corresponding to multiple segments. The video level segmenter 107 may generate a video level segmentation result based on the refined object query. For example, the video level segmenter 107 may obtain video features by concatenating all segment features (a total of K segments) output by the pixel decoder 103, and by multiplying the video features and the refined segment object query. to generate video predictions.

[0030] The second sub-model 106 can take all segment-level object queries as input. Specifically, each segment can be processed by its own set of object queries. Given a video segmented into multiple segments, multiple sets of object queries can be obtained from the segment-level segmenter 109. In order to further capture the global (i.e., the entire video) temporal connection, query track attention can be applied to all segment-level objects. Intuitively, since each object query is learned to encode an object in the segment, track attention is applied to all object queries, and each object is learned to be tracked across different segments by finding the trajectory of each object in the video. In addition to track attention, temporal ASPP can be used to capture object motion over different time spans. The resulting second sub-model 106 can iteratively stack track attention and temporal ASPP to refine the object query of the video, thereby obtaining temporally consistent predictions at the video level. The second sub-model 106 can enable the entire video to be received as input during reasoning, thereby encouraging temporal consistency in a completely offline manner.

[0031] The first sub-model 102 and the second sub-model 106 enable the system 100 to achieve much better temporally consistent segmentation results than existing segmentation techniques. For example, the first sub-model 102 and the second sub-model 106 enable the system 100 to achieve much better temporally consistent segmentation by utilizing track attention, which is commonly used for the task of video classification. The track attention utilized by the system 100 can be adapted to handle dense pixel features and object queries, aiming to improve intra-segment tracking results and cross-segment tracking results, respectively.

[0032] The system 100 including the first sub-model 102 and the second sub-model 106 built on top of any segment-level video segmenter can be used to reason about the entire video in an offline manner by refining the segment-level prediction results. To obtain a video-level pipeline from the refined object queries, a 1D convolutional layer can be utilized to generate temporal weighted category predictions motivated by the fact that object queries on the trajectory path should have the same category prediction.

[0033] Figure 2 The first sub-model 102 (e.g., intra-segment tracking sub-model) is shown in more detail. The main component of the first sub-model 102 is the axial track attention. The axial track attention decomposes the track attention in the height axis and the width axis, and effectively learns to track objects across frames in the same segment (e.g., intra-segment tracking). The axial decomposition significantly reduces the computational complexity of dense pixel features.

[0034] In the first sub-model 102, the input features 202 can be enriched by utilizing multi-scale deformable attention and axial track attention. The axial track attention deploys the track attention in a manner similar to the axial attention, which not only effectively captures the object motion information but also reduces the computational cost. The attention can be visualized relative to the reference point (e.g., point) 201 shown in frame 1. The first sub-model 102 iteratively stacks the multi-scale deformable attention and the axial track attention for N times (sequentially along the height axis (H axis) and the width axis (W axis)). w times, and outputs segment features 204 that are consistent in spatial domain and temporal domain.

[0035] Formally, an input video segment consisting of T frames can be forward propagated through a frame-level network backbone (e.g., ConvNeXt) to extract feature maps Where D, H, and W represent the dimension, height, and width of the feature map F, respectively. The feature map F can be extracted frame by frame via the network backbone, so the temporal information may not be exchanged between frames. The feature 202 can be reshaped into To obtain a sequence of TH pixel features x th Can be linearly projected to a set of query key-value vectors Axial attention can be performed along the trajectory (i.e., the probabilistic path of points between frames). Specifically, for each reference point at a specific temporal height th and its corresponding query q th , we can construct a set of trajectory points The set of trajectory points can represent the pooled information weighted by the trajectory probability. The axial trajectory extends for the duration of the video clip, and its points at different times t′ is defined as follows:

[0036]

[0037] This step computes the axial track attention (index h′) on the H axis independently for each frame. It does this by only focusing the track query q along the H axis. th With key k t′h′ The axial trajectory path of the reference point th across frames t′ in the segment is compared. To infer intra-frame (within the same segment) connections, the trajectories can be pooled for time t′. Specifically, the trajectory point can be linearly projected to obtain a new set of query key-value vectors:

[0038]

[0039] Where W q , W k and W vis the linear projection matrix for query, key and value. The reference point at temporal height th can then be updated by applying 1D attention along time t′:

[0040]

[0041] Using the above update rule, motion information can be propagated along the H axis in the video segment. In order to capture global information, the feature 202 can be further reshaped into And the same axial track attention (but along the W axis) can be applied continuously to capture width dynamics. The axial track attention effectively reduces the computational complexity of the original track attention from Reduce to This allows it to be applied to dense video feature maps and infer motion information across frames in the same segment.

[0042] In order to spatially enhance the features 202, multi-scale deformable attention can be used to exchange information of features at different levels. Specifically, multi-scale deformable attention can be applied to the feature map F (extracted by the network backbone) frame by frame, which effectively exchanges information across feature map scales (steps 32, 16, and 8) for each frame. Finally, the first sub-model 102 can be obtained by iteratively stacking multi-scale deformable attention and axial trajectory attention to ensure that the learned features 204 are spatially consistent across scales and temporally consistent across frames in the same segment.

[0043] In an embodiment, the first sub-model 102 may enable a segment-level segmenter to segment a video in a nearly online manner (i.e., segment by segment). Unlike the prior art that takes overlapping segments as input and uses video stitching to link predicted segment-level pipelines, the first sub-model 102 may use Hungarian matching to associate segment-level pipelines via object queries (at the segment level instead of the frame level, since our input segments are non-overlapping).

[0044] Figure 3 An example visualization 300 of learned axial trajectory attention is shown. In this short four-frame clip depicting the action of "playing basketball", the position of the basketball at frame 1 is selected as the reference point 301. The axial trajectory attention can be calculated sequentially along the H-axis and the W-axis. The H-axis trajectory attention and the W-axis trajectory attention can be multiplied to visualize the trajectory of the reference point 301 over time (i.e., bright spots correspond to high attention values ​​on the H-axis trajectory attention and the W-axis trajectory attention). Figure 3 As shown, the learned axial trajectory attention is able to capture the motion path of the basketball.

[0045] Figure 4The second sub-model 106 is shown in more detail. The second sub-model 106 can be implemented by iteratively performing trajectory attention and time-domain ASPP up to N c The second sub-model 106 can refine and better associate segment-level predictions to resolve inconsistencies. Specifically, given an object query in a video, (where the video is divided into K = L / T non-overlapping segments, and the kth segment has its own object query ), Hungarian matching can be used to first align the object query to the initial tracking result. The object query can then be used as input to the second sub-model 106, which can further refine the tracking result to capture the entire video temporal connection (i.e., across all segments). The second sub-model 106 contains two operations: track attention and temporal ASPP.

[0046] In an embodiment, the second sub-model 106 includes a track attention operation. For the kth segment, the object query C k Encode the fragment-level pipeline prediction (i.e., C k Each query in generates a category label for an object in the kth segment). Therefore, associating segment-level predictions can be similar to finding a trajectory path for an object query throughout the video. Trajectory attention can also be used to capture the whole video temporal connections between segments. Formally, for a video divided into K segments (each segment is processed by N object queries), each object query C kn ∈{C k} is first projected into the query key vector set Then, the trajectory query set can be calculated by calculating the probabilistic path of each object query

[0047]

[0048] In further querying the trajectory Project to After that, similar to the equation used to calculate the probability path for each object query, the entire video cross-segment connections can be aggregated along the trajectory path of the object query by the following equation:

[0049]

[0050] In an embodiment, the second sub-model 106 may include a temporal ASPP operation. Although the above-mentioned trajectory attention infers the temporal connection of the entire video, it can be further enriched by a short-term tracking module. The ASPP that has successfully captured multi-scale context information in the spatial domain can be extended to the temporal domain. Specifically, the temporal ASPP includes three parallel temporal hole convolutions, where these temporal hole convolutions are applied to the updated object query Z at different rates to capture object motions of different time spans.

[0051] In summary, video panoptic segmentation requires consistent segmentation (for both the "object" and "stuff" categories) and tracking of objects over time in a video. System 100 enriches off-the-shelf mask transformers by leveraging track attention. The deployed mask transformer takes short clips consisting of only a few frames as input and predicts clip-level segmentation. To enforce temporal consistency, system 100 employs a within-segment tracking sub-model and a cross-segment tracking sub-model, thereby effectively leveraging track attention.

[0052] The present disclosure adapts track attention for both dense pixel features and object queries, aiming to improve short-term tracking results and long-term tracking results, respectively. In particular, in the intra-fragment tracking sub-model, axial track attention can be performed, which effectively calculates track attention for tracking dense pixels sequentially along the height axis and the width axis. The axial decomposition significantly reduces the computational complexity of dense pixel features. In the cross-fragment tracking sub-model, since the object query in the mask transformer is learned to encode the object information, it is possible to capture long-term temporal connections by applying track attention to the object query (which learns to track each object across different fragments).

[0053] Figure 5 An example process 500 for implementing video segmentation is shown. Figure 5 1 and 2 are depicted as a series of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the depicted operations.

[0054] At 502, a video may be divided into a plurality of segments. Each of the plurality of segments may include a plurality of frames of the video. For example, the entire video may be divided into a set of non-overlapping segments. In order to enhance local consistency within adjacent frames of the same segment, the first sub-model may apply axial track attention to each of the plurality of segments. At 504, axial track attention may be applied. Axial track attention may be applied by a first sub-model (e.g., first sub-model 102 (i.e., intra-segment tracking sub-model)) to each of the plurality of segments. The first sub-model may be configured to track objects across frames within the same segment among the plurality of segments. At 506, segment features may be generated. The segment features may correspond to each of the plurality of segments. The segment features may be generated by the first sub-model.

[0055] At 508, an object query set may be generated. The object query set may correspond to each of the plurality of segments. The object query set may be generated based on segment features. The object query set may be generated by a transformer decoder. The transformer decoder may be a k-means transformer decoder configured to produce segment level predictions, wherein the object query is iteratively refined over multiple layers.

[0056] At 510, track attention may be applied by a second sub-model (e.g., second sub-model 106 (i.e., cross-segment tracking sub-model)). Track attention may be applied to refine a set of object queries corresponding to multiple segments. Track attention may be applied by a second sub-model. The second sub-model may be configured to track each object across multiple segments. The second sub-model may use track attention and temporal atrous spatial pyramid pooling (temporal ASPP) to improve long-term consistency. At 512, a video level segmentation result may be generated. The video level segmentation result may be generated based on the refined object query. For example, the video level segmentor may obtain video features by concatenating all segment features output by the first sub-model, and by multiplying the video features and the refined segment object query. to generate video predictions.

[0057] Figure 6 An example process 600 for implementing video segmentation is shown. Figure 6 1 and 2 are depicted as a series of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the depicted operations.

[0058] A first sub-model (e.g., first sub-model 102) may perform axial trajectory attention that efficiently computes trajectory attention for sequentially tracking dense pixels along a height axis. At 602, a pixel feature sequence may be generated. A pixel feature sequence may be generated along the height axis. The pixel feature sequence may be generated based on extracting features from each of a plurality of frames in a same segment among a plurality of segments. At 604, axial trajectory attention may be performed along the height axis. The axial trajectory attention may be performed by the first sub-model.

[0059] Figure 7 An example process 700 for implementing video segmentation is shown. Figure 7 1 and 2 are depicted as a series of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the depicted operations.

[0060] A first sub-model (e.g., first sub-model 102) may perform axial trajectory attention that efficiently computes trajectory attention for sequentially tracking dense pixels along a width axis. At 702, a pixel feature sequence may be generated. A pixel feature sequence may be generated along the width axis. The pixel feature sequence may be generated based on extracting features from each of a plurality of frames in a same segment among a plurality of segments. At 704, axial trajectory attention may be performed along the width axis. The axial trajectory attention may be performed by the first sub-model.

[0061] Figure 8 An example process 800 for implementing video segmentation is shown. Figure 8 1 and 2 are depicted as a series of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the depicted operations.

[0062] At 802, a video may be divided into a plurality of segments. Each of the plurality of segments may include a plurality of frames of the video. For example, the entire video may be divided into a set of non-overlapping segments. Features corresponding to a particular segment may be input into a first sub-model (e.g., first sub-model 102). The first sub-model may utilize axial trajectory attention and multi-scale deformable attention for enhancing local consistency within adjacent frames in the same segment. At 804, multi-scale deformable attention may be performed on feature maps extracted from frames in the same segment among the plurality of segments. Multi-scale deformable attention may be performed to exchange information across feature map scales and ensure spatial domain consistency of segment features.

[0063] The multi-scale deformable attention may be iteratively stacked with the axial track attention. At 806, the axial track attention may be applied. The axial track attention may be applied to each of the plurality of segments. The axial track attention may be applied to each of the plurality of segments along a height axis and a width axis. Applying the axial track attention may enhance local consistency within adjacent frames of the same segment.

[0064] Fig. 9 An example process 900 for implementing video segmentation is shown. Fig. 9 1 and 2 are depicted as a series of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the depicted operations.

[0065] At 902, a video may be divided into a plurality of segments. Each of the plurality of segments may include a plurality of frames of the video. For example, the entire video may be divided into a set of non-overlapping segments. The first sub-model (e.g., the first sub-model 102) may be enhanced by incorporating multi-scale deformable attention, which may be iteratively stacked with axial trajectory attention to ensure that learned segment features are consistent in both temporal and spatial domains. At 904, the multi-scale deformable attention and the axial trajectory attention may be iteratively performed. The multi-scale deformable attention and the axial trajectory attention may be iteratively performed by the first sub-model.

[0066] The first sub-model may output a segment feature (e.g., a spatiotemporal enhancement feature) for each of the plurality of segments. At 906, segment features having temporal consistency and spatial domain consistency may be output from the first sub-model. At 908, an object query set corresponding to each of the plurality of segments may be generated. The object query set may be generated by a transformer decoder based on the segment features. The object query set corresponding to each of the plurality of segments may be iteratively refined over a plurality of layers of the transformer decoder.

[0067] Fig.10 An example process 1000 for implementing video segmentation is shown. Fig.10 1 and 2 are depicted as a series of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the depicted operations.

[0068] The second sub-model (e.g., the second sub-model 106) can perform a track attention operation. For the k-th segment, the object query C k Encode the fragment-level pipeline prediction (i.e., C kEach query in the k-th segment generates a category label for an object in the k-th segment). Therefore, associating the segment-level prediction results can be similar to finding the trajectory path of the object query in the entire video. Trajectory attention can also be used to capture the entire video time domain connection between segments. At 1002, each object query in the object query set can be projected into a query key vector set. Formally, for a video divided into K segments (each segment is processed by N object queries), each object query C kn ∈{C k} is first projected into the query key vector set At 1004, a trajectory query set may be calculated. For example, the trajectory query set may be calculated by calculating a probability path for each object query.

[0069]

[0070] At 1006, the trajectory query set may be linearly projected into a new query key vector set. Project to After that, similar to the equation for calculating the probability path for each object query, the entire video cross-segment connection can be aggregated along the trajectory path of the object query. At 1008, the entire video temporal connection across multiple segments can be aggregated based on the new query key vector set. For example, the entire video cross-segment connection can be aggregated along the trajectory path of the object query by the following equation:

[0071]

[0072] Fig.11 An example process 1100 for implementing video segmentation is shown. Fig.11 2 is depicted as a sequence of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the operations depicted.

[0073] At 1102, an object query set may be generated. The object query set may correspond to each of a plurality of segments. The object query set may be generated based on segment features. The object query set may be generated by a transformer decoder. The transformer decoder may be a k-means transformer decoder configured to produce segment level predictions, wherein the object query is iteratively refined over a plurality of layers.

[0074] At 1104, track attention may be applied. Track attention may be applied to refine a set of object queries corresponding to a plurality of segments. Track attention may be applied by a second sub-model (e.g., second sub-model 106). The second sub-model may be configured to track each object across a plurality of segments. The second sub-model may use track attention and temporal atrous spatial pyramid pooling (temporal ASPP) to improve long-term consistency. At 1106, temporal ASPP may be performed on the set of object queries. Temporal ASPP may be performed on the set of object queries to capture object motion information over different time spans.

[0075] Fig.12 An example process 1200 for implementing video segmentation is shown. Fig.12 1 and 2 are depicted as a series of operations, but one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the depicted operations.

[0076] At 1202, an object query set may be generated. The object query set may correspond to each of a plurality of segments. The object query set may be generated based on segment features. The object query set may be generated by a transformer decoder. The transformer decoder may be a k-means transformer decoder configured to produce segment-level predictions, wherein the object query is iteratively refined over a plurality of layers.

[0077] To further capture global (i.e., entire video) temporal connections, track attention can be applied to all segment-level object queries. Intuitively, since each object query is learned to encode an object in a segment, applying track attention to all object queries learns to track each object across different segments by finding its trajectory in the video. In addition to track attention, temporal ASPP can be used to capture object motion over different time spans. The resulting second sub-model can iteratively stack track attention and temporal ASPP to refine object queries for a video, thereby obtaining temporally consistent predictions at the video level.

[0078] At 1204, track attention and temporal ASPP may be performed iteratively. Track attention and temporal ASPP may be performed iteratively to refine the object query set. Track attention and temporal ASPP may be performed iteratively for generating temporally consistent predictions at the video level. The second sub-model may enable the entire video to be received as input during inference, thereby encouraging temporal consistency in a completely offline manner.

[0079] At 1206, a video level segmentation result may be generated. The video level segmentation result may be generated based on the refined object query. For example, the video level segmenter may obtain the video features by concatenating all the segment features output by the first sub-model, and by multiplying the video features and the refined segment object query. to generate video predictions.

[0080] The system 100 is a unified method for video segmentation both near-online and offline (i.e., the cross-segment tracking submodel is used only in the offline setting). For the near-online setting (i.e., employing the within-segment tracking submodel), a segment size of 2 may be used. For the offline setting (i.e., employing the cross-segment tracking submodel), a video of length 24 (i.e., 12 segments) may be used. In an embodiment, only the first submodel 102 may be trained, while the segment level segmenter and the second submodel 106 are frozen due to memory limitations. During testing, the entire video may be directly inferred using the full model.

[0081] Ablation experiments are performed on the VIPSeg dataset using ResNet50. VIPSeg is a new large-scale video panoramic segmentation dataset for various wild scenes. The dataset contains 124 semantic categories (58 "object" categories and 66 "stuff" categories) and 3536 videos, where each video spans 3 to 10 seconds. On this benchmark, the main evaluation metric adopted is VPQ (Video Panoramic Quality).

[0082] Fig.13 Table 1300 is shown, which shows the ablation results on the intra-segment tracking sub-model. The number of multi-scale deformable attentions (#MSDeformAttn), the number of track attentions (#TrjAttn), and / or the number of axial track attentions (#AxialTrjAttn) are varied. w represents the number of blocks (i.e., repetitions). First, in N w = 2 (i.e., stacking two of the blocks) using one MSDeformAttn (multi-scale deformable attention) and one TrjAttn (trajectory attention), resulting in a performance of 45.3% VPQ. Replacing TrjAttn with the proposed AxialTrjAttn (axial trajectory attention sequentially along the H-axis and W-axis) yields a comparable performance of 45.4%. Stacking two AxialTrjAttn layers in each block results in a performance of 46.1% for the final setting. If two TrjAttn layers are stacked in a V100 GPU, out of memory may occur. Increasing or decreasing the number of blocks N wThe performance is slightly reduced. If one more AxialTrjAttn layer is used per block, the performance drops by 0.4%. Finally, if the iterative stacking scheme is changed to a sequential approach (i.e., stacking two MSDeformAttn and then stacking four AxialTrjAttn), the performance is also slightly reduced by 0.3%.

[0083] Fig.14 A computing device is shown that can be used in various aspects, such as Figure 1-Figure 4 Any of the services, networks, sub-models and / or devices shown in . Figure 1-Figure 4 , any or all of the components can be Fig.14 The system may be implemented by one or more instances of computing device 1400. Fig.14 The computer architecture shown illustrates a conventional server computer, workstation, desktop computer, laptop, tablet computer, network appliance, PDA, e-reader, digital cellular telephone, or other computing node and can be used to perform any aspects of the computer described herein, such as to implement the methods described herein.

[0084] The computing device 1400 may include a substrate or "motherboard" that is a printed circuit board to which a number of components or devices may be connected via a system bus or other electrical communication paths. One or more central processing units (CPUs) 1404 may operate in conjunction with a chipset 1406. The CPU(s) 1404 may be a standard programmable processor that performs the arithmetic and logic operations necessary for the operation of the computing device 1400.

[0085] (Multiple) CPUs 1404 can perform necessary operations by switching from one state to the next via manipulation of switch elements that distinguish and change discrete physical states. Switch elements can typically include electronic circuits that maintain one of two binary states (such as flip-flops), and electronic circuits that provide output states based on logical combinations of the states of one or more other switch elements (such as logic gates). These basic switch elements can be combined to create more complex logic circuits, including registers, adder-subtractors, arithmetic logic units, floating point units, etc.

[0086] CPU(s) 1404 may be augmented or replaced by other processing units, such as GPU(s) 1405. GPU(s) 1405 may include processing units dedicated to, but not necessarily limited to, highly parallel computing, such as graphics processing and other visualization-related processing.

[0087] Chipset 1406 may provide an interface between CPU(s) 1404 and the rest of the components and devices on the substrate. Chipset 1406 may provide an interface to random access memory (RAM) 1408 used as main memory in computing device 1400. Chipset 1406 may also provide an interface to computer-readable storage media such as read-only memory (ROM) 1420 or non-volatile RAM (NVRAM) (not shown) for storing basic routines that may help boot computing device 1400 and transfer information between various components and devices. ROM 1420 or NVRAM may also store other software components necessary for the operation of computing device 1400 according to various aspects described herein.

[0088] The computing device 1400 can operate in a networked environment using logical connections to remote computing nodes and computer systems through a local area network (LAN). Chipset 1406 may include functionality for providing network connectivity through a network interface controller (NIC) 1422, such as a Gigabit Ethernet adapter. NIC 1422 is capable of connecting computing device 1400 to other computing nodes through network 1416. It should be understood that multiple NICs 1422 may be present in computing device 1400, thereby connecting the computing device to other types of networks and remote computer systems.

[0089] The computing device 1400 can be connected to a mass storage device 1428 that provides non-volatile storage for the computer. The mass storage device 1428 can store system programs, applications, other program modules, and data that have been described in more detail herein. The mass storage device 1428 can be connected to the computing device 1400 by a storage controller 1424 connected to the chipset 1406. The mass storage device 1428 can be composed of one or more physical storage units. The mass storage device 1428 may include a management component 1410. The storage controller 1424 can be connected to the physical storage unit through a serial attached SCSI (SAS) interface, a serial advanced technology attachment (SATA) interface, a fiber channel (FC) interface, or other types of interfaces for physically connecting and transmitting data between the computer and the physical storage unit.

[0090] The computing device 1400 can store data on the mass storage device 1428 by transforming the physical state of the physical storage unit to reflect the stored information. The specific transformation of the physical state may depend on various factors and different implementations of this specification. Examples of these factors may include, but are not limited to, the technology used to implement the physical storage unit and whether the mass storage device 1428 is characterized as primary storage or secondary storage, etc.

[0091] For example, the computing device 1400 can store information to the mass storage device 1428 by issuing instructions via the storage controller 1424 to change the magnetic properties of a specific location in the disk drive unit, the reflective or refractive properties of a specific location in the optical storage unit, or the electrical properties of a specific capacitor, transistor or other discrete component in the solid-state storage unit. Other transformations of the physical media are possible without departing from the scope and spirit of the present specification, wherein the foregoing examples are provided only to make the description more convenient. The computing device 1400 can also read information from the mass storage device 1428 by detecting the physical state or properties of one or more specific locations in the physical storage unit.

[0092] In addition to the mass storage device 1428 described above, the computing device 1400 may also access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. It should be understood by those skilled in the art that computer-readable storage media may be any available media that provides storage for non-transitory data and can be accessed by the computing device 1400.

[0093] As an example and not limitation, computer-readable storage media may include volatile computer-readable storage media and non-volatile computer-readable storage media, temporary computer-readable storage media and non-temporary computer-readable storage media, and removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory or other solid-state memory technology, compact disk ROM (CD-ROM), digital versatile disk (DVD), high-definition DVD (HD-DVD), Blu-ray or other optical storage devices, cassettes, magnetic tapes, magnetic disk storage devices, other magnetic storage devices, or any other medium that can be used to store desired information in a non-temporary manner.

[0094] Mass storage devices (such as Fig.14 The mass storage device 1428 shown can store an operating system for controlling the operation of the computing device 1400. The operating system can include a version of the LINUX operating system. The operating system can include a version of the WINDOWS server operating system from Microsoft. According to a further aspect, the operating system can include a version of the UNIX operating system. Various mobile phone operating systems, such as IOS and ANDROID, can also be utilized. It should be understood that other operating systems can also be utilized. The mass storage device 1428 can store other systems or applications and data utilized by the computing device 1400.

[0095] The mass storage device 1428 or other computer-readable storage medium may also be encoded with computer-executable instructions that, when loaded into the computing device 1400, transform the computing device from a general-purpose computing system into a special-purpose computer capable of implementing the aspects described herein. As described above, these computer-executable instructions transform the computing device 1400 by specifying how the CPU(s) 1404 transition between states. The computing device 1400 may access a computer-readable storage medium storing computer-executable instructions that, when executed by the computing device 1400, may perform the methods described herein.

[0096] Computing devices (such as Fig.14 The computing device 1400 shown may also include an input / output controller 1432 for receiving and processing input from a plurality of input devices, such as a keyboard, a mouse, a touch pad, a touch screen, an electronic stylus, or other types of input devices. Similarly, the input / output controller 1432 may provide output to a display, such as a computer monitor, a flat panel display, a digital projector, a printer, a plotter, or other types of output devices. It should be understood that the computing device 1400 may not include Fig.14 All components shown in the Fig.14 Other components not explicitly shown in the figure or may be used in conjunction with Fig.14 A completely different architecture is shown in .

[0097] As described herein, a computing device may be a physical computing device such as Fig.14 The computing device 1400 of the embodiment of the present invention may further include a virtual machine host process and one or more virtual machine instances. The computer executable instructions may be indirectly executed by the physical hardware of the computing device by interpreting and / or executing instructions stored and executed in the context of the virtual machine.

[0098] It should be understood that these methods and systems are not limited to specific methods, specific components or specific implementations. It should also be understood that the terminology used herein is for the purpose of describing specific embodiments only and is not intended to be limiting.

[0099] As used in the specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" one particular value and / or to "about" another particular value. When such a range is expressed, another embodiment includes from a particular value and / or to another particular value. Similarly, when a value is expressed as being approximate by using the antecedent "about", it will be understood that the particular value forms another embodiment. It will also be understood that the endpoints of each range in a range are significant both relative to the other endpoint and independently of the other endpoint.

[0100] "Optional" or "optionally" means that the subsequently described event or circumstance may or may not occur, and that the description includes instances where said event or circumstance occurs and instances where it does not.

[0101] Throughout the description and claims of this specification, the word "include" and variations of the word (such as "comprising" and "including") mean "including but not limited to", and are not intended to exclude, for example, other components, integers or steps. "Exemplary" means "an example of..." and is not intended to convey an indication of a preferred embodiment or an ideal embodiment. "Such as" is not used in a limiting sense, but for the purpose of explanation.

[0102] Components that can be used to perform the described methods and systems are described. When describing combinations, subsets, interactions, groupings, etc. of these components, it should be understood that, although specific references to each of the various individual and common combinations and arrangements of these components may not be explicitly described, for all methods and systems, each is specifically contemplated and described herein. This applies to all aspects of the application, including but not limited to the operations in the described methods. Therefore, if there are various additional operations that can be performed, it should be understood that each of these additional operations can be performed using any specific embodiment or combination of embodiments in the described methods.

[0103] The presented methods and systems may be understood more readily by reference to the following detailed description of the preferred embodiments and the examples included therein and to the accompanying drawings and their descriptions.

[0104] As will be appreciated by those skilled in the art, these methods and systems may take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software aspects and hardware aspects. In addition, these methods and systems may take the form of a computer program product on a computer-readable storage medium in which computer-readable program instructions (e.g., computer software) are embodied. More particularly, the presented methods and systems may take the form of network-implemented computer software. Any suitable computer-readable storage medium may be utilized, including a hard disk, a CD-ROM, an optical storage device, or a magnetic storage device.

[0105] Embodiments of these methods and systems are described below with reference to block diagrams and flow charts of methods, systems, devices, and computer program products. It will be appreciated that each block in the block diagrams and flow charts and combinations of blocks in the block diagrams and flow charts can be implemented by computer program instructions, respectively. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed on the computer or other programmable data processing device create components for implementing the functions specified in one or more flow chart blocks.

[0106] These computer program instructions may also be stored in a computer-readable memory, which may direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including computer-readable instructions for implementing the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device, thereby producing a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flowchart blocks.

[0107] The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, in some embodiments, certain method or process blocks can be omitted. The methods and processes described herein are also not limited to any particular order, and the blocks or states associated therewith can be performed in other appropriate orders. For example, the blocks or states described can be performed in an order different from the specific description, or multiple blocks or states can be combined in a single block or state. The example blocks or states can be performed serially, in parallel, or in some other way. Blocks or states can be added to the described exemplary embodiments, or blocks or states can be deleted from the described exemplary embodiments. The example systems and components described herein can be configured in different ways than described. For example, compared with the described example embodiments, elements can be added to, removed from, or rearranged.

[0108] It will also be understood that various items are shown as being stored in memory or on storage devices while being used, and these items or portions thereof may be transferred between memory and other storage devices for the purpose of memory management and data integrity. Alternatively, in other embodiments, some or all of the software modules and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. In addition, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways, such as at least partially implemented in firmware and / or hardware, including but not limited to one or more application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the modules, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a computer-readable medium (such as a hard disk, memory, network, or portable media product) for reading by an appropriate device or via an appropriate connection. Systems, modules, and data structures may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless-based media and wired / cable-based media), and may take various forms (e.g., as part of a single analog signal or a multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products may also take other forms. Accordingly, the present invention may be practiced using other computer system configurations.

[0109] Although these methods and systems have been described in conjunction with preferred embodiments and specific examples, it is not intended to limit the scope to the specific embodiments set forth, as the embodiments herein are intended in all respects to be illustrative rather than restrictive.

[0110] Unless otherwise expressly stated, it is not intended that any method described herein be interpreted as requiring that its operations be performed in a specific order. Accordingly, in the event that a method claim does not actually recite the order in which its operations are to be followed, or in the event that the operations are not otherwise specifically stated in the claims or specification to be limited to a specific order, no order is intended to be inferred in any respect. This applies to any possible non-explicit basis for interpretation, including: logical issues regarding the arrangement of steps or operational flow; plain meaning derived from grammatical organization or punctuation; and the number or type of embodiments described in the specification.

[0111] It will be apparent to those skilled in the art that various modifications and variations may be made without departing from the scope or spirit of the present disclosure. Other embodiments will be apparent to those skilled in the art in view of the specification and practice described herein. It is intended that the specification and exemplary drawings be considered merely exemplary, with the true scope and spirit being indicated by the appended claims.

Claims

1. A method for implementing video segmentation, comprising: dividing the video into a plurality of segments, wherein each segment of the plurality of segments comprises a plurality of frames; applying, by a first sub-model, axial track attention to each of the plurality of segments, wherein the first sub-model is configured to track an object across frames within the same segment of the plurality of segments; generating, by the first sub-model, a segment feature corresponding to each segment of the plurality of segments; generating, by a transformer decoder, a set of object queries corresponding to each of the plurality of segments based on the segment features; applying, by a second sub-model, track attention to refine a set of object queries corresponding to the plurality of segments, wherein the second sub-model is configured to track each object across the plurality of segments; as well as A video level segmentation result is generated based on the refined object query.

2. The method according to claim 1, further comprising: generating a sequence of pixel features along a height axis based on extracting features from each frame in the same segment; as well as The axial trajectory attention is performed along the altitude axis.

3. The method according to claim 1, further comprising: generating a sequence of pixel features along a width axis based on extracting features from each frame in the same segment; as well as The axial trajectory attention is performed along the width axis.

4. The method according to claim 1, further comprising: Multi-scale deformable attention is performed on feature maps extracted from frames in the same segment to exchange information across feature map scales and ensure spatial domain consistency of the segment features.

5. The method according to claim 4, further comprising: Iteratively performing the multi-scale deformable attention and the axial trajectory attention by the first sub-model; as well as The segment features having temporal domain consistency and spatial domain consistency are output from the first sub-model. 6 . The method of claim 1 , wherein the set of object queries corresponding to each of the plurality of segments is iteratively refined over multiple layers of the transformer decoder.

7. The method of claim 1, wherein applying, by the second sub-model, track attention to refine the set of object queries corresponding to the plurality of segments further comprises: Projecting each object query in the object query set into a query key value vector set; Compute the set of trajectory queries by computing the probabilistic path for each object query; Linearly projecting the trajectory query set into a new query key value vector set; as well as Based on the new set of query key vectors, the entire video temporal connections across the multiple segments are aggregated.

8. The method according to claim 1, further comprising: A temporal atrous spatial pyramid pooling (ASPP) is performed on the object query set to capture object motion information of different time spans.

9. The method according to claim 8, further comprising: The track attention and the temporal ASPP are iteratively performed by the second sub-model to refine the object query set for generating temporally consistent predictions at the video level.

10. A system for implementing video segmentation, comprising: at least one processor; as well as at least one memory communicatively coupled to the at least one processor and comprising computer readable instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: dividing the video into a plurality of segments, wherein each segment of the plurality of segments comprises a plurality of frames; applying, by a first sub-model, axial track attention to each of the plurality of segments, wherein the first sub-model is configured to track an object across frames within the same segment of the plurality of segments; generating, by the first sub-model, a segment feature corresponding to each segment of the plurality of segments; generating, by a transformer decoder, a set of object queries corresponding to each of the plurality of segments based on the segment features; applying, by a second sub-model, track attention to refine a set of object queries corresponding to the plurality of segments, wherein the second sub-model is configured to track each object across the plurality of segments; and Based on the refined object query, a video level segmentation result is generated.

11. The system of claim 10, wherein the operations further comprise: generating a sequence of pixel features along a height axis based on extracting features from each frame in the same segment; as well as The axial trajectory attention is performed along the altitude axis.

12. The system of claim 10, wherein the operations further comprise: generating a sequence of pixel features along a width axis based on extracting features from each frame in the same segment; as well as The axial trajectory attention is performed along the width axis.

13. The system of claim 10, wherein the operations further comprise: Multi-scale deformable attention is performed on feature maps extracted from frames in the same segment to exchange information across feature map scales and ensure spatial domain consistency of the segment features.

14. The system of claim 10, wherein applying, by the second sub-model, track attention to refine the set of object queries corresponding to the plurality of segments further comprises: Projecting each object query in the object query set into a query key value vector set; Compute the set of trajectory queries by computing the probabilistic path for each object query; Linearly projecting the trajectory query set into a new query key value vector set; as well as Based on the new set of query key vectors, the entire video temporal connections across the multiple segments are aggregated.

15. The system of claim 10, the operations further comprising: A temporal atrous spatial pyramid pooling (ASPP) is performed on the object query set to capture object motion information of different time spans.

16. A non-transitory computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, cause the processor to perform operations comprising: dividing the video into a plurality of segments, wherein each segment of the plurality of segments comprises a plurality of frames; applying, by a first sub-model, axial track attention to each of the plurality of segments, wherein the first sub-model is configured to track an object across frames within the same segment of the plurality of segments; generating, by the first sub-model, a segment feature corresponding to each segment of the plurality of segments; generating, by a transformer decoder, a set of object queries corresponding to each of the plurality of segments based on the segment features; applying, by a second sub-model, track attention to refine a set of object queries corresponding to the plurality of segments, wherein the second sub-model is configured to track each object across the plurality of segments; as well as Based on the refined object query, a video level segmentation result is generated.

17. The non-transitory computer-readable storage medium of claim 16, the operations further comprising: generating a sequence of pixel features along a height axis based on extracting features from each frame in the same segment; as well as The axial trajectory attention is performed along the altitude axis.

18. The non-transitory computer-readable storage medium of claim 16, the operations further comprising: generating a sequence of pixel features along a width axis based on extracting features from each frame in the same segment; as well as The axial trajectory attention is performed along the width axis.

19. The non-transitory computer-readable storage medium of claim 16, the operations further comprising: Multi-scale deformable attention is performed on feature maps extracted from frames in the same segment to exchange information across feature map scales and ensure spatial domain consistency of the segment features.

20. The non-transitory computer-readable storage medium of claim 16, the operations further comprising: A temporal atrous spatial pyramid pooling (ASPP) is performed on the object query set to capture object motion information of different time spans.

Images