A spatiotemporal parallel sparse memory network for efficient video object segmentation

Abstract

This invention provides a spatiotemporally parallel sparse memory network for efficient video target segmentation, comprising a spatiotemporally parallel memory construction component and a parallel sparse memory reader. The spatiotemporally parallel memory construction component simultaneously mines temporal and spatial memory values ​​from multiple historical frames to comprehensively capture discriminative video target information. The temporal and spatial mappings of each memory frame are stacked along the temporal dimension to form a temporal memory bank and a spatial memory bank. The parallel sparse memory reader is used to uniformly and parallelly sample sparse memory frames from the temporal and spatial memory banks. This invention can accurately and efficiently segment video targets; it can enhance the ability to focus on dynamic regions, thereby improving segmentation accuracy and limiting the segmentation range of moving objects and static backgrounds; by aggregating spatial context information, it expands the receptive field of a single frame to a multi-frame receptive field; and by uniform sampling, it can directly and effectively reduce frame-level memory redundancy, thereby achieving acceleration without affecting performance.

Description

Technical Field

[0001] This invention relates to the field of video object segmentation technology, and in particular to a spatiotemporal parallel sparse memory network for efficient video object segmentation. Background Technology

[0002] Video object segmentation is the foundation of video understanding tasks in computer vision. Its purpose is to track objects in video sequences and separate foreground pixels from background pixels. It has wide applications in video editing, autonomous driving, robot navigation, and virtual reality.

[0003] However, video object segmentation is a challenging task, especially in semi-supervised settings where only the first frame of the video is labeled with an object mask, requiring the algorithm to predict the objects in the remaining frames. Semi-supervised video object segmentation tasks demand that algorithms accurately and continuously track and segment objects in the video, maintaining high accuracy even when the objects are moving rapidly, changing in shape, partially occluded, or have similar instances. Furthermore, considering the large scale of video data, the algorithm must also be efficient and robust enough to adapt to changes in lighting and background interference, meeting the needs of real-time applications.

[0004] In video object segmentation tasks, various methods exist, including propagation-based, detection-based, and matching-based methods. Among them, matching-based methods have attracted widespread attention due to their superior performance. These methods improve segmentation accuracy by storing all past frames and performing pixel-level matching within them to retrieve relevant information. However, these methods still suffer from inaccurate segmentation and low efficiency in complex video scenarios in real-world applications, as specifically manifested below:

[0005] 1. The memory mode based on independent frames does not take into account the temporal motion information between consecutive frames, which limits the ability to perceive moving targets.

[0006] 2. The limited receptive field of a single frame hinders the effective use of spatial context information, resulting in poor performance in distinguishing objects with similar appearances;

[0007] 3. As the video length increases, the computational burden of densely matching the query frame with all memory frames becomes heavy, and noise will inevitably be introduced, causing interference. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a spatiotemporal parallel sparse memory network for efficient video object segmentation. This invention can efficiently and accurately retrieve useful temporal motion information and spatial context information from these two memory groups in parallel.

[0009] The technical solution of this invention is as follows: a spatiotemporal parallel sparse memory network for efficient video target segmentation, comprising a spatiotemporal parallel memory construction component and a parallel sparse memory reader. The spatiotemporal parallel memory construction component simultaneously mines temporal and spatial memory values ​​from multiple historical frames to comprehensively capture discriminative video target information. The temporal and spatial mappings of each memory frame are stacked along the temporal dimension to form a temporal memory bank and a spatial memory bank. The parallel sparse memory reader is used to uniformly and parallelly sample sparse memory frames from the temporal memory bank and the spatial memory bank.

[0010] Preferably, the spatiotemporal parallel memory construction component includes a temporal memory construction module and a spatial memory construction module; wherein, the temporal memory construction module models the long-term motion of video targets by capturing inter-frame changes; and the spatial memory construction module enhances the ability to distinguish similar instances by aggregating more discriminative spatial context information in a local region.

[0011] Preferably, the time memory construction module includes a global attention enhancement unit and a motion perception memory enhancement unit.

[0012] Preferably, the Global Attention Enhancement Unit (GEA) includes a Global Average Pooling Layer (GAP), a Convolutional Layer (f0), and a sigmoid Layer (S0).

[0013] Preferably, the Global Average Pooling (GAP) layer aggregates past frames x t-I memory value V t-I To obtain the global embedding of the entire image, it is then passed to the convolutional layer f0 and the sigmoid layer S0 to compute the global attention mask. The global attention mask Selectively enhance encoder E V Various features in the image are used to merge complete image information into the memory value.

[0014] Preferably, the global attention enhancement unit calculates the global attention mask. The formula for calculation is:

[0015]

[0016] In the formula, For global attention mask; ⊙ represent element-wise summation and element-wise multiplication operations, respectively; f GAE V represents the global attention enhancement operation. t-I For past frames x t-I The memory value; Conv represents the convolutional layer operation; GAP represents the global average pooling layer aggregation operation; σ is the Sigmoid activation function.

[0017] Preferably, the motion-aware memory enhancement unit includes a convolutional layer f1, a sigmoid layer S1, and a global attention enhancement unit GEA.

[0018] Preferably, the motion-aware memory enhancement unit first calculates the past frame x t-I and query frame x t The changes between them, and the previously predicted mask m t-I Then concatenate the parts and input them into encoder E. V Then, the frame change information of the concatenated features is input into the convolutional layer f1 and the sigmoid layer S1 to generate a motion-aware attention mask; then, the motion-aware attention mask is combined with the global attention mask. After multiplication, the input to the Global Attention Enhancement Unit (GEA) is used to calculate an instance of enhanced motion perception memory value.

[0019] Preferably, the motion-sensing memory value The formula for calculation is:

[0020]

[0021] f GAE This indicates a global attention enhancement operation. For motion perception memory values; σ is the global attention mask; E is the sigmoid activation function; V To enhance the encoder; x t x t-I These represent the query frame and the past frame, respectively; ⊙ represents element-wise multiplication, and f1 represents the convolutional layer operation.

[0022] Preferably, the spatial memory building module includes a convolutional layer f2, a sigmoid layer S2, and a global attention enhancement unit (GEA).

[0023] Preferably, the spatial memory construction module will use the previous frame x t-I Predictive mask m t-I With query frame x t The data is stitched together to generate a spatial relationship diagram, which is then input into encoder E. V Spatial attention masks are generated through convolutional layer f2 and sigmoid layer S2; then the spatial attention mask is combined with motion-aware memory values.

[0024] The input Global Attention Augmentation Unit (GEA) generates global context embedding values.

[0025] Preferably, the global context embedding value The formula for calculation is:

[0026]

[0027] In the formula, The value is embedded in the global context; f GAE To enhance global attention operations, For motion-sensing memory values; ⊙ represents element-wise multiplication; σ is the Sigmoid activation function; f2 is the convolutional layer operation; E V To enhance the encoder; m t-I x for the previous frame t-I The prediction mask; x t For query frames.

[0028] Preferably, the parallel sparse memory reader includes a temporal memory sampling reader and a spatial memory sampling reader; the temporal memory sampling reader and the spatial memory sampling reader respectively retrieve data from the temporal memory database. and spatial memory bank A set of sparse memory frames is uniformly sampled along the time dimension; and soft weights are calculated by the sparse matching similarity between the query frame and the uniformly sampled memory frames. The formula for calculating the similarity is as follows:

[0029]

[0030] In the formula, Sim represents the similarity calculation function, U represents uniform sampling of the memory frame, (K L ) j Represents the key mapping of the memory, This represents the key mapping of the query frame, where i and j represent the indexes.

[0031] Preferably, the time memory sampling reader retrieves relevant temporal motion information from a uniformly sampled sparse time memory database based on a similarity score obtained through soft-weighted summation, and employs a top-k strategy to filter the similarity score. The time memory sampling reader is defined as follows:

[0032]

[0033] In the formula, This indicates the relevant timing motion information read. This is a value mapping for the query frame; topk selects the top k elements. Value mapping for the time memory; (K L ) j Z is the key mapping of the memory bank; Z is the normalization factor; [·,·] represents the concatenation operation; U represents uniform sampling of the memory frame.

[0034] Preferably, the spatial memory sampling reader efficiently retrieves the spatial context information of the target from a uniformly sampled sparse spatial memory bank based on a similarity score. The spatial memory sampling reader is defined as follows:

[0035]

[0036] In the formula, To retrieve the relevant spatial context information, This is a value mapping for the query frame; topk selects the top k elements. Value mapping for the spatial memory bank; (K L ) j For the key mapping of the memory bank; Z is the normalization factor; [·,·] denotes the concatenation operation; U denotes uniform sampling of the memory frame;

[0037] Finally, the obtained spatial context information Temporal motion information value mapping of query frames Perform concatenation to obtain a prediction mask m with sufficient target representation. t .

[0038] The beneficial effects of this invention are as follows:

[0039] 1. This invention simultaneously mines temporal and spatial memory values ​​from multiple historical frames to comprehensively capture distinctive video target information. The temporal and spatial mappings of each memory frame are stacked along the temporal dimension to form a temporal memory bank and a spatial memory group. It efficiently and accurately retrieves useful temporal motion information and spatial context information from these two memory groups in parallel, thereby enabling accurate and efficient segmentation of video targets.

[0040] 2. This invention can enhance the ability to focus on dynamic regions, thereby improving segmentation accuracy and limiting the segmentation range between moving objects and static backgrounds;

[0041] 3. This invention expands the receptive field of a single frame to a multi-frame receptive field by aggregating spatial context information;

[0042] 4. The uniform sampling of the present invention can directly and effectively reduce frame-level memory redundancy, thereby achieving acceleration without affecting performance. Attached Figure Description

[0043] Figure 1 This is a framework diagram of the parallel sparse memory network of the present invention;

[0044] Figure 2 This is a framework diagram of the spatiotemporal parallel memory construction component of the present invention;

[0045] Figure 3This is a framework diagram of the parallel sparse memory reader of the present invention;

[0046] Figure 4 This is a qualitative comparison chart of the experimental results of Embodiment 1 of the present invention and other methods on the YouTube-VOS2018 validation set;

[0047] Figure 5 This is a qualitative comparison of the experimental results of Embodiment 1 of the present invention and other methods on the YouTube-VOS2019 validation set; wherein, (a) is a qualitative comparison of the experimental results of other methods on the YouTube-VOS2019 validation set; (b) is a qualitative comparison of the segmentation results of the present invention under illumination changes on the YouTube-VOS2019 validation set; and (c) is a qualitative comparison of the segmentation results of the present invention method on the DAVIS2017 test set for objects with complex shapes.

[0048] Figure 6 This is a qualitative result diagram of Embodiment 1 of the present invention in a dynamic traffic scenario. Detailed Implementation

[0049] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0050] Example 1

[0051] like Figure 1 As shown, this embodiment provides a spatiotemporal parallel sparse memory network for efficient video target segmentation, including a spatiotemporal parallel memory construction component (TSPMC) and a parallel sparse memory reader (PSMR). The TSPMC simultaneously mines temporal and spatial memory values ​​from multiple historical frames to comprehensively capture discriminative video target information. The temporal and spatial mappings of each memory frame are stacked along the temporal dimension to form a temporal memory bank and a spatial memory bank. The PSMR is used to uniformly and parallelly sample sparse memory frames from the temporal and spatial memory banks.

[0052] As a preferred embodiment, such as Figure 2 As shown, the Spatiotemporal Parallel Memory Construction Component (TSPMC) includes a temporal memory construction module and a spatial memory construction module. The temporal memory construction module models the long-term motion of video targets by capturing inter-frame changes; the spatial memory construction module enhances the ability to distinguish similar instances by aggregating more discriminative spatial context information in a local region.

[0053] As a preferred embodiment, such as Figure 2 As shown, the time memory construction module includes a global attention enhancement unit and a motion perception memory enhancement unit.

[0054] As a preferred embodiment, such as Figure 2 As shown, the Global Attention Enhancement Unit (GEA) includes a global average pooling layer (GAP), a convolutional layer (f0), and a sigmoid layer (S0). Figure 2 As shown in (a), the Global Average Pooling (GAP) layer aggregates past frames x t-I memory value V t-I To obtain the global embedding of the entire image, it is then passed to the convolutional layer f0 and the sigmoid layer S0 to compute the global attention mask. The global attention mask Selectively enhance encoder E V Various features are used to merge complete image information into a memory value. The global attention enhancement unit calculates the global attention mask. The formula for calculation is:

[0055]

[0056] In the formula, For global attention mask; ⊙ represent element-wise summation and element-wise multiplication operations, respectively; f GAE V represents the global attention enhancement operation. t-I For past frames x t-I The memory value; Conv represents the convolutional layer operation; GAP represents the global average pooling layer aggregation operation; σ is the Sigmoid activation function.

[0057] As a preferred embodiment, such as Figure 2 As shown in (b), the motion-aware memory enhancement unit includes a convolutional layer f1, a sigmoid layer S1, and a global attention enhancement unit (GEA). The motion-aware memory enhancement unit first calculates the past frame x. t-I and query frame x t The changes between them, and the previously predicted mask m t-I Then concatenate the parts and input them into encoder E. V Then, the frame change information of the concatenated features is input into the convolutional layer f1 and the sigmoid layer S1 to generate a motion-aware attention mask; then, the motion-aware attention mask is combined with the global attention mask. After multiplication, the input to the Global Attention Enhancement Unit (GEA) is used to calculate an instance of enhanced motion perception memory value.

[0058] In this embodiment, the motion-sensing memory value The formula for calculation is:

[0059]

[0060] fGAE This indicates a global attention enhancement operation. For motion perception memory values; σ is the global attention mask; E is the sigmoid activation function; V To enhance the encoder; x t x t-I These represent the query frame and the past frame, respectively; ⊙ represents element-wise multiplication, and f1 represents the convolutional layer operation.

[0061] As a preferred embodiment, such as Figure 2 As shown in (c), the spatial memory construction module includes a convolutional layer f2, a sigmoid layer S2, and a global attention enhancement unit (GEA). The spatial memory construction module stores the previous frame x... t-I Predictive mask m t-I With query frame x t The data is stitched together to generate a spatial relationship diagram, which is then input into encoder E. V Spatial attention masks are generated through convolutional layer f2 and sigmoid layer S2; then the spatial attention mask is combined with motion-aware memory values. The input Global Attention Augmentation Unit (GEA) generates global context embedding values.

[0062] In this embodiment, the global context embedding value The formula for calculation is:

[0063]

[0064] In the formula, The value is embedded in the global context; f GAE To enhance global attention operations, For motion-sensing memory values; ⊙ represents element-wise multiplication; σ is the Sigmoid activation function; f2 is the convolutional layer operation; E V To enhance the encoder; m t-I x for the previous frame t-U The prediction mask; x t For query frames.

[0065] As a preferred embodiment, such as Figure 3 As shown, the parallel sparse memory reader includes a temporal memory sampling reader and a spatial memory sampling reader; the temporal memory sampling reader and the spatial memory sampling reader respectively retrieve data from the temporal memory bank. and spatial memory bank A set of sparse memory frames is uniformly sampled along the time dimension; and soft weights are calculated by the sparse matching similarity between the query frame and the uniformly sampled memory frames. The formula for calculating the similarity is as follows:

[0066]

[0067] In the formula, Sim is the similarity calculation function, U represents uniform sampling of the memory frame, (K L ) h Represents the key mapping of the memory, This indicates the key mapping of the query frame.

[0068] As a preferred embodiment, such as Figure 3 As shown, the time memory sampling reader retrieves relevant temporal motion information from a uniformly sampled sparse time memory database based on a similarity score calculated using soft weights, and employs a top-k strategy to filter the similarity score. The time memory sampling reader is defined as follows:

[0069]

[0070] In the formula, This indicates the relevant timing motion information read. This is a value mapping for the query frame; topk selects the top k elements. Value mapping for the time memory; (K L ) j Z is the key mapping of the memory bank; Z is the normalization factor; [·,·] represents the concatenation operation; U represents uniform sampling of the memory frame.

[0071] As a preferred embodiment, such as Figure 3 As shown, the spatial memory sampling reader efficiently retrieves the spatial context information of a target from a uniformly sampled sparse spatial memory bank based on similarity scores. The spatial memory sampling reader is defined as follows:

[0072]

[0073] In the formula, To retrieve the relevant spatial context information, This is a value mapping for the query frame; topk selects the top k elements. Value mapping for the spatial memory bank; (K L ) j For the key mapping of the memory bank; Z is the normalization factor; [·,·] denotes the concatenation operation; U denotes uniform sampling of the memory frame;

[0074] Finally, the obtained spatial context information Temporal motion information value mapping of query frames Perform concatenation to obtain a prediction mask m with sufficient target representation. t .

[0075] Example 2

[0076] This embodiment verifies the performance of Example 1 by acquiring multiple video object segmentation benchmark datasets, namely DAVIS2016, DAVIS2017, and YouTube-VOS2018. DAVIS2016 is a single-object segmentation benchmark containing 50 videos and 3455 labeled frames. DAVIS2017 is a multi-object benchmark extension of DAVIS2016, with training, validation, and test sets containing 60, 30, and 30 videos, respectively. YouTube-VOS2018 is a large-scale multi-object benchmark, with training and validation sets containing 3471 and 474 videos, respectively.

[0077] This embodiment uses region similarity. Contour accuracy and average score As an evaluation metric, it is also used to measure inference speed in frames per second (FPS).

[0078] This embodiment processes multiple video object segmentation benchmark datasets using the embodiment network and existing methods, and the results are shown in Tables 1 and 2. Simultaneously, this embodiment verifies the compatibility of the embodiment 1 network by using the Spatiotemporal Parallel Memory Construction Component (TSPMC) as a plug-in, and the results are shown in Table 3.

[0079] Table 1 provides a quantitative comparison of experimental results between the network of Example 1 and existing methods on the YouTube-VOS2018 validation set, DAVIS2017 validation set, and test set; where the subscripts S and U represent seen or unseen categories, respectively, and "*" indicates that the method is trained only on the DAVIS2017 and YouTube-VOS2018 validation sets.

[0080]

[0081]

[0082]

[0083]

[0084] Table 2. Quantitative comparison of experimental results of Example 1 and other methods on the YouTube-VOS2019 and DAVIS2016 validation sets.

[0085]

[0086] Table 3 shows the compatibility test results of the TSPMC module with different methods on the DAVIS2017 validation set.

[0087]

[0088]

[0089] As can be seen from Tables 1-3, Example 1 outperforms other methods in segmentation accuracy and efficiency on multiple datasets. Among them, TSPMC has strong compatibility and can be combined with different video object segmentation methods to improve its performance.

[0090] To visually and intuitively evaluate the segmentation effect of the method of the present invention, the experimental results of this embodiment 1 and other methods on the YouTube-VOS2018 and YouTube-VOS2019 validation sets are qualitatively compared, such as... Figure 4 and Figure 5 As shown in (a), the segmentation effect of Example 1 was tested under different complex scenarios, and the results are as follows. Figure 5 (b) Figure 5 (c) and Figure 6 As shown. Qualitative results indicate that the network in Example 1 performs better in tracking fast-moving objects and distinguishing similar objects, exhibits strong robustness to continuous changes in background lighting, effectively segments objects with complex shapes, and demonstrates excellent segmentation performance in challenging traffic scenarios.

[0091] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. An efficient video target segmentation method based on spatiotemporal parallel sparse memory networks, characterized in that: The system includes a spatiotemporal parallel memory construction component and a parallel sparse memory reader. The spatiotemporal parallel memory construction component includes a temporal memory construction module and a spatial memory construction module. The parallel sparse memory reader includes a temporal memory sampling reader and a spatial memory sampling reader. The spatiotemporal parallel memory construction component simultaneously mines temporal and spatial memory values ​​from multiple historical frames to comprehensively capture discriminative video target information. The temporal and spatial mappings of each memory frame are stacked along the temporal dimension to form a temporal memory bank and a spatial memory bank. The parallel sparse memory reader is used to uniformly and parallelly sample sparse memory frames from the temporal memory bank and the spatial memory bank. The time memory construction module models the long-term motion of video targets by capturing inter-frame changes; the time memory construction module includes a global attention enhancement unit and a motion perception memory enhancement unit. The Global Attention Enhancement Unit (GEA) includes a Global Average Pooling Layer (GAP) and a Convolutional Layer. The sigmoid layer S0; the global average pooling layer GAP aggregates past frames. memory value To obtain the global embedding of the entire image, it is then passed to the convolutional layer. f 0 And a sigmoid layer S0 to compute the global attention mask. The global attention mask mentioned Selectively enhance the encoder Various features in the image are used to merge complete image information into memory values; The motion-sensing memory enhancement unit includes convolutional layers. f 1. A sigmoid layer S1 and a global attention enhancement unit (GEA); The motion-sensing memory enhancement unit first calculates past frames. and query frame The changes between them, and the previously predicted mask. Then concatenate the parts and input them into the encoder. Then, the frame change information of the spliced ​​features is input into the convolutional layer. f The motion-aware attention mask is generated by layer 1 and the sigmoid layer S1; then the motion-aware attention mask is combined with the global attention mask. After multiplication, the input to the Global Attention Enhancement Unit (GEA) is used to calculate an instance of enhanced motion perception memory value. ; The spatial memory construction module aggregates discriminative spatial context information over a local region, and includes convolutional layers. f 2. A sigmoid layer S2 and a global attention enhancement unit (GEA).

2. The efficient video target segmentation method based on spatiotemporal parallel sparse memory network according to claim 1, characterized in that: The global attention enhancement unit calculates the global attention mask. The formula for calculation is: ； In the formula, For global attention mask; , These represent element-wise summation and element-wise multiplication operations, respectively. This indicates a global attention enhancement operation. For past frames The memory value; Indicates convolutional layer operations; This indicates a global average pooling layer aggregation operation; This is the Sigmoid activation function.

3. The efficient video target segmentation method based on spatiotemporal parallel sparse memory network according to claim 1, characterized in that: The motion perception memory value The formula for calculation is: ； This indicates a global attention enhancement operation. For motion perception memory values; For global attention mask; Use the Sigmoid activation function; To enhance the encoder; , These are the query frame and the past frame, respectively. This is an element-wise multiplication operation. This refers to operations on convolutional layers.

4. The efficient video target segmentation method based on spatiotemporal parallel sparse memory network according to claim 3, characterized in that: The aforementioned spatial memory construction module will use the previous frame Prediction mask With query frame The data is stitched together to generate a spatial relationship diagram, which is then input into the encoder. ; and through convolutional layers f 2. The sigmoid layer S2 generates a spatial attention mask; then the spatial attention mask is combined with the motion-aware memory value. The input Global Attention Augmentation Unit (GEA) generates global context embedding values. ; In the formula, Embedded values ​​for the global context; To enhance global attention operations, For motion perception memory values; This is an element-wise multiplication operation; Use the Sigmoid activation function; For convolutional layer operations; To enhance the encoder; The previous frame The prediction mask; For query frames.

5. The efficient video target segmentation method based on spatiotemporal parallel sparse memory network according to claim 1, characterized in that: The time memory sampling reader and the spatial memory sampling reader respectively retrieve data from the time memory bank. and spatial memory bank A sparse set of memory frames is uniformly sampled along the time dimension. The soft weights are calculated by using the sparse matching similarity between the query frame and the uniformly sampled memory frames. The formula for calculating the similarity is as follows: In the formula, For similarity calculation functions, This indicates uniform sampling of memory frames. Represents the key mapping of the memory, This indicates the key mapping of the query frame. i , j Indicates an index.

6. The efficient video target segmentation method based on spatiotemporal parallel sparse memory network according to claim 5, characterized in that: The time memory sampling reader retrieves relevant temporal motion information from a uniformly sampled sparse time memory database based on a similarity score calculated using soft weights, and employs a top-k strategy to filter the similarity score. The time memory sampling reader is defined as follows: In the formula, This indicates the relevant timing motion information read. Mapping values ​​for query frames; This indicates that the topmost one is selected. k One element; Value mapping for the time memory; For key mapping of the memory; Normalization factor; Indicates a serial operation; This indicates uniform sampling of memory frames.

7. The efficient video target segmentation method based on spatiotemporal parallel sparse memory network according to claim 6, characterized in that: The spatial memory sampling reader efficiently retrieves the spatial context information of a target from a uniformly sampled sparse spatial memory bank based on similarity scores. The spatial memory sampling reader is defined as follows: In the formula, To retrieve the relevant spatial context information, Mapping values ​​for query frames; To select the top one k One element; Value mapping for the spatial memory bank; For key mapping of the memory; It is a normalization factor; Indicates a serial operation; This indicates uniform sampling of memory frames; Finally, the obtained spatial context information Time-series motion information value mapping of query frames Perform concatenation to obtain a predictive mask with sufficient target representation. m t .

Images