Single-stage dense event description method based on dynamic time sequence anchor frame

Abstract

The invention discloses a single-stage dense video description method based on a dynamic time sequence anchor frame, which is used for solving the problems of inaccurate long video complex event positioning, incomplete description semantics, low training efficiency and the like. According to the method, an encoder-decoder architecture is adopted, and the method comprises the steps that firstly, semantic enhanced multi-dimensional primary video features are constructed through a visual feature backbone network and an MLP, and the semantic enhanced multi-dimensional primary video features are input into an encoder of Mamba to be subjected to bidirectional space-time scanning to generate refined features; and the decoder realizes interaction between event proposals and video feature fusion through self-attention and cross attention, optimizes attention calculation in combination with conditional space query and scale modulation, iteratively updates anchor frames and proposals, and predicts final model output through a boundary prediction head and a text description head. According to the method, positioning and description are decoupled, only pure visual input is needed, high efficiency and expansibility are considered, multi-scale and overlapping events can be accurately positioned, high-quality description is generated, and the method is suitable for scenes such as automatic explanation, video retrieval and intelligent security and protection.

Description

Technical Field

[0001] This invention pertains to algorithmic inventions within the computer vision branch of artificial intelligence. Specifically, it addresses the dense video description task aimed at video understanding, employing a method for dense event description based on dynamic temporal anchor boxes. This method, by overcoming the shortcomings of existing approaches, locates and describes complex events present in uncropped long videos. Background Technology

[0002] Video understanding tasks can be broadly categorized into three levels: the lowest level focuses on "element-level" processing, such as event recognition and classification of videos containing single events or actions, object detection or tracking in videos, and frame-level classification and keyframe extraction. A higher level focuses on "segment-level" tasks, such as abstract classification of video content, sentiment classification, video event localization and segmentation, and video segment understanding. The highest level of video understanding possesses both "element-level" and "segment-level" understanding capabilities, enabling the identification and comprehension of differences within videos, such as video question answering and dense video description tasks. This invention focuses on the dense video description task, which aims to locate different events in uncropped videos and generate corresponding semantic descriptions for each event. Dense video description algorithms have high research value and broad practical application prospects, with applications in automatic narration, human-computer interaction, video summarization, video retrieval, intelligent security, and visual impairment assistance.

[0003] Dense video description tasks comprise two subtasks: event localization and event description. They directly model the temporal correspondence between video content and natural language, making the identification and localization of all events within the video crucial. However, compared to most other video understanding tasks, dense video description tasks deal with videos that span longer time periods, encompass a greater amount of content, and exhibit more complex temporal relationships. Consequently, long videos often contain events with variable numbers, difficult boundary localization, significant variations in event duration, and temporal overlap. Incomplete event identification directly leads to the loss of video information, severely impacting model performance. Inaccurate event boundary identification causes the model to lose information within events, further hindering the generation of semantically complete descriptive information from event proposals. Significant variations in event duration require the model to model event proposals at different scales with varying weights, while event overlap or overlap necessitates improvements in the model's matching strategy. Therefore, designing deep learning models to specifically optimize for the aforementioned problems in unedited real videos and extracting semantically rich event proposal features has become crucial for solving dense video description tasks. Simultaneously, constructing an efficient video encoder to extract video information and adopting an output head oriented towards the application environment are equally important. Since dense video description tasks consist of two sub-tasks—event localization and event description—traditional methods typically construct these two sub-tasks sequentially from the encoded video. While these methods offer better interpretability and facilitate separate training of different modules, the modules in a two-stage model cannot mutually reinforce each other during training. The performance of the description task is largely limited by the localization task. Furthermore, multi-stage models heavily rely on techniques such as Non-Maximum Suppression (NMS) to remove redundant event predictions. To overcome these problems, since 2018, research on dense video description tasks has largely shifted towards single-stage models. Single-stage models input video features extracted by the backbone network into the encoder, fully modeling the temporal context information of the video to obtain refined video features. In the decoder, candidate event proposals are fully interacted with the refined video features and applied to both sub-tasks. Existing single-stage dense video description models still have some problems: 1) They are not good at localizing temporal boundaries and cannot handle videos of different scales or overlapping events effectively. 2) Candidate event proposals are used for both localization and description tasks, which may lead to suboptimal semantic information. Furthermore, initializing candidate event proposals with all zeros or a normal distribution requires a long training time to simultaneously model event localization and semantic information. 3) Attention mechanisms have quadratic computational complexity. Using global attention to model large-scale video features consumes a lot of computational resources and further prolongs the model's training time.Currently, some works use multimodal methods to incorporate audio information from videos and external corpora into the training process. Although this has improved the performance, the models are large, the inference speed is slow, and real videos often lack external annotation information, making them difficult to deploy in real-world scenarios.

[0004] This invention adheres to using purely visual information as input and, starting from the problems of single-stage models, designs a single-stage dense video description algorithm based on dynamic temporal anchor boxes (DTA-PDVC). This decouples the modeling process of candidate event proposals from the localization and description tasks, solving the complex event detection problem in real-world videos with a simple and lightweight model architecture, while also considering the model's efficiency and scalability. The effectiveness of our method is validated on large-scale datasets. Summary of the Invention

[0005] This invention provides a single-stage dense video description model based on dynamic temporal anchor boxes. It continues the encoder-decoder architecture of single-stage models. The encoder consists of multiple Vision Mamba blocks, processing primary video features through time-first and space-first scanning modes, modeling video contextual semantic information, and generating video features containing high-level semantic information. The decoder consists of multiple decoder layers, each with two attention modules: a self-attention module and a cross-attention module. The self-attention module is responsible for query updates, enabling candidate event proposals to be mutually aware and avoiding redundant predictions. The cross-attention module is responsible for extracting video features from candidate event proposals. We innovate on the attention calculation method, solving the problem of coupled construction of localization and description tasks in single-stage models, while also effectively detecting complex events in videos. The entire decoder is responsible for constructing event proposals and iterating temporal anchor boxes. Finally, the text output header, temporal localization header, and event counting header predict the model output to obtain the inference result. During model training, the Hungarian algorithm is used to select candidate event proposals that match the ground truth events, their losses are calculated, and backpropagation is used to update the model parameters. The model is continuously optimized through training. During the testing phase, a confidence threshold was designed to filter candidate event proposals for prediction by the real video generation model.

[0006] In the encoder, the Mamba-based encoder efficiently models video information while balancing the additional time consumption caused by the global attention approach in the decoder. In the decoder, the definition of dynamic temporal anchor boxes is key to solving the inherent problems of dense video description tasks. Temporal anchor boxes help event proposals perceive events in different temporal domains and match multiple events with varying spans that may overlap in uncropped videos. The location- and content-decoupled attention calculation mechanism and the dynamic update strategy of temporal anchor boxes are crucial for solving the information coupling construction problem in traditional single-stage models. Temporal anchor boxes are iteratively updated in the decoder layer and ultimately used directly for event temporal localization. The scale-modulated attention mechanism incorporates temporal scale information into the attention score, allowing candidate event proposals to focus on feature frames within the event domain with even weight distribution and smooth weight transitions at event boundaries. This gives the modeled event features the ability to perceive video context information, making the descriptions generated by the corresponding candidate event proposals more semantically fluent and coherent.

[0007] A dense video description method based on dynamic temporal anchor boxes specifically includes the following steps:

[0008] S1. Use video backbone networks such as 3D convolutional or dual-stream networks to extract primary video features. This is to extract spatial semantic information from video frames and to model local temporal dynamic information.

[0009] S2. Video consists of multiple frames of images. Compared to image data, video data contains additional temporal information, including the temporal sequence information of object dynamic changes. To model the contextual information of the video and capture the dynamic changes of objects, we map the primary video features extracted in step S1 to four different feature spaces using an MLP. The aim is to enhance the features to model the temporal dependencies in different dimensions of the video (including long-term dependencies, short-term dependencies, and causal dependencies).

[0010] S3. Arrange the different spatial features from step S2 into two orders based on spatiotemporal relationships: a temporal priority sequence and a spatial priority sequence. Input these orders into our Mamba-based encoder for bidirectional scanning modeling. This yields a refined video feature sequence with advanced semantics.

[0011] S4. Use the output of the previous encoder layer as the input of the next encoder layer, repeat step S3, and iteratively update the video features.

[0012] S5. Randomly initialize N candidate event proposals and timing anchor boxes. These serve as storage containers for different events present in the video. To match different events at the same location and handle event overlap, we introduce the concept of Pattern, which involves copying the existing candidate event proposals and timing anchor boxes P times.

[0013] S6. Input the candidate event proposals and timing anchor boxes into the decoder. An extension function expands the timing anchor boxes into a format used for attention calculation within the decoder. This format is a quadruple containing the event's start position, center position, end position, and length. This quadruple is then processed using a positional encoding function and a mapping function to obtain the positional encoding used for addition in self-attention. In the self-attention module, the content components of the query, key, and value are all projected from the same candidate event proposal, while the query and key also incorporate positional components. Perform self-attention calculation to process the candidate event proposals.

[0014] S7. In the cross-attention module, the query is provided by candidate event proposals that have undergone self-attention, while the key and value are refined video features. The query and key are concatenated to inject positional information. This concatenation-based approach allows the attention score to be composed of both content similarity and positional relevance, facilitating the explicit injection of positional information into candidate event proposals. To align the embedding spaces of the positional components in the query and key, this paper introduces a conditional space query mechanism. This mechanism learns a mapping function to obtain a scale vector generated by projecting content information, and then performs element-wise multiplication of this vector with the positional embedding vector. Cross-attention is then calculated.

[0015] S8. Use the output of the previous decoder layer as the input of the next decoder layer, repeat step S6, and iteratively update the candidate event proposals and the timing anchor boxes corresponding to each candidate event proposal.

[0016] S9. The temporal anchor boxes in the last decoder layer are used as predictions of event locations. The corresponding candidate event proposal features are input into the prediction of the event counter and event description header generation model. During model training, the Hungarian algorithm is used to select candidate event proposals that match the ground truth events. Their losses are calculated and backpropagated to update the model parameters. The model is continuously optimized through training. In the testing phase, a confidence threshold is designed to filter the predictions of candidate event proposals against the real video generation model.

[0017] The video backbone network in step S1 can be C3D, I3D, TSN, TSP, or CLIP video backbone networks to extract primary video features.

[0018] The two sequence scanning methods in step S3 can be chosen at will, not both.

[0019] In step S5, the initialization method for the temporal anchor boxes can be a random initialization method, a uniform initialization method in the event dimension, or the results of other video action localization models can be used to initialize the temporal anchor boxes of candidate event proposals.

[0020] In step S9, the text description header can be either an LSTM text output header or a DSA-LSTM text description header. The event localization header and the temporal anchor box boundary prediction header between the decoder layers are the same.

[0021] Beneficial effects

[0022] This invention proposes a dense video description method based on dynamic temporal anchor boxes. It can efficiently construct high-level features of videos, locate complex events in long videos, and generate high-quality text descriptions. Furthermore, it solves the problem of coupled construction of localization and description tasks in single-stage model candidate event proposals. The model's performance is validated on large-scale datasets, leading among all pure visual models and comparable to multimodal models. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method of the present invention;

[0024] Figure 2 This is a schematic diagram of the method model of the present invention;

[0025] Figure 3 This is a schematic diagram of the encoder scanning mode of the present invention;

[0026] Figure 4 This is a structural diagram of the decoder of the present invention;

[0027] Figure 5 This is a diagram illustrating the effects of the present invention; Detailed Implementation

[0028] The embodiments of the present invention will be further described below with reference to the accompanying drawings, but the implementation of the present invention is not limited thereto.

[0029] The flowchart of this invention is as follows Figure 1 As shown, the following is a further explanation.

[0030] like Figure 2 As shown, a method for describing dense events based on dynamic temporal anchor boxes specifically includes:

[0031] As shown in the image, the video shows a man standing on a snow-covered mountain preparing to ski from 0 to 14 seconds; from 15 to 55 seconds, the man skis down the slope; and from 0 to 38 seconds, there is intermittent narration. The input data is a sequence of video frames. , This represents the total number of frames in the video.

[0032] Step S1 includes the following steps:

[0033] S11 extracts primary video features from video frames, using a pre-trained C3D backbone network as an example. The video's 16 frames are grouped as input to extract action features. This video is divided into... Group, output visual feature frame sequence .

[0034] S12 interpolates and scales the feature frames in the temporal dimension to make the video frame sequence a fixed length. This yields a video frame sequence of uniform length over time.

[0035] Step S2 includes the following steps:

[0036] The primary features of the video are mapped to different feature spaces using a feedforward neural network. .in For the first The target feature dimensions of each branch, each It consists of fully connected layers and activation functions, and only adjusts the channel dimension of the features to ensure the integrity of the temporal dimension.

[0037] Step S3 includes the following steps:

[0038] like Figure 3 The diagram shows the construction of time-first and space-first traversal sequences, respectively. These sequences are then input into the encoder layer for bidirectional scanning. Note that we perform bidirectional scanning at two levels: bidirectional scanning at the Vision Mamba block level and bidirectional scanning within a Vision Mamba block. The intra-block scanning is implemented using Vim Block V2. Block-level scanning refers to matching the results of inputting the forward-order feature frame sequence with the reverse-order feature frame sequence into the encoder, then weighting and integrating the results to obtain the final encoder output.

[0039] Step S5 includes the following steps:

[0040] We preset the number of candidate event anchor boxes N to be 10 and P=2. We initialized the temporal anchor boxes using the output of the BMN model (a simple video action localization network) and initialized the candidate event proposals using a random initialization method. We also duplicated these 10 candidate event proposals and temporal anchor box pairs twice.

[0041] Step S6 includes the following steps:

[0042] Decoder model structure as follows Figure 4 As shown. Given a timing anchor box. and These are the corresponding content query and location query, respectively. This represents the dimension of the decoder embedding vector and the location query. Location Query The generation method is as follows: . The function expands the timing anchor box to The format in which and Let these represent the start time and end time of the event, respectively, and satisfy the following condition: Its specific transformation form can be defined as: . This represents a positional encoding operation, which generates a sinusoidal embedding vector based on the corresponding temporal position; this paper focuses on... The function is overloaded to ensure its expression is unambiguous. The calculation process of this positional encoding... . The function can map a single floating-point timing point to... A dimensional vector, that is, one that satisfies the mapping relationship. ; This represents a vector concatenation function. The module is responsible for dimensional vector mapping to 3D space, corresponding mapping relationship is .Should The module consists of two linear layers with ReLU activation functions, and the feature dimensionality reduction operation is performed in the first linear layer; it is worth noting that... The parameters are shared across all decoder layers. In the self-attention module, the content components of the query, key, and value are all projected from the same candidate event proposal, while the query and key also incorporate a positional component, which is calculated as follows: .

[0043] Step S7 includes the following steps:

[0044] In the S71 cross-attention module, this paper concatenates positional and content information to construct the cross-attention query and key. To align the embedding space of the positional components in the query and key, this paper introduces a conditional space query mechanism by learning a mapping function. This yields a scale vector generated from the content information projection, and this vector is then multiplied element-wise with the position embedding vector. The computational form of cross-attention is defined as follows: ; .in Representative moment Corresponding video refinement features, symbols This represents element-wise multiplication. This is another position encoding function used in the encoder, which maps a single floating-point time point to... A dimensional vector, that is, one that satisfies the mapping relationship. : , Then responsible for dimensional vector mapping to 3D space, corresponding mapping relationship is This paper employs an indirect video feature location encoding method in the decoder, aiming to maintain the consistency of location encoding in the model and thus reduce the learning difficulty of the model.

[0045] To incorporate scale information into the prior location information generated by the temporal anchor boxes, S72 introduces a learnable factor during the attention calculation process to modulate the location attention map (before softmax activation). The corresponding score is calculated as follows: .in, The function of this item is to scale the attention value; Represents timing anchor frame The corresponding event length, The reference length for content prediction serves as a learnable parameter of the model, influencing the calculation results. Its calculation method is defined as follows: The scale-modulated positional attention mechanism can help the model extract event features at different scales, while making the trend of attention score changes smoother.

[0046] Step S8 is as follows:

[0047] At the end of each decoder layer, a bias prediction head (implemented by an MLP) is set to predict the relative displacement of temporal anchor boxes; therefore, as the model training process progresses, the positions of the temporal anchor boxes are dynamically updated. The parameters of this bias prediction head can be shared across all layers of the decoder or set separately for different layers. The anchor boxes updated by the bias prediction head in the last layer are directly used as the event boundaries predicted by the model.

[0048] Step S9 is as follows:

[0049] During model training, the Hungarian algorithm is used to select candidate event proposals that match the ground truth events. The cost function of the Hungarian algorithm is... .in, This represents the generalized intersection-over-union (COUNO) loss between predicted and actual time series segments. This represents the focus loss between the predicted classification score and the true label. Based on the event pairs obtained from the matching, we calculate the ensemble prediction loss, which is a weighted sum of the gIOU loss, classification loss, event counting loss, and description generation loss: .in, It is the cross-entropy loss between the predicted word probability and the actual word (with the description text length as the normalization factor). This is the cross-entropy loss between the predicted event quantity distribution and the actual event quantity. Their loss is calculated and backpropagated to update the model parameters, iteratively optimizing the model through training. During the testing phase, a confidence threshold is designed to filter candidate event proposals for prediction by the real video generation model. The overall confidence score for each candidate event proposal is calculated as follows: ,in This represents the probability of generating the word. The average word confidence score is used to represent the average word confidence score. We found that the average word confidence score cannot effectively measure the quality of sentence-level descriptions because the description generation head tends to generate higher confidence scores for short sentences. Therefore, this paper introduces a modulation factor. To correct the impact of description length on confidence level, This is a balancing factor used to adjust the weight ratio between location and description confidence levels. Taking the testing process as an example, the final model output is as follows: Figure 2 As shown on the right.

[0050] In summary, by utilizing the aforementioned technical solutions, this invention proposes a method for describing dense events using dynamic temporal anchor boxes. Through a series of operations including feature extraction from videos, event proposal modeling, and textual description, all events can be described in textual form. This method can identify as many events as possible occurring in the video and sequentially provide highly accurate textual descriptions without requiring manual editing. This allows viewers to clearly understand all the events that occurred, overcoming the shortcomings of existing dense event description technologies.

Claims

1. A single-stage dense video description method based on dynamic temporal anchor frames, characterized in that, The method includes the following steps: S1. Constructing an enhanced primary feature sequence of video with strong local information correlation: Use a feature extraction network to extract primary visual features of the input video containing spatial distribution information and local motion information; introduce a multilayer perceptron (MLP) to construct a feature enhancement module, which adaptively projects the primary visual features to multiple high-dimensional feature spaces, so that the feature encoder can fully understand the different parts of the video and model long-term dependencies, short-term local correlations and causal logical relationships between events. S2. Constructing a refined video feature sequence with both robustness and expressiveness: To adapt to the processing characteristics of the Mamba architecture for input sequences, the enhanced video features are reconstructed into structured and ordered feature sequences according to the rules of "time-first" (suitable for videos with strong temporal dynamics) or "space-first" (suitable for videos with complex spatial scenes), ensuring the structured representation of the feature sequences and the effectiveness of subsequent modeling; a stacked Mamba encoder architecture is designed, and the reconstructed feature sequences are input into the encoder to perform multi-level bidirectional scanning iterative modeling. By utilizing the efficient long sequence modeling capability and gating mechanism of the Mamba architecture, long-range spatiotemporal correlation information in the feature sequences is adaptively captured, and finally, a refined video feature sequence with rich semantics and clear temporal structure is output; S3. Construct event proposals and model events in different time domains in the video: Based on the statistical patterns of the temporal distribution of video events, initialize N candidate event proposals and dynamic temporal anchor boxes covering different time lengths and starting positions. For complex scenarios such as multiple overlapping and nested events in the video, generate a high-density candidate set through P replication expansions to ensure complete coverage and accurate positioning of overlapping events. After inputting the candidate event set into the stacked decoder, the positional similarity is explicitly injected into the attention calculation results through temporal anchor boxes, guiding different candidate event proposals to focus on different time domains of the video. S4. Construct a dynamic temporal anchor box update mechanism and generate the final event prediction: At the end of the decoder layer, the bias prediction network predicts the deviation value of the temporal anchor box relative to the real event boundary, and adds it to the temporal anchor box of the current decoder layer as the input of the temporal anchor box of the corresponding candidate event proposal in the next decoder layer. The temporal anchor box is continuously updated between decoder layers to form a dynamic temporal anchor box that adapts to the localization requirements of event boundaries of different durations. Subsequently, the dynamic temporal anchor box output by the stacked decoder is used as the event temporal localization result. Simultaneously, the optimized candidate event proposal features are input into the customized event counter and the text description head of the DSA-LSTM architecture. The former predicts the number of key events in the video based on the multilayer perceptron to provide quantity constraints, and the latter combines event features and video semantic information to generate logical event description text, thus completing the final prediction of event information.

2. The single-stage dense video description method based on dynamic temporal anchor frames according to claim 1, characterized in that, The video primary feature extraction backbone network in step S1 can be a C3D, I3D, TSN, TSP, or CLIP network.

3. The single-stage dense video description method based on dynamic temporal anchor frames according to claim 1, characterized in that, In step S2, the primary video features are mapped to four high-dimensional feature spaces using MLP; and the bidirectional scanning in step S2 includes bidirectional scanning at the Vision Mamba block level and bidirectional scanning within a Vision Mamba block. The block-level bidirectional scanning is achieved by weighted integration of the results after the forward and reverse sequence of feature frames are input into the encoder.

4. The single-stage dense video description method based on dynamic temporal anchor frames according to claim 1, characterized in that, The temporal anchor box initialization method in step S3 is random initialization, uniform initialization of event dimensions, or initialization based on the results of other video action localization models. The candidate event proposal adopts the random initialization method.

5. The single-stage dense video description method based on dynamic temporal anchor frames according to claim 1, characterized in that, In step S3, the temporal anchor box is expanded into a quadruple containing the event start position, center position, end position and event length by the expansion function, and a self-attention position code is generated by the position encoding function and the mapping function. In the self-attention module, the content components of the query, key, and value are obtained by projecting the same candidate event proposal, and the query and key are additionally incorporated into the position component by addition.

6. The single-stage dense video description method based on dynamic temporal anchor frames according to claim 1, characterized in that, In step S3, the query of the cross-attention module is provided by the candidate event proposals output by the self-attention module, with the key and value being refined video features; The query and key are concatenated to inject location information, and a conditional space query mechanism is introduced. The scale vector is obtained by learning the mapping function, and element-wise multiplication is performed with the location embedding vector to align the embedding space.

7. The single-stage dense video description method based on dynamic temporal anchor frames according to claim 6, characterized in that, In cross-attention calculation, a learnable factor is introduced to scale the positional attention map, incorporating the scale information of event length into the attention score, making the attention score change smoother.

Images