A first-view video description generation method based on a space-time grouping attention mechanism

Abstract

This invention provides a first-view video description generation method based on a spatiotemporal grouping attention mechanism. The method extracts the feature spectrum of the input first-view video as regional features; a positional encoding module outputs the positional codes corresponding to the regional features; the regional features are then updated by adding their positional codes; a spatial grouping attention module merges the high and wide dimensions of the regional features into a spatial dimension, and then calculates grouping attention twice for all features in the spatial dimension to fully model the interaction relationships between all features in the spatial dimension; subsequently, the spatial and temporal dimensions are exchanged, and a temporal grouping attention module calculates grouping attention twice for all features in the temporal dimension and outputs it, then averages the results in the spatial dimension to obtain the first-view video feature code. This provides the decoder with video features containing more spatiotemporal information, adapting to the jitter and inter-frame variations present in the first-view video, thereby improving the quality of the description.

Description

Technical Field

[0001] This invention relates to video description generation technology, specifically to first-person video description generation technology based on a spatiotemporal grouping attention mechanism. Background Technology

[0002] Video description generation is a task in computer vision that involves abstracting features from a given video and converting them into natural language to structurally summarize and re-express the visual content. Simply put, given a video, the computer outputs a textual description of that video. This technology belongs to the field of video understanding and has broad application prospects. For example, with the rise of short video platforms on the internet, the number of online videos has grown rapidly, leading to a corresponding increase in demand for video understanding technology. This technology platform classifies and filters videos, helping internet users understand the content of the video based on the textual description. In video surveillance systems, the need for automatically generating textual descriptions of videos is also growing, recording and promptly reporting any abnormal situations detected through textual descriptions. In daily life, video description technology can bring convenience to visually impaired people, providing them with video description services through wearable vision-assistive devices, enabling them to understand their surroundings based on the textual and audio descriptions.

[0003] From video feature encoding to descriptive sentence generation, various effective models and methods have been designed, significantly improving model performance and effectively enhancing the quality of generated sentences. Most video description frameworks are designed as encoder-decoder structures. The encoder learns compressed video representations from multimodal features using various methods such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). The decoder generates sentences word-by-word based on the representations learned from the encoder, primarily using RNN and Transformer-based models. In video description systems, there are many variations of CNNs, RNNs, and Transformers. CNNs are used to generate feature vectors from image and video spatial data, and these vectors are then fed into an RNN / Transformer architecture through fully connected layers to generate word sequences. Most research in this area attempts to describe events using third-person video data, but these studies have demonstrated the difficulty of describing more fine-grained activities (such as cooking) because these activities require more detailed information than third-person video data. First-person view videos can generate more fine-grained information about the camera wearer's activities based on a closer perspective. However, unlike third-person videos, first-person videos suffer from problems such as motion blur, focus blur, and drastic changes between frames.

[0004] Most existing methods utilize CNNs to extract global features from videos. While these features contain relatively rich semantic information, they lose spatial information. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a more reliable video feature encoding method to improve the accuracy of description, addressing the difficulty of drastic changes between first-view video frames and the problem of spatial information loss in the global features used by current mainstream video description generation methods.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is a first-person video description generation method based on a spatiotemporal grouping attention mechanism, comprising the following steps:

[0007] The input first-view video is uniformly downsampled to obtain keyframes, and each keyframe is scaled down to a set size and then input into a pre-trained ResNet model and a position encoding module.

[0008] The ResNet model outputs the feature spectrum of its last convolutional layer to form the region features of the first-view video; the position encoding module outputs the position encoding corresponding to the region features; then the region features are updated by adding the result of their position encoding, and the updated region features are input to the spatial grouping attention module.

[0009] The spatial grouping attention module receives the input regional features, merges the high-dimensional and wide-dimensional features of the regional features into a spatial dimension, and then calculates grouping attention twice for all features in the spatial dimension, thereby completing the information interaction of regional features in space; the spatial grouping attention module outputs the regional features that have completed the spatial information interaction.

[0010] The spatial and temporal dimensions of the region features output by the spatial grouping attention module are swapped, and the region features with the swapped spatial and temporal dimensions are output to the temporal grouping attention module.

[0011] The temporal grouping attention module receives the input regional features, calculates grouping attention twice for all features in the time dimension, thereby completing the information interaction of regional features in time; the temporal grouping attention module outputs the regional features that have completed the information interaction in time.

[0012] The encoding of the first-view video features is obtained by averaging the regional features output by the temporal grouping attention module in the spatial dimension.

[0013] The encoder inputs the first-person view video features to the decoder to generate a sequence of words about the video content, thereby generating a video description sentence.

[0014] This invention aims to utilize CNNs to extract regional features while preserving spatial information. By employing semantic grouping and attention mechanisms, it reduces computational load while simultaneously aggregating and interacting with features, providing the decoder with video features containing more spatiotemporal information, thereby improving the quality of descriptive sentences.

[0015] The beneficial effect of this invention is that it achieves information interaction and fusion of features by grouping and aggregating regional features in both temporal and spatial dimensions. This grouping attention method overcomes the limitations of locality in convolution operations and the number of features that can be aggregated, allowing features with similar semantics to be grouped together. This approach can effectively address the issues of shakiness and large inter-frame variations present in first-person perspective videos, thereby improving the accuracy of video description and reducing computational complexity. Attached Figure Description

[0016] Figure 1 : Model diagram of the present invention;

[0017] Figure 2 : Schematic diagram of the grouped attention module. Detailed Implementation

[0018] The implementation of the video description generation method in this example mainly consists of five modules and two data operations, such as... Figure 1 As shown, the five modules are: ResNet residual network model, position encoding module, spatial grouping attention module, temporal grouping attention module, and decoder sentence generation module. The two data operations are swapping the temporal and spatial dimensions and averaging the spatial dimensions.

[0019] The steps for generating a video description are as follows:

[0020] Step 1: For a 5-second, 120-frame first-person perspective video, uniformly downsample 12 images as keyframes, and place each keyframe into a 224 pixel array. Size: 224.

[0021] Step 2: Extract the feature spectrum of the last convolutional layer for each key using a pre-trained ResNet model. The feature spectrum size for each frame is 8. 8 2048, these feature spectra are spliced ​​together into a size of Video region features .

[0022] Step 3: Calculate using equation (6) The location code is updated using equation (7). .

[0023] Step 4: Merge The spatial dimensions of height and width transform the scale into ;

[0024] Step 5: Calculate group attention twice for the 64 spatial features in the second dimension to complete the spatial information interaction, resulting in a new value of the same size. The characteristics of the two attention groups are 8 and 4 respectively.

[0025] Step 6: Swap the spatial and temporal dimensions to transform the feature size to 64. 12 2048.

[0026] Step 7: Calculate group attention twice for the 12 features in the second dimension to complete the temporal information interaction between features, resulting in a value of... The feature is that the number of groups for attention is 4 in both cases.

[0027] Step 8: Calculate the average value across the spatial dimension to obtain the size. First-person perspective video features.

[0028] Step 9: Input the first-person view video features into the decoder to generate a word sequence about the video content, thus generating a video description sentence. The decoder can be an autoregressive model such as a Long Short-Term Memory (LSTM) artificial neural network, a GRU recurrent neural network, or a Transformer neural network based on a self-attention mechanism.

[0029] Specifically, the group attention module, such as Figure 2 As shown, it includes a k-means module, a multi-head attention module, a fully connected layer, a GELU layer, and a layer normalization module.

[0030] Given a size of N The features X of D are given by N, where N represents the number of features and D represents the dimension of each feature. The K-Means module aggregates all features into g groups, resulting in a group of size g. D's grouping center The multi-head attention module will As a query Q, As keys K and values ​​V, their multi-head attention is used to update video features. The updated video features are then nonlinearized through a fully connected layer and a GELU layer. The original features are then added to the residual structure and finally output after layer normalization.

[0031] Specifically, the positional encoding module is used to calculate the positional information of each feature. Because the multi-head attention mechanism captures the positional relationships between features, it loses the relative spatial and temporal information of the video features, which is crucial for understanding the video content. Therefore, positional encoding is needed to compensate for the relative positional relationships between features. In this invention, we use learnable parameters as the positional encoding table, for a size of... The video features are defined as follows: L represents the frame number, H represents the height of the region feature, and W represents the width of the region feature. Therefore, a video has a total of [number of frames]. There are three features, where D represents the dimension of each feature. The feature in the j-th row and k-th column of the i-th frame is a three-dimensional feature, with the three dimensions being frame length, height, and width. These three dimensions correspond to three positional encoding tables. and Location coding for:

[0032] ;

[0033] , , These represent the corresponding values ​​in the encoding table for the i-th frame, j-th row, and k-th column, respectively. and Location code at the location;

[0034] Add the location code to the region features to obtain the updated video features:

[0035] ;

[0036] Specifically, multi-head attention is used in group attention. The formula for calculation is:

[0037] ;

[0038] ;

[0039] ;

[0040] So group attention Represented as:

[0041] ;

[0042] ;

[0043] in , and These are all learnable parameters. These represent query, key, and value, respectively; Concat represents tensor concatenation; and LayerNorm represents layer normalization. Representing vectors Size, For the activation function GELU, For cluster centers, The video features are input to this group attention module.

[0044] Specifically, the k-means process used is as follows:

[0045] Step 1: Select g feature samples at equal intervals as cluster centers;

[0046] Step 2: Calculate the cosine distance between each feature sample and each cluster center;

[0047] Step 3: Assign each feature sample to the cluster center that is closest to it. ;

[0048] Step 4: Calculate the mean of the samples in each class, and use it as the new cluster centers. ;

[0049] Step 5: If the class center no longer changes or a certain number of iterations are reached, the algorithm ends; otherwise, return to step 2.

Claims

1. A first-person video description generation method based on a spatiotemporal grouping attention mechanism, characterized in that, Includes the following steps: The input first-view video is uniformly downsampled to obtain keyframes, and each keyframe is scaled down to a set size and then input into a pre-trained ResNet model and a position encoding module. ResNet outputs the feature spectrum of its last convolutional layer to form the region features of the first-view video. The location encoding module outputs the location encoding corresponding to the region feature; then the region feature is updated by adding the result of its location encoding to the region feature, and the updated region feature is input to the spatial grouping attention module; The spatial grouping attention module receives the input regional features, merges the high-dimensional and wide-dimensional features of the regional features into a spatial dimension, and then calculates grouping attention twice for all features in the spatial dimension, thereby completing the information interaction of regional features in space; the spatial grouping attention module outputs the regional features that have completed the spatial information interaction. The spatial and temporal dimensions of the region features output by the spatial grouping attention module are swapped, and the region features with the swapped spatial and temporal dimensions are output to the temporal grouping attention module. The temporal grouping attention module receives the input regional features, calculates grouping attention twice for all features in the time dimension, thereby completing the information interaction of regional features in time; the temporal grouping attention module outputs the regional features that have completed the information interaction in time. The encoding of the first-view video features is obtained by averaging the regional features output by the temporal grouping attention module in the spatial dimension. The encoder of first-person view video features is input to the decoder to generate a sequence of words about the video content, thereby generating a video description sentence; The spatial grouping attention module and the temporal grouping attention module have the same structure, both including a k-means module, a multi-head attention module, a fully connected layer, a GELU activation function layer, and a layer normalization module; For an input size of N The input video features X are given by D, where N represents the number of features and D represents the dimension of each feature. The k-means module aggregates the input video features X into g groups according to a pre-set number of groups g, resulting in a dataset of size g. D's grouping center The multi-head attention module will input features As a query Q, The video features are updated by calculating their multi-head attention as keys K and values ​​V. The updated video features are then nonlinearized through a fully connected layer and a GELU activation function layer. The nonlinearized video features are then added to the original input video features X using a residual structure. Finally, the added video features are processed by a layer normalization module before being output.

2. The method as described in claim 1, characterized in that, The location encoding module is used to calculate the location information of each feature in the regional features of the first-view video, using learnable parameters as the location encoding table, for a size of The region features are defined as follows: L represents the frame number, H represents the height of the region feature, W represents the width of the region feature, and D represents the dimension of each feature. The region features in the i-th frame, j-th row, k-th column of the first-person view video are considered as follows: Corresponding to three position coding tables and Location coding for: ； in, , , These represent the corresponding values ​​in the encoding table for the i-th frame, j-th row, and k-th column, respectively. and Location code at the location; The specific implementation of updating region features by adding their location encoding results is as follows: 。 3. The method as described in claim 1, characterized in that, The decoder is a Long Short-Term Memory (LSTM) artificial neural network, a Recurrent Neural Network (GRU), or a Transformer neural network based on a self-attention mechanism.

Images