Video dense description generation method based on deep local self-attention network

Abstract

The present invention proposes a method for generating dense description of video based on deep local self-attention network. The steps of the present invention are as follows: 1. Data set preprocessing and data set partitioning, using a trained behavior recognition network to extract features from the video; 2. Constructing a sequence of structured dense description of the video and constructing text features of the sequence; 3. Constructing a deep local self-attention network for sequence-to-sequence modeling; 4. Training the network parameters in step (3) through the back propagation algorithm until the entire network model converges. The present invention also jointly models the two subtasks of event location and event description in dense video description into a sequence generation task, closely linking the two independent subtasks together to promote each other. Compared with most two-stage method models, there are not too many hyperparameters that need to be manually adjusted, the training model is more convenient, and the performance is further improved.

Description

Technical Field

[0001] This paper proposes a method for generating dense video descriptions based on a deep local self-attention network. The core of this method is to propose a deep local self-attention network that models both the temporal information of an event and the corresponding event description in text form. It also jointly models the two subtasks of event location and event description in dense video description as a sequence generation task, while simultaneously predicting both the occurrence event and the corresponding event description. The superiority of this model is verified in the deep learning task of dense video description generation. This method is the first to propose an end-to-end method for generating dense video descriptions. By jointly modeling two independent subtasks, the two independent subtask modules no longer constrain each other, but instead promote each other, thereby achieving better results. Background Art

[0002] Dense video captioning is a learning task involving computer vision and natural language processing. It takes an unedited video as input and outputs the start time of multiple events in the input video and the corresponding event descriptions through the model.

[0003] In general, dense video captioning can be decomposed into two subtasks: event localization and video segment description. Previous approaches have solved this task by decomposing it and using a two-stage "locate event occurrence time - describe video segment" approach. First, a time module locates the event occurrence, and then an event description module selects the video segment corresponding to the localized time to generate a description. This approach typically involves two independent modules with different underlying model architectures for event localization and event description, presenting captions for dense events based on the predicted event span.

[0004] In practical applications, online videos have become an important source of knowledge and skills for people. To help users locate interesting information, search engines and video platforms often display anchors at key moments, often accompanied by descriptions of the clip's content. This is a direct application of the dense video captioning task, and therefore solutions to improve this task are highly relevant to any video platform.

[0005] In summary, dense video description generation is a topic worthy of in-depth research. This patent intends to explore several key points in this task, solve the difficulties and key points of the current methods, and form a complete dense video generation system.

[0006] Previous approaches all first predict a set of event times with precise boundaries. By extracting fine-grained semantic clues and visual context from the predicted event times, a caption generator ultimately decodes a detailed sentence description. While this approach is simple and intuitive, it suffers from the following two issues:

[0007] 1. Taking title processing as a downstream task, the performance of the solution is highly dependent on the quality of the predicted event occurrence time, which limits the mutual promotion of the two subtasks.

[0008] 2. The prediction of event occurrence time relies on artificial post-processing algorithms (e.g., non-maximum suppression). These post-processing methods introduce additional hyperparameters that are highly dependent on thresholds, hindering the progress of fully end-to-end caption generation. Summary of the Invention

[0009] This paper provides a method for generating dense video descriptions based on a deep local self-attention network. This method primarily models both the time information of an event and the corresponding event description in text form. This method jointly models the two subtasks of event location and event description in dense video description into a single sequence generation task. This closely links the two tasks and allows for simultaneous prediction, enabling the two independent subtasks to mutually reinforce each other.

[0010] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0011] Step (1): Dataset preprocessing and data set division, using the trained behavior recognition network to extract features from the video.

[0012] Step (2): Construct a sequence of structured dense descriptions of the video and construct text features of the sequence.

[0013] Step (3): Construct a deep local self-attention network for sequence-to-sequence modeling.

[0014] Step (4): Train the network parameters in step (3) through the back propagation algorithm until the entire network model converges.

[0015] The process of data preprocessing and video feature extraction in step (1) is as follows:

[0016] Perform word segmentation on the description in the dataset, and obtain the number of words corresponding to the dataset as N v The dataset is divided into 70% for training and the remaining 30% for testing.

[0017] For the number of frames transmitted per second f vThe input video with a total length of t seconds is used to extract features using the trained behavior recognition network. The video is input into the behavior recognition network, and the network samples the video every a frames, and the output of the last layer of the network is extracted as the video feature. Splice all features into complete video features in d v Represents the set of real numbers dimension.

[0018] F=[f 1 ,f 2 ,…,…,f L ] (Formula 1)

[0019] Step (2) constructs a sequence of densely structured descriptions of the video and constructs text features of the sequence. The specific implementation process is as follows:

[0020] For video event m start time Seconds and end time seconds and the total video length is t seconds, the number of words w is b m , event description c m .

[0021] c m =[w1,w2,…,…,w b m] (Formula 2)

[0022] First, the start time of the video event m Seconds and end time Seconds are converted into the proportion of the video The precision is two decimal places, and the formula is as follows:

[0023]

[0024]

[0025] Start time End Time proportion With event description c m Merge to get C m , the formula is as follows:

[0026]

[0027] The sequence representation modeling methods of video structured dense description are respectively based on start time Merge in order of size, by end time Merge in order of size, by fragment duration Large and small merge, random merge, merge all events into a model and add the start symbol "[seq]" and the end symbol "[ / seq]" at the beginning and end of the constructed sequence respectively to obtain the sequence C of the video structured dense description, the number of words is The formula is as follows:

[0028] C=[[seq],C 1 ,C 2 ,…,…,C m ,[ / seq]] (Formula 6)

[0029] According to the pre-trained word vector model, each word in the sequence C of the video structured dense description is converted into a word vector containing semantic information Concatenate the word vectors of all words into text features

[0030]

[0031] Step (3) constructs a deep local self-attention network for sequence-to-sequence modeling, as follows:

[0032] The video feature F is input into the encoder, and the video feature F is obtained after passing through the encoder to obtain the video feature X′∈ During the nth decoding, the encoded video feature X′ is combined with the text features of the first n words At the same time, the decoder is input for decoding, and finally input into the fully connected function to output the answer probability Select the last column The word with the highest probability is used as the predicted answer for the n+1th word, and the next word is predicted iteratively until the end symbol "[ / seq]" is predicted. The input word for the first decoding is a start symbol "[seq]".

[0033] 3-1. Constructing an adaptive local mask matrix

[0034] In order to accurately locate the boundaries of events, especially those with shorter duration, and because different videos have different lengths, an adaptive local mask matrix is ​​constructed. To block global perception.

[0035] 3-1-1. First construct the initial matrix The formula is as follows:

[0036]

[0037] Where i and j are the horizontal and vertical coordinates of the matrix respectively.

[0038] 3-1-2. Video feature F is converted into a feature vector through full connection mapping The formula is as follows:

[0039] X = Linear(F) (Formula 9)

[0040] 3-1-3. The feature vector X passes through the pooling layer, ReLU activation layer, and linear layer in sequence to obtain the local factor α. The formula is as follows:

[0041] α=Linear(ReLU(Pooling(X))) (Formula 10)

[0042] 3-1-4. The local factor α is multiplied by the initial matrix M0, and the adaptive local mask matrix M is obtained by the function F(x) and the natural logarithm. l , the formula is as follows:

[0043]

[0044] M l =ln(F(αM0)) (Formula 12)

[0045] 3-2. Building an adaptive local self-attention encoder

[0046] The feature vector X and the adaptive local mask matrix M l As the input of SA (self-attention), the output The formula is as follows:

[0047] G=SA(X,M l ) (Formula 13)

[0048] The specific formula of SA is as follows:

[0049] G′=LN(X+MHA(X,X,X,M l )) (Formula 14)

[0050] G = LN(G′ + FFN(G′)) (Formula 15)

[0051] The input of MHA (multi-head attention) is X and the adaptive local mask matrix M l , the output is the feature vector G′ containing the information of the previous and next frames of the video. The input of FFN (feed-forward network) is the feature vector G′. The formula is as follows:

[0052] Q = Linear(X) (Formula 16)

[0053] K = Linear(X) (Formula 17)

[0054] V = Linear(X) (Formula 18)

[0055]

[0056]

[0057] in They are respectively obtained by mapping X through the fully connected layer.

[0058] 3-3. Deep Stacked Encoder

[0059] Take the feature vector X as the input encoder, stack the encoder in depth, and stack the output feature X of the Nth layer encoder (N) As the input of the N+1th layer encoder, it is iterated back and forth to finally obtain the encoded video feature X′.

[0060] 3-4. Build a decoder

[0061] Text features L of the first n words n After fully connected mapping and position encoding, it is converted into a feature vector The formula is as follows:

[0062]

[0063] where pos∈[1,2,…,…,t],i∈[0,1,2,…,…,d m / 2]

[0064] Y=Linear(L)+PE n (Formula 22)

[0065] The feature vector Y and the mask matrix M t ∈{-∞,0} n×n As the input of SA, the output The formula is as follows:

[0066]

[0067] Y′=SA(Y,M t ) (Formula 24)

[0068] Where i and j are the horizontal and vertical coordinates of the matrix respectively.

[0069] Take X′, Y′ as the input of GA (guided-attention) and output The formula is as follows:

[0070] Z=GA(X′,Y′) (Formula 25)

[0071] The specific formula of GA is as follows:

[0072] Z′=LN(Y′+MHA(X′,X′,Y′)) (Formula 26)

[0073] Z=LN(Z′+FFN(Z′)) (Formula 27)

[0074] 3-5. Deep Stacked Decoder

[0075] Take the video feature X′ and the text feature Y as the decoder input, stack the decoder in depth, and combine the video feature X′ and the output feature I of the Nth layer decoder. (N) As the input of the N+1 layer decoder, it is iterated back and forth to finally obtain the multimodal video text features

[0076] 3-6. Calculating the loss function

[0077] The multimodal video text feature I output from 3-5 is passed through the fully connected layer to obtain the predicted sequence The formula is as follows:

[0078] W n =Linear(I) (Formula 28)

[0079] The predicted sequence W n The corresponding target sequence L n Input them together into the maximum likelihood estimation loss function to calculate the gap between the predicted sequence and the true sequence. The specific formula is as follows:

[0080]

[0081] Step (4) trains the network parameters in step (3) through the back propagation algorithm until the entire network model converges, as follows:

[0082] According to the loss value generated by the maximum likelihood estimation loss function in step (3), the model parameters of the neural network in step (3) are gradient-backed using the back-propagation algorithm and continuously optimized until the entire network model converges.

[0083] The beneficial effects of the present invention are as follows:

[0084] The algorithm proposed in this invention changes the model of the previous two-stage method and proposes a sequence representation modeling method for video structured dense description. The two subtasks of event location and event description in dense video description are jointly modeled into a sequence generation task, and the two tasks are closely linked together for end-to-end generation modeling, which strengthens the connection between the two subtasks and enables them to promote each other. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 : Schematic diagram of joint modeling of event occurrence time and event description.

[0086] Figure 2 : Adaptive local self-attention encoder.

[0087] Figure 3 : An end-to-end dense video description network architecture. DETAILED DESCRIPTION

[0088] The detailed parameters of the present invention are further described below.

[0089] like Figure 1 、 2 As shown in ,3, the present invention provides a video dense description generation method based on deep local self-attention network.

[0090] The data preprocessing and video feature extraction in step (1) are specifically implemented as follows:

[0091] Dataset preprocessing: The descriptions in the dataset were segmented to obtain a vocabulary of 10,172 words. The dataset was divided into a training set consisting of 10,009 videos with corresponding event times and descriptions, and a test set consisting of 4,925 videos with corresponding event times and descriptions.

[0092] For the number of frames transmitted per second f v The input video with a total length of t seconds is used to extract features using the trained behavior recognition network. The video is input into the behavior recognition network, and the network samples the video every 64 frames. The output of the last layer of the network is extracted as the video feature. Splice all features into complete video features in d v Represents the set of real numbers dimension.

[0093] Step (2) constructs a sequence of structured dense descriptions of the video and constructs text features of the sequence. The specific implementation is as follows:

[0094] like Figure 1 As shown, for the video event m start time Seconds and end time seconds and the total video length is t seconds, the number of words is b m Event Descriptionc m First, the start time of the video event m Seconds and end time Seconds are converted into the proportion of the video The precision is two decimal places. Combine the start time and event description to get C m The sequence representation modeling methods for video structured dense description are respectively based on the start time Merge in order of size, by end time Merge in order of size, by fragment duration Large and small merges and random merges are used to merge all events into a model and add the start symbol "[seq]" and the end symbol "[ / seq]" at the beginning to obtain a sequence C of structured dense description of the video with b words.

[0095] The sequence C of densely structured description of the video is converted into a word vector containing semantic information according to the pre-trained word vector model Concatenate the word vectors of all words into text features

[0096] Step (3): Constructing a deep local self-attention network for sequence-to-sequence modeling is specifically implemented as follows:

[0097] like Figure 3 As shown, the video feature F is input into the encoder, and the video feature is obtained after the encoder is encoded. During the nth decoding, the encoded video feature X′ is combined with the text features of the first n words At the same time, it is input into the decoder for decoding, and finally input into the full connection function and activation function to output the answer probability Select the last column The word with the highest probability is used as the predicted answer for the n+1th word, and the next word is predicted iteratively until the end symbol "[ / seq]" is predicted. The input word for the first decoding is a start symbol "[seq]".

[0098] 3-1. Constructing an adaptive local mask matrix

[0099] like Figure 2 As shown in Figure 2, in order to accurately locate the boundaries of events, especially the boundaries of events with a shorter duration, and because the lengths of different videos are different, an adaptive local mask matrix is ​​constructed. To shield the global perception. First, construct the initial matrix The video feature F is converted into a feature vector through full connection mapping The feature vector X passes through the pooling layer, ReLU activation layer, and linear layer in turn to obtain the local factor α. The local factor α is multiplied by the initial matrix, and the mask matrix M is calculated by the function F(x) and the natural logarithm. l .

[0100] 3-2. Constructing an Adaptive Local Self-Attention Encoder

[0101] like Figure 2 As shown, the video feature F is transformed into a feature vector through full connection mapping Combine the feature vector X and the adaptive local mask M l As the input of SA, the output

[0102] 3-3. Deep Stacked Encoder

[0103] Taking the feature vector X as input, the encoder is stacked in depth, and the output features of the Nth layer encoder are used as the input of the N+1th layer encoder. The process is repeated repeatedly to finally obtain the encoded video features X′.

[0104] 3-4. Build a decoder

[0105] Text features L of the first n words n After fully connected mapping and position encoding, it is converted into a feature vector The feature vector Y and the mask matrix M t ∈{-∞,0} n×n As the input of SA, the output Take X′, Y′ as the input of GA and output

[0106] 3-5. Deep Stacked Decoder

[0107] Taking the video feature X′ and the text feature Y as input, the deep stack decoder is used to combine the video feature X′ and the output feature I of the Nth layer decoder. (N) As the input of the N+1 layer decoder, it is iterated back and forth to finally obtain the multimodal video text features

[0108] 3-6. Loss Function

[0109] The multimodal video text feature I output from 3-5 is passed through the fully connected layer to obtain the predicted sequence Together with the corresponding target sequence, it is input into the maximum likelihood estimation loss function and the loss value is calculated.

[0110] Step (4): Train the network parameters in step (3) through the back propagation algorithm until the entire network model converges.

[0111] According to the loss value generated by the maximum likelihood estimation loss function in step (3), the model parameters of the neural network in step (3) are gradient-backed using the back-propagation algorithm and continuously optimized until the entire network model converges.

[0112] Predict the dense video description sequence on the converged model, and map the multimodal video text features I output from steps 3-5 to Select the last column Sort and select the category with the highest score as the predicted answer for the n+1th word, and iteratively predict the next word until the end symbol "[ / seq]" is predicted, generating the final answer, that is, the dense video description sequence.

Claims

1. Video dense description generation method based on deep local self-attention network, characterized by The steps include: Step (1): Dataset preprocessing and data set partitioning, using the trained behavior recognition network to extract video features; Step (2): Construct a sequence of structured dense descriptions of the video and construct text features of the sequence; Step (3): Construct a deep local self-attention network for sequence-to-sequence modeling; Step (4): Train the network parameters in step (3) through the back propagation algorithm until the entire network model converges; The deep local self-attention network for sequence-to-sequence modeling described in step (3) is specifically implemented as follows: 3-1. Constructing an adaptive local mask matrix To block global perception; 3-1-1. First construct the initial matrix The formula is as follows: Where i and j are the horizontal and vertical coordinates of the matrix respectively. f v is the number of frames transmitted per second, t is the total duration, and a is the number of frames per second between video sampling intervals; 3-1-2. Video feature F is converted into a feature vector through full connection mapping The formula is as follows: X = Linear(F) (Formula 9) 3-1-3. The feature vector X passes through the pooling layer, ReLU activation layer, and linear layer in sequence to obtain the local factor α. The formula is as follows: α=Linear(ReLU(Pooling(X))) (Formula 10) 3-1-4. The local factor α is multiplied by the initial matrix M0, and the adaptive local mask matrix M is obtained by the function F(x) and the natural logarithm. l , the formula is as follows: M l =ln(F(αM0)) (Formula 12) 3-2. Constructing an Adaptive Local Self-Attention Encoder The feature vector X and the adaptive local mask matrix M l As the input of SA, the output The formula is as follows: G=SA(X,M l ) (Formula 13) The specific formula of SA is as follows: G'=LN(X+MHA(X,X,X,M l )) (Formula 14) G = LN(G' + FFN(G')) (Formula 15) The input of MHA is X and the adaptive local mask matrix M l , the output is the feature vector G' containing the information of the previous and next frames of the video. The input of FFN is the feature vector G'. The formula is as follows: Q = Linear(X) (Formula 16) K = Linear(X) (Formula 17) V = Linear(X) (Formula 18) in They are respectively obtained by mapping X through the fully connected layer; 3-3. Deep Stacked Encoder Take the feature vector X as the input encoder, stack the encoder in depth, and stack the output feature X of the Nth layer encoder (N) As the input of the N+1th layer encoder, it is iterated back and forth to finally obtain the encoded video feature X'; 3-4. Build a decoder Text features L of the first n words n After fully connected mapping and position encoding, it is converted into a feature vector The formula is as follows: where pos∈[1,2,…,…,t],i∈[0,1,2,…,…,d m / 2] Y=Linear(L)+PE n (Formula 22) The feature vector Y and the mask matrix M t ∈{-∞,0} n×n As the input of SA, the output The formula is as follows: Y' = SA(Y, M t ) (Equation 24) Where i and j are the horizontal and vertical coordinates of the matrix respectively; Take X', Y' as the input of GA and output The formula is as follows: Z=GA(X',Y') (Formula 25) The specific formula of GA is as follows: Z'=LN(Y'+MHA(X',X',Y')) (Formula 26) Z = LN(Z' + FFN(Z')) (Formula 27) 3-5. Deep Stacked Decoder Take the video feature X' and the text feature Y as the decoder input, stack the decoder in depth, and combine the video feature X' and the output feature I of the Nth layer decoder (N) As the input of the N+1 layer decoder, it is iterated back and forth to finally obtain the multimodal video text features 3-6. Calculating the loss function The multimodal video text feature I output from 3-5 is passed through the fully connected layer to obtain the predicted sequence The formula is as follows: W n = Linear(I) (Formula 28) The predicted sequence W n The corresponding target sequence L n Input them together into the maximum likelihood estimation loss function to calculate the gap between the predicted sequence and the true sequence. The specific formula is as follows: Where b is the number of words contained in the sequence C of the structured dense description of the video.

2. The method for generating dense description of video based on deep local self-attention network according to claim 1 is characterized in that The dataset preprocessing and dataset partitioning described in step (1) use the trained behavior recognition network to extract features from the video, as follows: Perform word segmentation on the description in the dataset, and obtain the number of words corresponding to the dataset as N v The data set is divided into 70% for training and the remaining 30% for testing; For the number of frames transmitted per second f v The input video with a total length of t seconds is used to extract features using the trained behavior recognition network; the video is input into the behavior recognition network, the network samples the video every a frames, and the output of the last layer of the network is extracted as the feature of the video Splice all features into complete video features in d v Represents the set of real numbers Dimensions; F=[f 1 ,f 2 ,…,…,f L ] (Formula 1).

3. The method for generating dense video description based on deep local self-attention network according to claim 1 or 2, characterized in that Step (2) constructs a sequence of densely structured descriptions of the video and constructs text features of the sequence. The specific implementation process is as follows: For video event m start time Seconds and end time seconds and the total video length is t seconds, the number of words w is b m , event description c m ; First, the start time of the video event m Seconds and end time Seconds are converted into the proportion of the video The precision is two decimal places. The formula is as follows: Start time End Time proportion With event description c m Merge to get C m , the formula is as follows: The sequence representation modeling methods of video structured dense description are respectively based on start time Merge in order of size, by end time Merge in order of size, by fragment duration Large and small merge, random merge, merge all events into a model and add the start symbol "[seq]" and the end symbol "[ / seq]" at the beginning and end of the constructed sequence respectively to obtain the sequence C of the video structured dense description, the number of words is The formula is as follows: C=[[seq],C 1 ,C 2 ,…,…,C m ,[ / seq]] (Formula 6) According to the pre-trained word vector model, each word in the sequence C of the video structured dense description is converted into a word vector containing semantic information Concatenate the word vectors of all words into text features L b =[l 1 ,l 2 ,…,…,l b ] (Formula 7).

4. The method for generating dense description of video based on deep local self-attention network according to claim 3 is characterized in that The deep local self-attention network for sequence-to-sequence modeling described in step (3) is constructed as follows: The video feature F is input into the encoder, and the video feature F is obtained after passing through the encoder. During the nth decoding, the encoded video feature X' is combined with the text features of the first n words At the same time, the decoder is input for decoding, and finally input into the fully connected function to output the answer probability Select the last column The word with the highest probability is used as the predicted answer for the n+1th word, and the next word is predicted iteratively until the end symbol "[ / seq]" is predicted; The first decoded input word is a start symbol "[seq]".

Images