A dense video subtitle generation method based on global video features

Abstract

The present invention discloses a method for dense video subtitle generation based on global video features. The present invention encodes the global features of a whole video by an adaptive clustering method, thereby guiding event location and subtitle generation in an end-to-end manner, omitting the step of using a priori thresholds to divide event proposals in the previous model, thereby greatly reducing the computational complexity while ensuring the accuracy of subtitle generation. When processing long sequence features, the present invention can adaptively cluster similar feature queries to reduce redundancy and save memory. At the same time, as a fast approximation of the complete self-attention mechanism in the traditional Transformer, this method also excels in encoding accuracy.

Description

Technical Field

[0001] The invention belongs to the field of video feature extraction, and in particular relates to a method for generating dense video subtitles based on global video features. Background Art

[0002] With the rapid development of multimedia platforms, more and more people are accustomed to getting information from videos. On average, tens of millions of videos are uploaded to the Internet every day, and reviewing these videos consumes a lot of time. Therefore, the work of automatically generating descriptive subtitles for videos has become very valuable, which can not only greatly reduce the time of video review, but also help visually impaired patients obtain information with the help of voice reading software. However, a video usually contains multiple interrelated events. Generating only a single short description for the video will lose a lot of information, so the task of dense video subtitle generation came into being. In general, this task aims to locate each event contained in the video and generate corresponding subtitles for it. The whole process mainly includes two subtasks, namely event location and subtitle generation. A competitive dense video subtitle generation model should have good performance on both subtasks.

[0003] Existing work usually adopts a two-stage tandem scheme of "event localization-subtitle generation", which usually requires the introduction of a priori thresholds to screen numerous event proposals, which inevitably increases the amount of computation and memory consumption; in addition, the quality of the subtitles generated by this scheme is heavily dependent on the accuracy of event localization, resulting in very unstable model performance. Summary of the invention

[0004] The purpose of the present invention is to overcome the above-mentioned shortcomings and provide a method for generating dense video subtitles based on global video features, which can improve the computational efficiency as much as possible while ensuring the accuracy of the generated video subtitles.

[0005] In order to achieve the above object, the present invention comprises the following steps:

[0006] Use pre-trained action recognition networks to extract primary encoding features of videos;

[0007] The primary coding features are processed, the visual center and weight are determined, and then several layers are stacked to obtain the global feature coding;

[0008] Taking global feature encoding as a guide, a parallel multi-head decoder is used to predict the number of events, locate events, and generate subtitles, and finally generate video subtitles.

[0009] Extracting primary coding features of the video through C3D model, two-stream network structure or time-sensitive video encoder.

[0010] The specific method to obtain the global feature encoding is as follows:

[0011] The video features in the primary coding features are processed using a local sensitive hashing method to determine the visual center of the video features;

[0012] Query each set of video features, obtain the top k video feature keys with the highest attention and determine the weights;

[0013] Repeat the above steps to assign weights to all encoded video features to obtain global feature encoding.

[0014] The specific method of using the local sensitive hashing method to process the video features in the primary coding features is as follows:

[0015] Calculate the hash value of the video feature query in each primary encoding feature;

[0016] Use Euclidean Locality Sensitive Hashing as the hash function:

[0017]

[0018] in, yes The weight, are hyperparameters, a and b is a random variable that satisfies ,and , , applied to H LSHs, and the hash value of each video component is obtained:

[0019]

[0020] Where B is a constant;

[0021] set up is the center of the video feature query with the same hash value, Represents the video feature query component Q i Which group do you belong to? j Group Vision Center Q c(j) It is expressed as the following formula:

[0022]

[0023] The corresponding cluster attention matrix Obtained as follows:

[0024]

[0025] The specific method of obtaining the top k video feature keys with the highest attention and determining the weights is as follows:

[0026] set up is a set of indicator vectors, where P ji = 1 if and only if the i The video feature key is j The group's attention is in the front k One of the keys, otherwise 0;

[0027] In this way, the j The most popular in the group k The keys of the digits are separated from other keys and the following attention coefficients are calculated for them:

[0028]

[0029] The improved attention matrix is ​​expressed as follows:

[0030]

[0031] in i It means the j The video feature group contains i Video feature query;

[0032] New video feature values It can be divided into two parts:

[0033]

[0034] in,

[0035]

[0036]

[0037] in Depend on Broadcast received.

[0038] The specific method to obtain the global feature encoding is as follows:

[0039] go through J The layer stack contains an adaptive clustering attention encoding layer to extract the final global feature encoding of the video ,The obtained global video features not only contain the background information of the entire video ,but should also be object-sensitive and event-sensitive.

[0040] The event number prediction uses the event number prediction head. The specific method is as follows:

[0041] Event query features Compress into a vector, and then use the fully connected layer to predict a fixed-length vector , where each element represents the probability that the number of events is that value;

[0042] In the inference phase, the confidence level is selected first. L inf The proposal is used as the final event segmentation result, and the confidence score of each event generated subtitle can be obtained by the following formula:

[0043]

[0044] in, Indicated in i The event generates t The probability of a target word, γ is the modulation factor, μ It is a balancing factor used to weaken the impact of subtitle length on the confidence score.

[0045] Event positioning uses the event proposal positioning header. The specific method is as follows:

[0046] The event proposal localization head is designed to generate box predictions and perform binary classification for each event-level feature. The box prediction is used to predict the starting position of each event-level feature; the binary classification is used to predict the foreground confidence of each event query. Both of these predictions are based on the application of multi-layer perceptrons to event-level features. The above obtained:

[0047]

[0048]

[0049] After the event proposal positioning head, a set of tuples is obtained To represent the detected events, Indicates the start and end time of the event proposal. Indicates event characteristics The positioning reliability.

[0050] Subtitle generation uses the subtitle generation header, the specific method is as follows:

[0051] Low-level features for initially encoded video F , in order to extract the features of different scales, F Input ResNet and extract the output of the network C3 to C5 stage, recorded as ,in M = 4, no. M The video feature map of each scale is obtained by applying a convolution with a kernel of 3×3 and a stride of 2 to the output of the C5 stage;

[0052] Semantic Query hi,t-1 and event-level features Splice as query, h i,t-1 Represents the hidden features in the caption generation LSTM, generating primary features for each scale D The basic process is as follows:

[0053]

[0054]

[0055] in g i Directly act on the query by linear mapping and sigmoid activation function It is obtained from the above, which represents the coordinates of the normalized reference point, that is, g i ∈[0,1] 2 , Map the normalized reference point to the feature map of the corresponding scale, represents the sampling offset, Representative for the i semantic event query, sampling point d exist m Attention on scale.

[0056] Compared with the prior art, the present invention uses an adaptive clustering method to encode the global features of the entire video when only the entire video is input, and then guides event location and subtitle generation in an end-to-end manner, omitting the step of using a priori thresholds to divide event proposals in the previous model, thereby greatly reducing the computational complexity while ensuring the accuracy of subtitle generation. When processing long sequence features, the present invention can adaptively cluster similar feature queries to reduce redundancy and save memory. At the same time, as a fast approximation of the complete self-attention mechanism in the traditional Transformer, this method also excels in encoding accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of global video feature extraction in the present invention;

[0058] Figure 2 The overall flow chart of the subtitle generation header in the present invention;

[0059] Figure 3 It is a model flow chart of the present invention. DETAILED DESCRIPTION

[0060] The present invention will be further described below in conjunction with the accompanying drawings.

[0061] See also Figure 1, Global video feature extraction based on adaptive clustering:

[0062] This paper uses an adaptive clustering method to encode the global features of the entire video by inputting only the entire video, and then guides event localization and subtitle generation in an end-to-end manner, omitting the step of using prior thresholds to divide event proposals in previous models, thereby greatly reducing the computational complexity while ensuring the accuracy of subtitle generation.

[0063] First, use the pre-trained action recognition network (C3D, TSN, TSP) to extract the primary coding features of the video This primary coding feature is then processed to obtain a representative global video feature.

[0064] Interpolation is used to rescale the temporal dimension of the video features to N , thereby obtaining the primary features of the video The primary features are then flattened and embedded into positional encoding as a

[0065] Input to the Transformer model:

[0066] (1)

[0067] (2)

[0068] in W Q , W K , W V are learnable parameters that map the primary features of the video to the encoder input space. For convenience, let the resulting video feature query be Q , Video feature key , video feature value . ACTAtt(·) in (2) is the core of this paper: the encoder based on adaptive clustering attention. The main idea is to first divide the video feature query into U video feature groups, where Then only the attention of these groups is calculated, and the same attention weight is assigned to the video feature queries of the same group. Furthermore, in order to make some video feature keys that should have received higher attention receive high attention, it is also necessary to k In general, in order to obtain the global video feature S according to formula (2), the specific steps are as follows (the following is a specific explanation of formula (2), where FW Q represent Q , FW K represent K , FW V represent V ):

[0069] In order to determine the video feature group, this paper first uses the locality sensitive hashing (LSH) method to process the video feature query. Considering that LSH is a powerful tool for solving the nearest neighbor search problem: if the adjacent vectors can obtain the same hash value with a high probability, that is, fall into the same hash bucket, while the hash values ​​of distant vectors are different, the hash scheme is called locally sensitive. Therefore, by controlling the relevant parameters and number of rounds of the hash function, this paper can sort all vectors with a distance less than 100 according to the hash value. ε The video feature query is greater than p Specifically, we first calculate the hash value of each video feature query, and this paper selects Euclidean local sensitive hashing as the hash function:

[0070] (3)

[0071] in Q j yes Q The weight, r is a hyperparameter, a and b is a random variable that satisfies a and , , Application H LSH, the hash value of each video component is as follows:

[0072] (4)

[0073] in B is a constant. From formula (3), we can see that the hash function can actually be regarded as a set of random normal vectors a and offset b The hyperplane, hyperparameters r Controls the spacing of the hyperplanes, r The larger the value, the larger the spacing. And formula (3) shows that H A hash function divides the space into several cells, and vectors falling into the same cell will get the same hash value.

[0074] In order to obtain the visual center, is the center of the video feature query with the same hash value,I i Is the category index, indicating the video feature query component Q i Therefore, j Group Vision Center Q c(j) It can be expressed as follows:

[0075] (5)

[0076] Based on this, the corresponding cluster attention matrix and video feature values It can be obtained as follows:

[0077] (6)

[0078] Further, for each set of video feature queries, find the top k The weight of the remaining part is still calculated according to the above clustering method.

[0079] Specifically, if is a set of indicator vectors, where P ji = 1 if and only if the i The video feature key is j The group's attention is in the front k One of the keys in the first column, otherwise it is 0. j The most popular in the group k The key is separated from other keys and the following attention coefficient is calculated for them (the purpose of this is to ensure that the previous k The sum of the values ​​corresponding to the video feature key and the other video feature keys is 1):

[0080] (8)

[0081] Formula (8) is actually j Among the video feature groups, the most popular k The total probability of the video feature keys. Then the attention matrix improved in the above way can be expressed as:

[0082] (9)

[0083] in i It means the j The video feature group contains i In other words, according to formula (6), we select each video feature group with the highest attention level in the front.k The video feature key is scaled with the attention coefficient and is compared with each video feature query in the video feature group. Q i Do the dot product and then use softmax Recalculate and obtain new weights. For video feature keys that do not belong to the above, the weights are still calculated only at the visual center of each video feature group according to formula (6). In general, the new video feature value It can be divided into two parts:

[0084] (10)

[0085] in,

[0086] (11)

[0087] (12)

[0088] in Depend on Broadcast received.

[0089] Based on this, the model performs the above operations on the encoded video features in each encoding layer to obtain a representative global video feature.

[0090] Assume that J By stacking the above encoding layers including adaptive clustering attention, the model can extract the final global feature encoding of the video The obtained global video features not only contain the background information of the entire video, but also should be object-sensitive and event-sensitive.

[0091] See also Figure 3 The parallel multi-head decoder uses the global video feature S obtained above as a guide to simultaneously perform three downstream subtasks: event number prediction, event location, and subtitle generation, thereby promoting the interaction of subtasks and ultimately generating accurate dense video subtitle descriptions for the video. Specifically, the decoder in this paper does not process the input event query sequentially and recursively, but processes it in parallel. L A learnable event query that aims to directly L The event-level features are queried from the global video features with rich clustering information conditioned on a learnable embedding. If the initialized learnable event query is represented as , then the calculation process of attention in each decoding layer can be expressed as:

[0092] (13)

[0093] (14)

[0094] Among them, Att(·) is the self-attention mechanism. It should be noted that the keys and values ​​in formula (14) are derived from the global video features output by the encoder. S , the output of the self-attention in the decoding layer is used as the query. This paper calls this attention mechanism the cross attention mechanism. For simplicity, only the attention part in one decoding layer is described here. J Output of the Transformer decoder obtained by iterative refinement of the layer decoding layer That is, event-level features guided by global video features.

[0095] Event Proposal Location Header

[0096] The event proposal localization head aims to generate box predictions and perform binary classification for each event-level feature. Specifically, the box prediction is used to predict the starting position of each event-level feature; the binary classification is used to predict the foreground confidence of each event query. Both of these predictions are based on the multi-layer perceptron applied to event-level features. The above obtained:

[0097] (15)

[0098] (16)

[0099] In this way, after the event proposal positioning head, the model can obtain a set of tuples To represent the detected events, Indicates the start and end time of the event proposal. Indicates event characteristics The positioning reliability.

[0100] See also Figure 2 , subtitle generation header

[0101] Low-level features for initially encoded video F , in order to extract the features of different scales, F Input ResNet and extract the output of the network C3 to C5 stage, recorded as ,in M = 4, no. M The video feature maps of different scales are obtained by applying a convolution with a kernel of 3×3 and a stride of 2 to the output of the C5 stage. Although the resolutions of feature maps of different scales are different, their number of channels can be converted to 256 through a 1×1 convolution operation. In this way, the multi-scale video primary features used to supplement the visual information of the subtitle generation head are obtained, denoted as X = It should be noted that the global video features processed by the adaptive clustering encoder are not selected as visual information supplements because richer visual information can be sampled from the features that have not been clustered.

[0102] Further, when generating i Event query t When there are words, we first need to generate features for each scale. D The basic process is as follows:

[0103] (17)

[0104] in,

[0105] (18)

[0106] Specifically, the semantic query h i,t-1 and event-level features Splice as a query, here h i,t-1 represents the hidden features in the subtitle generation LSTM. Then, according to formula (17), the primary features of each scale are generated D reference points, of which g i Directly act on the query by linear mapping and sigmoid activation function It is obtained from the above, which represents the coordinates of the normalized reference point, that is, g i ∈[0,1] 2 . Directly map the normalized reference point to the feature map of the corresponding scale, represents the sampling offset, Representative for the i semantic event query, sampling point d exist m Attention at different scales, both of which are obtained by applying a linear projection to the semantic event query.

[0107] In this way, based on semantic and event queries, we can obtain complementary video visual information sampled at different scales of primary video features. Next, based on the idea of ​​soft attention, these visual information sampling points can be weighted according to semantics and event queries:

[0108] (19)

[0109] (20)

[0110] inw a , W x , W ha are all learnable parameters. represents each component of the complementary visual information. Therefore, the weighted visual context feature can be expressed as:

[0111] (twenty one)

[0112] Next, this paper will supplement the contextual visual features z it , event-level features And the previous word embedding w i,t-1 Input into LSTM to get the time step t The hidden state of h it , and further use the fully connected layer to w it Then for the i Event Query e i , you can get the corresponding subtitles O i = ,in B i Indicates the length of the subtitle.

[0113] Event count prediction header

[0114] Considering the number of event queries L is a manually set hyperparameter. In the actual dense video caption generation task, it is not necessary to L Event queries generate subtitles. Too many events will lead to a lot of repetition in the generated subtitles, which is not readable; too few events will lead to the loss of important information. Therefore, this section designs an event number prediction head to predict an appropriate number of events for each video.

[0115] Specifically, the event count prediction head contains a maximum pooling head and a fully connected layer with softmax activation. First, the event query features Compress into a vector, and then use the fully connected layer to predict a fixed-length vector , where each element represents the probability of the number of events being that value. In the inference stage, select the confidence level in the front L inf The proposal is used as the final event segmentation result, and the confidence score of each event generated subtitle can be obtained by the following formula:

[0116] (twenty two)

[0117] in, Indicated in i The event generates t The probability of a target word, γ is the modulation factor, μ It is a balancing factor used to weaken the impact of subtitle length on the confidence score.

[0118] In order to demonstrate the superiority of the model of the present invention, this section compares the model of the present invention with some classic dense video subtitle generation models in terms of event localization accuracy, subtitle generation quality, and reasoning time.

[0119] Evaluation of event location accuracy: In earlier work, event proposals were generated in advance through a pre-trained model, not an end-to-end structure. Therefore, in this section, the present invention is compared with the classic two-stage model, that is, the pipeline structure model of "location-selection-description", to reflect the advantages of the subtask parallel strategy. Specifically:

[0120] 1.MT: As the baseline model of this article, this model is also based on the Transformer encoder-decoder structure. First, the video is encoded into an appropriate representation. The event proposal decoder decodes the encoding with different anchor points to generate event proposals. The subtitle decoder generates subtitles based on the output of the proposal decoder.

[0121] 2.MFT: Design event proposal generation and subtitle generation as a recurrent network so that the previous subtitle description can guide the current event proposal segmentation;

[0122] 3.SDVC: It takes into account the temporal dependency of events and uses reinforcement learning to provide two-level rewards in terms of event and plot coherence.

[0123] Then, the accuracy, recall rate and F1 results of the model event location are shown in the following table:

[0124] Table 1 Event localization results on ActivityNet validation set (using C3D encoding)

[0125]

[0126] The present invention is different from the existing methods that use the serial scheme of "event localization-subtitle generation". The present invention abandons this method and directly uses a parallel approach to output event proposal location, which can greatly reduce the setting of the prior threshold in the scheme and is more effective than the serial scheme. As can be seen from the figure, the event proposal location results of the present invention far exceed MFT and MT, and are comparable to SDVC, a non-end-to-end model with more parameters. In particular, when the IOU threshold is higher, the two versions of the model of the present invention show more competitive results and more accurate positioning performance. In addition, when the model uses a more sophisticated subtitle generation head, the average accuracy of event proposal location is also improved, which also shows that the subtasks corresponding to the two parallel heads influence and promote each other.

[0127] In order to analyze the role of global video features in guiding decoding operations, this experiment selects video clips and normalizes them into frame-level importance outputs through a fully connected layer. Different events and different frames in an event have different weights, which provides important guidance for the downstream decoder to generate event proposals and generate subtitles.

[0128] Considering that the ActivityNet dataset contains a total of 203 action classes, and the performance of subtitles may be related to the category of the action, this paper continues to explore the performance of the GDC model on different categories. Specifically, this paper selects 10 representative categories, evaluates the METEOR index of GDC-generated subtitles on them, and compares the performance on real event proposals and predicted event proposals. The experimental results are shown in Table 2:

[0129] Table 2 METEOR indicators for generating subtitles on different categories of actions

[0130]

[0131] As can be seen from the above table, the performance of the present invention on video data of different action types is different. Specifically, for some large-scale actions or activities with certain specificity in action backgrounds, such as driving bumper cars, playing squash, skating, etc., the subtitle results generated by the present invention are already very competitive. However, for some smaller and more detailed activities, such as karate, Zumba and other videos, the performance of the present invention is slightly reduced. However, based on the results in Table 2, it can be seen that GDC will be expected to achieve greater performance improvements by capturing more fine-grained features when performing preliminary encoding of videos through more advanced video feature extraction models. It should be noted that although for the method of extracting global video features using adaptive clustering, modifying some hyperparameters will lead to an improvement in subtitle accuracy, it will also increase memory consumption, which will be further analyzed in the following ablation experiment.

[0132] In summary, based on the results of quantitative and qualitative analysis, GDC has achieved very competitive performance on both datasets. It has significantly improved the existing dense video subtitle generation model in both event localization and subtitle generation tasks, which further proves the effectiveness of GDC.

[0133] The present invention is compared with the existing dense video subtitle generation method on the ActivityNet dataset and the YouCookII dataset. The experimental results show that the model of the present invention has taken the lead in terms of event positioning, subtitle generation and reasoning efficiency. The present invention ensures the accuracy of its generated subtitles in both predicted proposals and real proposals. These performances further confirm the effectiveness of the model of the present invention. In addition, the behavior of global video features in downstream tasks is further analyzed through ablation experiments. The visualized video feature group further demonstrates the guiding role of global video features and increases the interpretability of the model. Finally, the influence of the encoder's hyperparameters H and r and the number of event queries L in the decoder on the experimental results is analyzed. Finally, the selection of hyperparameters well balances the performance of the model in various indicators.

Claims

1. A method for generating dense video subtitles based on global video features, characterized in that: The following steps are involved: Use pre-trained action recognition networks to extract primary encoding features of videos; The primary coding features are processed, the visual center and weight are determined, and then several layers are stacked to obtain the global feature coding; Taking global feature encoding as a guide, a parallel multi-head decoder is used to predict the number of events, locate events, and generate subtitles, and finally generate video subtitles; the number of events is predicted using the event number prediction head, and the specific method is as follows: Event query features Compress into a vector, and then use the fully connected layer to predict a fixed-length vector , where each element represents the probability that the number of events is a feature of the event query; In the inference phase, the confidence level is selected first. L inf The proposal is used as the final event segmentation result, and the confidence score of each event generated subtitle can be obtained by the following formula: in, Indicated in i The event generates t The probability of a target word, γ is the modulation factor, μ It is a balancing factor used to reduce the impact of subtitle length on the confidence score; Event positioning uses the event proposal positioning header. The specific method is as follows: The event proposal localization head is designed to generate box predictions and perform binary classification for each event-level feature. The box prediction is used to predict the starting position of each event-level feature; the binary classification is used to predict the foreground confidence of each event query. Both of these predictions are based on the application of multi-layer perceptrons to event-level features. The above obtained: After the event proposal positioning head, a set of tuples is obtained To represent the detected events, Indicates the start and end time of the event proposal. Indicates event characteristics The reliability of the location; Subtitle generation uses the subtitle generation header, the specific method is as follows: Low-level features for initially encoded video F , in order to extract the features of different scales, F Input ResNet and extract the output of the network C3 to C5 stage, recorded as ,in M = 4, no. M The video feature map of each scale is obtained by applying a convolution with a kernel of 3×3 and a stride of 2 to the output of the C5 stage; Semantic Query h i,t-1 and event-level features Splice as query, h i,t-1 Represents the hidden features in the caption generation LSTM, generating primary features for each scale D The process is as follows: in g i Directly act on the query by linear mapping and sigmoid activation function It is obtained from the above, which represents the coordinates of the normalized reference point, that is, g i ∈[0,1] 2 , Map the normalized reference point to the feature map of the corresponding scale, represents the sampling offset, Representative for the i semantic event query, sampling point d exist m Attention on scale.

2. The method for generating dense video subtitles based on global video features according to claim 1, characterized in that: Extracting primary coding features of the video through C3D model, two-stream network structure or time-sensitive video encoder.

3. The method for generating dense video subtitles based on global video features according to claim 1, characterized in that: The specific method to obtain the global feature encoding is as follows: The video features in the primary coding features are processed using a local sensitive hashing method to determine the visual center of the video features; Query each set of video features, obtain the top k video feature keys with the highest attention and determine the weights; Repeat the above steps to assign weights to all encoded video features to obtain global feature encoding.

4. The method for generating dense video subtitles based on global video features according to claim 3, characterized in that: The specific method of using the local sensitive hashing method to process the video features in the primary coding features is as follows: Calculate the hash value of the video feature query in each primary encoding feature; Use Euclidean Locality Sensitive Hashing as the hash function: in, yes The weight, are hyperparameters, a and b is a random variable that satisfies ,and , , applied to H LSHs, and the hash value of each video component is obtained: Where, B is a constant; set up is the center of the video feature query with the same hash value, Represents the video feature query component Q i Which group do you belong to? j Group Vision Center Q c(j) It is expressed as the following formula: The corresponding cluster attention matrix Obtained as follows: 。 5. The method for generating dense video subtitles based on global video features according to claim 3, characterized in that: The specific method of obtaining the top k video feature keys with the highest attention and determining the weights is as follows: set up is a set of indicator vectors, where P ji = 1 if and only if the i The video feature key is j The group's attention is in the front k One of the keys, otherwise 0; In this way, the j The most popular in the group k The keys of the digits are separated from other keys and the following attention coefficients are calculated for them: The improved attention matrix is ​​expressed as follows: in i It means the j The video feature group contains i Video feature query; New video feature values It can be divided into two parts: in, in Depend on Broadcast received.

6. The method for generating dense video subtitles based on global video features according to claim 3, characterized in that: The specific method to obtain the global feature encoding is as follows: go through J The layer stack contains an adaptive clustering attention encoding layer to extract the final global feature encoding of the video ,The obtained global video features not only contain the background information of the entire video ,but should also be object-sensitive and event-sensitive.

Images