Convolutional attention weakly supervised temporal action localization method with re-attention mechanism

Abstract

The application discloses a convolution attention weakly supervised temporal action localization method with a re-attention mechanism, which utilizes convolution attention to mine global and local information and refines through re-attention for weakly supervised temporal action detection.The steps are as follows:1, data preprocessing, extracting the initial spatio-temporal features of video data.2, convolution attention model with re-attention mechanism, building a multi-stage model with multi-head self-attention mechanism, and introducing three auxiliary branches to learn actions, backgrounds and action contexts respectively.3, model training, training neural network parameters by using a back propagation algorithm.4, generating positioning detection results.The application proposes a multi-stage method for mining local and global temporal information in the field of weakly supervised temporal action localization by using a convolution attention mechanism and refining attention weights through re-attention, and simultaneously mines additional information through auxiliary branches, so that the performance is improved compared with the original method.

Description

TECHNICAL FIELD

[0001] The application relates to a ConvTransformer network model with a re-attention mechanism for weakly supervised temporal action localization (TAL). Wherein, ConvTransformer refers to the cascade of convolution and Transformer to overcome the respective problems of convolution and Transformer, the application designs a convolution attention structure, and refines the weight by re-attention on the importance weight after attention, and simultaneously proposes an action suppression branch to directly learn the background by suppressing the action. The application is in the form of end-to-end. Compared with the original method, the performance is greatly improved. BACKGROUND

[0002] In recent years, temporal action localization has attracted more and more attention. With the popularity of 5G, the upload and download speeds of the network have been greatly improved. At the same time, due to the continuous increase in the number of netizens and people's increasing habit of sharing their lives and experiences through the network, the content on the network has increased dramatically. Compared with text, the video form can contain more rich information in a short time. Many people have got used to brushing short videos as an entertainment way in their leisure time. These factors have led to a sharp increase in the number of videos on the network. Whether it is security monitoring, video review or unmanned driving, people need an effective method to obtain the desired information from the video. For video content analysis, the most important task is behavior analysis.

[0003] Early behavior analysis is mainly a behavior recognition task. First, the video is manually cropped to only retain the part containing the action, and then the algorithm is used to classify the action. However, there is no cropped video in the real environment, and it is time-consuming and laborious to manually crop, which is not realistic. Therefore, people began to explore behavior recognition on uncropped videos, that is, temporal behavior detection, which also realizes the video cropping work through the algorithm, but this improves the difficulty of the task and it is not easy to accurately locate the start and end time of the action.

[0004] With the continuous development of deep learning, people began to explore the method of combining deep learning and temporal behavior detection. Compared with traditional machine learning algorithms, deep learning has many advantages: 1. According to the adaptive learning of the network layer, the features are better represented 2. Through the learning of big data, it has good generalization ability 3. The features can be expressed layer by layer from low-level raw data to high-level semantic information. Combining deep learning and temporal behavior detection has developed into the current mainstream temporal action detection method.

[0005] Temporal Action Localization (TAL) is very important for video understanding, analysis, summarization and relocalization problems. Its purpose is to identify action instances (e.g., running, swimming or jumping, etc.) in a video file and localize the temporal boundaries (e.g., start and end times) of each action in a long untrimmed video. Therefore, TAL includes two processes: action recognition and action localization. The purpose of action recognition is to detect and classify actions in untrimmed videos. Since most videos in the real world are untrimmed (such as surveillance videos, online video platforms, etc.), and contain multiple action instances, in untrimmed videos, complete action names and annotations of the temporal boundaries of each action are needed. These annotations are used as labels to learn the recognition model through the training phase of full supervision learning (i.e., ground-truth labels). The annotation task is performed manually, which causes some difficulties: (a) expensive cost. (b) Time-consuming. (c) Difficult to assign accurate temporal boundaries to each action. (d) Subjective, depending on the person himself. These difficulties illustrate the importance of reducing annotation information by reducing the annotation information used in the training phase. For this purpose, some studies use semi-supervised learning, which uses some fully annotated samples to predict adaptive thresholds in the training and testing phases.

[0006] Some studies use single-frame supervised learning, which is only a single frame with an action, and the duration of the action is annotated within the time window of the action, while others use segment-level labels, where the annotation is the start and end timestamps of the segment with the action. In short, if the annotation is greatly reduced, the above difficulties can be avoided, and the long untrimmed video can be annotated with the least annotation using only the list of action labels (video-level labels) without using any other information. This type of setting is called weakly supervised learning.

[0007] Weakly supervised temporal action detection refers to using only video-level annotation information (such as video-level labels) for model training in the temporal action detection task, without using timestamp-level annotation information (such as the start and end timestamps of the action). This method can greatly reduce the cost of data annotation, because video-level annotation is simpler, less time-consuming, and can be quickly obtained through crowdsourcing, etc., while timestamp-level annotation requires professional personnel to spend a lot of time and effort to complete.

[0008] However, the current mainstream weakly supervised temporal action detection method also has some problems. Compared with the processing of static information of images, the behavior detection in videos often needs to process the content that needs to fuse the information in time sequence, and the positioning task required for detecting behavior is completed at the time sequence node. Therefore, effective time sequence information processing is often a difficulty that needs to be solved in the behavior detection task. The current weakly supervised method mostly follows the classification and positioning paradigm, first classifies each frame to obtain the time sequence class activation sequence TCAS, and then the scores of each frame are summarized to guide the frame-level prediction with a video-level classification loss. However, the classifier usually focuses on the extremely fragmented part with high discriminative fragments. On the other hand, the detector tries to find the entire action instance without losing any relevant fragments. This is the inherent contradiction between the classifier and the detector, which affects the model performance. The weakly supervised method works as a detector, but is usually based on a classifier. Moreover, the temporal convolution can only directly obtain the local information of the video due to the limitation of the convolution kernel size, and cannot learn the direct relationship between the fragments far away in time in the video. Therefore, this method cannot simulate the long-range interaction between the fragments, which may be important for action detection.

[0009] With the success of the Transformer model in natural language processing and image and other fields, many methods use multi-head self-attention (MHSA) to model long-range relationships in videos for action detection. This attention mechanism can establish direct one-to-one global relationships between each time slice (i.e., time token). Detect highly correlated and composite actions. However, existing methods rely on modeling this long-range relationship on the input frames themselves, and a time token only contains very few frames, which is usually too short compared to the duration of an action instance. In addition, in this setting, transformers need to explicitly learn the strong relationship between adjacent tokens due to temporal consistency, which is natural for temporal convolution (i.e., local inductive bias). And this kind of method is easily affected by irrelevant backgrounds, because there is no frame-level label under weak supervision, and the Transformer will calculate the attention weight between all frames, in this case, it is easy to set a larger weight for the background frame because of the classification error, and thus be disturbed by the background. Therefore, the pure transformer architecture may not be sufficient to model the complex temporal dependence of action detection, and some methods integrate Transformers and convolutions to alleviate these shortcomings, but still have the problem of incomplete positioning. SUMMARY

[0010] The present application proposes a weakly supervised temporal action localization method with re-attention convolutional attention to solve the above difficult problems. 1. A convolutional attention structure is designed to simultaneously mine local and global information by integrating convolution and attention. Convolution projects features to obtain feature maps with different feature dimensions at different stages and mine local information, and attention uses multi-head self-attention to calculate global importance weights. 2. A re-attention module is added behind the attention, and the output weights of the multi-head self-attention are selected according to the cumulative importance. The high importance weights selected are used to re-attention the output features of the multi-head self-attention to obtain more refined attention weights. At the same time, different re-attention thresholds are selected, and the obtained re-attention scores are averaged to obtain attention weights with different refinement degrees, so as to balance local and global information. 3. Three auxiliary branches are used to model actions, backgrounds and action contexts respectively, and the main structures of the three branches are similar, the difference lies in that filters are added in two branches to suppress actions or backgrounds to learn the part that is not suppressed, and when calculating the loss function, different labels are used to set different learning goals for each branch.

[0011] The technical solution adopted by the present application to solve its technical problems comprises the following steps:

[0012] Step (1), data preprocessing, extracting initial spatio-temporal features of video data

[0013] Preprocessing of video data set V: for each input video v in video data set V n , first extract image frames at 30FPS, and use TVL-1 algorithm to extract video optical flow. For the extracted image and optical flow, use the I3D model pre-trained based on the Kinetics data set to extract the image features and optical flow features corresponding to the image and optical flow respectively, and then stack the two features in the feature dimension to integrate the appearance features and motion features of the input video. In this way, the spatial information can be ensured while the temporal information is taken into account, and the final initial spatio-temporal features are obtained. Since the unpruned video has different temporal lengths, scaling operation is needed. The part exceeding the length of the too long video is truncated, and the too short video is linearly interpolated, and finally the features with fixed temporal length are obtained.

[0014] Step (2), build a convolutional attention network model with re-attention mechanism

[0015] Firstly, the video data features are down-sampled by convolution, and then the importance between nodes is adaptively learned according to the attention mechanism; the multi-head form is used to calculate the attention weight of the nodes, and finally the output weight of all nodes is obtained by fusion; the output weight of the multi-head self-attention is selected according to the cumulative importance in the re-attention module, and the output features of the multi-head self-attention are re-attended using the high importance weight selected, and different scales of cumulative importance can be selected by setting different thresholds;

[0016] By multiple stages, the features are mined in different feature dimensions, the features of each stage are projected to the same dimension by convolution and spliced, and finally input into the convolution classifier to obtain a class activation sequence, which indicates the activation probability of each class in each frame of the video, and the action classification score is obtained by the top-k aggregation strategy; meanwhile, three auxiliary branches are added: the background, action and action context are learned by the action suppression branch, the background suppression branch and the context branch respectively, each branch contains two convolution modules for adjusting the features to adapt to subsequent training, three class activation sequences are obtained by the convolution classifier, and the respective action classification scores are obtained by the top-k aggregation strategy; the two suppression branches use a filter to suppress the action and the background at the beginning. When calculating the loss function, different labels are used to set different learning goals for each branch.

[0017] Step (3), model training

[0018] According to the classification score obtained in step (2), the cross-entropy loss is used, the action classification score is constrained by the video-level class label, the action probability entropy of the background segment is maximized by the background entropy loss, and the class activation sequence is guided to suppress the background segment at the video segment level by the guidance loss. The model parameters are trained by using the back propagation algorithm

[0019] Step (4), generating the positioning detection result

[0020] For the class activation sequence and the classification score obtained by the background suppression branch, in the C candidate classes, the class whose video-level action classification score is lower than the set threshold is discarded. Next, the class activation sequence of the class with high confidence is selected, the continuous frames whose action classification score is higher than the threshold are taken as the predicted action instance, and the start time and end time of the predicted action instance are obtained. Then, the confidence score of the predicted action instance is determined by the external-internal comparison strategy.

[0021] The technical solution adopted by the application to solve the technical problems includes the following steps:

[0022] Step (1) of the data preprocessing, the initial spatio-temporal features of the video data are extracted, and the specific steps are as follows:

[0023] For each input video v in the video dataset V n , first extract image frames at 30FPS, and at the same time extract the optical flow of the video using the TVL-1 algorithm. For the extracted images and optical flow, use the I3D model pre-trained based on the Kinetics dataset to extract the image features and optical flow features corresponding to the images and optical flow respectively, and then stack the two features in the time sequence dimension to integrate the appearance features and motion features of the input video, so that the spatial information is guaranteed while the time sequence information is taken into account, and the final initial space-time features are obtained. Since the unpruned videos have different time sequence lengths, scaling operation is needed, and the part exceeding the length is truncated for too long videos, and linear interpolation is performed for too short videos, and finally the fixed time sequence length features are obtained.

[0024] The convolution attention method with re-attention mechanism in step (2) is specifically as follows:

[0025] 2-1, Convolution attention, adjust the feature dimension through time sequence convolution, mine global and local information in different dimensions, the early feature dimension is low, then it is increased, the global and local blocks capture global context information through multi-head self-attention, and the attention is calculated by calculating the similarity of Q and K. The similarity calculated by Q and K reflects the importance of the extracted V value, that is, the weight. The special point of the self-attention mechanism in the KQV model is that K=Q=V, which respectively represent query, key and value in the attention mechanism process. In the present application, a multi-head (Multi-Head) self-attention mechanism is adopted, the purpose of which is to let multiple heads adaptively learn different attention weights, and improve the robustness of the overall structure. Specifically, for the input X, for the jth attention block in the first stage, the time sequence convolution projects the feature dimension D of the input X to D0 to obtain X0, and then D0 is continuously expanded in the subsequent stages. Taking the first stage as an example, X0 passes through the multi-head attention layer in the global relationship block, and the multi-head attention layer is composed of H attention heads. For each head i∈{1,...,H}, the input X j is projected to Q ij = W ij Q X j , K ij = W ij K X j and V ij = W ij V X j , where W ij Q , W ij K , represent the weights of linear layers, and Dh = D0 / H represents the feature dimension of each head. Therefore, the self-attention of head i is calculated as

[0026]

[0027] wherein, is the weight, W ∈ R H×T×T , The outputs of different attention heads are then mixed with an additional linear layer to obtain the output features of the global module

[0028]

[0029] wherein W j O represents the weight of the linear layer. X j is the bias term of the linear layer, and the output feature size of the multi-head attention layer is the same as the input feature size.

[0030] 2-2, Re-attention mechanism. The output features M j and the output weight W j of the multi-head attention are input into the re-attention module. The re-attention module selects the output weight of the multi-head self-attention according to the cumulative importance, and uses the selected high importance weight to re-attend the output features of the multi-head self-attention. Then, an adaptive threshold is constructed using the cumulative distribution sampling method. The specific operation is as follows: the preliminary attention result W is normalized and sorted from large to small to obtain the sorted weight S and construct the cumulative importance I, j = 0,..., T, W ∈ R H×T×T , T represents the video time length; a fixed threshold τ' is determined for the cumulative importance result, and an adaptive threshold is generated for the preliminary attention result W. The label parameter b = [I < τ'] represents a binary mask indicating whether the selected frame exists.

[0031] Then, the label parameter b is used for re-attention. First, the selection matrix B is generated according to the label parameter b for re-attention. The selection matrix B is a binary matrix, wherein each item is B u,v , which means that the v-th frame will be used for updating the u-th frame. By replacing the self-attention module in the attention block with a masked self-attention module through the selection matrix B, a masked attention is obtained, as follows:

[0032]

[0033]

[0034] Where A is multiplied by the output feature M of multi-head self-attention to get the re-attention output score, score is passed through a fully connected layer and a masked softmax layer to produce the importance weight λ. According to the importance weight λ, the weight value is retrieved from the pruned output weight W to get the re-attention vector. The re-attention vector is defined as:

[0035]

[0036] W' = W ⊙ (J T -b) + λr (6)

[0037] Where b k represents the label parameters of the k-th frame; J is a matrix where each element is equal to 1.

[0038] The re-attention vector is obtained

[0039] Here, X0 is input into two re-attention modules with different τ' to obtain re-attention vectors W'1 and W'2 of different scales, and then averaged to obtain the final re-attention score. The re-attention score is multiplied by the output M of multi-head self-attention, and then added to X0 as a residual to obtain the output of the global-local block.

[0040] The processes of other stages are similar, except that the feature dimensions of the input features are different. The information mined by all stages is input into the mixer to be mixed to form a unified video representation, and finally input into the convolutional classifier.

[0041] 2-3, auxiliary branch. For the context branch, given the input feature X, first input BaseModule, which contains two frames-specific convolutions. Specifically, first generate an embedding through a convolution, and for adjacent frames of the embedding, generate a similarity mask by calculating the cosine similarity. The similarity mask is added to the convolution calculation process to obtain the frames-specific convolution. After the input feature X passes through the BaseModule, the result is input into the convolutional classifier to obtain the class activation sequence S, where the convolutional classifier is a temporal convolution. Then, the top-k aggregation strategy is used to calculate the video-level action classification score.

[0042] The action suppression branch and the background suppression branch are similar to the context branch, but have a filter module in the front end, which suppresses the action score for the action suppression branch, thereby explicitly modeling the background, and the remaining process is similar to the context branch and the background suppression branch, except that the input feature map is different, and except for the filter, the other modules share parameters with other branches. When calculating the loss, the difference between branches is that the first C classes of the label y of the action suppression branch are set to 0, and the last class, i.e. the background class, is set to 1. The C+1 classes of the label y of the context branch are all set to 1. The first C classes of the label y of the background suppression branch are set to 1, and the last class is set to 0.

[0043] Step (3) model training, specifically as follows:

[0044] 3-1, first generate the corresponding training label according to the label instance of the input video. For a video, its corresponding label is a binary vector containing C elements, corresponding to whether each category exists, and a binary value indicating whether the background class exists. Thus, a label containing C+1 classes is obtained. Different values are set for the background class for different branches to achieve different learning goals. For the action suppression branch, the label is a vector with C 0 elements concatenated with 1, indicating that only the background class exists, and the label is calculated with the action classification score obtained by the corresponding branch to calculate a cross-entropy loss. For the context branch, the video-level classification score s v 0 v 1 v C v ] and the classification label y, the video-level classification loss L cls is calculated by cross-entropy loss.

[0045]

[0046] wherein is the action classification score after softmax normalization. The background suppression branch and the action suppression branch obtain classification losses L fg and L bg , respectively. The convolutional attention branch with re-attention mechanism, i.e. the global-local branch, obtains a classification loss L gl through a similar process.

[0047] ​​​3-2, Attention-guided loss: Although the MIL learning mechanism is introduced to construct the video-level classification loss, which makes the network classify the action instances contained in the video, this way does not optimize the segment-level action classification results, which is not conducive to the subsequent accurate action time positioning. In order to make the action context branch distinguish the action instance segments and those ambiguous action context frames at the segment level, in addition to the video-level cross-entropy classification loss L cls , an attention-guided loss is also introduced. The action instance attention sequence is set as a binary indicator for each video segment, and it is used to guide the weighted CAS to suppress the action context and background segments at the video segment level. Specifically, the attention-guided loss L is:

[0048]

[0049] where p base (t) is the predicted segment-level action probability distribution obtained by applying the softmax function to the cas base output, represents the probability that the segment s(t) belongs to the last class, that is, the background class, is the output weight of the background suppression module filter.

[0050] Background entropy loss. Although the uncertainty modeling loss encourages the background segments to generate lower Logits for all actions, due to the relativity of the softmax function, the softmax scores of certain action classes may be higher. In order to prevent the background segments from having high softmax scores for any action class, the action probability entropy of the background segment is maximized through the background entropy loss, and the background segment is encouraged to have a uniform probability distribution for the action class, and the loss is calculated as follows:

[0051]

[0052] where, is the average action probability of the c-th background segment, p c (s n,j ) is the softmax score of the c-th class of s n,j .

[0053] The overall loss includes the classification loss, the guided loss and the background entropy loss of the four branches.

[0054] L total = L cls + L fg + L bg + αL gl + βL gui + ηL be (10)

[0055] Wherein, alpha, beta, eta are adjustment coefficients.

[0056] The step (4) generates a positioning detection result, and the specific process is as follows:

[0057] For the class activation sequence and the classification score of the background suppression branch, in the C candidate categories, the category with a video-level classification score lower than the threshold is discarded. Next, the class activation sequence with a high confidence category is selected, the continuous frames with higher scores are taken as the action instance, and the start time and end time are obtained. Then, the confidence score of the predicted action instance is determined through the external-internal comparison strategy.

[0058] The present application has the following advantages:

[0059] The present application provides a weakly supervised temporal action localization method with re-attention convolutional attention. Although many current class activation sequence-based network models have also achieved good positioning effects. However, due to the tendency of the classifier to select the segment with high discriminability, and the requirement of the locator to obtain the complete action interval, the weakly supervised temporal action localization based on the classifier as the locator is inevitably affected by this contradiction. At the same time, the locality of convolution makes the positioning more fragmented, which affects the performance. Therefore, this paper uses the structure of convolutional attention to simultaneously excavate local and global information. Convolution can excavate local information while adjusting the feature dimension to reduce the amount of calculation and make the feature adapt to subsequent training. The good learning ability of attention to long time series helps the network to learn complete action instances. However, under the weakly supervised setting, the attention mechanism may introduce a lot of background interference. Therefore, the present application adds a re-attention module after the attention module. The output weight of the multi-head self-attention is selected according to the cumulative importance, and the output feature of the multi-head self-attention is re-attended using the high importance weight selected. In addition, three branches with different and complementary learning objectives are introduced to learn actions, backgrounds and action contexts respectively to help improve the learning performance. The method of the present application has achieved greater performance improvement compared with traditional temporal behavior detection methods. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 The present application has a complete flowchart. DETAILED DESCRIPTION

[0061] The detailed parameters of the present application are further described as follows. Figure 1 As shown in the figure, the weakly supervised temporal action localization method with re-attention mechanism convolutional attention of the present application has the following specific implementation steps:

[0062] Step (1), data preprocessing, extracting the initial spatio-temporal features of the video data

[0063] Feature extraction is performed on the extracted images and optical flow, an I3D model pre-trained based on the Kinetics dataset is used to extract features corresponding to the images and optical flow respectively, the image features and optical flow features are stacked in the feature dimension, and initial spatio-temporal features are obtained Since the unpruned video has different time sequence lengths N, scaling operation is needed, too long video is truncated for the part exceeding the length, too short video is linearly interpolated, and finally the feature with fixed time sequence length is obtained

[0064] Step (2), convolution attention method with re-attention

[0065] Input The convolution attention method with re-attention includes two stages, each stage contains a local-global block and a re-attention block.

[0066] 2-1, the local-global block can be divided into a local block and a global block, the local block adjusts the feature dimension through a 1D convolution, the first stage projects the original 2048-dimensional feature to 256-dimensional, and the second stage up-samples the 256-dimensional feature to 384-dimensional. For the first stage, the 256-dimensional feature is then input to the global block, the attention of the global block is calculated based on the input feature First, three linear transformation operations of the same dimension are performed. Get Query, Key and Value, the dimension is 8x750x32, 8 is the number of heads of multi-head self-attention, and 32=256 / 8. Query and Key are multiplied to get an 8x750x750 similarity matrix, and then the similarity matrix and Value are multiplied to get an 8x750x32 new feature. The new feature is reshaped to 750x256. Then pass through a linear layer without changing the dimension. The feature after the target self-attention is obtained.

[0067] 2-2, the re-attention module selects the output weight of multi-head self-attention according to the cumulative importance, and uses the high importance weight selected to re-attention the output feature of multi-head self-attention. Then use the cumulative distribution sampling method to construct an adaptive threshold, the specific operation is: normalize the preliminary attention result W and sort it from large to small to get S and construct the cumulative importance I, A fixed threshold is determined for the cumulative importance result, then an adaptive threshold is generated for W, b is obtained, b=[I<τ'] represents a binary mask indicating whether the selected frame exists.

[0068] Then use b to pay attention again, first generate selection matrix B according to b for paying attention again, replace the self-attention module in the attention block with the masked self-attention module through B to obtain the masked attention, multiply A by the output M of the multi-head self-attention to obtain the paying attention again output score, score passes through the full connection layer and the mask softmax layer to generate the importance weight λ. According to λ, the weight value is retrieved from the pruned W to obtain the paying attention again vector W'. Input X0 into two paying attention again modules with different τ' to obtain paying attention again vectors W'1 and W'2 of different scales, the first paying attention again τ'=0.6, the first paying attention again τ'=0.7, then average to obtain the final paying attention again score scores, scores are multiplied by the output M of the multi-head self-attention, and then X0 is added as a residual to obtain the output of the paying attention again block, and after normalization, the output of each stage is obtained.

[0069] The inputs of the two stages are projected to 1024 dimensions through linear layers, the output of the first stage is added to the output of the second stage after a convolution to balance the feature resolution, the spliced fusion feature and the output of the second stage are obtained to obtain 2048 features, a classifier composed of a convolution is used to obtain a class activation sequence. The class activation sequence is aggregated by top-k, and the classification score is obtained by taking the mean value, and k is 750 / 8.

[0070] 2-3, Auxiliary branch. The context branch learns both action and background simultaneously. First, the 2048-dimensional feature is input into two convolutions. The first one projects the feature into 512 dimensions, and the second one projects it into 32 dimensions to obtain the feature embedding. For each frame of the feature embedding, the cosine similarity with the adjacent frame is calculated to obtain a similarity mask. Before calculating the convolution at each frame, the feature near it can be weighted. Thus, the traditional convolution becomes a frame-specific convolution. The input 2048-dimensional feature is input into two frame-specific convolutions, and then the activation function and the Dropout operation (Dropout = 0.7) are performed to obtain the classification feature, the dimension of which remains unchanged. The classification feature is input into the classifier to obtain the class activation sequence of the context branch. The class activation sequence is aggregated by top-k, and then the mean value is taken to obtain the classification score. The value of k is 750 / 8. The processes of the background suppression branch and the action suppression branch are similar to that of the context branch, but at the beginning, the 2048-dimensional feature is calculated for the action and background weights by a filtering module. The filtering module consists of two convolutions. The first one down-samples the 2048-dimensional feature to 512 dimensions, and the second one down-samples the 512-dimensional feature to 1. The kernel size of both convolutions is 1. The weight of T x 1 is obtained, which represents the probability of each frame being background or action. The weight is multiplied by the 2048-dimensional input feature to obtain the corresponding weighted feature of the two branches. The subsequent processes are the same as those of the context branch. For the background suppression branch, the value of k is 750 / 8, and for the action suppression branch, the value of k is 0.106 x 750.

[0071] Step (3), model training

[0072] 3-1, For a video, the corresponding label is a binary vector containing C elements, indicating whether each class exists. A binary value is concatenated to indicate whether the background class exists. Thus, a label containing C+1 classes is obtained. Different values are set for the background class for different branches to achieve different learning goals. For example, for the action suppression branch, the label is a vector with C 0 elements concatenated with 1, indicating that only the background class exists. The label and the class score obtained by the corresponding branch are calculated to obtain a cross-entropy loss. For example, for the context branch, the video-level classification score is obtained and the classification label y, the video-level classification loss L is calculated by cross-entropy loss cls .

[0073]

[0074] wherein, is the softmax normalized action classification score. The background suppression branch and the action suppression branch obtain classification losses L fg and L bgThe convolutional attention branch with the re-attention mechanism, i.e., the global-local branch, obtains the classification loss L by a similar process gl .

[0075] 3-2, Attention-guided loss: the C+1-th dimension of the class activation sequence represents the score of the C+1-th class, i.e., the background class in each frame, and the activation score p of the background is obtained by softmax normalization base (t), then 1-p base (t) represents the activation score of the action, and through the guidance loss, 1-p base (t) and the output weight distribution of the background suppression module filter are as close as possible, so as to guide the class activation sequence to suppress the background segment at the video segment level:

[0076]

[0077] where W fg (t) is the output weight of the background suppression module filter.

[0078] Background entropy loss. Due to the relativity of the softmax function, the softmax score of some action classes may be high. For each class of a video, the sum of the corresponding activation values of all background frame class activation sequences is calculated to obtain the background activation value score_bg∈R C+1 , and the cross-entropy loss between the label and score_bg is calculated, so that the background activation value has a uniform probability distribution:

[0079]

[0080] where, is the average action probability of the c-th background segment, p c (s n,j ) is the softmax score of the c-th class of s n,j .

[0081] The overall loss includes the classification loss of the four branches, the guidance loss and the background entropy loss.

[0082] L total =L cls +L fg +L bg +0.85L gl +0.002L gui +0.3L be

[0083] Step (4) generates the positioning detection result, specifically as follows:

[0084] For the class activation sequence and the classification score obtained by the background suppression branch, among the C candidate classes, the class with a video-level classification score lower than a threshold is discarded. Next, the class activation sequence with a high-confidence class is selected, and the continuous frames with higher scores are taken as the action instance to obtain the start time and end time. Then, the confidence score of the predicted action instance is determined by the external-internal comparison strategy.

Claims

1. A convolutional attention weakly supervised temporal action localization method with a re-attention mechanism, characterized in that: (1) Data preprocessing: extract image frames from video data and use the TVL-1 algorithm to extract optical flow; for the extracted image and optical flow, use the I3D model pre-trained based on the Kinetics dataset to extract the image and optical flow features corresponding to the image and optical flow, respectively, then concatenate the two features in the feature dimension to integrate the appearance features and motion features of the input video, scale the video features to a fixed time length, and obtain the complete video data features through linear interpolation; (2) Build a convolutional attention network model with a re-attention mechanism: First, downsample the video data features through convolution, then adaptively learn the importance between nodes according to the attention mechanism; use a multi-head form to calculate the attention weight of the nodes, and finally fuse to get the output weight of all nodes; the re-attention module selects the high importance weight from the output weight of the multi-head self-attention according to the cumulative importance, and uses the selected high importance weight to re-attention the output features of the multi-head self-attention, and by setting different thresholds, different scales of cumulative importance can be selected; Through multiple stages, the features are mined in different feature dimensions, the features of each stage are projected to the same dimension through convolution and concatenated, and finally input into the convolutional classifier to obtain the class activation sequence, which indicates the activation probability of each class in each frame of the video, and the action classification score is obtained through the top-k aggregation strategy; At the same time, three auxiliary branches are added: through the action suppression branch, the background suppression branch and the context branch, the background, the action and the action context are learned respectively, each branch contains two convolution modules to adjust the features to adapt to the subsequent training, three class activation sequences are obtained through the convolutional classifier, and the respective action classification scores are obtained through the top-k aggregation strategy; The two suppression branches use a filter to suppress actions and backgrounds at the beginning; (3) Model training: according to the action classification scores obtained in steps (1) and (2), use cross-entropy loss to constrain the action classification scores through video-level class labels, maximize the action probability entropy of background segments through background entropy loss, and guide the weighted class activation sequence to suppress background segments at the video segment level through guide loss. Use the back propagation algorithm BP to train the model parameters; (4) Generate localization detection results: for the class activation sequence and action classification score obtained by the background suppression branch, among the C candidate classes, discard the classes whose video-level action classification score is lower than the set threshold; select the class activation sequence of the class whose confidence is greater than the set threshold, and take the continuous frames whose action classification score is higher than the threshold as the predicted action instance to obtain the start time and end time of the predicted action instance; then, determine the confidence score of the predicted action instance through the external-internal comparison strategy; In step (2), the convolutional attention network model with a re-attention mechanism is built, and the specific implementation is as follows: 2-1. Convolutional attention: adopt multi-head self-attention mechanism to let multiple heads learn different attention weights respectively and adaptively. Specifically, for input X, for the jth attention block in the first stage, the temporal convolution projects the feature dimension D of the input X to D0 to obtain X0; the subsequent stages continuously expand D0; the first stage X0 passes through the multi-head attention layer in the global relation block, and the multi-head attention layer is composed of H attention heads. For each head i∈{1,...,H}, the input X j is projected to Q ij , ij Q X j , K ij and V ij K are projected to W j , ij ij V X j , where W ij Q , W ij K , represent the weights of the linear layer, D h = D0 / H represents the feature dimension of each head;​ Therefore, the self-attention of head i is calculated as: wherein, is an output weight, The outputs of different attention heads are then mixed with an additional linear layer to obtain the output features of the global module wherein W j O denotes the weight of the linear layer, X j is the bias term of the linear layer, the output feature size of the multi-head attention layer is the same as the input feature size; 2-2. Re-attention mechanism: selecting output features M of multi-head attention j and output weights W j The input re-attention module selects the output weights of the multi-head self-attention according to the accumulated importance, and uses the selected high-importance weights to re-attend to the output features of the multi-head self-attention. An adaptive threshold is constructed using the cumulative distribution sampling method, and the specific operation is as follows: The preliminary attention result W is normalized and sorted from large to small to obtain a sorting weight S, and a cumulative importance I is constructed, j = 0,..., T, W ∈ R H×T×T T represents the length of the video time sequence; a fixed threshold τ' is determined for the cumulative importance, an adaptive threshold is generated for the preliminary attention result W, and a flag parameter b = [I < τ'] is set to represent a binary mask indicating whether the selected frame exists; Then, using the marking parameter b, it is noted again that the selection matrix B is generated according to the marking parameter b for the re-attention, and the selection matrix B is a binary matrix, where each item is B u,v , which means that the v-th frame will be used for the update of the u-th frame; by replacing the self-attention module in the attention block with a masked self-attention module through the selection matrix B, the masked attention is obtained, as follows: Wherein, A is multiplied by the output feature M of the multi-head self-attention to obtain a re-attention output score, the score passes through a full connection layer and a mask softmax layer to generate an importance weight λ; the importance weight λ is used to retrieve a weight value from the pruned output weight W to obtain a re-attention vector, and the re-attention vector is defined as: W' = W © (J T - b) + λr (6) where b k denotes the landmark parameters of the k-th frame; J is a matrix where each element is equal to 1; and the attention vector is obtained by Here, X0 is input into two re-attention modules with different τ' to obtain re-attention vectors W'1 and W'2 of different scales, and then the re-attention vectors are averaged to obtain a final re-attention score; the re-attention score is multiplied by the output feature M of the multi-head self-attention, and then X0 is added as a residual to obtain the output of the re-attention module; The processes of other stages are similar, and the difference lies in that the feature dimensions of the input features are different; all the information mined in the stages is input into a mixer to be mixed to form a unified video representation, and finally input into a convolutional classifier; 2-3. Auxiliary branch: for the context branch, given the input feature X, first input the Base Module, which includes two frames-specific convolutions; specifically, first generate an embedding through a convolution, and then generate a similarity mask by calculating the cosine similarity between adjacent frames of the embedding; the similarity mask is added to the convolution calculation process to obtain the frames-specific convolution; after the input feature X passes through the Base Module, the result is input into a convolutional classifier to obtain a class activation sequence S, wherein the convolutional classifier is a temporal convolution; then a top-k aggregation strategy is used to calculate the video-level action classification score; The action suppression branch and the background suppression branch are similar to the context branch, but there is a filter module at the front end; for the action suppression branch, the filter module suppresses the action score, thereby explicitly modeling the background; the remaining processes are similar to those of the context branch and the background suppression branch, except that the feature maps input are different; in addition to the filter, the other modules share parameters with other branches; when calculating the loss, the first C classes of the label y of the action suppression branch are set to 0, and the last class, i.e., the background class, is set to 1; the C+1 classes of the label y of the context branch are all set to 1; the first C classes of the label y of the background suppression branch are set to 1, and the last class is set to 0.

2. The weakly supervised temporal action localization method with re- attentive convolutional attention according to claim 1, characterized in that Step (1) data preprocessing, which is implemented as follows: Feature extraction is performed on the extracted images and optical flow, and an I3D model pre-trained based on a Kinetics dataset is used to extract image features and optical flow features corresponding to the images and the optical flow, respectively; the image features and the optical flow features are stacked in the feature dimension, and initial spatio-temporal features are obtained; since the unpruned videos have different temporal lengths, scaling operation is needed; videos that are too long are truncated, and videos that are too short are linearly interpolated, and finally fixed temporal length features are obtained.

3. The weakly supervised temporal action localization method with re- attentive convolutional attention according to claim 1, characterized in that The model training in step (3) is implemented as follows: 3-1. First, generate the corresponding training label according to the label instance of the input video; for a video, its corresponding label is a binary vector containing C elements, corresponding to whether each category exists, and a binary value indicating whether the background class exists; thus obtaining a label containing C+1 classes; such as for the action suppression branch, the label is a vector with C 0 elements concatenated with 1, indicating that only the background class exists, and the label and the action classification score obtained by the corresponding branch are calculated to obtain a cross-entropy loss; such as for the context branch, the video-level action classification score and the classification label y, the video-level classification loss L is calculated by the cross-entropy loss cls ; wherein, is the softmax normalized action classification score; the background suppression branch and the action suppression branch obtain classification loss L fg and L bg ; the convolutional attention branch with re-attention mechanism, i.e., the global-local branch, obtains classification loss L gl ; 3-2. Attention-guided loss: To make the action context branch distinguish the action example segments from those ambiguous action context frames at segment level, besides the video-level cross-entropy classification loss L cls In addition, an attention-guided loss is introduced; the action example attention sequence is set as a binary indicator for each video segment, and it is used to guide the weighted CAS to suppress the action context and background segments at the video segment level. Specifically, the attention-guided loss L gui is: wherein, is a predicted segment-level action probability distribution obtained by applying a softmax function, denotes the probability that the segment s(t) belongs to the last class, i.e., the background class, w fg (t) is an output weight of the background suppression module filter; The background entropy loss maximizes the action probability entropy of the background segment, and the background segment is encouraged to have a uniform probability distribution for the action class, and the loss is calculated as follows: wherein, is the average action probability of the cth background segment, p c( s n,j) is the softmax score of the cth class for s n,j ​ Overall loss L total The classification loss, the guidance loss and the background entropy loss including four branches; L total = L cls + L fg + L bg + aL gl + βL gui + ηL be (10) Wherein, α, β, η are adjustment coefficients.

Images