A Weakly Supervised Temporal Action Localization Method Based on Fragment Similarity

Abstract

The application discloses a weakly supervised temporal action localization method based on segment similarity, comprising the following steps: after low-dimensional vectorization representation of an uncut video, inputting the uncut video into a feature extractor to obtain RGB features and optical flow features, wherein the RGB features and the optical flow features are fused through a fusion layer; constructing a similarity module based on a time sequence segment, the module evaluates the similarity between segments, learns global and local information in the video, and propagates the similarity between segments to learn the characteristics of action segments; constructing a pseudo label generation module, dynamically adjusting the weight of the weakly supervised learning mode, thereby dynamically training the whole temporal action localization network to complete the action localization task. The similarity propagation mode enables comprehensive information interaction between segments, and achieves more accurate positioning effect. The pseudo label generation module can provide more accurate pseudo supervision for the action localization network, and improves the positioning accuracy.

Description

Technical Field

[0001] This invention relates to the field of temporal action localization technology, and specifically to a weakly supervised temporal action localization method based on fragment similarity. Background Technology

[0002] Currently, video search engines typically rely on the text information attached to the video, but this text information is often incomplete and inaccurate. To enable computers to extract valuable information from massive amounts of video data, Temporal Action Localization (TAL) was developed. This task solves the problem of the time-consuming and laborious process of using human eyes to locate actions, allowing computers to find the location of actions and saving human resources.

[0003] Action Contexts (TALs) aim to locate the start and end points of action instances while identifying the action category for each instance. Most existing TALs typically achieve this by training a deep neural network under full supervision, where each frame is labeled either as an action or background frame. Generating a TAL model requires a large dataset for training. If trained under full supervision, each frame needs to be labeled as either an action or background frame, which presents several problems: first, frame-level annotation is extremely expensive; second, manual annotation is subjective and introduces significant errors at action boundaries; and third, manual annotation is prone to errors.

[0004] Because unedited videos contain a large number of background frames in addition to action segments (here, "background frames" refers broadly to video segments that do not belong to the category to be localized), weakly supervised methods only have video-level annotations and cannot distinguish between background and action frames. Furthermore, background segments often possess characteristics that interfere with action localization. In weakly supervised temporal action localization tasks, the task is essentially to infer action time intervals from video classification models. However, there is a significant gap between classification and localization tasks. Video classification aims to identify time segments that improve classification accuracy. These segments are the most distinctive and help differentiate the target class from other background classes. However, localization tasks aim to identify complete time segments, not just distinguishable parts. Therefore, this difference between classification and localization tasks leads to incomplete localized segments. On the other hand, localization tasks are trained with precise annotations of action boundaries, thus outputting clear and confident action boundaries during the inference phase. Meanwhile, classification models lack any boundary information, leading to blurred localization boundaries due to this difference between classification and localization tasks. Summary of the Invention

[0005] The purpose of this invention is to propose a weakly supervised temporal action localization method based on segment similarity to mine the feature similarity between video frames, eliminate feature uncertainty, and solve the problem of difficult action-background separation.

[0006] To achieve the above objectives, this application proposes a weakly supervised temporal action localization method based on fragment similarity, comprising:

[0007] After the unedited video is represented in low-dimensional vector form, it is input into the feature extractor to obtain RGB features and optical flow features. The RGB features and optical flow features are then fused through a fusion layer.

[0008] A similarity module based on temporal segments is constructed. This module evaluates the similarity between segments, learns global and local information in the video, propagates the similarity between segments, and learns the characteristics of action segments.

[0009] A pseudo-label generation module is constructed to dynamically adjust the weights of the weakly supervised learning method, thereby dynamically training the entire temporal action localization network to complete the action localization task.

[0010] Furthermore, after representing the unedited video in low-dimensional vector form using the I3D model, the feature extractor learns seamless temporal features from the RGB video data. Learning continuous temporal features from optical flow video data

[0011] Furthermore, the RGB features and optical flow features are fused through a fusion layer, specifically by fusing the temporal features. and time series characteristics Connect them into a complete feature vector It is embedded into the space of a temporal action localization task; it uses a 1D convolutional layer followed by a ReLU function; expressed as:

[0012]

[0013] in Indicates a convolution operator with activation, φ embed The trainable parameters of the representation layer; specifically, the dimension of the embedded features is the same as the dimension of the input features, i.e.

[0014] Furthermore, the similarity module obtains an affinity matrix through the cosine similarity between feature segments, and converts the affinity matrix into a transition probability matrix, enabling the random walk to find similar segments in the video; the affinity matrix is ​​obtained as follows:

[0015]

[0016]

[0017] in T is the transpose operator. τ represents matrix multiplication; Softmax represents softmax operation; Norm represents L1-norm operation for each row of the input matrix; τ is a hyperparameter that controls the smoothness of the normalization result. It is an affinity matrix, where each element represents the normalized cosine similarity between the row index and the column index.

[0018] Furthermore, the similarity module is designed with a guide mask generation part to determine the foreground and background segments, and uses a multilayer perceptron (MLP) to generate a video mask; based on the similarity between segments, it guides the search for possible foreground segments.

[0019] Furthermore, the specific implementation of the guiding mask generation part is as follows:

[0020] First, clear the diagonal elements of the affinity matrix Wn to zero:

[0021]

[0022] in ⊙ represents a matrix with all elements being 1, ⊙ represents element-wise multiplication, and E represents the identity matrix;

[0023] Then, the zeroed affinity matrix is ​​downsampled, i.e., max pooling is performed; the maximum similarity value between a single segment and all other segments is extracted as the unique representative value of that segment's similarity; this representative value is then fused using the input of another fusion layer, defined as:

[0024]

[0025] in yes Row vectors represent the similarity matrix between a single segment and all other segments; and maxpooling is performed on each row of the affinity matrix.

[0026] Based on the fusion layer's judgment of segment similarity, the mask of the video segment is obtained; As input to the MLP, the video mask is obtained after passing through the MLP.

[0027]

[0028] Where σ is the Sigmoid function, and w0 and w1 represent the weights of the neuron;

[0029] Furthermore, the similarity module includes a foreground weight generation part, which generates the fused feature vector. As input, refined video features X′ are obtained using a bipartite random walk (Bi Rrandom Walk) based on the WTAL task. n :

[0030]

[0031] in I represents the identity matrix, and ω represents the parameter [0, 1] which controls the trade-off between feature propagation and original features;

[0032] Refined Feature X′ n The foreground weight is obtained through the filtering module.

[0033]

[0034] Furthermore, in the pseudo-label generation module, the video segment generates a class activation sequence through a classification head; if the video-level label is yes, the class activation sequences of the label class and the background class are extracted, and binarized according to a threshold to obtain the pseudo-label of the label class activation sequence:

[0035]

[0036] θ is the threshold; except for pseudo-labels of the label class, pseudo-labels of other classes are all 0. Then, all pseudo-labels are merged to obtain...

[0037] Furthermore, after generating segment-level pseudo-labels in the pseudo-label generation module, the mean squared error loss is applied to force the activation sequence to fit the pseudo-labels. The mean squared error loss is:

[0038]

[0039] In the formula, N is the total number of segments.

[0040] Compared with existing technologies, the technical solutions adopted in this invention have the following advantages: they solve the problem that existing classifiers focus more on discriminative segments and easily ignore potential segments, enabling the action localization network to pay more attention to potential segments. The similarity propagation method allows for comprehensive information exchange between segments, achieving more accurate localization results. The pseudo-label generation module can provide more accurate pseudo-supervision for the action localization network, improving localization accuracy. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of a weakly supervised temporal action localization method based on fragment similarity;

[0043] Figure 2 This is a schematic diagram of the fragment similarity module designed in this invention;

[0044] Figure 3 This is a diagram of the pseudo-label generation module designed in this invention;

[0045] Figure 4 This is a model diagram for weakly supervised temporal action localization based on fragment similarity. Specific implementation methods

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the application; that is, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0047] like Figure 1 As shown, this embodiment provides a weakly supervised temporal action localization method based on fragment similarity, including:

[0048] Step 1: After representing the unedited video in low-dimensional vector form, input it into the feature extractor to obtain RGB features and optical flow features. The RGB features and optical flow features are then fused through a fusion layer.

[0049] Specifically, representing videos as low-dimensional vectors facilitates better network training. The I3D model is used to process unedited videos; it's a two-stream inflated 3D ConvNet based on 2D ConNet inflation, extending the filters and pooling kernels of very deep classification ConvNets to 3D. The feature extractor learns seamless temporal features from RGB video data. Learn continuous temporal features from optical flow video data

[0050] Concatenate the RGB features and flow features into a complete feature vector. It is embedded into the space of a specific temporal action localization task; it uses a 1D convolutional layer followed by a ReLU function. Represented as:

[0051]

[0052] in Indicates a convolution operator with activation, φ embed The trainable parameters of the representation layer. Specifically, the dimension of the embedded features is the same as the dimension of the input features, i.e.

[0053] This fusion layer is actually a convolutional network; the purpose of embedding the RGB features and optical flow features of the video into the fusion layer is to enable the effective combination of RGB features and optical flow features, in order to prepare for the subsequent video action classification and localization task.

[0054] Step 2: As Figure 2 As shown, a similarity module based on temporal segments is constructed. This module evaluates the similarity between segments, learns global and local information in the video, propagates the similarity between segments, and learns the characteristics of action segments.

[0055] Specifically, the similarity module calculates the cosine similarity between feature segments to obtain an affinity matrix (i.e., a similarity matrix), which is then converted into a transition probability matrix to facilitate random walks in finding similar segments in the video. It simply transposes the input video features and multiplies the original feature vector with the transposed feature vector to obtain an affinity matrix (also called a similarity matrix). Essentially, it transforms the semantic affinity between segment features into a transition probability matrix, enabling random walks to perceive similar segments in the video and encouraging interaction of class activation values ​​between similar segments. Intuitively, the more similar the features (containing the same action class) between segments, the higher the product value after multiplying the two. Therefore, the affinity matrix is ​​obtained by calculating the cosine similarity between segment features, i.e.:

[0056]

[0057]

[0058] in T is the transpose operator. This represents matrix multiplication. Softmax represents the softmax operation, and Norm represents the L1-norm operation for each row of the input matrix. τ is a hyperparameter that controls the smoothness of the normalization result. It is an affinity matrix, where each element represents the normalized cosine similarity of the corresponding segment features for the row and column indices.

[0059] The similarity module incorporates a guide mask generation part to distinguish between foreground and background segments, and utilizes a multilayer perceptron (MLP) to generate video masks to optimize video features. Based on the similarity between segments, the network is guided to find possible foreground segments. First, the diagonal elements of Wn are cleared using the following equation:

[0060]

[0061] in This represents a matrix with all elements equal to 1, ⊙ represents element-wise multiplication, and E represents the identity matrix. This zeroing operation is performed to eliminate interference caused by the segments themselves and to focus on the relationships between segments.

[0062] Then, the obtained segment similarity matrix is ​​downsampled to reduce data dimensionality and computational complexity, thereby improving efficiency. Specifically, this is achieved by performing a max-pooling operation on the similarity matrix. The maximum similarity value between a single segment and all other segments is extracted as the unique representative value of that segment's similarity. Using this as input to another fusion layer, the representative values ​​are fused, defined as:

[0063]

[0064] in yes Row vectors, representing the similarity matrix of a single segment to all other segments, are sparse representations of the entire video. Max pooling is then performed on each row of the affinity matrix.

[0065] Based on the similarity assessment of segments by the fusion layer, the mask of the video segment is obtained. As input to the MLP, the video mask is obtained after passing through the MLP.

[0066]

[0067] Where σ is the Sigmoid function, and w0 and w1 represent the weights of the neurons.

[0068] The similarity module also includes a foreground weight generation part. Through foreground weighting, the class activation sequence is further optimized, improving the accuracy of action localization. Local information is studied based on the original features and the feature similarity matrix. A bipartite random walk (Bi Rrandom Walk) based on the WTAL task is used; the fused RGB features and optical flow features X are then analyzed. n As input, the refined video features X′ are obtained. n Specifically, the random walk operation performed is as follows:

[0069]

[0070] in I represents the identity matrix, and ω represents the parameter [0, 1] which controls the trade-off between feature propagation and the original features. This random walk operation effectively propagates knowledge over time periods by aggregating similar features.

[0071] Refined Feature X′ n The foreground weight is obtained through the filtering module.

[0072]

[0073] The aim is to further optimize class activation sequences using local information, thereby improving the accuracy of action localization. The filtering module consists of multiple convolutional layers, which seek to identify action segments and suppress background noise.

[0074] Step 3: As Figure 3 As shown, a pseudo-label generation module is constructed to dynamically adjust the weights of the weakly supervised learning method, thereby dynamically training the entire temporal action localization network to complete the action localization task.

[0075] Specifically, by thresholding the class activation sequences of the label and background classes, their corresponding pseudo-ground truths are obtained, thus classifying the class activation sequences. Then, the pseudo-ground truth information for the remaining classes is defined as 0, and all pseudo-ground truth information is concatenated to obtain video segment-level pseudo-ground truth information. These generated pseudo-ground truth values ​​are used to provide supervision for the network to obtain pseudo-ground truth loss, thereby further improving the localization accuracy of action instances. Specifically, a video segment generates a class activation sequence through a classification head. If the video-level label is , the class activation sequences of the label and background classes are first extracted and binarized according to a threshold to obtain the pseudo-pseudo-labels of the label class activation sequence.

[0076]

[0077] θ is the threshold. Except for pseudo-labels of the label class, all other pseudo-labels are 0, i.e., 0. Then, all pseudo-labels are merged to obtain...

[0078] After generating segment-level pseudo-labels, the activation sequence of the original class is forced to fit the pseudo-labels as closely as possible, thereby applying mean squared error loss:

[0079]

[0080] Based on the above operations, the pseudo-label generation method eliminates the uncertainty of network localization actions and provides stronger pseudo-supervision for the network.

[0081] The above steps can leverage the semantic correlation between different video segments to mine feature similarity between video frames, eliminating feature uncertainty and addressing the problem of incomplete localization. Furthermore, by learning global information, generating segment masks, and learning local contextual information, foreground weights are obtained. This allows for better capture of action segments, suppressing background noise while identifying relevant action segments. Simultaneously, pseudo-label generation technology pays closer attention to the presence of the background, designing separate pseudo-labels for background classes to improve network learning performance and generalization ability, and to some extent, contribute to increased accuracy.

[0082] In this application, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0083] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A weakly supervised temporal action localization method based on segment similarity, characterized in that, The uncut video is input into a feature extractor after being represented by a low-dimensional vector to obtain RGB features and optical flow features, and the RGB features and the optical flow features are fused through a fusion layer. A similarity module based on a time sequence segment is constructed, which evaluates the similarity between segments, learns global and local information in the video, and propagates the similarity between segments to learn the characteristics of action segments. A pseudo-label generation module is constructed to dynamically adjust the weight of the weakly supervised learning method, thereby dynamically training the entire time sequence action localization network to complete the action localization task. The similarity module obtains an affinity matrix through the cosine similarity between feature segments, and converts the affinity matrix into a transition probability matrix, so that random walk can find similar segments in the video. The affinity matrix is obtained in the following manner: The similarity module designs a guiding mask generation part to determine foreground and background segments, and generates a video mask using a multi-layer perception (MLP). wherein is a transpose operator, denotes a matrix multiplication operation; Softmax denotes a softmax operation, Norm denotes an L1-norm operation on each row of the input matrix; τ is a hyper-parameter controlling the smoothness of the normalized result; is an affinity matrix, where each element represents the normalized cosine similarity of the row index and the column index; According to the similarity between segments, possible foreground segments are guided to be found. The guiding mask generation part is implemented in the following manner: First, the diagonal elements of the affinity matrix Wn are set to zero: wherein denotes an element-wise multiplication, and E denotes an identity matrix. Then, the affinity matrix after zeroing is down-sampled, that is, a max-pooling operation is performed. The maximum value of the similarity between a single segment and all other segments is extracted as the unique representative value of the similarity of the segment. wherein is a row vector representing the similarity matrix of a single segment with all other segments; and performing a max pooling operation on each row of the affinity matrix; According to the judgment of the fusion layer on the similarity of the segments, a mask of the video segments is obtained; and As the input of the MLP, the mask of the video is obtained after the MLP The representative values are fused as inputs to another fusion layer, and are defined as follows:

2. The weakly supervised temporal action localization method based on segment similarity according to claim 1, wherein, After the untrimmed videos are represented as low-dimensional vectors by the I3D model, the feature extractor learns seamless temporal features from the RGB video data Continuous temporal features are learned from the optical flow video data 3. The weakly supervised temporal action localization method based on segment similarity according to claim 1, wherein, The RGB feature and the optical flow feature are fused through a fusion layer, specifically, the time sequence feature and the time sequence feature are connected into a complete feature vector and embedded into a space of a time sequence action positioning task. Where σ is a Sigmoid function, and w0 and w1 represent the weights of neurons. wherein denotes a convolutional operator with activation, φ embed denotes the trainable parameters of the layer; in particular, the dimension of the embedded features is the same as the dimension of the input features, i.e.

4. The weakly supervised temporal action localization method based on segment similarity according to claim 1, wherein, The similarity module includes a foreground weight generation section that generates a fused feature vector As input, the refined video features X' are obtained using a two-part random walk based on the WTAL task improvement n : wherein I denotes the identity matrix and ω denotes a parameter [0, 1] controlling the trade-off between feature propagation and original features; Refined feature X' n Obtaining foreground weights thereof by a filtering module 5. The weakly supervised temporal action localization method based on segment similarity according to claim 1, wherein, A 1D convolution layer is used, followed by a ReLU function, and is represented as follows: θ is a threshold value; in addition to the pseudo label of the label class, the pseudo label of other classes is 0, and then all the pseudo labels are combined to obtain 6. The weakly supervised temporal action localization method based on segment similarity according to claim 5, wherein, In the pseudo-label generation module, a class activation sequence is generated for the video segment through a classification head. If the video-level label is yes, the class activation sequences of the label class and the background class are extracted, and are binarized according to a threshold to obtain the pseudo-label of the label class activation sequence: After generating the segment-level pseudo-label in the pseudo-label generation module, a mean square error loss is applied to force the activation sequence to fit the pseudo-label. The mean square error loss is as follows: Where N is the total number of segments.

Images