A Weakly Supervised Temporal Behavior Detection Method Based on Context Modeling and Background Suppression

Abstract

This invention discloses a weakly supervised temporal behavior detection method based on context modeling and background suppression, comprising the following steps: dividing a video into multiple non-overlapping segments, extracting spatial features and temporal motion features of the video scene, and constructing a video-level feature representation for each segment; designing a memory bank M as a learning source for action localization, and modeling the contextual information of the video based on a self-attention module; adding a background auxiliary class, and using a filtering module to suppress the input features of background frames; further jointly implementing video-level prediction by combining the refined segment features and foreground weights in an attention weighted pool. This scheme introduces a self-attention module, which models the potential temporal structure of action segments during the feature modeling and prediction stages, thereby refining action features of different attributes and ensuring the integrity of behavior instances; adding a background auxiliary class, using a filtering module to attenuate the input features from background frames, and creating negative samples for the background class, thereby learning the features of background segments, suppressing the influence of background noise, and improving the accuracy of action detection.

Description

Technical Field

[0001] This invention belongs to the field of video temporal action detection technology, specifically relating to a weakly supervised temporal action detection method based on context modeling and background suppression. Background Technology

[0002] With the popularization and development of multimedia, the internet, and filming equipment, video data is experiencing explosive growth. For video temporal action localization, a large amount of labeled information is needed for training. Accurate temporal boundary labeling is extremely expensive, requiring significant human and financial resources, which greatly limits the application of temporal action detection algorithms. Weakly supervised training has emerged as a solution. Weakly supervised action localization technology uses only video-level labels during training, further reducing the waste of human resources and time, as well as labeling errors, and offering excellent flexibility.

[0003] Most existing weakly supervised temporal action localization (WTAL) methods can be divided into two categories. One approach is inspired by weakly supervised image semantic segmentation tasks, treating weakly supervised temporal action detection as a video recognition task. It introduces a foreground-background separation attention mechanism to construct video-level features, and then applies an action classifier to recognize the video. The other approach formulates this task as a multiple instance learning (MIL) problem, treating the entire unedited video as a packet containing positive and negative instances, i.e., action instances and background frames (background frames refer to video segments that do not belong to the category to be detected). Segment classification is performed over time to generate class activation sequences (CAS), and then video-level predictions are obtained by temporarily merging CAS, and segment-level class scores are thresholded to generate action proposals.

[0004] Both of the above methods aim to learn an effective classification function to identify actions from action instances and background frames. However, existing methods do not fully simulate the action detection problem and still face challenges related to localization integrity and background interference.

[0005] (1) Localization Integrity Problem: For a continuous temporal behavior, recognizing an action often relies excessively on feature regions that are obviously helpful for classification, which will lead to incomplete localization. For example... Figure 1 Here is an example of a diving activity. (a) represents the actual location, and (b) is the prediction based on the MIL method. The MIL framework only captures the most discriminative locations in the entire diving activity, which can produce high classification confidence but does not produce good localization performance.

[0006] (2) Background interference: Background frames are trained to be classified as action frames in the video, even if they do not have action features. This inconsistency in classifying background frames as action frames leads to false positives and a decrease in detection performance. Currently, weakly supervised methods are trained directly based on video-level category labels, considering only action frames and not background frames. False positives are still inevitable, limiting the detection accuracy of the model. Summary of the Invention

[0007] To address the issues of incomplete localization and background interference in existing technologies, this invention proposes a weakly supervised temporal behavior detection method based on context modeling and background suppression. The method uses a self-attention module to model the contextual information of the video, thereby extracting action features to encourage smoother temporal classification scores and achieve complete localization. Furthermore, a filtering module is designed to suppress the input features of background frames to prevent interference from background noise.

[0008] This invention is achieved using the following technical solution: a weakly supervised temporal behavior detection method based on context modeling and background suppression, comprising the following steps:

[0009] Step A: First, divide the video into multiple non-overlapping segments, extract the video scene spatial features and temporal motion features, and then fuse them to construct a video-level feature representation;

[0010] Step B: Design a memory bank M as the learning source for action localization, and use a self-attention module to model the contextual information of the video to extract segment-level action features and train a segment-level classifier.

[0011] Step C: Add a background auxiliary class to suppress the input features of the background frame through the filtering module to prevent interference from background noise and obtain foreground attention weights;

[0012] Step D: Combining steps B and C, the network is iteratively optimized and trained. The refined segment-level action features and foreground attention weights are further combined in the attention weighting pool to achieve video-level prediction.

[0013] Furthermore, step A is specifically implemented in the following way:

[0014] Using a uniform sampling strategy to sample video V i The scene is divided into T non-overlapping segments, and scene spatial features are extracted based on a feature extractor. and time motion characteristics Then, the two stream segment-level features are fused to obtain x. i ∈R 2D i∈[1,T], and then construct video-level feature representations. D represents the feature dimension.

[0015] Furthermore, step B specifically includes the following steps:

[0016] (1) The video-level features obtained in step A Stored into memory M, M∈R T×2D Using E Q E k and E v The encoder generates queries, keys, and values ​​corresponding to the video segments respectively;

[0017] K i =E k (M)

[0018] V i =E v (M)

[0019] K i ∈R T×2D / m V i ∈R T×(C+1)2D These are keys and values, and m is a hyperparameter that controls memory read efficiency;

[0020] (2) Based on encoder E Q Video-level features Encoding a set of query Q i Q i ∈R T×2D / m Then, similarity scores are calculated between video segments with queries, and contextual information is aggregated using these similarity scores to obtain refined segment-level action features:

[0021]

[0022] Where I is the identity matrix used to store the original video information. and Maintain the same dimensions; extract global contextual information and obtain more discriminative features that are easy to classify and locate through information transfer between segments;

[0023] (3) Calculate Q i With K i The interactions between segments are analyzed, and the correlations between different segments are obtained, thus giving the network a global view. Finally, the similarity matrix V is obtained by aggregation. i o As shown below:

[0024]

[0025] Among them, V i o ∈R T×(C+1)2D ;

[0026] (4) The similarity matrix V i o Remodeled into a set of fragment-level classifiers It adapts to changes in the appearance or movement of each segment; utilizing V i o Calculate the sparsity loss L s Function to train a fragment-level classifier:

[0027]

[0028] Here, ||||1 represents the L1 loss, which encourages low similarity between the background frame and all action segments.

[0029] Furthermore, step C is specifically implemented in the following manner:

[0030] (1) As input to the filtering module, the filtering module comprises two temporal 1D convolutions and a sigmoid function, trained with a training objective opposite to the background class to suppress background frames, thereby obtaining foreground attention weights. W i ∈[0,1], It is a function with parameter Φ;

[0031] (2) Utilizing real behavior categories With predicted score p j Construct binary cross-entropy loss L for each class sup To train the filter module:

[0032]

[0033] Where, p j This indicates the predicted score.

[0034] Furthermore, in step D, video-level prediction is achieved by combining steps B and C, specifically through the following methods:

[0035] Applying the classifier to the corresponding segments yields video-level classification results. Obtained from the attention-weighted pool:

[0036]

[0037] in, Action classification loss is calculated from the predictions among N videos and the video label y. i constitute:

[0038]

[0039] Among them, L actLet C represent the action classification loss, and C represent the total number of action categories.

[0040] Furthermore, in step D, when iteratively optimizing the network during training, the following method is specifically adopted:

[0041] (1) Combine steps B and C to define the joint loss function;

[0042] L tol =λ1L sup +λ2L act +λ3L s

[0043] Where λ1, λ2 and λ3 are hyperparameters that need to be learned to balance the contribution of each loss function;

[0044] (2) Video localization reasoning:

[0045] 1) Video-level prediction score Set a threshold and discard items with confidence scores below the threshold θ. cls Category;

[0046] 2) Apply threshold θ to each remaining category act Apply foreground attention weights to generate action proposals:

[0047] To assign confidence to each action proposal, we first compute the class activation sequence CAS, and then pass it along the class dimension through Softmax to obtain the class score at each time point. Furthermore, the action proposal {(c,q,t)} s ,t e The confidence level q in )} is:

[0048]

[0049] Finally, since an action instance may appear multiple times in an unedited video, this approach uses nonmaximum-like suppression (NMS) to remove highly overlapping action proposals.

[0050] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0051] (1) To address the issue of localization integrity, this solution introduces self-attention to model the latent temporal structure of action segments during the feature modeling and prediction stages:

[0052] Traditional MIL methods, which treat video segments as independent instances, neglect to model the underlying temporal structure during feature modeling and prediction, resulting in low-quality action proposals generated from CAS. This proposed solution first designs a memory bank M as the learning source for action localization and introduces a self-attention module to model the contextual information of the video, thereby extracting action features to encourage smoother temporal classification scores and achieve complete localization.

[0053] (2) To address the background interference problem, a filtering module is designed:

[0054] Unedited videos contain a large number of background frames in addition to action segments. Weakly supervised methods only provide video-level annotations and cannot distinguish between background frames and action segments, leading to many background segments being mistakenly identified as the action to be detected. This solution adds a background auxiliary class and uses a filtering module to suppress the input features of background frames, preventing interference from background noise. Attached Figure Description

[0055] Figure 1 A schematic diagram of capturing action instances using the existing MIL framework; (a) represents the actual localization, and (b) represents the prediction based on the MIL method.

[0056] Figure 2 This is a schematic diagram of the overall network architecture of an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram illustrating the self-attention application principle of an embodiment of the present invention;

[0058] Figure 4 The following are schematic diagrams of positioning results displayed on THUMOS14 according to an embodiment of the present invention, wherein (a) is a schematic diagram of basketball playing results; (b) is a schematic diagram of shot put and discus throwing results; and (c) is a schematic diagram of ice dancing results. Detailed Implementation

[0059] To better understand the above-described objects, features, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the present invention; however, the present invention may be practiced in other ways than those described herein, and therefore, the present invention is not limited to the specific embodiments disclosed below.

[0060] This plan proposes an overall framework as follows: Figure 2 As shown, it includes a self-attention module and a filtering module. Before delving into its details, let's formally define the problem statement:

[0061] Problem description:

[0062] Suppose there are N training videos, and for each video Vi They all have real labels. i ∈R C+1 Where C+1 is the number of action categories; if action category j exists in the video, then y i (j) = 1, otherwise y i (j) = 0. During testing, the goal of temporal motion localization is to generate a set of motion proposals {(c,q,t)} for each video. s ,t e )}, where c represents the predicted category, q is the confidence score, and t s and t e These represent the start and end times of the action, respectively. The problem this invention aims to solve is to locate the start and end boundaries of behavioral instances in unedited videos and identify their corresponding behavioral categories using training data with only video-level category annotations. The key design points are as follows:

[0063] Key Point 1: How to model the integrity of behavioral instances:

[0064] In unedited videos, the lack of fine-grained temporal boundary annotations makes detecting complete and accurate behavioral instances extremely difficult. This solution introduces a self-attention module, which models the potential temporal structure of action segments during the feature modeling and prediction stages, thereby refining action features of different attributes. This encourages smooth segment classification scores and ensures the integrity of behavioral instances.

[0065] Key Point 2: How to suppress background interference:

[0066] For weakly supervised methods with only video-level annotations, the inability to distinguish between background frames and action segments leads to many background segments being mistakenly identified as the action to be detected. This solution adds a background auxiliary class, attenuates input features from background frames through a filtering module, and creates negative samples for the background class. This allows the method to learn the features of background segments, suppress the influence of background noise, and improve the accuracy of action detection.

[0067] This embodiment proposes a weakly supervised temporal sequence detection method based on context modeling and background suppression, including the following steps:

[0068] Step A: First, divide the video into multiple non-overlapping segments, extract the video scene spatial features and temporal motion features, and then fuse these features to construct video-level features;

[0069] Step B: Design a memory bank M as the learning source for action localization, and introduce a self-attention module to model the contextual information of the video, thereby extracting segment-level action features to achieve complete localization.

[0070] Step C: Add a background auxiliary class and use the filtering module to suppress the input features of the background frame to prevent interference from background noise and obtain foreground attention weights;

[0071] Step D: Combining steps B and C, the network is iteratively trained to achieve video-level prediction.

[0072] Specifically, the present invention will be described in detail below:

[0073] 1. Feature Extraction

[0074] Using a uniform sampling strategy to sample video V i The scene was divided into T non-overlapping 16-frame segments, and the I3D feature extractor was used to extract scene spatial features. and time motion characteristics Then, the fragment-level features of the RGB stream and the Flow stream are fused to obtain x. i ∈R 2D i∈[1,T], and then construct the video-level feature representation X. i e =[x1,...,x T ]∈R 2D×T D represents the feature dimension.

[0075] 2. Self-attention module

[0076] (1) Store the T segments and 2D-dimensional video-level feature representations into a memory bank M, M∈R. T×2D In China, E is used Q E k and E v The encoder generates queries (Q), keys (K), and values ​​(V) corresponding to the video segments respectively;

[0077] E k Aimed at reducing the dimensionality of fragments, its keys store information about the fragment's appearance and motion for efficient retrieval from memory, and it is implemented by a fully connected layer (FC). E v An MLP network consisting of two fully connected (FC) layers, with a bottleneck structure between the FC layers to reduce parameters, aims to encode each segment into a category-specific feature for classification.

[0078]

[0079] K i ∈R T×2D / m V i ∈R T×(C+1)2D These are keys and values, and m is a hyperparameter controlling memory read efficiency. Given the memory M and the input video, the following section describes how to perform video classification and background suppression.

[0080] (2) For video classification, the encoder E implemented by the FC layer Q Video-level features Encoding a set of query Q i Q i ∈R T×2D / m Then, similarity scores are calculated between video segments with queries, and contextual information is aggregated using these similarity scores to obtain refined segment features, such as... Figure 3 As shown, it represents the following:

[0081]

[0082] Where I is the identity matrix used to store the original video information. and Maintain the same dimensions; extract global contextual information and obtain more discriminative features for classification and localization through information transfer between segments. Figure 3 Self-attention is applied to each query fragment, and contextual information is aggregated by calculating similarity with other fragments. and This represents element-wise addition and matrix multiplication, where T and 2D represent the number of video segments and feature dimensions, respectively.

[0083] (3) Calculate Q i With K i The network analyzes the interactions between segments and obtains the correlations between different segments, thus providing a global view. Finally, these correlation scores are aggregated to obtain a similarity matrix, as follows:

[0084]

[0085] Among them, V i o ∈R T×(C+1)2D ;

[0086] (4) In order to perform classification later, the similarity matrix V i o Remodeled into a set of segment classifiers It adapts to changes in the appearance or movement of each segment; utilizing V i o Calculate the sparse loss function:

[0087]

[0088] Here, ||||1 represents the L1 loss, which encourages low similarity between the background frame and all action segments.

[0089] 3. Suppression Module

[0090] (1) In order to create negative samples of the background class, the video-level feature representation is used. As input to the filtering module, the filtering module comprises two temporal 1D convolutions and a sigmoid function, trained with a training objective opposite to the background class to suppress background frames, and returns foreground attention weights. W i ∈[0,1], It is a function with parameter Φ. W i Treat a set of fragments without any background activity as negative samples of the background class.

[0091] (2) In this process, real behavior categories are utilized. With predicted score p j Construct binary cross-entropy loss L for each class sup Apply constraints:

[0092]

[0093] (3) Finally, the classifier is applied to the corresponding segments, and the video-level classification results are obtained. Obtained from the attention-weighted pool, calculated as follows:

[0094]

[0095] in, Action classification loss is calculated from the predictions among N videos and the video label y. i constitute:

[0096]

[0097] 4. Network Training and Inference

[0098] (1) Combining steps 2 and 3, define the joint loss function:

[0099] L tol =λ1L sup +λ2L act +λ3L s (8)

[0100] Here, λ1, λ2, and λ3 are hyperparameters that need to be learned to balance the contribution of each loss function.

[0101] (2) After training the model, a two-step approach is used to achieve behavior localization;

[0102] Video-level prediction score Set a threshold and discard items with confidence scores below the threshold θ. cls Category;

[0103] Then, the threshold θ is applied to each of the remaining categories. act Apply foreground attention weights to generate action proposals;

[0104] To assign confidence to each proposal, we first compute the class activation sequence CAS, and then pass it through Softmax along the class dimension to obtain the class score at each time point. (t represents the fragment index, which is indicated by ";" due to the dynamic nature of the time position.) Then the action proposal {(c,q,t)} s ,t e The confidence level q in )} is calculated as follows:

[0105]

[0106] To remove motion proposals with high overlap (high overlap means that the same action appears multiple times in an unedited video; in this embodiment, only one is counted and represented), this solution uses nonmaximum-like suppression (NMS) for processing.

[0107] Implementation details:

[0108] This embodiment uses a dual-stream I3D network as a feature extractor and applies the TV-L1 algorithm to extract optical flow from RGB data, setting D = 1024. In formula (8), λ1 = λ2 = 0.8, λ3 = 0.2. During the inference process, the threshold θ cls Let θ be 0.1 (its value is generally between 0.1 and 1). act It is video V i The mean of the foreground weights for each category is calculated. Class-NMS with a threshold of 0.3 is used to remove highly overlapping proposals. This approach uses a PyTorch deep learning network framework, and the entire experiment was conducted on a single GTX 3060 GPU, trained using Adam optimization with a learning rate of 10. -4 The batch size is 20.

[0109] like Figure 4 As shown, Figure 4 The localization results are shown on THUMOS14. Each example has three plots with multiple sampled frames. The first plot indicates the real-world situation. The second and third plots represent the segmented activation sequences corresponding to the self-attention module and the filtering module, respectively. The horizontal axis represents the time step of the video, and the vertical axis represents the activation intensity, ranging from 0 to 1.

[0110] Figure 4 The results of the proposed algorithm on the THUMOS14 dataset are qualitatively presented. Figure 4(a) Involves frequent action instances; all frames in the video share similar elements, namely people and basketballs. This is addressed by introducing sparse L during context modeling. s Loss is found by identifying subtle differences between actions, between actions and context, and between contexts, thereby avoiding contextual confusion. Figure 4 (b) Contains action instances from two different classes, namely "Throw Discus" and "Shotput". Although the visual appearance and motion patterns are very similar across all frames, the method of the present invention can still locate most of the time intervals of multiple actions. Figure 4 (c) depicts a single action “ice dance” with a background challenge that looks very similar to the foreground. Even so, the model achieves separation of action from context through self-attention context modeling and the suppression of background frames by the filtering module.

[0111] This invention focuses on academic problems in weakly supervised action recognition and localization tasks, such as inaccurate action boundary localization due to background frame interference and incomplete action localization caused by the arbitrary neglect of some candidate segments. To better address these problems, this invention designs a context modeling framework and a learning background suppression paradigm to solve weakly supervised temporal action localization tasks. The solution to the first problem is to model the potential temporal structure of action segments during the feature modeling and prediction stages, thereby refining action features of different attributes to encourage smooth segment classification scores. The guiding principle for the second problem is to add an auxiliary background class and use a filtering module to suppress the input features of background frames, thus preventing interference from background noise. By combining high-quality classification scores with accurate foreground weights, a significant improvement in video-level prediction performance is achieved. Extensive experiments on the THUMOS14 and ActivityNet1.2 datasets demonstrate the effectiveness and feasibility of this method.

[0112] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments for application in other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A weakly supervised temporal behavior detection method based on context modeling and background suppression, characterized in that, Includes the following steps: Step A: First, divide the video into multiple non-overlapping segments, extract the video scene spatial features and temporal motion features, and then fuse them to construct a video-level feature representation; Step B: Design a memory bank M as the learning source for action localization, and use a self-attention module to model the contextual information of the video to extract segment-level action features and train a segment-level classifier. (1) The video-level features obtained in step A Stored into memory M, M∈R T×2D T represents the number of non-overlapping segments, D represents the feature dimension, and E represents the feature size. Q E k and E v The encoder generates queries, keys, and values ​​corresponding to the video segments respectively; K i ∈R T×2D / m V i ∈R T×(C+1)2D These are keys and values, and m is a hyperparameter that controls memory read efficiency; (2) Based on encoder E Q Video-level features Encoding a set of query Q i Q i ∈R T×2D / m Then, similarity scores are calculated between video segments with queries, and contextual information is aggregated using these similarity scores to obtain refined segment-level action features: Where I is the identity matrix used to store the original video information. and Maintain the same dimensions; extract global contextual information and obtain more discriminative features that are easy to classify and locate through information transfer between segments; (3) Calculate Q i With K i The interactions between segments are analyzed, and the correlations between different segments are obtained, thus giving the network a global view. Finally, the segments are aggregated to obtain a similarity matrix. As shown below: in, (4) Similarity matrix Remodeled into a set of fragment-level classifiers It adapts to changes in the appearance or movement of each segment; utilizing Calculate the sparse loss function to train the fragment-level classifier: Where || ||1 represents the L1 loss, which encourages low similarity between the background frame and all action segments; Step C: Add a background auxiliary class to suppress the input features of the background frame through the filtering module to prevent interference from background noise and obtain foreground attention weights; (1) As input to the filtering module, the filtering module comprises two temporal 1D convolutions and a sigmoid function, trained with a training objective opposite to the background class to suppress background frames, thereby obtaining foreground attention weights. W i ∈[0,1], It is a function with parameter Φ; (2) Utilizing real behavior categories With predicted score p j Construct a binary cross-entropy loss L for each class sup To train the filter module: Where, p j L represents the predicted score. sup Represents binary cross-entropy loss; Step D: Combining steps B and C, the network is iteratively optimized and trained. The refined segment-level action features and foreground attention weights are further combined in the attention weighting pool to achieve video-level prediction.

2. The weakly supervised temporal behavior detection method based on context modeling and background suppression according to claim 1, characterized in that: Step A is specifically implemented in the following way: Using a uniform sampling strategy to sample video V i The scene is divided into T non-overlapping segments, and scene spatial features are extracted based on a feature extractor. and time motion characteristics Then, the two stream segment-level features are fused to obtain x. i ∈R 2D i∈[1,T], and then construct video-level feature representations. D represents the feature dimension.

3. The weakly supervised temporal behavior detection method based on context modeling and background suppression according to claim 1, characterized in that: In step D, video-level prediction is achieved by combining steps B and C, specifically through the following methods: Applying the classifier to the corresponding segments yields video-level classification results. Obtained from the attention-weighted pool: in, Action classification loss is calculated by combining the predictions and the actual video labels y among N videos. i constitute: Among them, L act C represents the action classification loss, and C+1 represents the total number of action categories.

4. The weakly supervised temporal behavior detection method based on context modeling and background suppression according to claim 1, characterized in that: In step D, the following method is specifically adopted when iteratively optimizing the network training: (1) Combine steps B and C to define the joint loss function; L tol =λ1L sup +λ2L act +λ3L s Where λ1, λ2 and λ3 are hyperparameters that need to be learned to balance the contribution of each loss function; (2) Video localization reasoning: 1) Video-level prediction scores Set a threshold and discard items with confidence scores below the threshold θ. cls Category; 2) Threshold θ on each remaining category act Apply foreground attention weights to generate action proposals: To assign confidence to each action proposal, we first compute the class activation sequence CAS, and then pass it along the class dimension through Softmax to obtain the class score at each time point. Furthermore, the action proposal {(c,q,t)} s ,t e The confidence level q in )} is: Finally, nonmaximal suppression (NMS) is used to remove highly overlapping action proposals.

Images