A method for detecting timing behavior based on context aggregation and boundary generation

Abstract

The present application belongs to the technical field of computer vision and pattern recognition, and particularly relates to a time sequence behavior detection method based on context aggregation and boundary generation. A multi-path time sequence context feature aggregation module is designed to effectively aggregate long-term and short-term time context information, thereby enhancing the context representation of behavior events. Secondly, in order to generate accurate behavior boundaries, a multi-branch time boundary detector is designed to optimize the prediction results by utilizing the complementary relationship between the two boundary detectors. In addition, in order to accurately predict the confidence of densely distributed nominees, a nominee relationship perception module is designed to model the nominee relationship by utilizing global correlation, thereby enhancing the expressiveness and robustness of the nominee context. The present application can effectively detect videos with long duration and short duration, and can realize high recall rate and high precision capture of behavior boundaries.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition technology, specifically relating to a temporal behavior detection method based on context aggregation and boundary generation. Background Technology

[0002] In recent years, with the development of technology and the internet, it has become increasingly common for people to transmit information via video. Videos can be easily and quickly recorded using mobile phones, computers, and surveillance cameras, leading to an explosive growth in the number of videos. Therefore, accurately extracting the necessary information from massive amounts of video has become a crucial issue in video understanding. The purpose of temporal behavior detection is to locate the temporal boundaries of behaviors of interest in raw videos and classify these actions. This technology can be widely applied in video surveillance, video search and retrieval, autonomous driving, human-computer interaction, and video content recommendation, possessing significant research value and high practical value. In terms of social security, temporal behavior detection technology can automatically detect abnormal behavior segments, enabling timely discovery and prevention, reducing casualties and property damage, significantly reducing police workload, and improving efficiency. In video retrieval, temporal behavior detection technology can find specific behaviors from massive amounts of video, intelligently generating highlights and improving user experience. Furthermore, with massive amounts of videos uploaded and forwarded on the internet every day, temporal behavior detection can automatically retrieve violent videos and delete them promptly, restoring and purifying the online environment and preventing minors from being misled.

[0003] Temporal behavior detection methods consist of two steps: first, generating temporal behavior nominations, and then classifying the nominations. Currently, the mainstream temporal behavior nomination generation methods fall into two main categories: top-down and bottom-up approaches.

[0004] The "top-down" approach generates a large number of behavior nominations densely using sliding windows or uniformly distributed anchor points. However, this approach is limited and lacks flexibility when handling behavior instances with varying durations due to fixed-length segments. The "bottom-up" approach, based on boundaries, first evaluates the probability of potential behaviors at each time point in the video sequence and then matches them to form candidate nominations. However, this approach has the following problems: 1. It extracts temporal features based on local information, lacking modeling of global and long-term temporal relationships to obtain features covering behavior instances with different durations. 2. It mainly relies on shallow local features around the boundary to predict behavior boundaries, ignoring the capture of deep semantic features, resulting in inaccurate generated behavior boundaries and limiting performance improvement. 3. It does not consider the relationship between nominations in the confidence evaluation, failing to obtain sufficient semantic supplementation, thus leading to low performance. Summary of the Invention

[0005] This invention addresses the problems of existing technologies failing to fully utilize temporal context features, lacking modeling of global and long-term temporal relationships, lacking deep semantic feature capture in boundary detection, and not considering the relationship between nominees in confidence evaluation, resulting in unsatisfactory behavior detection performance. This invention provides a temporal behavior detection method based on context aggregation and boundary generation.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: A temporal behavior detection method based on context aggregation and boundary generation, comprising the following steps:

[0007] Step 1, Data Preparation: The datasets are derived from the general temporal behavior detection datasets ActivityNet 1.3 and THUMOS 14;

[0008] Step 2, Feature Encoding: A two-stream network is used to extract the spatiotemporal features of the input video, generating RGB features and optical flow features as the video feature sequence F;

[0009] Step 3: The video feature sequence F is input into the multi-path temporal context feature aggregation module, and is processed through:

[0010] The temporally global correlation structure constructs rich global contextual information on the entire video input features, generating temporally global feature A;

[0011] The long-range temporal feature correlation structure considers the aggregation of long-distance contextual features to generate long-range temporal feature B;

[0012] Local feature structure, through convolution operation, generates local feature C;

[0013] Finally, the three features are fused to obtain the multi-path temporal context feature D;

[0014] Step 4: Input the multi-path temporal context feature D into the multi-branch boundary generator, and generate two sets of boundary probability sequences P′ through the shallow boundary generator and the deep boundary generator in the multi-branch boundary generator, respectively. s ,P′ e and P″ s ,P″ e The final nomination boundary probability P is generated by fusion. s and P e ;

[0015] Step 5, Nomination Evaluation: Input the multi-path temporal context feature D from Step 3 into the nomination evaluation module to generate a nomination matching confidence map, providing confidence scores for densely distributed nomination boxes;

[0016] Step 6, Post-processing: Fuse the nomination boundary probabilities P generated in Step 4s P e The nominations generated in step 5 are matched with a confidence map to generate a final confidence score. Based on the final confidence score, the Soft-NMS algorithm is used to suppress redundant nominations.

[0017] Furthermore, in step 3 above:

[0018]

[0019] M s (F c )=σ(f([AvgPool(F c MaxPool(F) c (2)

[0020]

[0021] B = Activation(F) dilated-Causal (F)+Conv1D(F)) (4)

[0022] C = Conv(Conv(F)) (5)

[0023] Furthermore, step 4 above specifically involves: the shallow boundary generator consisting of two Conv1D convolutional layers, and using a sigmoid function to activate the output features to generate the probability P′ of the starting point. s and the probability of the endpoint P′ e The deep boundary generator is designed with an encoder-decoder U-shaped structure. It generates precise action boundaries by combining multi-scale features and fuses encoder and decoder features of the same scale through a series of dense convolutional blocks in a skip connection manner. Finally, the output features are activated using the sigmoid function to generate the probability P″ of the starting point. s "and the probability of the endpoint P" e ", fusion generates the final nomination boundary probability P s and P e The formula is (7):

[0024]

[0025] Furthermore, in step 5 above, for the input multi-path temporal context feature D, it is first converted into a boundary matching feature map, and then sampled to obtain a nomination matching feature map P. Then, adjacent nomination features are fused through the nomination relationship awareness module in the nomination evaluation, and finally the nomination matching confidence map is obtained by using the Sigmoid function, which is used for classification and regression respectively.

[0026] Furthermore, the nomination relationship-aware module is a variant of the residual module with a compressed excitation block. For the input feature map P, it first performs a global average pooling operation through the input feature layer, then uses two fully connected layers to capture the relationship between channels, sets a GELU layer between the two fully connected layers, and then performs a Sigmoid function to fix the values ​​in [0,1] to obtain the weight value of each channel of the input feature layer. Finally, it is superimposed with the input feature map P to obtain the final output feature map.

[0027] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0028] 1. This invention designs a multi-path temporal context feature aggregation module to effectively aggregate long-term and short-term temporal context information, thereby enhancing the contextual representation of behavioral events. Specifically: 1) Since global temporal information is related to the category of behavior, features suitable for behavior classification should be discriminative. Therefore, a global temporal correlation structure is designed to generate global features by constructing rich global context information on the entire video input features; 2) For long videos, long-term temporal modeling is essential. To enable boundary locations to focus on aggregating the information of their respective behavioral instances, and to simultaneously consider the aggregation of long-distance context, a long-range temporal feature correlation structure is designed to capture the long-distance context of the video and generate long-range temporal features; 3) For videos with shorter detection times, local information is more effective. A local feature structure is designed to generate local features through convolution operations. Finally, by combining the three types of features, the goal of effectively detecting both long-duration and short-duration videos is achieved.

[0029] 2. This invention designs a multi-branch temporal boundary detector, consisting of shallow and deep temporal boundary detectors. The shallow temporal boundary generator is composed of convolutional layers, while the deep temporal boundary generator is designed with a U-shaped structure. Integrating the deep temporal boundary generator solves the problem that the shallow temporal boundary generator only focuses on local mutations, resulting in high recall but low precision in the generated behavioral boundaries. Simultaneously, the complementary relationship between the two temporal boundary detectors optimizes the prediction results by determining whether two sets of start and end keypoints are simultaneously at a threshold or peak value. The start and end points are paired to generate nominations, thereby achieving high recall and high precision in capturing behavioral boundaries.

[0030] 3. The deep temporal boundary generator designed in this invention is a U-shaped encoder-decoder structure, comprising an encoder (downsampling), a decoder (upsampling), and skip connections. Since the temporal convolutional layers process features in each channel on an average basis, not all features obtained by the encoder are effective for boundary probability prediction. Therefore, an adaptive channel attention module is connected after each temporal convolutional layer in the encoder to enhance the ability to capture key features at the current temporal scale. Furthermore, some information is lost during the decoding process. To address this issue, we connect the multi-scale features of the decoder to capture fine-grained details and coarse-grained semantics, effectively extracting more information and thus generating accurate behavioral boundaries.

[0031] 4. This invention designs a nomination relationship awareness module, which models the nomination relationship through global correlation, aggregates the contextual information of adjacent nominations, and distinguishes the semantic information between sparse nominations. This effectively solves the problem that existing methods fuse adjacent nominations through convolution operations, resulting in a lack of distinctiveness and richness in the fused nominations, and enhances the expressiveness and robustness of the nomination context.

[0032] 5. Comprehensive experiments were conducted on two large datasets for general temporal behavior nomination generation and detection: ActivityNet1.3 and THUMOS14. This invention effectively improves the performance of temporal behavior nomination generation, and can generate behavior nominations with high accuracy and recall. Furthermore, when combined with existing action classifiers, it can also achieve good temporal behavior detection performance, which is superior to the current advanced methods. Attached Figure Description

[0033] Figure 1 This is the structure of the temporal behavior detection network MBGNet based on temporal context aggregation and multi-branch boundary generation in this embodiment of the invention;

[0034] Figure 2 This is a structural diagram of the multi-path temporal context feature aggregation module in an embodiment of the present invention;

[0035] Figure 3 This is a structural diagram of the multi-branch boundary generator in an embodiment of the present invention; Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0037] This invention discloses a temporal behavior detection method based on context aggregation and boundary generation to improve the performance of temporal behavior nomination generation and detection. First, a multi-path temporal context feature aggregation module is designed to effectively aggregate long-term and short-term temporal context information, thereby enhancing the contextual representation of behavioral events. Second, to generate accurate behavioral boundaries, a multi-branch temporal boundary detector is designed, utilizing the complementary relationship between two boundary detectors to optimize the prediction results. Furthermore, to accurately predict the confidence of densely distributed nominations, a nomination relationship awareness module is designed, using global correlation to model nomination relationships, enhancing the expressiveness and robustness of the nomination context.

[0038] See Figure 1 The present invention provides a temporal behavior detection method based on context aggregation and boundary generation, comprising the following steps:

[0039] Step 1: Data Preparation. Specifically, the ActivityNe1.3 dataset is a large dataset for general temporal action nomination generation and detection. Version 1.3 contains 19,994 undressed videos, 200 videos annotated with action subclasses, with each video containing an average of 1.41 action entities, for a total video duration of 849 hours. The dataset is divided into test, validation, and training sets in a 2:1:1 ratio. The THUMOS14 dataset contains 1010 validation videos and 1574 test videos. For action nomination or detection tasks, it includes 200 annotated undressed videos from the validation set and 213 annotated undressed videos from the test set, covering 20 action categories.

[0040] Step 2: Feature Encoding. Specifically, a two-stream network is used to extract the spatiotemporal features of the input video, generating RGB features and optical flow features, which are then used as the video sequence F.

[0041] Step 3: The video feature sequence F is input into the multi-path temporal context feature aggregation module, which generates global temporal feature A, long-range temporal feature B, and local feature C respectively. These three features are then fused to obtain the multi-path temporal context feature D. The specific implementation is as follows:

[0042] This invention designs a three-path temporal context feature aggregation module, referring to... Figure 2 .

[0043] Because temporal global information is related to the category of behavior, features suitable for action classification should be discriminative. Therefore, we design a temporally global correlation structure based on an attention mechanism to generate global features A by constructing rich global contextual information on the entire video input features. Specifically, for the original video features F, we first use average pooling and max pooling operations to aggregate the spatial information of the feature map, generating average pooling features and max pooling features respectively. Then, these two features are passed through a shared multilayer perceptron network, and the results are summed element-wise and then passed through the sigmoid function to obtain the channel attention feature map M. c (F). The original feature F is compared with the channel attention feature map M. c Multiplying (F) yields the channel attention feature F. c Then, for the channel attention feature F c Average pooling and max pooling operations are performed along the channel dimension to generate average pooling features and max pooling features respectively. These features are then concatenated and convolved through a standard convolutional layer to generate a spatial attention feature map M. s (F). Finally, the channel attention features F are simultaneously... c Spatial attention feature map M s (F) After multiplying, add it to the original feature F to obtain feature A containing global information.

[0044]

[0045] M s (F c )=σ(f([AvgPool(F c MaxPool(F) c (2)

[0046]

[0047] For long videos, long-term temporal modeling is essential. To enable boundary locations to focus on aggregating information about their respective action instances while simultaneously considering the aggregation of long-range context, we designed a long-range temporal feature correlation structure to generate the long-range contextual features B of the video. Specifically, we input the video feature sequence F into the long-range temporal feature correlation structure, expand the receptive field through dilated convolution, and introduce causal convolution on top of the dilated convolution to ensure that the receptive field covers every input in history and obtains the long-term memory of the time series. Furthermore, residual connections enable information to be transferred across layers, avoiding the problems of gradient vanishing and gradient exploding, thereby obtaining the long-range contextual features B of the video.

[0048] B = Activation(F) dilated-Causal(F)+Conv1D(F)) (4)

[0049] Since local information is more effective when detecting short-duration videos, this invention generates local features C by performing Conv1D operations on the original video features F.

[0050] C = Conv(Conv(F)) (5)

[0051] In summary, for the original video features F, the three-path temporal context feature aggregation module performs long-range temporal feature modeling for long videos and local feature capture for short videos, while simultaneously capturing global contextual temporal features, thus effectively aggregating long-term and short-term contextual information. These three features are then fused to obtain multi-path temporal contextual features, thereby achieving the goal of effectively detecting both long-duration and short-duration behaviors.

[0052] Step 4: Input the multi-path temporal context feature D into the multi-branch boundary generator, and generate two sets of boundary probability sequences P′ through the shallow boundary generator and the deep boundary generator, respectively. s ,P′ e and P″ s ",P" e "The prediction results are optimized by utilizing the complementary relationship between the two sets of boundary probability sequences. It is determined whether the two sets of start or end key points are simultaneously at the threshold or peak, and then fused to generate the final nomination boundary probability P." s and P e The specific implementation is as follows:

[0053] Design a multi-branch boundary generator. The multi-branch boundary generator consists of a shallow boundary generator and a deep boundary generator. The shallow boundary generator (SBG) consists of two Conv1D convolutional layers and uses a sigmoid function to activate the output features to generate the probability of starting points. The probability of the endpoint Where lv represents the number of time positions. It represents the probability that the nth time position is the starting point. It represents the probability that the nth time position is the end point of the action. Simultaneously, a deep boundary generator (DBG) with an encoder-decoder U-shaped structure was designed to address the problem that shallow boundary generators only focus on local mutations, resulting in high recall but low precision for the generated time boundaries. The structure of the deep boundary generator is shown below. Figure 3 As shown in (a).

[0054] The deep boundary generator generates precise action boundaries by combining multi-scale features and fuses encoder and decoder features of the same scale through a series of dense convolutional blocks in a skip connection manner, ensuring that all prior knowledge is accumulated and improving detection accuracy. Furthermore, an adaptive channel attention (ECA) module is connected after each temporal convolutional layer in the encoder to enhance the ability to capture key features at the current temporal scale. The detailed structure of the ECA module can be found in [link to ECA module]. Figure 3 As shown in (b), global average pooling is first performed to generate a tensor representing global spatial information, and then a one-dimensional convolutional layer is applied to encode the correlation of the channels.

[0055] Where, the encoder node x i,j The calculation formula is (6):

[0056]

[0057] Where H(*) represents the function corresponding to the basic convolutional unit, followed by the activation function, p(*) represents downsampling max-pooling, u(*) represents upsampling transposed convolution, and [*] represents the concatenation function.

[0058] The deep boundary generator ultimately uses the sigmoid function to activate the output features to generate the probability P″ of the starting point. s and the probability of the endpoint P″ e The final nomination boundary probability P is generated by fusion. s and P e The formula is (7):

[0059]

[0060] Step 5: Input the multi-path temporal context features into the nomination evaluation module to generate a nomination matching confidence map, providing confidence scores for densely distributed nomination boxes.

[0061] In the nomination evaluation module, a nomination relationship-aware module (PRAM) is designed to aggregate adjacent matches with different time scales and semantic densities. Since adjacent matches in the feature map have similar semantic information, the fused match lacks discriminativeness and richness. Therefore, the proposed nomination relationship-aware module PRAM is a variant of the residual module with compressed excitation blocks. It adaptively adjusts the weights of the feature map, improves the interdependence between channels, enables the network to perform feature recalibration, and enhances feature discriminative ability. For the input C*D*T feature map P, global average pooling is first performed through the input feature layer. Then, two fully connected (FC) layers are used to capture the relationships between channels, and a GELU layer is set between the two fully connected layers. Afterward, a Sigmoid function is performed to fix the value in [0,1]. The formula is as follows:

[0062]

[0063]

[0064] p′ c =F scale (p c ,s c ) = s c ·p c (10)

[0065] M = p c +p′ c (11)

[0066] The weight values ​​of each channel of the input feature layer are obtained, and finally superimposed with the input feature map P to obtain the final output feature map.

[0067] Step 6: Post-processing: Fuse the nomination boundary probabilities P generated in Step 4 s and P e The confidence map is matched with the nomination boundary generated in step 5 to generate the final confidence score, and the Soft-NMS algorithm is used to suppress redundant nominations based on the final confidence score.

[0068] The performance comparison results on ActivityNet 1.3 are shown in Table 1. Compared with the latest methods, the MBGNet proposed in this invention has a higher recall rate and AUC value, which proves the effectiveness of MBGNet on ActivityNet 1.3.

[0069] Table 1 compares the performance of the proposed method with other generation methods on the ActivityNet-1.3 dataset.

[0070]

[0071] Table 2 shows the performance comparison results of MBGNet on THUMOS14. Compared with the latest methods, the MBGNet proposed in this invention shows a higher recall rate, indicating that MBGNet significantly improves the performance of temporal behavior nomination generation on THUMOS14.

[0072] Table 2 compares the performance of the method of this invention with other generation methods on the THUMOS14 dataset.

[0073]

[0074] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for context-based aggregation and boundary generation based temporal behavior detection, the method comprising: Comprising the following steps: Step 1, data preparation: the data set is derived from the general time sequence behavior detection data set ActivityNet1.3 and THUMOS14 data set; Step 2, feature encoding: the two-stream network is used to extract the space-time features of the input video, and the RGB features and optical flow features are generated as the video feature sequence F; Step 3, the video feature sequence F is input into the multi-path time sequence context feature aggregation module, which respectively through: The time global correlation structure constructs rich global context information on the whole video input feature, and generates the time global feature A; The long-range time feature correlation structure considers the aggregation of long-distance context features, and generates the long-range time sequence feature B; The local feature structure generates the local feature C through convolution operation; Finally, the three features are fused to obtain the multi-path time sequence context feature D; Step 4, input the multi-path timing context feature D into the multi-branch boundary generator, respectively generate two groups of boundary probability sequences P' through the shallow boundary generator and the deep boundary generator in the multi-branch boundary generator s , P' e , and P" s " , P" e " , and fuse to generate the final nomination boundary probability P s and P e ; Step 5, nomination evaluation: the multi-path time sequence context feature D in step 3 is input into the nomination evaluation module to generate the nomination matching confidence map, which provides the confidence score for the dense distribution of nomination frame; Step 6, Post-processing: Fuse the nomination boundary probabilities P generated in Step 4 s , P e and the nomination match confidence maps generated in Step 5, generate final confidence scores, and suppress redundant nominations based on the final confidence scores using a Soft-NMS algorithm.

2. The method of claim 1, wherein: In the step 3, M s (F c ) = σ(f([AvgPool(F c ) ; MaxPool(F c )])) (2) B = Activation(F dilated-Causal (F) + Conv1D(F)) (4) C=Conv(Conv(F)) (5).

3. The method of claim 2, wherein: In step 4, the shallow boundary generator consists of two Conv1D convolution layers and uses sigmoid function to activate the output features to generate the probability P' of the start point s and the probability P' of the end point e ; the deep boundary generator is designed as an encoder-decoder U-shaped structure, which generates accurate action boundaries by combining multi-scale features, and fuses the encoder and decoder features of the same scale in a skip connection manner through a series of dense convolution blocks, and finally uses the sigmoid function to activate the output features to generate the probability P'' of the start point s and the probability P'' of the end point e , and fuses to generate the final nominated boundary probability P s and P e The formula is (7): 。 4. A method for detecting timing behavior based on context aggregation and boundary generation according to any one of claims 1-3, characterized in that: In the step 5, for the input multi-path time sequence context feature D, it is first converted into a boundary matching feature map, then sampled to obtain a nomination matching feature map P, then the adjacent nomination features are fused through the nomination relationship perception module in the nomination evaluation module, and finally the Sigmoid function is used to obtain the nomination matching confidence map, which is used for classification and regression respectively.

5. The method of claim 4, wherein: The nomination relationship perception module is a residual module variant with a compression excitation block. For the input feature map P, first perform global average pooling operation on the input feature layer, then use two fully connected layers to capture the relationship between channels, set GELU layer between the two fully connected layers, then perform Sigmoid to fix the value in [0, 1], obtain the weight value of each channel of the input feature layer, finally superimpose with the input feature map P to obtain the final output feature map.

Images