A temporal action detection method for videos

Abstract

The present invention provides a video-oriented temporal action detection method to solve the problem that the duration of action instances in actual data varies greatly. The present invention adopts the TAPP method to use the attention mechanism to focus on the distinctive part of the proposal and suppress the influence of the background on the proposal features; then constructs a temporal pyramid structure to convert a proposal feature sequence of any length into multiple fixed-length sequences without losing time information, and designs a multi-scale time function, which is applied to the temporal pyramid to generate the final proposal features, and constructs a temporal action proposal generation model and an action proposal classification model based on the TAPP method, which can effectively process temporal action proposals of indefinite length, thereby better completing the temporal action detection task in video data.

Description

Technical Field

[0001] The invention belongs to the field of video data processing and relates to a video-oriented temporal action detection method. Background Art

[0002] Temporal action detection is a challenging task in video understanding research, which requires simultaneously determining the temporal boundaries and action categories of target actions in untrimmed videos.

[0003] Most existing temporal action detection algorithms are divided into two stages: proposal generation and action classification. Since the duration of action instances in real data varies greatly, ranging from one second to hundreds of seconds, traditional action proposal generation methods output action proposals of varying lengths, and many proposals contain a large amount of background information. Studying a method that can effectively map action proposals of different durations to the same feature space has become the key to action proposal generation and classification tasks. Summary of the invention

[0004] The present invention provides a video-oriented temporal action detection method to solve the problem that the duration of action instances in actual data varies greatly.

[0005] The present invention is achieved through the following technical solutions.

[0006] A method for temporal action detection in video, comprising:

[0007] Step 1: Use the two-stream network to extract features from the original video to obtain a video feature sequence;

[0008] Step 2: Input the video feature sequence into the TAPP-based temporal action proposal generation model to predict the start probability and end probability of all time domain positions in the video feature sequence, thereby predicting the confidence score and completeness score of each proposal;

[0009] Step 3: Multiply the start probability, end probability, confidence score, and completeness score to obtain the final score of each proposal. Sort all proposals from large to small according to the final scores, and select the top N proposals as the final proposals.

[0010] Step 4: Input the final proposal and the video feature sequence obtained in step 1 into a proposal classification model based on TAPP to obtain a classification score, a completeness score and a boundary regressor for each proposal;

[0011] Step 5: multiply the classification scores of each proposal as the final classification score, and adjust the proposal boundary according to the boundary regressor; determine which action the proposal belongs to according to the final classification score, and use the adjusted proposal boundary as the start and end time of the action, thereby detecting the action in the video.

[0012] Beneficial effects of the present invention:

[0013] The present invention adopts the TAPP method and uses the attention mechanism to focus on the distinctive parts of the proposal and suppress the influence of the background on the proposal features; then a temporal pyramid structure is constructed to convert proposal feature sequences of arbitrary length into multiple fixed-length sequences without losing temporal information. At the same time, a multi-scale time function is designed and applied to the temporal pyramid to generate the final proposal features. Based on the TAPP method, a temporal action proposal generation model and an action proposal classification model are constructed, which can effectively process temporal action proposals of indefinite length, thereby better completing the temporal action detection task in video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 The present invention is a flow chart of a method for detecting temporal motion in video. DETAILED DESCRIPTION

[0015] The present invention will be further described below in conjunction with the accompanying drawings.

[0016] like Figure 1 As shown, a video-oriented temporal action detection method of this specific implementation method specifically includes the following steps:

[0017] Step 1: Use the two-stream network to extract features from the original video to obtain a video feature sequence;

[0018] Step 2: Input the video feature sequence into a temporal action proposal generation model based on TAPP (Temporal Attention-Pyramid Pooling), predict the start probability and end probability of all temporal positions in the video feature sequence, and thus predict the confidence score and completeness score of each proposal;

[0019] In the prior art, the BMN (Boundary-matching Network) model is an end-to-end temporal action proposal generation method, which can show good performance on mainstream temporal action detection datasets. In this embodiment, it is used as the basic structure and further improved, and it is combined with TAPP to establish a new BMN-TAPP model, that is, a temporal action proposal generation model based on TAPP.

[0020] Therefore, in this embodiment, the TAPP-based temporal action proposal generation model includes a feature enhancement module, a temporal evaluation module, and a proposal evaluation module; wherein:

[0021] The feature enhancement module uses a residual convolution block with a two-layer bidirectional LSTM to perform superposition processing on the video feature sequence, and obtains a bidirectional semantic relationship between features through the two-layer bidirectional LSTM to enhance the features;

[0022] The time domain evaluation module is used to predict the start and end probabilities of all time domain positions in the video feature sequence after superposition processing; in this embodiment, the time domain evaluation module includes two CovNets, each of which is composed of a one-dimensional convolution;

[0023] The proposal evaluation module first uses an exhaustive method to generate candidate proposals, and then uses the TAPP method to model the feature sequence fragments corresponding to each candidate proposal of different lengths into proposal features with the same feature dimension, and arranges the proposal features in a boundary matching feature map according to the length and starting position of the candidate proposal, and predicts the confidence score and completeness score of each candidate proposal.

[0024] In this embodiment, the candidate proposal refers to a preliminary obtained time segment that may contain an action.

[0025] [1,2],[1,3]…[1,n],[2,3],…[2,n],…[n-1,n], where n is the length of the video feature sequence.

[0026] The implementation principle of the feature enhancement module and the proposal evaluation module is as follows: use residual convolution blocks and two-layer bidirectional LSTM to process video feature sequences, and use the bidirectional relationship between frame features to capture long-term past and future information. Given the input feature sequence Fi∈RT×C, the feature enhancement module uses a residual module composed of two convolutional layers to further extract the semantic information of the frame, and then inputs the feature sequence Fu∈RT×Cu into the two-layer bidirectional LSTM through the residual convolution module. The hidden state sequences from the forward and backward LSTMs are concentrated on the channel dimension as the final feature sequence.

[0027] Step 3: Multiply the start probability, end probability, confidence score, and completeness score to obtain the final score of each proposal. Sort all proposals from large to small according to the final scores, and select the first N proposals as the final proposals. In this embodiment, N is 200.

[0028] Step 4: Input the final proposal and the video feature sequence obtained in step 1 into the proposal classification model based on TAPP.

[0029] In this embodiment, the TAPP-based proposal classification model includes a feature enhancement module, a TAPP module, and a classification module;

[0030] The feature enhancement module moves the boundary of the final proposal to both sides to obtain an extended proposal; in this embodiment, the length of the extended proposal is set to twice the length of the corresponding final proposal, so as to improve the quality of the video features;

[0031] The TAPP module models the feature sequences corresponding to the final proposal and the extended proposal into feature vectors respectively, and inputs them into the classification module for classification;

[0032] The classification module includes two parallel sub-networks consisting of two fully connected layers. The features corresponding to the final proposal and the extended proposal are respectively input into the two sub-networks, and the classification score, integrity score and boundary regressor of each proposal are output. In this embodiment, the classification module uses an integrity classifier and an action classifier to predict the label of the proposal; wherein a fully connected layer with a softmax operation is used as the action classifier, and a binary classifier corresponding to each action class is used as the integrity classifier.

[0033] Step 5: multiply the classification scores of each proposal as the final classification score, and adjust the proposal boundary according to the boundary regressor; determine which action the proposal belongs to according to the final classification score, and use the adjusted proposal boundary as the start and end time of the action, thereby detecting the action in the video.

[0034] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for temporal action detection in video, characterized in that: The following steps are involved: Step 1: Use the two-stream network to extract features from the original video to obtain a video feature sequence; Step 2: Input the video feature sequence into a temporal action proposal generation model based on TAPP, where TAPP is Temporal Attention-Pyramid Pooling; predict the start probability and end probability of all temporal positions in the video feature sequence, thereby predicting the confidence score and completeness score of each proposal; Step 3: Multiply the start probability, end probability, confidence score, and completeness score to obtain the final score of each proposal. Sort all proposals from large to small according to the final scores, and select the top N proposals as the final proposals. Step 4: Input the final proposal and the video feature sequence obtained in step 1 into a proposal classification model based on TAPP to obtain a classification score, a completeness score and a boundary regressor for each proposal; Step 5: multiply the classification scores of each proposal as the final classification score, and adjust the proposal boundary according to the boundary regressor; determine which action the proposal belongs to according to the final classification score, and use the adjusted proposal boundary as the start and end time of the action, thereby detecting the action in the video.

2. A method for video-oriented temporal action detection as claimed in claim 1, characterized in that: The TAPP-based temporal action proposal generation model includes a feature enhancement module, a temporal evaluation module, and a proposal evaluation module; wherein: The feature enhancement module uses a residual convolution block with a two-layer bidirectional LSTM to perform superposition processing on the video feature sequence, and obtains a bidirectional semantic relationship between features through the two-layer bidirectional LSTM to enhance the features; The time domain evaluation module is used to predict the start and end probabilities of all time domain positions in the video feature sequence after superposition processing; The proposal evaluation module first uses an exhaustive method to generate candidate proposals, and then uses the TAPP method to model the feature sequence fragments corresponding to each candidate proposal of different lengths into proposal features with the same feature dimension, and arranges the proposal features in a boundary matching feature map according to the length and starting position of the candidate proposal, and predicts the confidence score and completeness score of each candidate proposal.

3. A method for video-oriented temporal action detection as claimed in claim 2, characterized in that: The time domain evaluation module includes two CovNets, each of which is composed of one-dimensional convolution.

4. A video-oriented temporal action detection method as claimed in claim 2 or 3, characterized in that: The candidate proposal refers to a preliminary obtained time segment that may contain an action [1,2],[1,3],…[1,n],[2,3],…[2,n],…[n-1,n], where n is the length of the video feature sequence.

5. A method for video-oriented temporal action detection as claimed in claim 1, characterized in that: N is selected as 200.

6. A method for video-oriented temporal motion detection as claimed in claim 1, 2 or 3, characterized in that: The TAPP-based proposal classification model includes a feature enhancement module, a TAPP module, and a classification module; The feature enhancement module moves the boundary of the final proposal to both sides to obtain an extended proposal; The TAPP module models the feature sequences corresponding to the final proposal and the extended proposal into feature vectors respectively, and inputs them into the classification module for classification; The classification module consists of two double-layer fully connected sub-networks in parallel. The features corresponding to the final proposal and the extended proposal are respectively input into the two sub-networks, and the classification score, completeness score and boundary regressor of each proposal are output.

7. A method for video-oriented temporal action detection as claimed in claim 6, characterized in that: The length of the extended proposal is set to be twice that of the corresponding final proposal.

8. A method for video-oriented temporal motion detection as claimed in claim 6 or 7, characterized in that: The classification module uses an integrity classifier and an action classifier to predict the label of the proposal; wherein a fully connected layer with a softmax operation is used as the action classifier, and a binary classifier corresponding to each action class is used as the integrity classifier.

Images