Video action detection method based on spatiotemporal information and video context information mining

Abstract

The present invention provides a video action detection method based on spatiotemporal information and video context information mining, which belongs to the field of computer vision and pattern recognition technology and accurately realizes the positioning and classification of action instances in videos; the specific steps of the inventive method are as follows: (1) video feature extraction; (2) construction of a network model of the video action detection method based on spatiotemporal information and video context information mining; (3) construction of an objective function of the video action detection method based on spatiotemporal information and video context information mining; (4) video action detection method based on spatiotemporal information and video context information mining; the present invention efficiently locates and classifies action instances in videos based on known data sets, and the method converges quickly.

Description

Technical Field

[0001] The present invention relates to a video action detection method based on mining spatiotemporal information and video context information, and belongs to the technical field of computer vision and pattern recognition. Background Art

[0002] In recent years, with the popularity of short videos, understanding video content has become increasingly important. As an important branch of video understanding, temporal action detection has attracted widespread attention from academia and industry. Temporal action detection is one of the most challenging tasks in the field of computer vision. It involves detecting the start and end times of action instances and predicting their class labels. The temporal action detection task is generally divided into two stages: video feature extraction and action localization and detection. Because video processing is very difficult, video is usually frame-sampled to transform the video problem into an image problem. One of the most important goals of the temporal action detection task is to accurately locate each action in the video. The generation of temporal action candidates is key to the performance of temporal action detection algorithms. Current methods for generating temporal action candidates can be roughly divided into three categories. One category generates temporal candidates based on sliding windows of multiple scales, collectively referred to as the Anchor-base mechanism in this article. Another category of Actionness methods is based on action probability curves to define the temporal boundaries of temporal action candidates. Another category of Anchor-free methods is to find multiple points in a video as the center points of the action, and then extend a certain time scale forward and a certain scale backward from the center point, splice the three parts, and finally form an anchor, which represents a candidate proposal. Compared with the first two methods, this method not only reduces the data redundancy caused by the large number of anchors generated by the Anchor-base method, but also has the same positioning boundary accuracy as Actionness. The present invention adopts the Anchor-free method. This method only needs to generate a proposal at each time position, which represents the sum of the distances from the current position to the start position and the end position. This method does not have a large number of proposals, which greatly reduces the complexity of the model.

[0003] Previous temporal action detection algorithms all use two branches, a locator and a classifier, to perform positioning and classification prediction respectively, and then fuse the two results. The disadvantages of this approach are obvious. If the classification or positioning is inaccurate, it will affect the final result. In order to solve this problem, the present invention proposes a structure that uses more accurate classification features and the original positioning features to fuse and then perform positioning. This structure can make the final positioning result more accurate. The present invention uses multiple modules to improve the accuracy of classification. Therefore, the positioning result predicted by the locator will be more accurate, and then the classification and positioning of the entire video action segment will be more accurate. Summary of the Invention

[0004] The present invention provides a video action detection method based on mining spatiotemporal information and video context information, which can more accurately classify and locate the entire video action segment.

[0005] To achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions:

[0006] A video action detection method based on mining spatiotemporal information and video context information includes the following steps:

[0007] 1) Video feature extraction:

[0008] The action video to be detected is subjected to feature extraction by the pre-trained I3D network to obtain the spatiotemporal feature sequence V of the entire video. Each video is represented by a feature sequence and input into the network model.

[0009] 2) Network model construction of video action detection method based on spatiotemporal information and video context information mining:

[0010] 201. Using the feature sequence in step 1) as input to the temporal information enhancement module and the channel information enhancement module;

[0011] 202. The features of the temporal information enhancement module and the channel information enhancement module are fused to obtain a feature sequence after spatiotemporal information enhancement and used as the input of the temporal context capture module and the semantic context capture module;

[0012] 203. The features generated by the temporal context information capture module and the semantic context capture module are fused with the enhanced spatiotemporal features to obtain a feature sequence for classification and positioning;

[0013] 204. Fusing the classification features with the positioning features to obtain the final positioning features, and obtaining the final classification results and positioning offsets through the classifier and the locator;

[0014] 3) Constructing the objective function of the video action detection method based on spatiotemporal information and video context information mining:

[0015] 301. Focalloss is used to constrain the categories of action instances and the positive samples;

[0016] 302. DIoUloss is used to locate action instances in videos to constrain the difference between the predicted action center point and the real action midline point, as well as the difference between the predicted action offset and the real action offset.

[0017] In the preferred embodiment of the video action detection method based on mining of spatiotemporal information and video context information, the overall constraint function formula in step 301 is as follows: , and The loss functions for classification and boundary regression are used to constrain the predicted action instance results. is the total number of positive samples, , is the predicted classification result, is the true label.

[0018] In the preferred embodiment of the video action detection method based on mining of spatiotemporal information and video context information, the function formula in step 302 is as follows: , is the loss function, is the true label offset, is the predicted offset, d is the longest distance between the true label offset and the predicted offset, It is a hyperfunction that controls the weight of the calculated time length of the predicted action segment and the real action segment.

[0019] In the preferred embodiment of the video action detection method based on mining spatiotemporal information and video context information, the characteristic formula of the temporal information enhancement module is as follows: , where F is the feature enhanced by temporal information, FC is the fully connected layer, is the activation function, For three different convolution.

[0020] The preferred solution of the video action detection method based on spatiotemporal information and video context information mining, the feature of channel information enhancement The formula is as follows: ,in is an activation function, an S-shaped curve and continuous, mapping the feature vector to the (0, 1) interval, is the normalization function, For three different , V is the video feature vector.

[0021] The preferred embodiment of the video action detection method based on mining spatiotemporal information and video context information, the feature formula after spatiotemporal information enhancement is as follows: The characteristics of F temporal information enhancement module, The features of channel information enhancement, V is the video feature vector.

[0022] Features for classification and localization The formula is as follows: , is the spatiotemporal enhancement feature, is the feature of temporal context information, is the feature of semantic context information;

[0023] The characteristic formula of temporal context information is as follows: , the characteristic formula of semantic context information is as follows: , , ,in is a tensor reshaping operation, T is the length of the feature sequence or the number of edge attention vertices, N is the number of similar frames for each vertex or the number of edges, and E is the connection between several frames similar to each frame in the model. is a convolutional layer consisting of three layers of 2D convolution, It is a feature sequence formed by edge attention consisting of multiple 2D convolutions, activation functions, normalization layers, and pooling layers.

[0024] The preferred solution of the video action detection method based on spatiotemporal information and video context information mining is a relatively accurate classification feature constrained by loss. Fusion with positioning features to obtain the final positioning features , , 1D convolution is used for dimensionality reduction after concatenating two features. For splicing operations, is deconvolution.

[0025] The present invention proposes a new temporal action detection structure. First, a pre-trained I3D network is used to extract features from the action video to be detected to obtain the spatiotemporal feature sequence of the action to be detected. The feature sequence is then used as the input of the temporal information enhancement module and the channel information enhancement module. The temporal information enhancement module captures the temporal information of the entire video, and at the same time enhances the temporal information of the action part, and more clearly separates the action features and non-action features of the feature sequence. The channel information enhancement module captures the action information inside the video features of each frame, suppresses the influence of the non-action area inside each frame at the pixel level, and enhances the significance of the foreground action information. The features of the two modules are then fused to obtain a feature sequence after spatiotemporal information enhancement and used as the input of the temporal context capture module and the semantic context capture module. The temporal context capture module mines the temporal context information of the feature sequence to make the action feature segments of the feature sequence more obvious, while suppressing the background segments. The semantic context capture module establishes a connection between the feature information of several frames in the feature sequence that are similar to the features of each frame. The present invention regards the information of each frame as a point and the connection between frames as an edge. At the same time, edge attention is added to the edge to make the connection between frames more accurate. Then, the information similar to the current frame is fused, and the entire feature sequence has rich semantic context information. Finally, the features generated by the temporal context information capture module and the semantic context capture module are fused with the enhanced spatiotemporal features to obtain a feature sequence for classification and positioning. A more accurate classification feature is obtained, and the more accurate classification feature is fused with the positioning feature to obtain the final positioning feature. Finally, the final classification result and positioning offset are obtained through the classifier and locator.

[0026] The advantages of the present invention are:

[0027] 1) The temporal information enhancement module captures the temporal information of the entire video while also enhancing the temporal information of the action portion, clearly separating the action and non-action features of the feature sequence. Furthermore, the channel information enhancement module captures the action information within each frame's features, suppressing the influence of non-action areas within each frame at the pixel level and enhancing the saliency of foreground action information.

[0028] 2) The temporal context capture module mines the temporal context of feature sequences, making the action feature segments more prominent while suppressing background segments. Furthermore, the semantic context capture module connects the feature information of several frames in the feature sequence that are similar to each frame. This method treats each frame's information as a point and the connections between frames as edges. Edge attention is then added to the edges to make the connections between frames more accurate. Information similar to the current frame is then fused, resulting in a rich semantic context for the entire feature sequence.

[0029] 3) The features generated by the temporal context information capture module and the semantic context capture module are fused with the enhanced spatiotemporal features to generate a feature sequence for classification and localization. This yields relatively accurate classification features, which are then fused with localization features to form the final localization features. Finally, the classifier and localizer achieve better classification results and localization offsets.

[0030] 4) After the samples are trained with the model, the optimal weights can be obtained, and these weights are applied to the model, so that the classifier and locator can obtain the best classification and positioning results. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0032] Figure 1 It is a structural diagram of the present invention;

[0033] Figure 2 is a flow chart of the present invention;

[0034] Figure 3 and Figure 4 Comparison of the performance of the classic action recognition algorithm and the proposed method on the THUMOS14 and ActivityNet1.3 datasets;

[0035] Figure 5 This is the result predicted by the present invention in an action segment. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0037] Example 1 Reference Figure 1 and Figure 2 A video action detection method based on spatiotemporal information and video context information mining includes the following steps:

[0038] 1) Video feature extraction:

[0039] The action video to be detected is extracted from the entire video through the pre-trained I3D network to obtain the spatiotemporal feature sequence of the entire video. Each video is represented by a feature sequence and input into the network model.

[0040] In this embodiment, the video feature vector is expressed as follows: , The length of the video after the video is extracted into frames, and the action label of each frame is also obtained , the starting offset , the ending offset , and the middle position between the start and end ,at this time ,C refers to the number of action instances in each video;

[0041] 2) Network model construction of video action detection method based on spatiotemporal information and video context information mining:

[0042] 201. The feature sequence from step 1) is used as input to the temporal information enhancement module and the channel information enhancement module. The temporal information enhancement module captures the temporal information of the entire video and enhances the temporal information of the action portion. The channel information enhancement module captures the action information within each frame of video features. At the pixel level, it suppresses the influence of non-action areas within each frame and enhances the saliency of foreground action information.

[0043] 202. The features of the temporal information enhancement module and the channel information enhancement module are fused to obtain a feature sequence with enhanced spatiotemporal information, which is then used as the input for the temporal context capture module and the semantic context capture module. The temporal context capture module mines the temporal context information of the feature sequence to make the action feature segments of the feature sequence more prominent while suppressing the background segments. The semantic context capture module establishes a connection between the feature information of several frames in the feature sequence that are similar to the features of each frame. The present invention regards the information of each frame as a point and the connection between frames as an edge. At the same time, edge attention is added to the edges to make the connection between frames more accurate.

[0044] 203. The features generated by the temporal context information capture module and the semantic context capture module are fused with the enhanced spatiotemporal features to obtain a feature sequence for classification and positioning;

[0045] 204. Fusing the classification features with the positioning features to obtain the final positioning features, and obtaining the final classification results and positioning offsets through the classifier and the locator;

[0046] 3) Constructing the objective function of the video action detection method based on spatiotemporal information and video context information mining:

[0047] 301. Focalloss is used to constrain the categories of action instances and the positive samples;

[0048] 302. We use DIoU loss to constrain the difference between the predicted action center and the true action midline, as well as the predicted action offset and the true action offset, to localize action instances in videos. Soft-NMS is used to process all predictions to suppress redundant proposals, enabling classification and localization of action instances in videos.

[0049] 4) Video action detection based on spatiotemporal information and video context information mining.

[0050] In this embodiment, As the input of the temporal information enhancement module, the feature information of the action segment is enhanced by first convolving the feature and then multiplying it with the original feature, and the information of the non-action segment is suppressed to obtain the feature of the temporal information enhancement module. The formula is as follows: , where F is the feature enhanced by temporal information, FC is the fully connected layer, is the activation function, For three different convolution.

[0051] In this embodiment, As the input of the temporal information enhancement module, the original features are first processed by the convolution operation, and then the features after the convolution operation are converted into dimensions, followed by multiplication operation, and finally the multiplied features are convolved, normalized and activated. At the same time, the obtained features are multiplied with the original features. In this way, the action part inside each frame feature can be weighted at the pixel level, making the action pixels of each frame feature more significant, and the non-action pixels of each frame feature will be affected less, so as to obtain the channel information enhanced features. The formula is as follows: ,in is an activation function, an S-shaped curve and continuous, mapping the feature vector to the (0, 1) interval, is the normalization function, For three different , V is the video feature vector.

[0052] In this embodiment, the original features and the features enhanced by the time series information are And the characteristics of channel information enhancement After weighted fusion, the feature formula after spatiotemporal information enhancement is obtained as follows: The characteristics of F temporal information enhancement module, The features of channel information enhancement, V is the video feature vector.

[0053] Temporal context capture module, the previous operation can not well classify and locate the situation where the action and background are adjacent. Therefore, the present invention uses spatiotemporal enhancement features As the input of the temporal context capture module, in order to reduce the temporal information loss caused by the feature sequence passing through the temporal information enhancement module and the channel information enhancement module, and also to enable the subsequent convolution operation to have more accurate temporal information, the present invention proposes to use self-attention to capture the temporal context. For long-term dependence, the accuracy of temporal information can be increased, and the self-attention mechanism increases the weight of the action segment, making the action feature segment of the feature sequence more obvious, while suppressing the background segment, and finally obtaining features with temporal context information. The feature formula of temporal context information is as follows: ,Semantic context capture module,Semantic context capture module also uses spatiotemporal enhancement features , as the input of this module, the relationship between the frames similar to each frame in the video is modeled, represented by E, E represents the relationship between the current frame and some other similar frame features, T represents the time of the feature sequence, and N represents the number of all frames similar to the current frame;

[0054] Using side attention to increase the weight of the connection between frames can better handle situations where action and background adjacent features are difficult to classify and locate, and obtain features with semantic context information. The feature formula of semantic context information is as follows: , , ,in is a tensor reshaping operation, T is the length of the feature sequence or the number of edge attention vertices, N is the number of similar frames for each vertex or the number of edges, and E is the connection between several frames similar to each frame in the model. is a convolutional layer consisting of three layers of 2D convolution, It is a feature sequence formed by edge attention consisting of multiple 2D convolutions, activation functions, normalization layers, and pooling layers.

[0055] In this embodiment, in order to capture more accurate video context information, the spatiotemporal enhancement feature Features with temporal context information and features with semantic context information Fusion is performed to obtain the final features used for classification and positioning , features for classification and localization The formula is as follows: , For spatiotemporal enhancement features, is the feature of temporal context information, is the feature of semantic context information; classification and positioning module, this embodiment Use a lightweight convolutional neural network to perform classification operations first to obtain classification features , classification is performed under the constraints of the loss function, so the classification features already contain relatively accurate classification information. In order to make the positioning results more accurate, the more accurate classification features constrained by the loss are Fusion with positioning features to obtain the final positioning features , , 1D convolution is used for dimensionality reduction after concatenating two features. For splicing operations, For deconvolution, according to the features and , through a lightweight convolution operation, the predicted classification , use another lightweight convolution to get the offset from each time t to the start and end .

[0056] In this embodiment, in the objective function of the network, first, Focalloss constrains the classification of action instances in the video to make the classification more accurate, and then the DIoUloss function reduces the difference between the action center point and the real action center point, as well as the difference between the predicted action offset and the real action offset, so that the predicted action boundary will be more accurate. Since the positioning feature is formed by the fusion of the classification feature and the positioning feature constrained by Focalloss, the more accurate the classification result, the more accurate the positioning result will be, thereby improving the accuracy of positioning. The total constraint function formula in step 301 is as follows: , and The loss functions for classification and boundary regression are used to constrain the predicted action instance results. is the total number of positive samples, , is the predicted classification result, is the true label.

[0057] The function formula in step 302 is as follows: , is the loss function, is the true label offset, is the true label offset, d is the longest distance between the true label offset and the predicted offset, It is a hyperfunction that controls the weight of the calculated time length of the predicted action segment and the real action segment.

[0058] The present invention generates video features through 1), and after steps 2) and 3), forms a final video action detection method model based on classification and positioning serial of feature encoder. The process of the model is: a feature sequence with enhanced spatiotemporal information is obtained through a temporal information enhancement module and a channel information enhancement module, and the feature sequence is input into a temporal context capture module and a semantic context capture module to obtain features with video context information. This feature is passed through a classifier to obtain a more accurate classification feature of each frame, and finally the classification feature and positioning feature are spliced. At the same time, the spliced ​​features are reduced in dimension through convolution, and the fused features are used for action positioning to obtain the final video classification and positioning results, which are the detection results of the video by the present invention. In the above process, the classification and positioning operations are constrained by the objective function to obtain more accurate action classification and results.

[0059] In order to verify the effectiveness of the present invention, the evaluation was carried out on the temporal action detection datasets THUMOS14 and ActivityNet1.3. The specific experimental settings are as follows: in the dataset THUMOS14, it contains 101 video categories and consists of four parts: training, validation, test and background sets. Each part includes 13320, 1010, 1574 and 2500 uncut videos, 200 videos in the validation set are used for training, and 213 videos in the test set are used for testing; in the dataset ActivityNet1.3, it is a large-scale action dataset, containing 200 activity classes and approximately 20,000 videos of more than 600 hours. The dataset is divided into three subsets: 10024 videos for training, 4926 for validation, and 5044 for testing. The dataset is divided into training, testing and validation at a ratio of 2:1:1. From Figure 2 and Figure 3 It can be seen that the video action detection method based on the classification and positioning sequence of the feature encoder proposed in the present invention has good detection performance.

[0060] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A video action detection method based on spatiotemporal information and video context information mining, characterized in that: The following steps are involved: 1) Video feature extraction: The action video to be detected is subjected to feature extraction by the pre-trained I3D network to obtain the spatiotemporal feature sequence V of the entire video. Each video is represented by a feature sequence and input into the network model. 2) Network model construction of video action detection method based on spatiotemporal information and video context information mining:

201. Using the feature sequence in step 1) as input to the temporal information enhancement module and the channel information enhancement module; 202. The features of the temporal information enhancement module and the channel information enhancement module are fused to obtain a feature sequence after spatiotemporal information enhancement and used as the input of the temporal context capture module and the semantic context capture module; 203. The features generated by the temporal context information capture module and the semantic context capture module are fused with the enhanced spatiotemporal features to obtain a feature sequence for classification and positioning; 204. Fusing the classification features with the positioning features to obtain the final positioning features, and obtaining the final classification results and positioning offsets through the classifier and the locator; 3) Constructing the objective function of the video action detection method based on spatiotemporal information and video context information mining:

301. Focalloss is used to constrain the categories of action instances and the positive samples; 302. DIoUloss is used to locate action instances in videos to constrain the difference between the predicted action center point and the real action midline point, as well as the difference between the predicted action offset and the real action offset.

2. The video action detection method based on mining spatiotemporal information and video context information according to claim 1 is characterized in that: The overall constraint function formula in step 301 is as follows: , and The loss functions for classification and boundary regression are used to constrain the predicted action instance results. is the total number of positive samples, , is the predicted classification result, is the true label.

3. The video action detection method based on mining spatiotemporal information and video context information according to claim 1 is characterized in that: The function formula in step 302 is as follows: , is the loss function, is the true label offset, is the true label offset, d is the longest distance between the true label offset and the predicted offset, It is a hyperfunction that controls the weight of the calculated time length of the predicted action segment and the real action segment.

4. The video action detection method based on spatiotemporal information and video context information mining according to claim 1 is characterized in that ,The characteristic formula of the timing information enhancement module is as follows: , where F is the feature enhanced by temporal information, FC is the fully connected layer, is the activation function, For three different convolution.

5. The video action detection method based on spatiotemporal information and video context information mining according to claim 1 is characterized in that ,The features of channel information enhancement The formula is as follows: ,in is an activation function, an S-shaped curve and continuous, mapping the feature vector to the (0, 1) interval, is the normalization function, For three different , V is the video feature vector.

6. The video action detection method based on spatiotemporal information and video context information mining according to claim 1 is characterized in that ,The characteristic formula after spatiotemporal information enhancement is as follows: The characteristics of F temporal information enhancement module, The features of channel information enhancement, V is the video feature vector; Features for classification and localization The formula is as follows: , is the spatiotemporal enhancement feature, is the feature of temporal context information, is the feature of semantic context information; The characteristic formula of temporal context information is as follows: , the characteristic formula of semantic context information is as follows: , , ,in is a tensor reshaping operation, T is the length of the feature sequence or the number of edge attention vertices, N is the number of similar frames for each vertex or the number of edges, and E is the connection between several frames similar to each frame in the model. is a convolutional layer consisting of three layers of 2D convolution, It is a feature sequence formed by edge attention consisting of multiple 2D convolutions, activation functions, normalization layers, and pooling layers.

7. The video action detection method based on spatiotemporal information and video context information mining according to claim 6 is characterized in that , more accurate classification features after loss constraint Fusion with positioning features to obtain the final positioning features , , 1D convolution is used for dimensionality reduction after concatenating two features. For splicing operation, is deconvolution.

Images