An online end-to-end spatio-temporal action detection method and detector

Abstract

The application discloses an online end-to-end spatio-temporal action detection method and a detector, which directly obtains corresponding spatio-temporal action detection results from input video sequences without an additional post-processing process. The application constructs a detection network, and the network configuration comprises a feature extraction network, a feature cache area, a center point prediction branch, an enclosing frame regression branch and a sequence matching branch. High-dimensional abstract features of video frames are extracted through the feature extraction network, the video frame features are cached by using the feature cache module, and after sampling, the feature sequences are respectively transmitted into the action center point prediction branch, the human body enclosing frame regression branch and the sequence matching branch, so that the action instance center point and the human body action detection frame are obtained, and the spatio-temporal action detection result is formed in combination with the sequence matching. The application effectively utilizes the time sequence information contained in the video for action recognition, avoids repeated calculation, improves the detection efficiency, embodies the robustness and high efficiency in the video spatio-temporal detection task, and has strong portability and expansibility.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer software, and relates to a spatio-temporal action detection technology, in particular to an online end-to-end spatio-temporal action detection method and detector. BACKGROUND

[0002] Spatio-temporal action detection is a hot research topic in recent years, and its goal is to detect all action instances appearing in a video, distinguish the action categories, and locate the occurrence region of the action in time and space. Spatio-temporal detection is a very important research task in the field of computer vision, and has a wide range of application scenarios in real-world scenarios, such as pedestrian action detection in video monitoring scenarios and action detection of athletes in sports scenarios. The current mainstream solution is to use a detector to generate single-frame detection results on each video image independently, and then perform matching and splicing of all detection results in time sequence through a post-processing stage, in which a target tracking or dynamic programming based technology is often used to complete the matching process. Such methods often fail to utilize the unique time sequence information in the video, resulting in poor detection results, or use 3D convolutional neural networks for feature extraction, which brings high computational burden. At the same time, the post-processing stage based on dynamic programming and target tracking often only relies on the size of the overlapping region to match the candidate regions, which not only leads to poor connection effect, but also results in the inability to perform end-to-end reasoning.

[0003] In order to solve the above problems, many work solutions in recent years input a small video sequence, such as 5-8 frames, and complete continuous action detection within this period of time, and then use the overlapping regions in time sequence and space to connect, to obtain the final spatio-temporal action detection result. These solutions make good use of the unique spatio-temporal features in the video frames, and additionally use the overlapping regions in time sequence for matching and connection, further improving the detection effect. However, these methods involve a large amount of repeated and time-consuming calculations, and the entire framework cannot be end-to-end reasoned. With the growth of a single video, the post-processing event of such methods often occupies a large overhead, thereby limiting the application of the spatio-temporal detection framework in real-world scenarios. SUMMARY

[0004] The problem to be solved by the present application is that the existing spatio-temporal detection methods usually follow a multi-stage detection-post-processing mode, thereby resulting in the inability of the entire framework to complete end-to-end reasoning, and the inefficiency and performance deficiency in utilizing the unique time sequence information of the video.

[0005] The technical scheme of the present application is: an online end-to-end spatio-temporal action detection method, a detection network is constructed to directly detect an input video sequence to output a spatio-temporal action detection result, the detection network comprises a feature extraction network, a feature cache area, a center point prediction branch, an enclosing box regression branch and a sequence matching branch, the feature extraction network extracts high-dimensional abstract features of video frames, the feature cache module caches the extracted video frame features to obtain a feature sequence, the feature sequence is input into the action center point prediction branch, the target enclosing box regression branch and the sequence matching branch respectively, the action instance center point and the target action detection box are obtained by the action center point prediction branch and the target enclosing box regression branch, the action detection box is obtained from the action instance center point and the target action detection box, the action detection box is input into the sequence matching branch, the current frame action detection result is matched with the last frame detection result in time sequence, and the continuous spatio-temporal detection result is generated by splicing.

[0006] As a preferred mode, the detection network is specifically as follows:

[0007] 1) Feature extraction network: a 2D feature extraction network is adopted to extract features of the input image sequence, the video frame obtains a high-dimensional feature map, i.e. a video frame feature, through the feature extraction network, the feature extraction network comprises a down-sampling layer and an up-sampling layer, the video frame feature comprises two types, the feature after the up-sampling layer is taken as a high-resolution feature, and only the down-sampling layer is used, and the feature without the up-sampling layer is a low-resolution feature;

[0008] 2) Feature cache area: the feature cache area caches at most 64 video frame features, and the newly generated features are put into the buffer area to replace the earliest cached features in the order of the input video frame;

[0009] 3) Center point prediction branch: comprising a 2D branch and a 3D branch, the 2D branch obtains the position of the motion instance center point through the convolution processing of the high-resolution feature of the current frame, the 3D branch uniformly samples 8 frames from the low-resolution feature sequence in the feature cache area as an input feature sequence, performs spatio-temporal feature fusion through a stacked R(2+1)D module, i.e. a 3D sub-network, obtains an action category preference prediction result through a full connection layer, combines the results of the 2D branch and the 3D branch to obtain the action center point prediction result, including positioning and classification;

[0010] 4) Enclosing box regression branch: the enclosing box of the motion instance center point of the current frame is obtained from the high-resolution feature of the current frame through the enclosing box regression branch, the positioning and classification obtained by the center point prediction branch are combined to form the action instance detection result of the current frame, including the action category and the action enclosing box;

[0011] 5) Sequence matching branch: match the current frame action detection result with the previous frame detection result in time sequence, and generate continuous spatio-temporal detection result through adaptive matching splicing, and accordingly generate the spatio-temporal action detection result of the entire video sequence after the entire video sequence processing is completed.

[0012] Further, the construction of the detection network comprises the following steps:

[0013] 1) Generating training samples: taking 64 consecutive frames of a video as a sampling period, and sampling 8 frames in each period as an input video sequence, and the video frame is an RGB image;

[0014] 2) Configuring the detection network;

[0015] 3) Training stage: using Focal Loss to supervise the center point prediction branch of the detection network, using L1 Loss to supervise the bounding box regression branch, using Contrastive Loss to supervise the sequence matching branch, using AdamW optimizer, and updating the network parameters through the back propagation algorithm until the iteration number is reached;

[0016] 4) Test stage: inputting the video in the test set into the detection network to obtain the spatio-temporal action detection result in the entire video to verify the effect of the detection network.

[0017] The application also proposes an online end-to-end spatio-temporal action detector, comprising a data storage device, and a computer program is configured in the data storage device, and the computer program is implemented when the detection method is executed.

[0018] The application designs a brand-new online end-to-end spatio-temporal action detection framework, uses a 2D feature extraction network to extract single-frame features from the original video, based on the invariance of the convolutional neural network, uses a feature cache area to cache the extracted video features, and only needs to put the current frame features into the cache area each time, so that the detection efficiency is improved; for action center point detection, the application sends high-dimensional features into a 2D convolutional layer to obtain accurate action positioning results and rough action classification results, and sends continuous multi-frame low-dimensional features into a 3D sub-network to obtain corresponding action category prediction, and the combination of the two can realize accurate action center point positioning and action category classification prediction results; on this basis, the application performs action bounding box regression near each predicted center point to obtain single-frame action detection results; finally, the sequence matching branch is used to match the current frame detection result with the previous detection result, and the action detection process can be completed in the video stream processing process, and the action detection result of the entire video can be directly obtained after the video processing is completed, without the need for a post-processing stage.

[0019] Compared with the prior art, the application has the following advantages:

[0020] The application proposes an online end-to-end spatio-temporal action detection method and detector, which can gradually complete spatio-temporal positioning and classification tasks in the stream processing process, and compared with the previous two-stage method, the post-processing process is saved, and it is more simple and efficient.

[0021] The application extracts high-dimensional features by using a 2D feature extraction network and realizes feature reuse by using a feature cache area, and an action recognition task is realized by using a lightweight 3D subnetwork, so that the calculation complexity is reduced on the premise of utilizing the time sequence information specific to the video.

[0022] The application has good robustness and high efficiency in the video action spatio-temporal positioning task, is more simple and efficient than the mainstream spatio-temporal detection framework based on detection-tracking at present, and has strong expansibility and portability. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 It is a schematic diagram of the overall operation process of the application.

[0024] Figure 2 It is a schematic diagram of the detection framework overview of the application.

[0025] Figure 3 It is a schematic diagram of the detection network of the application.

[0026] Figure 4 It is the construction mode of the feature extraction network and the feature cache area of the application.

[0027] Figure 5 It is a schematic diagram of the center point prediction branch of the application.

[0028] Figure 6 It is a schematic diagram of the bounding box regression branch of the application.

[0029] Figure 7 It is a schematic diagram of the sequence matching branch of the application.

[0030] Figure 8 It is the visualization result of the application, (a) is the heat map of each action category obtained by the 2D branch, (b) is the prediction score of each action category by the 3D branch, (c) is the heat map of each action category obtained by fusing the 2D branch and the 3D branch, and (d) shows the detection result and the real result. DETAILED DESCRIPTION

[0031] The application proposes a new spatio-temporal action detection framework, the whole detection process can be inferred on the online end-to-end, and has high efficiency and usability, which can meet the real-time scene, called OEEDet. Specifically, the application defines several key principles to design the detection framework. First, in order to process continuous video streams, the detection process should be carried out in an online manner, and action detection should be carried out as soon as a new video frame is received, without obtaining any side information from future frames; second, in order to ensure the detection efficiency and detection accuracy, the whole spatio-temporal detection framework should be unified as an end-to-end learnable framework, without any post-processing stage, and the separate post-processing connection step is avoided to realize efficient processing and joint optimization; finally, in order to improve the accuracy of action recognition, the detector should make full use of long-term time sequence information to help identify the action category.

[0032] The online end-to-end spatio-temporal action detection method and detector of the application are proved to have high accuracy and real-time performance through training and testing on UCF101-24 and JHMDB spatio-temporal detection data sets, and Python 3 programming language and Pytorch 1.6.0 deep learning framework are used in the experiment.

[0033] Figure 1 The whole detection process of the application is described, Figure 2 The detection framework of the application is an overview, the application uses a feature cache area to save the high-dimensional video frame features extracted by the 2D feature extraction network, avoids repeated calculation and improves the detection efficiency. The cache area includes high-resolution current frame features and last frame features, and low-resolution continuous 64 frame sequence features. In the center point prediction branch, the 2D convolution layer is used to abstract the current frame features to obtain the action center point positioning and action rough classification results, and the continuous 64 frame sequence spatio-temporal features are sent into the 3D sub-network to obtain the action category preference, and the two are combined to detect the action center point; the bounding box regression branch predicts the action bounding box near each action center point to generate single-frame action detection results; the sequence matching branch matches the prediction results of the current frame with the previous prediction results and generates spatio-temporal action detection results along the time sequence dimension. In the detection process, the matching and connection of the spatio-temporal action detection results can be completed step by step, and when the video processing is completed, any post-processing stage is not required. The specific implementation is introduced as follows.

[0034] 1) Generate training samples: the detector only detects the current frame each time, and does not depend on the subsequent unknown video sequence. In order to make full use of the time sequence information in the video, 8 frames are uniformly sampled in the previous 64 continuous frames as the input video sequence. In order to ensure the real-time performance of the whole framework, we only use RGB images as network input without using time-consuming optical flow information.

[0035] The videos in UCF101-24 and JHMDB datasets are cropped into video frames and saved on the hard disk. To ensure the efficiency of real-time detection, only the original RGB images are used as network input without additional optical flow information. In the training process, we use the labeled videos for training, randomly sample 64 consecutive frames each time and uniformly sample 8 frames from them as training samples. For each training sample video segment, the corresponding K-frame RGB image sequence is taken as input and denoted as I. Due to the limited training samples and the large amount of data required by the deep model as network supervision signal, in order to train the model more effectively, data augmentation processing is performed during the training process, such as random expansion, color variation, random horizontal flip, random cropping, color jittering, etc. to increase the size of the data by data augmentation. The resolution of the input is uniformly adjusted to 320*224 by bilinear interpolation, the input channel number is 3, representing the RGB three channels, the obtained video frame is subtracted from the COCO dataset RGB three channel mean and divided by the three channel standard deviation to realize normalization, and finally converted to Tensor form, processed in batches and shuffled data loading order. The distributed training framework is used to complete the training process on 8 NVIDIA TESLA graphics cards.

[0036] 2) Configure the detection network, and the overall detection network is as shown in Figure 3

[0037] 2.1) Feature extraction network and feature buffer area: use the 2D feature extraction network DLA-34 to perform high-dimensional feature extraction on the input video frame I in 1) and place it in the feature buffer area. Specifically, the video frame feature is where W is the width of the input picture resolution, H is the height of the input picture resolution, R is the down-sampling rate of feature extraction, and C is the number of feature channels; the feature extraction network includes a down-sampling layer and an up-sampling layer, corresponding to R=4, C=64 and R=32, C=512 respectively. The feature after the up-sampling layer is taken as a high-resolution feature, and only the down-sampling layer is used without the up-sampling layer. The feature is a low-resolution feature, and both types of features are saved in the feature buffer area. In the detection process, high-resolution and low-resolution features are extracted using the feature extraction network when the video frame arrives, and are placed in the feature buffer area.

[0038] ​The feature extraction network extracts features from the input image sequence, resulting in a high-dimensional feature map with 64 channels and a resolution reduced to one-quarter of the original. The feature buffer of this invention stores previously generated high-dimensional video frame features, avoiding redundant computation and thus improving detection efficiency. Unlike current mainstream action recognition models that directly use 3D feature extraction networks to process the original video sequence, this invention only uses a 2D feature extraction network to process the arriving original video frames and buffers the extracted video frame features. In subsequent modules, the buffer features are sampled before the 3D sub-network is used to process the high-dimensional features, thus efficiently utilizing the temporal information contained in the video.

[0039] To further improve efficiency, the feature buffer stores two different types of high-dimensional features: high-resolution and low-resolution. The high-resolution features of the current frame and the previous frame are used to accurately locate the center point of the action and the corresponding bounding box, with a resolution of 80*56 and 64 feature channels. The low-resolution features of the 64 consecutive frames in the buffer are used to distinguish the action categories, with a resolution of 10*7 and 512 feature channels.

[0040] like Figure 4 As shown, the feature buffer can cache already computed 2D feature maps, thus avoiding redundant computation and improving detection efficiency. During training, 8 frames are uniformly sampled as input within a continuous 64-frame range. However, during testing, since the network does not need to be updated, previously processed video frame features can be cached in the feature buffer. Only the currently arriving video frame is used for feature extraction and stored in the buffer. Features from 8 frames are uniformly sampled in the buffer as low-resolution spatiotemporal features, and high-resolution feature maps of the current and previous frames are obtained. For the feature extraction network, we use the 2D framework DLA-34. Before training begins, its pre-trained model parameters on the COCO object detection dataset are loaded, and an upsampling process is added to facilitate the collection of high-resolution and low-resolution feature maps for subsequent action center point prediction.

[0041] 2.2) Center point prediction branch: The center point prediction branch generates the location of the action center point and the corresponding action category in the current frame. The continuous low-resolution features contain long temporal and spatial information, which is helpful for the classification of action category. However, due to the low resolution, they cannot cover enough spatial positioning information. On the contrary, the high-resolution features of the current frame contain fine-grained spatial features, which can be easily used to estimate the specific location of the action center point. On this basis, our center point prediction branch simultaneously utilizes the fine spatial detail information of the high-resolution features and the continuous temporal feature information of the low-resolution features to complete the positioning of the action center point and the estimation of the action category. Specifically, the branch consists of a 3D path and a 2D path, which perform action recognition and center estimation, respectively. Unlike conventional representation methods based on 2D images or based on 3D video sequences, the present application adopts a hybrid representation method of 2D and 3D features, which is more efficient than methods that completely rely on 3D video sequences and more accurate and effective than methods based on 2D images, and more effectively realizes the modeling of action instances on video frames.

[0042] As shown in detail in Figure 5 , the center point prediction branch is divided into a 2D branch and a 3D branch. The 3D branch uniformly samples 8 frames from the 64-frame low-resolution feature sequence in the feature buffer and concatenates them along the channel to form a feature map F ST of W / 32*H / 32*(512*8), where W and H represent the original input image size, W=320, H=224, which is then input into a 3D subnetwork containing 4 stacked R(2+1)D spatio-temporal convolution blocks for action recognition. The processing result is subjected to a global average pooling layer and a fully connected layer to obtain a one-dimensional action category prediction vector V, whose score represents the confidence of each action category. When sampling features from the feature buffer, the 3D branch does not need to wait for the buffer to be full. When the number of frames in the buffer is less than 64, the existing frame features are copied and uniformly filled to ensure that the 8 frames sampled as input feature sequences are input into the 3D subnetwork.

[0043] The 3D branch only focuses on the subtask of action recognition, and the subtask of action positioning is completed by the 2D branch. Since the spatio-temporal features F ST constructed by the 3D branch span a long time interval, the action instance can have a large spatial displacement, so the spatio-temporal features cannot accurately describe the location of the action. In addition, unlike the action classification task, the action positioning requires high-resolution feature maps with more detailed information. Therefore, we use the high-resolution feature map F HR (H / 4*W / 4*64) of the current frame as the input of the 2D branch. The 2D branch subjects the input feature map F HR to a 3*3 convolution layer Conv L1Feature fusion is performed to obtain a fused feature map of H / 4*W / 4*256, which is then subjected to a ReLu activation layer, and finally transmitted to a 1*1 convolution layer Conv L2 to obtain a feature map of H / 4*W / 4*C (C represents the number of action categories), and a final heat map is obtained through a Sigmoid activation layer 24 in UCF101-24 and 21 in JHMDB dataset.

[0044] As a preferred mode, the 2D branch fuses the high-resolution features of the previous frame and the current frame as input, and the high-resolution feature map F t of the current frame and the high-resolution feature map F t-1 of the previous frame are mixed to obtain W / 4*H / 4*128, which is subjected to a 3*3 convolution layer Conv L1 Feature fusion is performed to obtain a fused feature map of W / 4*H / 4*256, which is then subjected to a ReLu activation layer, and finally transmitted to a 1*1 convolution layer Conv L2 to obtain a feature map of W / 4*H / 4*C, and a final heat map is obtained through a Sigmoid activation layer

[0045] Finally, the results of the 2D branch and the 3D branch are combined to obtain the action center point prediction. The present application adopts a simple and effective scheme: first, the one-dimensional action category prediction vector V generated by the 3D branch is expanded along the W-H axis to generate a 2D prediction matrix M, and then M and the heat map are directly added together, so that the long sequence action classification information is incorporated into the center point positioning estimation heat map with position detail features, thereby helping to suppress inaccurate classification results in the middle. Facilitating accurate positioning and classification of the center point of the action instance.

[0046] During training, the present application supervises the fusion results of the 2D branch and the 3D branch, and specifically uses a Focal Loss to set a loss function item l center , and a heat map supervision signal is generated by a Gaussian kernel function Through the Gaussian kernel function, the points near the center point are assigned values, and the closer the position to the center point, the closer the response value to 1.

[0047] In the inference stage, a 3*3 maximum pooling operation is used to filter out the point with the largest local response from the generated heat map, and finally 10 points with the highest scores are selected as candidate center points, representing the category prediction of the action instance and the position prediction of the corresponding center point.

[0048] In the implementation of the present application, we found that combining the current frame features with the high-resolution features of the previous frame will help capture short-term motion information and thus better estimate the center point heat map. Therefore, the 2D branch of the present application preferably inputs the high-resolution feature maps of the adjacent two frames at the same time.

[0049] 2.3) bounding box regression branch: Since the spatial position regression of the detection box does not need to use the timing information, the present application uses the high-resolution feature map of the current frame as the branch input, as shown in Figure 6 The detection box regression branch only inputs the high-resolution feature map of the current frame, performs spatial feature fusion through a 3*3 convolution to obtain a feature map of W / 4*H / 4*256, then passes through a ReLu activation layer, and then is transmitted to a 1*1 convolution to obtain a detection box length-width prediction result of W / 4*H / 4*2. The length and width of the detection box are read from the center point position of the action instance of the current frame obtained in 2.2) to generate the detection box.

[0050] During the training process, the detection box regression branch uses L1 Loss to set a loss function item l bbox In this branch, only positive examples are supervised, and a detection box size supervision signal is designed for each frame of each action instance, and the L1 distance with the predicted displacement is used as the loss penalty.

[0051] In the inference stage, the detection box regression is performed on the center point of the action instance generated in 4), and the action detection result of the current frame can be obtained.

[0052] 2.4) sequence matching branch: The sequence matching branch directly forms the final spatio-temporal detection result in the detection process. In the technical field, the spatio-temporal detection result generated along the time dimension is called tube. Currently, the mainstream method is to connect the single-frame detection structure through a separate post-processing stage. These methods often rely on dynamic programming or target tracking technology, and pay more attention to the overlapping area of adjacent frame detection boxes while ignoring the semantic level information. The matching branch of the present application is composed of a learnable matching module and an online tube generation module, which can gradually generate tubes during the detection process and successfully utilize the high-level semantic information.

[0053] The prediction result of the current frame is matched with the previous prediction result to generate a spatio-temporal action detection result along the time dimension. Unlike the conventional method of first performing single-frame action detection on all video frames and then connecting the single-frame detection results through a post-processing stage to form the final action detection result, the present method gradually completes the matching and linking of the spatio-temporal action detection result during the detection process, and does not require any post-processing stage after the video processing is completed, realizing an online end-to-end action detection process.

[0054] Specifically as Figure 7As shown, the learnable matching module obtains the current frame action detection result B from the center point prediction branch and the bounding box regression branch j where j is the index of the current frame action detection bounding box. The sequence matching branch of the present application aims to match B j to the generated tube, so that a complete tube can be formed step by step during the detection process. We assume that T i is the i-th generated tube, T i is described as follows:

[0055] T i = {c i , L bbox , S i , B' i , E' i}

[0056] where c i is the action category represented by the tube, L bbox is the detection box coordinates corresponding to each frame in the tube, S i represents the average prediction score of the tube, B' i represents the position coordinates of the last detection box in L bbox , and E' i represents the RoI feature of the entire tube.

[0057] In order to effectively match the detection result B j of the current frame with T i , the present application proposes a learnable matching module to calculate their similarity. Specifically, the present application uses RoI Align to encode each B j to obtain the corresponding RoI feature E j , and we take the average RoI feature of the entire tube as the high-order feature representation E' i of T i . In order to distinguish different categories of action instances, we fuse the one-hot encoding of the category indication in the RoI feature. E j and E' i are input into a Siamese network to calculate the cosine similarity matrix M sim of the features, which consists of three fully connected layers. Using ground-truth annotations on the training set, the network can be effectively trained through Contrastive Loss.

[0058] In the inference stage, in addition to the above-mentioned cosine similarity matrix M sim , we also use the IoU matrix M iou to calculate B'i and B j overlap region size, in addition to which we extend the prediction score of B j into a two-dimensional matrix M score . The three together complete the sequence matching process. In the experiment, we observed that the two additional matrices can help us constrain the temporal smoothness of the generated tube and remove unreliable bounding boxes with lower confidence. Therefore, we add the three matrices together to form the matching score map MSM.

[0059] The best matching <T i , B j > is screened out using MSM, and B j is updated to the matched T i , so that the spatiotemporal detection results can be gradually formed in the processing of the streaming video. When screening the best match, each tube T i is traversed in order, and the B j with the highest score in the i-th row of MSM and satisfying the following conditions is selected as the matched action detection box: B j has not been selected, the predicted action category of B j and T i is the same, the score at the corresponding position of M sim is greater than 0, the score at the corresponding position of M iou is greater than 0.5, and the score at the corresponding position of M score is greater than 0.01. The successfully matched B j is updated to the corresponding T i to complete the update of the tube. Finally, we retain those tubes with an average score greater than 0.01 and a length greater than 30 as the final spatiotemporal detection results.

[0060] 3) Training stage, using Focal Loss as the loss function of the center point prediction branch, using L1 Loss as the loss function of the bounding box regression branch, and using Contrastive Loss as the loss function of the sequence matching branch. In the first stage, we discard the sequence matching branch and focus on the single-frame action detection results. After adding Focal Loss and L1 Loss, the total loss is optimized using the AdamW optimizer with an initial learning rate of 5e-4 (the learning rate of the feature extraction network is reduced by 10 times). When the loss is stable, the learning rate is reduced by 10 times. The training is completed on 8 TITIAN TESLA GPUs, with a single-card BatchSize of 16, and the total number of training rounds is not more than 50. In the second stage, we add the sequence matching branch and freeze the other parts without updating. We use Contrastive Loss as the loss function and continue to iterate for 20 rounds to complete the training.

[0061] 4) Test stage, in the test stage, the application does not perform any data enhancement processing, but still subtracts the respective mean values of the three channels of the COCO dataset and divides by the standard deviation of the three channels to achieve normalization. Similarly, we use bilinear interpolation to unify the input image to 320*224. Under the premise of using only RGB data as input, finally on the UCF101-24 dataset, frameAP@0.5 reaches 74.1, videoAP@0.2 reaches 78.7, videoAP@0.5 reaches 54.0, videoAP@0.75 reaches 21.6, videoAP@0.5:0.95 reaches 25.6, and the visualization is as shown in Figure 8 (a) is the heat map of each action category obtained by the 2D branch, (b) is the prediction score of each action category by the 3D branch, (c) is the heat map of each action category obtained by fusing the 2D branch and the 3D branch, (a)-(c) correspond to the same video detection. From (b), it can be seen that the region with a score of 0.11 has the maximum response in (c). (d) shows the detection result and the true result, and the action label IceDancing is below the frame line in the figure, and the detection result above also has the result of IceDancing. It can be seen that after the enhancement of the action category prediction vector by the 3D branch, the detector of the application has better discriminability for the corresponding action category.

Claims

1. An online end-to-end spatio-temporal action detection method, characterized in that The detection network directly detects the spatio-temporal action detection result for the input video sequence, and the detection network comprises a feature extraction network, a feature cache area, a center point prediction branch, a bounding box regression branch and a sequence matching branch. The feature extraction network extracts high-dimensional abstract features of the video frames. The feature cache module caches the extracted video frame features to obtain a feature sequence. The feature sequence is transmitted into the action center point prediction branch, the target bounding box regression branch and the sequence matching branch respectively. The action center point prediction branch and the target bounding box regression branch obtain the action instance center point and the target action detection box. The action detection box is transmitted into the sequence matching branch. The current frame action detection result is matched with the last frame detection result in time sequence, and the continuous spatio-temporal detection result is generated by splicing. The detection network is specifically configured as follows: 1) The feature extraction network: a 2D feature extraction network is used to extract features of the input image sequence. The video frame is processed by the feature extraction network to obtain a high-dimensional feature map, i.e. a video frame feature. The feature extraction network comprises a down-sampling layer and an up-sampling layer. The video frame feature comprises two types. The feature after the up-sampling layer is used as a high-resolution feature, and the feature only after the down-sampling layer is used as a low-resolution feature. 2) The feature cache area: the feature cache area caches at most 64 video frame features. The newly generated feature is put into the buffer area to replace the earliest cached feature according to the order of the input video frame. 3) The center point prediction branch: the center point prediction branch comprises a 2D branch and a 3D branch. The 2D branch is processed by the high-resolution feature of the current frame to obtain the position of the motion instance center point. The 3D branch uniformly samples 8 frames from the low-resolution feature sequence in the feature cache area as an input feature sequence. The 3D sub-network is processed by the stacked R(2+1)D module to perform spatio-temporal feature fusion. The action category preference prediction result is obtained by a full connection layer. The results of the 2D branch and the 3D branch are combined to obtain the action center point prediction result, including positioning and classification. The center point prediction branch simultaneously uses the fine spatial detail information of the high-resolution feature and the continuous time sequence feature information of the low-resolution feature to complete the positioning of the action center point and the estimation of the action category. The 3D branch completes the sub-task of action category recognition, and the 2D branch completes the sub-task of action positioning. 4) The bounding box regression branch: the bounding box regression branch is processed by the high-resolution feature of the current frame to obtain the bounding box of the motion instance center point of the current frame. The positioning and classification obtained by the center point prediction branch are combined to form the action instance detection result of the current frame, including the action category and the action bounding box. In the 2D branch, high-resolution features of the previous frame and the current frame are fused as input, and the input features sequentially pass through a 3*3 convolution layer, a ReLU activation layer, a 1*1 convolution layer, and a Sigmoid layer to obtain a center point prediction heat map with C channels , C represents the action category; 8 frames are uniformly sampled from a 64-frame low-resolution feature sequence in the 3D feature buffer and spliced along the channel to form a feature map, which is input into a 3D subnetwork for action recognition. The 3D subnetwork includes 4 stacked R(2+1)D spatiotemporal convolution blocks and a global average pooling layer, and finally outputs a one-dimensional action category prediction vector V through a fully connected layer. The dimension of the prediction vector V is C. Combine the results of 2D branch and 3D branch: first extend the one-dimensional action category prediction vector V generated by the 3D branch along the W-H axis to generate a two-dimensional prediction matrix M, then add the prediction matrix M and the heat map generated by the 2D branch, so that the long sequence action classification information is included in the center point positioning estimation heat map with position detail features, which helps to suppress inaccurate classification results in the middle 5) The sequence matching branch: the current frame action detection result is matched with the last frame detection result in time sequence. The continuous spatio-temporal detection result is generated by adaptive matching and splicing. After the whole video sequence is processed, the spatio-temporal action detection result of the whole video sequence is generated accordingly. The network is a DLA-34 fused with a deformable convolution.

2. The online end-to-end spatio-temporal action detection method of claim 1, wherein the feature extraction is characterized by ​ 3. The online end-to-end spatio-temporal action detection method of claim 1, wherein In the 3D branch of the center point prediction branch, when the number of frames cached in the feature cache area is less than 64 frames, the existing frames are copied to uniformly fill in to realize uniform sampling of 8 frames as the input feature sequence.

4. The online end-to-end spatio-temporal action detection method of claim 1, wherein In the adaptive matching and splicing process, the sequence matching branch uses RoI to represent the action detection result of a single frame image, and uses a twin network to process the detection results of the two frames before and after, matches the same area on the two frames, uses Contrastive Loss for supervised training, and after matching, splices to generate continuous spatio-temporal detection results. In the sequence matching process, the detection result of the current frame is matched with the existing action sequence, and the network prediction similarity, the size of the overlapping part and the corresponding prediction score of the two regions are considered comprehensively. The prediction score is the confidence of the detection network for the region, that is, the prediction score of the center point prediction branch.

5. The online end-to-end spatio-temporal action detection method of claim 4, wherein The sequence matching branch is composed of a learnable matching module and an online tube generation module, and the tube refers to the spatio-temporal detection result connected along the time dimension; The current frame action detection result is obtained by the center point prediction branch and the bounding box regression branch where j is the index of the current frame action detection bounding box, and the sequence matching branch aims to match the current frame action detection result with the sequence of the previous frame action detection results in an online manner connected to the generated tube, so that a complete tube is gradually formed during the detection process for the ith generated tube, The description is as follows: wherein, an action category represented by the tube, a bounding box coordinate corresponding to each frame of the tube, an average prediction score represented by the tube, representing a position coordinate of the last bounding box in the tube, a RoI feature representing the tube as a whole; The learnable matching module calculates the detection result of the current frame With The similarity of the tube, using RoI Align to encode each , get the corresponding RoI feature , and take the average RoI feature of the whole tube as High-order feature representation In order to distinguish different categories of action instances, the one-hot encoding of the category indication is fused in the RoI feature, and finally and Input a Siamese network to calculate the cosine similarity matrix of the features , which consists of three fully connected layers, and uses ground-truth annotations on the training set to train the network through Contrastive Loss In addition, using the IoU matrix The overlap area size is calculated And The prediction score of is expanded into a two-dimensional matrix The similarity matrix , the IoU matrix , the two-dimensional matrix The three matrices are added to form a matching score map MSM, the best matching , is screened out using the MSM, and is updated into the matched , and the spatiotemporal detection result is gradually formed in the processing process of the streaming video.

6. The online end-to-end spatio-temporal action detection method of claim 1, wherein The construction of the detection network includes the following steps: 1) Generate training samples: take 64 consecutive frames of a video as a sampling period, sample 8 frames in each period as an input video sequence, and the video frame is an RGB image; 2) Configure the detection network; 3) Training stage: use Focal Loss to supervise the center point prediction branch of the detection network, use L1 Loss to supervise the bounding box regression branch, use Contrastive Loss to supervise the sequence matching branch, use AdamW optimizer, and update the network parameters through the back propagation algorithm until the iteration number is reached; 4) Test stage: input the video in the test set into the detection network to obtain the spatio-temporal action detection result in the whole video to verify the effect of the detection network.

7. An online end-to-end spatio-temporal action detector characterized by The computer program is configured in the data storage device, and the computer program is executed to realize the detection method of any one of claims 1-4. The computer program is configured in the data storage device, and the computer program is executed to realize the detection method of any one of claims 1-4.

Images