TSF network structure and video behavior recognition method based on stage feature fusion

Abstract

This invention discloses a TSF network structure and video action recognition method based on stage feature fusion, which relates to the field of video action recognition technology. First, the video to be identified is sliced ​​to obtain K video segments; the video frames of each video segment are sampled; the video frames obtained from each video segment are proportionally input into the slow channel and the fast channel to extract the temporal and spatial features of each video segment; through the neighborhood feature fusion method, a prediction function is used to obtain K prediction scores; the K prediction scores are aggregated to obtain a prediction score for the video to be identified; and based on the prediction score of the video to be identified, the recognition result of the video to be identified is obtained. The solution of the invention can more fully extract long-term features and achieve higher accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of video action recognition, and more particularly to a TSF network structure based on stage feature fusion and a video action recognition method. Background Art

[0002] Existing deep learning approaches to action recognition in videos fall into two main categories: video recognition methods that extract and classify spatiotemporal features, and pose estimation methods that extract skeleton information for retraining. Because neural networks can learn features from data, and this learning method aligns with how humans perceive the world, the semantic features learned by neural networks can often be used for action recognition.

[0003] ConvNets have significant advantages in image detection and classification, achieving numerous breakthroughs. ConvNets have also been introduced to the field of video action recognition. ConvNets possess excellent modeling capabilities for large-scale supervised datasets, but unlike images, videos possess rich temporal features. Therefore, end-to-end deep convolutional neural networks offer no significant advantages over traditional hand-crafted feature extraction for video action recognition. The main drawback of mainstream ConvNet architectures for video action recognition is their inability to model long-term temporal structures. Several approaches have been proposed to address this, primarily relying on dense temporal sampling with predefined sampling intervals. SlowFast is a recently proposed action recognition algorithm that uses a modified 3D ResNet as its backbone network to construct two fast and slow channels. By setting different sampling frequencies, the fast and slow channels can effectively extract partial spatiotemporal information. However, this relies on dense uniform sampling, resulting in high computational complexity, poor long-term feature extraction, and the potential for overfitting. Another solution is to segment the time domain by segmenting the video data, sparsely sample each slice to extract spatiotemporal features, and then fuse the spatiotemporal features separately to reach a consensus. Although it can effectively model long-term structures, the features extracted from the fragments are relatively independent. The fused fragment consensus models the entire video, ignoring that spatiotemporal information is accumulated over time. Each fragment has staged spatiotemporal features related to the context. Therefore, it can model long-term structures, but cannot fully extract long-term information.

[0004] Several challenges remain in the existing technologies: 1) how to model long-term temporal structures and learn the spatiotemporal characteristics of videos; and 2) how to fully capture the spatiotemporal characteristics of long-term structures. Therefore, overcoming these challenges is a pressing issue for those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention proposes a network framework called the Temporal Segmented Fast-Slow Fusion Network (TSF). This framework adopts and improves the idea of ​​using asymmetric sampling for the fast and slow channels, sparsely sampling the two channels proportionally and evenly in each video segment. On this basis, a segmented structure is used to aggregate information through a neighborhood feature fusion strategy to perform long-term temporal modeling at the video level. This invention solves the shortcoming of the dual-stream network architecture that cannot extract long-term features and achieves video-level recognition results.

[0006] In order to achieve the above object, the present invention provides the following specific technical solutions:

[0007] A TSF network structure based on stage feature fusion, including the following parts:

[0008] The video slicing module is used to slice the video to be identified to obtain K video segments;

[0009] The spatiotemporal feature extraction module is used to sample the video frames of each video segment; the video frames sampled from each video segment are input into the Slow channel and the Fast channel in proportion to extract the spatiotemporal features of each video segment;

[0010] The prediction module is used to obtain K prediction scores using the prediction function through the neighborhood feature fusion method;

[0011] The aggregation module is used to aggregate the K prediction scores to obtain a prediction score of the video to be identified; and obtain a recognition result of the video to be identified based on the prediction score of the video to be identified.

[0012] Based on the above TSF network structure, a video behavior recognition method based on stage feature fusion is further disclosed, including the following steps:

[0013] Step 1: Slice the video to be identified to obtain K segments of video;

[0014] Step 2: Sample the video frames of each video segment; input the sampled video frames of each video segment into the Slow channel and the Fast channel in proportion to extract the spatiotemporal features of each video segment;

[0015] Step 3: Using the neighborhood feature fusion method, based on the spatiotemporal features of each video, a prediction function is used to obtain K prediction scores for the K videos.

[0016] Step 4: Aggregate the K prediction scores to obtain a stage consensus among them and obtain the prediction score of the video to be identified; and obtain the recognition result of the video to be identified based on the prediction score of the video to be identified.

[0017] Optionally, in step 2, an asymmetric sparse sampling method is used to sample the video frames of each video, which not only captures rich information but also enables end-to-end learning of long time series with reasonable time and computing resources.

[0018] Optionally, in step 3, the K prediction scores include a preliminary prediction score for the first video segment and K-1 staged prediction scores for the remaining K-1 videos.

[0019] Optionally, in step 3, the method of neighborhood feature fusion is specifically as follows:

[0020] Based on the spatiotemporal characteristics of the first video, a preliminary prediction score is directly obtained through the prediction function;

[0021] Starting from the second video, the spatiotemporal features obtained from each video are concatenated with the spatiotemporal features of all previous videos in a feature matrix, and then the prediction function is used to obtain the stage-by-stage prediction score.

[0022] Optionally, in step 3, the prediction function uses a Softmax function to obtain a prediction score.

[0023] Optionally, in step 4, the K prediction scores are aggregated using a piecewise consensus function.

[0024] Optionally, in step 4, the predicted score of the video to be identified includes the predicted score of each behavior label, and the predicted score of each behavior label is used as the confidence level. The behavior category corresponding to the behavior label with the highest confidence level is selected as the identification result of the video to be identified.

[0025] Optionally, the back propagation algorithm is used to iteratively optimize the parameters of the prediction function based on the loss function. The loss function is expressed as:

[0026]

[0027] Where C is the number of action classification categories, y i is the confidence of the i-th action, G i is the value obtained by the phase consistency function of the i-th type of action, G j is the value obtained by the stage consistency function of the j-th type of action, G i It can be expressed as:

[0028]

[0029] Through the above technical solutions, it can be seen that the present invention discloses a TSF network structure and video behavior recognition method based on stage feature fusion, which has the following beneficial effects compared with the existing technology:

[0030] The present invention proposes a neighborhood feature fusion strategy, and on this basis, provides a TSF network structure and its corresponding video behavior recognition method, which continuously performs phased splicing and fusion of the spatiotemporal features of the video along the time dimension and obtains prediction results. Finally, the prediction results of each stage are aggregated to obtain the overall prediction result. By summarizing and judging the phased information to reach a phased consensus, it not only takes into account the contextual connection between video segments, but also integrates the video-level features, greatly enhancing the network's ability to model long-term data and fully extracting long-term domain features. Compared with the existing technology, the TSF network structure not only converges quickly, but also can more fully extract long-term domain features with higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0032] Figure 1 Schematic diagram of the TSF network structure of the present invention;

[0033] Figure 2 This is a step diagram of the video behavior recognition method of the present invention;

[0034] Figure 3 This is a diagram of the duration frequency of the UCF 101 dataset;

[0035] Figure 4 A schematic diagram of feature fusion in an embodiment;

[0036] Figure 5(a) is a schematic diagram of the loss function curve of the TSF network structure;

[0037] Figure 5(b) is a schematic diagram of the loss function curve of the SlowFast network structure;

[0038] Figure 6 Schematic diagram of video recognition accuracy results for various network structure models. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] Traditional two-stream networks work well for short-term actions. However, in real life, some behaviors span long time periods, representing long-term actions. Without effective extraction of long-term features, effective detection is unsuccessful. The most straightforward approach to addressing this problem is dense video sampling, but this significantly increases computational complexity and still fails to achieve video-level prediction for long-term segments.

[0041] The embodiment of the present invention discloses a TSF network structure based on stage feature fusion, see Figure 1 , used for video recognition, including the following parts:

[0042] The video slicing module is used to slice the video to be identified to obtain K video segments;

[0043] The spatiotemporal feature extraction module is used to sample the video frames of each video segment; the video frames sampled from each video segment are input into the Slow channel and the Fast channel in proportion to extract the spatiotemporal features of each video segment;

[0044] The prediction module is used to obtain K prediction scores using the prediction function through the neighborhood feature fusion method;

[0045] The aggregation module is used to aggregate K prediction scores to obtain the prediction score of the video to be identified; and obtain the recognition result of the video to be identified based on the prediction score of the video to be identified.

[0046] The embodiment of the present invention also discloses a video behavior recognition method based on stage feature fusion, see Figure 2 , including the following steps:

[0047] Step 1: Slice the video to be identified and obtain K segments of video (T1, T2, ..., T K ) is a video segment sequence.

[0048] Step 2: Use an asymmetric sparse sampling method to sample the video frames of each video segment. Input the sampled video frames of each video segment into the slow channel and the fast channel in proportion to extract the spatiotemporal features of each video segment. The video frames input into the slow channel and the fast channel preferably adopt a ratio of slow:fast = 2:16.

[0049] Step 3: Based on the spatiotemporal features of each video, a prediction function is used to obtain K prediction scores for K videos using the neighborhood feature fusion method, including the preliminary prediction score of the first video and K-1 stage prediction scores for the remaining K-1 videos.

[0050] use Indicates the T iThe segment's stage-wise prediction category score, where M i =(T1,…,T i ), represents the first i segments, and W represents the network weight parameter. Based on the stage-by-stage consensus, a prediction function is used to predict the probability of each action classification in the segment. In a specific embodiment, the prediction function can use a Softmax function to obtain a prediction score.

[0051] Among them, the method of neighborhood feature fusion is specifically as follows:

[0052] Based on the spatiotemporal characteristics of the first video, a preliminary prediction score is directly obtained through the prediction function;

[0053] Starting from the second video, the spatiotemporal features obtained from each video are concatenated with the spatiotemporal features of all previous videos in a feature matrix, and then the prediction function is used to obtain the stage-by-stage prediction score.

[0054] Step 4: Use the segmented consensus function to aggregate the K prediction scores to obtain a prediction score for the video to be identified. The prediction score for the video to be identified includes a prediction score for each behavior tag. The prediction score for each behavior tag is used as the confidence level. The behavior category corresponding to the behavior tag with the highest confidence level is selected as the recognition result for the video to be identified. For example, the prediction scores for tag 0 are: running 0.95, tag 1 falling 0.96, tag 2 fighting 0.94, and tag 3 smoking 0.99. The highest score is found, corresponding to tag 3, with a confidence level of 0.99. Therefore, the behavior recognition result for the video to be identified is smoking.

[0055] In one embodiment, the above process also includes a parameter optimization step. Finally, combined with the multi-classification cross entropy loss function, the final overall loss function can be expressed as:

[0056]

[0057] Where C is the number of action classification categories, y i is the confidence of the i-th action, G i is the value obtained by the phase consistency function of the i-th type of action, G j The value obtained for the stage consistency function of the j-th type of action can be expressed as:

[0058]

[0059] During network training, the weight parameter W can be optimized iteratively using the backpropagation algorithm in multiple stages. The network's loss function is the loss of the entire video-level result, rather than the loss of the video segment. The overall model parameters are updated by optimizing the loss of the video-level result:

[0060]

[0061] Specific embodiments are listed below to verify the beneficial effects of the present invention:

[0062] 1. Dataset collection.

[0063] UCF 101 is a commonly used basic dataset. The UCF 101 action recognition dataset is collected from YouTube and contains 101 types of actions. Each type of action is performed by 25 people, and each person performs 4-7 groups, totaling 13,320 videos. UCF 101 has a very large diversity in the collection of actions, including camera movement, appearance changes, posture changes, object scale changes, background changes, fiber changes, etc. The 101 types of actions can be divided into five categories: human-object interaction, human body movements, human-human interaction, musical instrument playing, and sports. The total length of this dataset exceeds 27 hours. Since most of the videos in this dataset contain camera movement, background mixing, partial occlusion, poor lighting conditions and low-quality frames, it is challenging in action recognition tasks.

[0064] This example uses the UCF 101 dataset, whose duration distribution frequency is as follows: Figure 3 It can be seen that the video length of most behaviors in the dataset is mostly in the range of 2-10S. Therefore, it is speculated that when the algorithm can extract long-term domain features for training, the detection effect of the algorithm will be improved.

[0065] 2. Video slicing and parameter setting.

[0066] Split the video into 3 Clips and merge them in 3 stages, such as Figure 4 As shown in the figure, the slow channel uniformly samples two frames per video segment, while the fast channel uniformly samples 16 frames. This approach leverages the biological characteristics and advantages of both the fast and slow channels while avoiding the significant overhead of dense sampling. Compared to previous work using densely sampled frames, this significantly reduces the computational cost of the convolutional network, enabling the extraction of sufficient long-term features within limited time and reasonable computing resources.

[0067] 3. Extraction of spatiotemporal features and score prediction.

[0068] The first clip is fed into the network, and its spatiotemporal features are extracted. A preliminary prediction score is directly obtained through the prediction function. Starting from the second clip, the temporal features obtained from each clip are concatenated with the feature matrix of the previous clip, and then passed through the prediction function to obtain a stage-by-stage prediction score. Finally, the three stage-by-stage prediction scores of the temporal network are combined using a consensus algorithm using weighted average fusion to obtain the final prediction score. The final weighted average fusion result S can be expressed as:

[0069]

[0070] Where k is the number of segments, s i is the prediction result of stage i, f i is the weight function, f i =i.

[0071] 4. Performance testing of TSF network structure.

[0072] The Top-K metric is used to measure the performance of the model, where K represents the number of samples with the highest confidence after the Softmax function. Usually, K takes values ​​of 1, 3, 5, etc., that is, the correct result is included in the top K prediction confidences. When K = 1, the prediction with the largest prediction confidence value is considered to be the prediction result. In this embodiment, K = 1 is set, and the accuracy is calculated by calculating the ratio of the number of correctly classified samples to the total number of classified samples. The calculation formula is:

[0073]

[0074] Where True is the number of correct identifications and All is the total number of tests.

[0075] In the TSF network architecture, the TSF network input is set to RGB images sampled from the video, with each frame randomly horizontally flipped and RGB jittered, and the image size is fixed to 224×224. The spatial and temporal stream networks set the training set batch_size to 64, where the batch_size represents the sample capacity of each batch. The initial learning rate lr is set to 0.01. The learning rate is subsequently adjusted using the cosine annealing strategy as the iteration progresses. The calculation formula is as follows:

[0076]

[0077] in and Represent the maximum and minimum values ​​of the learning rate, respectively, and define the range of the learning rate; T cur It indicates how many epochs are currently executed, T i represents the total number of epochs in the i-th run. The network optimization method is stochastic gradient descent (SGD) with a momentum of 0.9. A dropout layer is added after each fully connected layer to prevent overfitting during network training. The dropout rate is set to 0.3 for the spatial stream network and 0.5 for the temporal stream network. Training is terminated when the training loss converges.

[0078] 5. Performance comparison of TSF network structure and existing technologies.

[0079] This example compares the SlowFast network structure with the TSF network structure, with the segment K set to 2. When the RGB image is input into the SlowFast network structure and the TSF network structure, the training loss function curve is as follows: Figure 5(a)-Figure 5(b) , where Figure 5(a) is the loss function curve of the TSF network structure, and Figure 5(b) is the loss function curve of the SlowFast network structure. The horizontal axis is the number of iterations in the network training phase, and the vertical axis is the loss value. Figure 5(a)-Figure 5(b) As can be seen in the figure, the loss decreases with increasing iterations, indicating that the classification results gradually approach the true results. After 5,000 iterations, the loss curve for the TSF network begins to flatten, indicating network convergence. After 60,000 iterations, the loss curve for the SlowFast network stabilizes, indicating network convergence.

[0080] In the test, continuous RGB images are used as input to the trained neural network, and the accuracy results are as follows Figure 6 shown.

[0081] The TSF network structure converged before SlowFast during training, achieving an accuracy rate 8.41 percentage points higher than SlowFast and 5.13 percentage points higher than the TSN network structure. This demonstrates that the TSF network structure not only converges faster but also more fully extracts long-term features, leading to better results. In summary, the TSF network structure has higher accuracy than SlowFast and allows for faster model training, demonstrating its practical value.

[0082] Furthermore, the performance of the TSF network structure on UCF101 is compared with that of other mainstream behavior recognition algorithm models. Figure 6 It can be seen that the TSF network structure proposed in this paper has a significant improvement in accuracy. This also proves that the TSF network structure can fully learn long-term features, where the fusion of neighborhood features makes the temporal features gradually significant, thereby effectively identifying behaviors at the video level.

[0083] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0084] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video behavior recognition method based on stage feature fusion, characterized in that: The following steps are involved: Step 1: Slice the video to be identified to obtain K segments of video; Step 2: sampling the video frames of each video; The video frames sampled from each video are input into the Slow channel and the Fast channel in proportion to extract the spatiotemporal features of each video; Step 3: Using the neighborhood feature fusion method, based on the spatiotemporal features of each video, a prediction function is used to obtain K prediction scores for the K videos. Step 4: Aggregate the K prediction scores to obtain the prediction score of the video to be identified; obtain the recognition result of the video to be identified based on the prediction score of the video to be identified; In step 3, the K prediction scores include the preliminary prediction score of the first video segment and the K-1 staged prediction scores of the remaining K-1 videos; The specific method of neighborhood feature fusion is: Based on the spatiotemporal characteristics of the first video, a preliminary prediction score is directly obtained through the prediction function; Starting from the second video, the spatiotemporal features obtained from each video are concatenated with the spatiotemporal features of all previous videos to form a feature matrix, and then the prediction function is used to obtain the stage-by-stage prediction score. Based on the loss function, the back propagation algorithm is used to optimize the parameters of the prediction function iteratively; the loss function is expressed as: Where C is the number of action classification categories, y i is the confidence of the i-th action, G i is the value obtained by the phase consistency function of the i-th type of action, G j is the value obtained by the stage consistency function of the j-th type of action, G i It can be expressed as:

2. The video behavior recognition method based on stage feature fusion according to claim 1 is characterized in that: In step 2, an asymmetric sparse sampling method is used to sample the video frames of each video segment.

3. The video behavior recognition method based on stage feature fusion according to claim 1 is characterized in that: In step 3, the prediction function is the Softmax function.

4. The video behavior recognition method based on stage feature fusion according to claim 1 is characterized in that: In step 4, the K prediction scores are aggregated using a piecewise consensus function.

5. The video behavior recognition method based on stage feature fusion according to claim 1 is characterized in that: In step 4, the prediction score of the video to be identified includes the prediction score of each behavior label, and the prediction score of each behavior label is used as the confidence level. The behavior category corresponding to the behavior label with the highest confidence level is selected as the recognition result of the video to be identified.

6. A TSF network structure of a video behavior recognition method based on the stage feature fusion according to any one of claims 1 to 5, characterized in that: Includes the following sections: The video slicing module is used to slice the video to be identified to obtain K video segments; The spatiotemporal feature extraction module is used to sample the video frames of each video; The video frames sampled from each video are input into the Slow channel and the Fast channel in proportion to extract the spatiotemporal features of each video; The prediction module is used to obtain K prediction scores using the prediction function through the neighborhood feature fusion method; The aggregation module is used to aggregate K prediction scores to obtain the prediction score of the video to be identified; and obtain the recognition result of the video to be identified based on the prediction score of the video to be identified.

Images