Behavior Recognition Method Based on Self-Supervised Spatiotemporal Attention Network
Through the combination of multi-scale feature extraction of self-supervised space-time attention network and space-time attention module, the problem of insufficient feature extraction in behavior recognition is solved, and the effect of high accuracy and rapid recognition is achieved.
Patent Information
- Application Number
- CN202310619495.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-29
- Publication Date
- 2025-08-05
- Estimated Expiration
- 2043-05-29
AI Technical Summary
The prior art fails to effectively distinguish key information from important action areas when feature extraction exists in behavior recognition, the recognition accuracy is low and the speed is slow, and the data set annotation is time-consuming.
The self-supervised space-time attention network is adopted to train the network through a combination of multi-scale feature extraction module and space-time attention module, and the self-supervised method is used to perform convolution operations at different scales, improve the image quality of the sampled frame, capture the remote dependence of the sampled frame in the video, and realize the full utilization of the sampled frame image information and space-time information.
It improves the accuracy and speed of behavior recognition, solves the problem of failing to effectively distinguish key information from important action areas during feature extraction, and reduces the data set labeling time, and has the advantages of high recognition accuracy and fast speed.
Smart Images
Figure CN116630776B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing technology, and specifically relates to a behavior recognition method based on a self-supervised spatiotemporal attention network. Background Art
[0002] A method for training action recognition models based on web data has been published. This method applies a teacher network to the dataset to obtain pseudo-labels, filters out irrelevant samples with low confidence, and employs a joint training strategy to address domain differences between multiple data sources and formats in supervised learning networks to achieve recognition results. This method suffers from issues with feature extraction, such as ineffectively distinguishing key information from important action areas. Furthermore, a too slow sampling rate results in overly pronounced changes between actions (weak discriminability), while a too fast sampling rate results in less pronounced changes between actions (strong discriminability), potentially leading to loss of feature signals and affecting action recognition results. Summary of the Invention
[0003] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a behavior recognition method based on a self-supervised spatiotemporal attention network with high recognition accuracy and fast recognition speed.
[0004] The technical solution adopted to solve the above technical problems is composed of the following steps:
[0005] (1) Processing the dataset
[0006] We use the public dataset Kinetics-400 as the training set V and UCF101 as the test set, V∈{v1,v2,…,v t}, v t is the frame at time t of the video, t∈{1,2,…,N}, N is the total length of the video frame, a finite positive integer, a video is sampled into view A and view B, Where C is the number of feature channels, H represents the feature height, W represents the feature width, view A is sampled from 90% of the frames of the original video, and view B is sampled from 40% of the frames of the original video.
[0007] (2) Building a behavior recognition network
[0008] The behavior recognition network consists of a student network and a teacher network in parallel. The student network is composed of a multi-scale feature extraction module and a spatiotemporal attention module in series. The teacher network and the student network have the same structure. View A passes through the teacher network and the student network, and view B passes through the student network.
[0009] The multi-scale feature extraction module is composed of residual module 1, residual module 2, and residual module 3 connected in parallel.
[0010] The spatiotemporal attention module is composed of a self-attention module and a spatial attention module in series.
[0011] (3) Training the network
[0012] 1) Determine the objective function
[0013] The objective function includes the loss function The highest accuracy T1 and the top five accuracy T5 of the evaluation function are used to determine the loss function.
[0014]
[0015] Among them, f TA represents the value of view A input to the teacher network, f SA represents the value of view A input to the student network, f SB Represents the value that View B inputs to the student network.
[0016] The highest accuracy rate T1 is determined by the following formula:
[0017]
[0018] Among them, TP represents the number of correct positive example predictions, FP represents the number of incorrect negative example predictions, FN represents the number of incorrect positive example predictions, TN represents the number of correct negative example predictions, the sum of TP, FP, FN, and TN is the total number of samples, and the sum of TP and TN is the number of correctly predicted samples. TP, FP, TN, FN∈N, where N ranges from 0 to 101, and TP, FP, TN, and FN are not all 0.
[0019] Determine the top five accuracy T5 as follows:
[0020]
[0021] Among them, A5 represents the number of all correct labels included in the top five classification probabilities.
[0022] 2) Training the network
[0023] The training set is input into the action recognition network for training. The video frame size of the training set is adjusted to 224×224. During the training process, the data batch is 16. The Kinetics-400 dataset is trained for 50 rounds with a learning rate of 0.001. The learning rate is decayed to 0.0001 in the 40th round and the learning rate is decayed to 0.00001 in the last 10 rounds. The temperature hyperparameter τ is set to 0.7 and the training is carried out until the loss function converges.
[0024] Perform forward and back propagation on the teacher network and the student network, update the weights of the two networks, and update the parameters of the student network through the exponential moving average EMA m is a finite positive integer, and the student network weight is determined by the following formula
[0025]
[0026]
[0027] where w t is the weight of the teacher network, α is the decay coefficient of the moving average, and α is (0,1).
[0028] (4) Behavior Recognition
[0029] Input the test set into the trained action recognition network and output the action recognition results in the video.
[0030] In step (2) of the present invention, in constructing the behavior recognition network, the residual module 1 is composed of a first pooling layer, a first convolution layer with a convolution kernel of 7×7, and a second convolution layer with a convolution kernel of 5×5, which are connected in series in sequence. The residual module 2 is composed of a second pooling layer and a third convolution layer with a convolution kernel of 3×3, which are connected in series. The residual module 3 is composed of the third pooling layer. The multi-scale feature extraction module outputs X:
[0031]
[0032] x ab =conv(Mix P )
[0033] Mix P =M P (x j )+A P (x j )
[0034] Among them, a is the order of the residual module, a∈{1,2,3}, b is the order of the convolution layer on the residual module, b∈{1,2}, σ represents the sigmoid function, x j represents the output of the original branch in the residual connection, M P is the maximum pooling, A P For average pooling, the output is mixed pooling Mix P .
[0035] In step (2) of constructing the behavior recognition network of the present invention, the spatial attention module is composed of branch module 1 and branch module 2 in parallel.
[0036] The branch module 1 of the present invention is composed of a fourth convolution layer with a convolution kernel of 1×1, and the branch module 2 is composed of a fifth convolution layer with a convolution kernel of 1×1, a fourth pooling layer, and a normalization layer connected in series.
[0037] In the step (3) of training the network in the present invention, the teacher network and the student network are forward propagated and back propagated to update the weights of the two networks. The parameters of the student network are updated by the exponential moving average EMA. The value of m ranges from 50 to 100, and the student network weight is determined by the following formula
[0038]
[0039]
[0040] where w t is the weight of the teacher network, α is the decay coefficient of the moving average, and α is (0,1).
[0041] In the step (3) of training the network in the present invention, the teacher network and the student network are forward propagated and back propagated to update the weights of the two networks. The parameters of the student network are updated by the exponential moving average EMA. The best value of m is 80, and the student network weight is determined by the following formula
[0042]
[0043]
[0044] where w t is the weight of the teacher network, α is the decay coefficient of the moving average, and the best value of α is 0.5.
[0045] The present invention combines a multi-scale feature extraction module with a spatiotemporal attention module, uses a self-supervised approach to train the network, and improves the image quality of the original sampling frame by performing convolution operations on the input image at different scales. It fully explores the spatiotemporal information of different views, captures the long-range dependencies of the sampling frames in the video, and fully utilizes the image information and spatiotemporal information of the sampling frames, as well as accurately identifies the behavior. This solves the problem of not effectively distinguishing between key information and important action areas when extracting behavior recognition features, as well as the time-consuming problem of data set annotation. In addition, this training method can explore unknown or hidden information in the data. The present invention was compared with the prior art in an experiment. The experiment showed that compared with the comparative experiment, the highest accuracy and top five accuracy rates of the present invention were better than those of the comparative experiment. The present invention has the advantages of high recognition accuracy and fast recognition speed, and can be promoted and used in the field of image processing technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 This is a flowchart of Example 1 of the present invention.
[0047] Figure 2 Schematic diagram of the structure of the self-supervised spatiotemporal attention network of Example 1 of the present invention.
[0048] Figure 3 yes Figure 2 Schematic diagram of the structure of the multi-scale feature extraction module.
[0049] Figure 4 yes Figure 2 Schematic diagram of the spatiotemporal attention module structure. DETAILED DESCRIPTION
[0050] The present invention will be further described in detail below with reference to the accompanying drawings and examples, but the present invention is not limited to the following embodiments.
[0051] Example 1
[0052] The behavior recognition method based on the self-supervised spatiotemporal attention network of this embodiment consists of the following steps (see Figure 1 ):
[0053] (1) Processing the dataset
[0054] We use the public dataset Kinetics-400 as the training set V and UCF101 as the test set, V∈{v1,v2,…,v t}, v t is the frame at time t of the video, t∈{1,2,…,N}, N is the total length of the video frame, a finite positive integer, a video is sampled into view A and view B, Where C is the number of feature channels, H represents the feature height, W represents the feature width, view A is sampled from 90% of the frames of the original video, and view B is sampled from 40% of the frames of the original video.
[0055] (2) Building a behavior recognition network
[0056] exist Figure 2 In the embodiment, the behavior recognition network is composed of a student network and a teacher network in parallel. The student network is composed of a multi-scale feature extraction module and a spatiotemporal attention module in series. The teacher network and the student network have the same structure. View A passes through the teacher network and the student network, and view B passes through the student network.
[0057] exist Figure 3 In the embodiment, the multi-scale feature extraction module is composed of residual module 1, residual module 2, and residual module 3 in parallel.
[0058] In this embodiment, the residual module 1 is composed of a first pooling layer, a first convolution layer with a 7×7 convolution kernel, and a second convolution layer with a 5×5 convolution kernel, which are connected in series. The residual module 2 is composed of a second pooling layer and a third convolution layer with a 3×3 convolution kernel, which are connected in series. The residual module 3 is composed of the third pooling layer. The multi-scale feature extraction module outputs X:
[0059]
[0060] x ab =conv(Mix P )
[0061] MiX p =M p (x j )+A P (x j )
[0062] Among them, a is the order of the residual module, a∈{1,2,3}, b is the order of the convolution layer on the residual module, b∈{1,2}, σ represents the sigmoid function, x j represents the output of the original branch in the residual connection, M P is the maximum pooling, A P For average pooling, the output is mixed pooling Mix P .
[0063] exist Figure 4 In this embodiment, the spatiotemporal attention module is composed of a self-attention module and a spatial attention module in series. The self-attention module has been published in existing publications. The spatial attention module of this embodiment is composed of branch module 1 and branch module 2 in parallel. Branch module 1 of this embodiment is composed of the fourth convolutional layer with a convolution kernel of 1×1, and branch module 2 is composed of the fifth convolutional layer with a convolution kernel of 1×1, the fourth pooling layer, and the normalization layer in series.
[0064] (3) Training the network
[0065] 1) Determine the objective function
[0066] The objective function includes the loss function The highest accuracy T1 and the top five accuracy T5 of the evaluation function are used to determine the loss function.
[0067]
[0068] Among them, f TA represents the value of view A input to the teacher network, f SA represents the value of view A input to the student network, f SB Represents the value that View B inputs to the student network.
[0069] The highest accuracy rate T1 is determined by the following formula:
[0070]
[0071] Among them, TP represents the number of correct positive example predictions, FP represents the number of incorrect negative example predictions, FN represents the number of incorrect positive example predictions, TN represents the number of correct negative example predictions, the sum of TP and FP, FN, and TN is the total number of samples, and the sum of TP and TN is the number of correctly predicted samples. TP, FP, TN, and FN∈N, where N ranges from 0 to 101. In this embodiment, N is 5, and TP, FP, TN, and FN are not all 0 at the same time.
[0072] Determine the top five accuracy T5 as follows:
[0073]
[0074] Among them, A5 represents the number of all correct labels included in the top five classification probabilities.
[0075] 2) Training the network
[0076] The training set is input into the action recognition network for training. The video frame size of the training set is adjusted to 224×224. During the training process, the data batch is 16. The Kinetics-400 dataset is trained for 50 rounds with a learning rate of 0.001. The learning rate is decayed to 0.0001 in the 40th round and the learning rate is decayed to 0.00001 in the last 10 rounds. The temperature hyperparameter τ is set to 0.7 and the training is carried out until the loss function converges.
[0077] Perform forward and back propagation on the teacher network and the student network, update the weights of the two networks, and update the parameters of the student network through the exponential moving average EMA The value of m ranges from 50 to 100. In this embodiment, the value of m is 80. The student network weight is determined as follows:
[0078]
[0079]
[0080] where w t is the weight of the teacher network, α is the attenuation coefficient of the moving average, and the value of α is (0, 1). In this embodiment, the value of α is 0.5.
[0081] (4) Behavior Recognition
[0082] Input the test set into the trained action recognition network and output the action recognition results in the video.
[0083] Completed the action recognition method based on self-supervised spatiotemporal attention network.
[0084] Example 2
[0085] The behavior recognition method based on the self-supervised spatiotemporal attention network of this embodiment consists of the following steps (see Figure 1 ):
[0086] (1) Processing the dataset
[0087] This step is the same as in Example 1.
[0088] (2) Building a behavior recognition network
[0089] This step is the same as in Example 1.
[0090] (3) Training the network
[0091] 1) Determine the objective function
[0092] The objective function includes the loss function The highest accuracy T1 and the top five accuracy T5 of the evaluation function are used to determine the loss function.
[0093]
[0094] Among them, f TA represents the value of view A input to the teacher network, f SA represents the value of view A input to the student network, f SB Represents the value that View B inputs to the student network.
[0095] The highest accuracy rate T1 is determined by the following formula:
[0096]
[0097] Among them, TP represents the number of correct positive example predictions, FP represents the number of incorrect negative example predictions, FN represents the number of incorrect positive example predictions, TN represents the number of correct negative example predictions, the sum of TP, FP, FN, and TN is the total number of samples, and the sum of TP and TN is the number of correctly predicted samples. TP, FP, TN, and FN∈N, where N ranges from 0 to 101. In this embodiment, N is 0, and TP, FP, TN, and FN are not 0 at the same time.
[0098] Determine the top five accuracy T5 as follows:
[0099]
[0100] Among them, A5 represents the number of all correct labels included in the top five classification probabilities.
[0101] 2) Training the network
[0102] The training set is input into the action recognition network for training. The video frame size of the training set is adjusted to 224×224. During the training process, the data batch is 16. The Kinetics-400 dataset is trained for 50 rounds with a learning rate of 0.001. The learning rate is decayed to 0.0001 in the 40th round and the learning rate is decayed to 0.00001 in the last 10 rounds. The temperature hyperparameter τ is set to 0.7 and the training is carried out until the loss function converges.
[0103] Perform forward and back propagation on the teacher network and the student network, update the weights of the two networks, and update the parameters of the student network through the exponential moving average EMA The value of m ranges from 50 to 100. In this embodiment, the value of m is 50. The student network weight is determined as follows:
[0104]
[0105]
[0106] where w t is the weight of the teacher network, α is the attenuation coefficient of the moving average, and the value of α is (0, 1). In this embodiment, the value of α is 0.1.
[0107] The other steps are the same as those in Example 1.
[0108] Completed the action recognition method based on self-supervised spatiotemporal attention network.
[0109] Example 3
[0110] The behavior recognition method based on the self-supervised spatiotemporal attention network of this embodiment consists of the following steps (see Figure 1 ):
[0111] (1) Processing the dataset
[0112] This step is the same as in Example 1.
[0113] (2) Building a behavior recognition network
[0114] This step is the same as in Example 1.
[0115] (3) Training the network
[0116] 1) Determine the objective function
[0117] The objective function includes the loss function The highest accuracy T1 and the top five accuracy T5 of the evaluation function are used to determine the loss function.
[0118]
[0119] Among them, f TA represents the value of view A input to the teacher network, f SA represents the value of view A input to the student network, f SB Represents the value that View B inputs to the student network.
[0120] The highest accuracy rate T1 is determined by the following formula:
[0121]
[0122] Among them, TP represents the number of correct positive example predictions, FP represents the number of incorrect negative example predictions, FN represents the number of incorrect positive example predictions, TN represents the number of correct negative example predictions, the sum of TP and FP, FN, and TN is the total number of samples, and the sum of TP and TN is the number of correctly predicted samples. TP, FP, TN, and FN∈N, where N ranges from 0 to 101. In this embodiment, N is 101, and TP, FP, TN, and FN are not 0 at the same time.
[0123] Determine the top five accuracy T5 as follows:
[0124]
[0125] Among them, A5 represents the number of all correct labels included in the top five classification probabilities.
[0126] 2) Training the network
[0127] The training set is input into the action recognition network for training. The video frame size of the training set is adjusted to 224×224. During the training process, the data batch is 16. The Kinetics-400 dataset is trained for 50 rounds with a learning rate of 0.001. The learning rate is decayed to 0.0001 in the 40th round and the learning rate is decayed to 0.00001 in the last 10 rounds. The temperature hyperparameter τ is set to 0.7 and the training is carried out until the loss function converges.
[0128] Perform forward and back propagation on the teacher network and the student network, update the weights of the two networks, and update the parameters of the student network through the exponential moving average EMA The value of m ranges from 50 to 100. In this embodiment, the value of m is 100. The student network weight is determined as follows:
[0129]
[0130]
[0131] where w tis the weight of the teacher network, α is the attenuation coefficient of the moving average, and the value of α is (0, 1). In this embodiment, the value of α is 0.9.
[0132] The other steps are the same as those in Example 1.
[0133] Completed the action recognition method based on self-supervised spatiotemporal attention network.
[0134] In order to verify the beneficial effects of the present invention, the behavior recognition method based on the self-supervised spatiotemporal attention network in Example 1 of the present invention (hereinafter referred to as Example 1) is compared with “Li Tao, Xueting Wang, and Toshihiko Yamasaki. Animproved inter-intra contrastive learning framework on self-supervised video representation. IEEE Trans. Circuits Syst. Video Technol, 32(8): 5266–5280, 2022.” (hereinafter referred to as comparative experiment 1), “Ishan Dave, Rohit Gupta, Mamshad Nayeem Rizve, and Mubarak Shah. Tclr: Temporal contrastive learning for video representation. Computer Vision and Image Understanding, 219, 2022.” (hereinafter referred to as comparative experiment 2), “Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. Vivit: A video vision transformer. In 2021 IEEE / CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 6816–6826. IEEE, 2021." (referred to as comparative experiment 3), "Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale visiontransformers.In2021IEEE / CVF International Conference on Computer Vision,ICCV2021,Montreal,QC,Canada,October 10-17,2021,pages 6804–6815.IEEE,2021.A comparative experiment was conducted with two researchers: "Kanchana Ranasinghe, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Michael S. Ryoo. Self-supervised video transformer. In IEEE / CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 2864–2874. IEEE, 2022." (Comparative Experiment 5). The results of UCF101 action recognition were evaluated using the highest accuracy and top-five accuracy.
[0135] The experimental and calculation results are shown in Table 1.
[0136] Table 1 Experimental results of Example 1 and comparative experiments 1-5
[0137] Experimental group Highest accuracy T1 (%) Top five accuracy T5 (%) Comparative test 1 55.1 72.1 Comparative Test 2 56.2 72.2 Comparative test 3 64.6 89.8 Comparative test 4 67.1 90.8 Comparative test 5 75.3 90.3 Example 1 77.4 91.2
[0138] As can be seen from Table 1, compared with Comparative Experiments 1-5, Example 1 significantly improves the scores of various evaluation indicators. The highest accuracy and top five accuracy of Example 1 are 22.3% and 19.1% higher than those of Comparative Experiment 1, 21.2% and 19% higher than those of Comparative Experiment 2, 13% and 1.4% higher than those of Comparative Experiment 3, 10.3% and 0.4% higher than those of Comparative Experiment 4, and 2.1% and 0.9% higher than those of Comparative Experiment 5. The above experiments show that compared with the comparative experiments, the highest accuracy and top five accuracy of the present invention are both better than those of the comparative experiments and can be used for behavior recognition.
Claims
1. An action recognition method based on self-supervised spatiotemporal attention network, characterized by It consists of the following steps: (1) Processing the dataset We use the public dataset Kinetics-400 as the training set V and UCF101 as the test set, V∈{v1,v2,…,v t }, v t is the frame at time t of the video, t∈{1,2,…,N}, N is the total length of the video frame, which is a finite positive integer. A video is sampled into view A and view B, A∈{x1,x2,…,x A }, B∈{x1,x2,…,x B }, Where C is the number of feature channels, H represents the feature height, W represents the feature width, view A is sampled from 90% of the frames of the original video, and view B is sampled from 40% of the frames of the original video; (2) Building a behavior recognition network The action recognition network consists of a student network and a teacher network in parallel. The student network is composed of a multi-scale feature extraction module and a spatiotemporal attention module in series. The teacher network and the student network have the same structure. View A passes through the teacher network and the student network, and view B passes through the student network. The multi-scale feature extraction module is composed of residual module 1, residual module 2, and residual module 3 in parallel; The spatiotemporal attention module is composed of a self-attention module and a spatial attention module in series; (3) Training the network 1) Determine the objective function The objective function includes the loss function The highest accuracy T1 and the top five accuracy T5 of the evaluation function are used to determine the loss function. Among them, f TA represents the value of view A input to the teacher network, f SA represents the value of view A input to the student network, f SB Represents the value of view B input to the student network; The highest accuracy rate T1 is determined by the following formula: Among them, TP represents the number of correct positive example predictions, FP represents the number of incorrect negative example predictions, FN represents the number of incorrect positive example predictions, TN represents the number of correct negative example predictions, the sum of TP, FP, FN, and TN is the total number of samples, and the sum of TP and TN is the number of correctly predicted samples. TP, FP, TN, FN∈N, where N ranges from 0 to 101, and TP, FP, TN, and FN are not all 0. Determine the top five accuracy T5 as follows: Among them, A5 represents the number of all correct labels included in the top five classification probabilities; 2) Training the network The training set is input into the action recognition network for training. The video frame size of the training set is adjusted to 224×224. During the training process, the data batch size is 16. The Kinetics-400 dataset is trained for 50 rounds with a learning rate of 0.
001. The learning rate is decayed to 0.0001 after 40 rounds and to 0.00001 for the next 10 rounds. The temperature hyperparameter τ is set to 0.7 and the training is carried out until the loss function converges. Perform forward and back propagation on the teacher network and the student network, update the weights of the two networks, and update the parameters of the student network through the exponential moving average EMA m is a finite positive integer, and the student network weight is determined by the following formula where w t is the weight of the teacher network, α is the decay coefficient of the moving average, and α is (0,1); (4) Behavior Recognition Input the test set into the trained action recognition network and output the action recognition results in the video.
2. The behavior recognition method using a self-supervised spatiotemporal attention network according to claim 1, characterized in that: In step (2) of constructing the behavior recognition network, the residual module 1 is composed of the first pooling layer, the first convolution layer with a convolution kernel of 7×7, and the second convolution layer with a convolution kernel of 5×5, which are connected in series. The residual module 2 is composed of the second pooling layer and the third convolution layer with a convolution kernel of 3×3, which are connected in series. The residual module 3 is composed of the third pooling layer. The multi-scale feature extraction module outputs X: x ab =conv(Mix P ) Mix P =M P (x j )+A P (x j ) Among them, a is the order of the residual module, a∈{1,2,3}, b is the order of the convolution layer on the residual module, b∈{1,2}, σ represents the sigmoid function, x j represents the output of the original branch in the residual connection, M P is the maximum pooling, A P For average pooling, the output is mixed pooling Mix P .
3. The behavior recognition method using a self-supervised spatiotemporal attention network according to claim 1, characterized in that: In step (2) of constructing the behavior recognition network, the spatial attention module is composed of branch module 1 and branch module 2 in parallel.
4. The behavior recognition method using a self-supervised spatiotemporal attention network according to claim 3, characterized in that: The branch module 1 is composed of a fourth convolution layer with a convolution kernel of 1×1, and the branch module 2 is composed of a fifth convolution layer with a convolution kernel of 1×1, a fourth pooling layer, and a normalization layer connected in series.
5. The behavior recognition method using a self-supervised spatiotemporal attention network according to claim 1, characterized in that: In step (3) of training the network, forward propagation and back propagation are performed on the teacher network and the student network to update the weights of the two networks. The parameters of the student network are updated by exponential moving average EMA. The value of m ranges from 50 to 100, and the student network weight is determined by the following formula where w t is the weight of the teacher network, α is the decay coefficient of the moving average, and α is (0,1).
6. The behavior recognition method using a self-supervised spatiotemporal attention network according to claim 5, characterized in that: In step (3) of training the network, forward propagation and back propagation are performed on the teacher network and the student network to update the weights of the two networks. The parameters of the student network are updated by exponential moving average EMA. The value of m is 80, and the student network weight is determined by the following formula where w t is the weight of the teacher network, α is the decay coefficient of the moving average, and α is set to 0.5.