A spatio-temporal attention video behavior recognition method based on time adaptation

Abstract

The application discloses a kind of spatio-temporal attention video behavior identification method based on time adaptation, belong to computer vision in behavior identification technical field.The method is distinguished to the importance of feature in time dimension by time adaptation, and filters out representative typical feature, to guide more attention is allocated on important feature to assist accurate identification.Spatio-temporal attention obtains spatio-temporal information from important feature class and secondary feature class based on energy function, selects the feature with representative information, and then fuses two kinds of features based on spatio-temporal dimension weight to obtain fusion feature, finally obtains video behavior identification result according to fusion feature.Spatio-temporal attention not only does not introduce parameter, but also can directly deduce three-dimensional weight, so it can improve the performance of video behavior identification network.The entire module has the nature of plug and play, can be easily embedded into mainstream 2DCNNs, to form a dedicated video behavior identification network.

Description

TECHNICAL FIELD

[0001] The present application relates to a time adaptive spatio-temporal attention video behavior recognition method, belonging to the technical field of behavior recognition in computer vision. BACKGROUND

[0002] With the continuous development of information technology, video data has become an indispensable part of modern society. Especially with the continuous development of video-related hardware devices, the amount of video data has increased significantly, which has caused problems in the storage, processing and analysis of video data. Compared with traditional two-dimensional images, video data contains more rich information, including the position, speed, direction of objects and the relationship between objects, etc. These information provides great potential and possibility for intelligent analysis and mining of video data.

[0003] In the past, traditional video monitoring and understanding mainly relied on manual completion, but this way not only consumes time and effort, has time delay and is easy to have subjective color, and sometimes even missed, which cannot meet the needs of modern society. With the continuous emergence of computer vision technology with deep learning as the core, new methods and tools are provided for intelligent analysis and processing of video data, and the behavior recognition ability based on video data has been greatly improved. These progress and innovation provide the possibility for the realization of fast, accurate intelligent analysis and automatic processing of video data, making it a research hotspot in the field of computer vision.

[0004] Deep learning technology has made great achievements in the field of two-dimensional image recognition, and more and more scholars have joined the research. The video behavior recognition task based on video data has different time information from traditional two-dimensional images. How to reasonably use the time dimension information is an important research direction of the video behavior recognition task. Although the existing methods have mined the information of the time dimension, they do not consider the continuity feature between video frames. When applied to the video behavior recognition task, all video frames are treated equally without distinguishing their importance to the behavior target. At the same time, the importance of the action of a behavior at different time points is also different for the behavior recognition result. The existing methods do not distinguish the importance of the features in the time dimension, which will waste more hardware and software resources and have a certain impact on the recognition result. For example, for human behaviors such as running and typing, the feature is obvious and the change is small, so the information gap between video frames is not large. There will be a problem of video frame redundancy, and a few randomly extracted images can also well recognize the behavior. But for behaviors with large changes and similar to other actions, video frames with rich features need to be strictly selected. For example, for the behavior of shooting, the video frames containing the two typical action features of jumping and throwing the ball play a decisive role in the recognition result, and the action of holding the ball alone cannot accurately determine the result. Therefore, a method is needed to distinguish the importance of video frames at different time points. SUMMARY

[0005] In order to further improve the accuracy of video behavior recognition, the application provides a time-adaptive spatio-temporal attention video behavior recognition method, which comprises the following steps:

[0006] Step 1, construct a time-adaptive module and a spatio-temporal attention module, and embed them into a two-dimensional convolutional neural network to obtain a time-adaptive spatio-temporal attention video behavior recognition network;

[0007] Step 2, extracting features of the image frames to be identified to obtain the original feature map of each image frame; the image to be identified is an image obtained by video decoding the video to be identified;

[0008] Step 3, learning the weight of the original feature map in the time dimension based on the time-adaptive module to obtain the importance of each time dimension feature to the result, thereby distinguishing important features and secondary features, and further determining the time dimension weight coefficient of each image frame;

[0009] Step 4, multiplying the time dimension weight coefficient obtained in step 3 with the original feature map to obtain a feature map fused with the time dimension weight, and determining the important feature frame and the secondary feature frame according to the time dimension weight coefficient;

[0010] Step 5, obtain the spatio-temporal correlation feature based on the spatio-temporal attention module, and assign the spatio-temporal dimension weight to the important feature frame class and the secondary feature frame class respectively;

[0011] Step 6, assign two learnable parameters to the important feature frame class and the secondary feature frame class respectively, and then perform pixel-by-pixel addition to obtain the fusion feature;

[0012] Step 7, based on the fusion feature, use the two-dimensional convolutional neural network to obtain the recognition result of the video behavior.

[0013] Optionally, the step 3 comprises:

[0014] Step 3.1, reshape the original feature map, and adjust the feature map with the size of NTxCxHxW to the size of NxTxCxHW; wherein N is the batch size, T is the number of sampled video frames, C, H and W represent the channel number, height and width of the feature map respectively;

[0015] Step 3.2, obtain the local saliency information and the background global information by using the maximum pooling and the average pooling respectively, and determine the parameters α and β based on the features obtained by the two kinds of pooling operations, the two parameters are between 0 and 1;

[0016] The adaptive learning process is expressed as:

[0017]

[0018] Wherein, X and Y are the input and output feature maps of the time adaptive module respectively, Maxpool and Avgpool represent the maximum pooling operation and the average pooling operation respectively; the size of the input feature map X is TxCxHW, and the size of the output feature map Y is Txlxl;

[0019] Step 3.3, determine the time dimension weight coefficient of each image frame based on the output feature map.

[0020] Optionally, the step 5 comprises:

[0021] Step 5.1, obtain the spatio-temporal correlation feature of the important feature and the secondary feature respectively:

[0022]

[0023] Wherein, X' represents the important feature or the secondary feature after the time weight assignment, is the corresponding output feature map, i.e. the spatio-temporal correlation feature, and E is the energy function;

[0024]

[0025] In the formula, μ and σ 2respectively represent the mean and variance of each pixel point in the feature map;

[0026] Step 5.2, according to the spatio-temporal correlation characteristics of the important features and the secondary features, determine the spatio-temporal feature coefficients A and B respectively assigned to the important feature frame class and the secondary feature frame class.

[0027] Optionally, the mean and variance of the pixel points are respectively:

[0028]

[0029]

[0030] wherein M is the product of the time dimension size, length and width of the feature map, X i represents each pixel point in the feature map.

[0031] Optionally, the two-dimensional convolutional neural network in step 1 includes ResNet, MobileNet, ShuffleNet and EfficientNet.

[0032] Optionally, in the method, the two-dimensional convolutional neural network selects a corresponding specific network according to the actual application scenario:

[0033] If the requirement for recognition speed is higher than the requirement for recognition accuracy in the actual application scenario, a small-capacity lightweight network is selected, including MobileNet.

[0034] If the requirement for recognition speed is lower than the requirement for recognition accuracy in the actual application scenario, a network with moderate or larger capacity is selected, including the ResNet series network.

[0035] Optionally, in step 1, the time adaptive module and the spatio-temporal attention module are embedded in any position of the two-dimensional convolutional neural network to obtain a spatio-temporal attention video behavior recognition network based on time adaptation.

[0036] Optionally, if the two-dimensional convolutional neural network adopts the network structure of ResNet-50, the time adaptive module and the spatio-temporal attention module are embedded in the last residual unit in the last stage.

[0037] Optionally, the method further comprises training the spatio-temporal attention video behavior recognition network based on time adaptation, and when the trained network is used for video behavior recognition, the loss function adopts Focal Loss:

[0038] FL(p t )=-α(1-p t ) γ log(p t )

[0039]

[0040] wherein the values of parameters a and g are respectively 0.25 and 2; p represents a prediction probability.

[0041] The application also provides application of the above method in video monitoring and target tracking.

[0042] The application has the following advantages:

[0043] The time self-adaption distinguishes the importance of features in the time dimension without introducing complex operations, filters out representative typical features, and thus guides to allocate more attention on important features to assist accurate recognition. The spatio-temporal attention obtains spatio-temporal information from important feature classes and secondary feature classes respectively, so as to select features with representative information, and obtains spatio-temporal correlation information of important features and secondary features based on an energy function, and then allocates spatio-temporal dimension weights, and subsequently fuses the two types of features based on the spatio-temporal dimension weights to obtain fused features, and finally obtains a video behavior recognition result according to the fused features. In the application, the spatio-temporal attention not only does not introduce parameters, but also can directly infer three-dimensional weights, so as to improve the performance of the video behavior recognition network. The entire module has a plug-and-play property, and can be easily embedded into mainstream 2D CNNs to form a dedicated video behavior recognition network. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 Structure diagram of the spatio-temporal attention video behavior recognition module based on time self-adaption.

[0046] Figure 2 Time self-adaption structure diagram.

[0047] Figure 3 Spatio-temporal attention structure diagram.

[0048] Figure 4 Spatio-temporal attention video behavior recognition network based on time self-adaption.

[0049] Figure 5 Comparison diagram of accuracy rate of mainstream method and the method of the present application on UCF101 dataset and HMDB51 dataset. DETAILED DESCRIPTION

[0050] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0051] Embodiment one:

[0052] The embodiment provides a spatial-temporal attention method based on temporal adaptation (STATA), which comprises the following steps:

[0053] Step 1, constructing a temporal adaptation module and a spatial-temporal attention module, and embedding the two modules into a two-dimensional convolutional neural network to obtain a spatial-temporal attention video behavior recognition network based on temporal adaptation;

[0054] Step 2, performing feature extraction on the image frames to be identified to obtain original feature maps of each image frame; the image to be identified is an image obtained by video decoding a video to be identified;

[0055] Step 3, learning the weight of the original feature map in the time dimension based on the temporal adaptation module to obtain the importance of each time dimension feature to the result, thereby distinguishing important features and secondary features, and further determining the time dimension weight coefficient of each image frame;

[0056] Step 4, multiplying the time dimension weight coefficient obtained in step 3 with the original feature map to obtain a feature map fused with the time dimension weight, and determining important feature frames and secondary feature frames according to the time dimension weight coefficient;

[0057] Step 5, obtaining spatial-temporal correlation features based on the spatial-temporal attention module, and assigning spatial-temporal dimension weights to important feature frame classes and secondary feature frame classes respectively;

[0058] Step 6, adding important features and secondary features pixel by pixel according to the spatial-temporal dimension weights of the important feature frame classes and the secondary feature frame classes to obtain fused features;

[0059] Step 7, based on the fused features, using a two-dimensional convolutional neural network to classify to obtain a recognition result of the video behavior.

[0060] Embodiment two:

[0061] The embodiment provides a spatial-temporal attention method based on temporal adaptation, as shown in Figure 1 , the method comprises the following steps:

[0062] Step1: Select a commonly used two-dimensional convolutional neural network as the main network of the video behavior recognition algorithm, and then embed the proposed time-adaptive spatiotemporal attention module into the network to form a dedicated video behavior recognition network.

[0063] Step2: Morph the feature map obtained from the original network, and then the time-adaptive spatiotemporal attention module learns the weights of the input feature map in the time dimension, obtaining the importance of each time dimension feature to the result, and then classifying into important feature class and secondary feature class.

[0064] Step3: According to the classification result, important features and secondary features are extracted respectively. The spatiotemporal attention module is designed according to the energy function, which can model the time and space information without introducing parameters. Finally, the two types of features are fused to obtain spatiotemporal features with good expression.

[0065] Step4: The video data after data preprocessing is sent into the dedicated time-adaptive spatiotemporal attention video behavior recognition network. The time-adaptive spatiotemporal attention module can obtain more appropriate and sufficient feature expression, so as to realize more accurate classification of behaviors.

[0066] The main network of behavior recognition can be any commonly used two-dimensional convolutional neural network, such as ResNet, MobileNet, ShuffleNet, EfficientNet, etc.

[0067] Referring to Figure 2 , the above method is applied to the UCF101 dataset and uses the ResNet50 network for behavior recognition as an example to illustrate that the UCF101 dataset is a commonly used public dataset in the field of behavior recognition. The method comprises:

[0068] A.1, select the main network. The main network should be selected according to the deployment scene requirements. If speed is pursued, a small-capacity lightweight network such as MobileNet should be selected. If accuracy is pursued, a network with moderate or larger capacity such as ResNet series can be selected. In this embodiment, ResNet-50 is selected as the main network of behavior recognition.

[0069] B.1, build a time-adaptive spatiotemporal attention video behavior recognition module.

[0070] The step B.1 comprises:

[0071] (1) Temporal adaptation. Firstly, the feature map is reshaped to adjust the size from NTxCxHxW to NxTxCxHW. Where N is the batch size, T is the number of sampled video frames, and C, H, W represent the channel number, height, and width of the feature map, respectively. The max-pooling operation can obtain the local saliency information of the target, and the average-pooling operation can retain the global information such as the background. In the CBAM experiment, the effect of using both kinds of pooling operations is better than using one kind of method alone, and they can complement each other. This shows that in the process of extracting image features, max-pooling and average-pooling play different roles. Based on this, the proposed temporal adaptation module uses an adaptive mechanism based on the CAM idea. The temporal adaptation learns the features obtained by the max-pooling and average-pooling operations through the network, and allocates trainable parameters a and β to the feature maps obtained by different pooling operations, respectively. The two parameters are between 0 and 1. The adaptive learning process of the two weights can be expressed as

[0072]

[0073] Where X and Y are the input and output feature maps of the adaptive operation, and Maxpool and Avgpool represent the max-pooling operation and the average-pooling operation, respectively. The size of the input feature map is TxCxHW, and the size of the output feature map changes to Txlxl after the pooling operation, which realizes dimension reduction in the channel dimension and the spatial dimension. The final output feature not only retains the original information, but also adaptively adjusts the inclination of the pooled features to accurately extract useful features as much as possible.

[0074] (2) Spatio-temporal attention. T frames are randomly sampled from the original video, and then the time dimension weight feature obtained from the temporal adaptation structure is multiplied with the original feature map. The weighted feature map is separated into important features and secondary features according to the weight coefficient size. The feature maps with the top 4 weight sizes are defined as important features, and the remaining feature maps are defined as secondary features. A mask is set to 1 for the important features and 0 for the secondary features, so that the important features are retained and the information of the remaining 4 time point feature maps is set to zero, which is used as the important feature frame class. Conversely, for the secondary feature frame class, the mask is exactly the opposite of the previous one, and the information of the feature maps with the top 4 weight sizes is retained.

[0075] Traditional channel attention distinguishes each channel but treats all positions equally; spatial attention distinguishes different positions but treats all channels equally. However, in the human cognitive system, the two mechanisms work together, which is more conducive to information selection. In CBAM, the weights of the channel dimension and the spatial dimension are estimated in turn, and then the information of the two dimensions is combined to realize the joint action of the two dimensions. However, this method cannot directly obtain the real three-dimensional weight, and it spends a lot of effort. Yang et al. proposed a simple parameter-free attention module (SimAM) to directly obtain the three-dimensional weight. This method evaluates each neuron and then assigns a unique weight. In neuroscience theory, according to the richness of the information carried by the neuron, each neuron will exhibit a unique firing pattern and will inhibit the surrounding neurons, so that the target neuron and other neurons are linearly separable, and the target neuron can be given a higher importance. Based on the knowledge of neuroscience, an energy function is defined, and a parameter-free attention module is constructed to directly infer the three-dimensional weight from the current neuron. The parameter-free attention proposed by Yang et al. realizes the different treatment of spatial positions, but this approach ignores the unique time dimension characteristics of video information, limiting the ability of attention to learn the correlation information between time and space. Inspired by the idea of parameter-free attention, the original SimAM applied to the feature map with a size of NxCxHxW is transplanted to the feature map with a size of NxCxHxW after deformation, which can realize the synchronous modeling of time and space features, i.e. spatiotemporal attention. Spatiotemporal attention relies on the theory that the lower the energy, the more important the weight is, and then uses the weight to enhance the feature.

[0076] The core of spatiotemporal attention can be expressed by the following formula, and the spatiotemporal correlation features of important features and secondary features are obtained by the following formula respectively:

[0077]

[0078] wherein X' represents the important feature or the secondary feature after the time weight assignment, is the output feature map, i.e. the spatiotemporal correlation feature, and E is the energy function. According to the theory of neuroscience, the energy function of each neuron can be defined as:

[0079]

[0080] In the formula, μ and σ 2 represent the mean and variance of each pixel point in the feature map, which can be expressed as:

[0081]

[0082]

[0083] M is the product of the feature map time dimension size, length, and width, X i represents each pixel point in the feature map.

[0084] The non-parametric space-time attention is applied on the deformed feature map with a size of N x T x C x HW, the feature map after the space-time attention obtains the space-time correlation information of each element, then two learnable parameters A and B are allocated to the important feature frame class and the secondary feature frame class respectively, and the important features are amplified or the non-important features are suppressed by means of the two learnable parameters. Finally, the two groups of features are added pixel by pixel to realize feature fusion.

[0085] C.1, embedding a time adaptive space-time attention video behavior recognition module.

[0086] The designed time adaptive space-time attention video behavior recognition module has the nature of plug and play, and can be easily embedded into any position in the mainstream two-dimensional convolutional neural network. In the network structure of ResNet-50, the feature map obtained from the deeper Block has higher semantic information. Therefore, the last residual unit in the last stage is modified, and the time adaptive space-time attention video behavior recognition module is embedded to realize the supplement of time information and space information, thereby forming a time adaptive space-time attention network.

[0087] D.1, data preprocessing.

[0088] The step D.1 comprises:

[0089] (1) video decoding. The original data set belongs to the video type, and before being sent into the network for processing, the video stream needs to be converted into a picture sequence, that is, the continuous three-dimensional structure is decomposed into a discrete two-dimensional image structure. A complete continuous video is extracted into a discrete image frame.

[0090] (2) sparse sampling. First, a video is divided into K continuous segments, then a frame is randomly sampled from the segmented segments, and the input of the network only uses the frame extracted from each segment. The segmented sparse sampling strategy reduces the redundant information and ensures that the data has a certain time interval.

[0091] (3) Data augmentation. In the case of limited data, using a small number of samples to train easily leads to overfitting problems. To alleviate this problem, data augmentation is performed on the data. The main data augmentation operations are random cropping, random flipping, etc. Random cropping is to randomly select a position in the image for cropping, and the cropped image size is 224x224. Random flipping is to horizontally or vertically flip the cropped image. Scale jittering is to first adjust the size of the original image to 340x256, then randomly select the width and height values of the to-be-cropped region from the four preset values, and finally downsize the cropped image to 224x224.

[0092] Taking the UCF101 dataset as an example, it contains a total of 13320 video data and a total of 101 categories. According to the rules provided by the official website, the dataset is divided into three groups of training set and test set.

[0093] E.1, training the network model.

[0094] The step E.1 comprises:

[0095] (1) Set the experimental parameters. The batch size is 24. In the training phase, the SGD optimization algorithm is used, and the momentum is set to 0.9. To alleviate the overfitting phenomenon, the Dropout parameter is set to 0.5. The incomplete pre-training model strategy is used, the initial learning rate is set to 0.001, and the learning rate is decayed at the 10th and 25th epochs, and the decay rate is 0.1.

[0096] (2) Set the loss function. The expression of the conventional binary cross-entropy loss function can be expressed as

[0097]

[0098] Where p and y represent the predicted probability and the true label, respectively. The cross-entropy loss function gives the same weight to easy-to-classify samples and difficult-to-classify samples, which leads to slow convergence of the loss function and even optimization problems. Therefore, the cross-entropy loss function in the original TSN method is replaced by Focal Loss. This loss function gives higher weights to samples of minority classes to improve the imbalance problem between positive and negative samples. The expression of Focal Loss is:

[0099] FL(p t )=-α(1-p t ) γ log(p t )

[0100]

[0101] Focal Loss introduces two new parameters based on the cross-entropy loss function, using alpha to balance positive and negative samples, and using gamma to alleviate difficult sample classification problems, the values of alpha and gamma are 0.25 and 2 respectively; p represents the prediction probability.

[0102] F.1, complete network training, and test and deploy the best-performing model.

[0103] The step F.1 comprises:

[0104] (1) When deploying the model, the model with the highest accuracy is deployed, and the average value of the output of the three UCF101 test data sets is taken as the final result.

[0105] As Figure 3 described, the embodiments compare the behavior recognition accuracy of classifying all behaviors of the UCF101 data set using different methods, wherein:

[0106] The TSN RGB method can refer to“Wang L, Xiong Y, Wang Z, et al. Temporal segment networks: Towards good practices for deep action recognition [C] / / Proceedings of the European Conference on Computer Vision. 2016: 20-36.”;

[0107] The P-TSN method can refer to“Shi Y, Zeng Z. Action recognition based on feature propagation of temporal segmentation network [J]. Journal of Computer-Aided Design & Computer Graphics, 2020, 32(04): 582-589.”;

[0108] The MIF method can refer to“Zhang B, Ge S, Wang Q, et al. Research on behavior recognition method based on multi-order information fusion [J] / / Acta Automatica Sinica, 2021, 47(03): 609-619.”;

[0109] The AMA method can refer to“Yu M, Chen Y. AMA: attention-based multi-feature aggregation module for action recognition [J] / / Signal, Image and Video Processing, 2022: 1-8.”.

[0110] The four methods and the method of the present application all use the commonly used two-dimensional convolutional neural network ResNet-50 as the main network to classify the video behaviors. Among the four methods, the TSN RGB method is the earliest to propose a sparse sampling strategy. It divides the complete video, and then randomly selects a video frame from the segmented segment, thereby establishing a long-distance time information connection. In Figure 3 In order to ensure the fairness of the comparison, the original BN-Inception network is replaced by ResNet-50, and only the RGB frame is used as the input of the network. This method finally only trains the extracted video frame, and each segment will get the preliminary prediction result of its own behavior class, and then fusion is generated to form the final video-level prediction. The P-TSN method is a feature propagation-based time domain segmentation network, which uses TSN as the basic skeleton and designs a two-stream time domain segmentation network that uses feature propagation to express spatial information and uses FlowNet to express motion information. Although it can effectively obtain spatial appearance information and temporal motion information, the operation of first separating time and space information and then fusing them is not conducive to establishing the association between time and space. The MIF method is a behavior recognition method based on multi-order information fusion. This method is also based on the ResNet-50 network, but it is a dual-stream network model. This method obtains local information through a two-order dual-stream model to form multi-order information with first-order information, thereby forming a global expression of the video. The AMA method integrates the acquisition of spatial-temporal information and channel features into a unified 2DCNNs framework. Its spatial-temporal aggregation structure not only presents spatial features and long-distance temporal relationships, but also captures the mutual dependence of global context. The channel excitation structure can selectively emphasize useful features and suppress less useful features. This method is based on the attention mechanism to aggregate time, space and channel features, and can be plug-and-play. As can be seen, although the four existing methods focus on the acquisition of spatial-temporal information, they ignore the screening of the importance of features, which can easily cause uneven resource allocation, so it is difficult to further improve the recognition accuracy; Figure 3 The behavior recognition accuracies finally obtained by the four methods and the method of the present application are given by Figure 3It can be known that the behavior recognition accuracy of the method is 95.8%, which is higher than the four existing methods listed, and the existing methods are difficult to improve the accuracy of behavior recognition, and the time adaptive spatio-temporal attention module designed in the application can select the feature map with representative information to reasonably obtain more rich and accurate spatio-temporal information. The time adaptive module is used in the time dimension to obtain the weight of each video feature map, then the feature map is realized in the time dimension according to the importance, and is divided into two categories of important and unimportant, then the spatio-temporal attention module is used for the two types of feature maps to obtain the space and time information, finally the spatio-temporal features of the two parts are fused. The method has the nature of plug and play, can be easily embedded into mainstream 2D CNNs, and good performance is achieved on the UCF101 data set.

[0111] Part of the steps in the embodiments of the application can be realized by software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk.

[0112] The above only describes the preferred embodiments of the application and is not intended to limit the application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A spatio-temporal attention video action recognition method based on time adaptation, characterized in that, The method comprises: Step 1, constructing a time adaptive module and a space-time attention module, and embedding both into a two-dimensional convolutional neural network to obtain a space-time attention video behavior recognition network based on time adaptation; Step 2, performing feature extraction on the image frames to be identified to obtain original feature maps of each image frame; the image to be identified is an image obtained by video decoding on a video to be identified; Step 3, learning the weight of the original feature map in the time dimension based on the time adaptive module to obtain the importance of each time dimension feature to the result, thereby distinguishing important features and secondary features, and further determining the time dimension weight coefficient of each image frame; Step 4, multiplying the time dimension weight coefficient obtained in step 3 with the original feature map to obtain a feature map fused with the time dimension weight, and determining important feature frames and secondary feature frames according to the time dimension weight coefficient; Step 5, obtaining space-time correlation features based on the space-time attention module, and assigning space-time dimension weights to important feature frame classes and secondary feature frame classes respectively; Step 6, assigning two learnable parameters to important feature frame classes and secondary feature frame classes respectively, and then performing pixel-by-pixel addition to obtain fused features; Step 7, based on the fused features, using a two-dimensional convolutional neural network to classify to obtain the recognition result of the video behavior; The step 3 comprises: Step 3.1, reshaping the original feature map to adjust the feature map with a size of NTxCxHxW to a size of NxTxCxHW; wherein N is the batch size, T is the number of sampled video frames, and C, H and W represent the channel number, height and width of the feature map respectively; Step 3.2, obtaining local saliency information and background global information by using maximum pooling and average pooling respectively, and learning parameters a and β based on the features obtained by the two kinds of pooling operations; the two parameters are between 0 and 1; The adaptive learning process is expressed as: Wherein, X and Y are the input and output feature maps of the time adaptive module, and Maxpool and Avgpool represent the maximum pooling operation and the average pooling operation respectively; the size of the input feature map X is TxCxHW, and the size of the output feature map Y is Tx1x1; Step 3.3, determining the time dimension weight coefficient of each image frame based on the output feature map.

2. The method of claim 1, wherein, The step 5 comprises: Step 5.1, obtaining space-time correlation features of important features and secondary features respectively: Wherein, X' represents the important features or secondary features after time weight assignment, For the corresponding output feature map, that is, the space-time correlation feature, E is the energy function; In the formula, μ and σ 2 respectively represent the mean and variance of each pixel point in the feature map; Step 5.2, determining space-time feature coefficients A and B assigned to important feature frame classes and secondary feature frame classes respectively according to the space-time correlation features of important features and secondary features.

3. The method of claim 2, wherein, The mean and variance of the pixel points are: where M is the product of the characteristic map temporal dimension size, length, and width, X i represents each pixel point in the characteristic map.

4. The method of claim 2, wherein, The two-dimensional convolutional neural network in step 1 comprises ResNet, MobileNet, ShuffleNet and EfficientNet.

5. The method of claim 2, wherein, In the method, the two-dimensional convolutional neural network selects a corresponding specific network according to the actual application scenario: If the requirement for recognition speed is higher than the requirement for recognition accuracy in the actual application scenario, a small-capacity lightweight network is selected, including MobileNet; If the requirement for recognition speed is lower than the requirement for recognition accuracy in the actual application scenario, a network with moderate or larger capacity is selected, including the ResNet series network.

6. The method of claim 2, wherein, In the method, the time adaptive module and the space-time attention module are embedded into any position of the two-dimensional convolutional neural network in step 1 to obtain a space-time attention video behavior recognition network based on time adaptation.

7. The method of claim 6, wherein, If the two-dimensional convolutional neural network adopts the network structure of ResNet-50, the time adaptive module and the space-time attention module are embedded into the last residual unit in the last stage.

8. The method of claim 1, wherein, The method further comprises training the space-time attention video behavior recognition network based on time adaptation, and when the trained network is used for video behavior recognition, a loss function adopts Focal Loss: FL(p t ) = -a(1 - p t ) γ log(p t ) wherein the values of parameters α and γ are 0.25 and 2 respectively; and p represents a prediction probability.

9. Use of the method according to any one of claims 1-8 in video monitoring and target tracking.

Images