Human behavior recognition method based on temporal aggregation network

Abstract

The present invention discloses a method for human behavior recognition based on a time aggregation network, which solves the problem that the existing technology is limited by the size of the convolution kernel and cannot fully explore the dependencies between behavioral features within a long time range, fully aggregate and strengthen time information, and obtain more expressive features, resulting in the inability to significantly improve the accuracy of behavior recognition. The implementation steps of the present invention are: constructing a time aggregation module; constructing a time aggregation network; generating a training set; training the time aggregation network; and recognizing human behavior. The present invention uses self-attention to construct a time aggregation module and introduces the time aggregation module into the existing human behavior recognition network ResNet‑50, so that the present invention can aggregate time information over a long distance and extract more discriminative features, thereby improving the recognition accuracy of human behavior.

Description

Technical Field

[0001] The present invention belongs to the field of video processing technology, and further relates to a human behavior recognition method based on a time aggregation network in the field of computer vision technology. The present invention can be used for human behavior recognition in videos. Background Art

[0002] Due to its powerful content-carrying and expressive capabilities, video has become an indispensable information medium in our lives. Human behavior recognition uses computers to process and analyze input videos to identify human behaviors. Human behavior recognition has practical value in areas such as video content retrieval and recommendation, automated surveillance video analysis, and human-computer interaction.

[0003] Nanjing University of Aeronautics and Astronautics has disclosed a method for human behavior recognition in its patent application, "A Human Behavior Recognition Method Based on ViT of a Dual-Stream Network Architecture" (patent application number: CN202211446673.0, application publication number CN115797827A). The main implementation steps of this method are: preprocessing the input video to convert the video stream data into image RGB frames and TVL1 optical flow images; using the establishment of a feature extraction model for the image RGB frames and TVL1 optical flow images; performing Softmax classification on the features of the RGB frames and TVL1 optical flow images, and performing a weighted sum of the results between multiple frames to obtain the result; inputting the corresponding video data, obtaining the corresponding RGB frames and corresponding optical flow images of the video after the previous preprocessing, and inputting them into the trained dual-stream ViT model to obtain the classification result. The shortcoming of this method is that the image RGB frame feature extraction model processes each image RGB frame with the same weight. In fact, the contribution of each image RGB frame to the recognition of human behavior is different. For example, when playing basketball, the image RGB frame containing the interactive relationship between the person and the basketball is more important for recognizing the basketball than the image RGB frame containing only the person. Using the above method, the image RGB frame with low contribution will affect the network's ability to extract discriminative features, resulting in a lower human recognition rate.

[0004] Shandong Normal University disclosed a human behavior recognition method in its patent application "Human behavior recognition method based on adaptive spatiotemporal convolutional network" (patent application number: CN202111628110.9, application publication number CN114463837A). The implementation steps of this method are: obtaining skeleton data; performing classification operations based on the skeleton data and the adaptive spatiotemporal convolutional network, outputting the classification results, and obtaining human behavior recognition results based on the classification results; wherein, the construction process of the adaptive spatiotemporal convolutional network includes: constructing a multi-layer spatiotemporal convolution block, wherein the fifth and eighth layers are added with residual blocks of temporal convolution, and each of the remaining spatiotemporal convolution blocks includes two different spatial convolution blocks and multi-scale temporal convolution blocks, and motion information is extracted through the two different spatial convolution blocks; according to the motion information and the multi-scale temporal convolution block, the motion information is extracted and aggregated to obtain time domain information. The shortcoming of this method is that it uses the multi-scale temporal convolution block in the adaptive spatiotemporal convolutional network to re-extract and aggregate the motion information input into it to obtain time domain information. Since the convolution kernel size of the multi-scale temporal convolution block is limited, it cannot fully explore the dependency between behavioral features within a long time range, resulting in the network being unable to fully aggregate and strengthen temporal information and obtain more expressive features. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose a human behavior recognition method based on a time aggregation network to solve the problem that the existing technologies are insufficient in capturing long-distance time information, resulting in low accuracy in human behavior recognition.

[0006] To achieve the above objectives, the idea of ​​the present invention is to construct a time aggregation module, splice two input features in the channel dimension to obtain spliced ​​features, input the spliced ​​features into the second low-dimensional mapping group and the third low-dimensional mapping group for feature mapping, calculate the product of the output features of the second low-dimensional mapping group and the output features of the third low-dimensional mapping group, and then use the softmax function for normalization to obtain an attention matrix. The attention matrix is ​​used to weight the features input to the first input layer and the spliced ​​features, and the two weighted features are fused. The above-mentioned self-attention modeling modeling has the dependency relationship between the features of each moment and the features of any moment, realizing the full aggregation of time information within a long time range. By fusing the two weighted features, the information interaction between the two input features is promoted. A time aggregation network is constructed based on the time aggregation module, and the time aggregation module is used to process the shallow features and deeper features input therein, so that the entire network can extract more expressive features and improve the recognition accuracy of human behavior.

[0007] To achieve the above object, the specific steps of the present invention are as follows:

[0008] Step 1: Build a time aggregation module:

[0009] Build a time aggregation module including a first input layer, a second input layer, a splicing unit, a first low-dimensional mapping group, a second low-dimensional mapping group, a third low-dimensional mapping group, a fourth low-dimensional mapping group, a first computing unit, a second computing unit, a third computing unit, a first high-dimensional mapping group, a second high-dimensional mapping group, an adder, a pooling layer, a first output layer, and a second output layer; wherein: the first input layer, the first low-dimensional mapping group, the first computing unit, the first high-dimensional mapping group, the adder, and the first output layer are connected in series in sequence; the second input layer, the splicing unit, the fourth low-dimensional mapping group, the third computing unit, the second high-dimensional mapping group, the pooling layer, and the second output layer are connected in series in sequence; the splicing unit is respectively connected to the first input layer, the second low-dimensional mapping group, and the third low-dimensional mapping group; the second computing unit is respectively connected to the second low-dimensional mapping group, the third low-dimensional mapping group, the first computing unit, and the third computing unit; and the first input layer is connected to the adder;

[0010] Step 2: Build a time aggregation network:

[0011] Build a temporal aggregation network including an input layer, a first convolution block, a second convolution block, an adder, a splicing unit, a fully connected layer, an output layer, a pooling layer, and a temporal aggregation module; wherein: the input layer, the first convolution block, the second convolution block, the adder, the splicing unit, the fully connected layer, and the output layer are connected in series in sequence; the first input layer, the second input layer, the first output layer, and the second output layer of the temporal aggregation module are connected to the second convolution block, the pooling layer, the adder, and the splicing unit respectively; the pooling layer is connected to the first convolution block;

[0012] Step 3: Generate training set:

[0013] Select videos of at least 50 human behavior categories to form a sample set, with each category containing at least 50 videos, and each video corresponding to a specific human behavior label. Select a video clip from each video in the sample set, pre-process the selected video clips, and obtain a training set;

[0014] Step 4: Train the temporal aggregation network:

[0015] The training set is input into the temporal aggregation network, and the cross-entropy loss function is used to calculate the loss value between the predicted label output by the temporal aggregation network and the human behavior label. The network parameters of the temporal aggregation network are iteratively updated through the mini-batch stochastic gradient descent algorithm until the cross-entropy loss function of the network converges, thus obtaining a trained temporal aggregation network.

[0016] Step 5: Identify human behavior:

[0017] At least two video clips are selected from each human behavior video to be identified, each video clip is preprocessed, and the preprocessed video clips are input into the trained time aggregation network; the average vector of the probability value vector output by the network is calculated, and the human behavior category corresponding to the maximum value in the average vector is taken as the recognition result of the human behavior video to be identified.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] The present invention constructs a time aggregation network based on the time aggregation module, which overcomes the problem of low behavior recognition accuracy caused by the existing technology being limited by the convolution kernel size and insufficient ability to capture long-distance time information. The present invention enables modeling the dependency between behavioral features within a long-distance time range, fully aggregates time information, and promotes information interaction between the two input features of the time aggregation module, thereby obtaining more expressive features, thereby improving the network's human behavior recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of the present invention;

[0021] Figure 2 This is a schematic structural diagram of the time aggregation module of the present invention;

[0022] Figure 3 Schematic diagram of the structure of the time aggregation network of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] Reference Figure 1 The specific steps for implementing the embodiment of the present invention are further described.

[0025] Step 1: Build a time aggregation module.

[0026] Reference Figure 2 , further describes the structure of the time aggregation module constructed by the present invention.

[0027] Build a time aggregation module including a first input layer, a second input layer, a splicing unit, a first low-dimensional mapping group, a second low-dimensional mapping group, a third low-dimensional mapping group, a fourth low-dimensional mapping group, a first computing unit, a second computing unit, a third computing unit, a first high-dimensional mapping group, a second high-dimensional mapping group, an adder, a pooling layer, a first output layer, and a second output layer; wherein: the first input layer, the first low-dimensional mapping group, the first computing unit, the first high-dimensional mapping group, the adder, and the first output layer are connected in series in sequence; the second input layer, the splicing unit, the fourth low-dimensional mapping group, the third computing unit, the second high-dimensional mapping group, the pooling layer, and the second output layer are connected in series in sequence; the splicing unit is respectively connected to the first input layer, the second low-dimensional mapping group, and the third low-dimensional mapping group; the second computing unit is respectively connected to the second low-dimensional mapping group, the third low-dimensional mapping group, the first computing unit, and the third computing unit; and the first input layer is connected to the adder.

[0028] The first, second, third, and fourth low-dimensional mapping groups are each composed of a convolutional layer and a dimensionality change layer in series. The convolution kernel size of the convolutional layers in the first to fourth low-dimensional mapping groups is set to 3x1x1, the stride is set to 1, and the dimensionality change layers in the first to fourth low-dimensional mapping groups are all implemented using the reshape function.

[0029] The first computing unit, the second computing unit, and the third computing unit are all composed of a multiplication unit and a normalization layer connected in series. The normalization layers in the first to third computing units are all implemented by a softmax function.

[0030] The first and second high-dimensional mapping groups are both composed of a dimensionality change layer and a convolution layer in series. The convolution kernel size of the convolution layers in the first and second high-dimensional mapping groups is set to 3x1x1, the stride is set to 1, and the dimensionality change layers in the first and second high-dimensional mapping groups are both implemented using the reshape function.

[0031] Input feature maps to the first input layer of the temporal aggregation module The second input layer input feature map Use the splicing unit to connect X and X SAM Cascade along the channel dimension to obtain the concatenated features Use the second low-dimensional mapping group and the third low-dimensional mapping group to process X respectively C Specifically, in the second low-dimensional mapping group, a convolution layer with a kernel size of 3×1×1 and a stride of 1 is used to process X C Compared with the convolution layer with a convolution kernel size of 1×1×1, the convolution layer with a convolution kernel size of 3×1×1 has a larger size in the time dimension, a larger time receptive field, and therefore a stronger time modeling capability. Then, the reshape function is used to process the feature map output by the above convolution layer, so that the dimension of the final output feature map is In the third low-dimensional mapping group, change the parameters of the reshape function and use the same method as above to process X C , so that the dimension of the final output feature map is The output features of the second low-dimensional mapping group and the third low-dimensional mapping group are input to the second calculation unit to calculate the attention matrix A. Specifically, the product of the output features of the second low-dimensional mapping group and the output features of the third low-dimensional mapping group is calculated, and then the product result is normalized using the softmax function. In fact, the value in A represents the feature map X C The correlation degree between the features of each moment in the is higher when the features have a stronger correlation with other moments, and lower when the features have a weaker correlation with other moments.

[0032] Use the first high-dimensional mapping group to process X. The processing process of the first high-dimensional mapping group is the same as that of the second low-dimensional mapping group. Then, A and the features output by the first high-dimensional mapping group are input into the first computing unit. The processing process of the first computing unit is the same as that of the second computing unit. The first computing unit is input into the first high-dimensional mapping group. In the first high-dimensional mapping group, the feature map input therein is first reorganized into C×T×W×H using the reshape function. Then, a convolution layer with a convolution kernel size of 3×1×1 is used for processing. Finally, the output of the first high-dimensional mapping group is element-wise added to X to obtain the feature X′ of the first input layer.

[0033] Process X using the fourth low-dimensional mapping group C , the processing process of the fourth low-dimensional mapping group is the same as the second low-dimensional mapping group, and then the features output by A and the fourth high-dimensional mapping group are input to the third calculation unit. The processing process of the third calculation unit is the same as the second calculation unit mentioned above, and the second calculation unit is input to the second high-dimensional mapping group. In the first high-dimensional mapping group, the reshape function is first used to reorganize the feature maps input therein into Then, a convolution layer with a convolution kernel size of 3×1×1 is used for processing, and finally the output of the second high-dimensional mapping group is input to the pooling layer to obtain the feature X′ of the second input layer. SAM .

[0034] Step 2: Build a time aggregation network based on the time aggregation module.

[0035] Reference Figure 3 , further describing the structure of the time aggregation network constructed by the present invention.

[0036] Build a time aggregation network including an input layer, a first convolution block, a second convolution block, an adder, a splicing unit, a fully connected layer, an output layer, a pooling layer, and a time aggregation module; wherein: the input layer, the first convolution block, the second convolution block, the adder, the splicing unit, the fully connected layer, and the output layer are connected in series in sequence; the first input layer, the second input layer, the first output layer, and the second output layer of the time aggregation module are respectively connected to the second convolution block, the pooling layer, the adder, and the splicing unit; the pooling layer is connected to the first convolution block.

[0037] The first convolution block is composed of conv1, conv2_x, conv3_x, and conv4_x in the ResNet-50 network in the prior art; the second convolution block is implemented using conv5_x in the ResNet-50 network.

[0038] The structural parameters of conv1, conv2_x, conv3_x, conv4_x, and conv5_x in the ResNet-50 are shown in Table 1. conv1, conv2_x, conv3_x, conv4_x, and conv5_x are each composed of a convolutional layer with different convolution kernels and parameters. Taking conv2_x as an example, the symbols and their corresponding meanings in Table 1 are explained. 1x1x1,64 represents a convolutional layer composed of 64 convolution kernels of the same structure, all of which are 1x1x1 in size. Similarly, 3x3x3,64 represents a convolutional layer composed of 64 convolution kernels of the same structure, all of which are 3x3x3 in size. 1x1x1,256 represents a convolutional layer composed of 256 convolution kernels of the same structure, all of which are 1x1x1 in size. [] indicates that all convolution layers within it are connected in series to form a convolution group. x3 indicates that three convolution groups of the same structure are connected in series to form conv2_x.

[0039] Table 1 List of conv1, conv2_x, conv3_x, conv4_x, conv5_x structure parameters

[0040]

[0041] Step 3: Generate a training set.

[0042] In step 3.1, at least 50 videos of human behavior categories are selected to form a sample set. Each category contains at least 50 videos, and each video corresponds to a specific human behavior label.

[0043] Specifically, the UCF-101 training set and the HMDB-51 training set are used as sample sets. For the UCF-101 and HMDB-51 datasets, the official definition of the training set and test set division criteria is given. Based on the official definition, the training set is extracted from the UCF-101 and HMDB-51 datasets, and the two extracted training sets are used as sample sets.

[0044] In step 3.2, a video segment is selected from each video in the sample set, and the selected video segment is downsampled with a step size of 2 to obtain an image sequence corresponding to the video segment.

[0045] The selecting a video segment from each video in the sample set refers to performing a random sampling on each video in the sample set to obtain a video segment corresponding to the video and including 64 consecutive frames of images.

[0046] In step 3.3, each image in each image sequence is randomly cropped and scaled in turn, and all processed images form a training set.

[0047] The random cropping and scaling transformation processing of each image in each image sequence in sequence refers to randomly cropping each image in each image sequence, setting the long side and short side of the cropped image to 0.5 to 1 times the short side of the original image, and then performing a scale scaling transformation so that the size of the transformed image is 112×112.

[0048] Step 4: Train the temporal aggregation network.

[0049] The training set is input into the temporal aggregation network, and the cross-entropy loss function is used to calculate the loss value between the predicted label output by the temporal aggregation network and the human behavior label. The network parameters of the temporal aggregation network are iteratively updated using the mini-batch stochastic gradient descent algorithm until the cross-entropy loss function converges to obtain a trained temporal aggregation network.

[0050] The parameters of the small-batch stochastic gradient descent method are set as follows: the momentum coefficient is set to 0.9, the batch size is set to 32, the initial learning rate is set to 0.01, and the learning rate decays to the original 0.1 after every 15 epochs. The training process has a total of 90 epochs.

[0051] Step 5: Identify human behavior.

[0052] In step 5.1, the same method as in step 3.2 is used to select at least two video clips from each human behavior video to be identified, and the same method as in step 3.2 is used to preprocess each video clip to generate an image sequence with the same number as the video clips.

[0053] Specifically, each human behavior video to be identified is randomly sampled four times to obtain four video clips containing 64 consecutive frames of images corresponding to the video.

[0054] In step 5.2, each image sequence is input into the trained temporal aggregation network. The network outputs the probability value vector corresponding to the image sequence. The average value of all probability value vectors is calculated to obtain the average value vector.

[0055] Specifically, four image sequences corresponding to a video of a human action to be recognized are fed into a trained temporal aggregation network. Each image sequence is fed into the trained temporal aggregation network, and the network outputs a probability vector. The dimension of the probability vector is the number of human action categories, and the value of a particular dimension of the probability vector is the probability that the recognition result belongs to the human action category corresponding to that dimension. Following this process, a total of four probability vectors are obtained. The average of these four vectors is calculated to obtain an average vector.

[0056] In step 5.3, the human behavior category corresponding to the maximum value in the average value vector is used as the recognition result of the human behavior video to be recognized.

[0057] The effects of the present invention will be further described below in conjunction with simulation experiments.

[0058] 1. Simulation experiment conditions:

[0059] The hardware platform for the simulation experiment of the present invention is: 4 TITAN X GPUs.

[0060] The software platform for the simulation experiment of the present invention is: PyTorch deep learning framework and Ubuntu 18.04 operating system.

[0061] The simulation experiments in this paper use the UCF-101 human action recognition video dataset to train the temporal aggregation network. UCF-101, collected from YouTube, contains 13,320 videos from 101 different action categories, divided into 25 groups. The action categories include shaving, long jump, skating, skiing, soccer penalty kicks, handstand walks, and handstand push-ups. These can be divided into five types: single-person body movement, person-object interaction, person-to-person interaction, sports, and instrument playing.

[0062] The simulation experiments in this paper used the HMDB-51 human behavior recognition video dataset to train a temporal aggregation network. The HMDB-51 dataset contains a large number of real videos collected from various scenarios. The dataset contains 6,849 video clips from 51 behavior categories, with each category containing at least 101 video clips. The behavior categories in the dataset include laughing, chewing, smoking, pull-ups, hugging, and playing golf. They can be divided into five categories: common facial movements, facial movements related to objects, routine human behaviors, human behaviors that interact with objects, and human behaviors that interact with humans.

[0063] 2. Simulation content and results analysis:

[0064] The simulation experiment used the present invention and five existing human behavior recognition methods (Two-Stream ConvNet, C3D, TSN, Taylor RNN, and RF3D) to identify human behavior recognition videos from the preprocessed UCF-101 and HMDB-51 human behavior recognition video datasets. For each human behavior recognition video, after processing using each of the above methods, a mean value vector was obtained. The human behavior category corresponding to the maximum value in the mean value vector was used as the recognition result of the human behavior video to be identified by the method.

[0065] The existing technology Two-Stream ConvNet human action recognition method refers to the video human action recognition method based on two-stream networks proposed by Karen Simonyan et al. in "Two-Stream Convolutional Networks for Action Recognition in Videos. In Advances in Neural Information Processing Systems (NeurIPS), 2014", referred to as Two-Stream ConvNet.

[0066] The existing C3D human action recognition method refers to the human action recognition method using 3D convolutional networks to learn spatiotemporal features proposed by Du Tran et al. in "Learning Spatiotemporal Features with 3D Convolutional Networks. IEEE International Conference on Computer Vision (ICCV), 2015", referred to as C3D.

[0067] The existing technology TSN human action recognition method refers to the human action recognition method based on time segment network proposed by Limin Wang et al. in "Temporal Segment Networks: Towards Good Practices for Deep Action Recognition. European Conference on Computer Vision (ECCV), 2016", referred to as TSN.

[0068] The existing technology Taylor RNN human behavior recognition method refers to the human behavior recognition method based on Taylor theory recursive neural network proposed by Guan X et al. in "Mind theremainder: Taylor's theorem view on recurrent neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2021", referred to as TaylorRNN.

[0069] The existing RF3D human behavior recognition method refers to the human behavior recognition method based on 3D convolutional network based on residual frames proposed by Tao L et al. in "Rethinking Motion Representation: Residual Frames With 3D ConvNets. IEEE Transactions on Image Processing, 2021", referred to as RF3D.

[0070] To verify the simulation effect of the present invention, the following human behavior recognition accuracy formula was used to calculate the method used in the simulation experiment of the present invention and five different human behavior recognition methods. After recognizing all videos in the UCF-101 and HMDB-51 test sets, the human behavior recognition accuracy in each test set was calculated, and all the calculated results are plotted in Table 2. Ours in Table 2 represents the simulation experiment results of the present invention.

[0071]

[0072] Where Accuracy represents the accuracy of human behavior recognition, ∑ represents the summation operation, M represents the number of human behavior categories, i represents the i-th human behavior category, where i∈{1,2,...,M-1,M}, and N represents the number of detected human behavior videos.

[0073] Table 2 List of recognition accuracy of the present invention and five methods

[0074]

[0075] From Table 2, it can be seen that the human behavior recognition accuracy of the present invention exceeds that of the prior art on the UCF-101 and HMDB-51 datasets, proving that the present invention can improve the human behavior recognition accuracy.

Claims

1. A human behavior recognition method based on time aggregation network, characterized in that: Construct a time aggregation network with the time aggregation module as the main body; the specific steps of the recognition method include the following: Step 1: Build a time aggregation module: Build a time aggregation module including a first input layer, a second input layer, a splicing unit, a first low-dimensional mapping group, a second low-dimensional mapping group, a third low-dimensional mapping group, a fourth low-dimensional mapping group, a first computing unit, a second computing unit, a third computing unit, a first high-dimensional mapping group, a second high-dimensional mapping group, an adder, a pooling layer, a first output layer, and a second output layer; wherein: the first input layer, the first low-dimensional mapping group, the first computing unit, the first high-dimensional mapping group, the adder, and the first output layer are connected in series in sequence; the second input layer, the splicing unit, the fourth low-dimensional mapping group, the third computing unit, the second high-dimensional mapping group, the pooling layer, and the second output layer are connected in series in sequence; the splicing unit is respectively connected to the first input layer, the second low-dimensional mapping group, and the third low-dimensional mapping group; the second computing unit is respectively connected to the second low-dimensional mapping group, the third low-dimensional mapping group, the first computing unit, and the third computing unit; and the first input layer is connected to the adder; Step 2: Build a time aggregation network: Build a temporal aggregation network including an input layer, a first convolution block, a second convolution block, an adder, a splicing unit, a fully connected layer, an output layer, a pooling layer, and a temporal aggregation module; wherein: the input layer, the first convolution block, the second convolution block, the adder, the splicing unit, the fully connected layer, and the output layer are connected in series in sequence; the first input layer, the second input layer, the first output layer, and the second output layer of the temporal aggregation module are connected to the second convolution block, the pooling layer, the adder, and the splicing unit respectively; the pooling layer is connected to the first convolution block; Step 3: Generate training set: Select videos of at least 50 human behavior categories to form a sample set, with each category containing at least 50 videos, and each video corresponding to a specific human behavior label. Select a video clip from each video in the sample set, pre-process the selected video clips, and obtain a training set; Step 4: Train the temporal aggregation network: The training set is input into the temporal aggregation network, and the cross-entropy loss function is used to calculate the loss value between the predicted label output by the temporal aggregation network and the human behavior label. The network parameters of the temporal aggregation network are iteratively updated through the mini-batch stochastic gradient descent algorithm until the cross-entropy loss function of the network converges, thus obtaining a trained temporal aggregation network. Step 5: Identify human behavior: At least two video clips are selected from each human behavior video to be identified, each video clip is preprocessed, and the preprocessed video clips are input into the trained time aggregation network; the average vector of the probability value vector output by the network is calculated, and the human behavior category corresponding to the maximum value in the average vector is taken as the recognition result of the human behavior video to be identified.

2. The human behavior recognition method based on time aggregation network according to claim 1 is characterized in that: The first low-dimensional mapping group, the second low-dimensional mapping group, the third low-dimensional mapping group, and the fourth low-dimensional mapping group described in step 1 are all composed of a convolution layer and a dimensionality change layer connected in series; the convolution kernel size of the convolution layer in the first to fourth low-dimensional mapping groups is set to 3x1x1, and the step size is set to 1; the dimensionality change layer in the first to fourth low-dimensional mapping groups is implemented by the reshape function.

3. The human behavior recognition method based on time aggregation network according to claim 1 is characterized in that: The first computing unit, the second computing unit, and the third computing unit described in step 1 are all composed of a multiplication unit and a normalization layer connected in series; the normalization layers in the first to third computing units are all implemented by a softmax function.

4. The human behavior recognition method based on time aggregation network according to claim 1 is characterized in that: The first high-dimensional mapping group and the second high-dimensional mapping group described in step 1 are both composed of a dimensionality change layer and a convolution layer in series; the convolution kernel size of the convolution layer in the first and second high-dimensional mapping groups is set to 3x1x1, the step size is set to 1, and the dimensionality change layer in the first and second high-dimensional mapping groups is implemented by a reshape function.

5. The human behavior recognition method based on time aggregation network according to claim 1 is characterized in that: The first convolution block described in step 2 is composed of conv1, conv2_x, conv3_x, and conv4_x in the ResNet-50 network of the prior art; the second convolution block is implemented using conv5_x in the ResNet-50 network.

6. The human behavior recognition method based on time aggregation network according to claim 1 is characterized in that: The preprocessing of the selected video clips in step 3 refers to downsampling the selected video clips with a step size of 2 to obtain an image sequence corresponding to the video clips; and performing random cropping and scaling transformation processing on each image in each image sequence in turn.

7. The human behavior recognition method based on time aggregation network according to claim 1 is characterized in that: The selecting a video segment from each video in the sample set refers to performing a random sampling on each video in the sample set to obtain a video segment corresponding to the video and including 64 consecutive frames of images.

8. The human behavior recognition method based on time aggregation network according to claim 6 is characterized in that: The random cropping and scaling transformation processing of each image in each image sequence in sequence refers to randomly cropping each image in each image sequence, setting the long side and short side of the cropped image to 0.5 to 1 times the short side of the original image, and then performing a scale scaling transformation so that the size of the transformed image is 112×112.

9. The human behavior recognition method based on time aggregation network according to claim 1 is characterized in that: The cross entropy loss function described in step 4 is as follows: Among them, L C represents the loss value of the cross entropy loss function, ∑ represents the summation operation, C represents the total number of categories of human behavior labels in the training set, i represents the sequence number of the category, x i represents the probability that the human behavior label belongs to the i-th category, log(·) represents the logarithmic operation with the natural constant e as the base, y i Represents the probability that the predicted label belongs to the i-th category.

Images