A human behavior recognition method based on spatial and timing dual-channel fusion model

Abstract

The application discloses a human behavior recognition method based on a space and time sequence dual-channel fusion model, constructs a deep learning network architecture fusing an ER3D model and a Space-TimeTransformer model, and obtains an initialized network through end-to-end pretraining on a public human behavior recognition dataset, then reads a human behavior training video, carries out fixed interval sampling and data enhancement on the training video, feeds the video frames after data enhancement into the pre-trained network architecture for training, and generates a human behavior recognition model.In the test stage, the human behavior video in the test set is read, sampled and cut, and then fed into the trained recognition model for recognition, and the whole process realizes end-to-end detection of human behaviors in the video.According to the characteristics that the ER3D model and the Space-TimeTransformer model can effectively model the space information and the time sequence information contained in the video frames, a deep learning network architecture capable of accurately recognizing human behaviors is constructed and trained.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of behavior recognition, and particularly relates to a human behavior recognition method based on a space and time dual-channel fusion model. BACKGROUND

[0002] Human behavior recognition aims to understand human behavior, mainly studies how to determine the behavior of a target from a video image and assign a label to each behavior, such as common behaviors such as walking, sitting, lying, running, jumping, etc., which is a highly attractive and challenging task in computer vision. Human behavior recognition, as an emerging frontier research field, involves multiple disciplines such as pattern recognition, image processing, target detection, and deep learning, and it can be widely applied in many fields, such as intelligent monitoring, information retrieval, human-computer interaction, etc.

[0003] Traditional methods use manual video feature extraction, then feature coding on the extracted features, and classification using a corresponding classifier. Hand-crafted features are time-consuming and labor-intensive, not suitable for all types of data sets, and difficult to extend and deploy. With the development of deep learning, people try to let computers extract information from videos. Deep learning-based methods can automatically learn features from raw data, and then input the learned features into the model for training, thereby achieving efficient classification and recognition of input videos.

[0004] However, there are still many difficulties in practical applications. First of all, the diversity of behavior recognition poses. Everyone wears different clothes, different cameras are at different angles, and the time span of each person's action is different, and the way of holding objects is also different. In addition, occlusion also brings great challenges to behavior recognition. The boundaries of key behaviors are also uncertain, such as wiping the mouth and drinking water, which have a strong confusion effect, and standing at different angles will bring many wrong judgments. SUMMARY

[0005] In order to solve the above problems existing in the prior art and better model the spatial information and time sequence information in human behavior video data, the application provides a human behavior recognition method based on a space and time dual-channel fusion model. The method constructs a deep learning network architecture that fuses an ER3D model and a Space-Time Transformer model, pre-trains it using public data sets, performs fixed interval sampling and data augmentation on the training videos, inputs the data augmented video frames into the pre-trained network architecture for training, and generates a human behavior recognition model. The fusion model can recognize and classify human behaviors in videos with high accuracy, and the entire process realizes end-to-end detection of human behaviors in videos.

[0006] The specific technical scheme adopted by the present application is as follows:

[0007] A human behavior recognition method based on a space and time sequence dual-channel fusion model mainly includes the following steps:

[0008] Step 1, build a space and time sequence dual-channel fusion model;

[0009] Step 2, read the video data in the human behavior data training set;

[0010] Step 3, fixed interval sampling and data enhancement operation are performed on the video data read in the step 2, and video frame sequences are obtained;

[0011] Step 4, input the video frame sequences obtained in the step 3 into the space and time sequence dual-channel fusion model which has been pre-trained, and perform model training;

[0012] Step 5, read the human behavior video data in the test data set;

[0013] Step 6, sample and crop the video data read in the step 5 to obtain video frame sequences;

[0014] Step 7, the five groups of video frame sequences obtained in the step 6 are sequentially input into the model trained in the step 4, and the result of each prediction is averaged to obtain the category of the human behavior in the test video.

[0015] Further, the space and time sequence dual-channel fusion model built in the step 1 is composed of an ER3D model and a Space-Time Transformer model;

[0016] The ER3D model optimizes and improves the traditional residual network ResNet-52, uses depth separable convolution and reverse bottleneck structure to reduce the calculation parameter quantity of the model, increases the basic channel number and the convolution kernel size, and at the same time reduces the use of activation function and normalization layer, then expands the size of the convolution kernel from two dimensions to three dimensions to adapt to video data. Finally, the output feature size is 1x768, and the cross-entropy loss function often used in multi-classification tasks is selected as the loss function, and its expression is:

[0017]

[0018] In the formula, N represents the number of samples, C represents the number of categories, represents the label value of the i-th sample, which is 1 when the sample belongs to category c, and 0 otherwise, represents the probability that the model predicts the i-th sample as category c;

[0019] The Space-Time Transformer model adopts a Vision Transformer as a backbone network for feature extraction, proposes two video frame blocking methods of sequential blocking and compressed blocking, and uses a multi-head attention mechanism to extract the temporal action features between video frames. The size of the feature vector used for classification is 1x768, and the operation process of the attention mechanism in the Space-Time Transformer model is as follows:

[0020]

[0021]

[0022] where Q, K, and V are vectors obtained by multiplying the encoded input sequence features and three learnable matrices, l = 1, 2, …, L represents the number of encoders, a = 1, 2, …, A represents the number of heads in the multi-head attention mechanism, D h = D / A represents the dimension of each head in the multi-head attention mechanism, K (0,0) and V (0,0) are separated because they are used to generate the first position in the sequence after linear mapping, and Z (0,0) only participates in the dot product operation of the attention mechanism once.

[0023] Further, the step 3 specifically includes the following process:

[0024] Since the length of each video in the training set is not fixed, a 16-second continuous video segment is first randomly cropped from each video, then 32 video frames are generated from the 16-second video segment by extracting one frame every 0.5 seconds, and then the 32 video frames are sampled at fixed intervals d1 = 8 and d2 = 4. The remaining 4 video frames after sampling will be sent to the SR3D model later, and the remaining 8 video frames will be sent to the Space-Time Transformer model later.

[0025] The short side length of all video frames retained after fixed interval sampling is resized to a random value in [256, 320], and the long side length is also scaled by the same proportion. Then a picture with a size of 224x224 is randomly cropped from it. In order to do data augmentation, the picture will also be horizontally flipped.

[0026] Further, the step 4 specifically includes the following process:

[0027] The video frames sampled and enhanced in step 3 are input into the fusion model pre-trained by the UCF101 dataset for training.

[0028] In the pre-training and training process of the fusion model, the two channels of the fusion model respectively use an ER3D spatial feature extraction model and a Space-Time Transformer time sequence feature extraction model, because the feature maps used for classification extracted by the two models are of the same size, both are 1x768, therefore, the two feature maps are fused before passing through a fully connected layer, and the size of the new feature map after fusion is 1x1536, which contains more comprehensive video information.

[0029] The advantage of the present application is that the human behavior recognition method based on the spatial and time sequence dual-channel fusion model provided by the present application fuses the spatial features and time sequence features of the video frames extracted by the two channels before classification in the fully connected layer, utilizes the complementarity between the different features extracted, and effectively improves the behavior recognition accuracy by fusing the advantages between different features. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is the ER3D model structure diagram provided by the present application.

[0031] Figure 2 is the Space-Time Transformer model structure diagram provided by the present application.

[0032] Figure 3 is a structure diagram of the spatial and time sequence dual-channel fusion model provided by the present application. DETAILED DESCRIPTION

[0033] The human behavior recognition method based on the spatial and time sequence dual-channel fusion model provided by the present application will be described in detail below in combination with the drawings and specific embodiments.

[0034] Referring to Figure 3 The human behavior recognition method based on the spatial and time sequence dual-channel fusion model provided by the present application embodiment comprises the following steps:

[0035] Step 1, build a spatial and time sequence dual-channel fusion model.

[0036] The spatial and time sequence dual-channel fusion model built is composed of an ER3D model and a Space-Time Transformer model. Among them, the ER3D model optimizes and improves the traditional residual network ResNet-52, uses a depth separable convolution and a reverse bottleneck structure to reduce the calculation parameter amount of the model, increases the basic channel number and the convolution kernel size, and at the same time reduces the use of activation function and normalization layer, then expands the size of the convolution kernel from two dimensions to three dimensions to adapt to video data. Finally, the feature size is 1x768, and the loss function selects the cross-entropy loss function often used in multi-classification tasks, and its expression is:

[0037]

[0038] where N represents the number of samples, C denotes the number of classes, denotes the label value of the ith sample, which is 1 when the sample belongs to class c, otherwise 0, denotes the probability that the model predicts the ith sample as class c;

[0039] The Space-Time Transformer model uses the Vision Transformer as the backbone network for feature extraction, proposes two video frame blocking methods of sequential blocking and compressed blocking, and uses the multi-head attention mechanism to extract the time sequence action features between video frames. The size of the feature vector used for classification is 1x768, and the operation process of the attention mechanism in the Space-Time Transformer model is as follows:

[0040]

[0041]

[0042] where Q, K, and V are the results obtained by multiplying the encoded input sequence features and three parameter learnable matrices, l = 1, 2, …, L represents the number of encoders, a = 1, 2, …, A represents the number of heads in the multi-head attention mechanism, D h = D / A represents the dimension of each head in the multi-head attention mechanism, K (0,0) and V (0,0) are separated because after linear mapping, the first position in the sequence is added to Z (0,0) which is only involved in one attention mechanism dot product operation.

[0043] Step 2, read the human behavior video data in the training set.

[0044] In this embodiment, the training set consists of 400 short videos containing human behavior, the resolution of the videos is 1080x720, the length is about 90 seconds, each video is labeled with human behavior categories, and there are a total of 20 categories.

[0045] Step 3, fixed interval sampling and data enhancement operation are performed on the video data in the training set.

[0046] Since the length of each video in the training set is not fixed, a 16-second continuous video segment is first randomly cropped from each video, then 32 video frames are generated from the 16-second video segment by taking a frame every 0.5 seconds, and then the 32 video frames are sampled at fixed intervals d1 = 8 and d2 = 4. The remaining 4 video frames will be sent to the SR3D model later, and the remaining 8 video frames will be sent to the Space-Time Transformer model later.

[0047] The short side length of all video frames after fixed interval sampling is resized to a random value in [256, 320], and the long side length is also scaled by the same proportion. Then a picture with a size of 224x224 is randomly cropped from it, and a horizontal flip is also performed on the picture for data augmentation.

[0048] Step 4, use the video frame sequence generated in step 3 to train the spatial and temporal dual-channel fusion model built in step 1 to obtain the recognition model.

[0049] In this embodiment, the spatial and temporal dual-channel fusion model built in step 1 needs to be pre-trained in advance. The model pre-training stage uses the UCF101 dataset, which has a total of 13320 video segments, mainly including human-object interaction, human-human interaction, and body movement only. After pre-training, the video frame sequence generated in step 3 is used to further train the model.

[0050] Step 5, read the human behavior video data in the test set.

[0051] In this embodiment, the training set consists of 100 short videos containing human behavior, with a resolution of 1080x720 and a length of about 90 seconds.

[0052] Step 6, sample and crop the data in the test set.

[0053] In this embodiment, for each video, 5 continuous 16-second video segments are uniformly sampled along the time axis, and then the remaining operations on the video segments are synchronized with step 3.

[0054] The scaling process of the video frame is synchronized with step 3, and then the center cropping method is used to obtain a picture with a size of 224x224.

[0055] Step 7, use the 5 groups of video frame sequences obtained in step 6 as inputs to the model trained in step 4, and average the results of each prediction to obtain the category of the human behavior in the video.

[0056] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application in any form. Any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application are within the scope of the technical solutions of the present application.

Claims

1. A human behavior recognition method based on a spatial and temporal dual-channel fusion model, characterized in that, The method comprises the following steps: Step 1, build a space and time dual-channel fusion model: The built space and time dual-channel fusion model is composed of an ER3D model and a Space-Time Transformer model: The ER3D model optimizes and improves the traditional residual network ResNet-52, uses depth separable convolution and reverse bottleneck structure to reduce the calculation parameter quantity of the model, increases the basic channel number and the convolution kernel size, and reduces the use of activation function and normalization layer, then extends the size of the convolution kernel from two dimensions to three dimensions to adapt to video data, finally outputs a feature size of 1x768, and the loss function selects the cross-entropy loss function often used in multi-classification tasks, and its expression is: The Space-Time Transformer model uses a Vision Transformer as the backbone network for feature extraction, proposes two video frame blocking methods of sequential blocking and compressed blocking, and uses a multi-head attention mechanism to extract the time sequence action features between video frames, and finally the feature vector size for classification is 1x768, and the operation process of the attention mechanism in the Space-Time Transformer model is: where N represents the number of samples, C denotes the number of classes, denotes the label value of the i-th sample, which is 1 when the sample belongs to class c, and 0 otherwise, denotes the probability that the model predicts the i-th sample to be class c. Step 2, read the video data in the human behavior data training set; where Q, K, V are three vectors obtained by multiplying the input sequence feature encoding and three parameter learnable matrices, l = 1, 2, …, L represents the number of encoders, a = 1, 2, …, A represents the number of heads in the multi-head attention mechanism, D h = D / A represents the dimension of each head in the multi-head attention mechanism, K (0,0) and V (0,0) are separated out because the first position in the sequence after linear mapping is added to Z (0,0) involved in the dot product operation of the attention mechanism only once; Step 3, fixed interval sampling and data enhancement operation are performed on the video data read in step 2 to obtain a video frame sequence; step 3 specifically comprises the following process: Because the length of each video in the training set is not fixed, 16-second continuous video segments are first randomly cropped from each video, then 32 video frames are generated from the 16-second video segments by taking one frame every 0.5 seconds, and then the 32 video frames are sampled at fixed intervals d1=8 and d2=4, After sampling, the remaining 4 video frames will be sent to the ER3D model, and the remaining 8 video frames will be sent to the Space-Time Transformer model; The short side length of all video frames retained after fixed interval sampling is resized to a random value in [256, 320], and the long side length is also scaled by the same proportion, then a picture with a size of 224x224 is randomly cropped from it, and the picture is also horizontally flipped for data augmentation; Step 4, input the video frame sequence obtained in step 3 into the space and time dual-channel fusion model that has been pre-trained, and perform model training; Step 5, read the human behavior video data in the test data set; Step 6, sample and crop the video data read in step 5, and step 6 specifically comprises the following process: The video in the test data set is uniformly sampled along the time axis to obtain 5 16-second continuous video segments, and the remaining operations on the video segments are the same as step 3; The scaling process of the video frame is the same as step 3, and then a picture with a size of 224x224 is obtained by using center cropping; ​ Step 7, the 5 groups of video frame sequences obtained in step 6 are sequentially taken as inputs of the model trained in step 4, and the results of each prediction are averaged to obtain the category of the human body behavior in the test video.

2. The human behavior recognition method based on the spatial and timing dual-channel fusion model according to claim 1, wherein, The step 4 specifically includes the following process: The video frame obtained in step 3 is input into the fusion model pre-trained by the UCF101 data set for training.

3. The human behavior recognition method based on the spatial and temporal dual-channel fusion model according to claim 2, wherein, In the pre-training and training process of the fusion model, the two channels of the fusion model respectively use the ER3D spatial feature extraction model and the Space-Time Transformer time series feature extraction model. Since the sizes of the feature maps extracted by the two models for classification are the same, both are 1x768, the two feature maps are fused before passing through the full connection layer, and the size of the new feature map after fusion is 1x1536, which contains more comprehensive video information.

Images