A human behavior recognition method based on hybrid neural network

Abstract

The present invention relates to the field of intelligent information processing and pattern recognition technology, and in particular to a method for human behavior recognition based on a hybrid neural network. Construct a training sample set and a test sample set: send the training sample set to a hybrid neural network for training, use the overall recognition accuracy of the test sample set as an indicator to judge the performance, and obtain the model with the highest accuracy as the final training model; perform a preprocessing operation on the pending video sequence, import the image frames and optical flow frames of the pending video sequence into the final training model, and obtain the behavior in the pending video sequence. The present invention achieves the purpose of fully extracting the scene information, motion information and time series information of the video sequence through the deep fusion of the neural network structure, thereby forming a recognition accuracy and robustness that is superior to any single network structure; and realizes the recognition of human behavior for complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent information processing and pattern recognition, and in particular to a human behavior recognition method based on a hybrid neural network. Background Art

[0002] Human behavior recognition technology is widely used in many fields such as medical health, intelligent transportation, security monitoring, intelligent robots, etc. In recent years, with the continuous development of neural networks and the further improvement of computer computing power, the use of neural networks for human behavior recognition has gradually become a research hotspot for researchers.

[0003] Due to the rise of convolutional neural networks in the fields of images, convolutional neural networks can extract information from images very well, so convolutional neural networks have great advantages in extracting appearance information and operation information. However, compared with images, video sequences have time dimension characteristics. When processing time dimension information, convolutional neural networks cannot effectively extract their temporal characteristics. Due to its unique memory, recurrent neural networks can effectively extract contextual relationships to achieve the purpose of extracting time series, but recurrent neural networks may experience gradient vanishing or gradient explosion when extracting long time series. The improved recurrent neural network LSTM adds a forget gate to effectively solve the problem of gradient vanishing and gradient explosion, but it is easy to cause an increase in the number of parameters when extracting scene information and motion information.

[0004] At present, the methods of human behavior recognition based on deep learning can be roughly summarized into three methods: two-stream convolutional neural network, 3D convolutional neural network, and recurrent neural network. Two-stream convolutional neural network extracts its appearance information and motion information respectively and then performs fusion classification to obtain the classification result. It can extract the scene information and motion information of the video well. Although the 3D convolutional neural network can directly input the video into the network for feature extraction, the number of parameters caused by the complex structure of the 3D convolution kernel also increases greatly with the increase of network depth. It cannot perform long-term feature extraction and the extracted motion information and timing information are not very sufficient. Although the subsequent improved 3D convolutional neural network has solved the problem of too large parameters and limited feature extraction ability to a certain extent, it does not have a good effect on all problems. The improved LSTM on the recurrent extension network can obtain long-term timing information, but it is insufficient in extracting scene and running information and introduces a large number of parameters. It slows down the recognition speed of the model. Later researchers combined LSTM with other neural networks to achieve good recognition results.

[0005] Solving the difficulty of human behavior recognition involves a large number of theories and methods in image processing, pattern recognition, and artificial intelligence. To this end, the present invention proposes a recognition method based on a hybrid neural network combining a convolutional neural network, LSTM, and a multi-head attention mechanism, which has better feature extraction adequacy and human behavior recognition accuracy than existing methods. Summary of the invention

[0006] The purpose of the present invention is to provide a human behavior recognition method based on a hybrid neural network, which is used to solve the problems of insufficient feature extraction and insufficient accuracy of human behavior recognition in the prior art.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A human behavior recognition method based on a hybrid neural network comprises the following steps:

[0009] S1. Construct training sample set and test sample set:

[0010] Obtain video data including behavior labels, perform preprocessing operations on the video data, obtain image frames in the video data, and calculate the optical flow of adjacent frames to obtain optical flow frames;

[0011] Dividing the video data into a training sample set and a test sample set;

[0012] S2. Send the training sample set to the hybrid neural network for training. During the training process, the training results are verified by the test sample set. During the verification, the parameters are not updated and are saved as a weight file.

[0013] S3. Use the weight file with the highest verified accuracy to initialize the overall hybrid neural network, use the overall recognition accuracy of the test sample set as an indicator to judge the performance, and obtain the model with the highest accuracy as the final training model;

[0014] S4. Using a camera to collect a pending video sequence with human behavior, and performing a preprocessing operation on it, to obtain image frames and optical flow frames of the pending video sequence;

[0015] S5. Import the image frames and optical flow frames of the pending video sequence into the final training model to obtain the behavior in the pending video sequence.

[0016] Further preferably, in S2 and S4, the method of obtaining the image frame and the optical flow frame by preprocessing mainly includes:

[0017] s21. Extract one frame from every five frames of the video sequence to form an RGB image frame;

[0018] s22. Calculate the brightness change value at the moment by combining the gradient of all corresponding pixels of each two adjacent image frames in the RGB image frame data set with the time information;

[0019] s23. Track the corner points of brightness changes through a detection frame with 5*5 pixels to obtain optical flow images in the x and y directions, forming optical flow image frames in the x and y directions.

[0020] Further preferably, the hybrid neural network in S2 is a network model with ResNet101 as the backbone, and the 3*3 convolution of ResNet101 is replaced by a cot block module to form a combined network of convolutional neural network and attention mechanism.

[0021] Further preferably, in S2, the training process includes:

[0022] All image frames and optical flow frame data in the video data are defined as , where C, H, and W are the number of channels, height, and width of the image respectively;

[0023] The spatial resolution of the sampled frame is cropped to a size of 224×224 and randomly flipped and converted into a tensor input to the hybrid neural network;

[0024] The batch size during training is set to 64; the network input size of the image frame is (64, 3, 224, 224), and the output is (64, 101); the network input size of the optical flow frame is (64, 20, 224, 224), and the output is (64, 101); 64 is the batch size, 3 is the number of channels of the image frame, 20 is the number of channels of the optical flow frame, and 224 and 224 are the spatial resolution sizes of the image;

[0025] Back propagation is used to update parameters during training;

[0026] The classifier is implemented as a fully connected layer with its output dimension set to the total number of categories.

[0027] The present invention has at least the following beneficial effects:

[0028] The present invention fully utilizes the local receptive field, weight sharing, and time or space sub-sampling feature extraction methods of convolutional neural networks, the long-distance modeling ability of the attention mechanism, and the interactive advantages between features through the fusion of static and dynamic contexts, and the LSTM structure of the recurrent neural network to extract time series information in addition to motion information through a hybrid neural network.

[0029] The present invention deeply integrates the above neural network structures to fully extract the scene information, motion information and time series information of the video sequence, thereby forming a recognition accuracy and robustness that is superior to any single network structure;

[0030] The neural network model trained as above can be used for identifying human behaviors in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0032] Figure 1 It is the overall flow chart of the present invention;

[0033] Figure 2 Schematic diagram of the basic structure of hybrid neural network for behavior recognition.

[0034] Figure 3 Schematic diagram of the feature extraction process of the self-attention mechanism. Among them, x is the original feature map, Q, K, V are all feature maps after 1*1 convolution, k1 is the static model obtained by group convolution, W is the full-fat matrix formed by the concatenation of k1 and Q, D is the dynamic context formed by the product of W and V, and Out is the fusion feature output of the static context k1 and the dynamic context.

[0035] Figure 4 To demonstrate the Figure 2 Schematic diagram of the processing results of each step. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] Please refer to Figure 1 ,The present invention is a human behavior recognition method based on hybrid neural network.

[0038] Specifically, first obtain a data sample set containing video data and labels and preprocess it to obtain image frames and calculate the optical flow of its adjacent frames and divide it into a training sample set and a test sample set; secondly, send the training data set into the hybrid neural network, set the learning rate to train the entire network, and iterate 100 times in total. When it reaches 90 times, the learning rate is reduced to 1 / 10 of the original to accelerate the convergence of the network. Use the classifier to generate the recognition score, and update the network parameters through back propagation. The model is verified every two trainings, and the verification data set is used as the test data set, and the parameters are saved as a weight file; finally, the model with the highest accuracy is used to verify the test sample set video to obtain the recognition result, and the overall recognition accuracy of the test sample set is used as an indicator to judge the performance.

[0039] The following is combined with Figure 2 and Figure 4 The present invention is described in detail:

[0040] (1) The dataset used in this invention is UCF101, which contains 101 categories and 13,320 videos. This dataset is composed of videos uploaded by users, which contain camera motion and cluttered backgrounds. It contains 101 categories and 13,320 videos. The videos in each category are divided into 25 groups, each of which can contain 4-7 videos. Videos from the same group have some similar features, such as background, characters, etc.

[0041] (2) The present invention uses a dual-stream architecture that integrates a spatiotemporal network. A frame of the preprocessed image is randomly extracted and input into the spatial stream network to extract its scene information. Ten frames of the calculated optical flow in the x and y directions are taken and superimposed and sent to the temporal stream network to extract its motion information and timing information.

[0042] The specific implementation process of the experiment of the present invention is as follows:

[0043] (1) The network model uses ResNet101 as the backbone. The jump connection of ResNet101 can effectively avoid the gradient vanishing and gradient exploding problems caused by the network being too deep. The 3*3 convolution of ResNet101 is replaced by the cot block module to form a combination of convolutional neural network and attention mechanism. The structural diagram of the cot block module is shown in Figure 3.

[0044] (2) Use Matlab to extract frames from the original video data and calculate the optical flow in the x and y directions, and save them as RGB and FLOW files respectively. Obtain the name of the original video folder as the label of the sample data set. Divide the preprocessed sample data set into a training sample set and a test sample set in a ratio of 7:3 for training and testing of the hybrid neural network respectively.

[0045] (3) All image frames and optical flow frame data in the UCF101 data sample set are defined as , where C, H, and W are the number of channels, height, and width of the image, respectively. The spatial resolution of the sampled frame is cropped to a size of 224×224 and randomly flipped before being converted into a tensor and input into the overall hybrid neural network. The batch size in the overall network learning process is set to 64, the RGB network input size is (64, 3, 224, 224), and the output is (64, 101); the Flow network input size is (64, 20, 224, 224), and the output is (64, 101), where 64 is the batch size, 3 is the number of channels of the RGB frame, 20 is the number of channels of the optical flow frame, and 224 and 224 are the spatial resolution sizes of the image.

[0046] (4) The training was conducted 350 times in total. The initial learning rate was set to 0.001 and was reduced by 10 times after 300 training times. Back propagation was used to update the parameters during the training process. The classifier was implemented as a fully connected layer with an output dimension set to the total number of categories. The total number of categories in the UCF101 data sample set was 101. The training was validated on the test sample set after every two training sessions. The parameters were not updated during the validation and were saved as a weight file in the *.pth format.

[0047] (5) Use the weight file with the highest verified accuracy to initialize the overall network, test it on the test sample set, and use the full-size image as the input frame size with a spatial resolution of 256×256. Send the sampled frame as input to the overall network to obtain the recognition result. Compare the recognition result with the label corresponding to the video data. If the two are the same, the recognition is correct. Otherwise, the recognition is incorrect. Count the videos that are correctly recognized. Finally, obtain the ratio of the number of correctly recognized videos to the number of videos in the test sample set. This ratio is the recognition accuracy. The recognition accuracy of the test sample set is used as an indicator to judge the performance.

[0048] According to the above steps, the present invention proposes a method for video human behavior recognition based on a convolutional neural network, an attention mechanism and a recurrent neural network, using ResNet101 as the backbone model to insert an attention mechanism and LSTM to construct a network model. It can be seen that the method of the present invention can comprehensively extract scene information, motion information and timing information in the video, and achieves the highest recognition accuracy of 97.3% among similar methods on the UCF101 data sample set. While achieving a high recognition accuracy, the network parameters are greatly reduced, so it is more lightweight when applied to actual scenarios.

[0049] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A human behavior recognition method based on a hybrid neural network, characterized in that: The following steps are involved: S1. Construct training sample set and test sample set: Obtain video data including behavior labels, perform preprocessing operations on the video data, obtain image frames in the video data, and calculate the optical flow of adjacent frames to obtain optical flow frames; Dividing the video data into a training sample set and a test sample set; S2. Send the training sample set to the hybrid neural network for training. During the training process, the training results are verified by the test sample set. During the verification, the parameters are not updated and are saved as a weight file. The hybrid neural network is a dual-stream architecture that integrates a spatiotemporal network. A frame of the preprocessed image is randomly extracted and input into the spatial stream network to extract its scene information. The calculated optical flow frames in the x and y directions are superimposed and sent to the temporal stream network to extract its motion information and timing information. The spatial stream network includes a network model with ResNet101 as the backbone, and replaces the 3*3 convolution of ResNet101 with a cotblock module to form a network combining a convolutional neural network and an attention mechanism; The time stream network includes the recurrent neural network LSTM; S3. Use the weight file with the highest verified accuracy to initialize the overall hybrid neural network, use the overall recognition accuracy of the test sample set as an indicator to judge the performance, and obtain the model with the highest accuracy as the final training model; S4. Using a camera to collect a pending video sequence with human behavior, and performing a preprocessing operation on it, to obtain image frames and optical flow frames of the pending video sequence; S5. Import the image frames and optical flow frames of the pending video sequence into the final training model to obtain the behavior in the pending video sequence.

2. The method for human behavior recognition based on hybrid neural network according to claim 1, characterized in that: in, In S2 and S4, the method of obtaining the image frame and the optical flow frame by preprocessing mainly includes: s21. Extract one frame from every five frames of the video sequence to form an RGB image frame; s22. Calculate the brightness change value at the moment by combining the gradient of all corresponding pixels of each two adjacent image frames in the RGB image frame data set with the time information; s23. Track the corner points of brightness changes through a detection box with 5*5 pixels, obtain the optical flow images in the x and y directions, and form the optical flow frames in the x and y directions.

3. A method for human behavior recognition based on a hybrid neural network according to claim 1, characterized in that: In S2, the training process includes: All image frames and optical flow frame data in the video data are defined as , where C, H, and W are the number of channels, height, and width of the image respectively; The spatial resolution of the sampled frame is cropped to a size of 224×224 and randomly flipped and converted into a tensor input to the hybrid neural network; The batch size during training is set to 64; the network input size of the image frame is (64, 3, 224, 224), and the output is (64, 101); the network input size of the optical flow frame is (64, 20, 224, 224), and the output is (64, 101); 64 is the batch size, 3 is the number of channels of the image frame, 20 is the number of channels of the optical flow frame, and 224 and 224 are the spatial resolution sizes of the image; Back propagation is used to update parameters during training; The classifier is implemented as a fully connected layer with its output dimension set to the total number of categories.

Images