Human body behavior recognition method and recognition system based on deep learning

Abstract

The invention discloses a deep learning-based human behavior recognition method and recognition system, and the method is characterized in that the method comprises the following steps: S1, data collection: collecting video data containing a plurality of human behaviors through a high-definition camera or a video database, and covering different scenes, illumination conditions and human postures; s2, data preprocessing: firstly extracting a three-dimensional coordinate sequence of human skeleton key points by using a Medape pre-training model, and then performing mean filtering and normalization preprocessing on the collected data to improve the generalization ability of the model; according to the method, the advantages of the DCN and the LSTM are combined, the improved double-flow convolution model is added with the space-time soft attention mechanism, the problems that space-time feature extraction is insufficient and feature design is complex in human body action recognition are effectively solved, the lightweight degree of the model is remarkably improved, and the human body action recognition accuracy is improved. And a new method is provided for research and application of human behavior recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of human behavior recognition, and in particular to a human behavior recognition method and recognition system based on deep learning. Background Art

[0002] Human behavior recognition is widely used in many fields such as virtual reality, smart home control, and human-computer interaction. In recent years, with the development of deep learning, compared with the traditional method of manually designing features, deep learning methods can automatically learn features from data, especially on large-scale data sets, and are more robust. The data sources for human behavior recognition mainly include pictures, video streams, wearable sensor devices, and bone sequences. Considering that the actual detection process is easily affected by factors such as light changes and object occlusion, the human skeleton sequence information is used as the object of feature extraction. With the continuous improvement of hardware equipment, the Kinect depth camera can directly output the key point data of the human skeleton.

[0003] With the continuous development of deep learning, the use of ordinary RGB cameras and human posture recognition technologies such as Openpose and Alphapose can directly obtain motion information such as the optical flow posture of the human skeleton, which makes the research on human behavior recognition based on skeletons more efficient. The encoding form of skeleton data often adopts graph structures, such as ST-GCN and TSN. However, the existing methods still have the problem of insufficient extraction of action features, low recognition accuracy under complex backgrounds and changing lighting conditions, and complex models and insufficient lightweight. Summary of the invention

[0004] In view of the above problems, the present invention proposes a human behavior recognition method and recognition system based on deep learning, which uses the MediaPipe pre-trained model for data preprocessing, and introduces a two-stream deformable convolutional neural network (DCN) and a long short-term memory network (LSTM) model to learn the continuous time series features of human behavior video frames. Through this deep learning framework method, the system can more accurately identify and classify different actions while significantly improving the lightweight degree of the model.

[0005] The present invention proposes a human behavior recognition method and recognition system based on deep learning, which is characterized by comprising the following steps:

[0006] S1: Data collection: Use high-definition cameras or video databases to collect video data containing a variety of human behaviors, covering different scenes, lighting conditions, and human postures;

[0007] S2: Data preprocessing: Use the MediaPipe pre-trained model to extract the coordinate data of the key points of the human skeleton and preprocess the collected data;

[0008] S3: Feature extraction: A two-stream deep learning model that integrates a deformable convolutional neural network and a long short-term memory network is used to extract temporal features from the preprocessed data and learn the spatial hierarchical features in the image sequence;

[0009] S4: Model training: Use labeled datasets to train the above deep learning model so that the model can recognize and learn different human behaviors;

[0010] S5: Behavior recognition: Apply the trained model to real-time camera or video data to identify and classify human behaviors in real time;

[0011] S6: Post-processing: Post-processing is performed based on the real-time recognition and classification results to generate human-computer interaction instructions;

[0012] The data preprocessing in S2 includes video frame extraction, mean filtering and normalization to enhance the generalization ability of the model. The spatial position change information of N skeleton joints of the human body within a duration T is the human skeleton behavior data:

[0013]

[0014] Among them, S t is the human skeleton behavior data, a 3D joint coordinate is J = (x, y, z), N is the number of joints of each skeleton, and the filtering process of the spatial coordinate data of a certain point of the original skeleton data is:

[0015]

[0016] Where S k is the coordinate data of the skeleton key points before filtering, and the mean of 5 continuous data is taken as the filtered data S t ;

[0017] The normalization algorithm normalizes the coordinates of the key points, and the normalized coordinate data is:

[0018]

[0019] P1 represents the initial position of the wrist, P1 represents the initial position of the shoulder joint, ||P1-P2|| represents the length of a person's arm, x 1.t represents the position of the wrist on the x-axis at time t, x 2.t Represents the position of the shoulder on the x-axis at time t;

[0020] The deep learning model in S3 is a two-stream deformable convolutional neural network (DCN) and a long short-term memory network (LSTM) model, in which the deformable convolution introduces a 2D offset variable Δp in the convolution operation. n, so that the convolution kernel can be adjusted according to the shape of the input data to adapt to the diversity and complexity of the action, effectively capture the tiny movement of the action and the change of the action shape, integrate more information in space, improve the ability of feature extraction and recognition accuracy, after the offset, each point on the output feature map can be expressed as:

[0021]

[0022] in, represents the output feature map after the L-th layer convolution operation, the 0th layer is the input S, the i-th output feature value, f(.) is the activation function (ReLU is used here), is the convolution kernel weight at position p of the Lth layer, S[.] is the input feature map data, p0 is the center position of the current convolution operation, is the learnable offset of the Lth layer at position p, is the bias of the Lth layer, G is the sampling position set of the standard convolution kernel,

[0023] G={(-1,-1),(-1,0),(-1,1),(0,-1),(0,0),(0,1),(1,-1),(1,0),(1,1)},

[0024]

[0025] in, is the output feature map after the l-th layer pooling operation, is the output feature map after the l-th layer convolution operation, and MaxPool is the maximum pooling operation;

[0026] The long short-term memory neural network is used to extract temporal features and classify them. The spatial attention model can automatically explore and utilize the importance differences of different joints. Each joint in the frame is assigned a different weight according to the joint selection gate. The importance score of the Nth joint can be expressed as:

[0027]

[0028] Among them, U I , W SI and W hI is the learnable parameter matrix, b I , b uI is the bias vector. is the hidden variable of the LSTM layer;

[0029] For the Nth joint point, the normalized activation value of the joint point selection gate is:

[0030]

[0031] The input to the main LSTM network is conditioned as:

[0032] S' t =(s' t,1 ,…,s' t,N ) T ,s' t.N =α t.n ·S t.N ,

[0033] Different frames are assigned different attention weights β t ,

[0034]

[0035] Among them, s t is the input feature vector of the current time step t, w x* With w h* is the learned weight matrix, is the hidden state of the previous time step t-1, b * is a bias term used to adjust the output value, and RELU is the activation function;

[0036] The two-stream network is fused in time and space, and the loss function is the cross entropy function. The probability that the current predicted sample accounts for all categories in the data set is:

[0037]

[0038] in, and are the predicted probability vectors of the spatial stream and the temporal stream for the current input, λ is a constant between 0 and 1, and n is the number of categories;

[0039] S4: Model training: Use labeled datasets to train deep learning models so that the models can recognize and learn different human behaviors;

[0040] S5: Behavior recognition: Apply the trained model to the image or video data captured by the real-time camera to recognize and classify human behavior in real time;

[0041] S6 uses behavior continuity analysis, abnormal behavior detection, and behavior modification based on contextual information to improve the accuracy and practicality of recognition;

[0042] An information data processing terminal for realizing a human behavior recognition method and recognition system based on deep learning;

[0043] A computer-readable storage medium includes instructions, which, when executed on a computer, enable the computer to execute a human behavior recognition method and recognition system based on deep learning.

[0044] Beneficial effects of the present invention:

[0045] The present invention realizes the accurate recognition process of human behavior through sophisticated network architecture design, effective data preprocessing strategy, advanced training technology and optimization method; the improved two-stream convolutional model effectively solves the problems of insufficient spatiotemporal feature extraction and complex feature design in human action recognition, and achieves excellent performance on multiple data sets, solves the problems of insufficient spatiotemporal feature extraction and complex feature design under conditions of complex background, occlusion and lighting changes, and improves the lightweight degree of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of a human behavior recognition method and recognition system based on deep learning in the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0049] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0050] Example 1

[0051] A human behavior recognition method and recognition system based on deep learning, which uses the MediaPipe pre-trained model for data preprocessing, introduces a two-stream deformable convolutional neural network (DCN) and a long short-term memory network (LSTM) model to learn the continuous time series features of human behavior video frames. Through this deep learning framework method, the system can more accurately identify and classify different actions while significantly improving the lightweight of the model.

[0052] The present invention proposes a human behavior recognition method and recognition system based on deep learning, which is characterized by comprising the following steps:

[0053] S1: Data collection: Use high-definition cameras or video databases to collect video data containing a variety of human behaviors, covering different scenes, lighting conditions, and human postures;

[0054] S2: Data preprocessing: Use the MediaPipe pre-trained model to extract the coordinate data of the key points of the human skeleton and preprocess the collected data;

[0055] S3: Feature extraction: A two-stream deep learning model that integrates a deformable convolutional neural network and a long short-term memory network is used to extract temporal features from the preprocessed data and learn the spatial hierarchical features in the image sequence;

[0056] S4: Model training: Use labeled datasets to train the above deep learning model so that the model can recognize and learn different human behaviors;

[0057] S5: Behavior recognition: Apply the trained model to real-time camera or video data to identify and classify human behaviors in real time;

[0058] S6: Post-processing: Post-processing is performed based on the real-time recognition and classification results to generate human-computer interaction instructions;

[0059] The data preprocessing in S2 includes video frame extraction, mean filtering and normalization to enhance the generalization ability of the model. The spatial position change information of N skeleton joints of the human body within a duration T is the human skeleton behavior data:

[0060]

[0061] Among them, S t is the human skeleton behavior data, a 3D joint coordinate is J = (x, y, z), N is the number of joints of each skeleton, and the filtering process of the spatial coordinate data of a certain point of the original skeleton data is:

[0062]

[0063] Where S k is the coordinate data of the skeleton key points before filtering, and the mean of 5 continuous data is taken as the filtered data S t ;

[0064] The normalization algorithm normalizes the coordinates of the key points, and the normalized coordinate data is:

[0065]

[0066] P1 represents the initial position of the wrist, P1 represents the initial position of the shoulder joint, ||P1-P2|| represents the length of a person's arm, x 1.t represents the position of the wrist on the x-axis at time t, x 2.t Represents the position of the shoulder on the x-axis at time t;

[0067] The deep learning model in S3 is a two-stream deformable convolutional neural network (DCN) and a long short-term memory network (LSTM) model, in which the deformable convolution introduces a 2D offset variable Δp in the convolution operation. n , so that the convolution kernel can be adjusted according to the shape of the input data to adapt to the diversity and complexity of the action, effectively capture the tiny movement of the action and the change of the action shape, integrate more information in space, improve the ability of feature extraction and recognition accuracy, after the offset, each point on the output feature map can be expressed as:

[0068]

[0069] in, represents the output feature map after the L-th layer convolution operation, the 0th layer is the input S, the i-th output feature value, f(.) is the activation function (ReLU is used here), is the convolution kernel weight at position p of the Lth layer, S[.] is the input feature map data, p0 is the center position of the current convolution operation, is the learnable offset of the Lth layer at position p, is the bias of the Lth layer, G is the sampling position set of the standard convolution kernel,

[0070] G={(-1,-1),(-1,0),(-1,1),(0,-1),(0,0),(0,1),(1,-1),(1,0),(1,1)},

[0071]

[0072] in, is the output feature map after the l-th layer pooling operation, is the output feature map after the l-th layer convolution operation, and MaxPool is the maximum pooling operation;

[0073] The long short-term memory neural network is used to extract temporal features and classify them. The spatial attention model can automatically explore and utilize the importance differences of different joints. Each joint in the frame is assigned a different weight according to the joint selection gate. The importance score of the Nth joint can be expressed as:

[0074]

[0075] Among them, U I , W SI and W hI is the learnable parameter matrix, b I , b uI is the bias vector. is the hidden variable of the LSTM layer;

[0076] For the Nth joint point, the normalized activation value of the joint point selection gate is:

[0077]

[0078] The input to the main LSTM network is conditioned as:

[0079] S' t =(s' t,1 ,…,s' t,N ) T ,s' t.N =α t.n ·S t.N ,

[0080] Different frames are assigned different attention weights β t ,

[0081]

[0082] Among them, s t is the input feature vector of the current time step t, w x* With w h* is the learned weight matrix, is the hidden state of the previous time step t-1, b * is a bias term used to adjust the output value, and RELU is the activation function;

[0083] The two-stream network is fused in time and space, and the loss function is the cross entropy function. The probability that the current predicted sample accounts for all categories in the data set is:

[0084]

[0085] in, and are the predicted probability vectors of the spatial stream and the temporal stream for the current input, λ is a constant between 0 and 1, and n is the number of categories;

[0086] S4: Model training: Use labeled datasets to train deep learning models so that the models can recognize and learn different human behaviors;

[0087] S5: Behavior recognition: Apply the trained model to the image or video data captured by the real-time camera to recognize and classify human behavior in real time;

[0088] S6 uses behavior continuity analysis, abnormal behavior detection, and behavior modification based on contextual information to improve the accuracy and practicality of recognition;

[0089] An information data processing terminal for realizing a human behavior recognition method and recognition system based on deep learning;

[0090] A computer-readable storage medium includes instructions, which, when executed on a computer, enable the computer to execute a human behavior recognition method and recognition system based on deep learning.

[0091] Example 2

[0092] The Kinetics-700 dataset is used as training data. This dataset contains about 700 different human behavior categories, each with more than 600 video samples. Smaller datasets such as UCF101 and HMDB51 are used for verification and testing to evaluate the generalization ability of the model.

[0093] All video data were first converted to grayscale images to reduce the amount of computation and normalized to 224x224 pixels. To enhance the stability of the model, data augmentation techniques were used, including random cropping, rotation ±-30°, scaling 0.8-1.2 times, and horizontal flipping. The timeline was randomly sampled, and segments of different lengths were selected from each video for training.

[0094] The architecture of the variable convolutional neural network (DCN) consists of multiple convolution blocks, a 13-layer deep network, including 10 convolutional layers, 2 fully connected layers and 1 output layer. These 13 layers are divided into 6 groups, of which the 1st, 2nd, 3rd and 4th groups are convolutional layers. Each convolutional layer consists of 1 DCN, 1 CNN and 1 maximum pooling, which are used to extract the features of the input data. The 5th layer is a fully connected layer, which maps the human behavior features obtained after screening to the sample label space through a weight matrix. The model ends with the output layer, which uses the softmax classifier as the output layer to obtain real numbers with values ​​between 0 and 1, and normalizes them to ensure that the sum is 1;

[0095] The long short-term memory neural network with the attention mechanism consists of 1 LSTM layer, 2 fully connected layers and 1 normalization unit;

[0096] After the feature extraction network, add a fully connected layer whose number of neurons matches the number of behavior categories. This layer uses the ReLU activation function and is subsequently connected to a Softmax classifier to output the probability of each behavior category.

[0097] A 5-fold cross-validation strategy was used to evaluate the performance of the model. In each cross-validation, the dataset was randomly divided into training set, validation set, and test set with a division ratio of 8:1:1. The initial learning rate was set to 0.001 and dynamically adjusted according to the performance on the validation set after every 5 epochs. If the validation loss did not improve within 3 consecutive epochs, the learning rate was halved. The weight decay coefficient was set to 0.0001 to prevent overfitting. If the validation loss did not improve within 10 consecutive epochs, the training process was terminated early.

[0098] The model was pre-trained on a large video dataset, Kinetics-400. Then, the model was fine-tuned using Kinetics-700, and the parameters of the last few layers were adjusted to adapt to the new task and data distribution.

[0099] On the Kinetics-700 test set, the model achieved a Top-1 accuracy of 92.5%, which is about 5 percentage points higher than the baseline model (without the attention mechanism and DCN layer). On the UCF101 and HMDB51 datasets, the model also showed good generalization ability, reaching 96.8% and 78.2% Top-1 accuracy respectively. In addition, by optimizing the network structure and algorithm acceleration technology, the model achieved low computing resource consumption and recognition delay while maintaining high accuracy.

[0100] In summary, by combining the advantages of DCN and LSTM, the improved two-stream convolutional model effectively solves the problems of insufficient spatiotemporal feature extraction and complex feature design in human action recognition. The excellent performance of this model on the UCF101 and HMDB5 datasets not only verifies its effectiveness, but also provides a new method for the research and application of human action recognition.

[0101] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. The various components mentioned in the present invention are common technologies in the existing field. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A human behavior recognition method and recognition system based on deep learning, characterized in that: The following steps are involved: S1: Data collection: Use high-definition cameras or video databases to collect video data containing a variety of human behaviors, covering different scenes, lighting conditions, and human postures; S2: Data preprocessing: preprocess the collected data to improve the generalization ability of the model; S3: Feature extraction: Use deep learning models to extract features from preprocessed data and learn spatial hierarchical features in images; S4: Model training: Use labeled datasets to train deep learning models so that the models can recognize and learn different human behaviors; S5: Behavior recognition: Apply the trained model to the image or video data captured by the real-time camera to recognize and classify human behavior in real time; S6: Post-processing: Combining the results of target detection and posture recognition, post-process the recognition results to generate the final prediction result interactive action of the human behavior recognition model.

2. According to the method and system for human behavior recognition based on deep learning according to claim 1, it is characterized in that: Data preprocessing in S2 includes video frame extraction, mean filtering, and normalization operations to enhance the generalization ability of the model.

3. According to the method and system for human behavior recognition based on deep learning of claim 2, it is characterized in that: The deep learning model in S3 is a hybrid architecture of a deformable convolutional neural network and a long short-term memory neural network, which is used to capture both spatial and temporal features. The deformable convolutional neural network is responsible for extracting spatial features from continuous video frames, and the long short-term memory neural network is combined with a spatiotemporal attention mechanism to further process these features and capture longer temporal dependencies.

4. According to claim 3, a method and system for human behavior recognition based on deep learning, characterized in that: During the training process of S4, data enhancement and regularization techniques are used to prevent overfitting.

5. According to the method and system for human behavior recognition based on deep learning of claim 4, it is characterized in that: In S5, a fully connected layer or classifier is used to finally classify the extracted features and identify specific human behaviors.

6. The method and system for human behavior recognition based on deep learning according to claim 5, characterized in that: S6 uses behavior continuity analysis, abnormal behavior detection, and behavior correction based on contextual information to improve recognition accuracy and practicality.

7. An information data processing terminal, characterized in that: Used to implement a human behavior recognition method and recognition system based on deep learning as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that: It includes instructions, which, when executed on a computer, enable the computer to execute a human behavior recognition method and recognition system based on deep learning as described in any one of claims 1 to 6.

Images