A method and system for deep learning-based human behavior recognition using ZYNQ

Abstract

This invention discloses a method and system for deep learning-based human behavior recognition using ZYNQ, comprising the following steps: 1) acquiring a video dataset of human behavior; 2) performing data preprocessing, processing the human estimation information collected from the video dataset in step 1) using a human pose estimation algorithm, and dividing the collected data into a training set and a test set; 3) constructing a human behavior recognition model, using a designed deep learning neural network to extract and process features from the dataset in step 2), thereby training the model; 4) setting training parameters and training the designed deep learning network; 5) according to actual needs, using ZYNQ as the core platform, transferring the trained model, and finally deploying it to achieve online human behavior recognition. This invention is beneficial for recognizing complex, long-term reasoning behaviors in video scenarios and has practical application value.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition, and in particular relates to a method and system for deep learning-based human behavior recognition based on ZYNQ. Background Technology

[0002] Visual information, as one of the most readily available forms of information in real life, has seen a surge of research and applications, particularly in video. Among these areas of focus, human behavior recognition within videos has become a key area of ​​interest, with applications such as intelligent elderly care, sports event judging, and sign language recognition. Compared to interactions between humans and objects, these tasks focus more on the movements of the human body itself. Since videos consist of multiple frames with temporal relationships, simultaneously capturing the spatial semantic features of the images within the video and the temporal motion features between multiple frames has become crucial for human behavior recognition.

[0003] The biggest difference between human behavior recognition in videos and image classification is that video classification requires temporal modeling. Therefore, various behavior recognition networks have explored different combinations of temporal and spatial modeling. The Two Stream CNN, proposed in 2014, attempted to use a two-stream network: one for spatial feature representation and the other for temporal feature representation. The spatial and temporal features extracted by the two networks were then fused to obtain the classification. However, this method suffers from separating the extraction of spatiotemporal features, ignoring the connection between them, and lacking the ability to capture long-range temporal information. Subsequently, many temporal modeling models emerged. The TSN model, proposed in 2016, divides the video into multiple segments and performs sparse sampling. Predictions are made on each segment, and the results are fused to obtain a video-level prediction. This mechanism enables the model to capture long-range temporal information; however, this method still does not link the temporal and spatial dimensions of behavioral features, lacking spatiotemporal fusion capabilities. Since the introduction of C3D in 2015, 3D Convolutional Neural Networks (3D CNNs) have been used for spatiotemporal feature extraction, including models such as R3D, I3D, NL-I3D, SlowFast, NL-SlowFast, and X3D. 3D convolution expands spatial semantic features into spatiotemporal features by adding a temporal dimension to the convolutional kernel. Although 3D CNN models can effectively fuse spatiotemporal features, they learn features only within a small sliding window rather than across the entire video. Therefore, they struggle to achieve video-level predictions. Furthermore, 3D CNN models are computationally expensive, demanding on high-performance computing platforms, difficult to train, and very time-consuming in actual inference and prediction processes.

[0004] Human behavior can be divided into two categories. One type of behavior can be determined from a single static frame in a video; this is called non-inference behavior. The other type requires identifying the features of multiple frames in a video and understanding their motion relationships; this is called inference-dependent behavior. Identifying inference-dependent behavior places higher demands on the temporal relationship modeling of human behavior recognition models. Therefore, human behavior recognition methods applied to video scenarios must consider not only their spatiotemporal feature fusion capabilities but also their feature set. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for deep learning-based human behavior recognition based on ZYNQ, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the method provided by the present invention includes the following steps:

[0007] Step 1: Obtain a video dataset of human behavior;

[0008] Step 2: Perform data preprocessing. Process the human estimation information collected from the video dataset in Step 1 using the human pose estimation algorithm, and divide the collected data into training set and test set.

[0009] Step 3: Construct a human behavior recognition model. Use the designed deep learning neural network to extract and process features from the dataset in Step 2, thereby training the model. The deep learning neural network is developed by modifying the bidirectional recurrent gate control network, adding a spatiotemporal attention mechanism to enhance the network's ability to process information in both time and space. Furthermore, by adding another branch to the CNN network, human feature information is extracted, and the two feature sets are fused. This increases the number of feature extraction methods and yields more complete feature information.

[0010] Step 4: Set training parameters and train the designed deep learning network;

[0011] Step 5: Based on actual needs, using ZYNQ as the core board, transfer the trained model and finally complete the deployment to achieve online human behavior recognition.

[0012] Furthermore, in step one, the publicly available human motion dataset of HMDB51 is obtained, and the obtained dataset is saved in different file directories according to different actions, and labeled with action names.

[0013] Furthermore, in step two, the data is processed using the OpenPose algorithm to extract skeletal key point information of human behavior, and the data is labeled and classified according to time sequence and action type. The obtained dataset is then divided into training set and test set in an 8:2 ratio.

[0014] Furthermore, the specific steps for constructing the human behavior recognition model in step three are as follows:

[0015] 1) Design a bidirectional recurrent gate neural network with a spatiotemporal attention mechanism

[0016] The bidirectional recurrent gate neural network for spatiotemporal attention mechanism mainly includes: temporal and spatial attention mechanism modules and a bidirectional GRU neural network;

[0017] 2) Designing CNN network branches

[0018] The main purpose of designing CNN networks is to generate different feature information and improve the accuracy of feature information.

[0019] 3) Design a multi-scale feature fusion module

[0020] The multi-scale feature fusion module is composed of a feature fusion module connected together and an upsampling module connected together;

[0021] 4) Design the classifier module

[0022] Video images are classified using a classifier and a human behavior recognition model.

[0023] 5) Construct a model network based on a bidirectional recurrent gate network. This network mainly includes a spatiotemporal attention mechanism, CNN network branches, multi-scale feature fusion, and a classifier module. A deep learning neural network is used to extract feature vectors from the preprocessed video images. The feature vectors are then input into the classifier of the human behavior recognition model. The video images are classified by the classifier and the human behavior recognition model to obtain the classification results of the video images with updated sub-behavior category labels. The classification results of the video images with updated sub-behavior category labels are then categorized and merged according to the major categories in the sample library to obtain the behavior category to which the human behavior in the video image belongs.

[0024] Furthermore, in step four, the activation function and loss function for model training are estimated using the difference between the real image and the image after passing through the deep learning-based convolutional neural network. At the same time, an optimization function needs to be selected to iteratively train the deep learning convolutional neural network. During training, important parameters such as the learning rate, the number of iterations, and the batch of training data need to be set.

[0025] Furthermore, in step five, to implement human behavior recognition on ZYNQ, the trained model needs to be converted into a model adapted for ARM execution. The converted model is then deployed to ZYNQ, and the algorithm is run on the ARM side. After deployment, the camera on the ZYNQ board can be driven to collect images of human behavior in real time. The collected images are then recognized by the algorithm model to obtain real-time human action information.

[0026] The specific beneficial effects of this invention are as follows:

[0027] (1) The advantage of this invention is that it uses ZYNQ as the processor for the algorithm to perform the algorithm processing.

[0028] ZYNQ is a system composed of ARM and FPGA. Using ZYNQ, the computing units on the FPGA side can be used to combine deep learning networks to achieve hardware acceleration and improve the running speed of algorithms.

[0029] (2) The advantage of this invention lies in the design of deep learning neural networks.

[0030] The advantage of this deep learning neural network lies in its ability to obtain temporal and spatial correlation information in human behavior, acquire feature information in different dimensions, and fully extract the required feature information.

[0031] (3) The advantage of this invention is that the structure is modularized and the backbone network of the feature extraction, spatiotemporal feature extraction and feature fusion modules can be flexibly replaced with other networks. Lightweight or high-precision networks can be selected according to computing resources. Attached Figure Description

[0032] Figure 1 This is a diagram illustrating the overall design framework of the method of the present invention;

[0033] Figure 2 A diagram showing the model structure of the deep learning network designed for this invention;

[0034] Figure 3 This is a framework diagram of the residual module designed for this invention. Detailed Implementation

[0035] The present invention will be further described below with reference to specific embodiments.

[0036] This embodiment provides a method and system for human behavior recognition based on ZYNQ deep learning, which includes the following steps:

[0037] Step 1: Obtain a video dataset of human behavior, as follows:

[0038] 1) Obtain public open-source video datasets by downloading HMDB51, UCF101, and Jester video datasets;

[0039] 2) The obtained public open-source video datasets are organized according to a custom file structure standard. The first-level folder directory of the HMDB51 video dataset is named according to the category to which human behavior belongs, and each folder contains videos in the .avi format belonging to that category.

[0040] Step Two: Extract key information from the videos in the video set from Step One to create a dataset, as detailed below:

[0041] 1) First, the video dataset is processed into video files. The Openpose algorithm is used to extract video frames from each video. The extracted video frame data is then used to extract key point information of the human body. From Openpose, information on 18 key points of the human body can be obtained. The two-dimensional coordinates of the key point information are converted into one-dimensional coordinate information and labeled.

[0042] 2) Based on the labeled dataset information, group the consecutive images in chronological order, label the grouped datasets again, and finally randomly divide the labeled datasets into training and test sets in an 8:2 ratio. Then convert them into .csv files and save them so that each group of data corresponds to a label.

[0043] Step 3: Design a deep learning neural network model

[0044] 1) Design a bidirectional recurrent gate neural network with a spatiotemporal attention mechanism

[0045] The bidirectional recurrent gate neural network with spatiotemporal attention mechanism mainly includes: temporal and spatial attention mechanism modules and bidirectional GRU neural network. These two modules mainly use the bidirectional GRU module as the main framework, and add the spatiotemporal attention mechanism module on this basis. The temporal attention mechanism mainly uses the GRU network as the main body, extracts feature information through fully connected layers, and adds the ReLU function after the fully connected layers. The spatial attention mechanism uses the tanh function and adds a fully connected layer after the function. Then, it is fused with the bidirectional GRU network to obtain the network framework of this branch.

[0046] 2) Designing CNN network branches

[0047] The main purpose of designing a CNN network is to generate different feature information and improve the accuracy of the feature information. By using two 1*1 convolutional kernels as residual modules, if the network input and the filters of the convolutional kernels are the same, there is no need to add a third convolutional kernel. If they are different, a third convolutional kernel needs to be added to process them. The activation function of the residual network is ReLU. In the design process of the CNN model, three residual blocks are used to process the input data, and finally, the network results are processed using an FC layer.

[0048] 3) Design a multi-scale feature fusion module

[0049] The multi-scale feature fusion module consists of a feature fusion module connected to an upsampling module. The convolution kernel size of the feature fusion module is 1*1, the stride is 1, and the padding is 0. The Concat function is used to fuse the final output results, and the fused features are then fed into the classifier module.

[0050] 4) Design the classifier module

[0051] The principle behind classifying video images using classifiers and human behavior recognition models is as follows:

[0052] SoftMax is constructed as the activation function of this invention, which adds a probability to each classification. The expression of the SoftMax function is as follows:

[0053]

[0054] in The meaning refers to the output value of the function at the i-th node, where is the total number of categories of the output nodes mentioned above. For example, the original input of a neural network... , , ...This function can transform the output value of a single node into a probability distribution.

[0055] The loss function corresponding to the Softmax activation function is the cross-entropy function, and its expression is as follows:

[0056]

[0057] 5) Construct a model network based on a bidirectional recurrent gate network. This network mainly includes a spatiotemporal attention mechanism, CNN network branches, multi-scale feature fusion, and a classifier module. A deep learning neural network is used to extract feature vectors from the preprocessed video images. The feature vectors are then input into the classifier of the human behavior recognition model. The video images are classified by the classifier and the human behavior recognition model to obtain the classification results of the video images with updated sub-behavior category labels. The classification results of the video images with updated sub-behavior category labels are then categorized and merged according to the major categories in the sample library to obtain the behavior category to which the human behavior in the video image belongs.

[0058] Step 4: The network framework designed in Step 3 is then trained by setting training parameters. The initial learning rate is set to 0.1, the number of iterations is set to 25, and the batch size is set to 256. The SGD algorithm is then used for iterative training of the network. The dataset is the dataset obtained in Step 2.

[0059] Step 5: Port and deploy the model trained in Step 4. The specific steps are as follows:

[0060] 1) Perform framework conversion on the trained model, convert the obtained TensorFlow model into an ONNX model, deploy the converted ONNX model on the ZYNQ main control circuit, and call the model through the algorithm.

[0061] 2) Image acquisition is performed using a camera driven by ZYNQ. The acquired images are processed in real time using the OpenPose algorithm. The processed data is then used to perform real-time human behavior judgment using a pre-deployed model to obtain accurate recognition results.

[0062] In summary, this invention proposes a method and system for human behavior recognition based on ZYNQ deep learning. It can effectively reduce data redundancy, capture long-range temporal information, and obtain more complete human behavior features by fusing multiple feature extraction models. Through the fusion and processing of feature information, the differences in human behavior are highlighted, which is beneficial for the recognition of human behavior. Some modules can be flexibly replaced according to the scenario. It has broad research and practical application value and is worth promoting.

Claims

1.A method for human behavior recognition based on deep learning of ZYNQ, characterized in that, It comprises the following steps: Step one: obtain the video dataset of human behavior; Step two: data preprocessing, the video dataset in step one is processed by the human pose estimation algorithm to collect human estimation information, and the collected data is divided into training set and test set; It specifically includes: in step two, the openpose algorithm is used to extract the key point information of the human behavior skeleton, the two-dimensional coordinates of the key point information are converted into one-dimensional coordinate information, and the information is labeled and classified according to time sequence and action type, and the obtained dataset is divided into training set and test set according to the ratio of 8:2; Step three: constructing a human behavior recognition model, using the designed deep learning neural network to extract and process the data in step two, and realizing the training of the model; wherein the deep learning neural network is modified by a bidirectional recurrent gate control network, a space-time attention mechanism is added, the processing capacity of the network for time and space information is enhanced, and another branch of CNN network is added to extract human feature information, and the two features are fused, which aims to increase the feature extraction method and obtain more complete feature information; The specific steps of constructing the human behavior recognition model in step three are as follows: 1) design a bidirectional recurrent gate neural network with space-time attention mechanism, which includes time and space attention mechanism module and bidirectional GRU neural network; 2) design a CNN network branch, which is designed to produce different feature information and improve the accuracy of feature information; 3) design a multi-scale feature fusion module, which is connected by a feature fusion module and an up-sampling module; 4) design a classifier module, which classifies and processes video images through the classifier and human behavior recognition model 5) build a model network based on bidirectional recurrent gate network as the main body, which includes space-time attention mechanism, CNN network branch, multi-scale feature fusion and classifier module, adopts deep learning neural network to extract feature vector of preprocessed video image; input the feature vector into the classifier of human behavior recognition model, classify and process the video image through the classifier and human behavior recognition model, and get the classification result of the video image with updated sub-behavior class label; classify and merge the classification result of the video image with updated sub-behavior class label according to the classification of the specimen library, and get the behavior class of the human behavior in the video image; Step four: set the training parameters and train the designed deep learning network; Step five: according to the actual needs, take ZYNQ as the core board, migrate the trained model, finally complete the deployment, and realize online human behavior recognition. 2.The method of claim 1, wherein, In step one, the public human motion dataset of HMDB51 is obtained, the obtained dataset is saved in different file directories according to different actions, and is labeled with action name. 3.The method of claim 1, wherein, The activation function and the loss function of the training of the model in the fourth step are estimated by the difference between the real image and the image after the deep learning-based convolutional neural network. Meanwhile, an optimization function needs to be selected to iteratively train the deep learning convolutional neural network. During training, training parameters need to be set, including learning rate, number of iterations, and batch size of input training data. 4.The method of claim 1, wherein, In the fifth step, human behavior recognition on ZYNQ requires converting the trained model into a model suitable for ARM execution, deploying the converted model to ZYNQ, running the algorithm on the ARM side, and after deployment, driving the camera on the ZYNQ board to collect real-time human behavior image, and identifying the collected image through the algorithm model to obtain real-time human motion information.

Images