A human motion state classification method based on improved convolutional neural network
By improving the convolutional neural network, combining biological experimental data and short-time Fourier transform, the human body's movement state classification model is optimized, and the problems of model simplification and overfitting in the existing technology are solved, achieving higher classification accuracy Sex and robustness.
Patent Information
- Application Number
- CN201910366410.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-05-05
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-05-05
AI Technical Summary
In the prior art, the classification model of human motion states is too simplified, making it difficult to accurately identify the real human motion state, and conventional convolutional neural networks are insufficient in the optimization of human motion state characteristics, resulting in complex classifier design and easy to overfit.
Based on the improved convolutional neural network and combined with biological experimental data, a more accurate human motion model was established. Through the structure of 4 convolutional layers, pooling layers and fully connected layers, LRelu activation function and short-time Fourier were used. Transform, optimize network structure and parameters, reduce calculation amount, prevent overfitting, and improve robustness.
It improves the accuracy and robustness of human movement state classification, reduces the number of network parameters, improves the training speed and classification accuracy, and avoids the complexity of the feature extraction process and the problem of model overfitting.
Smart Images

Figure CN110045348B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of human motion state classification based on radar detection, and in particular relates to a human motion state classification method based on an improved convolutional neural network. Background Art
[0002] Human target detection has important applications in anti-terrorism, post-disaster search and rescue, and other occasions. Radar-based human motion state classification technology has been a hot topic in recent years. For the human body in motion, life detection radar uses the swing of the human trunk and limbs to modulate the radar transmission signal micro-Doppler, extract the echo signal micro-Doppler features, and achieve detection of the human body and motion state identification. Since the modulation process of human motion on the echo signal is a non-stationary process, and the amplitude of limb swing is much larger than the vibration amplitude of the chest cavity, the harmonics are many and complex, and it is difficult to use traditional Fourier spectrum for analysis. Therefore, the focus of research is mainly on micro-Doppler feature extraction. For example, the University of Pennsylvania in the United States uses life detection radar to detect human activities behind a wall and performs Hilbert transform analysis on the echo signal. First, the signal and noise are separated by EMD decomposition to improve the signal-to-noise ratio, and then Hilbert spectrum analysis is performed. Experiments have shown that this spectrum analysis method has better time and frequency resolution than the traditional time-frequency analysis method, and can effectively extract the micro-Doppler features of human breathing, heartbeat, and gait. The University of California and the Netherlands Physical Electronic Laboratory jointly studied a method for estimating human gait parameters based on traditional features. Huang Binke et al. extracted the step frequency as a motion feature through the echo of the limbs, and used it as a feature quantity to identify the human motion state. Zhang Yi et al. analyzed the Doppler frequency of the hands and legs when the human body was walking, and used it as a feature for identification. However, the current models used for human motion state identification are mostly based on mathematical models, that is, using simple sine functions to simulate the motion trajectory of various parts of the human body. The human motion model is too simplistic and idealized, and has a low degree of consistency with the real human motion state. At the same time, the selection of micro-Doppler features for gait motion is usually achieved by using specific feature types. The process of manually extracting features is relatively complicated, and the algorithm complexity is high. When different classifiers are used for classification, the results may be quite different, and the selection of classifiers requires high requirements. Other human activity classification methods, such as using SVD, are simple linear classifiers implemented by observing the average movement speed of the human body, which has low robustness. Combining the advantages of deep learning, the use of convolutional neural networks for motion state identification can effectively simplify the feature extraction process and effectively avoid the problem of redesigning the classifier when the number of features changes. However, conventional convolutional neural networks do not optimize the network design for the characteristics of human motion states, so they require more data sets for training and more network layers, which easily leads to overfitting of the network.
[0003] The improved convolutional neural network of the present invention optimizes the network structure and parameters for the classification of human motion states. The results show that the improved convolutional neural network can improve the classification recognition rate and robustness. Summary of the invention
[0004] In response to the problems existing in the prior art, the present invention is based on biological experimental data, takes into account the relationship between the flexure angles of each joint and the positional relationship of each body part, and establishes a more accurate motion model of the human body. On this basis, a human motion state classification method based on an improved convolutional neural network is proposed for the classification of human motion states.
[0005] The present invention is implemented in this way: an improved convolutional neural network, wherein the improved convolutional neural network has 4 convolutional layers, each convolutional layer is followed by an LRelu activation function layer, the pooling layer after the first three convolutional layers adopts 2×2 maximum pooling, and the pooling layer after the last convolutional layer adopts 4×4 maximum pooling;
[0006] The first convolutional layer uses 128 5×5 convolution kernels with a pad of 2; the second and third convolutional layers use 128 3×3 convolution kernels with a pad of 1; the last convolutional layer uses 64 3×3 convolution kernels with a pad of 1;
[0007] There are two fully connected layers after the four convolutional layers; the first fully connected layer fully connects the 4×4×64 feature map and the 256×1 vector, and then activates it with the LRelu activation function. The second fully connected layer reduces the dimension to 2×1, and the output layer uses softmax to achieve binary classification.
[0008] Furthermore, the improved convolutional neural network consists of a convolutional layer, a pooling layer, and a fully connected layer;
[0009] Convolution layer, convolution kernel is used to convolve the input image to obtain the feature map of the input image;
[0010] Pooling layer reduces network parameters, reduces computational complexity, and prevents overfitting;
[0011] Fully connected layer and output layer, used to output network categories.
[0012] Another object of the present invention is to provide a human motion state classification method based on the improved convolutional neural network, the human motion state classification method comprising:
[0013] In the first step, the convolution layer uses 128 convolution kernels to convolve the input image to obtain 128 feature maps of the input image. Multiple feature maps express multiple features of the extracted image.
[0014] In the second step, due to the large similarity between adjacent pixels of the image and the high image redundancy, the feature map is pooled to reduce the image size and the number of parameters. It is then passed through three convolutional pooling layers to extract deeper features of the image.
[0015] In the third step, we finally get 64 4×4 feature maps. The first fully connected layer uses 256 neurons to fully connect with the feature map, and the second fully connected layer uses 2 neurons to fully connect with the first fully connected layer. After the network converges after training, the image category can be output according to the input image.
[0016] Further, the calculation formula of the radar echo signal of each part of the human body in the human body motion state classification method is:
[0017]
[0018] Where P t is the radar transmission power, L s is the transmission and reception loss, G r and G v are the transmitting and receiving antenna patterns, λ, f 0 are the wavelength and frequency of the radar signal, c is the speed of light, σ i The RCS of each scattering center of the human body target; calculate the echo of each part of the human body separately, and sum up the echo signals to calculate the total echo signal of the human body.
[0019] Furthermore, the time-frequency feature extraction of the human motion state classification method uses short-time Fourier transform to perform time-frequency analysis on the target radar echo; first, a window function is selected, and the signal is divided into many small time intervals by moving the window function along the time axis to ensure that the signal is a stable signal within the time interval, and the power spectrum at each moment is calculated. The short-time Fourier transform of the echo signal s(t) is:
[0020]
[0021] Where g(f) is the window function. Common window functions include Hamming window, Gaussian window and Hanning window. When g(t) = 1, the short-time Fourier transform is converted into the traditional Fourier transform. STFT can roughly represent the signal energy, which is defined as the square of the STFT modulus value, that is:
[0022] SPEC(t,w)=|STFT f (t,w)| 2 .
[0023] Another object of the present invention is to provide a training method for the improved convolutional neural network, the improved convolutional neural network training method comprising: after the network model is established, the network can be trained on a subset of the data set, the training set uses human actions (walking and running), and the verification set and the test set each use the actions of one person. For each action, the data set is randomly divided, so it will not be verified and tested on the action data set of the same person.
[0024] Network training is an iterative process. In each training, the loss and accuracy curves and the network's hyperparameters are adjusted in the optimal direction. The network training batch is 100, and it is tested on the validation set after every 500 iterations. From the beginning of training, the loss value on the validation set and the test set decreases exponentially while the accuracy increases.
[0025] Another object of the present invention is to provide a human motion state classification and processing platform using the improved convolutional neural network.
[0026] In summary, the advantages and positive effects of the present invention are as follows: First, the present invention simplifies the human target motion into a composite motion of a typical motion form, and proposes an empirical model of human walking and running based on the biomechanical experimental data measured by R.Boulic, M.Thalaman and D.Thalman. Based on the XYZ convention, the Euler rotation matrix is used to establish and improve the human motion animation model of the stationary stepping state and the walking and running state, avoiding the problem that the traditional gait model mostly uses a simple sine function to fit the motion of various parts of the human body, making the model too simplistic, idealized and too low in consistency with the real human walking and running. Secondly, based on the continuous wave signal, the overall radar echo model of the human walking and running target and the radar echo model of each part of the target are constructed, the instantaneous frequency characteristics and instantaneous motion characteristics of the moving human target are theoretically analyzed, the radar echo characteristics under different motion states of the human body are analyzed, and the corresponding time-frequency diagram is generated by short-time Fourier transform, which lays a theoretical foundation for the feature extraction of the moving human target and the recognition of the walking and running state of the human body. The time-frequency graph of human body movements (walking and running) is divided into a training set, a validation set and a test set, and the improved convolutional neural network is input for training, and the network parameters are adjusted to make the model converge so that the network can correctly classify the human body movement state. At the same time, the improved convolutional neural network of the present invention can reduce the number of network parameters, improve the training speed and classification accuracy, and improve the robustness compared with the conventional convolutional neural network. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a flow chart of a method for classifying human motion states based on an improved convolutional neural network provided in an embodiment of the present invention.
[0028] Figure 2 Schematic diagram of a human body model provided in an embodiment of the present invention (including 17 reference joint points and component lengths).
[0029] Figure 3 Schematic diagram of the RCS of an ellipsoid provided in an embodiment of the present invention.
[0030] Figure 4 It is a schematic diagram of the network structure of the improved convolutional neural network provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0032] The application principle of the present invention is described in detail below in conjunction with the accompanying drawings.
[0033] like Figure 1 As shown, the human motion state classification method based on the improved convolutional neural network provided by the embodiment of the present invention includes the following steps:
[0034] S101: Based on the measured biomechanical experimental data, an empirical model of human walking and running is proposed; and based on the XYZ convention, the Euler rotation matrix is used to establish and improve the human motion animation model of the state of marking time, walking and running;
[0035] S102: Secondly, based on the continuous wave signal, the overall radar echo model of the human walking and running target and the radar echo model of each part of the target were constructed. The radar echo characteristics of the human body in different motion states were analyzed using the Matlab simulation tool and the corresponding time-frequency diagram was generated through short-time Fourier transform, which laid a theoretical foundation for the feature extraction of moving human targets and the recognition of human walking and running states.
[0036] S103: Divide the time-frequency graph of human motion into a training set, a validation set, and a test set, input the improved convolutional neural network for training, and adjust the network parameters to make the model converge so that the network can correctly classify the human motion state.
[0037] The application principle of the present invention is further described below in conjunction with the accompanying drawings.
[0038] 1. Human gait model. In order to simulate human motion, a motion model describing the motion of interest and an object model describing the human body are required. The present invention establishes an empirical model of human walking and running based on the measured biomechanical experimental data, and establishes and improves the human dynamic motion model of the state of marking time and the state of walking and running based on the XYZ convention using the Euler rotation matrix. The model provides the three-dimensional spatial position and direction of each component of the human body walking in the form of a time function.
[0039] The three-dimensional orientation of a human body part can be determined by positioning these 17 joint points in the human body reference coordinate system centered on the spine origin, such as Figure 2As shown. The present invention uses a sphere to more realistically equate the head and human reference points (elbow joint, shoulder joint, knee joint, ankle joint), and uses an ellipsoid to equate the remaining body parts (trunk, upper arm, lower arm, shoulder, thigh, lower leg, etc.), and obtains the position of each body reference point in the gait model according to the motion trajectory function, the angle conversion relationship and the XYZ conventional Euler rotation matrix, and uses the joint flexion angle as the rotation angle and the body part length as the vector modulus. After the rotation transformation, the positions of the 17 reference points of the human body model can be obtained. In order to accurately calculate the positions of the 17 reference points of the human body based on multiple angles, the outermost angle must be considered first, that is, the left and right and front and back rotations of the trunk are processed first, and then the flexion calculations of the joints on the limbs are performed on this basis. Finally, after considering these angles, the translation is processed to finally establish a set of dynamic human gait models.
[0040] 2. Target radar echo model
[0041] After establishing the motion model of human walking and running, it is necessary to calculate the RCS (Radar Cross Section) of each body part. The present invention uses a sphere to more realistically equate the head and human reference points (elbow joint, shoulder joint, knee joint, ankle joint), and uses an ellipsoid to equate the remaining body parts (trunk, upper arm, lower arm, shoulder, thigh, calf, etc.). The schematic diagram of the radar cross section of the ellipsoid is shown in Figure 3 shown.
[0042] The RCS of a sphere is calculated as:
[0043]
[0044] In the formula, r sphere represents the radius of the sphere, and λ represents the radar reflection wavelength.
[0045] The RCS of an ellipsoid is calculated as:
[0046]
[0047] In the formula, a, b, and c represent the lengths of the three radii of the ellipsoid along the x, y, and z axes, respectively. The incident angle and azimuth reflect the orientation characteristics of the ellipsoid relative to the radar. In general, a is approximately equal to b, and formula (2) is simplified to:
[0048]
[0049] After finding the RCS of each part of the human body, the radar echo model of the human body multi-scattering center gait model can be established. The human body and the radar move towards each other on the same horizontal plane. According to the RCS of each scattering center of the human body and the echo delay τ from each scattering center to the radar i, the radar echo signals of various parts of the human body can be calculated in the following form:
[0050]
[0051] Where P t is the radar transmission power, L s is the transmission and reception loss, G r和 G v are the transmitting and receiving antenna patterns, λ, f 0 are the wavelength and frequency of the radar signal, c is the speed of light, σ i The RCS of each scattering center of the human body target is calculated. The echo of each part of the human body is calculated separately, and the total echo signal of the human body can be calculated by summing up the echo signals.
[0052] 3. Time-frequency feature extraction
[0053] Short-time Fourier transform is used to perform time-frequency analysis on the target radar echo. First, a window function is selected and the signal is divided into many small time intervals by moving the window function along the time axis to ensure that the signal is a stable signal within the time interval and calculate the power spectrum at each moment. The short-time Fourier transform of the echo signal s(t) is
[0054]
[0055] Where g(f) is the window function. Common window functions include Hamming window, Gaussian window and Hanning window. When g(t) = 1, the short-time Fourier transform is converted into the traditional Fourier transform. STFT can roughly represent the signal energy, which is defined as the square of the STFT modulus value, that is:
[0056] SPEC(t,w)=|STFT f (t,w)| 2 (6)
[0057] 4. Classification of human motion state based on improved convolutional neural network
[0058] Since the time-frequency graph does not contain as much target information as the optical image, such as the target's outline, texture, color, size, etc., when classifying the walking and running states of a person through the time-frequency graph, a very complex network structure and a large number of features are not required to achieve a high classification accuracy. Therefore, the present invention proposes the following improved convolutional neural network structure, such as Figure 4 shown.
[0059] There are 4 convolutional layers, each followed by an LRelu activation function layer. The pooling layer after the first three convolutional layers uses 2×2 maximum pooling, and the pooling layer after the last convolutional layer uses 4×4 maximum pooling. The first convolutional layer uses 128 5×5 convolution kernels with a pad of 2, the second and third convolutional layers use 128 3×3 convolution kernels with a pad of 1, and the last convolutional layer uses 64 3×3 convolution kernels with a pad of 1. There are 2 fully connected layers after the 4 convolutional layers. The first fully connected layer fully connects the 4×4×64 feature map with the 256×1 vector, and then activates it with the LRelu activation function. The second fully connected layer reduces the dimension to 2×1, and the output layer uses softmax to achieve binary classification.
[0060] Convolutional neural networks consist of convolutional layers, pooling layers, and fully connected layers. They can be used for image classification. Like other machine learning algorithms, the parameters of convolutional neural networks are learned by training on a large number of data sets, and do not need to be set manually. The convolutional layer uses convolution kernels to convolve the input image to obtain the feature map of the input image. Each pixel on the feature map is obtained by sliding convolution of the convolution kernel on the input image. There can be multiple convolution kernels in a convolutional layer to extract multiple features of the image. Although the input may be a three-channel image, after convolution with multiple convolution kernels, many feature maps can be obtained. Each feature map is the result of convolution of a convolution kernel with the input image. The number of feature maps obtained after convolution of the number of convolution kernels is the number of feature maps. After the convolution layer, the activation function layer is passed. The activation function is a nonlinear function that can increase the nonlinear expression ability of the network. Since the similarity between adjacent pixels of the image is high and the redundancy is large, the pooling layer can reduce the network parameters, reduce the amount of calculation, and prevent overfitting. After multiple convolutional pooling layers, and then through the fully connected layer and output layer, the network category can be output.
[0061] 5. Improving the training of convolutional neural networks
[0062] After the network model is established, the network can be trained on a subset of the data set. The training set uses human actions (walking and running), and the validation set and test set each use the actions of one person. For each action, the data set is randomly divided, so the validation and test will not be on the same person's action data set.
[0063] Network training is an iterative process. During each training, the loss and accuracy curves and the network's hyperparameters (non-learning parameters, such as the basic learning rate) are adjusted in the optimal direction. The network training batch is 100, and it is tested on the validation set after every 500 iterations. The figure below shows the loss and accuracy curves during training. The loss function represents the network's error. From the beginning of training, the loss value decreases exponentially while the accuracy increases on the validation set and the test set. If the loss value decreases quickly during training, the learning rate will decrease appropriately according to the current learning strategy. If the loss value decreases slowly, the learning rate will increase appropriately. It is a dynamically adjusted parameter.
[0064] The optimization method used for learning is Adam, β 1 =0.9,β 2 =0.999,ε=1×10 -8 The network is trained on the dataset for 25-75 epochs. If the loss on the validation set starts to increase, it indicates that the CNN is overfitting. At this time, the network model obtained before overfitting is used for testing. The deep learning framework uses caffe and is trained on a GTX1080 graphics card. A complete training session takes about 5 hours.
[0065] The application effect of the present invention is described in detail below in conjunction with experiments.
[0066] 1. Experimental results
[0067] Human simulation conditions: set the distance R between the human and the radar 0 =10m, radar carrier frequency is f 0 = 15GHz, human body height H = 1.8m, the length of other parts is greater than Figure 2 As shown in the figure, the number of sampling points is 2048, the number of cycles is 4, and STFT is used for time-frequency analysis.
[0068] The application effect of the present invention is described in detail below in conjunction with tests.
[0069] 640 time-frequency graphs of walking and running are used, 500 time-frequency graphs are used for network training, and 140 are used to test the trained network. AlexNet, VGGNet and the improved network of the present invention are trained respectively. The network parameters are adjusted to make the network converge and the 140 time-frequency graphs are tested respectively with the trained models, wherein the trained AlexNet model can correctly classify 114 time-frequency graphs, with an accuracy of 114 / 140=81.4%. The trained VGGNet model can correctly classify 119 time-frequency graphs, with an accuracy of 119 / 140=85%. The trained improved network model of the present invention can correctly classify 126 time-frequency graphs, with an accuracy of 138 / 140=98.57%. Because the network structures of AlexNet and VGGNet are more complex than the improved network of the present invention, and a large amount of data sets are required to train the network model. However, the data sets used in the experiment are limited, so AlexNet and VGGNet are not as good as the method of the present invention in the problem of time-frequency graph classification, because there are too many features of network learning, resulting in overfitting of the model.
[0070] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A human motion state classification method based on an improved convolutional neural network, characterized in that: The human body motion state classification method comprises: In the first step, the convolution layer uses 128 convolution kernels to convolve the input image to obtain 128 feature maps of the input image. Multiple feature maps express multiple features of the extracted image. In the second step, due to the large similarity between adjacent pixels of the image and the high image redundancy, the feature map is pooled to reduce the image size and the number of parameters. It is then passed through three convolutional pooling layers to extract deeper features of the image. In the third step, 64 4×4 feature maps are finally obtained. The first fully connected layer uses 256 neurons to fully connect with the feature map, and the second fully connected layer uses 2 neurons to fully connect with the first fully connected layer. After the network is trained and converged, the category of the image can be output according to the input image; An improved convolutional neural network, the improved convolutional neural network has 4 convolutional layers, each convolutional layer is followed by an LRelu activation function layer, the pooling layers after the first three convolutional layers adopt 2×2 maximum pooling, and the pooling layer after the last convolutional layer adopts 4×4 maximum pooling; The first convolutional layer uses 128 5×5 convolution kernels with a pad of 2; the second and third convolutional layers use 128 3×3 convolution kernels with a pad of 1; the last convolutional layer uses 64 3×3 convolution kernels with a pad of 1; There are two fully connected layers after the four convolutional layers. The first fully connected layer fully connects the 4×4×64 feature map with the 256×1 vector, and then activates it with the LRelu activation function. The second fully connected layer reduces the dimension to 2×1, and the output layer uses softmax to achieve binary classification. The improved convolutional neural network consists of a convolutional layer, a pooling layer, and a fully connected layer; Convolution layer, convolution kernel is used to convolve the input image to obtain the feature map of the input image; Pooling layer reduces network parameters, reduces computational complexity, and prevents overfitting; The fully connected layer and the output layer are used to output the network category; the calculation formula of the radar echo signal of each part of the human body in the human body motion state classification method is: Where P t is the radar transmission power, L s is the transmission and reception loss, G r和 G v are the transmitting and receiving antenna patterns, λ and f0 are the wavelength and frequency of the radar transmitting signal, c is the speed of light, σ i The RCS of each scattering center of the human target; the echo of each part of the human body is calculated respectively, and the total echo signal of the human body can be calculated by summing up the echo signals; the time-frequency feature extraction of the human motion state classification method, the target radar echo is subjected to time-frequency analysis by short-time Fourier transform; first, a window function is selected, and the signal is divided into many small time intervals by moving the window function along the time axis to ensure that the signal is a stable signal within the time interval, and the power spectrum at each moment is calculated. The short-time Fourier transform of the echo signal s(t) is: Where g(f) is the window function. Common window functions include Hamming window, Gaussian window and Hanning window. When g(t) = 1, the short-time Fourier transform is converted into the traditional Fourier transform. STFT can roughly represent the signal energy, which is defined as the square of the STFT modulus value, that is: SPEC(t,w)=|STFT f (t,w)| 2 。 2. A method for improving the training of a convolutional neural network as claimed in claim 1, characterized in that: The improved convolutional neural network training method includes: after the network model is established, the network can be trained on a subset of the data set, the training set uses human actions (walking and running), and the verification set and the test set each use the actions of one person; for each action, the data set is randomly divided, so the verification and test will not be performed on the action data set of the same person; Network training is an iterative process. In each training, the loss and accuracy curves and the network's hyperparameters are adjusted in the optimal direction. The network training batch is 100, and it is tested on the validation set after every 500 iterations. From the beginning of training, the loss value on the validation set and the test set decreases exponentially while the accuracy increases.
3. A human motion state classification and processing platform using the improved convolutional neural network described in claim 1.
