Human behavior recognition method based on inertial sensor

Abstract

A human behavior recognition method based on inertial sensors, which relates to the field of behavior recognition technology, first collects inertial sensor data; then filters and denoises the original data to generate data; secondly, performs action segmentation and extracts features from the data segments; finally, a hierarchical recognition model is constructed based on a support vector machine and a random forest model, and sub-classifiers are combined into a hierarchical model according to prior knowledge, and a feature stratification method is proposed, and different features are selected for different classifiers in the model for classification, and the final behavior recognition result is obtained by layer-by-layer recognition. The present invention effectively improves the robustness of the classification model, significantly improves the recognition accuracy, and has obvious advantages.

Description

Technical Field

[0001] The invention belongs to the field of human behavior recognition based on inertial sensors and relates to a human behavior recognition method. Background Art

[0002] Human behavior recognition technology can fully reflect human behavior status and physiological information, and has broad application prospects in the fields of sports tracking, fitness training, daily monitoring, medical rehabilitation, human-computer interaction, virtual reality, and intelligent environment. Behavior recognition based on inertial sensors uses sensors such as accelerometers, gyroscopes, and direction sensors to collect physical information such as acceleration, angular velocity, and direction generated by human movement to identify current human behavior.

[0003] Current research mainly focuses on basic movements such as standing, sitting, walking, running, going upstairs, and going downstairs. However, transition movements between basic movements, such as standing-sitting, sitting-standing, standing, standing-squatting, and squatting-standing, are often ignored by researchers due to their low incidence and short duration. However, in the fields of elderly care and rehabilitation training, it is particularly necessary to monitor transition movements. For example, falling is an accident that is prone to occur in the elderly, and it usually occurs in the transition from sitting to standing. On the other hand, when the activity changes, if it is not handled properly, the performance of the human behavior recognition system will be affected. Therefore, it is very important to detect and recognize transition movements. So far, only a small number of researchers have considered transition movements in the recognition system.

[0004] Reyes-Ortiz et al. considered the impact of transition actions and combined the probability output of SVM continuous activity prediction with a heuristic filtering method for activity probability estimation within a 2.56s time window. The system error rate was 3.34%, and they pointed out that the misclassification of transition actions increased the system error rate by 4.13%. However, in the study, all transition actions were treated as a single class and no distinction was made. Summary of the invention

[0005] In order to overcome the shortcomings of the existing technology, the present invention provides a human behavior recognition method based on inertial sensors, which is based on inertial sensors such as accelerometers and gyroscopes, and takes transition actions into consideration in the recognition system, and can provide more comprehensive and higher accuracy recognition results.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A human behavior recognition method based on inertial sensors comprises the following steps:

[0008] Step (1): Construct a hierarchical human behavior recognition model. The model contains five sub-classifiers: static and dynamic action classifier c1, periodic and transition action classifier c2, static action classifier c3, dynamic periodic action classifier c4, and transition action classifier c5. Use the feature selection method to select the feature subset corresponding to the feature composition of each sub-classifier from the feature set F. The feature subset F i For the classifier c i The corresponding classification feature subsets, where i = 1, 2, ..., 5, are used to train the sub-classifiers in the hierarchical recognition model using the training data set;

[0009] Step (2): collecting accelerometer and gyroscope sensor data;

[0010] Step (3): using a low-pass filtering method to filter and denoise the data collected by the accelerometer and gyroscope sensors;

[0011] Step (4): Generate acceleration amplitude, linear acceleration, gravity acceleration, vertical and horizontal acceleration data;

[0012] Step (5): Segment the data to obtain data segments;

[0013] Step (6): Based on the feature set F p Extract features from the data segment and obtain the feature vector corresponding to the data segment N p is the feature set F p The number of features in F p is the union of the feature subsets of each subclassifier in the hierarchical recognition model in step (1), F p =F1∪F2∪...∪F5;

[0014] Step (7): Transform the feature vector V p Input the hierarchical recognition model for human behavior recognition, select the corresponding feature subsets for different sub-classifiers in the model for classification, and classify layer by layer to obtain the final recognition category;

[0015] The steps of constructing and training the human behavior hierarchical recognition model in step (1) are as follows:

[0016] Step (1-1): Collect accelerometer and gyroscope sensor data under different human behaviors, including static A s 、Dynamic cycle A d , Transition Action A t Three categories, collection J s is the number of predefined static action categories, the set J dis the number of predefined dynamic periodic action categories, the set J t is the number of predefined transition action categories, A=A s ∪A d ∪A t , is the set of all behavior categories;

[0017] Step (1-2): Filter and de-noise the collected sensor data using a low-pass filtering method;

[0018] Step (1-3): Generate acceleration amplitude, linear acceleration, gravity acceleration, vertical and horizontal acceleration data;

[0019] Step (1-4): Segment different behavior data to obtain data segments;

[0020] Step (1-5): According to the feature set F, extract features from each data segment to obtain the feature vector V corresponding to the data segment. k =[v1, v2, ..., v N ]∈R N , N is the number of features in the feature set F, and marks the behavior category a to which the feature vector belongs j , a j ∈A, k = 1, 2, ..., K, K is the number of data segments, and the feature vectors extracted from all data segments together constitute the sample set S, S = [V1; V2; ...; V K ];

[0021] Step (1-6): construct a hierarchical recognition model for human behavior, which includes five sub-classifiers, namely, static and dynamic action classifier c1, periodic and transition action classifier c2, static action classifier c3, dynamic periodic action classifier c4, and transition action classifier c5. The sub-classifiers are combined into a hierarchical recognition model based on prior knowledge.

[0022] Step (1-7): Divide the collected sample set S into feature selection data set S F , model training set S T and the model validation set S V , based on the feature selection set S F and S F The behavior category to which each sample belongs is selected by using the feature selection method, and some features suitable for each sub-classifier in the hierarchical recognition model are selected from the feature set F to form the corresponding feature subset. The feature subset F i For the classifier c i The corresponding classification feature subset, where i = 1, 2, ..., 5;

[0023] Step (1-8): Use the model training set ST , S T The behavior category to which each sample belongs and the feature subset of each sub-classifier are trained, and the sub-classifiers in the hierarchical model are trained, and the model validation set S is used V Perform model validation.

[0024] Furthermore, in the step (1-1), the collected sensor data includes data in different directions of the accelerometer and gyroscope sensors, and the collected behavior set A includes static actions (standing, sitting), dynamic periodic actions (walking, running, going upstairs, going downstairs) and transition actions (stand-sit, sit-stand, stand-squat, squat-stand), a total of 10 actions.

[0025] Furthermore, in the step (1-3), the acceleration amplitude data is synthesized using the accelerometer x-, y-, and z-axis data, the linear acceleration and gravity acceleration are separated using a low-pass filter, and the horizontal and vertical accelerations are calculated using the linear acceleration and gravity acceleration.

[0026] Furthermore, in the steps (1-4), the collected static actions and dynamic periodic actions are segmented using a sliding window method with a window length of L and an overlapping coverage rate of p%; for transition actions, the fuzzy approximate entropy of the acceleration amplitude is calculated, and the threshold is set to T to detect the starting point and the ending point of the transition action, and then segmentation is performed.

[0027] Furthermore, in the step (1-6), a static and dynamic action classifier c1, a period and transition action classifier c2, and a static action classifier c3 are constructed based on a support vector machine algorithm, and a dynamic period action classifier c4 and a transition action classifier c5 are constructed based on a random forest algorithm. Subclassifiers are combined into a hierarchical recognition model based on prior knowledge. For an input sample, the static and dynamic action classifier c1 is first used to identify whether the input sample is a static action or a dynamic action; if it is a static action, the static action classifier c3 is used to identify the specific static action category of the input sample; if it is a dynamic action, the period and transition action classifier c2 is used to identify whether the input sample is a dynamic period action or a transition action; if it is a dynamic period action, the dynamic period action classifier c4 is used to identify the specific period action category of the input sample; if it is a transition action, the transition action classifier c5 is used to identify the specific transition action category of the input sample.

[0028] Furthermore, in the step (1-7), let the sample subset for Each eigenvector in If it is a static action, it is marked as static category a s If it is a dynamic periodic action or a transition action, it is marked as a dynamic category a m , forming a vector All vectors constitute the feature selection set For sample subset The number of eigenvectors in is based on Perform feature selection on the feature set F, and select some features suitable for the static and dynamic action classifiers c1 to form a feature subset F1;

[0029] From S F Select the feature vectors belonging to dynamic periodic actions or transition actions to form a sample subset for Each eigenvector in If it is a dynamic periodic action, it is marked as dynamic periodic category a d , if it is a transition action, it is marked as transition action category a t , forming a vector All vectors constitute the feature selection set For sample subset The number of eigenvectors in is based on Perform feature selection on the feature set F, and select some features suitable for the period and transition action classifier c2 to form a feature subset F2;

[0030] From S F Filter out the feature vectors belonging to static actions and form a sample subset For each feature vector in S3 Mark the category of data collection Composition vector All vectors constitute the feature selection set For sample subset The number of eigenvectors in is based on Perform feature selection on the feature set F, and select some features suitable for the static action classifier c3 to form a feature subset F3;

[0031] From S F Select the feature vectors belonging to dynamic periodic actions and form a sample subset for Each eigenvector in Mark the category of data collection Composition vector All vectors constitute the feature selection set For sample subset The number of eigenvectors in is based on Perform feature selection on the feature set F, and select some features suitable for the dynamic periodic action classifier c4 to form a feature subset F4;

[0032] From S F Filter out the feature vectors belonging to transition actions and form a sample subset for Each eigenvector in Mark the category of data collection Composition vector All vectors constitute the feature selection set For sample subset The number of eigenvectors in is based on Feature selection is performed on the feature set F, and some features suitable for the transition action classifier c5 are selected to form a feature subset F5.

[0033] Furthermore, in the steps (1-7), the feature selection algorithm adopted is an integration method and a packaging method based on random forests. When selecting the feature subset of each sub-classifier, the integration method is used to obtain the weight coefficient of each feature in the feature set F, and the packaging method is used to sort the top M features with the highest weight coefficients. All of these top M features are input into the current sub-classifier to obtain the highest accuracy of the current sub-classifier on the feature selection set, and the threshold of each classifier is set to β% of the highest accuracy. The forward sequential search method is used to determine the features and number of the current sub-classifier. If the number of features is less than 5, the top 5 features are selected.

[0034] Furthermore, in the step (1-8), let the sample subset for For each sample in, according to the features in feature subset F1, select the corresponding eigenvalue to form a feature vector N1 is the number of features in the feature subset F1. If it belongs to a static action, it is marked as a static category a. s If it is a dynamic periodic action or a transition action, it is marked as a dynamic category a m , forming a vector All vectors constitute the training set For sample subset The number of eigenvectors in , based on Train static and dynamic action classifier c1;

[0035] From S T Filter out samples belonging to dynamic periodic actions or transition actions to form a sample subset for For each sample in, according to the features in feature subset F2, select the corresponding eigenvalue to form a feature vector N2 is the number of features in feature subset F2. If it belongs to dynamic periodic action, it is marked as dynamic periodic category a. d , if it is a transition action, it is marked as dynamic category a t , forming a vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train the dynamic cycle and transition action classifier c2;

[0036] From S T Filter out samples belonging to static actions and form a sample subset for For each sample in, according to the features in feature subset F3, select the corresponding eigenvalue to form a feature vector is the number of features in feature subset F3, marking the category when the data was collected Composition vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train static action classifier c3;

[0037] From S T Filter out samples belonging to dynamic periodic actions to form a sample subset for For each sample in, according to the features in feature subset F4, select the corresponding eigenvalue to form a feature vector N4 is the number of features in feature subset F4, marking the category when the data was collected Composition vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train the dynamic periodic action classifier c4;

[0038] From S T Filter out samples belonging to transition actions and form a sample subset for For each sample in, according to the features in feature subset F5, select the corresponding eigenvalue to form a feature vector is the number of features in feature subset F5, marking the category when the data was collected Composition vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train the transition action classifier c5.

[0039] Furthermore, in the steps (1-8), for the model validation set S V For each sample in, according to the features in feature subset F1, select the corresponding eigenvalue to form a feature vector Input static and dynamic action classifier c1. If the output result of c1 is static, then select the corresponding feature value to form the feature vector according to the features in feature subset F3. Input static action classifier c3 to get the final recognition result; if the output result of c1 is dynamic, then according to the features in feature subset F2, select the corresponding feature value to form the feature vector Input the period and transition action classifier c2. If the output result of c2 is a dynamic periodic action, then according to the features in the feature subset F4, select the corresponding feature value to form the feature vector Input the dynamic periodic action classifier c4 to obtain the final recognition result; if the output result of c2 is a transition action, then according to the features in the feature subset F5, select the corresponding feature value to form the feature vector Input transition action classifier c5 to get the final recognition result, N i is the feature subset F i The number of features in is i=1, 2, ..., 5.

[0040] Furthermore, in step (4), the acceleration amplitude data is synthesized using the accelerometer x-, y-, and z-axis data, the linear acceleration and gravity acceleration are separated using a low-pass filter, and the vertical and horizontal accelerations are calculated using the linear acceleration and gravity acceleration.

[0041] Furthermore, in step (5), the current data segment is obtained using a sliding window method with a window length of L and an overlapping coverage of p%.

[0042] Furthermore, in step (7), for the input feature vector V p , according to the features in feature subset F1, select the corresponding eigenvalues ​​to form the feature vector Input static and dynamic action classifier c1. If the output result of c1 is static, then select the corresponding feature value to form the feature vector according to the features in feature subset F3. Input static action classifier c3 to get the final recognition result; if the output result of c1 is dynamic, then according to the features in feature subset F2, select the corresponding feature value to form the feature vector Input the period and transition action classifier c2. If the output result of c2 is a dynamic periodic action, then according to the features in the feature subset F4, select the corresponding feature value to form the feature vector Input the dynamic periodic action classifier c4 to obtain the final recognition result; if the output result of c2 is a transition action, then according to the features in the feature subset F5, select the corresponding feature value to form the feature vector Input transition action classifier c5 to get the final recognition result, N i is the feature subset F i The number of features in is i=1, 2, ..., 5.

[0043] The beneficial effects of the present invention are mainly manifested in: effectively improving the robustness of the classification model, significantly improving the recognition accuracy, and having obvious advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the behavior recognition method of the present invention;

[0045] Figure 2 It is a hierarchical recognition model of the behavior recognition algorithm of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings.

[0047] Reference Figure 1 and Figure 2 , a human behavior recognition method based on inertial sensor, comprising the following steps:

[0048] Step (1): A hierarchical human behavior recognition model is constructed based on support vector machine and random forest. The model contains five sub-classifiers, namely static and dynamic action classifier c1, periodic and transition action classifier c2, static action classifier c3, dynamic periodic action classifier c4, and transition action classifier c5. The feature selection method is used to select some features suitable for each sub-classifier from the feature set F to form a feature subset. The feature subset F i For the classifier c i The corresponding classification feature subsets, where i = 1, 2, ..., 5, are used to train the sub-classifiers in the hierarchical recognition model using the training data set, and the hierarchical model performance is evaluated.

[0049] Step (2): Use the built-in inertial sensor of a smartphone with the head facing downward and the screen facing the thigh and fixed in the right trouser pocket to collect accelerometer and gyroscope sensor data.

[0050] Step (3): Use a moving mean filter method with a window length of 5 to perform low-pass filtering and denoising preprocessing on the data collected by the accelerometer and gyroscope sensors.

[0051] Step (4): Generate acceleration amplitude, linear acceleration, gravity acceleration, vertical and horizontal acceleration data.

[0052] Step (5): Use a sliding window with a window length of 100 and an overlap coverage of 50% to segment the data into data segments.

[0053] Step (6): Based on the feature set F p Extract features from the data segment and obtain the feature vector corresponding to the data segment N p is the feature set F p The number of features in F p is the union of the feature subsets of each subclassifier in the hierarchical recognition model in step (1), F p =F1∪F2∪…∪F5.

[0054] Step (7): Transform the feature vector V p Input hierarchical recognition model, for V p , according to the features in feature subset F1, select the corresponding eigenvalues ​​to form the feature vector Input static and dynamic action classifier c1. If the output result of c1 is static, then select the corresponding feature value to form the feature vector according to the features in feature subset F3. Input static action classifier c3 to get the final recognition result; if the output result of c1 is dynamic, then according to the features in feature subset F2, select the corresponding feature value to form the feature vector Input the period and transition action classifier c2. If the output result of c2 is a dynamic periodic action, then according to the features in the feature subset F4, select the corresponding feature value to form the feature vector Input the dynamic periodic action classifier c4 to obtain the final recognition result; if the output result of c2 is a transition action, then according to the features in the feature subset F5, select the corresponding feature value to form the feature vector Input transition action classifier c5 to get the final recognition result, N i is the feature subset F i The number of features in is i=1, 2, ..., 5.

[0055] The specific steps of step 1 of this implementation method for training and evaluating the hierarchical human behavior recognition model based on support vector machine and random forest are as follows:

[0056] Step (1-1): Collect accelerometer and gyroscope sensor data. This implementation method uses the built-in inertial sensor of the Redmi K20 mobile phone with the head facing down, the screen facing the thigh, and fixed in the right trouser pocket to collect the raw data of the accelerometer and gyroscope sensors for the selection of classifier features of the behavior recognition model, and the training and verification of the model. The number of people involved in data sampling was 10, and the collected actions included static actions (standing, sitting), dynamic periodic actions (walking, running, going upstairs, going downstairs) and transition actions (stand-sit, sit-stand, stand-squat, squat-stand), a total of 10 actions, and the sensor sampling frequency was 50Hz.

[0057] Step (1-2): Perform low-pass filtering and denoising preprocessing on the accelerometer and gyroscope sensor data, and use a moving average smoothing method with a window length of 5 to filter out the noise and burrs of the acceleration and angular velocity data.

[0058] Step (1-3): Generate acceleration amplitude, linear acceleration, gravity acceleration, vertical and horizontal acceleration data. The data generation method in step (4) is the same as this step.

[0059] The acceleration amplitude calculation formula is:

[0060]

[0061] Among them, C x , C y , C z are the accelerations of the accelerometer in the x, y, and z axes, respectively.

[0062] A first-order low-pass filter is used to separate gravity and linear acceleration. The calculation formula is:

[0063] g i (n) = αg i (n-1)+(1-α)C i (n)i=x,y,z (2)

[0064] c i (n) = C i (n)-g ( (n)i=x,y,z (3)

[0065] Among them, C i is the acceleration, g i is the acceleration due to gravity, c i is the linear acceleration, α is a coefficient, which is related to the sampling period and the time constant, and its value is greater than 0 and less than 1. In this implementation method, α=0.8.

[0066] Use linear acceleration and gravity acceleration to calculate vertical and horizontal acceleration. is the linear acceleration vector, v is the projection of the linear acceleration vector in the direction of gravity, and h is the projection of the linear acceleration vector in the horizontal direction. The calculation formula is:

[0067]

[0068]

[0069] The vertical acceleration v is positive when the angle between the linear acceleration vector and the direction of gravity is less than 90°, and negative when it is greater than 90°. The horizontal velocity is the acceleration projection of the linear acceleration on the plane perpendicular to gravity, and the positive and negative directions cannot be defined, so it is always positive.

[0070] Step (1-4): Segment the data. The basic actions (static actions and dynamic periodic actions) in the collected data are segmented using a sliding window with a window length of 100 points (2 seconds) and an overlap coverage of 50%.

[0071] For the transition action data, the fuzzy approximate entropy of the acceleration amplitude is calculated, and the threshold is set to detect the starting point and the end point of the transition action, and then segment it. For the time series data of W points {x(i), i = 1, 2, ..., W, under the condition of given pattern dimension m and similarity tolerance r, the calculation process of its fuzzy approximate entropy is:

[0072] Decompose the time series {x(i), i=1, 2, ..., W} into W-m+1 m-dimensional vectors, that is:

[0073]

[0074] definition for The mean of , that is:

[0075]

[0076] Calculate the value of any two m-dimensional vectors and The distance between It is defined as the maximum value of the difference between the two elements with the same index, that is:

[0077]

[0078] Introduce fuzzy membership function and calculate two vectors and Similarity

[0079]

[0080] Where r0 is the similarity tolerance r multiplied by the standard deviation of the signal x(i).

[0081] Statistics of each m-dimensional vector The average similarity with other vectors

[0082]

[0083] Find the average of all similarities The average value of:

[0084]

[0085] Add 1 to the dimension, changing it to m+1, and repeat the calculation process of formulas (6) to (11) to obtain Φ m+1 (r).

[0086] The fuzzy approximate entropy of the calculation sequence is calculated as follows:

[0087] FuzzyEn(m,r,W)=lnΦ m (r)-lnΦ m+1 (r) (12)

[0088] This implementation method calculates the fuzzy approximate entropy value of a data segment of 2 seconds, 1 second before and 1 second after the current moment of the acceleration amplitude, wherein the embedding dimension during the fuzzy approximate entropy calculation is m=2, the similarity tolerance is r=0.15, and this implementation method sets the threshold value to 0.1 for detection and segmentation.

[0089] The total data segments obtained by segmentation of this implementation method are K=8092, and each data segment contains data of 12 axes in total, including three-axis linear acceleration, three-axis gravitational acceleration, three-axis angular velocity, acceleration amplitude, vertical acceleration, and horizontal acceleration.

[0090] Step (1-5): Extract features from the segmented data segments according to the feature set F. The extracted features are 10 features, namely, mean, variance, standard deviation, root mean square, skewness, kurtosis, first-order autocorrelation coefficient, quartile difference, zero-crossing rate and fuzzy approximate entropy. The extracted data segments include three-axis linear acceleration, three-axis gravity acceleration, three-axis angular velocity, acceleration amplitude, vertical acceleration and horizontal acceleration, for a total of 12 axes of data. Since the acceleration amplitude and horizontal acceleration are always positive values, there is no need to extract the zero-crossing rate feature, so the number of features in the feature set F is N = 12 × 10-2 = 118.

[0091] Steps (1-6): Construct a hierarchical recognition model based on support vector machine and random forest. Figure 2As shown in the figure, the model includes five sub-classifiers, namely, static and dynamic action classifier c1, periodic and transition action classifier c2, and static action classifier c3, which are constructed based on support vector machine, periodic action classifier c4, and transition action classifier c5, which are constructed based on random forest. The sub-classifiers are combined into a hierarchical recognition model based on prior knowledge. For an input sample, the static and dynamic action classifier c1 is first used to identify whether the input sample is a static action or a dynamic action; if it is a static action, the static action classifier c3 is used to identify the specific static action category of the input sample; if it is a dynamic action, the periodic and transition action classifier c2 is used to identify whether the input sample is a dynamic periodic action or a transition action; if it is a dynamic periodic action, the dynamic periodic action classifier c4 is used to identify the specific periodic action category of the input sample; if it is a transition action, the transition action classifier c5 is used to identify the specific transition action category of the input sample.

[0092] Step (1-7): Divide the entire data set into a feature selection data set and a model training and verification data set. In this implementation method, the data of 3 of the 10 experimenters are divided into a feature selection data set S F The remaining 7 people’s data are the model training and verification data set S TV The feature selection dataset contains 2378 samples, and the model training and validation dataset contains 5714 samples.

[0093] Using feature selection dataset S F Feature selection is performed on the feature set F, and different feature subsets are selected for different classifiers in the hierarchical recognition model. The feature selection of each classifier is performed by combining the ensemble method (Embedded) and the wrapper method (Wrapper). First, the weight coefficient of each feature is obtained by using the ensemble method based on random forest and the average impurity reduction as the evaluation standard, and the top 20 features with the highest coefficient of each classifier are obtained; then the 20 features of each classifier are sorted by the wrapper method based on random forest. All the 20 features selected by each classifier are input into each classifier, and the accuracy of each classifier on the feature selection data set is obtained as the highest accuracy of each classifier. The threshold of each classifier is set to 98% or more of the highest accuracy, and the features and number of each classifier are determined by the forward sequential search method. If the number of features is less than 5, the first 5 features are selected. Finally, this implementation method selects the first 5 features for the static and dynamic action classifier c1, the periodic and transition action classifier c2, the static action classifier c3, and the transition action classifier c5, and selects the first 7 features for the periodic action classifier c4. Repeated features are counted only once, and the final number of selected features is 22. Table 1 shows the feature subsets selected by each classifier.

[0094] Table 1 Feature subsets selected by each classifier

[0095]

[0096] Step (1-8): Model training and validation. TV Divide into training set S T and the test set S V , use the training set to train the model, and use the model to identify the behavior data in the test set. Use the model validation data set as the data set for training the classifier. To avoid overfitting of the training model, the cross-validation method is usually used. This implementation method uses the 10-fold cross-validation method to verify the model performance, that is, all samples are randomly divided into 10 groups, and 9 groups are selected each time as the training set S T For model training, another group is used as a validation set S V , evaluate the various indicators of the model.

[0097] In using S T When training the model, let the sample subset for For each sample in, according to the features in feature subset F1, select the corresponding eigenvalue to form a feature vector N1 is the number of features in the feature subset F1. If it belongs to a static action, it is marked as a static category a. s If it is a dynamic periodic action or a transition action, it is marked as a dynamic category a m , forming a vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train static and dynamic action classifier c1;

[0098] From S T Filter out samples belonging to dynamic periodic actions or transition actions to form a sample subset for For each sample in, according to the features in feature subset F2, select the corresponding eigenvalue to form a feature vector N2 is the number of features in feature subset F2. If it belongs to dynamic periodic action, it is marked as dynamic periodic category a. d , if it is a transition action, it is marked as dynamic category a t , forming a vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train the dynamic cycle and transition action classifier c2;

[0099] From S TFilter out samples belonging to static actions and form a sample subset for For each sample in, according to the features in feature subset F3, select the corresponding eigenvalue to form a feature vector N3 is the number of features in feature subset F3, marking the category when the data was collected Composition vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train static action classifier c3;

[0100] From S T Filter out samples belonging to dynamic periodic actions to form a sample subset for For each sample in, according to the features in feature subset F4, select the corresponding eigenvalue to form a feature vector N4 is the number of features in feature subset F4, marking the category when the data was collected Composition vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train the dynamic periodic action classifier c4;

[0101] From S T Filter out samples belonging to transition actions and form a sample subset for For each sample in, according to the features in feature subset F5, select the corresponding eigenvalue to form a feature vector N5 is the number of features in feature subset F5, marking the category when the data was collected Composition vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train the transition action classifier c5.

[0102] In using S V When verifying model performance, for the model validation set S V For each sample in, according to the features in feature subset F1, select the corresponding eigenvalue to form a feature vector Input static and dynamic action classifier c1. If the output result of c1 is static, then select the corresponding feature value to form the feature vector according to the features in feature subset F3. Input static action classifier c3 to get the final recognition result; if the output result of c1 is dynamic, then according to the features in feature subset F2, select the corresponding feature value to form the feature vector Input the period and transition action classifier c2. If the output result of c2 is a dynamic periodic action, then according to the features in the feature subset F4, select the corresponding feature value to form the feature vector Input the dynamic periodic action classifier c4 to obtain the final recognition result; if the output result of c2 is a transition action, then according to the features in the feature subset F5, select the corresponding feature value to form the feature vector Input transition action classifier c5 to get the final recognition result, N i is the feature subset F i The number of features in is i=1, 2, ..., 5.

[0103] Table 2 is the confusion matrix obtained by 10-fold cross validation, where the horizontal axis is the recognition category and the vertical axis is the true category. Table 3 shows the accuracy, recall rate and F1-Score of each action.

[0104] Table 2 Cross-validation confusion matrix of hierarchical recognition model

[0105]

[0106]

[0107] Table 3 Precision, recall and F1-Score of each action

[0108]

[0109] It can be seen from the confusion matrix that for the classification of the three major categories of static actions, periodic dynamic actions, and transition actions, the system only has a very small amount of cross-category misclassification, indicating that the hierarchical design of this implementation method has good discrimination performance. In the recognition of periodic actions, there are certain misclassifications for the three actions of walking, going upstairs, and going downstairs. This is because the three actions themselves have certain similarities, and due to the limitations of the experimental collection environment, the data collection of going up and downstairs is carried out continuously on the stairs of the floor. The stairs are composed of steps and platforms at the connection. The up and downstairs behavior at the platform is similar to the walking behavior, which leads to confusion between going up and downstairs and walking behavior. Despite this, for the three actions of walking, going upstairs, and going downstairs, an average precision of 96.6%, an average recall of 96.3%, and an average F1-Score of 0.964 were still achieved. Excellent recognition performance was also achieved for the four transition actions. The overall recognition accuracy of the system is 98.1%. It can be seen that the hierarchical recognition model of this implementation method has high recognition performance.

Claims

1. A human behavior recognition method based on inertial sensor, characterized in that: The method comprises the following steps: Step (1): Construct a hierarchical human behavior recognition model. The model contains five sub-classifiers: static and dynamic action classifier c1, periodic and transition action classifier c2, static action classifier c3, dynamic periodic action classifier c4, and transition action classifier c5. Use the feature selection method to select the feature subset corresponding to the feature composition of each sub-classifier from the feature set F. The feature subset F i For subclassifier c i The corresponding classification feature subsets, where d = 1, 2, ..., 5, are used to train the sub-classifiers in the hierarchical recognition model using the training data set; Step (2): collecting accelerometer and gyroscope sensor data; Step (3): using a low-pass filtering method to filter and denoise the data collected by the accelerometer and gyroscope sensors; Step (4): Generate acceleration amplitude, linear acceleration, gravity acceleration, vertical and horizontal acceleration data; Step (5): Segment the data to obtain data segments; Step (6): Based on the feature set F p Extract features from the data segment and obtain the feature vector corresponding to the data segment N p is the feature set F p The number of features in F p is the union of the feature subsets of each subclassifier in the hierarchical recognition model in step (1), F p =F1∪F2∪…∪F5; Step (7): Transform the feature vector V p Input the hierarchical recognition model for human behavior recognition, select the corresponding feature subsets for different sub-classifiers in the model for classification, and classify layer by layer to obtain the final recognition category; In the step (1), the steps of constructing and training the human behavior hierarchical recognition model are as follows: Step (1-1): Collect accelerometer and gyroscope sensor data under different human behaviors, including static A s 、Dynamic cycle A d , Transition Action A t Three categories, collection J s is the number of predefined static action categories, the set J d is the number of predefined dynamic periodic action categories, the set J t is the number of predefined transition action categories, A=A s ∪A d ∪A t , is the set of all behavior categories; Step (1-2): Filter and de-noise the collected sensor data using a low-pass filtering method; Step (1-3): Generate acceleration amplitude, linear acceleration, gravity acceleration, vertical and horizontal acceleration data; Step (1-4): Segment different behavior data to obtain data segments; Step (1-5): According to the feature set F, extract features from each data segment to obtain the feature vector V corresponding to the data segment. k =[v1,v2,…,v N ]∈R N , N is the number of features in the feature set F, and marks the behavior category a to which the feature vector belongs j , a j ∈A, k = 1, 2, ..., K, K is the number of data segments, and the feature vectors extracted from all data segments together constitute the sample set S, S = [V1; V2; ...; V K ]; Step (1-6): construct a hierarchical recognition model for human behavior, which includes five sub-classifiers, namely, static and dynamic action classifier c1, periodic and transition action classifier c2, static action classifier c3, dynamic periodic action classifier c4, and transition action classifier c5. The sub-classifiers are combined into a hierarchical recognition model based on prior knowledge. Step (1-7): Divide the collected sample set S into feature selection data set S F , model training set S T and the model validation set S V , based on the feature selection set S F and S F The behavior category to which each sample belongs is selected by using the feature selection method, and some features suitable for each sub-classifier in the hierarchical recognition model are selected from the feature set F to form the corresponding feature subset. The feature subset F i For subclassifier c i The corresponding classification feature subset, where i = 1, 2, ..., 5; Step (1-8): Use the model training set S T , S T The behavior category to which each sample belongs and the feature subset of each sub-classifier are trained, and the sub-classifiers in the hierarchical model are trained, and the model validation set S is used V Conduct model validation; In the step (1-1), the sensor data collected include data in different directions of the accelerometer and the gyroscope sensor, and the collected behavior set A includes static actions of standing and sitting, dynamic periodic actions of walking, running, going upstairs and going downstairs, and transitional actions of standing-sitting, sitting-standing, standing-squatting, and squatting-standing, a total of 10 actions; In the step (1-3), the acceleration amplitude data is synthesized using the accelerometer x-, y-, and z-axis data, the linear acceleration and the gravitational acceleration are separated using a low-pass filter, and the vertical and horizontal accelerations are calculated using the linear acceleration and the gravitational acceleration; In the steps (1-4), the collected static actions and dynamic periodic actions are segmented by a sliding window method with a window length of L and an overlap coverage of p%; the fuzzy approximate entropy of the acceleration amplitude is calculated for the transition action, and the threshold is set to T to detect the starting point and the end point of the transition action, and then segmentation is performed; In the steps (1-6), a static and dynamic action classifier c1, a period and transition action classifier c2, and a static action classifier c3 are constructed based on a support vector machine algorithm; a dynamic periodic action classifier c4 and a transition action classifier c5 are constructed based on a random forest algorithm; and the sub-classifiers are combined into a hierarchical recognition model based on prior knowledge. For an input sample, the static and dynamic action classifier c1 is first used to identify whether the input sample is a static action or a dynamic action; if it is a static action, the static action classifier c3 is used to identify the specific static action category of the input sample; if it is a dynamic action, the period and transition action classifier c2 is used to identify whether the input sample is a dynamic periodic action or a transition action; if it is a dynamic periodic action, the dynamic periodic action classifier c4 is used to identify the specific periodic action category of the input sample; if it is a transition action, the transition action classifier c5 is used to identify the specific transition action category of the input sample.

2. The human behavior recognition method based on inertial sensor according to claim 1, characterized in that: In the steps (1-7), let the sample subset for Each eigenvector in If it is a static action, it is marked as static category a s If it is a dynamic periodic action or a transition action, it is marked as a dynamic category a m , forming a vector All vectors constitute the feature selection set For sample subset The number of eigenvectors in is based on Perform feature selection on the feature set F, and select some features suitable for the static and dynamic action classifiers c1 to form a feature subset F1; From S F Select the feature vectors belonging to dynamic periodic actions or transition actions to form a sample subset for Each eigenvector in If it is a dynamic periodic action, it is marked as dynamic periodic category a d , if it is a transition action, it is marked as transition action category a t , forming a vector All vectors constitute the feature selection set For sample subset The number of eigenvectors in is based on Perform feature selection on the feature set F, and select some features suitable for the period and transition action classifier c2 to form a feature subset F2; From S F Filter out the feature vectors belonging to static actions and form a sample subset For each feature vector in S3 Mark the category of data collection Composition vector All vectors constitute the feature selection set For sample subset The number of eigenvectors in is based on Perform feature selection on the feature set F, and select some features suitable for the static action classifier c3 to form a feature subset F3; From S F Select the feature vectors belonging to dynamic periodic actions and form a sample subset for Each eigenvector in Mark the category of data collection Composition vector All vectors constitute the feature selection set For sample subset The number of eigenvectors in is based on Perform feature selection on the feature set F, and select some features suitable for the dynamic periodic action classifier c4 to form a feature subset F4; From S F Filter out the feature vectors belonging to transition actions and form a sample subset for Each eigenvector in Mark the category of data collection Composition vector All vectors constitute the feature selection set For sample subset The number of eigenvectors in is based on Feature selection is performed on the feature set F, and some features suitable for the transition action classifier c5 are selected to form a feature subset F5.

3. The human behavior recognition method based on inertial sensor as claimed in claim 2, characterized in that: In the steps (1-7), the feature selection algorithm adopted is an integration method and a packaging method based on random forests. When selecting the feature subset of each sub-classifier, the integration method is used to obtain the weight coefficient of each feature in the feature set F, and the packaging method is used to sort the first M features with the highest weight coefficients. All of these first M features are input into the current sub-classifier to obtain the highest accuracy of the current sub-classifier on the feature selection set, and the threshold of each classifier is set to β% of the highest accuracy. The forward sequential search method is used to determine the features and number of the current sub-classifier. If the number of features is less than 5, the first 5 features are selected.

4. The human behavior recognition method based on inertial sensor according to claim 1, characterized in that: In the steps (1-8), let the sample subset for For each sample in, according to the features in feature subset F1, select the corresponding eigenvalue to form a feature vector N1 is the number of features in the feature subset F1. If it belongs to a static action, it is marked as a static category a. s If it is a dynamic periodic action or a transition action, it is marked as a dynamic category a m , forming a vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train static and dynamic action classifier c1; From S T Filter out samples belonging to dynamic periodic actions or transition actions to form a sample subset for For each sample in, according to the features in feature subset F2, select the corresponding eigenvalue to form a feature vector N2 is the number of features in feature subset F2. If it belongs to dynamic periodic action, it is marked as dynamic periodic category a. d , if it is a transition action, it is marked as dynamic category a t , forming a vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train the dynamic cycle and transition action classifier c2; From S T Filter out samples belonging to static actions and form a sample subset for For each sample in, according to the features in feature subset F3, select the corresponding eigenvalue to form a feature vector N3 is the number of features in feature subset F3, marking the category when the data was collected Composition vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train static action classifier c3; From S T Filter out samples belonging to dynamic periodic actions to form a sample subset for For each sample in, according to the features in feature subset F4, select the corresponding eigenvalue to form a feature vector N4 is the number of features in feature subset F4, marking the category when the data was collected Composition vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train the dynamic periodic action classifier c4; From S T Filter out samples belonging to transition actions and form a sample subset for For each sample in, according to the features in feature subset F5, select the corresponding eigenvalue to form a feature vector N5 is the number of features in feature subset F5, marking the category when the data was collected Composition vector All vectors constitute the training set For sample subset The number of eigenvectors in is based on Train the transition action classifier c5.

5. The human behavior recognition method based on inertial sensor according to claim 4, characterized in that: In the steps (1-8), for the model validation set S V For each sample in, according to the features in feature subset F1, select the corresponding eigenvalue to form a feature vector Input static and dynamic action classifier c1. If the output result of c1 is static, then select the corresponding feature value to form the feature vector according to the features in feature subset F3. Input static action classifier c3 to get the final recognition result; if the output result of c1 is dynamic, then according to the features in feature subset F2, select the corresponding feature value to form the feature vector Input the period and transition action classifier c2. If the output result of c2 is a dynamic periodic action, then according to the features in the feature subset F4, select the corresponding feature value to form the feature vector Input the dynamic periodic action classifier c4 to obtain the final recognition result; if the output result of c2 is a transition action, then according to the features in the feature subset F5, select the corresponding feature value to form the feature vector Input transition action classifier c5 to get the final recognition result, N i is the feature subset F i The number of features in is i=1,2,…,5.

6. The human behavior recognition method based on inertial sensor according to any one of claims 1 to 5, characterized in that: In the step (4), the acceleration amplitude data is synthesized using the accelerometer x-, y-, and z-axis data, the linear acceleration and the gravitational acceleration are separated using a low-pass filter, and the vertical and horizontal accelerations are calculated using the linear acceleration and the gravitational acceleration.

7. The human behavior recognition method based on inertial sensor according to any one of claims 1 to 5, characterized in that: In the step (5), the current data segment is obtained by using a sliding window method with a window length of L and an overlapping coverage rate of p%.

8. The human behavior recognition method based on inertial sensor according to any one of claims 1 to 5, characterized in that: In step (7), for the input feature vector V p , according to the features in feature subset F1, select the corresponding eigenvalues ​​to form the feature vector Input static and dynamic action classifier c1. If the output result of c1 is static, then select the corresponding feature value to form the feature vector according to the features in feature subset F3. Input static action classifier c3 to get the final recognition result; if the output result of c1 is dynamic, then according to the features in feature subset F2, select the corresponding feature value to form the feature vector Input the period and transition action classifier c2. If the output result of c2 is a dynamic periodic action, then according to the features in the feature subset F4, select the corresponding feature value to form the feature vector Input the dynamic periodic action classifier c4 to obtain the final recognition result; if the output result of c2 is a transition action, then according to the features in the feature subset F5, select the corresponding feature value to form the feature vector Input transition action classifier c5 to get the final recognition result, N i is the feature subset F i The number of features in is i=1,2,…,5.

Images