Activity recognition method and system based on recurrent graph and cnn-lstm

Abstract

The application relates to a method and system for activity recognition based on a recurrent graph and a CNN-LSTM, which comprises the following steps: acquiring three-axis acceleration sensor original data of long-time daily activities; performing preprocessing and normalization operations on the original data; dividing the data into a training set and a test set; using an SMOTE algorithm to expand the samples of minority classes in the training set; using an ENN algorithm to clean all the samples in the expanded training set; using an improved recurrent graph algorithm to convert the three-axis acceleration sensor time sequence data of the samples into corresponding two-dimensional recurrent matrices in sequence; using the recurrent matrix samples to train a CNN-LSTM classification model; and using the CNN-LSTM classification model to recognize activities. The application can effectively solve the problem of low activity recognition accuracy caused by the unevenness of various types of samples and the poor multi-dimensional time sequence feature extraction effect in the existing activity recognition based on three-axis acceleration sensor data.

Description

Technical Field

[0001] This invention relates to the fields of wearable devices and human activity recognition technology, and in particular to an activity recognition method and system based on recursive graphs and CNN-LSTM. Background Technology

[0002] Tracking and identifying users' daily activities can effectively reflect their behavioral habits and daily energy consumption, providing important guidance for health monitoring, elderly and child care, and virtual reality interaction. Existing technologies for daily activity identification are mainly based on two approaches: computer vision and wearable devices. Computer vision-based technologies have limited application scenarios, high costs, and privacy concerns, while wearable device-based technologies are low-cost and allow for monitoring anytime, anywhere, thus having wider applications. Wearable device-based technologies collect body movement data through sensors to construct datasets; this type of data is essentially a multi-dimensional time series. Therefore, identifying daily activities using wearable devices is essentially classifying multi-dimensional time series. Existing time series analysis methods mainly include time-domain analysis, frequency-domain analysis (spectral analysis, wavelet analysis, etc.), joint time-frequency analysis, and nonlinear analysis methods (chaotic analysis, recursive quantitative analysis, fractal analysis, complex network analysis, etc.). For different applications, it is necessary to select the appropriate analysis method, which obviously requires a lot of experimentation. These analysis methods can be used to extract various features characterizing time series. However, for time series classification applications, it is necessary to select the best classifier from among many classifiers (such as neural networks, support vector machines, and KNN (K-nearest neighbors algorithm)), which requires a lot of testing. It can be seen that the design process of traditional time series classification algorithms is time-consuming and labor-intensive, and since the extracted features are only shallow features of the time series, it is difficult to achieve high accuracy for more complex multidimensional time series classification.

[0003] Therefore, there is an urgent need to propose an activity recognition method based on recursive graphs and CNN-LSTM to overcome the aforementioned shortcomings of existing technologies. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the technical defects existing in the prior art, and to propose an activity recognition method and system based on recursive graph and CNN-LSTM, which solves the problem of poor performance of existing technologies in multidimensional time series feature extraction and improves the classification accuracy of daily activity recognition methods.

[0005] To address the aforementioned technical problems, this invention provides an activity recognition method based on recursive graphs and CNN-LSTM, comprising:

[0006] S1: Acquire triaxial accelerometer data of the target user's daily activities over a long period of time, and label the raw data collected under various activities with activity type.

[0007] S2: Perform preprocessing and normalization operations on the raw data obtained in step S1;

[0008] S3: Divide the data obtained in step S2 into several samples of the same length, and divide all the extracted samples into training set and test set according to a preset ratio;

[0009] S4: Expand the minority class samples in the training set using the SMOTE algorithm;

[0010] S5: Use the ENN algorithm to clean all samples in the expanded training set, filter and remove noisy samples;

[0011] S6: Using the improved recursive graph algorithm, the acceleration sensing time series data of each dimension of the training set samples obtained in step S5 and the test set samples obtained in step S3 are sequentially converted into corresponding two-dimensional recursive matrices.

[0012] S7: Use the recursive matrix samples transformed from the training set and test set samples to train and test the constructed classification model to obtain the trained classification model, wherein the classification model is constructed by combining CNN and LSTM.

[0013] S8: Deploy the classification model obtained in step S7 on the cloud platform and use the classification model to identify human daily activities.

[0014] In one embodiment of the present invention, in step S1, a triaxial accelerometer on a wearable device is used to acquire triaxial accelerometer data of a target user during a long period of daily activities.

[0015] In one embodiment of the present invention, in step S2, the original data is preprocessed using a median filter and a third-order Butterworth low-pass filter, and the maximum and minimum values ​​of each dimension of the triaxial accelerometer data are normalized sequentially.

[0016] In one embodiment of the present invention, step S4, the method of expanding the minority class samples in the training set using the SMOTE algorithm, includes:

[0017] For each sample in the minority class, calculate the Euclidean distance between that sample and other samples in the same class;

[0018] Sort the Euclidean distances of each sample to other samples of the same class in ascending order. Select the k nearest samples from the same class of each sample. Based on the oversampling factor N, for each minority class sample, randomly select N samples from its k nearest neighbors to generate a new sample.

[0019] In one embodiment of the present invention, in step S6, the acceleration sensing time series data of each dimension of the training set samples obtained in step S5 and the test set samples obtained in step S3 are sequentially converted into corresponding two-dimensional recursive matrices R′ using an improved recursive graph algorithm, and the temporal correlation between different sub-states is extracted from the time series. m represents the dimension of the extracted state, τ represents the extraction delay, and n represents the time series index.

[0020] In one embodiment of the present invention, in step S7, each sample is converted into a corresponding two-dimensional recursive matrix according to each dimension using an improved recursive graph method. That is, a sample is converted into a combination of multiple two-dimensional recursive matrices, and the combination is used as a sample for training and testing.

[0021] In one embodiment of the present invention, in step S7, when training the constructed CNN-LSTM model, the transformed recursive matrix samples in the training set are input into a 3-layer CNN network. The receptive field is expanded by stacking multiple layers of convolutional kernels to extract hidden local features and global information in the input data. The convolutional kernels of a large number of channels in each CNN layer are used to extract the channel-related features of the input data. The features extracted by the 3-layer CNN network are flattened and input into the LSTM layer to further extract the time-related information contained in the samples, thereby realizing the extraction of multi-dimensional feature correlation and time-related features of the time series.

[0022] Furthermore, this invention also provides an activity recognition system based on recursive graphs and CNN-LSTM, comprising:

[0023] The data acquisition module is used to acquire triaxial accelerometer data of the target user's daily activities over a long period of time, and to label the raw sensor data collected under various activities with the activity type.

[0024] The data processing module is used for preprocessing and normalizing the raw data;

[0025] The sample sorting module is used to divide the data processed by the data processing module into several samples of the same length, and divide all the extracted samples into training set and test set according to a preset ratio.

[0026] The sample generation module is used to augment the minority class samples in the training set using the SMOTE algorithm;

[0027] The sample cleaning module is used to clean all samples in the expanded training set using the ENN algorithm, filtering and removing noisy samples.

[0028] The sample conversion module is used to convert the acceleration sensing time series data of each dimension of the training set samples cleaned by the sample cleaning module and the test set samples cleaned by the sample sorting module into corresponding two-dimensional recursive matrices using an improved recursive graph algorithm. The new samples obtained after the conversion contain multiple two-dimensional recursive matrices.

[0029] The model training module is used to train and test the constructed classification model using the transformed recursive matrix samples in the training and test sets to obtain the trained classification model, wherein the classification model is constructed by combining CNN and LSTM.

[0030] The recognition module is used to deploy the trained classification model on the cloud platform and use the classification model to recognize human daily activities.

[0031] In one embodiment of the present invention, the sample conversion module is further configured to use an improved recursive graph algorithm to sequentially convert the acceleration sensing time series data of each dimension of the training set samples cleaned by the sample cleaning module and the test set samples processed by the sample sorting module into corresponding two-dimensional recursive matrices R′, and extract the temporal correlation between different states from the time series, wherein, m represents the dimension of the extracted state, τ represents the extraction delay, and n represents the time series index.

[0032] In one embodiment of the present invention, when training the constructed CNN-LSTM model, the model training module is used to input the transformed recursive matrix samples from the training set into a 3-layer CNN network. By stacking multiple layers of convolutional kernels, the receptive field is expanded, and hidden local features and global information in the input data are extracted. The convolutional kernels of a large number of channels in each CNN layer are used to extract the channel-related features of the input data. The features extracted by the 3-layer CNN network are flattened and input into the LSTM layer to further extract the time-related information contained in the samples, thereby realizing the extraction of multi-dimensional feature correlation and time-related features of the time series.

[0033] The technical solution of the present invention has the following advantages compared with the prior art:

[0034] 1. To address the problem of imbalanced sample sizes across various categories collected in real-life situations, this invention combines the SMOTE and ENN algorithms to balance the training set; it performs an improved recursive graph transformation on the training and test set samples to obtain a two-dimensional recursive matrix corresponding to multiple channels; and it uses the samples transformed from the improved recursive graph for model training, parameter optimization, and model testing; thus solving the problem of poor performance of existing technologies in extracting multi-dimensional time series features and improving the classification accuracy of daily activity recognition methods.

[0035] 2. This invention converts a one-dimensional time series into a two-dimensional recursive matrix for analysis. Each element in the two-dimensional recursive matrix sample is the Euclidean distance between different sub-state trajectories of the original time series, reflecting the correlation between different sub-states. Therefore, it has temporal correlation information, which can obtain more information from the original time series and greatly help feature extraction in classification. It is a good supplement to existing feature extraction methods in the field of daily activity recognition.

[0036] 3. This invention uses a CNN-LSTM combined model for classification. Since the transformed samples can reflect the temporal correlation information of the original time series, the local features and global information of the recursive matrix are first extracted using CNN, and then the time-related information in the transformed samples is further extracted using the LSTM model to improve classification accuracy. Attached Figure Description

[0037] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0038] Figure 1 This is a flowchart illustrating an activity recognition method based on recursive graphs and CNN-LSTM proposed in this invention.

[0039] Figure 2 This is a structural block diagram of the classification model proposed in this invention.

[0040] Figure 3 This is a block diagram of the 3-layer CNN network structure proposed in this invention.

[0041] Figure 4 This is a confusion matrix result diagram of the identification method proposed in this invention classified on the public sample set WISDOM. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0043] First, the abbreviations used in the following text are explained as follows:

[0044] CNN stands for Convolutional Neural Network.

[0045] LSTM stands for Long Short Term Memory, which is a neural network that uses long short-term memory.

[0046] SMOTE stands for Synthetic Minority Over-sampling Technique.

[0047] ENN stands for Edited Nearest Neighbors, which is a technology for editing nearest neighbors.

[0048] Reference Figure 1 As shown, this embodiment of the invention provides an activity recognition method based on recursive graphs and CNN-LSTM, including the following steps:

[0049] S1: Acquire triaxial accelerometer data of the target user's daily activities over a long period of time, and label the raw sensor data collected under various activities with activity type.

[0050] S2: Perform preprocessing and normalization operations on the raw data obtained in step S1;

[0051] S3: Divide the data obtained in step S2 into several samples of the same length, and divide all the extracted samples into training set and test set according to a preset ratio;

[0052] S4: Expand the minority class samples in the training set using the SMOTE algorithm;

[0053] S5: Use the ENN algorithm to clean all samples in the expanded training set, filter and remove noisy samples;

[0054] S6: Using the improved recursive graph algorithm, the acceleration sensing time series data of each dimension of the training set samples obtained in step S5 and the test set samples obtained in step S3 are sequentially converted into corresponding two-dimensional recursive matrices.

[0055] S7: Use the transformed recursive matrix samples from the training and test sets to train and test the constructed classification model to obtain a trained classification model, wherein the classification model is constructed by combining CNN and LSTM.

[0056] S8: Deploy the classification model obtained in step S7 on the cloud platform and use the classification model to identify human daily activities.

[0057] To address the issue of imbalanced sample sizes across various categories collected in real-life situations, this invention combines the SMOTE and ENN algorithms to balance the training set. It then performs an improved recursive graph transformation on the training and test sets to obtain two-dimensional recursive matrices corresponding to multiple channels. Finally, it utilizes the samples transformed from the improved recursive graph for model training, parameter optimization, and model testing. This solves the problem of poor performance in extracting multi-dimensional time series features using existing technologies and improves the classification accuracy of daily activity recognition methods.

[0058] In step S1, a data acquisition device on the wearable device is used to acquire triaxial accelerometer data of the target user's daily activities over a long period. For example, the accelerometer on the wearable device can be used to collect triaxial acceleration data of the body at a frequency of 20Hz to obtain raw data of the human body's daily activities over a long period. In other words, this invention utilizes an integrated accelerometer to collect data generated by body activity, achieving comprehensive acquisition of multidimensional physiological health information of the human body; the wearable device is inexpensive and consumes little power, meeting the needs of daily monitoring.

[0059] In step S2, a median filter and a third-order Butterworth low-pass filter are used to preprocess the raw data. For example, a median filter and a third-order Butterworth low-pass filter are used to preprocess the signal to reduce the influence of noise. Then, maximum / minimum normalization is performed sequentially on each dimension of the triaxial accelerometer data. Assume that a certain dimension of the time series of the sample is x = {x1, x2, ..., x...} n The normalization formula is as follows:

[0060]

[0061] In the formula, y={y1, y2,…,y n} represents the normalized data, x max and x min These are the maximum and minimum values ​​in this dimension, respectively.

[0062] In step S4, the method of expanding the minority class samples in the training set using the SMOTE algorithm is employed to balance the dataset. The SMOTE algorithm process is as follows:

[0063] 1) For each sample in the minority class, suppose one of the samples is x i = (a1, a2, ..., a n Calculate the Euclidean distance from the given sample to all samples in the same class set. Assume another sample is x. j = (b1, b2, ..., b n), then x and x j Euclidean distance ρ i，j Defined as:

[0064]

[0065] Where n is the time series length of the sample, a1, a2, ..., a n x i Sampled values ​​at different times, b1, b2, ..., b n x j Sampled values ​​at different times.

[0066] The Euclidean distance ρ from each sample in the minority class to all other samples can be calculated using the above formula. i，j ;

[0067] 2) Sort each sample by its distance to other samples of the same class in ascending order, and select the k nearest neighbors for each sample. Then, based on the oversampling factor N, for each minority class sample, randomly select N samples from its k nearest neighbors to generate a new sample. Assume that one randomly selected nearest neighbor is... It is combined with the original sample x to construct a new sample x according to the following formula. new .

[0068]

[0069] Where k and N are the parameters of the SMOTE algorithm, which can be adjusted according to the actual situation, and rand(0,1) is a random vector with the same dimension as x.

[0070] This invention addresses the problem in existing research where imbalanced datasets negatively impact model classification accuracy and prevent the learning of effective prior information. By combining the SMOTE algorithm with the ENN algorithm, it can better generate effective minority class samples, balance the class ratio of the dataset, and provide a simple and effective method for data augmentation of imbalanced datasets.

[0071] In step S6, an improved recursive graph algorithm is used to transform multi-channel one-dimensional time series data into multiple two-dimensional correlation matrix data. The improved recursive graph is a method for extracting correlations between different states from a time series. The dataset of this invention has a sample dimension of 3 (channels) * 80 (time sampling points). The time series in each dimension needs to be transformed using the improved recursive graph method, thus the final result is 3 (channels) * 78 * 78 (78 is the dimension of the two-dimensional recursive matrix obtained according to the improved recursive graph algorithm). Assume that the time series in one dimension is x = (x1, x2, ..., x...). n If n-(m-1)τ sub-state sequences s are extracted from it, then n-(m-1)τ sub-state sequences s are extracted from it. i:

[0072]

[0073] It includes two hyperparameters: m is the dimension of the extracted state, and τ is the extraction delay.

[0074] After obtaining the set of substate sequences, the Euclidean distance d between these substates is calculated. i，j =||s i -s j || Perform calculations, R i，j R represents the correlation between these substates. i，j The calculation is as follows:

[0075]

[0076] Where ε is the distance threshold, and Θ(*) is the step function, when ε-d i，j <0, then R i，j =0; otherwise it is R i，j =1.

[0077] Traditional recursive graph algorithms often suffer from the loss of effective detail information due to the distance threshold ε and step function. Therefore, improvements are made to the traditional recursive graph algorithm, specifically to improve R in the recursive graph algorithm. i，j The calculation is as follows:

[0078]

[0079] Through the above steps, a time series from one channel is transformed into a correlation matrix R′ that reveals different time states of the time series.

[0080]

[0081] The improved recursive graph algorithm described above transforms multi-channel time series into multiple binary correlation matrices, extracting the temporal correlation between different states from the time series and preserving richer nonlinear dynamic details.

[0082] In the above two-dimensional recursive matrix, each element R i，j All of these reflect the original time series x = (x1, x2, ..., x...). n The relevant information between the two sub-state trajectories in ) is based on d i，j =||s i -s j The magnitude of || represents the degree of correlation between any two trajectory states extracted from the time series. Therefore, the recursive matrix can be used to measure the fluctuation and change of a one-dimensional time series on the time axis.

[0083] Each of the above samples is converted into a corresponding two-dimensional recursive matrix according to each dimension using an improved recursive graph method. That is, a multi-dimensional time series sample is transformed into a combination of multiple two-dimensional recursive matrices, and the combination is used as a sample for training and testing.

[0084] This invention converts a one-dimensional time series into a two-dimensional recursive matrix for analysis. Each element in the sample of the two-dimensional recursive matrix is ​​the Euclidean distance between the trajectories of different sub-states of the original time series, reflecting the correlation between different sub-states. Therefore, it has temporal correlation information, which can obtain more information from the original time series and greatly help feature extraction in classification. It is a good supplement to the existing feature extraction methods in the field of daily activity recognition.

[0085] In step S7, the constructed classification model is trained and tested using the transformed recursive matrix samples from the training and test sets to obtain a trained classification model, wherein the classification model is constructed by combining CNN and LSTM; the construction of the classification model is as follows. Figure 2 As shown, the application scenario uses the WISDM dataset as an example. First, each transformed recursive matrix sample (dimension (3, 78, 78)) is input into a 3-layer CNN network for feature extraction, resulting in a feature dimension of (64, 9, 9). The features (64, 9, 9) extracted by the CNN network are reconstructed sequentially into (64, 81) channels. Then, the reconstructed features are input into an LSTM layer with 128 hidden units to further extract time-related features. The feature dimension after passing through the LSTM layer is (128, 81), and it is then input into a flattening layer to transform the features into a one-dimensional form (1, 10368). Next, the input features (1, 10368) are non-linearly mapped to a higher-dimensional representation space through two fully connected layers. The feature dimension after passing through fully connected layer 1 is (1, 128), and the feature dimension after passing through fully connected layer 2 is (1, 6). Finally, the features (1, 6) from fully connected layer 2 are input into a Softmax activation layer, transforming the output features into a probability distribution to predict the categories of six daily human activities.

[0086] The above 3-layer CNN network structure is as follows Figure 3As shown, each transformed recursive matrix sample (dimension (3, 78, 78)) is first input into a 2D convolutional layer 1 for feature extraction. The 2D convolutional layer 1 has 16 channels (filters) and a kernel size of (3, 3), and the convolution operation maintains the size of the input sample. After passing through the 2D convolutional layer 1, the extracted feature dimension is (16, 78, 78). Then, the extracted intermediate features are input into activation layer 1 to enhance the non-linear expressive power of the network. The features output from activation layer 1 are then input into max pooling layer 1, with a pooling window size of (2, 2) and a stride of (2, 2). After passing through max pooling layer 1, the feature dimension is (16, 39, 39). Subsequently, the intermediate features are sequentially input into 2D convolutional layer 2, activation layer 2, max pooling layer 2, 2D convolutional layer 3, activation layer 3, and max pooling layer 3. In the above steps, only the number of channels (filters) of the kernels in the 2D convolutional layers is set to 16 / 32 / 64 sequentially; all other hyperparameters remain consistent. Ultimately, the intermediate feature dimensions extracted by the 3-layer CNN network are (64, 9, 9).

[0087] When training the constructed CNN-LSTM model, the transformed two-dimensional recursive matrix from the training set is input into a three-layer CNN network. The receptive field is expanded by stacking multiple layers of convolutional kernels to extract hidden local features and global information from the input samples. The convolutional kernels with a large number of channels in each CNN layer are used to extract the channel-related features of the input samples. The features extracted by the three-layer CNN network are flattened and input into the LSTM layer to further extract the time-related information contained in the samples, thereby realizing the extraction of multi-dimensional feature correlation and time-related features of time series.

[0088] CNN modules are well-suited for handling two-dimensional data. Through convolutional layers, they can extract local features from two-dimensional input samples. Furthermore, by stacking multiple convolutional layers, the receptive field is expanded, allowing for the extraction of higher-dimensional, abstract global information. In addition, since each convolutional layer contains a large number of kernels, the number of kernel channels far exceeds the dimensionality of the input data, enabling the extraction of multi-channel correlation features from multi-dimensional recurrent matrices.

[0089] Because multi-layered CNNs can extract local features and multi-channel correlation features from multi-dimensional recursive matrices, and each element in the recursive matrix reflects the relevant information between different sub-states in the corresponding dimension of the time series, the local data implicitly reveals the fluctuations and changes of the corresponding dimension of the time series along the time axis. LSTM models can handle data with sequential changes well, so LSTM models are introduced to further extract the time-related information implicit in the features extracted by CNNs.

[0090] This invention combines CNN and LSTM to construct a classification model, transforming multidimensional time series data into multiple two-dimensional data. By using multiple convolutional kernels to operate on the time series data across multiple channels, features can be adaptively extracted from each channel, thus achieving feature extraction from multidimensional time series and learning the relationships between different features. Based on this, LSTM is further used to extract time-related information, enabling the learning of relationships between various features along the time axis, thereby improving the classification accuracy of daily activity recognition methods.

[0091] In step S8, wearable devices are used to collect data on physical activities generated in a short period of time, which is then uploaded to the OneNet cloud platform via the NB-IoT wireless module. The raw data collected is retrieved from the cloud platform webpage, and a trained daily activity classification model (CNN-LSTM model) is used to classify the daily activities of the human body.

[0092] This invention trains a CNN-LSTM model using a publicly available dataset, then saves the trained model parameters for deployment on a cloud platform for the recognition and classification of daily human activities. This embodiment applies the method of this invention to the publicly available WISDOM dataset (a six-category dataset: going upstairs, going downstairs, sitting, standing, jogging, and walking), collected by a smartphone's built-in inertial sensor, and trains and tests the CNN-LSTM model according to the method of this invention. Figure 4 As shown in the classification confusion matrix results, the identification method proposed in this invention achieves classification accuracy rates of 98.82%, 99.63%, 99.90%, 99.97%, 97.28%, and 99.22% for six daily activities: going downstairs, jogging, sitting, standing, going upstairs, and walking, respectively, which can more accurately predict various daily human activities.

[0093] This invention applies to wearable devices, utilizing them to collect necessary raw data and upload it to a cloud platform. Bluetooth, ZigBee, and Wi-Fi modules are easily limited by transmission distance and are typically only usable indoors. NB-IoT, however, runs on mobile cellular networks. NB-IoT wireless modules have a wide coverage area, making them well-suited for both indoor and outdoor use, enabling real-time monitoring of daily human activities.

[0094] This invention automatically retrieves raw data from cloud platform web pages, processes the raw data, deploys a deep learning-based classification model, analyzes the daily activity status of the human body in real time, and uploads the classification results to the cloud platform, providing a complete Internet of Things solution for fields such as patient rehabilitation management, elderly and child monitoring, and virtual reality interaction.

[0095] The following describes an activity recognition system based on recursive graphs and CNN-LSTM disclosed in an embodiment of the present invention. The activity recognition system based on recursive graphs and CNN-LSTM described below can be referred to in correspondence with the activity recognition method based on recursive graphs and CNN-LSTM described above.

[0096] The present invention also provides an activity recognition system based on recursive graphs and CNN-LSTM, comprising:

[0097] The data acquisition module is used to acquire triaxial accelerometer data of the target user's daily activities over a long period of time, and to label the raw sensor data collected under various activities with the activity type.

[0098] The data processing module is used for preprocessing and normalizing the raw data;

[0099] The sample sorting module is used to divide the data processed by the data processing module into several samples of the same length. The resulting samples are triaxial accelerometer time series. All the extracted samples are divided into training set and test set according to a preset ratio.

[0100] The sample generation module is used to augment the minority class samples in the training set using the SMOTE algorithm;

[0101] The sample cleaning module is used to clean all samples in the expanded training set using the ENN algorithm, filtering and removing noisy samples.

[0102] The sample conversion module is used to convert the acceleration sensing time series data of each dimension of the training set samples cleaned by the sample cleaning module and the test set samples cleaned by the sample sorting module into corresponding two-dimensional recursive matrices using an improved recursive graph algorithm. The new samples obtained after the conversion contain multiple two-dimensional recursive matrices.

[0103] The model training module is used to train and test the constructed classification model using the recursive matrix samples transformed from the training set and the test set, so as to obtain the trained classification model, wherein the classification model is constructed by combining CNN and LSTM.

[0104] The recognition module is used to deploy the trained classification model on the cloud platform and use the classification model to recognize human daily activities.

[0105] In one embodiment of the present invention, the sample conversion module is further configured to use an improved recursive graph algorithm to sequentially convert the acceleration sensing time series data of each dimension in the samples in the training set cleaned by the sample cleaning module and the samples in the test set processed by the sample sorting module into corresponding two-dimensional recursive matrices R′, and extract the temporal correlation between different states from the time series, wherein, m represents the dimension of the extracted state, τ represents the extraction delay, and n represents the time series index.

[0106] In one embodiment of the present invention, when training the constructed CNN-LSTM model, the model training module is used to input the transformed recursive matrix samples from the training set into a 3-layer CNN network. By stacking multiple layers of convolutional kernels, the receptive field is expanded, and hidden local features and global information in the input data are extracted. The convolutional kernels of a large number of channels in each CNN layer are used to extract the channel-related features of the input data. The features extracted by the 3-layer CNN network are flattened and input into the LSTM layer to further extract the time-related information contained in the samples, thereby realizing the extraction of multi-dimensional feature correlation and time-related features of the time series.

[0107] The activity recognition system based on recursive graphs and CNN-LSTM in this embodiment is used to implement the aforementioned activity recognition method based on recursive graphs and CNN-LSTM. Therefore, its specific implementation can be referred to the description of the corresponding embodiments, which will not be elaborated here.

[0108] Furthermore, since the activity recognition system based on recursive graphs and CNN-LSTM in this embodiment is used to implement the aforementioned activity recognition method based on recursive graphs and CNN-LSTM, its function corresponds to that of the above method, and will not be repeated here.

[0109] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An activity recognition method based on recursive graphs and CNN-LSTM, characterized in that: include: S1: Acquire triaxial accelerometer data of the target user's daily activities over a long period of time, and label the raw data collected under various activities with activity type. S2: Perform preprocessing and normalization operations on the raw data obtained in step S1; S3: Divide the data obtained in step S2 into several samples of the same length, and divide all the extracted samples into training set and test set according to a preset ratio; S4: Expand the minority class samples in the training set using the SMOTE algorithm; S5: Use the ENN algorithm to clean all samples in the expanded training set, filter and remove noisy samples; S6: Using an improved recursive graph algorithm, the acceleration sensing time series data of each dimension of the training set samples obtained in step S5 and the test set samples obtained in step S3 are sequentially converted into corresponding two-dimensional recursive matrices. Extracting the temporal correlation between different sub-states from a time series. , This represents the dimension of the extracted state. Indicates the extraction delay. Represents a time series index; S7: Use the recursive matrix samples transformed from the training set and test set samples to train and test the constructed classification model to obtain the trained classification model, wherein the classification model is constructed by combining CNN and LSTM; wherein, each sample is transformed into the corresponding two-dimensional recursive matrix according to each dimension through the improved recursive graph method, that is, a sample is transformed into a combination of multiple two-dimensional recursive matrices, and the combination is used as a sample for training and testing. S8: Deploy the classification model obtained in step S7 on the cloud platform and use the classification model to identify human daily activities.

2. The activity recognition method based on recursive graphs and CNN-LSTM according to claim 1, characterized in that: In step S1, the triaxial accelerometer on the wearable device is used to acquire triaxial accelerometer data of the target user during long-term daily activities.

3. The activity recognition method based on recursive graphs and CNN-LSTM according to claim 1, characterized in that: In step S2, the original data is preprocessed using a median filter and a third-order Butterworth low-pass filter, and the maximum and minimum values ​​of each dimension of the triaxial accelerometer data are normalized sequentially.

4. The activity recognition method based on recursive graphs and CNN-LSTM according to claim 1, characterized in that: In step S4, the method of expanding the minority class samples in the training set using the SMOTE algorithm includes: For each sample in the minority class, calculate the Euclidean distance between that sample and other samples in the same class; Sort the Euclidean distances of each sample to other samples of the same class in ascending order, and select the nearest samples from each sample's class. One sample, based on the oversampling factor For each minority class sample, from its Randomly select from the nearest neighbor samples One sample is used to generate a new sample.

5. The activity recognition method based on recursive graphs and CNN-LSTM according to claim 1, characterized in that: In step S7, when training the constructed CNN-LSTM model, the transformed recursive matrix samples in the training set are input into the 3-layer CNN network. The receptive field is expanded by stacking multiple layers of convolutional kernels to extract the hidden local features and global information in the input data. The convolutional kernels with a large number of channels in each CNN layer are used to extract the channel-related features of the input data. The features extracted by the 3-layer CNN network are flattened and then input into the LSTM layer to further extract the time-related information contained in the sample, thereby realizing the extraction of multi-dimensional feature correlation and time-related features of time series.

6. An activity recognition system based on recursive graphs and CNN-LSTM, characterized in that: include: The data acquisition module is used to acquire triaxial accelerometer data of the target user's daily activities over a long period of time, and to label the raw sensor data collected under various activities with the activity type. The data processing module is used for preprocessing and normalizing the raw data; The sample sorting module is used to divide the data processed by the data processing module into several samples of the same length, and divide all the extracted samples into training set and test set according to a preset ratio. The sample generation module is used to augment the minority class samples in the training set using the SMOTE algorithm; The sample cleaning module is used to clean all samples in the expanded training set using the ENN algorithm, filtering and removing noisy samples. The sample transformation module uses an improved recursive graph algorithm to sequentially convert the acceleration sensing time series data of each dimension of the training set samples (cleaned by the sample cleaning module) and the test set samples (processed by the sample processing module) into corresponding two-dimensional recursive matrices. The transformed sample contains multiple two-dimensional recursive matrices; the temporal correlation between different sub-states is extracted from the time series. , This represents the dimension of the extracted state. Indicates the extraction delay. Represents a time series index; The model training module is used to train and test the constructed classification model using the transformed recursive matrix samples from the training and test sets to obtain a trained classification model, wherein the classification model is constructed by combining CNN and LSTM; wherein each sample is converted into a corresponding two-dimensional recursive matrix according to each dimension through an improved recursive graph method, that is, a sample is converted into a combination of multiple two-dimensional recursive matrices, and the combination is used as a sample for training and testing. The recognition module is used to deploy the trained classification model on the cloud platform and use the classification model to recognize human daily activities.

7. The activity recognition system based on recursive graphs and CNN-LSTM according to claim 6, characterized in that: When training the constructed CNN-LSTM model, the model training module is used to input the transformed recursive matrix samples from the training set into the 3-layer CNN network. By stacking multiple layers of convolutional kernels, the receptive field is expanded, and the hidden local features and global information in the input data are extracted. The convolutional kernels with a large number of channels in each CNN layer are used to extract the channel-related features of the input data. The features extracted by the 3-layer CNN network are flattened and then input into the LSTM layer to further extract the time-related information contained in the sample, thereby realizing the extraction of multi-dimensional feature correlation and time-related features of time series.

Images