A data glove-based dynamic gesture recognition method and system

Abstract

The application provides a kind of dynamic gesture recognition method and system based on data glove, it is related to signal processing and pattern recognition field, the method includes the following steps: S1: the sensing data information of different gesture actions when subject wears data glove is acquired, the sampling frequency of host computer program is adjusted to 30Hz, different gesture actions are collected under dynamic gesture acquisition mode 1-2s, and record and save;S2: the training of gesture recognition model is carried out to the sensing data information collected, MS-1D-CNN neural network is used for training, and the data of three modes are extracted and fused, to change the uniform length of sample to 60 after resampling mode, and then input into CNN;S3: the starting and ending position points of gesture are analyzed by using multi-section threshold detection process.The method can accurately and real-time recognize the gesture actions when subject wears data glove.

Description

A dynamic gesture recognition method and system based on data gloves Technical Field

[0001] This invention relates to the field of signal processing and pattern recognition technology, and more specifically, to a dynamic gesture recognition method and system based on data gloves. Background Technology

[0002] With the development of information technology, gesture recognition technology has gradually shifted from manual recognition to machine intelligent recognition, and it plays an important role in inventions such as AR / VR and smart homes. Current gesture recognition technologies are mainly divided into vision-based recognition, electromyography (EMG) sensor-based recognition, and inertial sensing system-based recognition. These three technologies have different characteristics due to their different application scenarios. Vision recognition technology mainly relies on optical sensor gesture images, extracts and fuses features, and then uses classification algorithms for gesture recognition. Its disadvantage is that it is highly dependent on the clarity of optical images and the complexity of the environment, and its performance is not good enough in many complex scenarios. EMG sensor signal recognition technology mainly detects muscle activity through EMG sensors, extracts features, and then performs recognition. Its drawback is that it can restrict the wearer's movements to a certain extent. Inertial sensor recognition technology is an important research direction in wearable computing. It does not rely on a limited environment, mainly relying on IMUs to capture human body and gesture movements, extracting features through posture calculation and other methods, and using machine learning methods such as SVM for pattern classification. It is relatively limited by sensor accuracy, and inertial sensors are difficult to place on the hand. Furthermore, placing multiple IMUs (Inertial Measurement Units) poses a challenge to hand dexterity. Summary of the Invention

[0003] The problem addressed by this invention is how to accurately and in real-time recognize the hand gestures of subjects wearing data gloves.

[0004] To address the above problems, this invention provides a dynamic gesture recognition method based on a data glove, comprising the following steps:

[0005] S1: Acquire sensor data of different hand gestures of the subject while wearing data gloves, adjust the sampling frequency of the host computer program to 30Hz, collect different hand gestures for 1-2 seconds in dynamic gesture acquisition mode, and record and save them.

[0006] S2: Train the gesture recognition model using the collected sensor data information. The MS-1D-CNN neural network is used for training. The data is divided into three modalities for feature extraction and fusion. The sample length is changed to 60 by resampling before being fed into the CNN.

[0007] S3: Analyze the start and end points of gestures using a multi-segment threshold detection process;

[0008] S4: Apply the trained gesture recognition model to the QT client program to recognize the subject's gestures in real time.

[0009] In the above method, sensor data from a data glove is acquired. The subject wears the data glove and makes corresponding hand gestures. The program samples data points and labels the gesture types; 30-60 data points are selected for each gesture. The raw data is preprocessed, including filtering and denoising, data slicing, and normalization. Data analysis is performed on a large amount of gesture sensor data to design a dynamic gesture recognition algorithm. A multi-stream MS-1D-CNN neural network is used to train the dynamic gestures. The training model with the highest recognition rate on the validation set is selected for the real-time recognition stage. This invention's recognition method can accurately recognize the hand gestures of subjects wearing data gloves in real time. This method has the advantages of high recognition rate and fast analysis speed, and can be widely applied to various scenarios.

[0010] Furthermore, in step S1, the data glove connects to and transmits data to the host computer through its internal wireless communication module;

[0011] The host computer collects multiple sensor data through a serial port, including the bending angle data of the subject's five fingers, and the triaxial acceleration and triaxial angular velocity data of the inertial measurement unit on the back of the data glove.

[0012] In the host computer program, a dataset called Data_Dynamic is generated from the batch data and then stored.

[0013] Further, step S2 includes:

[0014] S21: Divide the dynamic gesture dataset obtained in step S1 into a training set and a test set in a 4:1 ratio, and generate the corresponding dataset file.

[0015] S22: Using a dynamic gesture dataset, design a Multi-Stream 1D-CNN neural network model, extract the fusion features of multimodal data, adjust the hyperparameters of the training, and save the recognition rate model;

[0016] S23: Input the processed dynamic gesture data sample test set into the Multi-Stream1D-CNN neural network for testing, obtain the accuracy of the test set and the prediction time, and select the model.

[0017] Furthermore, the threshold detection process in step S3 includes:

[0018] S31: Continuously receives sensor data from the data glove via Bluetooth host computer, and performs gesture segmentation for continuous gesture data streams;

[0019] S32: The sliding window method is used to capture the inertial measurement unit data of the data glove. The number of acceleration fluctuations of the inertial measurement unit within a window is determined. If the number of fluctuations exceeds the threshold, the dynamic hand gesture action is determined to have started.

[0020] S33: Read the data from the inertial measurement unit of the data glove. In the sliding window, determine the number of non-fluctuation points in the window. If the number exceeds the threshold, determine that the dynamic gesture action has ended.

[0021] A dynamic gesture recognition system based on a data glove, comprising:

[0022] Data acquisition module: used to acquire sensor data of different hand gestures of the subject when wearing data gloves, adjust the sampling frequency of the host computer program to 30Hz, collect different hand gestures for 1-2 seconds in dynamic gesture acquisition mode, and record and save them.

[0023] Training module: Used to train the gesture recognition model on the collected sensor data. It uses MS-1D-CNN neural network for training. The data is divided into three modalities for feature extraction and fusion. The sample length is changed to 60 by resampling before being fed into the CNN.

[0024] Analysis module: Used to analyze the start and end points of gestures using a multi-segment threshold detection process;

[0025] Action recognition module: Used to apply the two trained gesture recognition models to the QT client program to recognize the subject's gestures in real time.

[0026] Furthermore, the data glove connects to and transmits data to a host computer via its internal wireless communication module;

[0027] The host computer collects multiple sensor data through a serial port, including the bending angle data of the subject's five fingers, and the triaxial acceleration and triaxial angular velocity data of the inertial measurement unit on the back of the data glove.

[0028] In the host computer program, a dataset called Data_Dynamic is generated from the batch data and then stored.

[0029] Furthermore, the training module includes:

[0030] Partitioning Unit: Used to divide the dynamic gesture dataset obtained from the data acquisition module into training and test sets in a 4:1 ratio, and generate corresponding dataset files;

[0031] Feature extraction unit: Using a dynamic gesture dataset, design a Multi-Stream 1D-CNN neural network model, extract fusion features from multimodal data, adjust training hyperparameters, and save the recognition rate model;

[0032] Test unit: Used to input the processed dynamic gesture data sample test set into the Multi-Stream 1D-CNN neural network for testing, obtain the accuracy of the test set and the prediction time, and select the model.

[0033] Furthermore, the analysis module includes:

[0034] Segmentation Unit: Used to continuously receive sensor data from the data glove via Bluetooth host computer, and to segment the continuous gesture data stream into gesture segments;

[0035] First judgment unit: The sliding window method is used to capture the inertial measurement unit data of the data glove, and the number of acceleration fluctuations of the inertial measurement unit within a window is judged. If the number of fluctuations exceeds the threshold, the dynamic hand gesture action is judged to have started.

[0036] The second judgment unit is used to read the data from the inertial measurement unit of the data glove. In the sliding window, it judges the number of non-fluctuation points in the window. If the number exceeds the threshold, it determines that the dynamic gesture action has ended.

[0037] The present invention employing the above technical solution has the following beneficial effects:

[0038] This invention enables the recognition of dynamic gestures using a Multi-Stream 1D-CNN neural network model, and provides simplified real-time feedback. The method extracts dynamic gesture data segments through an active-end threshold detection algorithm, performs preprocessing operations such as filtering and normalization, and leverages the characteristics of multi-sensor data to comprehensively mine and fuse multimodal data, achieving a high recognition rate for dynamic gestures. The model training maintains both high recognition rate and speed, making it suitable for embedded system recognition solutions. This method offers advantages such as easy model training, strong robustness, and high recognition rate, making it worthy of widespread application. Attached Figure Description

[0039] Figure 1 is a flowchart of the dynamic gesture recognition method based on data gloves provided in Embodiment 1 of the present invention;

[0040] Figure 2 is a flowchart of the dynamic gesture recognition method based on data gloves provided in Embodiment 1 of the present invention;

[0041] Figure 3 is a schematic diagram of the data glove data processing flow in the dynamic gesture recognition method based on data gloves provided in Embodiment 1 of the present invention;

[0042] Figure 4 is an architecture diagram of the Multi-Stream 1D-CNN neural network in the dynamic gesture recognition method based on data gloves provided in Embodiment 1 of the present invention;

[0043] Figure 5 is a structural diagram of the dynamic gesture recognition system based on data gloves provided in Embodiment 2 of the present invention;

[0044] Figure 6 is a structural diagram of the dynamic gesture recognition system based on data gloves provided in Embodiment 2 of the present invention. Detailed Implementation

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0046] The following are specific embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0047] Example 1

[0048] This embodiment provides a dynamic gesture recognition method based on a data glove, as shown in Figures 1 and 2. The method includes the following steps:

[0049] S1: Acquire sensor data of different hand gestures of the subject while wearing data gloves, adjust the sampling frequency of the host computer program to 30Hz, collect different hand gestures for 1-2 seconds in dynamic gesture acquisition mode, and record and save them.

[0050] S2: Train the gesture recognition model using the collected sensor data information. The model is trained using a 1D-CNN neural network. The data is divided into three modalities for feature extraction and fusion. The sample length is changed to 60 by resampling before being fed into the CNN.

[0051] S3: Analyze the start and end points of gestures using a multi-segment threshold detection process;

[0052] S4: Apply the trained gesture recognition model to the QT client program to recognize the subject's gestures in real time.

[0053] In step S1, the data glove connects to the host computer and transmits data through its internal wireless communication module.

[0054] The host computer collects multiple sensor data through a serial port, including the bending angle data of the subject's five fingers, and the triaxial acceleration and triaxial angular velocity data of the inertial measurement unit on the back of the data glove.

[0055] In the host computer program, a dataset called Data_Dynamic is generated from the batch data and then stored.

[0056] Specifically, data is collected and identified in real time based on the data gloves worn by the subjects, and the recognition accuracy and performance are improved by accurately and effectively segmenting gestures.

[0057] Step S2 includes:

[0058] S21: Divide the dynamic gesture dataset obtained in step S1 into a training set and a test set in a 4:1 ratio, and generate the corresponding dataset file.

[0059] S22: Using a dynamic gesture dataset, design a Multi-Stream 1D-CNN neural network model, extract the fusion features of multimodal data, adjust the hyperparameters of the training, and save the recognition rate model;

[0060] S23: Input the processed dynamic gesture data sample test set into the Multi-Stream1D-CNN neural network for testing, obtain the accuracy of the test set and the prediction time, and select the model.

[0061] The threshold detection process in step S3 includes:

[0062] S31: Continuously receives sensor data from the data glove via Bluetooth host computer, and performs gesture segmentation for continuous gesture data streams;

[0063] S32: The sliding window method is used to capture the inertial measurement unit data of the data glove. The number of acceleration fluctuations of the inertial measurement unit within a window is determined. If the number of fluctuations exceeds the threshold, the dynamic hand gesture action is determined to have started.

[0064] S33: Read the data from the inertial measurement unit of the data glove. In the sliding window, determine the number of non-fluctuation points in the window. If the number exceeds the threshold, determine that the dynamic gesture action has ended.

[0065] Referring to Figure 3, the specific dynamic gesture recognition process using the data glove first involves transmitting data to a host computer via the sensing data glove for preprocessing. This preprocessing includes normalization, data segmentation, filtering, and handling outliers. Specifically, a corresponding host computer program is developed for the serial port data from the lower-level machine, performing timed and periodic data point sampling for different gestures; uniformly labeling the collected gesture data files with action categories; and dividing and saving various gesture data CSV files. Then, multi-stream one-dimensional convolutional gesture recognition is used to understand the meaning of the subject's gestures.

[0066] Specifically, the subjects need to wear data gloves. They connect to the host computer and transmit data through the wireless communication module of the gloves by clicking a button. At the same time, the data acquisition software on the PC acquires 11 channels of sensor data through the serial port, namely the bending angle data of the thumb, index finger, middle finger, ring finger and little finger, as well as the three-axis acceleration and three-axis angular velocity data of the inertial measurement unit on the back of the glove.

[0067] Specifically, the host computer program needs to preprocess the received data, using an extreme value removal and mean filtering algorithm to smooth the six-axis signals and five-channel bending sensor signals from the inertial measurement unit, improving the accuracy of the data in training and inference. The acquired gesture sensing data is then segmented and resampled, scaling the length of each data sample to 60 units. Finally, the MAX-MIN normalization method is used to eliminate dimensions from the 11-channel sensor data, resulting in a series of data samples.

[0068] After steps S1 to S4, the data sample format is obtained as N*11 dimensions, where N is the number of samples, and each sample includes five strain sensors, three acceleration sensors, and a triangular velocity sensor. In the host computer program, the data access module generates the dataset Data_Dynamic from the batch data and stores it locally. Multiple one-dimensional convolutional modules are used to extract data features from elasticity and inertia data separately, which are then fused and classified to avoid feature engineering. The dynamic gesture dataset DATA_DYNAMIC is divided into training and test sets at a 4:1 ratio, generating corresponding dataset files. The input to the neural network is defined as batch*60*11. The multi-stream one-dimensional convolutional neural network will receive three modal stream inputs with specifications of batch*60*5, batch*60*3, and batch*60*3. For each sample data, the inner product of the data, its neighborhood data, and the weights at the corresponding positions of the convolutional kernel matrix is ​​calculated step by step to form the feature map of the next layer. Using one-dimensional convolution, where the kernel matrix is ​​a one-dimensional vector, we will explain the working principle of the convolution model using single-kernel convolution as an example: The input layer is typically set to layer 0.

[0069]

[0070] Where M represents the size of the time window after data preprocessing, and the output of the convolutional layer is:

[0071]

[0072] i = 1, ..., M-φ+1;

[0073] Where f(·) is the activation function; b is the bias term; q is the one-dimensional convolution kernel vector; and φ is the length of q. The data is then fed into the pooling layer MP to reduce the dimensionality of the feature space and suppress overfitting. The MP output is:

[0074]

[0075] Where R represents the size of the pooling window, and the output of the pooling layer is the feature map P output by the convolutional layer.

[0076] After processing by the above algorithm, a set of CNN outputs x can be obtained, which can be flattened into n*1 dimensional sequence data DATA_CNN_OUT.

[0077] The data output DATA_CNN_OUT is fed into the softmax function. The output one-dimensional feature maps are concatenated end-to-end to form a one-dimensional feature vector.

[0078] s = [p 1 ,…,p n ];

[0079] Where n represents the number of convolution kernels. The feature vector s is processed by the fully connected layer, and the output is:

[0080] h = f(Ws + ε);

[0081] Where W is the weight matrix of the fully connected layer, and ε is the bias vector. The result is fed into the sotfmax layer, yielding the final output:

[0082]

[0083] Where x is the input to the softmax layer, i.e., the output of the fully connected layer; K is the number of activity categories, and K is 10. The result of the softmax layer is the probability of each category, which sums to 1.

[0084] Referring to Figure 4, during the algorithm recognition process, the newly input sensor signal also needs to be preprocessed in the same way as described above. Then, the one-dimensional recognition vector is input into the neural network model that has been trained, so that the data of the data glove can be read and recognized in real time, realizing the recognition of dynamic gestures of the data glove.

[0085] Specifically, the interface is developed based on QT to implement human-computer interaction, and the corresponding gesture recognition results can be fed back in real time for the gestures made by the person wearing sensor gloves.

[0086] This method designs a Multi-Stream 1D-CNN neural network model to recognize dynamic gestures and provides simplified real-time feedback. The method extracts dynamic gesture data segments through an active-end threshold detection algorithm, performs preprocessing operations such as filtering and normalization, and leverages the characteristics of multi-sensor data to comprehensively mine and fuse multimodal data, achieving a high recognition rate for dynamic gestures. The model training maintains both high recognition rate and speed, making it suitable for embedded system recognition solutions. This method offers advantages such as easy model training, strong robustness, and high recognition rate, making it worthy of wider application.

[0087] Example 2

[0088] This embodiment provides a dynamic gesture recognition system based on a data glove, as shown in Figures 5 and 6. The system includes:

[0089] Data acquisition module: used to acquire sensor data of different hand gestures of the subject when wearing data gloves, adjust the sampling frequency of the host computer program to 30Hz, collect different hand gestures for 1-2 seconds in dynamic gesture acquisition mode, and record and save them.

[0090] Training module: Used to train the gesture recognition model on the collected sensor data. It uses MS-1D-CNN neural network for training. The data is divided into three modalities for feature extraction and fusion. The sample length is changed to 60 by resampling before being fed into the CNN.

[0091] Analysis module: Used to analyze the start and end points of gestures using a multi-segment threshold detection process;

[0092] Action recognition module: Used to apply the two trained gesture recognition models to the QT client program to recognize the subject's gestures in real time.

[0093] The data glove connects to and transmits data to the host computer via its internal wireless communication module.

[0094] The host computer collects multiple sensor data through a serial port, including the bending angle data of the subject's five fingers, and the triaxial acceleration and triaxial angular velocity data of the inertial measurement unit on the back of the data glove.

[0095] In the host computer program, a dataset called Data_Dynamic is generated from the batch data and then stored.

[0096] Referring to Figure 6, the training module includes:

[0097] Partitioning Unit: Used to divide the dynamic gesture dataset obtained from the data acquisition module into training and test sets in a 4:1 ratio, and generate corresponding dataset files;

[0098] Feature extraction unit: Using a dynamic gesture dataset, design a Multi-Stream 1D-CNN neural network model, extract fusion features from multimodal data, adjust training hyperparameters, and save the recognition rate model;

[0099] Test unit: Used to input the processed dynamic gesture data sample test set into the Multi-Stream 1D-CNN neural network for testing, obtain the accuracy of the test set and the prediction time, and select the model.

[0100] The analysis module includes:

[0101] Segmentation Unit: Used to continuously receive sensor data from the data glove via Bluetooth host computer, and to segment the continuous gesture data stream into gesture segments;

[0102] First judgment unit: The sliding window method is used to capture the inertial measurement unit data of the data glove, and the number of acceleration fluctuations of the inertial measurement unit within a window is judged. If the number of fluctuations exceeds the threshold, the dynamic hand gesture action is judged to have started.

[0103] The second judgment unit is used to read the data from the inertial measurement unit of the data glove. In the sliding window, it judges the number of non-fluctuation points in the window. If the number exceeds the threshold, it determines that the dynamic gesture action has ended.

[0104] This system uses a Multi-Stream 1D-CNN neural network model to recognize dynamic gestures and provides simplified real-time feedback. Dynamic gesture data segments are extracted using an active-end threshold detection algorithm. Preprocessing operations such as filtering and normalization are performed, and the characteristics of multi-sensor data are utilized to comprehensively mine and fuse multi-modal data, achieving a high recognition rate for dynamic gestures. The model training maintains both high recognition rate and speed, making it suitable for embedded system recognition solutions. Data acquisition and real-time recognition are performed based on the subject wearing a data glove, and accurate and effective gesture segmentation improves recognition accuracy and performance.

[0105] While the disclosure is as stated above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of this disclosure, and all such changes and modifications will fall within the protection scope of this invention.

Claims

1. A dynamic gesture recognition method based on a data glove, characterized in that, The steps include: S1: Acquire sensor data of different hand gestures of the subject while wearing data gloves, adjust the sampling frequency of the host computer program to 30Hz, collect different hand gestures for 1-2 seconds in dynamic hand gesture acquisition mode, and record and save them. S2: Train the gesture recognition model using the collected sensor data information. The MS-1D-CNN neural network is used for training. The data is divided into three modalities for feature extraction and fusion. The sample length is changed to 60 by resampling before being fed into the CNN. The process includes: S21: Dividing the dynamic gesture dataset obtained in step S1 into a training set and a test set at a 4:1 ratio, generating corresponding dataset files; S22: Using the dynamic gesture dataset, designing an MS-1D-CNN neural network model, extracting fusion features from multimodal data, adjusting training hyperparameters, and saving the recognition rate model; S23: Inputting the processed dynamic gesture data sample test set into the MS-1D-CNN neural network for testing, obtaining the accuracy and prediction time of the test set, and selecting the model; S3: Analyzing the start and end points of gestures using a multi-segment threshold detection process; the threshold detection process includes... The process includes: S31: Continuously receiving sensor data from the data glove via Bluetooth host computer, and segmenting the gesture data stream into segments; S32: Capturing inertial measurement unit (IMU) data from the data glove using a sliding window method, determining the number of IMU acceleration fluctuations within a window, and determining the start of a dynamic gesture action if the number exceeds a threshold; S33: Reading IMU data from the data glove, determining the number of non-fluctuation points within the sliding window, and determining the end of the dynamic gesture action if the number exceeds a threshold; S4: Applying the trained gesture recognition model to the QT client program to recognize the subject's gesture actions in real time.

2. The dynamic gesture recognition method based on data gloves according to claim 1, characterized in that, In step S1, the data glove connects to and transmits data to the host computer through its internal wireless communication module; the host computer acquires multiple sensor data through a serial port, namely the bending angle data of the subject's five fingers, and the three-axis acceleration and three-axis angular velocity data of the inertial measurement unit on the back of the data glove; the host computer program generates a dataset Data_Dynamic from the batch data and stores the dataset.

3. A dynamic gesture recognition system based on a data glove, executing the dynamic gesture recognition method based on a data glove as described in claim 1, characterized in that, include: Data acquisition module: used to acquire sensor data of different hand gestures of the subject when wearing data gloves, adjust the sampling frequency of the host computer program to 30Hz, collect different hand gestures for 1-2 seconds in dynamic gesture acquisition mode, and record and save them. Training module: Used to train the gesture recognition model on the collected sensor data. It uses an MS-1D-CNN neural network for training, extracts and fuses features from three modalities of data, and feeds the data into the CNN after resampling to make the sample length uniform to 60. Analysis module: Used to analyze the start and end points of gestures using a multi-segment threshold detection process. Action recognition module: This module is used to apply the two trained gesture recognition models to the QT client program to recognize the subject's gestures in real time.

4. The dynamic gesture recognition system based on data gloves according to claim 3, characterized in that, The data glove connects to and transmits data to the host computer via its internal wireless communication module; the host computer acquires multiple sensor data through a serial port, namely the bending angle data of the subject's five fingers, and the three-axis acceleration and three-axis angular velocity data of the inertial measurement unit on the back of the data glove; the host computer program generates a dataset Data_Dynamic from the batch data and stores the dataset.

5. The dynamic gesture recognition system based on data gloves according to claim 4, characterized in that, The training module includes: a partitioning unit, used to partition the dynamic gesture dataset obtained from the data acquisition module into a training set and a test set at a ratio of 4:1, and generate corresponding dataset files; a feature extraction unit, using the dynamic gesture dataset, to design an MS-1D-CNN neural network model, extract fusion features of multimodal data, adjust the hyperparameters of the training, and save the recognition rate model; and a testing unit, used to input the processed dynamic gesture data sample test set into the MS-1D-CNN neural network for testing, obtain the accuracy and prediction time of the test set, and select the model.

6. The dynamic gesture recognition system based on data gloves according to claim 3, characterized in that, The analysis module includes: a segmentation unit, used to continuously receive sensor data from the data glove via a Bluetooth host computer, and segment the continuous gesture data stream into gesture segments; a first judgment unit, used to capture inertial measurement unit data from the data glove using a sliding window method, and judge the number of acceleration fluctuations of the inertial measurement unit within a window. If the number exceeds a threshold, the dynamic gesture action is determined to have started; and a second judgment unit, used to read the data from the inertial measurement unit of the data glove, and judge the number of non-fluctuation points in the sliding window. If the number exceeds a threshold, the dynamic gesture action is determined to have ended.

Images