Tactile gesture recognition method based on pressure sensor

Abstract

The invention relates to the technical field of man-machine interaction, in particular to a tactile gesture recognition method based on a pressure sensor, which comprises the following steps: acquiring sensor data generated by tactile gestures through a pressure sensor array at a preset sampling frequency, and implementing time extension and spatial change data enhancement processing on the preprocessed sensor data to obtain a tactile gesture recognition result. Inputting the data subjected to data enhancement processing into a CNN-LSTM-Attention hybrid neural network model for feature extraction and classification, performing weighted fusion on time sequence features by adopting a sequence aggregation module, performing gesture classification by adopting a classification decision-making module, outputting gesture category probability distribution, and identifying a touch gesture type in real time based on a model output result. And an identification result is displayed through a visual interface. Data are collected through the array pressure sensor, spatial-temporal features are extracted in combination with the CNN-LSTM-Attention hybrid network, real-time and high-precision recognition of various touch gestures is achieved, the influence of noise and sensor drift is reduced, and multi-modal gesture recognition is supported.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a tactile gesture recognition method based on a pressure sensor that integrates a deep learning architecture of convolutional neural network (CNN), long short-term memory network (LSTM) and attention mechanism. Background Technology

[0002] With the rapid development of artificial intelligence technology, tactile information recognition has significant application value in fields such as human-computer interaction, virtual reality, and intelligent robots. Tactile gesture recognition technology enables people to interact naturally and intuitively with computers, smart devices, and the virtual world by sensing and interpreting human hand movements.

[0003] Common tactile information recognition technologies based on pressure sensors mainly employ traditional machine learning methods or combine them with deep learning algorithms. Existing solutions typically use classic algorithms such as Support Vector Machines (SVM) and Random Forests to extract features and classify two-dimensional pressure distribution data collected by pressure sensor arrays. These methods have a certain recognition effect when processing single types of tactile information, but they have significant technical shortcomings when facing complex and varied tactile interaction scenarios, and still have many deficiencies in terms of sensitivity, recognition accuracy, and robustness.

[0004] In existing technologies, from the perspective of sensor usage, visual gesture recognition using computer vision algorithms, which captures images or depth information of user gestures, is highly sensitive to lighting conditions and camera angles. If the gesture is outside the camera's field of view or there are strong shadows, reflections, or other interference, the recognition accuracy will decrease significantly. Sound wave-based tactile recognition, which uses microphones or ultrasonic sensors to sense the sound or sound waves generated by hand movements and analyzes features to identify gestures, is easily affected by environmental noise. Background noise, changes in sound sources, and other factors can lead to a decrease in recognition rate. Tactile recognition based on flexible tactile sensors, which captures pressure distribution using piezoresistive sensors, has insufficient temporal resolution for dynamic gestures (such as swiping or continuous touching) and struggles to balance large range and high precision, making it difficult to capture minute pressure changes or rapid dynamic gestures.

[0005] From the perspective of deep learning technology implementation, there are defects such as insufficient recognition accuracy, lack of time dimension information, poor algorithm robustness, weak environmental adaptability and insufficient real-time performance. Specifically: (1) Traditional machine learning methods rely on manually designed feature extraction, which is difficult to effectively capture the complex spatiotemporal features in pressure distribution data, resulting in low recognition accuracy, especially when dealing with subtle tactile differences, resulting in insufficient recognition accuracy; (2) Existing systems have low sampling frequency and cannot accurately record the temporal features of gestures, resulting in difficulty in recognizing dynamic gestures such as sliding direction and continuous drawing, resulting in a lack of time dimension information; (3) Most solutions rely on a single-structure neural network and do not fully combine spatiotemporal features, resulting in insufficient classification accuracy for complex gestures, resulting in poor algorithm robustness; (4) Sensors are easily affected by external factors such as temperature and humidity, resulting in poor signal stability, resulting in weak environmental adaptability; (5) Traditional algorithms have high computational complexity when processing high-dimensional pressure sensor array data, and the implementation of offline temporal image recognition is difficult to meet the needs of real-time interaction, affecting user experience, resulting in insufficient real-time performance. Despite the breakthroughs made by deep learning technology in fields such as computer vision and speech recognition, its application in the field of tactile information recognition is still in its infancy and is difficult to meet the actual needs of complex tactile interaction scenarios.

[0006] Therefore, the existing field of tactile gesture recognition urgently needs to develop a tactile information recognition technology based on deep learning and using pressure sensors with high sensitivity and high spatiotemporal resolution. This technology can effectively solve the technical defects of existing technologies, such as low recognition accuracy, poor robustness, and limited functionality, and provide more accurate, stable, and comprehensive tactile information recognition capabilities for applications such as intelligent robots and touch devices. Summary of the Invention

[0007] To address the technical problems of insufficient recognition accuracy, poor robustness, and limited functionality in existing tactile recognition technologies, this invention provides a tactile information recognition method based on an off-grid capacitive pressure sensor array using CNN-LSTM deep learning. This method collects data using a high-sensitivity 16×16 array pressure sensor and extracts spatiotemporal features using a CNN-LSTM-Attention hybrid network. This enables real-time, high-precision recognition of various tactile gestures, reducing the impact of noise and sensor drift, supporting multimodal gesture recognition, optimizing the network structure and data processing flow to ensure low recognition latency, and using data augmentation and regularization techniques to adapt the system to new users and environments, improving generalization ability. This method can be widely applied in fields such as intelligent robots, human-computer interaction systems, and touch devices.

[0008] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0009] This invention provides a tactile gesture recognition method based on a pressure sensor, comprising the following steps:

[0010] Sensor data generated by tactile gestures is collected using a pressure sensor array at a preset sampling frequency, and the collected sensor data is preprocessed.

[0011] Data augmentation processing, including temporal expansion and spatial transformation, is applied to the preprocessed sensor data. The temporal expansion adjusts the time series length through interpolation and sampling, while the spatial transformation is achieved by detecting the gesture bounding box and performing rotational or translational geometric transformations.

[0012] The data after data augmentation is input into a CNN-LSTM-Attention hybrid neural network model for feature extraction and classification. The hybrid neural network model uses a convolutional feature extraction module to extract spatial features, a temporal modeling module to extract temporal features, a sequence aggregation module to weight and fuse temporal features, and a classification decision module to classify gestures and output the probability distribution of gesture categories.

[0013] The model output results are used to identify tactile gesture types in real time, and the identification results are displayed through a visual interface.

[0014] As a further aspect of the present invention, when collecting sensor data generated by tactile gestures, the data acquisition unit collects the sensor data through serial communication. The pressure sensor array adopts a 16×16 array pressure sensor with a total of 256 sensor nodes and a sampling frequency of 100Hz. Each frame of data contains 256 pressure values. After parsing the signal, the timing data of 256 channels is sent to the host computer.

[0015] As a further embodiment of the present invention, in the tactile gesture recognition method based on pressure sensor, the pressure sensor is a 16*16 array pressure sensor, PD packaged, 80mm in length, model SYCSPMX1616X1Y2PD-80; the data acquisition unit is a 256-channel acquisition unit module arranged in rows and columns, model SCM161610.

[0016] As a further aspect of the present invention, the collected sensor data is preprocessed, including data calibration, outlier filtering, and noise suppression. The data calibration uses an adaptive baseline calibration algorithm to calculate the sensor offset value, the outlier filtering replaces data exceeding the threshold based on historical data, and the noise suppression filters low-amplitude noise through a mask threshold.

[0017] As a further aspect of the present invention, the data calibration includes the following steps:

[0018] Historical data sequences are stored using a fixed-length circular buffer. ,in, The buffer's historical length;

[0019] Through calibration function Calculate the first Calibration offset values ​​of each sensor ,in, For the first Historical data sequences from each sensor This is the calibration coefficient, with a value of 1.01.

[0020] By calibration equation Obtain the calibrated data and ensure that the calibrated data is non-negative. The data values ​​after calibration. This is the raw sensor data.

[0021] As a further aspect of the present invention, the outlier filtering includes the following steps:

[0022] Set outlier threshold When the raw data When using a historical normal value replacement function Perform the replacement, where is the historical normal value lookup function:

[0023] .

[0024] As a further aspect of the present invention, the noise suppression includes the following steps:

[0025] Set noise mask threshold Through noise suppression function Filter low-amplitude noise.

[0026] As a further aspect of the present invention, the spatial change includes the following steps:

[0027] An activation point detection function is applied to the sensor data to identify the valid gesture region in each frame; wherein, the activation point detection function is: In the formula, For position exist The pressure value at any moment, The stress activation threshold, ;

[0028] Calculate the temporal activation map and the gesture bounding box boundary, wherein the temporal activation map is: The gesture bounding box includes , , and ,in:

[0029] ;

[0030] ;

[0031] ;

[0032] ;

[0033] The determined gesture bounding box area is:

[0034] ;

[0035] ;

[0036] ;

[0037] In the formula, BBox represents the bounding box area, and height and width represent the length and width of the bounding box.

[0038] Within the sensor space constraints, the effective translation area of ​​the current bounding box: , In the formula, H=16, W=16, which are the array size of the sensor;

[0039] Data translation is performed based on the bounding box. The data translation operation function is as follows:

[0040] ;

[0041] in, and The translation amount, , .

[0042] As a further aspect of the present invention, the CNN-LSTM-Attention hybrid neural network adopts an end-to-end learning architecture, including:

[0043] Input layer: Receives a four-dimensional pressure data tensor of shape (batch_size, n, height, width), supports variable-length sequence input, where n is the time series length, and height and width are the length and width of the pressure sensor array, which are 16×16.

[0044] Convolutional feature extraction module: includes a two-layer convolutional neural network. The first layer uses 32 3×3 convolutional kernels, padded with 1s to maintain the spatial size, and adopts the ReLU activation function. The second layer uses 64 3×3 convolutional kernels to expand the 32-channel feature map to 64 channels. Combined with max pooling and dropout layers, it outputs a 1024-dimensional spatial feature vector as the input for temporal modeling.

[0045] Temporal modeling module: includes a two-layer unidirectional LSTM network with an input dimension of 1024, 32 hidden units, an inter-layer dropout rate of 0.5, supports variable-length sequence processing, and outputs a temporal feature sequence with shape (batch_size, n, 32).

[0046] Sequence aggregation module: It uses an attention mechanism to calculate the attention score through a 32→1 linear transformation, and combines the sequence length mask and Softmax normalization to output a 32-dimensional global feature representation;

[0047] Classification Decision Module: A single-layer fully connected network is used for gesture classification. The fully connected layer is mapped to the number of gesture categories, and the Softmax activation function is used to output the probability distribution of gesture types.

[0048] As a further aspect of the present invention, the training optimization strategy of the CNN-LSTM-Attention hybrid neural network includes:

[0049] The AdamW adaptive optimizer is used with an initial learning rate of 1e-3, combined with a weight decay mechanism.

[0050] The loss function used is label-smoothed cross-entropy loss, with a smoothing parameter of 0.05;

[0051] The training batch size is 32, the number of epochs is 50, and the data is split into 80% training set, 10% validation set, and 10% test set.

[0052] Regularization techniques include: L2 weight decay with a weight decay coefficient of 1e-4; multi-layer dropout strategy with a dropout rate of 0.2 for convolutional layers, a dropout rate of 0.5 between LSTM layers, and dropout regularization before the classification layer; batch normalization; and variable-length sequence masking.

[0053] As a further aspect of the present invention, the tactile gesture recognition method based on pressure sensors supports the recognition of one or more combinations of gesture categories, including swipe up, swipe down, swipe left, swipe right, two-finger operation, clockwise, counterclockwise, hook, cross, single click, double click, and long press.

[0054] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0055] This invention presents a tactile gesture recognition method based on pressure sensors. Utilizing a pressure sensor array and a CNN-LSTM-Attention deep learning architecture, it exhibits outstanding recognition accuracy. The hybrid neural network architecture effectively extracts spatiotemporal features, achieving fine-grained gesture recognition. Furthermore, the model supports dynamic sequence length input, avoiding the computational waste of fixed-length methods and improving processing efficiency. The real-time system employs a buffer to process data, ensuring parallel execution of data acquisition, preprocessing, and recognition. The CNN-LSTM-Attention architecture has low computational complexity, and combined with dropout layers and batch normalization, it accelerates the inference process. The real-time recognition system interface intuitively displays pressure distribution and gesture trajectory, providing users with immediate feedback. The hybrid architecture of this invention also significantly enhances the feature representation capability of tactile information and achieves strong generalization ability through data augmentation and regularization techniques, requiring only a small amount of real data to adapt to new users and environments. This invention's tactile gesture recognition method based on pressure sensors, through the integration of deep learning and optimized preprocessing, achieves high accuracy, low latency, strong generalization, and high robustness, possessing broad application value in fields such as intelligent robots, touch devices, and human-computer interaction systems, thus promoting technological advancement.

[0056] These or other aspects of the invention will become more apparent from the following description of embodiments. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. In the drawings:

[0058] Figure 1 This is a framework diagram of the tactile information recognition system in a tactile gesture recognition method based on a pressure sensor according to the present invention.

[0059] Figure 2 This is a schematic diagram of the CNN-LSTM-Attention model architecture in a tactile gesture recognition method based on a pressure sensor according to the present invention.

[0060] Figure 3 This is a diagram of the real-time recognition system interface in a tactile gesture recognition method based on a pressure sensor according to the present invention.

[0061] Figure 4 This is a heat map of the trajectory of example data (counterclockwise circle gesture) in a tactile gesture recognition method based on a pressure sensor according to the present invention.

[0062] Figure 5This is a time series diagram of pressure intensity, which is example data (counterclockwise circle gesture) in a tactile gesture recognition method based on a pressure sensor according to the present invention.

[0063] Figure 6 This is a schematic diagram of the confusion matrix in a tactile gesture recognition method based on a pressure sensor according to the present invention.

[0064] Figure 7 This is a T-sne diagram in a tactile gesture recognition method based on a pressure sensor according to the present invention. Detailed Implementation

[0065] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0066] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0067] See Figures 1 to 7 As shown, embodiments of this application also provide a tactile gesture recognition method based on a pressure sensor, including the following steps:

[0068] Step S10: Collect sensor data generated by tactile gestures through a pressure sensor array at a preset sampling frequency, and preprocess the collected sensor data.

[0069] Step S20: Perform time-expansion and spatial-variation data augmentation processing on the preprocessed sensor data, wherein the time expansion is adjusted by interpolation and sampling to adjust the length of the time series, and the spatial variation is performed by detecting the gesture bounding box and performing rotation or translation geometric transformations.

[0070] Step S30: Input the data after data augmentation into the CNN-LSTM-Attention hybrid neural network model for feature extraction and classification. The hybrid neural network model uses a convolutional feature extraction module to extract spatial features, a temporal modeling module to extract temporal features, a sequence aggregation module to weight and fuse temporal features, and a classification decision module to classify gestures and output the probability distribution of gesture categories.

[0071] Step S40: Based on the model output results, identify the type of tactile gesture in real time and display the identification results through a visual interface.

[0072] This invention presents a tactile gesture recognition method based on pressure sensors, specifically a tactile information recognition method using an off-cell capacitive pressure sensor array based on CNN-LSTM deep learning. It aims to address the technical problems of insufficient recognition accuracy, poor robustness, and limited functionality in existing tactile recognition technologies. This invention improves the accuracy and robustness of tactile information recognition by designing and optimizing a CNN-LSTM-Attention hybrid neural network structure to effectively extract spatiotemporal features from the pressure sensor array data, thereby enhancing the recognition accuracy for complex tactile information and strengthening the system's resistance to interference factors such as noise and sensor drift. This invention achieves multimodal tactile information recognition, establishing a recognition framework capable of simultaneously recognizing multiple tasks such as sliding direction and gesture pattern drawing. It enhances the system's real-time performance and response speed by optimizing the network structure and algorithm flow, reducing computational complexity, ensuring real-time tactile information recognition, and meeting the real-time response requirements of human-computer interaction. Furthermore, it improves the system's generalization ability and adaptability by enhancing the system's adaptability to new users, new environments, and new gesture types through improved training strategies and data processing methods, thereby increasing the practicality and promotional value of the technology. By solving the above-mentioned technical problems, this invention aims to provide an accurate, stable, comprehensive, and real-time tactile information recognition solution for application fields such as intelligent robots, touch devices, and human-computer interaction systems, thereby promoting the development and application of tactile interaction technology.

[0073] In this embodiment, the tactile gesture recognition method based on pressure sensors of the present invention recognizes tactile information using a pressure sensor array based on a CNN-LSTM-Attention deep learning architecture. It collects tactile information through a 16×16 pressure sensor array and combines it with deep learning algorithms to accurately recognize various gestures such as sliding direction and gesture pattern drawing. Figure 1 As shown, the tactile gesture recognition method of the pressure sensor in this embodiment is implemented based on a tactile information recognition system. The tactile information recognition system of this invention includes a data acquisition module, a data preprocessing module, a deep learning module, a recognition result output module, and a visualization module. The data acquisition module is responsible for collecting spatial-temporal pressure data generated on a 16×16 flexible pressure sensor array after a specific gesture is performed using a signal acquisition device. The data preprocessing module performs filtering, normalization, and data augmentation on the raw data, including spatial expansion and temporal expansion. The deep learning module uses a CNN-LSTM-Attention hybrid network structure for feature extraction and pattern recognition, and generates parameters for the tactile recognition model after training, verification, and testing. By inputting sample data or real-time data into the tactile recognition model, the recognition result output module outputs the final tactile information recognition result and displays it on a visualization platform.

[0074] In this embodiment, when collecting sensor data generated by tactile gestures, the data acquisition unit collects the sensor data via serial communication. The pressure sensor array uses a 16×16 array pressure sensor with a total of 256 sensor nodes and a sampling frequency of 100Hz. Each frame of data contains 256 pressure values. After parsing the signal, the timing data of 256 channels is sent to the host computer. The pressure sensors and data acquisition unit used in this embodiment are shown in Table 1.

[0075] Table 1. Hardware List of Pressure Sensors and Data Acquisition Units

[0076] Hardware type Hardware Name model pressure sensor Saigan High Sensitivity Second Generation Flexible 16*16 Array Pressure Sensor - PD Package - 80mm Length SYCSPMX1616X1Y2PD-80 Data acquisition device Saigan 256-channel version 1.0 collector module - determinant SCM161610

[0077] In the tactile gesture recognition method based on pressure sensors in this embodiment, the pressure sensor is a 16*16 array pressure sensor, PD packaged, 80mm in length, model SYCSPMX1616X1Y2PD-80; the data acquisition unit is a 256-channel acquisition module with row and column arrangement, model SCM161610.

[0078] In step S10 of the present invention, the collected sensor data is preprocessed, including data calibration, outlier filtering and noise suppression. The data calibration uses an adaptive baseline calibration algorithm to calculate the sensor offset value, the outlier filtering replaces the data exceeding the threshold based on historical data, and the noise suppression filters low-amplitude noise through a mask threshold.

[0079] The data calibration includes the following steps:

[0080] Historical data sequences are stored using a fixed-length circular buffer. ,in, This is the buffer's historical length; when the buffer is full, the oldest data is automatically overwritten.

[0081] Through calibration function Calculate the first Calibration offset values ​​of each sensor ,in, For the first Historical data sequences from each sensor This is the calibration coefficient, with a value of 1.01.

[0082] By calibration equation Obtain the calibrated data and ensure that the calibrated data is non-negative. The data values ​​after calibration. This is the raw sensor data.

[0083] Data calibration employs an adaptive baseline calibration algorithm to eliminate initial offset and environmental interference from the sensor array. A calibration function is used to calculate the calibration offset value corresponding to the sensor at a specific location. Then, the original sensor value is subtracted from the calibration offset value to obtain the calibrated value, ensuring that the data is non-negative and updating the value.

[0084] The outlier filtering includes the following steps:

[0085] To prevent high-amplitude noise from contaminating the data, an outlier threshold is set. When the raw data When replacing outliers in the original data with normal values ​​from historical data, the replacement function uses historical normal values. Perform the replacement, where is the historical normal value lookup function:

[0086] .

[0087] The noise suppression includes the following steps:

[0088] To prevent low-noise contamination of data, noise suppression is applied to the raw data by using a noise mask. In this embodiment, a noise mask threshold is set. Through noise suppression function Filter low-amplitude noise.

[0089] In step S20 of this invention, to improve the model's generalization ability and reduce overfitting, the training dataset is effectively expanded while maintaining the gesture features. Two data augmentation operations are employed: temporal expansion and spatial transformation. Temporal expansion adjusts the time series length through interpolation and sampling, taking equally spaced positions. The interpolation operation uses data from the previous frame for expansion, while the sampling operation deletes data from the current frame at equally spaced positions, thus achieving temporal data augmentation while preserving the overall gesture contour. Spatial transformation involves geometric transformations such as rotation and translation of the sensor array data. First, the gesture information contained in each 16×16 data frame matrix needs to be extracted. Gesture bounding boxes need to be detected to identify the effective gesture region in each frame, and an activation point detection function is applied to the original data.

[0090] In this embodiment, the spatial change includes the following steps:

[0091] An activation point detection function is applied to the sensor data to identify the valid gesture region in each frame; wherein, the activation point detection function is: In the formula, For position exist The pressure value at any moment, The stress activation threshold, ;

[0092] Calculate the temporal activation map and the gesture bounding box boundary, wherein the temporal activation map is: The gesture bounding box includes , , and ,in:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] The determined gesture bounding box area is:

[0098] ;

[0099] ;

[0100] ;

[0101] In the formula, BBox represents the bounding box area, and height and width represent the length and width of the bounding box.

[0102] Within the sensor space constraints, the effective translation area of ​​the current bounding box: , In the formula, H=16, W=16, which are the array size of the sensor;

[0103] Data translation is performed based on the bounding box. The data translation operation function is as follows:

[0104] ;

[0105] in, and The translation amount, , .

[0106] In this case, pressure sensor data was acquired using a sampling frequency of 100 frames per second. Each gesture sample supported variable-length sequence input, fully preserving the complete temporal information of the gesture. The original dataset was constructed through manual annotation, with each gesture category containing 10–30 manually collected original samples. The remaining samples were generated using various data augmentation techniques to ensure the diversity and generalization ability of the dataset.

[0107] As shown in Table 2, the dataset covers 19 different gesture categories, totaling 48,930 training samples. The sequence length distribution is as follows: the shortest gesture data is 11 frames, the longest gesture data is 676 frames, and the average is 106.9 frames. The pressure sensor-based tactile gesture recognition method supports the recognition of one or more combinations of gesture categories, including swipe up, swipe down, swipe left, swipe right, two-finger operation, clockwise, counterclockwise, hook, cross, single click, double click, and long press.

[0108] Table 2 Gesture Data Statistics

[0109] Types of gestures Sample size Average length (frames) Slide up 3275 141.3 decline 3175 139.5 Swipe left 3130 142.7 Swipe right 3260 139.5 Swipe up with two fingers 1385 70.5 Two fingers slide down 1385 70.7 Swipe left with two fingers 1680 83.9 Swipe right with two fingers 1575 78.8 Top left 1840 80.5 Bottom left 1840 80.4 Top right 1825 80.3 Bottom right 1825 80.1 clockwise 1580 111.0 counterclockwise 1155 116.9 hook 2020 95.5 fork 5820 109.4 Click 5045 52.5 double click 4395 77.4 Long press 2720 232.0

[0110] In step S30 of the present invention, the CNN-LSTM-Attention hybrid neural network adopts an end-to-end learning architecture, including:

[0111] Input layer: Receives a four-dimensional pressure data tensor of shape (batch_size, n, height, width), supports variable-length sequence input, where n is the length of the time series, and height and width are the length and width of the pressure sensor array, which are 16×16.

[0112] Convolutional feature extraction module: Includes a two-layer convolutional neural network. The first layer uses 32 3×3 convolutional kernels, padded with 1s to maintain the spatial size, and adopts the ReLU activation function. The second layer uses 64 3×3 convolutional kernels to expand the 32-channel feature map to 64 channels. Combined with max pooling and dropout layers, it outputs a 1024-dimensional spatial feature vector as the input for temporal modeling.

[0113] Specifically, the convolutional feature extraction module uses a two-layer convolutional neural network combined with optional batch normalization techniques to extract spatial features, including:

[0114] First convolutional layer: 32 3×3 convolutional kernels, padded with 1 to maintain the spatial size, using the ReLU activation function;

[0115] First batch normalization layer: Normalizes 32 feature channels (optional) to accelerate convergence and improve training stability;

[0116] First pooling layer: 2×2 max pooling, downsampling the 16×16 feature map to 8×8;

[0117] First dropout layer: dropout rate of 0.2 to prevent overfitting;

[0118] The second convolutional layer consists of 64 3×3 convolutional kernels, which expand the 32-channel feature map to 64 channels.

[0119] Second batch normalization layer: Normalizes 64 feature channels (optional);

[0120] Second pooling layer: 2×2 max pooling, which further downsamples the 8×8 feature map to 4×4;

[0121] Second dropout layer: dropout rate of 0.2 to prevent overfitting;

[0122] Feature flattening: The 64×4×4=1024-dimensional spatial feature vector is used as the input for time series modeling.

[0123] The temporal modeling module consists of a two-layer unidirectional LSTM network with an input dimension of 1024, 32 hidden units, and an inter-layer dropout rate of 0.5. It supports variable-length sequence processing and outputs a temporal feature sequence of shape (batch_size, n, 32). Specifically, the temporal modeling module uses a two-layer unidirectional Long Short-Term Memory (LSTM) network to extract temporal dependencies, including:

[0124] Two-layer LSTM network: 1024 input dimensions, 32 hidden units, and 0.5 dropout rate between layers;

[0125] Variable-length sequence support: Efficient variable-length sequence processing is achieved using pack_padded_sequence and pad_packed_sequence;

[0126] Output features: a time-series feature sequence with shape (batch_size, n, 32).

[0127] The sequence aggregation module employs an attention mechanism to calculate the attention score through a 32→1 linear transformation, and combines this with sequence length masking and Softmax normalization to output a 32-dimensional global feature representation. This module supports two aggregation strategies to effectively integrate temporal features:

[0128] Attention mechanism model: The attention score is calculated by using a 32→1 linear transformation, and adaptive weighted fusion is performed on the effective time steps by combining the sequence length mask and Softmax normalization.

[0129] Last time step mode: directly extracts the features of the last valid time step of each sequence;

[0130] Context vector: Outputs a 32-dimensional global feature representation.

[0131] Classification Decision Module: This module uses a single-layer fully connected network for gesture classification. The fully connected layer maps to the number of gesture categories, and the Softmax activation function outputs the probability distribution of gesture types. Specifically, the classification decision module uses a single-layer fully connected network to achieve the final gesture classification.

[0132] Fully connected layer: 32-dimensional mapping to num_classes, supporting dynamic configuration of the number of classes (default 8 classes, expandable to any number of classes);

[0133] Dropout layer: Dropout regularization is applied during training;

[0134] Output layer: Outputs the probability distribution of each category through the Softmax activation function, supporting fine-grained recognition of multiple gesture types.

[0135] In this embodiment, the training optimization strategy of the CNN-LSTM-Attention hybrid neural network includes:

[0136] The AdamW adaptive optimizer is used with an initial learning rate of 1e-3, combined with a weight decay mechanism.

[0137] The loss function uses label-smoothed cross-entropy loss with a smoothing parameter of 0.05 to reduce the risk of overfitting.

[0138] The training batch size is 32, the number of epochs is 50, and the data is split into 80% training set, 10% validation set, and 10% test set; the early stopping mechanism is patience=10, and the minimum improvement threshold min_delta=1e-4 to prevent overfitting.

[0139] Regularization techniques include: L2 weight decay, with a weight decay coefficient of 1e-4, effectively constraining model complexity; multi-layer dropout strategy, with a dropout rate of 0.2 for convolutional layers, a dropout rate of 0.5 between LSTM layers, and dropout regularization before the classification layer; batch normalization and variable-length sequence masking; the variable-length sequence masking automatically hides the padding part to avoid invalid computation.

[0140] In this embodiment, the model parameter scale of the entire network contains 162,985 trainable parameters, which are distributed as follows:

[0141] CNN feature extraction module: approximately 18,816 parameters (37.6%).

[0142] LSTM module: approximately 30,000 parameters (60.0%).

[0143] Attention mechanism module: 33 parameters (0.1%);

[0144] Classification decision module: 256 parameters (0.5%), supports multi-type gesture recognition.

[0145] The real-time recognition system employs a data caching mechanism. Raw data for each frame is stored in a data buffer, and subsequent data processing, gesture prediction, and data saving all retrieve data from this buffer, ensuring both real-time performance and recognition accuracy. Figure 3 As shown, the system supports multiple visualization displays: (1) Real-time pressure heat map / curve graph displays the current pressure distribution; (2) Heat overlay graph displays the heat map overlay of each frame of the captured gesture, presenting the overall outline of the gesture; (3) Captured gesture curve graph displays the pressure curve of the captured gesture; (4) Real-time feedback of recognition results.

[0146] The method in this embodiment includes 19 gesture types; on the test set, the average recognition accuracy reaches 98.94%; thanks to the support for variable-length sequence input and the model architecture, the average recognition latency is less than 100ms, meeting the real-time requirements of human-computer interaction; the CNN-LSTM-Attention hybrid architecture effectively integrates spatial and temporal features, improving recognition performance compared to a single network structure; through data augmentation techniques, only a small amount of raw gesture data from a single person (10-30 samples for each gesture) needs to be collected to create a large amount of data, so that the trained model can also accurately recognize other people; at the same time, through regularization and other techniques, the training process achieves a good fit, and the loss trends of the training set and validation set are synchronized; through regularization and other techniques, the model has a certain anti-interference ability and can make correct recognition even for certain signal crosstalk and residual signals.

[0147] like Figure 6 The confusion matrix results show that the average recognition accuracy for various gestures is 98.94%, and as... Figure 7 The t-SNE diagram shown demonstrates that the model possesses clear recognition boundaries for the 19 gestures, proving the effectiveness and practicality of the technical solution of this invention. In this embodiment, different model architectures are used for learning and training, including but not limited to: the Transformer-based spatiotemporal attention mechanism can use Vision Transformer (ViT) instead of CNN for spatial feature extraction, dividing the 16×16 sensor array into 4×4 patches for processing; or Temporal Transformer can be used instead of LSTM for temporal modeling, capturing long-term dependencies through a self-attention mechanism. For spatiotemporal integrated processing based on 3D convolutional neural networks, 3D-CNN is used to directly process the (T×16×16) three-dimensional data volume while simultaneously extracting spatiotemporal features.

[0148] In this embodiment, different data preprocessing methods are used to acquire training data, including but not limited to: time-frequency domain feature extraction based on wavelet transform, performing wavelet transform on time-series data to extract time-frequency domain features; or using other data augmentation methods to preprocess the data, such as different geometric transformations, color transformation operations, and multi-sample data augmentation based on interpolation. When modeling the sensor array based on graph neural networks, the 16×16 sensor array is modeled as a graph structure, with sensor nodes connected through adjacency relationships, and spatial correlation features are extracted using graph convolutional neural networks (GCN). Different learning tasks are adopted, such as changing the implemented gesture actions, such as implementing other gestures based on this case, such as three-finger gestures, which essentially only changes the input of the training data. This model can complete similar gesture tasks. Alternatively, sensors with different numbers of channels can be used for data acquisition, and the length and width of the sensor array can be set to other values. Using multiple sensors with different input dimensions does not essentially change the training content of the model. For such inputs, only simple data processing is required, and the model in this case can still be used for training.

[0149] This invention presents a tactile gesture recognition method based on pressure sensors. Utilizing a pressure sensor array and a CNN-LSTM-Attention deep learning architecture, it exhibits outstanding recognition accuracy. The hybrid neural network architecture effectively extracts spatiotemporal features, achieving fine-grained gesture recognition. Furthermore, the model supports dynamic sequence length input, avoiding the computational waste of fixed-length methods and improving processing efficiency. The real-time system employs a buffer to process data, ensuring parallel execution of data acquisition, preprocessing, and recognition. The CNN-LSTM-Attention architecture has low computational complexity, and combined with dropout layers and batch normalization, it accelerates the inference process. The real-time recognition system interface intuitively displays pressure distribution and gesture trajectory, providing users with immediate feedback. The hybrid architecture of this invention also significantly enhances the feature representation capability of tactile information and achieves strong generalization ability through data augmentation and regularization techniques, requiring only a small amount of real data to adapt to new users and environments. This invention's tactile gesture recognition method based on pressure sensors, through the integration of deep learning and optimized preprocessing, achieves high accuracy, low latency, strong generalization, and high robustness, possessing broad application value in fields such as intelligent robots, touch devices, and human-computer interaction systems, thus promoting technological advancement.

[0150] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A tactile gesture recognition method based on a pressure sensor, characterized in that, Includes the following steps: Sensor data generated by tactile gestures is collected using a pressure sensor array at a preset sampling frequency, and the collected sensor data is preprocessed. Data augmentation processing, including temporal expansion and spatial transformation, is applied to the preprocessed sensor data. The temporal expansion adjusts the time series length through interpolation and sampling, while the spatial transformation is achieved by detecting the gesture bounding box and performing rotational or translational geometric transformations. The data after data augmentation is input into a CNN-LSTM-Attention hybrid neural network model for feature extraction and classification. The hybrid neural network model uses a convolutional feature extraction module to extract spatial features, a temporal modeling module to extract temporal features, a sequence aggregation module to weight and fuse temporal features, and a classification decision module to classify gestures and output the probability distribution of gesture categories. The model output results are used to identify tactile gesture types in real time, and the identification results are displayed through a visual interface.

2. The tactile gesture recognition method based on a pressure sensor as described in claim 1, characterized in that, When collecting sensor data generated by tactile gestures, the data acquisition unit collects sensor data through serial communication. The pressure sensor array adopts a 16×16 array pressure sensor with a total of 256 sensor nodes and a sampling frequency of 100Hz. Each frame of data contains 256 pressure values. After parsing the signal, the timing data of 256 channels is sent to the host computer.

3. The tactile gesture recognition method based on a pressure sensor as described in claim 1, characterized in that, The collected sensor data is preprocessed, including data calibration, outlier filtering, and noise suppression. Data calibration uses an adaptive baseline calibration algorithm to calculate sensor offset values. Outlier filtering replaces data exceeding the threshold based on historical data. Noise suppression filters low-amplitude noise through a mask threshold.

4. The tactile gesture recognition method based on a pressure sensor as described in claim 3, characterized in that, The data calibration includes the following steps: Historical data sequences are stored using a fixed-length circular buffer. ,in, The buffer's historical length; Through calibration function Calculate the first Calibration offset values ​​of each sensor ,in, For the first Historical data sequences from each sensor This is the calibration coefficient, with a value of 1.

01. By calibration equation Obtain the calibrated data and ensure that the calibrated data is non-negative. The data values ​​after calibration. This is the raw sensor data.

5. The tactile gesture recognition method based on a pressure sensor as described in claim 3, characterized in that, The outlier filtering includes the following steps: Set outlier threshold When the raw data When using a historical normal value replacement function Perform the replacement, where is the historical normal value lookup function: 。 6. The tactile gesture recognition method based on a pressure sensor as described in claim 3, characterized in that, The noise suppression includes the following steps: Set noise mask threshold Through noise suppression function Filter low-amplitude noise.

7. The tactile gesture recognition method based on a pressure sensor as described in claim 1, characterized in that, The spatial changes include the following steps: An activation point detection function is applied to the sensor data to identify the valid gesture region in each frame; wherein, the activation point detection function is: In the formula, For position exist The pressure value at any moment, The stress activation threshold, ; Calculate the temporal activation map and the gesture bounding box boundary, wherein the temporal activation map is: The gesture bounding box includes , , and ,in: ； ； ； ； The determined gesture bounding box area is: ； ； ； In the formula, BBox represents the bounding box area, and height and width represent the length and width of the bounding box. Within the sensor space constraints, the effective translation area of ​​the current bounding box: , In the formula, H=16, W=16, which are the array size of the sensor; Data translation is performed based on the bounding box. The data translation operation function is as follows: ； in, and The translation amount, , .

8. The tactile gesture recognition method based on a pressure sensor as described in claim 1, characterized in that, The CNN-LSTM-Attention hybrid neural network adopts an end-to-end learning architecture, including: Input layer: Receives a four-dimensional pressure data tensor of shape (batch_size, n, height, width), supports variable-length sequence input, where n is the time series length, and height and width are the length and width of the pressure sensor array, which are 16×16. Convolutional feature extraction module: includes a two-layer convolutional neural network. The first layer uses 32 3×3 convolutional kernels, padded with 1s to maintain the spatial size, and adopts the ReLU activation function. The second layer uses 64 3×3 convolutional kernels to expand the 32-channel feature map to 64 channels. Combined with max pooling and dropout layers, it outputs a 1024-dimensional spatial feature vector as the input for temporal modeling. Temporal modeling module: includes a two-layer unidirectional LSTM network with an input dimension of 1024, 32 hidden units, an inter-layer dropout rate of 0.5, supports variable-length sequence processing, and outputs a temporal feature sequence with shape (batch_size, n, 32). Sequence aggregation module: It uses an attention mechanism to calculate the attention score through a 32→1 linear transformation, and combines the sequence length mask and Softmax normalization to output a 32-dimensional global feature representation; Classification Decision Module: A single-layer fully connected network is used for gesture classification. The fully connected layer is mapped to the number of gesture categories, and the Softmax activation function is used to output the probability distribution of gesture types.

9. The tactile gesture recognition method based on a pressure sensor as described in claim 8, characterized in that, The training optimization strategy for the CNN-LSTM-Attention hybrid neural network includes: The AdamW adaptive optimizer is used with an initial learning rate of 1e-3, combined with a weight decay mechanism. The loss function used is label-smoothed cross-entropy loss, with a smoothing parameter of 0.05; The training batch size is 32, the number of epochs is 50, and the data is split into 80% training set, 10% validation set, and 10% test set. Regularization techniques include: L2 weight decay with a weight decay coefficient of 1e-4; multi-layer dropout strategy with a dropout rate of 0.2 for convolutional layers, a dropout rate of 0.5 between LSTM layers, and dropout regularization before the classification layer; batch normalization; and variable-length sequence masking.

10. The tactile gesture recognition method based on a pressure sensor as described in claim 1, characterized in that, The pressure sensor-based tactile gesture recognition method supports the recognition of one or more combinations of gesture categories, including swipe up, swipe down, swipe left, swipe right, two-finger operation, clockwise, counterclockwise, hook, cross, single click, double click, and long press.

Images