Gesture recognition method and system based on multi-sensor fusion and adaptive learning

Abstract

This application proposes a gesture recognition method and system based on multi-sensor fusion and adaptive learning, belonging to the field of wearable smart devices and pattern recognition. Based on the irregular curved surface shape and biomechanical characteristics of the human hand, this application employs a flexible pressure sensor array that conforms to the skin of the joints on the back of the hand to capture finger joint movement information. Based on a data-driven K-Means adaptive clustering grouping strategy, it enables multimodal heterogeneous sensor data to automatically form optimal feature subgroups without pre-setting grouping rules, thereby directly learning efficient feature representations from the data to achieve high-precision, real-time gesture recognition. This application includes a flexible pressure sensor fabrication module, a data acquisition module, a signal conditioning module, a main control communication module, a data preprocessing module, a K-Means clustering grouping module, a grouped independent feature extraction module, a dynamic feature fusion module, and a real-time recognition module.

Description

Technical Field

[0001] This application relates to the field of wearable smart devices and pattern recognition, and specifically proposes a gesture recognition method and system based on multi-sensor fusion and adaptive learning. Background Technology

[0002] Gesture recognition is a research hotspot in the field of human-computer interaction, possessing advantages such as being natural, intuitive, and requiring no additional input devices. It has broad application prospects in virtual reality, remote control, assisted communication, and smart homes. Existing gesture recognition technologies are mainly divided into three types: vision-based recognition, electromyography (EMG) sensor-based recognition, and data glove-based recognition.

[0003] Specifically, vision-based recognition technology mainly relies on optical sensors (such as cameras and depth cameras) to capture gesture images. After feature extraction and fusion, classification algorithms are used for gesture recognition. The disadvantages of this technology are that it is highly dependent on the clarity of optical images and ambient lighting conditions. Its performance drops sharply in scenarios with complex backgrounds, occlusion, and low light. In addition, it consumes a lot of computing resources and has a high risk of privacy leakage.

[0004] Recognition technology based on electromyography (EMG) sensors primarily detects surface electromyographic signals generated by muscle activity using EMG sensors, and then extracts time-domain or frequency-domain features for identification. The drawbacks of this technology are that the sensor needs to be in close contact with the skin, and the electrode placement is strictly controlled, making it inconvenient to wear. Furthermore, the EMG signals are easily interfered with by factors such as sweat and changes in skin impedance, resulting in poor comfort during long-term wear.

[0005] Glove-based recognition technology is an important research direction in wearable computing. It is not dependent on a limited environment, relying primarily on sensors integrated into the glove to capture hand movements. Features are extracted through methods such as pose calculation, and pattern classification is performed using machine learning or deep learning algorithms. For example, the previously published patent application, CN120872160B, discloses a gesture recognition method based on a smart glove. This method employs three types of sensors: flexible resistive sensors, inertial measurement sensors, and stretch sensors. The collected data is pre-classified into three categories according to sensor type: tactile data, inertial data, and stretch data. Feature extraction is performed using CNN-LSTM for each category, and then the data is fused for recognition through an attention mechanism. However, the technical solution has the following shortcomings: (1) The sensor type is fixed to three types, which increases the system complexity and manufacturing cost; (2) The data grouping method is pre-set and does not take into account the actual correlation between the sensor channels under different gestures. This may lead to channels that are physically close but belong to different sensor types being forcibly separated, or channels that are physically far apart but have similar motion patterns being unable to be aggregated, and feature extraction lacks specificity; (3) The use of attention mechanism for feature fusion has high computational complexity and is not conducive to the real-time deployment of edge devices.

[0006] For example, the previously published patent application CN121468612A discloses a multifunctional intelligent sensing system suitable for humanoid robots. It uses a flexible sensing sleeve to wrap the outer surface of the robot's fingers, arranging pressure sensing units, strain sensing units, and temperature sensing units for monitoring the interaction between the robot's fingers and the environment / human body, and for safety warnings. However, this technical solution is geared towards the rigid or quasi-rigid outer surface of robots, and its sensing requirements focus on contact force threshold monitoring and temperature anomaly detection, rather than the capture and recognition of fine human hand gestures. Human hand skin has biological characteristics such as temperature regulation, sweat secretion, and soft tissue deformation, which are fundamentally different from the robot's outer surface in terms of shape, material, and sensing requirements. Therefore, this solution cannot be directly applied to human hand gesture recognition scenarios.

[0007] While existing deep learning methods have achieved some success in gesture recognition, most of these methods employ fixed network structures, making it difficult to adaptively adjust to different sensor configurations and data distributions. For multimodal heterogeneous sensor data, existing techniques typically use pre-fixed grouping by sensor type, failing to fully exploit the inherent structural features of the data, thus limiting the specificity of feature extraction and the model's generalization ability. Summary of the Invention

[0008] The purpose of this invention is to address the problems existing in the prior art and provide a gesture recognition method and system based on multi-sensor fusion and adaptive learning. Addressing the irregular curved surface shape and biomechanical characteristics of the human hand, a highly sensitive flexible pressure sensor array conforming to the skin of the joints on the back of the hand is used to capture fine movement information of the finger joints. A data-driven K-Means adaptive clustering grouping strategy replaces the traditional fixed grouping method based on sensor type, enabling multimodal heterogeneous sensor data to automatically form optimal feature subgroups without the need for preset grouping rules. This allows for the direct learning of efficient feature representations from the data, achieving high-precision, real-time gesture recognition.

[0009] To achieve the above design objectives, this application proposes a gesture recognition system based on multi-sensor fusion and adaptive learning, which includes: A flexible pressure sensor fabrication module for fabricating MWCNT / PDMS composite flexible pressure sensors; The data acquisition module is used to acquire multimodal sensor data of the wearer's hand gestures; The signal conditioning module is used to amplify and filter the sensor output signal; The main control communication module is used for analog-to-digital conversion, data packaging, and transmission. The data preprocessing module is used to perform Min-Max normalization, fixed-length window slicing, and zero-padding on the raw multidimensional time series data; The K-Means clustering module is used to adaptively divide multiple flexible pressure sensor channels into K feature subgroups based on the statistical characteristics of the time-series features of each sensor channel, and to fix the inertial measurement unit channels into independent feature subgroups. The grouped independent feature extraction module is used to extract Conv1D-LSTM features from each feature subgroup, and the parameters of each branch are trained independently. The dynamic feature fusion module is used to perform global average pooling dimension alignment, channel concatenation, and fully connected classification on the outputs of each branch; The real-time recognition module is used to load pre-trained models, employ a sliding window mechanism to infer real-time data, and output recognition results and confidence scores.

[0010] Based on the aforementioned gesture recognition system based on multi-sensor fusion and adaptive learning, this application proposes a gesture recognition system method based on multi-sensor fusion and adaptive learning, which includes the following steps: Step (1): Construct a multimodal sensor data acquisition system for gesture capture; A flexible MWCNT / PDMS composite pressure sensor was fabricated and ergonomically integrated with an inertial measurement unit in a wearable device. On the outer surface of the back of the glove, a multimodal sensor data acquisition system for capturing human hand gestures is constructed, and signal conditioning is completed through two-stage series filtering and adaptive gain amplification; Step (2): Construct an adaptive deep learning network; An adaptive mapping from multimodal heterogeneous sensor data to gesture category labels is constructed, and a multi-branch Conv1D-LSTM neural network driven by K-Means clustering is trained. First, the K-Means algorithm is used to adaptively divide the multiple flexible pressure sensor channels into K feature subgroups based on the statistical characteristics of the temporal features of each sensor channel. The inertial measurement unit channel is fixedly divided into an independent feature subgroup. Then, an independent Conv1D-LSTM feature extraction branch is configured for each subgroup. The convolution kernel parameters, the number of LSTM hidden units, and the dropout rate of each branch are trained independently to learn the exclusive temporal feature representation of each subgroup. Finally, the dynamic feature fusion module performs dimension alignment and channel concatenation on the output of each branch, and outputs the gesture recognition result through a fully connected layer and a Softmax classifier. Step (3), real-time gesture recognition; The trained model is deployed in the host computer program, and the gesture action is recognized online by combining the sliding window real-time inference mechanism. The optimal gesture recognition model is used to process the real-time collected multimodal sensor data, output the recognition results and confidence scores, and display them on the host computer interface.

[0011] Step (1) includes the following steps: Step (1.1): Fabrication of the MWCNT / PDMS composite flexible pressure sensor; Step (1.2): Attach and fix the 9 flexible pressure sensors to the designated positions on the outer surface of the glove, and place the 1 inertial measurement unit at the center of the back of the glove; Step (1.3): Amplify and filter the signals output by the flexible pressure sensor and the inertial measurement unit; The amplification module is based on the LM358 operational amplifier chip and uses a non-inverting amplifier configuration. The gain is adjustable in the range of 1-20 times through surface-mount trimmer resistors. Gain adjustment is achieved through surface-mount trimmer resistors. The LM358 non-inverting amplifier circuit adopts a single operational amplifier structure. The input signal is connected to the non-inverting input terminal, and the feedback network is composed of gain resistors. The filtering adopts a two-stage series filter. The first stage is a three-point comparator filter, and the second stage is a Kalman filter. The process noise covariance matrix of the Kalman filter is Q=diag(0.01), and the observation noise covariance is R=0.1. Step (1.4): The conditioned signal is converted from analog to digital and packaged by the main control unit, and then transmitted to the host computer through the communication module.

[0012] Step (1.1) includes the following preparation process: Step (1.1.1): Weigh 15g of PDMS solution; The solution was obtained by mixing PDMS prepolymer and curing agent at a mass ratio of 10:1 and then degassing under vacuum. Then, 0.25g of MWCNT powder and 4.75g of the above PDMS solution were weighed and 10mL of n-hexane was added as a dispersant. Step (1.1.2): Place the mixture in an ultrasonic cleaner and ultrasonically treat it for 30 minutes to make MWCNT uniformly dispersed in the solvent and form a stable conductive filler suspension; Step (1.1.3): Transfer the suspension to a magnetic stirrer and mechanically stir for 30 minutes to promote dispersion and accelerate the volatilization of n-hexane until the mixture is a uniform black paste. Step (1.1.4): Pour the paste mixture into a polytetrafluoroethylene mold, place it in a constant temperature oven, and cure it at 70°C for 6 hours; Step (1.1.5): After demolding, a flexible pressure sensing unit is obtained; conductive silver paste is used to fix wires as electrodes on both sides of the sensing unit. After the conductive silver paste has fully cured, silicone sealant is applied to the surface to prevent the wires from falling off. Conductive silver paste is used to fix wires as electrodes on both sides of the sensing unit. After the conductive silver paste has fully cured, silicone sealant is applied to the surface to prevent the wires from falling off during finger bending. Step (1.1.6): After the sealant has fully cured, the sensing unit is encapsulated in a microbag made of self-adhesive bandage, and then sewn onto the outer surface of the glove (back of the hand) at the corresponding position of the finger joint.

[0013] In step (1.1.2), before ultrasonic treatment, the MWCNT powder can be pre-dried in a vacuum oven at 70 to 100°C for 1-2 hours to remove adsorbed moisture.

[0014] In step (1.2), the specific arrangement of the 9 flexible pressure sensors is as follows: 1 channel of the little finger metacarpophalangeal joint MCP, 1 channel of the little finger proximal interphalangeal joint PIP, 1 channel of the ring finger metacarpophalangeal joint MCP, 1 channel of the ring finger proximal interphalangeal joint PIP, 1 channel of the middle finger metacarpophalangeal joint MCP, 1 channel of the middle finger proximal interphalangeal joint PIP, 1 channel of the index finger metacarpophalangeal joint MCP, 1 channel of the index finger proximal interphalangeal joint PIP, and 1 channel between the thumb metacarpophalangeal joint and the proximal interphalangeal joint.

[0015] In step (1.3), the three-point comparison filter works by taking three consecutive sampling points x(t-1), x(t), and x(t+1). If |x(t)-x(t-1)|>δ and |x(t)-x(t+1)|>δ, where δ is a preset threshold, then x(t) is determined to be spike interference and replaced with [x(t-1)+x(t+1)] / 2; otherwise, x(t) is retained.

[0016] Step (2) includes the following steps: Step (2.1): Establish a data preprocessing module; The data preprocessing module is used to normalize, slice, and convert the format of the original 15-dimensional time series data to generate neural network input tensors; The 15-dimensional time-series data consists of outputs from 9 flexible pressure sensors and 6 inertial measurement units, denoted as X(t)=[p1(t),p2(t),...,p9(t),a] x (t),a γ (t),a u (t),g x (t),g γ (t),g u [(t)], where p i (t) represents the output of the i-th pressure sensor, ax (t), a γ (t), a u (t) represents the output of the triaxial accelerometer, g x (t), g γ (t), g u (t) represents the output of the three-axis gyroscope; During the data collection phase, 100 sets of gesture data of different durations were collected for each English letter; The normalization method used is Min-Max normalization, which normalizes each channel of data independently, as shown in the following formula: x' i = (x i - x i ,min) / (x i ,max - x i ,min) Where, x i For the raw data of the i-th channel, x i min and x i ,max are the minimum and maximum values ​​of the channel in the training set, respectively. After normalization, the data of each channel is mapped to the interval [0,1]. After encoding, statistical analysis was performed on the data samples to verify the uniformity of the label distribution. The encoded data was divided into training and test sets in an 8:2 ratio, with 20% of the data allocated to the test set. Step (2.2): Establish the K-Means clustering grouping module; The statistical characteristics of each channel are calculated from the raw time-series data of all multi-channel flexible pressure sensors in the training set, and a feature description matrix is ​​constructed. The inertial measurement unit channels do not participate in clustering and are directly and fixedly divided into independent feature subgroups. The K-Means++ initialization strategy is used for clustering. Step (2.3): Establish a grouped independent feature extraction module; The grouping method is data-driven. It automatically learns the feature correlation between channels through K-Means clustering, and groups channels with similar physical locations or similar motion patterns into the same subgroup. The channel composition of each subgroup is not fixed and is adaptively adjusted as the training data changes. The grouped independent feature extraction module consists of K Conv1D-LSTM branches, with each feature subgroup independently inputting into one branch; Each branch structure is dynamically configured based on the number of channels in the input subgroup: For subgroups with ≥3 channels, configure Conv1D layer (3 kernels) → BatchNorm layer → ReLU activation → MaxPool1D layer → LSTM layer (32 hidden units) → Dropout layer; For a subgroup with 2 channels, configure the Conv1D layer (kernel size 2) → BatchNorm layer → ReLU activation → MaxPool1D layer → LSTM layer (32 hidden units) → Dropout layer. For subgroups with 1 channel, no Conv1D layer is configured; instead, an LSTM layer (hidden unit 32) → Dropout layer is used directly. Each branch is configured with an independent fully connected layer Dense(32) after the LSTM layer to compress the temporal features of each branch; The convolution kernel parameters, LSTM hidden unit number and fully connected layer parameters of each branch are trained independently, and the dropout rate Pk is set independently according to the characteristics of each subgroup. Step (2.4): Establish a dynamic feature fusion module; The dynamic feature fusion module is used to perform dimension alignment and concatenation on the feature vectors output by each branch; It adopts a dimension alignment and channel splicing strategy based on global average pooling, which has low computational complexity and fast inference speed, making it suitable for real-time gesture recognition scenarios; Dimension alignment uses global average pooling to map the outputs of each branch to a unified dimension D; the formula for global average pooling is: H' k = (1 / T')·Σ t H k (t); where T' is the length of the LSTM output sequence, H k (t) represents the LSTM hidden state of the k-th branch at time t, H' k The feature vector after pooling; Concatenation involves concatenating K feature vectors of dimension D along the channel dimension to form a fused feature vector F, which has a dimension of K×D. The fused feature vector F passes through two fully connected layers. The first fully connected layer has 128 units and uses ReLU activation function; the second fully connected layer has 64 units and uses ReLU activation function. Finally, it passes through the Softmax activation function to output the probability distribution of each category. Step (2.5): Use the cross-entropy loss function and the L2 regularization term to establish the backpropagation loss function; Specifically, the backpropagation loss function L is established using the following equation (1): L = Lce + λ·Lreg (1) Where Lce is the cross-entropy loss, and is obtained from the following equation (2): (2) Where y is the one-hot encoding of the actual label. Let λ be the probability distribution of the model output, λ be the regularization coefficient, and Lreg be the L2 regularization term. The formula for calculating the L2 regularization term is: Lreg = Σ k ||W k || 2 _F; where, ||·|| 2 _F represents the Frobenius norm, W k For all trainable weight parameters in the k-th branch; The regularization coefficient λ is 0.001. The purpose of the regularization coefficient λ is to prevent the model from overfitting. By adding a penalty term to the weight parameters in the loss function, the model is encouraged to learn a more concise feature representation. Step (2.6): Train the neural network and calculate the loss function L; Training stops when the number of training iterations reaches the set number or the loss error is less than the set threshold, thus obtaining the optimal gesture recognition model.

[0017] Step (2) includes the following execution flow: Step (2.2.1): Perform statistical analysis on the time-series data of each channel of the multi-channel flexible pressure sensor in the training set, and extract the data range r of each channel. i (The difference between the maximum and minimum values ​​of the time series data in this channel) and standard deviation σ i Construct an N×2 statistical feature matrix N is the number of channels of the multi-channel flexible pressure sensor; the inertial measurement unit channels do not participate in clustering and are directly and fixedly divided into independent feature subgroups; Step (2.2.2): Based on the N×2 statistical feature matrix constructed in step (2.2.1), calculate the Euclidean distance between the statistical features of each channel of the multi-channel flexible pressure sensor, and construct the distance matrix. D ij =||s i -s j ||2,s i =[r i ,σ i [ ] represents the statistical feature vector of the i-th channel; Step (2.2.3): Use the distance matrix D as the input for K-Means clustering to cluster the statistical feature vectors of each channel of the multi-channel flexible pressure sensor; the inertial measurement unit channels do not participate in clustering and are fixedly divided into independent feature subgroups. Step (2.2.4): The K-Means++ initialization strategy is used to select initial cluster centers. First, a sample is randomly selected as the first cluster center. Then, for each sample x, the shortest distance D(x) between it and the selected cluster centers is calculated with probability D(x).2 / ΣD(x') 2 Select the next cluster center; repeat until K initial centers are selected; Step (2.2.5): Execute the standard K-Means iterative algorithm until the change in cluster centers is less than the preset threshold 1e-4 or the maximum number of iterations of 300 is reached; Step (2.2.6): Perform the above steps for all integer values ​​of K in the range of 2 to 8, calculate the silhouette coefficient corresponding to each K value, and select the K value that maximizes the silhouette coefficient as the optimal number of clusters.

[0018] In step (2.4), the parameters of each branch of Conv1D-LSTM are initialized independently. The initialization method is He initialization, that is, the weight parameters W are randomly sampled from a normal distribution with a mean of 0 and a variance of 2 / n_in, where n_in is the input dimension. The convolution kernel weights, LSTM gate weights and bias parameters of each branch are updated independently during training and do not share parameters. The strategy of independent parameter training enables each branch to learn the exclusive feature representation of its own subgroup.

[0019] In summary, this application has the following advantages and beneficial effects compared to the prior art: 1. This application addresses the irregular curved surface shape and biomechanical characteristics of the human hand by employing a MWCNT / PDMS composite flexible pressure sensor attached to the outer surface of the glove (back of the hand). This sensor can detect hand movement deformation through the glove fabric, thereby capturing the subtle bending changes of the finger joints when performing the 26 English letters gesture with high sensitivity.

[0020] 2. Based on the K-Means clustering grouping strategy and the statistical characteristics of the time-series features of each sensor channel, this application can adaptively divide multiple flexible pressure sensor channels into feature subgroups and fix the inertial measurement unit channels into independent feature subgroups. The data-driven grouping results reflect the inherent correlation structure of the data rather than artificially preset sensor category boundaries, making feature extraction more targeted, training more efficient, and convergence faster.

[0021] 3. This application adopts a group-independent Conv1D-LSTM branch, with each branch parameter initialized and trained independently, learning the exclusive feature representation of its own subgroup. This can effectively avoid feature interference between different subgroups and the loss of feature information in fixed fusion strategies.

[0022] 4. This application is based on a dynamic feature fusion module and a global average pooling-based dimension alignment and channel splicing strategy, which has lower computational complexity, faster inference speed, and is more suitable for real-time gesture recognition scenarios in resource-constrained wearable devices.

[0023] 5. This application uses a dual filtering mechanism (three-point comparison filter + Kalman filter) in each module, which can remove occasional interference and suppress high-frequency noise, while keeping the total delay within 40ms to meet the requirements of real-time interaction.

[0024] 6. This application adopts an adaptive data augmentation strategy based on confusion matrix analysis for easily confused gesture categories. Gaussian noise amplification is performed only on the category pair with the highest misclassification frequency, rather than amplifying all categories indiscriminately. This improves the ability to distinguish easily confused categories while reducing the disruption to the sample distribution of easily distinguishable categories.

[0025] 7. This application selects Conv1D-LSTM as the core recognition model, which can achieve a higher recognition accuracy (98.64%) on 26 types of gesture recognition tasks. At the same time, Conv1D-LSTM can capture local temporal features and long-term temporal dependencies simultaneously. Its dynamic gating mechanism and linear computational complexity make it both robust and lightweight in dynamic sequences and small sample scenarios. Attached Figure Description

[0026] The present application will now be further described in conjunction with the following figures; Figure 1 This is a schematic diagram of the fabrication and packaging process of a flexible pressure sensor; Figure 2 is a schematic diagram of monitoring using a flexible sensor; wherein, Figure 2(a) is a schematic diagram of the performance test platform; Figure 2(b) is a schematic diagram of the sensor's bending and recovery process at different angles of the finger; Figure 2(c) is a cyclic response curve at different angles; Figure 2(d) is a graph of the response time and recovery time of the flexible sensor under 500 Pa pressure; Figure 2(e) is a graph of the test results of the flexible sensor undergoing 2000 bending-release cycles within the bending range of 0° to 90°; Figure 3 This is a circuit diagram of a gesture recognition system; Figure 4 This is a schematic diagram of a signal acquisition device installed in a smart glove; where, Figure 4 (a) is a schematic diagram of the enlarged module; Figure 4 (b) is a schematic diagram of the data processing module; Figure 4 (c) is a schematic diagram of the amplification module and data processing module assembled in a smart glove; Figure 5 This is a schematic diagram of a gesture recognition system based on multi-sensor fusion and adaptive learning; Detailed Implementation

[0027] To further illustrate the technical means employed in this application to achieve the intended design purpose, the following preferred embodiments are presented in conjunction with the accompanying drawings. Specific details are set forth in the following description to provide a full understanding of the invention. However, the invention can be implemented in many ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0028] like Figures 1 to 5 As shown, this application proposes a gesture recognition system based on multi-sensor fusion and adaptive learning, which includes: A flexible pressure sensor fabrication module for fabricating MWCNT / PDMS composite flexible pressure sensors; The data acquisition module is used to acquire multimodal sensing data of the wearer's hand gestures; specifically, it may include 9 flexible pressure sensors and 1 inertial measurement unit (6 channels in total), totaling 15-dimensional sensing data. The signal conditioning module is used to amplify and filter the sensor output signal; specifically, it may include an LM358 non-inverting amplifier circuit, a three-point comparator filter, and a Kalman filter. The main control communication module is used for analog-to-digital conversion, data packaging and transmission; specifically, it can use an ESP32-S3 chip and Bluetooth wireless communication. The data preprocessing module is used to perform Min-Max normalization, fixed-length window slicing, and zero-padding on the raw 15-dimensional time series data; The K-Means clustering module is used to adaptively divide multiple flexible pressure sensor channels (such as 9-dimensional channels) into K feature subgroups based on the statistical characteristics of the time-series features of each sensor channel, and to fix the inertial measurement unit channels into independent feature subgroups. The grouped independent feature extraction module is used to extract Conv1D-LSTM features from each feature subgroup, and the parameters of each branch are trained independently. The dynamic feature fusion module is used to perform global average pooling dimension alignment, channel concatenation, and fully connected classification on the outputs of each branch; The real-time recognition module is used to load pre-trained models, employ a sliding window mechanism to infer real-time data, and output recognition results and confidence scores.

[0029] To address the aforementioned gesture recognition system based on multi-sensor fusion and adaptive deep learning, this application proposes the following gesture recognition method based on multi-sensor fusion and adaptive deep learning, which includes the following steps: Step (1): Construct a multimodal sensor data acquisition system for gesture capture; A flexible MWCNT / PDMS composite pressure sensor was fabricated and ergonomically integrated with an inertial measurement unit in a wearable device. On the outer surface of the back of the glove, a multimodal sensor data acquisition system for capturing human hand gestures is constructed, and signal conditioning is completed through two-stage series filtering and adaptive gain amplification; Step (1.1): Fabrication of the MWCNT / PDMS composite flexible pressure sensor; The MWCNTs mentioned are multi-walled carbon nanotubes with a diameter of 10-20 nm, a length of 10-30 μm, and a purity greater than 95%. The PDMS mentioned is polydimethylsiloxane, model number Dow Corning Sylgard 184; In the MWCNT / PDMS composite material, the mass fraction of MWCNT is 5%, which balances the conductivity and flexibility of the composite material. The preparation process is as follows: Step (1.1.1): Weigh 15g of PDMS solution; The solution was obtained by mixing PDMS prepolymer and curing agent at a mass ratio of 10:1 and then degassing under vacuum. Then, 0.25g of MWCNT powder and 4.75g of the above PDMS solution were weighed and 10mL of n-hexane was added as a dispersant. Step (1.1.2): Place the mixture in an ultrasonic cleaner and ultrasonically treat it for 30 minutes to make MWCNT uniformly dispersed in the solvent and form a stable conductive filler suspension; Furthermore, before ultrasonic treatment, MWCNT powder can be pre-dried in a vacuum oven at 80°C for 2 hours to remove adsorbed moisture; Step (1.1.3): Transfer the suspension to a magnetic stirrer and mechanically stir for 30 minutes to promote dispersion and accelerate the volatilization of n-hexane until the mixture is a uniform black paste. Step (1.1.4): Pour the paste mixture into a polytetrafluoroethylene mold, place it in a constant temperature oven, and cure it at 70°C for 6 hours; Step (1.1.5): After demolding, a flexible pressure sensing unit is obtained; conductive silver paste is used to fix wires as electrodes on both sides of the sensing unit. After the conductive silver paste has fully cured, silicone sealant is applied to the surface to prevent the wires from falling off. The silicone sealant uses addition-curing silicone rubber compatible with the PDMS matrix, and the curing conditions are room temperature curing for 24 hours; Conductive silver paste is used to fix wires as electrodes on both sides of the sensing unit. After the conductive silver paste has fully cured, silicone sealant is applied to the surface to prevent the wires from falling off during finger bending. Step (1.1.6): After the sealant has completely cured, the sensing unit is encapsulated in a microbag made of self-adhesive bandage, and then sewn onto the outer surface of the glove (back of the hand) at the corresponding position of the finger joint. In the aforementioned MWCNT / PDMS composite material, the MWCNT mass fraction is 5wt%, and under this ratio, the composite material has a sensitivity of 2.25 × 10⁻⁶ kPa in the pressure range of 0-50 kPa. -2 kPa -1 The rate of change in resistance is 10% within the bending angle range of 0°-90°. The performance parameters of the flexible pressure sensor were tested using a flexible electronic tester (THE FT2000, Shanghai Mifang Technology Co., Ltd.): the response time under 500Pa pressure was 501ms, and the recovery time was 342ms; it could identify different bending angles such as 20°, 40°, 60°, and 80° within the bending angle range of 0°-90°, and the resistance increased with the increase of the bending angle; no significant performance degradation was observed after 1000 bending-release cycles, demonstrating fast response and excellent stability. Step (1.2): Attach and fix the 9 flexible pressure sensors to the designated positions on the outer surface of the glove (back of the hand), and place the 1 inertial measurement unit at the center of the back of the glove; The gloves described may be five-finger knitted gloves made of spandex fiber, which have good elasticity and breathability; The specific arrangement of the nine flexible pressure sensors is as follows: one at the metacarpophalangeal joint (MCP) of the little finger, one at the proximal interphalangeal joint (PIP) of the little finger, one at the metacarpophalangeal joint (MCP) of the ring finger, one at the proximal interphalangeal joint (PIP) of the ring finger, one at the metacarpophalangeal joint (MCP) of the middle finger, one at the proximal interphalangeal joint (PIP) of the middle finger, one at the metacarpophalangeal joint (MCP) of the index finger, one at the proximal interphalangeal joint (PIP) of the index finger, and one between the metacarpophalangeal joint and the proximal interphalangeal joint of the thumb. The nine flexible pressure sensors are fixed to the outer surface of the glove (back of the hand) at designated locations by sewing.

[0030] The specific arrangement of the above-mentioned nine flexible pressure sensors is determined based on the anatomical structure of the human hand. The little finger metacarpophalangeal joint (MCP) sensor is located on the dorsal side of the junction between the base of the little finger and the palm. The little finger proximal interphalangeal joint (PIP) sensor is located in the middle of the dorsal side of the second phalanx of the little finger. The ring finger metacarpophalangeal joint (MCP) sensor is located on the dorsal side of the junction between the base of the ring finger and the palm. The ring finger proximal interphalangeal joint (PIP) sensor is located in the middle of the dorsal side of the second phalanx of the ring finger. The middle finger metacarpophalangeal joint (MCP) sensor is located on the dorsal side of the junction between the base of the middle finger and the palm. The middle finger proximal interphalangeal joint (PIP) sensor is located in the middle of the dorsal side of the second phalanx of the middle finger. The index finger metacarpophalangeal joint (MCP) sensor is located on the dorsal side of the junction between the base of the index finger and the palm. The index finger proximal interphalangeal joint (PIP) sensor is located in the middle of the dorsal side of the second phalanx of the index finger. The thumb sensor is located in the middle of the dorsal side between the thumb metacarpophalangeal joint and the proximal interphalangeal joint. The above arrangement is designed based on the following: When a person performs the gestures of the 26 English letters, the change in the bending angle of the finger joints is the main source of information to distinguish different gestures. Two sensors are arranged on each of the little finger, ring finger, middle finger, and index finger to capture the two-level bending information of the metacarpophalangeal joint and the proximal interphalangeal joint. The thumb has one sensor arranged because of its palmar movement characteristics to capture its relative positional relationship with other fingers. The above sensors are attached to the outer surface of the back of the glove and sense the motion deformation information of the skin on the back of the hand through the glove fabric. The inertial measurement unit can be a JY901S module, which is mounted on the top of the ESP32-S3 chip via pin headers and fixed to the center of the back of the glove. The JY901S module is a nine-axis attitude sensor, which includes a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer. It uses the data from its three-axis accelerometer and three-axis gyroscope, a total of 6 channels, to capture the overall spatial motion attitude of the palm. The JY901S inertial measurement unit has a built-in voltage regulation circuit, with an operating voltage range of 3.3V-5V, and its pin levels are compatible with 3.3V and 5V embedded systems. The module integrates a high-precision gyroscope, accelerometer, and geomagnetic sensor, and is equipped with a high-performance microprocessor. Utilizing dynamic calculation algorithms and Kalman dynamic filtering technology, it can quickly calculate the module's real-time motion attitude. The amplification module and the data processing module are interconnected via a flexible printed circuit (FPC) cable; The flexible pressure sensor is attached to the outer surface of the glove (back of the hand), and indirectly contacts the wearer's skin on the back of the hand through the glove fabric. Since the skin on the back of the hand transmits motion deformation information through the fabric, the sensor is required to have high flexibility to adapt to the large bending of the finger joints, and at the same time be able to sense joint movement with high sensitivity through the glove fabric. Step (1.3): Amplify and filter the signals output by the flexible pressure sensor and the inertial measurement unit; The amplification module is based on the LM358 operational amplifier chip and uses a non-inverting amplifier configuration. The gain is adjustable in the range of 1-20 times through surface-mount trimmer resistors. Gain adjustment is achieved through surface-mount trimmer resistors. The LM358 non-inverting amplifier circuit uses a single operational amplifier structure. The input signal is connected to the non-inverting input terminal, and the feedback network consists of gain resistors. The filtering uses a two-stage series filter. The first stage is a three-point comparator filter used to remove occasional spike interference. The operation mode of the three-point comparator filter is as follows: For three consecutive sampling points x(t-1), x(t), and x(t+1), if |x(t)-x(t-1)|>δ and |x(t)-x(t+1)|>δ, where δ is a preset threshold, then x(t) is determined to be spike interference and replaced with [x(t-1)+x(t+1)] / 2; otherwise, x(t) is retained. The second stage is a Kalman filter, used to suppress high-frequency noise and smooth the signal. The state equation and observation equation of the Kalman filter are respectively: X(k) = A·X(k-1) + W(k), Z(k) = H·X(k) + V(k); where X(k) is the system state vector at time k, containing two state variables: signal amplitude and signal rate of change; Z(k) is the observed value at time k; A is the state transition matrix; H is the observation matrix; W(k) and V(k) are the process noise and observation noise, respectively, and their covariance matrices Q and R are determined through experimental calibration. The total delay of the two-stage cascaded filter is no more than 40ms, which meets the requirements of real-time gesture recognition. The threshold δ of the three-point comparison filter was determined through experimental calibration. The calibration method was to collect sensor data for 8 seconds in a static, gesture-free state, calculate the noise standard deviation σ_n, and take δ=3σ_n; in the example, the calibration result δ=2.67. The process noise covariance matrix of the Kalman filter is Q=diag(0.01), and the observation noise covariance is R=0.1. The total delay of the two-stage filtering is 40ms, which is equivalent to 2 sampling periods at a sampling rate of 50Hz. Step (1.4): The conditioned signal is converted from analog to digital and packaged by the main control unit, and then transmitted to the host computer through the communication module; The data processing module is based on the ESP32-S3 chip. In addition to the ESP32-S3 chip, it also integrates a battery charge and discharge management chip, a voltage regulation chip, and a serial communication chip. All components work together to ensure stable system operation. The ESP32-S3 chip has an ADC reference voltage of 3.3V, a resolution of 12 bits, and an input voltage range of 0-3.3V. The sampling frequency is 50Hz, which means that 15 channels of data are collected every 20ms. The Bluetooth wireless communication baud rate is 115200, and the data packet transmission interval is 20ms; Step (2): Construct an adaptive deep learning network; An adaptive mapping from multimodal heterogeneous sensor data to gesture category labels is constructed, and a multi-branch Conv1D-LSTM neural network driven by K-Means clustering is trained. First, the K-Means algorithm is used to adaptively divide the multiple flexible pressure sensor channels into K feature subgroups based on the statistical characteristics of the temporal features of each sensor channel. The inertial measurement unit channel is fixedly divided into an independent feature subgroup. Then, an independent Conv1D-LSTM feature extraction branch is configured for each subgroup. The convolution kernel parameters, the number of LSTM hidden units, and the dropout rate of each branch are trained independently to learn the exclusive temporal feature representation of each subgroup. Finally, the dynamic feature fusion module performs dimension alignment and channel concatenation on the output of each branch, and outputs the gesture recognition result through a fully connected layer and a Softmax classifier. Step (2.1): Establish a data preprocessing module; The data preprocessing module is used to normalize, slice, and convert the format of the original 15-dimensional time series data to generate neural network input tensors. The 15-dimensional time-series data consists of outputs from 9 flexible pressure sensors and 6 inertial measurement units, denoted as X(t)=[p1(t),p2(t),...,p9(t),a] x (t),a γ (t),a u (t),g x (t),g γ (t),g u [(t)], where p i (t) represents the output of the i-th pressure sensor, a x (t), a γ (t), a u (t) represents the output of the triaxial accelerometer, g x (t), g γ (t), g u (t) represents the output of the three-axis gyroscope; During the data collection phase, 100 sets of gesture data of different durations were collected for each English letter; The normalization method used is Min-Max normalization, which normalizes each channel of data independently, as shown in the following formula: x' i = (x i - x i ,min) / (x i ,max - x i ,min) Where, x i For the raw data of the i-th channel, x i min and x i ,max are the minimum and maximum values ​​of the channel in the training set, respectively. After normalization, the data of each channel is mapped to the interval [0,1]. The slicing method divides continuous time-series data into fixed-length time windows, denoted by T and S. Specifically, the window length T is 60, and the step size S is 30, meaning that at a 50Hz sampling rate, each window covers a 1.2-second time segment, with 50% overlap between adjacent windows. When a window slides to the end of the data and is less than T in length, it is padded with zeros to reach the length T. Each window generates an input tensor of shape (60, 15), where 60 is the time step and 15 is the number of channels. When a window slides to the end of the data and is less than 60 in length, it is padded with zeros at the end of the time dimension. Labels are converted to vector form using One-Hot encoding, and N-bit state registers are used to represent N different states, each state corresponding to an independent register bit, with only one bit active at any given time. Before normalization, the original time-series data is cleaned by replacing NaN values ​​and infinity values ​​with finite values ​​to avoid abnormal data interfering with subsequent standardization and model training. After encoding, statistical analysis was performed on the data samples to verify the uniformity of the label distribution. The encoded data was divided into training and test sets in an 8:2 ratio, with 20% of the data allocated to the test set. A stratified sampling strategy was used during the division to ensure that the proportion of each gesture category sample was consistent between the training and test sets. Step (2.2): Establish the K-Means clustering grouping module; The K-Means clustering and grouping module is used to adaptively divide the 9 flexible pressure sensor channels into K feature subgroups and to fix the inertial measurement unit channels into independent feature subgroups. For each channel of the raw time-series data of all 9 flexible pressure sensor channels in the training set, the data range r (the difference between the maximum and minimum values ​​of the time-series data of that channel) and standard deviation σ are calculated, and an N×2 statistical feature matrix S=[r,σ] is constructed. Based on this statistical feature matrix, the Euclidean distance between each channel is calculated, and a distance matrix D is constructed. The inertial measurement unit channels do not participate in clustering and are directly fixed into independent feature subgroups. The K-Means++ initialization strategy is used for clustering, with a maximum number of iterations of 300 and a convergence threshold of 1e-4. Clustering is performed on each flexible pressure sensor channel within the range of K values, and the profile coefficient is calculated. The K value that maximizes the profile coefficient is selected as the optimal number of groups. In this embodiment, the optimal number of groups for the 9 flexible pressure sensors is K=3, plus the fixed group of the inertial measurement unit, for a total of 4 feature subgroups. The above division is based on a similarity measure of the temporal characteristics of each flexible pressure sensor channel; specifically, a statistical feature vector s=[r,σ] is constructed based on the data range r and standard deviation σ of each channel, and the Euclidean distance D is used to determine the similarity. ij =√((r i -r j ) 2 +(σ i -σ j ) 2 The difference between channels is measured; the data range reflects the dynamic amplitude of the channel in the gesture, and the standard deviation reflects the signal fluctuation of the channel; channels with similar motion patterns (such as similar bending amplitude and stability) are closer in the statistical feature space and are thus grouped into the same subgroup; the inertial measurement unit describes the overall motion of the palm, which is fundamentally different from the pressure sensor that describes the local bending of the fingers in terms of motion scale, so it is fixed and independently grouped. The K value is automatically determined using the silhouette coefficient method. The silhouette coefficient is calculated as follows: for each sample i, calculate its silhouette coefficient s(i) = [b(i) - a(i)] / max{a(i), b(i)}; where a(i) is the average distance from sample i to other samples in the same cluster, and b(i) is the average distance from sample i to all samples in the nearest heterogeneous cluster; the silhouette coefficients of all samples are averaged to obtain the overall silhouette coefficient S, and the K value that maximizes S is selected as the optimal number of clusters. The K value can also be verified using the elbow rule, which calculates the sum of squares within the cluster for different K values ​​as follows: SSE=Σ i Σ x ∈C i ||x-μ i || 2 Plot the curve of SSE as a function of K, and select the K value at the inflection point of the curve; The optimal number of groups for the 9-channel flexible pressure sensor was determined to be K=3 using the contour coefficient method. Including the fixed group of the inertial measurement unit, there are a total of 4 feature subgroups. The grouping results are determined during the training phase and are used in the inference phase. Specifically, the K=3 grouping results of the 9-channel flexible pressure sensor are as follows: Subgroup 1 includes p4(t), p5(t), and p7(t); Subgroup 2 includes p6(t), p8(t), and p9(t); Subgroup 3 includes p1(t), p2(t), and p3(t); the inertial measurement unit channel is fixedly divided into subgroup 4, which includes a x (t), a γ (t), a u (t), g x (t), g γ (t), g u (t) Six inertial measurement channels; The specific execution flow of the K-Means clustering module in this step is as follows: Step (2.2.1): Perform statistical analysis on the time series data of each channel of the multi-channel flexible pressure sensor in the training set, extract the data range (difference between the maximum and minimum values) and standard deviation of each channel, and construct a feature description matrix; the inertial measurement unit channels do not participate in clustering and are directly fixed into independent feature subgroups; Step (2.2.2), the N×2 statistical feature matrix constructed based on step (2.2.1) Calculate the Euclidean distance between the statistical characteristics of each channel of the multi-channel flexible pressure sensor, and construct a distance matrix. D ij =||s i -s j ||2=√((r i -r j ) 2 +(σ i -σ j ) 2 ), s i =[r i ,σ i [ ] represents the statistical feature vector of the i-th channel; Step (2.2.3): Use the distance matrix D as the input for K-Means clustering to cluster the statistical feature vectors of each channel of the multi-channel flexible pressure sensor; the inertial measurement unit channels do not participate in clustering and are fixedly divided into independent feature subgroups. Step (2.2.4): The K-Means++ initialization strategy is used to select initial cluster centers. First, a sample is randomly selected as the first cluster center. Then, for each sample x, the shortest distance D(x) between it and the selected cluster centers is calculated with probability D(x). 2 / ΣD(x') 2 Select the next cluster center; repeat until K initial centers are selected; Step (2.2.5): Execute the standard K-Means iterative algorithm until the change in cluster centers is less than the preset threshold 1e-4 or the maximum number of iterations of 300 is reached; Step (2.2.6): Perform the above steps for all integer values ​​of K in the range of 2 to 8 for the 9 flexible pressure sensor channels, calculate the profile coefficient corresponding to each K value, and select the K value that maximizes the profile coefficient as the optimal number of clusters; in this embodiment, the value of K is in the range of 2 to 4; Step (2.3): Establish a grouped independent feature extraction module; The grouping method is data-driven. It automatically learns the feature correlation between channels through K-Means clustering, and groups channels with similar physical locations or similar motion patterns into the same subgroup. The channel composition of each subgroup is not fixed and is adaptively adjusted as the training data changes. The grouped independent feature extraction module consists of K Conv1D-LSTM branches, with each feature subgroup independently inputting into one branch; Each branch structure is dynamically configured based on the number of channels in the input subgroup: The parameters of each branch are initialized independently using the He initialization method, where the weight parameters W are randomly sampled from a normal distribution with a mean of 0 and a variance of 2 / n_in, where n_in is the input dimension. The convolutional kernel weights, LSTM gate weights, and bias parameters of each branch are updated independently during training and do not share parameters. The strategy of independent parameter training enables each branch to learn the exclusive feature representation of its own subgroup, avoiding feature interference between different subgroups. The input shape of the Conv1D layer is (60, C). k ), where C k The number of channels in the k-th subgroup is (60, 32); the MaxPool1D layer reduces the temporal dimension from 60 to 30; the output shape of the LSTM layer is (30, 32); the fully connected layer Dense(32) compresses (30, 32) into (32,). The main reasons for choosing Conv1D-LSTM to build the gesture recognition model are: Conv1D-LSTM can simultaneously capture the local temporal features and long-range temporal dependencies of one-dimensional sequence data; and its dynamic gating mechanism and linear computational complexity make it both robust and lightweight in scenarios involving dynamic sequences, small samples or long sequences. Specifically, the k-th Conv1D-LSTM branch includes: the k-th Conv1D layer (the kernel size is dynamically selected according to the number of channels), the k-th batch normalization layer, the k-th ReLU activation layer, the k-th MaxPool1D layer, the k-th fully connected layer Dense(32), and the k-th Dropout layer. The kernel size of the kth Conv1D layer is dynamically selected based on the number of channels in the subgroup: F=3 when the number of channels is ≥3, F=2 when the number of channels is 2, and no Conv1D layer is configured when the number of channels is 1. The number of filters is Nk, the stride is 1, the padding method is same padding, and the activation function is ReLU. The calculation formula for the kth Conv1D layer is: Z k (t) = ReLU(BN(Conv1D(X k (t); W k , b k )));wherein, X k (t) represents the input time series data of the k-th feature subgroup, W k Let b be the convolution kernel weight matrix. k The bias vector is BN(·), which is the batch normalization operation. The formula for batch normalization is: BN(x) = γ·(x-μ_B) / √(σ 2 _B+ε) + β; Wherein, μ_B and σ 2 _B represents the mean and variance of the current batch of data, γ and β are learnable scaling and translation parameters, and ε=1e-5 is a small constant to prevent division by zero; The pooling size of the kth MaxPool1D layer is 2, and the stride is 2, which is used to reduce the temporal dimension. The number of hidden units in the k-th LSTM layer is Hk, and the return sequence is set to True, which means that the complete temporal hidden state sequence is output. The internal operation process of the LSTM unit is as follows: f t = σ(Wf·[h t-1 , x t ] + bf) i t = σ(Wi·[h t-1 , x t ] + bi) C̃ t = tanh(Wc·[h t-1 , x t ] + bc) C t = f t ⊙Ct-1 + i t ⊙C̃ t o t = σ(Wo·[h t-1 , x t ] + bo) h t = o t ⊙tanh(C t ) Among them, f t i t o t The outputs of the forget gate, input gate, and output gate are respectively, C t In cellular state, h t For the hidden state, σ is the sigmoid function, and ⊙ is the Hadamard product; The dropout rate of the k-th Dropout layer is Pk. During the training phase, the neuron output is randomly set to zero with probability Pk. During the inference phase, the outputs of all neurons are multiplied by (1-Pk). In this embodiment, Pk=0.3. Each branch processes data dynamically according to the number of channels in the input subgroup: for subgroups with ≥3 channels, the temporal data passes through the Conv1D layer, batch normalization layer, ReLU activation layer, MaxPool1D layer, LSTM layer, and Dropout layer in sequence, outputting a temporal feature vector H. k For a subgroup with 2 channels, the temporal data is sequentially processed through a Conv1D layer (with a kernel size of 2), a batch normalization layer, a ReLU activation layer, a MaxPool1D layer, an LSTM layer, and a Dropout layer, outputting a temporal feature vector H. k For the subgroup with 1 channel, the time series data is directly input into the LSTM and Dropout layers without passing through the Conv1D layer, and the output is the time series feature vector H. k Thus, the K branches yield the final feature vector set {H1, H2, ..., H} output by the grouped independent feature extraction module. k}; The kernel size is dynamically selected based on the number of channels in the subgroup: For subgroups with ≥3 channels, F=3, at a sampling rate of 50Hz, 3 sampling points cover a time span of 60ms, which can capture local dynamic changes in gestures; for subgroups with 2 channels, F=2, which can capture finer-grained local features; for subgroups with 1 channel, no Conv1D layer is configured, and LSTM is directly used to extract temporal features, avoiding excessive compression of single-channel information by convolution operations. Step (2.4): Establish a dynamic feature fusion module; The dynamic feature fusion module is used to perform dimension alignment and concatenation on the feature vectors output by each branch; It adopts a dimension alignment and channel splicing strategy based on global average pooling, which has low computational complexity and fast inference speed, making it suitable for real-time gesture recognition scenarios; Dimension alignment uses global average pooling to map the outputs of each branch to a unified dimension D; the formula for global average pooling is: H' k = (1 / T')·Σ t H k (t); where T' is the length of the LSTM output sequence, H k (t) represents the LSTM hidden state of the k-th branch at time t, H' k The feature vector after pooling; Concatenation involves concatenating K feature vectors of dimension D along the channel dimension to form a fused feature vector F, which has a dimension of K×D. The fused feature vector F is first compressed by a fully connected layer Dense (32), then transformed nonlinearly by a fully connected layer Dense (128), and overfitting is prevented by a Dropout layer (dropout rate 0.5); finally, the probability distribution of each category is output by the Softmax activation function. The formula for the Softmax function is: ; where z i This represents the logits value of the i-th class in the output of the fully connected layer. Let be the predicted probability of the i-th class; Specifically, global average pooling is used to map the LSTM output (30,32) of each branch to a fixed-dimensional vector of (32,); the four (32,) vectors are concatenated according to the channel dimension to form a fused feature vector of (128,). The fully connected layer structure is: Dense(32, ReLU)→Dense(128, ReLU)→Dropout(0.5)→Dense(26, Softmax); Step (2.5): Use the cross-entropy loss function and the L2 regularization term to establish the backpropagation loss function; Specifically, the backpropagation loss function L is established using the following equation (1): L = Lce + λ·Lreg (1) Where Lce is the cross-entropy loss, and is obtained from the following equation (2): (2) Where y is the one-hot encoding of the actual label. Let λ be the probability distribution of the model output, λ be the regularization coefficient, and Lreg be the L2 regularization term. The formula for calculating the L2 regularization term is: Lreg = Σ k ||Wk || 2 _F; where, ||·|| 2 _F represents the Frobenius norm, W k For all trainable weight parameters in the k-th branch; The regularization coefficient λ is 0.001. The purpose of the regularization coefficient λ is to prevent the model from overfitting. By adding a penalty term to the weight parameters in the loss function, the model is encouraged to learn a more concise feature representation. Step (2.6): Train the neural network and calculate the loss function L; Training stops when the number of training iterations reaches the set number or the loss error is less than the set threshold, thus obtaining the optimal gesture recognition model. Specifically, the Adam optimizer is used, and its parameter update formula is as follows: m t = β1·m t ₋1 + (1-β1)·g t v t = β2·v t ₋1 + (1-β2)·g t 2 m̂ t = m t / (1-β1 t ) v̂ t = v t / (1-β2 t ) θ t = θ t ₋1 - α·m̂ t / (√v̂ t + ε) Among them, g t Let m be the gradient at time t. t and v t First and second moments are estimated, respectively, with β1=0.9 and β2=0.999. α is the learning rate, and ε=1e-8. The initial learning rate is 0.001, halved every 50 iterations (lr=0.001 for iterations 1-50, lr=0.0005 for iterations 51-100). Training parameters are set as follows: batch_size=16, epochs=50, and the loss error threshold is 1×10⁻⁶. -6 ; During training, Gaussian noise is added to easily confused gesture categories for data augmentation. The easily confused gesture categories are determined through confusion matrix analysis in the early stage of training, and in this example, they are determined to be the letters D / M / N / U / V. The specific method for data augmentation is as follows: Calculate the noise intensity σ of each sensor group based on the standard deviation of each group. k Adding a factor equal to 10% of the standard deviation of the group to the training samples of each group, following the N(0, σ) standard deviation. k 2 Adaptive Gaussian noise with a distribution of N(0, σ) is added to the normalized data for training samples of easily confused categories. 2 Gaussian noise with a distribution of σ=0.05 was used, and the sample size was increased by 5 times. The amplification method consists of two stages: In the first stage, for the original sample x belonging to easily confused categories (D / M / N / U / V), five enhanced samples x' = x + ε are generated; where ε follows the order N(0, 0.05). 2 In the second stage, global augmentation is performed on all training samples of the 26 letter gestures, generating 3 augmented samples for each sample, where the adaptive noise intensity σ is... k The standard deviation of the sensor group is 10%; The purpose of the above data augmentation is to simulate potential interference in real-world scenarios, allowing the model to be exposed to more diverse gesture data during training, thereby enhancing the model's ability to generalize to various gestures. The loss function changes during training as follows: loss = 3.078 in the 1st epoch and loss = 0.1964 in the 50th epoch; the accuracy changes on the validation set as follows: accuracy = 0.3291 in the 1st epoch and accuracy = 0.9864 in the 50th epoch; after training, the optimal model weights, sensor grouping information, and grouping normalizer parameters are serialized and saved together for model loading and data preprocessing in the subsequent real-time inference stage. To further verify the model's recognition ability, a comparative experiment was conducted between the Conv1D-LSTM model and recognition models constructed using Random Forest, ResNet, and MobileNet algorithms. The recognition accuracies of the four models were as follows: Random Forest 97.69%, ResNet 97.31%, MobileNet 96.54%, and Conv1D-LSTM 98.64%. To further verify the model's accuracy and generalization ability, a 10-fold cross-validation method was used, with an average validation accuracy of 97.54%, indicating that the Conv1D-LSTM model has excellent generalization ability in gesture recognition tasks. Step (3), real-time gesture recognition; The trained model is deployed in the host computer program, and the gesture action is recognized online by combining the sliding window real-time inference mechanism. The optimal gesture recognition model is used to process the real-time collected multimodal sensor data, output the recognition result and confidence score, and display them on the host computer interface. The host computer interface simultaneously displays the current recognition result, confidence bar chart, real-time waveform graphs of each sensor channel, and historical records of the last 50 recognition results; when the recognition results of 40 consecutive frames are all the same gesture and the confidence is higher than 0.8, the control command corresponding to the gesture is triggered to output. Real-time inference uses a sliding window mechanism with a window length T=60 and a step size S=1. That is, every time a new sampling point is received, the window is slid forward by 1 point to identify the latest 60 sampling points. The confidence level is the maximum probability value output by Softmax. When the confidence level is lower than the preset threshold of 0.8, it is judged as an "unknown gesture," and no recognition result is output. Using this embodiment, test results across multiple sets of data show that the recognition accuracy on the test set reaches 98.64%, the macro-averaged precision is 98.64%, the macro-averaged F1 score is 98.13%, and the macro-averaged recall is 98.12%. After 20 actual wear demonstrations, the recognition accuracy rates for each letter were as follows: A-90%, B-95%, C-95%, D-100%, E-95%, F-90%, G-90%, H-80%, I-100%, J-100%, K-85%, L-100%, M-70%, N-85%, O-90%, P-85%, Q-100%, R-75%, S-70%, T-80%, U-75%, V-65%, W-90%, X-95%, Y-100%, Z-100%, with an overall average accuracy rate of 88.46%. The real-time gesture recognition program can be developed using Python. Its core function is as follows: after clicking the "Start Collection" button on the host computer interface, the program saves the data transmitted from the glove to the computer; then, clicking the "Classify" button imports the collected data into the trained model for classification, and finally displays the recognition results on the interface in real time; the real-time recognition shows that the recognition confidence of gesture "E" is 0.84 and the recognition confidence of gesture "I" is 1.0.

[0031] Based on the implementation of the above steps, the gesture recognition method based on multi-sensor fusion and adaptive deep learning can address the essential differences in sensing requirements between the irregular curved surface of the human hand and the outer surface of the robot. It adopts a high-sensitivity flexible pressure sensor array that fits against the outer surface of the glove and indirectly contacts the skin on the back of the hand through the glove fabric. Through a data-driven adaptive grouping strategy (multiple flexible pressure sensor channels are adaptively clustered based on the data range and standard deviation of each channel, while inertial measurement unit channels are fixed and independently grouped, and the grouping results are adaptively adjusted with the training data rather than being manually preset), it configures a Conv1D-LSTM feature extraction branch with an adaptive number of channels for each subgroup. It also achieves dynamic feature fusion through global average pooling and fully connected layers, thus realizing high-precision, real-time gesture recognition.

[0032] As described above, similar technical solutions can be derived from the solutions presented in conjunction with the accompanying drawings and description. Any solutions that do not depart from the structure of this invention still fall within the scope of the claims of this application.

Claims

1. A gesture recognition system based on multi-sensor fusion and adaptive learning, characterized in that: Including, A flexible pressure sensor fabrication module for fabricating MWCNT / PDMS composite flexible pressure sensors; The data acquisition module is used to acquire multimodal sensor data of the wearer's hand gestures; The signal conditioning module is used to amplify and filter the sensor output signal; The main control communication module is used for analog-to-digital conversion, data packaging, and transmission. The data preprocessing module is used to perform Min-Max normalization, fixed-length window slicing, and zero-padding on the raw multidimensional time series data; The K-Means clustering module is used to adaptively divide multiple flexible pressure sensor channels into K feature subgroups based on the statistical characteristics of the time-series features of each sensor channel, and to fix the inertial measurement unit channels into independent feature subgroups. The grouped independent feature extraction module is used to extract Conv1D-LSTM features from each feature subgroup, and the parameters of each branch are trained independently. The dynamic feature fusion module is used to perform global average pooling dimension alignment, channel concatenation, and fully connected classification on the outputs of each branch. The real-time recognition module is used to load pre-trained models, employ a sliding window mechanism to infer real-time data, and output recognition results and confidence scores.

2. A gesture recognition method based on multi-sensor fusion and adaptive learning, applying the gesture recognition system based on multi-sensor fusion and adaptive learning as described in claim 1, characterized in that: Includes the following steps, Step (1): Construct a multimodal sensor data acquisition system for gesture capture; A flexible MWCNT / PDMS composite pressure sensor was fabricated and ergonomically integrated with an inertial measurement unit in a wearable device. On the outer surface of the back of the glove, a multimodal sensor data acquisition system for capturing human hand gestures is constructed, and signal conditioning is completed through two-stage series filtering and adaptive gain amplification; Step (2): Construct an adaptive deep learning network; An adaptive mapping from multimodal heterogeneous sensor data to gesture category labels is constructed, and a multi-branch Conv1D-LSTM neural network driven by K-Means clustering is trained. First, the K-Means algorithm is used to adaptively divide the multiple flexible pressure sensor channels into K feature subgroups based on the statistical characteristics of the temporal features of each sensor channel. The inertial measurement unit channel is fixedly divided into an independent feature subgroup. Then, an independent Conv1D-LSTM feature extraction branch is configured for each subgroup. The convolution kernel parameters, the number of LSTM hidden units, and the dropout rate of each branch are trained independently to learn the exclusive temporal feature representation of each subgroup. Finally, the dynamic feature fusion module performs dimension alignment and channel splicing on the output of each branch, and outputs the gesture recognition result through a fully connected layer and a Softmax classifier; Step (3), real-time gesture recognition; The trained model is deployed in the host computer program, and the gesture action is recognized online by combining the sliding window real-time inference mechanism. The optimal gesture recognition model is used to process the real-time collected multimodal sensor data, output the recognition results and confidence scores, and display them on the host computer interface.

3. The gesture recognition method based on multi-sensor fusion and adaptive learning according to claim 2, characterized in that: Step (1) includes the following steps: Step (1.1): Fabrication of the MWCNT / PDMS composite flexible pressure sensor; Step (1.2): Attach and fix the 9 flexible pressure sensors to the designated positions on the outer surface of the glove, and place the 1 inertial measurement unit at the center of the back of the glove; Step (1.3): Amplify and filter the signals output by the flexible pressure sensor and the inertial measurement unit; The amplification module is based on the LM358 operational amplifier chip and uses a non-inverting amplifier configuration. The gain is adjustable in the range of 1-20 times through surface-mount trimmer resistors. Gain adjustment is achieved through surface-mount trimmer resistors. The LM358 non-inverting amplifier circuit adopts a single operational amplifier structure. The input signal is connected to the non-inverting input terminal, and the feedback network is composed of gain resistors. The filtering adopts a two-stage series filter. The first stage is a three-point comparator filter, and the second stage is a Kalman filter. The process noise covariance matrix of the Kalman filter is Q=diag(0.01), and the observation noise covariance is R=0.

1. Step (1.4): The conditioned signal is converted from analog to digital and packaged by the main control unit, and then transmitted to the host computer through the communication module.

4. The gesture recognition method based on multi-sensor fusion and adaptive learning according to claim 3, characterized in that: Step (1.1) includes the following preparation process, Step (1.1.1): Weigh 15g of PDMS solution; The solution was obtained by mixing PDMS prepolymer and curing agent at a mass ratio of 10:1 and then degassing under vacuum. Then, 0.25g of MWCNT powder and 4.75g of the above PDMS solution were weighed and 10mL of n-hexane was added as a dispersant. Step (1.1.2): Place the mixture in an ultrasonic cleaner and ultrasonically treat it for 30 minutes to make MWCNT uniformly dispersed in the solvent and form a stable conductive filler suspension; Step (1.1.3): Transfer the suspension to a magnetic stirrer and mechanically stir for 30 minutes to promote dispersion and accelerate the volatilization of n-hexane until the mixture is a uniform black paste. Step (1.1.4): Pour the paste mixture into a polytetrafluoroethylene mold, place it in a constant temperature oven, and cure it at 70°C for 6 hours; Step (1.1.5): After demolding, a flexible pressure sensing unit is obtained; conductive silver paste is used to fix wires as electrodes on both sides of the sensing unit. After the conductive silver paste has fully cured, silicone sealant is applied to the surface to prevent the wires from falling off. Conductive silver paste is used to fix wires as electrodes on both sides of the sensing unit. After the conductive silver paste has fully cured, silicone sealant is applied to the surface to prevent the wires from falling off during finger bending. Step (1.1.6): After the sealant has fully cured, the sensing unit is encapsulated in a microbag made of self-adhesive bandage, and then sewn onto the outer surface of the glove (back of the hand) at the corresponding position of the finger joint.

5. The gesture recognition method based on multi-sensor fusion and adaptive learning according to claim 3, characterized in that: In step (1.1.2), before ultrasonic treatment, the MWCNT powder can be pre-dried in a vacuum oven at 70 to 100°C for 1-2 hours to remove adsorbed moisture.

6. The gesture recognition method based on multi-sensor fusion and adaptive learning according to claim 3, characterized in that: In step (1.2), the specific arrangement of the 9 flexible pressure sensors is as follows: 1 channel of the little finger metacarpophalangeal joint MCP, 1 channel of the little finger proximal interphalangeal joint PIP, 1 channel of the ring finger metacarpophalangeal joint MCP, 1 channel of the ring finger proximal interphalangeal joint PIP, 1 channel of the middle finger metacarpophalangeal joint MCP, 1 channel of the middle finger proximal interphalangeal joint PIP, 1 channel of the index finger metacarpophalangeal joint MCP, 1 channel of the index finger proximal interphalangeal joint PIP, and 1 channel between the thumb metacarpophalangeal joint and the proximal interphalangeal joint.

7. The gesture recognition method based on multi-sensor fusion and adaptive learning according to claim 3, characterized in that: In step (1.3), the three-point comparison filter works by taking three consecutive sampling points x(t-1), x(t), and x(t+1). If |x(t)-x(t-1)|>δ and |x(t)-x(t+1)|>δ, where δ is a preset threshold, then x(t) is determined to be spike interference and replaced with [x(t-1)+x(t+1)] / 2; otherwise, x(t) is retained.

8. The gesture recognition method based on multi-sensor fusion and adaptive learning according to claim 2, characterized in that: Step (2) includes the following steps: Step (2.1): Establish a data preprocessing module; The data preprocessing module is used to normalize, slice, and convert the format of the original 15-dimensional time series data to generate neural network input tensors; The 15-dimensional time-series data consists of outputs from 9 flexible pressure sensors and 6 inertial measurement units, denoted as X(t)=[p1(t),p2(t),...,p9(t),a] x (t),a γ (t),a u (t),g x (t),g γ (t),g u [(t)], where p i (t) represents the output of the i-th pressure sensor, a x (t), a γ (t), a u (t) represents the output of the triaxial accelerometer, g x (t), g γ (t), g u (t) represents the output of the three-axis gyroscope; During the data collection phase, 100 sets of gesture data of different durations were collected for each English letter; The normalization method used is Min-Max normalization, which normalizes the data independently for each channel. The formula is as follows: x' i = (x i - x i ,min) / (x i ,max - x i ,min) Where, x i For the raw data of the i-th channel, x i min and x i ,max are the minimum and maximum values ​​of the channel in the training set, respectively. After normalization, the data of each channel is mapped to the interval [0,1]. After encoding, statistical analysis was performed on the data samples to verify the uniformity of the label distribution. The encoded data was divided into training and test sets in an 8:2 ratio, with 20% of the data allocated to the test set. Step (2.2): Establish the K-Means clustering grouping module; The statistical characteristics of each channel are calculated from the raw time-series data of all multi-channel flexible pressure sensors in the training set, and a feature description matrix is ​​constructed. The inertial measurement unit channels do not participate in clustering and are directly and fixedly divided into independent feature subgroups. The K-Means++ initialization strategy is used for clustering. Step (2.3): Establish a grouped independent feature extraction module; The grouping method is data-driven. It automatically learns the feature correlation between channels through K-Means clustering, and groups channels with similar physical locations or similar motion patterns into the same subgroup. The channel composition of each subgroup is not fixed and is adaptively adjusted as the training data changes. The grouped independent feature extraction module consists of K Conv1D-LSTM branches, with each feature subgroup independently inputting into one branch; Each branch structure is dynamically configured based on the number of channels in the input subgroup: For subgroups with ≥3 channels, configure Conv1D layer → BatchNorm layer → ReLU activation → MaxPool1D layer → LSTM layer → Dropout layer; For a subgroup with 2 channels, configure Conv1D layer → BatchNorm layer → ReLU activation → MaxPool1D layer → LSTM layer → Dropout layer; For the subgroup with 1 channel, no Conv1D layer is configured; instead, the LSTM layer → Dropout layer is used directly. Each branch is configured with an independent fully connected layer Dense(32) after the LSTM layer to compress the temporal features of each branch; The convolution kernel parameters, LSTM hidden unit number and fully connected layer parameters of each branch are trained independently, and the dropout rate Pk is set independently according to the characteristics of each subgroup. Step (2.4): Establish a dynamic feature fusion module; The dynamic feature fusion module is used to perform dimension alignment and concatenation on the feature vectors output by each branch; It adopts a dimension alignment and channel splicing strategy based on global average pooling, which has low computational complexity and fast inference speed, making it suitable for real-time gesture recognition scenarios; Dimension alignment uses global average pooling to map the outputs of each branch to a unified dimension D; the formula for global average pooling is: H' k = (1 / T')·Σ t H k (t); where T' is the length of the LSTM output sequence, H k (t) represents the LSTM hidden state of the k-th branch at time t, H' k The feature vector after pooling; Concatenation involves concatenating K feature vectors of dimension D along the channel dimension to form a fused feature vector F, which has a dimension of K×D. The fused feature vector F passes through two fully connected layers. The first fully connected layer has 128 units and uses ReLU activation function; the second fully connected layer has 64 units and uses ReLU activation function. Finally, it passes through the Softmax activation function to output the probability distribution of each category. Step (2.5): Use the cross-entropy loss function and the L2 regularization term to establish the backpropagation loss function; Specifically, the backpropagation loss function L is established using the following equation (1): L = Lce + λ·Lreg (1) Where Lce is the cross-entropy loss, and is obtained from the following equation (2): (2) Where y is the one-hot encoding of the actual label. Let λ be the probability distribution of the model output, λ be the regularization coefficient, and Lreg be the L2 regularization term. The formula for calculating the L2 regularization term is: Lreg = Σ k ||W k || 2 _F; where, ||·|| 2 _F represents the Frobenius norm, W k For all trainable weight parameters in the k-th branch; The regularization coefficient λ is 0.

001. The purpose of the regularization coefficient λ is to prevent the model from overfitting. By adding a penalty term to the weight parameters in the loss function, the model is encouraged to learn a more concise feature representation. Step (2.6): Train the neural network and calculate the loss function L; Training stops when the number of training iterations reaches the set number or the loss error is less than the set threshold, thus obtaining the optimal gesture recognition model.

9. The gesture recognition method based on multi-sensor fusion and adaptive learning according to claim 2, characterized in that: Step (2) includes the following execution flow: Step (2.2.1): Perform statistical analysis on the time-series data of each channel of the multi-channel flexible pressure sensor in the training set, and extract the data range r of each channel. i and standard deviation σ i Construct an N×2 statistical feature matrix Where N is the number of channels of the multi-channel flexible pressure sensor; the inertial measurement unit channels do not participate in clustering and are directly and fixedly divided into independent feature subgroups; Step (2.2.2): Based on the N×2 statistical feature matrix constructed in step (2.2.1), calculate the Euclidean distance between the statistical features of each channel of the multi-channel flexible pressure sensor, and construct the distance matrix. D ij =||s i -s j ||2,s i =[r i ,σ i [ ] represents the statistical feature vector of the i-th channel; Step (2.2.3): Use the distance matrix D as the input for K-Means clustering to cluster the statistical feature vectors of each channel of the multi-channel flexible pressure sensor; the inertial measurement unit channels do not participate in clustering and are fixedly divided into independent feature subgroups. Step (2.2.4): The K-Means++ initialization strategy is used to select initial cluster centers. First, a sample is randomly selected as the first cluster center. Then, for each sample x, the shortest distance D(x) between it and the selected cluster centers is calculated with probability D(x). 2 / ΣD(x') 2 Select the next cluster center; repeat until K initial centers are selected; Step (2.2.5): Execute the standard K-Means iterative algorithm until the change in cluster centers is less than the preset threshold 1e-4 or the maximum number of iterations of 300 is reached; Step (2.2.6): Perform the above steps for all integer values ​​of K in the range of 2 to 8, calculate the silhouette coefficient corresponding to each K value, and select the K value that maximizes the silhouette coefficient as the optimal number of clusters.

10. The gesture recognition method based on multi-sensor fusion and adaptive learning according to claim 8, characterized in that: In step (2.4), the parameters of each branch of Conv1D-LSTM are initialized independently. The initialization method is He initialization, that is, the weight parameters W are randomly sampled from a normal distribution with a mean of 0 and a variance of 2 / n_in, where n_in is the input dimension. The convolution kernel weights, LSTM gate weights and bias parameters of each branch are updated independently during training and do not share parameters. The strategy of independent parameter training enables each branch to learn the exclusive feature representation of its own subgroup.

Images