A wearable gesture interaction system and method based on multimodal flexible sensing

Abstract

The present invention discloses a wearable gesture interaction system and method based on multimodal flexible sensing. The wearable gesture interaction system includes: two modal flexible sensors, namely a flexible strain sensor and a flexible acceleration sensor, for capturing gesture simulation signals indicating finger bending state, three-dimensional hand posture, and hand swiping; a data acquisition module for converting the gesture simulation signals into digital gesture signals and outputting them in real time via a wireless communication module; and a host computer for receiving the digital gesture signals, performing gesture activity segmentation and preprocessing to obtain a Grammar angle field map of the gesture signals. A deep learning model based on a self-attention mechanism is then used to extract and fuse multimodal features to achieve gesture classification for gesture human-computer interaction. The present invention uses multimodal flexible sensors combined with deep learning to improve the accuracy of gesture recognition. Furthermore, the system is comfortable to wear and relatively low in cost, effectively meeting user requirements.

Description

Technical Field

[0001] The present invention relates to the fields of intelligent sensing technology and human-computer interaction, and in particular to a wearable gesture interaction system based on multimodal flexible sensing and a method thereof. Background Art

[0002] With the rapid development of IoT technology and ubiquitous computing, gesture recognition based on wearable devices has attracted the attention of researchers as an important component of human-computer interaction. Wearable sensors, unrestricted by ambient light, noise, and other conditions, can acquire the wearer's hand status data in real time, perform gesture recognition, and understand gesture intent. These technologies are widely used in fields such as health monitoring, virtual reality, and smart homes. Traditional wearable devices often use rigid microelectromechanical systems as sensing units. These devices have a noticeable foreign body sensation when worn, are difficult to fabricate into irregular or personalized shapes, lack flexibility and stretchability, and cannot fit the wearer tightly. In specific applications, they cannot accurately perceive the wearer's gesture status information in real time, resulting in a poor user experience. Therefore, gesture recognition based on flexible sensing has great development potential, contributing to more comfortable, convenient, and natural human-computer interaction applications.

[0003] Currently, models used for gesture recognition fall into two main categories: those based on classic machine learning algorithms and those based on deep learning algorithms. Commonly used classic machine learning algorithms, such as decision trees, random forests, and support vector machines, have been extensively studied in the field of wearable gesture recognition. However, these classic algorithms require complex and time-consuming feature engineering, which not only requires manual design and extraction of features but also requires feature selection or dimensionality reduction to identify highly representative features. Deep learning-based methods avoid this feature engineering step and utilize a multi-layered nonlinear network structure to extract higher-level feature information layer by layer, facilitating the recognition of more complex gestures.

[0004] However, gesture recognition models based on deep learning algorithms also face some unavoidable problems. Gesture recognition methods using recurrent neural networks (RNNs) can only extract features according to the temporal order of the input gesture data. The calculation of the next step during training is highly dependent on the results of the previous step, making training time-consuming. Gesture recognition methods using convolutional neural networks (CNNs) often use one-dimensional convolution kernels, making it difficult for the network to fully utilize the rich high-dimensional data features. Alternatively, the one-dimensional gesture time series data is rearranged and simply listed, lacking interpretability. Alternatively, the gesture time series data is converted into two-dimensional images through time-frequency Fourier transforms, resulting in a significant increase in computational complexity, making it unsuitable for gesture-based human-computer interaction applications that require high portability and real-time performance.

[0005] Furthermore, existing gesture-based human-computer interaction applications often rely on gesture recognition methods based on a single sensor modality, making it difficult to distinguish between similar hand shapes. For example, in American Sign Language, the finger curves of these sign language signals are essentially the same, differing only in the orientation of the hand. Using multimodal sensors for gesture recognition can help improve its effectiveness and enrich practical application scenarios for gesture-based human-computer interaction. Multimodal gesture data not only suffers from data redundancy, but also exhibits significant differences in the features expressed by different data. Current multimodal fusion strategies for gesture recognition primarily include sensor-level, feature-level, and decision-level fusion. Feature-level fusion is the most prevalent fusion method, employing either handcrafted features or deep learning features to construct multimodal gesture recognition models. However, existing multimodal gesture recognition models based on flexible sensors only learn feature relationships within a single modality and lack appropriate mechanisms to handle nonlinear correlations between modalities, resulting in poor gesture recognition performance. Summary of the Invention

[0006] In order to overcome the shortcomings of the existing technology, the present invention proposes a wearable gesture interaction system and method based on multimodal flexible sensing, so as to effectively extract gesture-specific information of a single flexible sensing modality and nonlinear related information of gestures between multiple flexible sensing modalities, thereby recognizing refined gestures, improving the accuracy of gesture recognition and making the wearable device comfortable to wear.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The wearable gesture interaction system based on multimodal flexible sensing of the present invention is characterized by comprising:

[0009] The flexible sensing module includes a flexible strain sensor and a flexible acceleration sensor, which are used to convert the finger bending state, three-dimensional hand posture and gesture state data of the hand swiping into analog signal output;

[0010] The data acquisition module obtains the output value of the flexible strain sensor through a multi-channel analog-to-digital conversion module, reads the output values ​​of the three axial accelerations of the flexible acceleration sensor through an integrated circuit bus, and then converts the output value of the flexible sensor module into a digital signal and outputs it through a wireless transmission main module;

[0011] The host computer receives the digital signal output by the data acquisition module from the module through wireless transmission, and after performing gesture activity segmentation and preprocessing on the digital signal, obtains the Gram angle field map of the gesture signal, and then inputs it into the deep learning network model for recognition. After mapping the recognition result into human-computer interaction instructions, it is displayed on the interactive interface.

[0012] The deep learning network model includes: an input unit, multi-stream branches, a multimodal fusion unit and an output unit;

[0013] The input unit is used to divide the Gram angle field map of the preprocessed gesture signal according to the sensing modality to obtain the Gram angle field map under each sensing modality;

[0014] The multi-stream branch is used to receive and process the Gram angular field map under each sensing modality, and output the hidden layer features under each sensing modality;

[0015] The number of branches of the multi-stream branch is equal to the number of sensing modalities, and each branch includes: a normal convolution layer, a second convolution layer, a third convolution layer, and a long short-term memory layer in sequence, wherein the second convolution layer and the third convolution layer are each composed of a basic convolution module and a downsampling convolution module;

[0016] The common convolution layer includes: a two-dimensional convolution layer for receiving the Gram angular field map of all channels under a single sensing modality and outputting a feature map;

[0017] The basic convolution module includes a first channel separation module and a first depth-wise separable convolution layer for feature extraction; the first channel separation module receives the feature map output by the previous layer and divides the feature map into two identical branches according to the number of channels of the feature map for processing, wherein the feature map of one branch remains unchanged, and the feature map of the other branch is input into the first depth-wise separable convolution layer and processed in sequence by 1*1 convolution, 3*3 depth-wise convolution, and 1*1 convolution, and then spliced ​​with the unchanged feature map, and finally the spliced ​​feature map is randomly shuffled by the channel and output;

[0018] The downsampling convolution module includes a second channel separation module and two depth-separable convolution layers for feature extraction; the second channel separation module receives the feature map output by the previous layer, and divides the downsampled feature map into two identical branches according to the number of channels of the feature map for processing, wherein the feature map of one branch is input into the second depth-separable convolution layer, and is sequentially processed by a 3*3 depth convolution with a stride of 2 and a 1×1 convolution with a stride of 1, and then outputs the second feature map; the feature map of the other branch is input into the third depth-separable convolution layer, and is sequentially processed by a 1*1 Conv, a 3×3 depth convolution with a stride of 2, and a 1*1 convolution, and then output after channel cascade with the second feature map;

[0019] The long short-term memory layer includes a forget gate, an input gate, and an output gate, which are used to receive the feature map output by the previous layer and extract the temporal features within the modality, and then convert them into hidden layer state outputs within the modality;

[0020] The multimodal fusion unit receives the short-term memory layer output of each branch in the multi-stream branch, and fuses the nonlinear features between different modalities using a self-attention mechanism to obtain a multimodal fusion feature;

[0021] The output unit includes a fully connected layer unit and a Softmax unit, wherein the fully connected layer unit is used to receive the multimodal fusion features and perform dimensionality reduction processing; the Softmax unit is used to receive the reduced dimensionality features output by the fully connected layer unit and perform category probability prediction through cross entropy loss.

[0022] The wearable gesture interaction system of the present invention is also characterized in that the flexible sensing module includes: a flexible strain sensor based on a nanofiber substrate and a liquid metal circuit and a flexible acceleration sensor based on a nanofiber substrate and a liquid metal circuit;

[0023] The flexible strain sensor based on the nanofiber substrate and the liquid metal circuit is a sandwich structure, comprising: a strain-sensitive flexible substrate, a strain-sensitive patterned circuit, and strain-sensitive package pins; wherein the strain-sensitive patterned circuit is the middle layer, and the upper and lower layers are the strain-sensitive flexible substrate respectively;

[0024] The strained flexible substrate is a thermoplastic polyurethane nanofiber membrane obtained by electrospinning a thermoplastic polyurethane solution;

[0025] The preparation process of the thermoplastic polyurethane solution is:

[0026] The polyurethane particles were dissolved in dimethylformamide and tetrahydrofuran solution, and then stirred thoroughly with a magnetic swivel at room temperature to obtain a uniform thermoplastic polyurethane solution;

[0027] The strain sensing patterned circuit is formed by printing liquid metal on the upper surface of the underlying strain flexible substrate through mask printing;

[0028] The strain sensing package pins include positive and negative pins. After a rigid wire is placed in a position reserved for the patterned circuit, polyvinyl alcohol gel is dripped onto the rigid wire. After the polyvinyl alcohol gel is cured, the rigid wire is fixed to the strain sensing patterned circuit.

[0029] The flexible triaxial acceleration sensor based on the nanofiber substrate and the liquid metal circuit has a sandwich structure, comprising: an accelerometer flexible substrate, an accelerometer chip, an accelerometer patterned circuit, and accelerometer package pins; the accelerometer patterned circuit and the accelerometer chip form the middle layer, and the upper and lower layers form the accelerometer flexible substrate;

[0030] The accelerometer flexible substrate is a thermoplastic polyurethane nanofiber membrane obtained by electrostatically spinning a thermoplastic polyurethane solution;

[0031] The accelerometer patterned circuit is formed by printing liquid metal on the upper surface of the lower flexible substrate through mask printing, and serves as a peripheral circuit for reading the accelerometer chip;

[0032] The accelerometer chip is placed in a position reserved for the accelerometer patterned circuit, and polyvinyl alcohol gel is dripped onto the pins of the accelerometer chip; after the polyvinyl alcohol gel is solidified, the accelerometer chip is fixed on the patterned circuit;

[0033] The accelerometer package pins include three pins: a ground line, a serial data line, and a serial clock line. After a rigid wire is placed in a reserved position on the patterned circuit, polyvinyl alcohol gel is dripped onto the rigid wire. After the polyvinyl alcohol gel solidifies, the rigid wire is fixed to the accelerometer patterned circuit.

[0034] The wearable gesture interaction method based on multimodal flexible sensing of the present invention is characterized in that it includes the following steps:

[0035] Step 1: Get gesture data:

[0036] The real-time gesture signals output by the multimodal flexible sensor of different gesture actions are collected by flexible wearable devices, and the real-time gesture signals are segmented and preprocessed to obtain the Gram matrix of the gesture signal in each sensing channel. , thus forming a multimodal gesture dataset , where Stres represents the Gram matrix of the output signal of the flexible strain sensor in the multimodal flexible sensor, and ACC represents the Gram matrix of the output signal of the flexible acceleration sensor in the multimodal flexible sensor. A set of labels representing gesture categories;

[0037] Step 2: Construct a deep learning network model, including: an input unit, a multi-stream branch with two parallel network branches, a multimodal fusion unit, and an output unit; wherein the number of branches of the multi-stream branch is equal to the number of sensing modalities, and each branch includes: a normal convolution layer, a second convolution layer, a third convolution layer, and a long short-term memory layer in sequence; the normal convolution layer includes: 1 convolution kernel with the number of , a 2D convolutional neural network layer with a kernel size of 3*3 and a Relu activation function layer; the second convolution layer and the third convolution layer are both composed of a basic convolution module and a downsampling convolution module; the basic convolution module includes: a first channel separation module, a first depth-separable convolution layer for feature extraction; the downsampling convolution module includes: a second channel separation module, two depth-separable convolution layers for feature extraction; the long short-term memory layer is a single-layer long short-term memory network with a hidden layer size of h and a dropout layer with a dropout probability of p; the output unit includes: a fully connected layer and a Softmax layer;

[0038] Step 2.1 Hand gesture sample dataset Input units are input and divided according to sensing modes to obtain the Gram angle field map under each sensing mode;

[0039] In step 2.2, the multi-stream branch receives the Gram angular field map under each sensing modality and inputs the Gram angular field map Stres into one branch. After being processed by the ordinary convolution layer, the second convolution layer, the third convolution layer and the long short-term memory layer in sequence, the final output length is The hidden layer features of

[0040] The Gram angle field map ACC is input into another branch, and passes through the normal convolution layer, the second convolution layer, the third convolution layer and the long short-term memory layer in sequence. The final output length is The hidden layer features of

[0041] Step 2.3 The multimodal fusion unit will be of length The hidden layer features and length are The hidden layer features of are concatenated into a new feature map H of length n, and the query Q, key value K, and weight vector S are obtained using equations (1), (2), and (3);

[0042] (1)

[0043] (2)

[0044] (3)

[0045] In formulas (1) to (3), b represents the bias term, , d represents the attention length, tanh represents the activation function; Represents the nonlinear transformation of the fully connected layer; softmax represents the softmax operation;

[0046] Step 2.4 The output unit inputs the weight vector S into the fully connected layer for dimensionality reduction, and then processes it through the Softmax layer to obtain The gesture with the highest probability;

[0047] Step 3: Set the network training hyperparameters including the number of sample batches, learning rate, and maximum number of epochs.

[0048] A cross entropy loss function is used as the loss function, and the deep learning network model is trained using the Adam optimizer. The loss function is calculated to update the model parameters until the maximum number of iterations is reached, thereby obtaining a trained gesture recognition model for gesture recognition.

[0049] The wearable gesture interaction method based on multimodal flexible sensing according to the present invention is also characterized in that the real-time gesture signal activity segment segmentation and preprocessing in step 1 includes:

[0050] According to the real-time acceleration values ​​of the three axes output by the flexible acceleration sensor, the collected real-time gesture signals are segmented, and the strain sensing signals and accelerometer signals of the gesture activity segment are extracted, and then gesture samples with a sampling length of L are obtained. Each gesture sample consists of the original signals of 5 strain sensing channels and 3 accelerometer channels. The original signal of any channel is recorded as Vorg={Vorg (1), Vorg (2),…,Vorg (l),…,Vorg (L)}, where Vorg (l) represents the lth original signal of any channel.

[0051] The original signal Vorg is filtered, normalized and resampled to obtain a signal with a length of The single-channel signal of the gesture sample Sample={ Sample (1), Sample (2),…,Sample ( ) ,…,Sample ( )}, where Sample ( ) indicates the The original signal after filtering and resampling;

[0052] Normalize the single-channel signal Sample of the gesture sample so that each element Sample ( ) is between (-1, 1), and then the normalized single-channel signal Sample is converted into a Gram matrix and used as the input of the deep learning network model.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention adopts a flexible and stretchable sensor based on a nanofiber substrate and liquid metal as the sensing unit of the wearable device. Based on the signal characteristics of the flexible sensor, a data acquisition module and data preprocessing method are designed to obtain real-time and accurate multimodal gesture motion data. This solves the problems of traditional wearable devices such as strong foreign body sensation, single sensing modality, and inability to personalize sensor size and properties. It constructs a multimodal gesture wearable device that is comfortable, safe, and real-time.

[0055] 2. The present invention uses the Gram angular field to map the one-dimensional gesture data of each sensing channel from a rectangular coordinate system to a polar coordinate system. Through the Gram matrix, it is quickly converted into a two-dimensional image similar to the real image, which can provide high-order features for deep learning models. It provides interpretability for the conversion of one-dimensional time series data of gestures into two-dimensional data while reducing the amount of computation.

[0056] 3. This paper designs a multi-stream branched CNN-LSTM network to extract features from each sensing modality. Within the CNN module, a lightweight convolutional unit is constructed using a basic module based on deep convolution and a downsampling module. Reluctant linear unit (ReLU) is used as the activation function to achieve pixel-level spatial information modeling. The addition of an LSTM layer after the CNN overcomes the lack of temporal dependency of features extracted by the CNN network, resulting in a model with excellent noise immunity.

[0057] 4. The present invention designs a multimodal gesture feature fusion mechanism based on self-attention, measures gesture data of different modalities, gives different attention to each hidden state, mines nonlinear correlation information between multiple modalities, solves the problem of multimodal data redundancy, and effectively improves the accuracy of multimodal gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a system block diagram of the present invention;

[0059] Figure 2a This is a schematic structural diagram of the flexible strain sensor of the present invention;

[0060] Figure 2b is a cross-sectional view of the flexible acceleration sensor of the present invention;

[0061] Figure 2c Schematic diagram of the flexible acceleration sensor of the present invention;

[0062] Figure 3 This is a schematic diagram of the layout of the flexible sensing module of the present invention;

[0063] Figure 4a The Chinese gesture representation involved in the embodiment of the present invention;

[0064] Figure 4b It is the custom human-machine control instruction gesture expression involved in the embodiment of the present invention;

[0065] Figure 5 This is a flowchart of gesture segmentation and preprocessing of the present invention;

[0066] Figure 6 This is a diagram of the gesture recognition deep learning network structure of the present invention;

[0067] Figure 7a It is the basic convolution block of the present invention;

[0068] Figure 7b is the downsampling convolution block of the present invention;

[0069] Figure 8 2 is a comparison diagram of the effects of the method of the present invention and other methods.

[0070] Numbers in the figure: 1 strain flexible substrate, 2 strain sensing patterned circuit, 3 strain sensing package pin, 4 accelerometer flexible substrate, 5 accelerometer chip, 6 accelerometer patterned circuit, 7 polyvinyl alcohol gel, 8 accelerometer package pin, 9 flexible strain sensor, 10 flexible acceleration sensor. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of the present invention easier to understand, they are further described in detail with reference to specific embodiments and the accompanying drawings.

[0072] In this embodiment, a wearable gesture interaction system and method based on multimodal flexible sensing has the advantages of wearing comfort, safety and high stretchability in terms of sensors; in terms of gesture interaction, through the Gram angle field and improved CNN-LSTM network, combined with the self-attention mechanism, it can effectively extract the specific information of a single modality and the nonlinear related information between multiple modalities, and can perceive the gesture state in real time, recognize refined gestures, and use the gesture recognition results for human-computer interaction. Specifically, Figure 1 As shown, a wearable gesture interaction system based on multimodal flexible sensing includes:

[0073] The flexible sensing module includes a flexible strain sensor and a flexible acceleration sensor, which are used to convert the finger bending state, three-dimensional hand posture and gesture state data of the hand swiping into analog signal output;

[0074] The data acquisition module obtains the output value of the flexible strain sensor through the multi-channel analog-to-digital conversion module, and reads the output values ​​of the three axial accelerations of the flexible acceleration sensor through the integrated circuit bus. The output value of the flexible sensor module is then converted into a digital signal and output through the wireless transmission main module.

[0075] The host computer receives the digital signal output by the data acquisition module from the module through wireless transmission, and after segmenting and preprocessing the digital signal into gesture activity segments, obtains the Gram angle field map of the gesture signal, and then inputs it into the deep learning network model for recognition. After mapping the recognition results into human-computer interaction instructions, they are displayed on the interactive interface.

[0076] Among them, the deep learning network model includes: input unit, multi-stream branches, multi-modal fusion unit and output unit;

[0077] The input unit is used to divide the Gram angle field map of the preprocessed gesture signal according to the sensing modality to obtain the Gram angle field map under each sensing modality;

[0078] The multi-stream branch is used to receive and process the Gram angular field map under each sensing modality, and then output the hidden layer features under each sensing modality;

[0079] The number of multi-stream branches is equal to the number of sensing modalities. Each branch includes: a normal convolution layer, a second convolution layer, a third convolution layer, and a long short-term memory layer. The second and third convolution layers are composed of a basic convolution module and a downsampling convolution module.

[0080] The common convolution layer includes: a two-dimensional convolution layer that receives the Gram angular field map of all channels under a single sensing modality and outputs a feature map;

[0081] The basic convolution module includes a first channel separation module and a first depth-wise separable convolution layer for feature extraction. The first channel separation module receives the feature map output by the previous layer and divides the feature map into two identical branches according to the number of channels of the feature map for processing. The feature map of one branch remains unchanged, and the feature map of the other branch is input into the first depth-wise separable convolution layer. After being processed by 1*1 convolution, 3*3 depth-wise convolution, and 1*1 convolution in sequence, it is spliced ​​with the unchanged feature map. Finally, the spliced ​​feature map is randomly shuffled by channel and output.

[0082] The downsampling convolution module includes a second channel separation module and two depth-wise separable convolution layers for feature extraction. The second channel separation module receives the feature map output by the previous layer and divides the downsampled feature map into two identical branches according to the number of channels of the feature map for processing. The feature map of one branch is input into the second depth-wise separable convolution layer and sequentially processed by a 3*3 depth-wise convolution with a stride of 2 and a 1×1 convolution with a stride of 1 to output the second feature map. The feature map of the other branch is input into the third depth-wise separable convolution layer and sequentially processed by a 1*1 Conv, a 3×3 depth-wise convolution with a stride of 2, and a 1*1 convolution, and then output after channel concatenation with the second feature map.

[0083] The long short-term memory layer includes a forget gate, an input gate, and an output gate. It is used to receive the feature map output by the previous layer and extract the temporal features within the modality, and then convert them into the hidden state output within the modality.

[0084] The multimodal fusion unit receives the short-term memory layer outputs of each branch in the multi-stream branch, and uses the self-attention mechanism to fuse the nonlinear features between different modalities to obtain the multimodal fusion features;

[0085] The output unit includes a fully connected layer unit and a Softmax unit. The fully connected layer unit is used to receive multimodal fusion features and perform dimensionality reduction processing; the Softmax unit is used to receive the reduced dimensionality features output by the fully connected layer unit and perform category probability prediction through cross entropy loss.

[0086] The flexible sensing module of the present invention is described in detail below: The flexible sensing module includes a flexible strain sensor based on a nanofiber substrate and a liquid metal circuit and a flexible acceleration sensor based on a nanofiber substrate and a liquid metal circuit;

[0087] The flexible strain sensor based on nanofiber substrate and liquid metal circuit is a sandwich structure, such as Figure 2a As shown, it includes: a strain-sensitive flexible substrate 1, a strain-sensitive patterned circuit 2, and strain-sensitive package pins 3. The middle layer is the strain-sensitive patterned circuit, and the upper and lower layers are the strain-sensitive flexible substrate. The strain-sensitive flexible substrate is a thermoplastic polyurethane (TPU) nanofiber membrane obtained by electrospinning a thermoplastic polyurethane (TPU) solution. It has high stretchability, high reliability, customizable tailoring, and is environmentally friendly. The thermoplastic polyurethane solution is prepared by dissolving polyurethane particles in a 1:1 volume ratio of dimethylformamide and tetrahydrofuran, followed by thorough stirring with a magnetic spinner at room temperature to obtain a uniform TPU solution. The strain-sensitive patterned circuit is printed on the upper surface of the lower flexible substrate by mask printing of liquid metal. The mask is obtained by engraving the designed circuit pattern on a 0.2mm polyvinyl chloride flexible plate using an engraving machine. The circuit pattern is a 1. The flexible strain sensor is a 5mm wide and 30mm long straight line. The liquid metal is gallium-based liquid metal. The strain sensing package pins include positive and negative pins. A rigid wire is placed in a reserved position on the patterned circuit, and polyvinyl alcohol gel is dripped onto the rigid wire. After the polyvinyl alcohol gel solidifies, the rigid wire is fixed to the patterned circuit. The flexible strain sensor is sensitive to external tensile force, specifically, the output resistance of the sensor increases with the increase of external tensile force. The flexible strain sensor is stretchable and undergoes mechanical deformation within 200% with increasing external tensile force. It can be comfortably attached to the human finger joint, and the bending state of the finger can be easily and conveniently detected.

[0088] The flexible acceleration sensor based on nanofiber substrate and liquid metal circuit is a sandwich structure, the cross-section of which is shown in the figure below. Figure 2b As shown, the overall schematic diagram is as follows Figure 2c As shown, the device comprises an accelerometer flexible substrate 4, an accelerometer chip 5, an accelerometer patterned circuit 6, polyvinyl alcohol gel 7, and accelerometer package pins 8. The accelerometer patterned circuit and accelerometer chip form the middle layer, while the upper and lower layers form the accelerometer flexible substrate. The accelerometer flexible substrate is a thermoplastic polyurethane nanofiber membrane obtained by electrospinning a TPU solution. The thermoplastic polyurethane solution is prepared by dissolving polyurethane particles in a 1:1 volume ratio of dimethylformamide and tetrahydrofuran, followed by thorough magnetic stirring at room temperature to obtain a uniform TPU solution. The accelerometer patterned circuit is a gallium-based liquid metal that is printed onto the upper surface of the lower flexible substrate using a mask. The patterned circuit serves as the peripheral circuit for reading the accelerometer chip. The accelerometer chip is placed in the reserved position on the patterned circuit, and the polyvinyl alcohol gel is dripped onto the accelerometer chip pins. After the polyvinyl alcohol gel solidifies, the accelerometer chip is fixed to the patterned circuit. The accelerometer package pins include ground, serial data, and serial clock. After placing a rigid wire in a designated location on the patterned circuit, polyvinyl alcohol gel is dripped onto the wire. Once the polyvinyl alcohol gel solidifies, the wire is fixed to the patterned circuit, allowing for convenient connection to an external microprocessor pin for accelerometer readings. The device comfortably adheres to the surface of the hand, enabling real-time detection of hand posture information.

[0089] In this embodiment, a wearable gesture interaction method based on multimodal flexible sensing includes the following steps:

[0090] Step 1: Get gesture data:

[0091] The real-time gesture signals output by the multimodal flexible sensor of different gesture actions are collected by flexible wearable devices, and the real-time gesture signals are segmented and preprocessed to obtain the Gram matrix of the gesture signal in each sensing channel. , thus forming a multimodal gesture dataset , where Stres represents the Gram matrix of the output signal of the flexible strain sensor in the multimodal flexible sensor, and ACC represents the Gram matrix of the output signal of the flexible acceleration sensor in the multimodal flexible sensor. A set of labels representing gesture categories;

[0092] In this example, eight adult users with normal finger function participated in data collection. Each participant wore the flexible sensor five times before data collection to ensure that the wearable device could fit their hand well when performing gestures.

[0093] The layout of the flexible sensing module is as follows Figure 3 As shown, the flexible strain sensor 9 on the fingers and the flexible acceleration sensor 10 on the back of the hand have a total of five flexible strain sensing channels and three accelerometer channels, namely five channels for Stres and three channels for Acc. A flexible strain sensor is attached to each finger, and a flexible acceleration sensor is attached to the center of the palm surface, collecting hand motion signals at a frequency of 30 Hz.

[0094] Gestures such as Figure 4a 、 Figure 4b As shown in Figure 1, before data collection, users receive a certain introduction to gestures so that they can successfully complete the experiment. In this embodiment, gestures are divided into two categories, such as Figure 4a There are 10 commonly used Chinese sign language gestures, such as Figure 4b It is a customizable gesture expression for 5 human-machine control commands.

[0095] The real-time gesture signal activity segment segmentation and preprocessing in step 1 include:

[0096] According to the real-time acceleration values ​​of the three axes output by the flexible acceleration sensor, the collected real-time gesture signals are segmented, and the strain sensing signals and accelerometer signals of the gesture activity segment are extracted, and then gesture samples with a sampling length of L are obtained. Each gesture sample consists of the original signals of 5 strain sensing channels and 3 accelerometer channels. The original signal of any channel is recorded as Vorg={Vorg (1), Vorg (2),…,Vorg (l),…,Vorg (L)}, where Vorg (l) represents the lth original signal of any channel.

[0097] The original signal Vorg is filtered, normalized and resampled to obtain a signal with a length of The single-channel signal of the gesture sample Sample={ Sample (1), Sample (2),…,Sample ( ) ,…,Sample ( )}, where Sample ( ) indicates the The original signal after filtering and resampling;

[0098] In the specific implementation, the gesture segmentation and preprocessing process is as follows Figure 5As shown. First, perform gesture segmentation and perform windowing on the gesture signal. The window size is is 10, and formula (1) calculates the magnitude of the acceleration of each point in the window;

[0099] (1)

[0100] In formula (1) represents the acceleration amplitude value of the i-th sampling point, 、 、 Respectively represent the acceleration values ​​of the three axes;

[0101] Then, according to formula (2), the first-order difference value of the acceleration amplitude between the i-th sampling point and the corresponding point in the forward window is calculated:

[0102] (2)

[0103] In formula (2), Represents the first-order difference of the acceleration amplitude between the i-th sampling point and the corresponding point in the forward window;

[0104] Then, after obtaining the acceleration amplitude difference of each point in the window, the first-order difference variance of the acceleration amplitude value of each sampling point in the window is calculated according to formula (3): ;

[0105] (3)

[0106] Finally, the start threshold THst is set. When the first-order difference variance of the acceleration amplitude value of each sampling point in a window is When ACCX is greater than the start threshold THst, it indicates the beginning of an active segment, which ends when ACCX is less than the start threshold THst. Taking into account the duration of the gesture and avoiding the influence of meaningless jitter on the recognition result, only active segments longer than 0.8 seconds are considered valid gestures. The original signal of any channel of a valid gesture is recorded as Vorg={Vorg (1), Vorg (2), …, Vorg (l), …, Vorg (L)}, where Vorg (l) represents the lth original signal of any channel;

[0107] After segmenting the gesture activity segments, the activity segment signals must be preprocessed. Preprocessing is divided into two steps: smoothing filtering and cutoff filtering. During the gesture data collection process, factors such as hand tremors and fluctuations in the flexible sensor due to environmental influences can cause a certain amount of noise in the data. Specifically, this manifests as slight ups and downs in certain data segments. To filter out this noise so that the gesture data can better express the characteristics of the gesture, in this embodiment, equation (4) is used to smooth the data.

[0108] (4)

[0109] In formula (4), is the tth original signal, is the filtered data of the t-th original signal, 𝛽 is a coefficient in the range of (0,1), and is 0.8 in this example;

[0110] Cutoff filtering requires the wearer to fully open their palm before performing a gesture, straighten their knuckles, and maintain this position for a period of time to obtain the approximate minimum sensor response value for each joint. The minimum sensor response value for a particular strain channel is denoted as Vmin. Similarly, the wearer clenches their fist and bends each joint to the maximum angle to obtain the maximum sensor response value. The minimum sensor response value for a particular strain channel is denoted as Vmax, which is then normalized according to Equation (5).

[0111] (5)

[0112] In formula (5), represents the original signal of the dth strain channel after smoothing and filtering, express The value after cutoff filtering. In this example, the accelerometer channel does not perform cutoff filtering.

[0113] Because gestures and wearers are different, the length of each gesture signal varies. In order to ensure the input of the subsequent network model, the gesture signal is regularized to a uniform length using linear interpolation. In this example, we determine the average length of all gesture signals. is 75. The final gesture sample i-th channel signal is expressed as ={ (1), (2),…, ( ),…, ( )}, ( ) represents the gesture sample i-th channel The original signal after filtering and resampling; each gesture sample has 8 channels of sample signals.

[0114] Each channel signal of the gesture sample Normalize so that each element ( ) is between (-1, 1), and then the normalized single-channel signal Sample is converted into a Gram matrix and used as the input of the deep learning network model.

[0115] In a specific embodiment, according to formula (6), the single-channel signal Sample of the gesture sample is Sample={ Sample (1), Sample (2),…, Sample ( ) ,…,Sample ( )}, each point Sample ( ) corresponding timestamp , the amplitude value is converted to the radius in polar coordinates and angles After converting the coordinates into polar coordinates, calculate any two angles according to formula (7) and The trigonometric transformation of the sum, that is, using the cosine function of the sum of the two angles, generates * The Gram angle field matrix GASF.

[0116] (6)

[0117] (7)

[0118] Step 2: Build a deep learning network model. The overall structure is as follows: Figure 6 As shown, it includes: an input unit, a multi-stream branch with two parallel network branches, a multimodal fusion unit and an output unit; wherein the number of branches of the multi-stream branch is equal to the number of sensing modalities, and each branch includes: a normal convolution layer, a second convolution layer, a third convolution layer and a long short-term memory layer in sequence; a normal convolution layer includes: 1 convolution kernel with the number of , a 2D convolutional neural network layer with a kernel size of 3*3 and a Relu activation function layer; for example, the number of convolution kernels in the ordinary convolution layer is set to =24, the filling size is 2*2, and the stride is 1*1;

[0119] The second convolution layer and the third convolution layer are composed of a basic convolution module and a downsampling convolution module; Figure 7a As shown in , the basic convolution module includes: a first channel separation module, a first depth-separable convolution layer for feature extraction; Figure 7b As shown in the figure, the downsampling convolution module includes: a second channel separation module, two depth-wise separable convolution layers for feature extraction;

[0120] For example, in the basic convolution block, the first channel separation module receives the feature map output by the previous layer and divides the feature map into two equal parts according to the number of channels. , remain unchanged, Perform 1*1 convolution, batch normalization, relu function activation, 3*3 depth convolution, batch normalization and 1*1 convolution, batch normalization, relu function activation in sequence, and finally Splice them together and then shuffle them evenly according to the channels; the step size of all convolution processes is 1;

[0121] In the downsampling convolution block, the second channel separation module receives the feature map output by the previous layer and divides the feature map into two equal parts according to the number of channels. , It undergoes a depthwise convolution with a stride of 2 and a kernel size of 3*3, batch normalization, and a 1*1 convolution with a stride of 1, batch normalization, and ReLU activation. It undergoes a 1*1 convolution with a stride of 1, batch normalization, ReLU function activation, a depthwise convolution with a stride of 2 and a kernel size of 3*3, batch normalization, and a 1*1 convolution with a stride of 1, batch normalization, and ReLU function activation.

[0122] The long short-term memory layer is a single-layer long short-term memory network with a hidden layer size of h and a dropout layer with a dropout probability of p;

[0123] For example, h=256, p=0.2;

[0124] The output unit includes: fully connected layer and Softmax layer;

[0125] Step 2.1 Hand gesture sample dataset Input units are input and divided according to sensing modes to obtain the Gram angle field map under each sensing mode;

[0126] Step 2.2 The multi-stream branch receives the Gram angular field map under each sensing modality and inputs the Gram angular field map Stres into one branch. After being processed by the normal convolution layer, the second convolution layer, the third convolution layer and the long short-term memory layer in sequence, the final output length is Hidden layer features of; For example, =256;

[0127] The Gram angle field map ACC is input into another branch, and passes through the normal convolution layer, the second convolution layer, the third convolution layer and the long short-term memory layer in sequence. The final output length is Hidden layer features of; For example, =256;

[0128] Step 2.3 The multimodal fusion unit is The hidden layer features and length are The hidden layer features of are concatenated into a new feature map H of length n, and the query Q, key value K, and weight vector S are obtained using equations (8), (9), and (10);

[0129] (8)

[0130] (9)

[0131] (10)

[0132] In equations (8) to (10), b represents the bias term, , d represents the attention length, tanh represents the activation function; Represents the nonlinear transformation of the fully connected layer; softmax represents the softmax operation;

[0133] For example, the length of the final concatenated feature map is n=512; the attention length is b=128;

[0134] Step 2.4 The output unit inputs the weight vector S into the fully connected layer for dimensionality reduction, and then passes it through the Softmax layer to obtain The gesture with the highest probability;

[0135] Step 3: Set the network training hyperparameters including the number of sample batches, learning rate, and maximum number of epochs.

[0136] The cross entropy loss function is used as the loss function, and the Adam optimizer is used to train the deep learning network model. The loss function is calculated to update the model parameters until the maximum number of iterations is reached, thereby obtaining a trained gesture recognition model for gesture recognition.

[0137] In this embodiment, the proposed neural network is implemented using the pytorch deep learning framework, with batch size = 40, epoch = 100, and learning rate lr = 0.001. When the loss function Loss reaches the minimum or the number of iterations equals epoch, the training is stopped, thereby obtaining the best gesture recognition classification model.

[0138] In this embodiment, in order to quantitatively evaluate the effectiveness of the present invention, the gesture recognition method of the present invention is compared with classic machine learning algorithms and deep learning algorithms, including five methods: k-Nearest Neighbor (KNN), Bayesian, Support Vector Machine (SVM), Convolutional Neural Network (CNN), and CNN-LSTM. The obtained accuracy, recall rate, and F score are compared. Figure 8 As shown. Classical machine learning algorithms cannot effectively recognize complex sign languages. The CNN model ignores the temporal relationship between time series data, resulting in low evaluation indicators. The LSTM model only considers the temporal relationship between data and ignores the fine-grained features of the data, resulting in insufficient feature proposals and affecting the overall recognition effect. The CNN-LSTM model combines the advantages of the CNN model and the LSTM model, which improves its various indicators. However, because it ignores the complementarity between hidden layer features of different modalities and the association between different channels of the same modality, the various indicators no longer improve after reaching a certain level. The purpose of designing the ablation experiment is to verify the effectiveness of the feature fusion method used in this model. This model is compared with a deep learning model that does not use the self-attention mechanism (the self-attention feature fusion module is replaced by a fully connected network, called no attention), and its recognition effect is shown as follows. Figure 8 As shown in the figure, we can see that by using the self-attention mechanism, different weights are assigned to the hidden layer outputs of each modality, which helps to achieve information complementarity between different modalities and improve the gesture recognition effect.

[0139] Combining the above comparative experiments and recognition results, we can draw the following conclusions: 1) This invention utilizes multimodal flexible sensors to construct a gesture recognition system, improving wearable comfort and portability while meeting the needs of personalized sensor shapes. 2) The proposed deep learning network model effectively integrates multimodal sensor data, improving gesture recognition accuracy.

Claims

1. A wearable gesture interaction system based on multimodal flexible sensing, characterized in that: include: The flexible sensing module includes a flexible strain sensor and a flexible acceleration sensor, which are used to convert the finger bending state, three-dimensional hand posture and gesture state data of the hand swiping into analog signal output; The data acquisition module obtains the output value of the flexible strain sensor through a multi-channel analog-to-digital conversion module, reads the output values ​​of the three axial accelerations of the flexible acceleration sensor through an integrated circuit bus, and then converts the output value of the flexible sensor module into a digital signal and outputs it through a wireless transmission main module; The host computer receives the digital signal output by the data acquisition module from the module via wireless transmission, and after segmenting and preprocessing the digital signal into gesture activity segments, obtains the Gram angle field map of the gesture signal, inputs it into the deep learning network model for recognition, and maps the recognition result into human-computer interaction instructions, which are then displayed on the interactive interface; The deep learning network model includes: an input unit, multi-stream branches, a multimodal fusion unit and an output unit; The input unit is used to divide the Gram angle field map of the preprocessed gesture signal according to the sensing modality to obtain the Gram angle field map under each sensing modality; The multi-stream branch is used to receive and process the Gram angular field map under each sensing modality, and output the hidden layer features under each sensing modality; The number of branches of the multi-stream branch is equal to the number of sensing modalities, and each branch includes: a normal convolution layer, a second convolution layer, a third convolution layer, and a long short-term memory layer in sequence, wherein the second convolution layer and the third convolution layer are each composed of a basic convolution module and a downsampling convolution module; The common convolution layer includes: a two-dimensional convolution layer for receiving the Gram angular field map of all channels under a single sensing modality and outputting a feature map; The basic convolution module includes a first channel separation module and a first depth-wise separable convolution layer for feature extraction; the first channel separation module receives the feature map output by the previous layer and divides the feature map into two identical branches according to the number of channels of the feature map for processing, wherein the feature map of one branch remains unchanged, and the feature map of the other branch is input into the first depth-wise separable convolution layer and processed in sequence by 1*1 convolution, 3*3 depth-wise convolution, and 1*1 convolution, and then spliced ​​with the unchanged feature map, and finally the spliced ​​feature map is randomly shuffled by the channel and output; The downsampling convolution module includes a second channel separation module and two depth-separable convolution layers for feature extraction; the second channel separation module receives the feature map output by the previous layer, and divides the downsampled feature map into two identical branches according to the number of channels of the feature map for processing, wherein the feature map of one branch is input into the second depth-separable convolution layer, and is sequentially processed by a 3*3 depth convolution with a stride of 2 and a 1×1 convolution with a stride of 1, and then outputs the second feature map; the feature map of the other branch is input into the third depth-separable convolution layer, and is sequentially processed by a 1*1 Conv, a 3×3 depth convolution with a stride of 2, and a 1*1 convolution, and then output after channel cascade with the second feature map; The long short-term memory layer includes a forget gate, an input gate, and an output gate, which are used to receive the feature map output by the previous layer and extract the temporal features within the modality, and then convert them into hidden layer state outputs within the modality; The multimodal fusion unit receives the short-term memory layer output of each branch in the multi-stream branch, and fuses the nonlinear features between different modalities using a self-attention mechanism to obtain a multimodal fusion feature; The output unit includes a fully connected layer unit and a Softmax unit, wherein the fully connected layer unit is used to receive the multimodal fusion features and perform dimensionality reduction processing; the Softmax unit is used to receive the reduced dimensionality features output by the fully connected layer unit and perform category probability prediction through cross entropy loss.

2. The wearable gesture interaction system according to claim 1, characterized in that: The flexible sensing module includes: a flexible strain sensor based on a nanofiber substrate and a liquid metal circuit and a flexible acceleration sensor based on a nanofiber substrate and a liquid metal circuit; The flexible strain sensor based on the nanofiber substrate and the liquid metal circuit is a sandwich structure, comprising: a strain-sensitive flexible substrate, a strain-sensitive patterned circuit, and strain-sensitive package pins; wherein the strain-sensitive patterned circuit is the middle layer, and the upper and lower layers are the strain-sensitive flexible substrate respectively; The strained flexible substrate is a thermoplastic polyurethane nanofiber membrane obtained by electrospinning a thermoplastic polyurethane solution; The preparation process of the thermoplastic polyurethane solution is: The polyurethane particles were dissolved in dimethylformamide and tetrahydrofuran solution, and then stirred thoroughly with a magnetic swivel at room temperature to obtain a uniform thermoplastic polyurethane solution; The strain sensing patterned circuit is formed by printing liquid metal on the upper surface of the underlying strain flexible substrate through mask printing; The strain sensing package pins include positive and negative pins. After a rigid wire is placed in a position reserved for the patterned circuit, polyvinyl alcohol gel is dripped onto the rigid wire. After the polyvinyl alcohol gel is cured, the rigid wire is fixed to the strain sensing patterned circuit. The flexible triaxial acceleration sensor based on the nanofiber substrate and the liquid metal circuit has a sandwich structure, comprising: an accelerometer flexible substrate, an accelerometer chip, an accelerometer patterned circuit, and accelerometer package pins; the accelerometer patterned circuit and the accelerometer chip form the middle layer, and the upper and lower layers form the accelerometer flexible substrate; The accelerometer flexible substrate is a thermoplastic polyurethane nanofiber membrane obtained by electrostatically spinning a thermoplastic polyurethane solution; The accelerometer patterned circuit is formed by printing liquid metal on the upper surface of the lower flexible substrate through mask printing, and serves as a peripheral circuit for reading the accelerometer chip; The accelerometer chip is placed in a position reserved for the accelerometer patterned circuit, and polyvinyl alcohol gel is dripped onto the pins of the accelerometer chip; after the polyvinyl alcohol gel is solidified, the accelerometer chip is fixed on the patterned circuit; The accelerometer package pins include three pins: ground, serial data line, and serial clock line. After a rigid wire is placed in a position reserved for the patterned circuit, polyvinyl alcohol gel is dripped onto the rigid wire; after the polyvinyl alcohol gel is cured, the rigid wire is fixed on the accelerometer patterned circuit.

3. A wearable gesture interaction method based on multimodal flexible sensing, characterized in that: The following steps are involved: Step 1: Get gesture data: The real-time gesture signals output by the multimodal flexible sensor of different gesture actions are collected by flexible wearable devices, and the real-time gesture signals are segmented and preprocessed to obtain the Gram matrix of the gesture signal in each sensing channel. , thus forming a multimodal gesture dataset , where Stres represents the Gram matrix of the output signal of the flexible strain sensor in the multimodal flexible sensor, and ACC represents the Gram matrix of the output signal of the flexible acceleration sensor in the multimodal flexible sensor. A set of labels representing gesture categories; Step 2: Construct a deep learning network model, including: an input unit, a multi-stream branch with two parallel network branches, a multimodal fusion unit, and an output unit; wherein the number of branches of the multi-stream branch is equal to the number of sensing modalities, and each branch includes: a normal convolution layer, a second convolution layer, a third convolution layer, and a long short-term memory layer in sequence; the normal convolution layer includes: 1 convolution kernel with the number of , a 2D convolutional neural network layer with a kernel size of 3*3 and a Relu activation function layer; the second convolution layer and the third convolution layer are both composed of a basic convolution module and a downsampling convolution module; the basic convolution module includes: a first channel separation module, a first depth-separable convolution layer for feature extraction; the downsampling convolution module includes: a second channel separation module, two depth-separable convolution layers for feature extraction; the long short-term memory layer is a single-layer long short-term memory network with a hidden layer size of h and a dropout layer with a dropout probability of p; the output unit includes: a fully connected layer and a Softmax layer; Step 2.1 Hand gesture sample dataset Input units are input and divided according to sensing modes to obtain the Gram angle field map under each sensing mode; In step 2.2, the multi-stream branch receives the Gram angular field map under each sensing modality and inputs the Gram angular field map Stres into one branch. After being processed by the ordinary convolution layer, the second convolution layer, the third convolution layer and the long short-term memory layer in sequence, the final output length is The hidden layer features of The Gram angle field map ACC is input into another branch, and passes through the normal convolution layer, the second convolution layer, the third convolution layer and the long short-term memory layer in sequence. The final output length is The hidden layer features of Step 2.3 The multimodal fusion unit will be of length The hidden layer features and length are The hidden layer features of are concatenated into a new feature map H of length n, and the query Q, key value K, and weight vector S are obtained using equations (1), (2), and (3); (1) (2) (3) In formulas (1) to (3), b represents the bias term, , d represents the attention length, tanh represents the activation function; Represents the nonlinear transformation of the fully connected layer; softmax represents the softmax operation; Step 2.4 The output unit inputs the weight vector S into the fully connected layer for dimensionality reduction, and then processes it through the Softmax layer to obtain The gesture with the highest probability; Step 3: Set the network training hyperparameters including the number of sample batches, learning rate, and maximum number of epochs. A cross entropy loss function is used as the loss function, and the deep learning network model is trained using the Adam optimizer. The loss function is calculated to update the model parameters until the maximum number of iterations is reached, thereby obtaining a trained gesture recognition model for gesture recognition.

4. The wearable gesture interaction method based on multimodal flexible sensing according to claim 3 is characterized in that: The real-time gesture signal activity segment segmentation and preprocessing in step 1 include: According to the real-time acceleration values ​​of the three axes output by the flexible acceleration sensor, the collected real-time gesture signals are segmented, and the strain sensing signals and accelerometer signals of the gesture activity segment are extracted, and then gesture samples with a sampling length of L are obtained. Each gesture sample consists of the original signals of 5 strain sensing channels and 3 accelerometer channels. The original signal of any channel is recorded as Vorg={Vorg (1), Vorg (2),…,Vorg (l),…,Vorg (L)}, where Vorg (l) represents the lth original signal of any channel. The original signal Vorg is filtered, normalized and resampled to obtain a signal with a length of The single-channel signal of the gesture sample Sample={ Sample (1), Sample (2),…,Sample ( ) ,…,Sample ( )}, where Sample( ) indicates the The original signal after filtering and resampling; Normalize the single-channel signal Sample of the gesture sample so that each element Sample ( ) is between (-1, 1), and then the normalized single-channel signal Sample is converted into a Gram matrix and used as the input of the deep learning network model.

Images