Real-time gesture recognition system based on 16-way imu array and spatiotemporal feature fusion and application

Abstract

The application discloses a kind of real-time gesture recognition systems based on 16-way IMU array and space-time feature fusion, including data acquisition module, data preprocessing module, space-time feature extraction module and real-time identification display module.The IMU array is composed of 16 six-axis inertial measurement units, and 96-dimensional original motion data is continuously sent in fixed format by UDP protocol;Data preprocessing module uses Euclidean alignment and sliding window segmentation technology, window slicing is carried out according to 30 frame length, 15 frame step, and 96-dimensional data is reconstructed into 6×16×30 three-dimensional tensor.The space-time feature extraction module supports ListenNet, DARNET and three kinds of deep network structures based on multi-channel time sequence patch visual Transformer (ViT), realizes the fusion modeling of local structure feature and global time sequence dependence.Has the advantages of strong data structure retention, excellent cross-subject generalization ability, simple deployment, high real-time performance, etc., applicable to wearable interaction, robot control, sign language recognition and other scenes.

Description

Technical Field

[0001] This invention belongs to the field of wearable human-computer interaction, specifically relating to a real-time gesture recognition system and its application based on the fusion of a 16-channel IMU array and spatiotemporal features. Background Technology

[0002] With the increasing demand for applications such as virtual reality, robot interaction, and remote control, gesture recognition based on wearable devices has become an important interaction method. Currently, there are three main types of methods: those based on flexible sensors or data gloves, vision-based methods, and IMU sensor-based methods. Gesture recognition methods based on flexible sensors or data gloves can capture local finger deformations, but the system complexity is high; vision-based gesture recognition methods have high accuracy, but are susceptible to lighting interference; gesture recognition methods based on IMU sensors have advantages in terms of resistance to lighting interference and high mobility, and are the mainstream trend in wearable interaction. Although IMU-based gesture recognition methods have significant advantages in terms of resistance to lighting interference and wearability, existing technologies still have many shortcomings in system structure design and algorithm modeling, specifically in the following aspects:

[0003] First, most existing IMU gesture recognition solutions typically use a small number of IMUs or a single IMU node for data acquisition. Even if multiple IMU sensors are used, the multi-channel data is often simply spliced ​​or flattened into a one-dimensional vector during the data modeling stage. This fails to explicitly preserve the spatial topological relationship of the IMU array in the hand, making it difficult for the model to fully characterize the coordinated motion features between different fingers and joints, thus limiting the recognition accuracy and ability to distinguish complex gestures.

[0004] Secondly, there are significant differences among users in terms of hand size, wearing position, and range of motion. Existing technologies mostly rely on individualized training or simple normalization to alleviate cross-subject differences, lacking a unified and effective data alignment mechanism. This results in insufficient generalization ability of the model in cross-user and cross-scenario applications, making it difficult to meet the needs of multi-user shared systems in practical applications.

[0005] Furthermore, existing IMU gesture recognition systems often focus on offline algorithm verification, lacking system-level design for real-time data reception, abnormal data processing, sliding window slicing, and low-latency inference. In real communication environments, they are easily affected by network jitter, data packet loss, or format abnormalities, making it difficult to operate stably for a long time.

[0006] In addition, although some existing methods introduce deep learning models for temporal modeling, they are mostly limited to RNN, LSTM or one-dimensional convolutional networks, which makes it difficult to simultaneously take into account local structural feature modeling and global temporal dependency modeling, and the utilization efficiency of high-dimensional complex spatiotemporal data generated by multiple IMU arrays is not high.

[0007] Therefore, there is an urgent need for a gesture recognition system that can explicitly maintain the spatial structure of the IMU array, enhance cross-subject generalization ability, have a stable real-time processing mechanism, and integrate local structural features and global temporal features to meet the actual needs of application scenarios such as wearable interaction and real-time control. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a real-time gesture recognition system based on the fusion of a 16-channel IMU array and spatiotemporal features, which solves the problem of IMU gesture recognition mentioned in the background art of the prior art.

[0009] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0010] A real-time gesture recognition system based on fusion of 16-channel IMU array and spatiotemporal features includes a data acquisition module, which is used to acquire 96-dimensional six-axis motion data from 16 IMU sensors in a fixed format in real time and transmit it to the data preprocessing module via UDP packets.

[0011] The data preprocessing module is used to perform Euclidean alignment, normalization, sliding window slicing and tensor reconstruction on the 96-dimensional data to obtain a 6×16×30 structured gesture spatiotemporal sequence.

[0012] The spatiotemporal feature extraction module is used to extract temporal relationships and structural features from the processed spatiotemporal sequence based on a neural network.

[0013] The real-time recognition and display module is used to display the classification results output by the deep network in real time and provide gesture category images or text prompts.

[0014] The data packet format of the data acquisition module is as follows: it begins with a fixed prefix "subpackage 1 / 1," followed by 96 floating-point numbers, which correspond to the six-axis values ​​of acceleration and angular velocity of 16 IMUs.

[0015] The Euclidean alignment includes: channel-by-channel centering, covariance matrix calculation, eigenvalue decomposition, and rotation transformation of the input time series, so that the sequences collected by different subjects are projected into a common Euclidean space.

[0016] The sliding window slicing method is as follows: using a window length of 30 frames and a step size of 15 frames, overlapping time-series samples are generated from continuous IMU stream data.

[0017] The tensor reconstruction process includes: restoring each frame's 96-dimensional vector into a two-dimensional matrix by arranging it in a "6-axis × 16 sensor" configuration, and stacking it along the time dimension to form a three-dimensional tensor with a shape of (6, 16, 30).

[0018] The spatiotemporal feature extraction module includes any of the following structures:

[0019] The ListenNet model employs temporal convolution and local attention mechanisms.

[0020] The DARNet model employs deep residual structures and multi-scale feature enhancement.

[0021] A visual Transformer model based on temporal patch partitioning and multi-head self-attention mechanism; wherein the Transformer model adopts a window size of 30 frames and a patch size of 1 frame; the embedding dimension is 384-dimensional; and it contains 3 Transformer encoding layers and 6 attention heads.

[0022] The real-time recognition and display module has a category adaptive mechanism, which automatically generates category names and quantities by detecting the directory structure of the dataset.

[0023] The real-time recognition and display module performs an inference operation every 30 frames and refreshes the recognition interface immediately after the inference is completed, ensuring that the recognition delay is less than 20ms.

[0024] It also includes error detection and packet dropping mechanisms, which automatically discard the data frame and update the acquisition status when the packet prefix is ​​incorrect, the dimensions are insufficient, or the format is incorrect.

[0025] The application of a real-time gesture recognition system based on 16-channel IMU array and spatiotemporal feature fusion is applicable to scenarios including wearable interaction, robot control, and sign language recognition.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. Real-time recognition follows a process of receiving 30 frames, performing EA+ normalization, reconstructing tensors, performing deep network forward inference, obtaining classification results (such as A, B, ZH, etc.), displaying corresponding icons on the interface, and updating the state machine (such as continuous recognition / change trend). The platform supports GPU inference (such as RTX 3060), CPU inference (x86 or ARM), and ONNX / TensorRT acceleration, and can achieve an average inference latency of less than 20 milliseconds.

[0028] 2. This system can automatically read training data and automatically generate a list of class names, a mapping from class to number, and a mapping from number to icon path, so that new gesture categories can be added without programmers having to manually modify the code.

[0029] 3. The system automatically loads gesture categories through a class directory adaptive mechanism, enabling low-latency real-time prediction. This invention has advantages such as strong data structure preservation, excellent cross-subject generalization ability, simple deployment, and high real-time performance, making it suitable for wearable interaction, robot control, sign language recognition, and other scenarios. Attached Figure Description

[0030] Figure 1 This is a block diagram of the overall system structure of the present invention.

[0031] Figure 2 This is a schematic diagram of the IMU array distribution of the present invention.

[0032] Figure 3 This is a schematic diagram of the data preprocessing process of the present invention.

[0033] Figure 4 This is a schematic diagram of the tensor reconstruction of (6,16,30) in this invention.

[0034] Figure 5 This is the ViT-based spatiotemporal feature extraction structure of the present invention. Detailed Implementation

[0035] The structure and working process of the present invention will be further described below with reference to the accompanying drawings.

[0036] The purpose of this invention is to disclose a real-time gesture recognition system based on a 16-channel IMU array and spatiotemporal feature fusion, comprising a data acquisition module, a data preprocessing module, a spatiotemporal feature extraction module, and a real-time recognition and display module. The IMU array consists of 16 six-axis inertial measurement units, continuously transmitting 96-dimensional raw motion data in a fixed format via UDP protocol. The data preprocessing module employs Euclidean alignment and sliding window segmentation techniques, slicing the data into windows with a length of 30 frames and a step size of 15 frames, and reconstructing the 96-dimensional data into a 6×16×30 three-dimensional tensor to maintain the spatial layout and temporal structure of the IMU. The spatiotemporal feature extraction module supports three types of deep network structures: ListenNet, DARNET, and Visual Transformer (ViT) based on multi-channel temporal patches, achieving fusion modeling of local structural features and global temporal dependencies. The system automatically loads gesture categories through a class directory adaptive mechanism, enabling low-latency real-time prediction. This invention has the advantages of strong data structure preservation, excellent cross-subject generalization ability, simple deployment, and high real-time performance, and is suitable for wearable interaction, robot control, sign language recognition and other scenarios.

[0037] Specific embodiments, such as Figures 1 to 5 As shown:

[0038] This embodiment is based on a real-time gesture recognition system that fuses a 16-channel IMU array with spatiotemporal features, such as... Figure 1As shown, the system comprises seven main modules: a 16-channel IMU acquisition device, a data receiving and parsing module, a data preprocessing module, a tensor reconstruction module, a deep network feature fusion module, a real-time recognition module, and a class directory adaptive module. These modules can be deployed on a host computer, embedded computing devices, or mobile terminals. The data acquisition module collects motion information, while the host computer handles the calculation and display. The entire system uses UDP data packets to construct an end-to-end real-time gesture recognition process.

[0039] like Figure 2 As shown, the IMU acquisition device uses 16 independent IMU sensors, each providing 6-dimensional data, including triaxial accelerations ax, ay, az and triaxial angular velocities gx, gy, gz. Therefore, the data dimension per frame is 16 IMUs × 6 dimensions = 96 dimensions. Considering the typical kinematic structure of each finger, three IMU measurement channels are configured for each finger, plus one IMU measurement channel for the back of the hand. Taking the index finger as an example, three IMU measurements are obtained from sensors located at the distal interphalangeal (DIP) joint, proximal interphalangeal (PIP) joint, and metacarpophalangeal (MCP) joint. In this way, a total of 16 IMU measurements are obtained, which can fully capture the curvature of each finger and facilitate subsequent classification and identification.

[0040] Each frame of data acquired by the 16-channel IMU array is sent in string format: "subpackage 1 / 1,ax1,ay1,az1,gx1,gy1,gz1, ax2,ay2,..., gz16". Its key features include a fixed prefix "subpackage 1 / 1," for frame synchronization; 96 floating-point numbers separated by commas; and a frame rate that can be set from 30Hz to 200Hz. To ensure real-time performance, this embodiment employs an independent thread to continuously listen on port 5001, and utilizes three main measures: a data buffer queue, a timeout discard strategy, and a mechanism to prevent the main thread from locking up, ensuring smooth interface operation and continuous execution of the inference model. Frame format verification is enabled; frames with insufficient length, incorrect prefixes, or illegal characters are discarded, and the packet loss rate is recorded. Furthermore, the data receiving thread is independent of the model inference thread, making inference unaffected by network jitter.

[0041] like Figure 3 As shown, the preprocessing module receives continuous 96-dimensional IMU data and performs the following steps sequentially:

[0042] The first step is Euclidean Alignment (EA). Different users exhibit shifts in their movement amplitude and inertial characteristics. To improve cross-subject recognition capabilities, this invention employs an EA preprocessing method. The steps are as follows:

[0043] Matrix X (T×96) is constructed using only axial data.

[0044] Mean centralization:

[0045]

[0046] Calculate the covariance matrix:

[0047]

[0048] Eigenvalue decomposition:

[0049]

[0050] Rotational transformation:

[0051]

[0052] EA can project all user data into a unified Euclidean space, making gestures more consistent across different users.

[0053] The second step is normalization, which involves performing min-max or Z-score normalization on the data of each channel to avoid model bias caused by differences in the gain of different sensors.

[0054] The third step is sliding window slicing, where continuous data is divided into multiple time windows, each with 30 frames and a step size of 15 frames. Its advantages are that it can cover long actions, align action start points, improve model robustness, and increase the number of training samples.

[0055] like Figure 4 As shown, in this embodiment, a 96-dimensional frame of data is rearranged into a 6×16 matrix, with the frame matrix M = reshape(96,(16,6)). Then, 30 frames are stacked along the time dimension, resulting in a tensor T with dimensions of (6 channels, 16 IMU nodes, 30 time frames). Its advantage lies in simultaneously preserving the IMU array topology and temporal order, improving the temporal modeling effect of the Transformer, and enabling convolution and attention mechanisms to utilize spatial information. This structure differs from the traditional "flattened vector" and is significantly innovative.

[0056] like Figure 5 As shown, this embodiment supports three types of models: ListenNet, DARNet, and Vision Transformer (ViT), which can be selected according to latency and accuracy requirements.

[0057] The ListenNet model is a lightweight temporal convolutional network with a structure that includes multi-scale convolutional modules, local attention channel enhancement modules, and temporal pooling layers, making it suitable for low-computing devices.

[0058] The DARNet model employs deep residual structures, dilated convolutions, and multi-scale hybrid convolution kernels to extract long sequence structural features and improve the recognition ability of complex gestures.

[0059] The ViT model is the most important module. It first divides the 30 frames of data into patches, each patch having a dimension of 6 × 16 = 96 dimensions, which are then projected onto a 384-dimensional vector.

[0060]

[0061] Next, positional encoding and multi-head self-attention mechanism are added to capture global temporal dependencies. Then, the model passes through a 3-layer Transformer Encoder and finally through mean pooling and a fully connected classifier to output the gesture recognition result. This model can learn complex temporal-spatial coupling structures, which is the core innovation of the system.

[0062] This embodiment designs a graphical interface for real-time display of gesture recognition results. Real-time recognition follows a process of receiving 30 frames, performing EA+ normalization, reconstructing the tensor, performing deep network forward inference, obtaining classification results (such as A, B, ZH, etc.), displaying the corresponding icon on the interface, and updating the state machine (such as continuous recognition / changing trend). The platform supports GPU inference (such as RTX 3060), CPU inference (x86 or ARM), and ONNX / TensorRT acceleration, achieving an average inference latency of less than 20 milliseconds.

[0063] To facilitate expansion, the system in this embodiment has the function of automatically reading training data and automatically generating a list of class names, a mapping of categories to numbers, and a mapping of numbers to icon paths, so that new gesture categories can be added without programmers having to manually modify the code.

[0064] To further verify the effectiveness of this scheme, this embodiment uses experiments to compare and illustrate its recognition accuracy, cross-subject generalization ability, and real-time performance. The experimental data consisted of 96-dimensional six-axis temporal data acquired using a 16-channel IMU array. The data was sliced ​​with a window length of 30 frames and a stride of 15 frames, and reconstructed into a (6,16,30) three-dimensional tensor before being input into a spatiotemporal feature extraction network (e.g., ListenNet, DARNet, or ViT model). The ViT model could employ a 3-layer Transformer encoder, 6 attention heads, and an embedding dimension of 384. The experimental environment was a GPU inference platform (RTX 3050).

[0065] (1) Experimental setup: The sampling frame rate was set to 60Hz, with a window of 30 frames corresponding to a 0.50s action segment and a step size of 15 frames corresponding to a 0.25s update cycle; the number of gesture categories was 22; the dataset was divided into "training / validation / testing = 64:16:20" for same-subject evaluation and "leave-one-subject-out" cross-subject evaluation was conducted. Evaluation metrics included Top-1 accuracy (Acc), macro-average F1 (Macro-F1), and single-window end-to-end inference latency (including preprocessing and network forward pass).

[0066] (2) Comparison scheme setup: To highlight the comprehensive effect of this scheme of "Euclidean alignment + structure-preserving tensor reconstruction + spatiotemporal feature fusion", the following comparison scheme is set up:

[0067] A) Existing solution 1: Do not perform Euclidean alignment, only normalize, flatten the 96×30 sequence into a vector and input it into LSTM / 1D-CNN (do not explicitly retain the topology of the 16-channel IMU).

[0068] B) Existing Solution 2: Retain the ViT backbone, but cancel tensor reconstruction and replace it with "flattened sequence patch" (weakening spatial structure constraints).

[0069] C) Ablation scheme: Same network structure as this scheme, but remove the Euclidean alignment (EA) step and only retain normalization and sliding window;

[0070] D) This solution: Execute EA + normalization + (6,16,30) tensor reconstruction + spatiotemporal feature fusion network (one of ListenNet / DARNet / ViT).

[0071] (3) Simulation results: Under the same data partitioning and training rounds, the accuracy comparison results are shown in Table 1; further statistical analysis of end-to-end real-time performance indicators is shown in Table 2. As can be seen from Table 1, the Acc and Macro-F1 of this scheme are significantly better than existing schemes that do not retain the topology or do not perform Euclidean alignment under cross-subject evaluation; as can be seen from Table 2, this scheme can meet the real-time requirement of single-window inference latency of less than 20ms on both GPU / CPU platforms.

[0072] Table 1 Comparison of Recognition Performance

[0073]

[0074] Table 2 Real-time performance comparison (single window end-to-end latency)

[0075]

[0076] Furthermore, in a robust simulation incorporating network jitter / packet loss, 5% of UDP data packets can be randomly dropped as interference. After adopting the "error detection and dropping mechanism + sliding window overlap" strategy of this scheme, the cross-subject Acc can still be maintained at around 92.3% (Macro-F1=0.919); while the existing scheme 1 drops to 83.5% (Macro-F1=0.827) under the same conditions, indicating that this scheme still has better stability and availability when there is slight packet loss in the real communication link.

[0077] It should be understood that the above experimental parameters, comparison schemes and experimental data are only used to illustrate the technical effects that this solution can achieve. This solution is not limited to the specific implementation methods described above. Any equivalent substitutions, modifications or alterations made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of this solution shall fall within the protection scope of this solution.

[0078] This solution is not limited to the specific embodiments described above. Devices and structures not described in detail herein should be understood as being implemented in a manner common to the art. Any person skilled in the art can make many possible variations and modifications to this solution, or modify it into equivalent embodiments, without departing from the scope of this solution, using the methods and techniques disclosed above. This does not affect the substantive content of this solution. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this solution, without departing from its scope, still fall within the protection scope of this solution.

Claims

1. A real-time gesture recognition system based on 16-channel IMU array and spatiotemporal feature fusion, characterized in that: It includes a data acquisition module, which is used to acquire 96-dimensional six-axis motion data from 16 IMU sensors in a fixed format in real time and transmit it to the data preprocessing module via UDP packets; The data preprocessing module is used to perform Euclidean alignment, normalization, sliding window slicing and tensor reconstruction on the 96-dimensional data to obtain a 6×16×30 structured gesture spatiotemporal sequence. The spatiotemporal feature extraction module is used to extract temporal relationships and structural features from the processed spatiotemporal sequence based on a neural network. The real-time recognition and display module is used to display the classification results output by the deep network in real time and provide gesture category images or text prompts.

2. The real-time gesture recognition system based on 16-channel IMU array and spatiotemporal feature fusion as described in claim 1, characterized in that: The data packet format of the data acquisition module is as follows: it begins with a fixed prefix "subpackage 1 / 1," followed by 96 floating-point numbers, which correspond to the six-axis values ​​of acceleration and angular velocity of 16 IMUs.

3. The real-time gesture recognition system based on 16-channel IMU array and spatiotemporal feature fusion as described in claim 1, characterized in that: The Euclidean alignment includes: channel-by-channel centering, covariance matrix calculation, eigenvalue decomposition, and rotation transformation of the input time series, so that the sequences collected by different subjects are projected into a common Euclidean space.

4. The real-time gesture recognition system based on 16-channel IMU array and spatiotemporal feature fusion as described in claim 1, characterized in that: The sliding window slicing method is as follows: using a window length of 30 frames and a step size of 15 frames, overlapping time-series samples are generated from continuous IMU stream data.

5. The real-time gesture recognition system based on 16-channel IMU array and spatiotemporal feature fusion as described in claim 1, characterized in that: The tensor reconstruction process includes: restoring each frame's 96-dimensional vector into a two-dimensional matrix by arranging it in a "6-axis × 16-sensor" configuration, and stacking it along the time dimension to form a three-dimensional tensor with a shape of (6, 16, 30).

6. The real-time gesture recognition system based on 16-channel IMU array and spatiotemporal feature fusion as described in claim 1, characterized in that: The spatiotemporal feature extraction module includes any of the following structures: The ListenNet model employs temporal convolution and local attention mechanisms. The DARNet model employs deep residual structures and multi-scale feature enhancement. A visual Transformer model based on temporal patch partitioning and multi-head self-attention mechanism; wherein the Transformer model adopts a window size of 30 frames and a patch size of 1 frame; the embedding dimension is 384-dimensional; and it contains 3 Transformer encoding layers and 6 attention heads.

7. The real-time gesture recognition system based on 16-channel IMU array and spatiotemporal feature fusion as described in claim 1, characterized in that: The real-time recognition and display module has a category adaptive mechanism, which automatically generates category names and quantities by detecting the directory structure of the dataset.

8. The real-time gesture recognition system based on 16-channel IMU array and spatiotemporal feature fusion according to claim 1, characterized in that: The real-time recognition and display module performs an inference operation every 30 frames and refreshes the recognition interface immediately after the inference is completed, ensuring that the recognition delay is less than 20ms.

9. The real-time gesture recognition system based on 16-channel IMU array and spatiotemporal feature fusion according to claim 1, characterized in that: It also includes error detection and packet dropping mechanisms, which automatically discard the data frame and update the acquisition status when the packet prefix is ​​incorrect, the dimensions are insufficient, or the format is incorrect.

10. The application of a real-time gesture recognition system based on 16-channel IMU array and spatiotemporal feature fusion, characterized in that: This system is suitable for scenarios including wearable interaction, robot control, and sign language recognition.

Images