Wearable device for 3D position tracking and gesture recognition
The smart ring with IMU and machine learning algorithms addresses limitations of conventional systems by providing precise 3D position tracking and gesture recognition, enhancing user mobility and interaction in spatial computing environments.
Patent Information
- Application Number
- KR1020267035865
- Authority / Receiving Office
- KR · KR
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-03-28
- Filing Date
- 2025-03-20
- Publication Date
- 2026-11-09
AI Technical Summary
Conventional systems for tracking a user's three-dimensional position and recognizing gestures are limited by environmental factors and require external equipment, restricting user mobility and versatility.
A wearable device, such as a smart ring, equipped with an inertial measurement unit (IMU) and machine learning algorithms for precise 3D position tracking and gesture recognition, utilizing a two-layer LSTM model for error compensation and gesture prediction.
Enables accurate, real-time gesture recognition and position tracking with high accuracy and robustness, allowing intuitive interaction in spatial computing environments.
Smart Images

Figure P1020267035865_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a wearable device, and specifically to a device for tracking a user's three-dimensional position and recognizing gestures. Background Technology
[0002] The field of wearable technology has made significant progress in recent years and is increasingly focusing on enhancing user interaction and immersion in various spatial computing applications such as virtual reality (VR), augmented reality (AR), and mixed reality (MR). A key component of these technologies is the ability to accurately track a user's three-dimensional (3D) position and recognize user gestures in real time.
[0003] Conventional systems often rely on external cameras, sensors, or markers to track movement and gestures. However, these systems can be limited by environmental factors such as lighting conditions or occlusion. Furthermore, the requirement for external equipment can restrict user mobility and limit the applicability of the technology in different environments.
[0004] Therefore, there is a need for wearable devices that overcome these limitations by providing precise 3D position tracking and gesture recognition capabilities. Such devices will provide greater freedom of movement and versatility, enabling users to interact with digital content in a more natural and intuitive way. The problem to be solved
[0005] The objective of the present invention is to provide a smart wearable capable of tracking a user's three-dimensional position and recognizing gestures.
[0006] Another objective of the present invention is to provide a smart wearable capable of predicting gestures and movements.
[0007] Another objective of the present invention is to provide a smart wearable capable of interpreting gestures for various operations within a virtual reality or augmented reality environment.
[0008] Another objective of the present invention is to provide a smart wearable capable of using machine learning techniques to interpret a user's gestures and three-dimensional position. means of solving the problem
[0009] This summary is provided to introduce aspects related to the present invention concerning protective casings for rings, which are further described in the detailed description below. This summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to determine or limit the scope of the claimed subject matter.
[0010] In one embodiment of the present disclosure, a wearable device includes an inertial measurement unit (IMU) for recording motion data of a user wearing the wearable device. The wearable device further includes a central processing unit (CPU) configured to implement a machine learning (ML) data preprocessing algorithm module for processing the motion data. The motion data is processed to perform at least one of removing outliers that exceed the physical limits of human movement; synchronizing timestamps of accelerometers and gyroscopes; normalizing acceleration and angular velocity values; performing feature extraction on the motion data; and creating a fixed-size sliding window for gesture recognition. The wearable device further includes an ML-based position tracking and gesture recognition module for estimating the user's gesture along with a confidence score of estimation based on the classification of the motion data.
[0011] In one embodiment, the ML-based position tracking and gesture recognition module includes an ML model trained using training data generated through time warping, and the training data is provided using a curriculum learning method.
[0012] In one embodiment, the ML model determines the user's zero-velocity point from the motion data.
[0013] In one embodiment, the ML model implements a fixed threshold zero velocity detector for collecting motion data based on the identification of the user's movement. The user's movement is identified through the processing of angular velocity measured by the gyroscope of the IMU, acceleration measured by the accelerometer, and hand orientation.
[0014] In one embodiment, the ML model is one of logistic regression (LR), support vector machine (SVM), random forest (RF), simple LSTM, or 6-layer LSTM (6-LSTM).
[0015] In one embodiment, the ML model is a two-layer LSTM in which each layer performs integration to account for errors present in the linear acceleration signal and compensates for said errors to predict the displacement estimate.
[0016] In one embodiment, the two-layer LSTM comprises the following:
[0017] A first LSTM layer that determines a velocity estimate by performing a first integral of raw acceleration data;
[0018] An error compensation layer that processes the above-mentioned speed estimate, gyroscope data, and zero-velocity detection features to generate corrected values of the above-mentioned speed estimate, the above-mentioned gyroscope data, and the above-mentioned zero-velocity detection; and
[0019] A second LSTM layer that determines the user's position estimate by performing a second integral using the above speed estimate, the above gyroscope data, and the corrected value of the above zero velocity detection.
[0020] In one embodiment, the wearable device is a smart ring.
[0021] In one embodiment, the wearable device includes a Bluetooth (BLE) communication unit for communicating with another wearable device that records the user's motion data.
[0022] In one embodiment, a method for estimating a user's position includes the step of providing raw acceleration data acquired by an inertial measurement unit (IMU) of a wearable device to a first LSTM layer of a two-layer LSTM model. The method further includes the step of performing a first integral of the raw acceleration data to determine a velocity estimate value by the first LSTM layer. The method further includes the step of providing the velocity estimate value, the gyroscope data, and zero velocity detection features to an error compensation layer to obtain the velocity estimate value, the gyroscope data, and the corrected value of the zero velocity detection. The method further includes the step of providing the velocity estimate value, the gyroscope data, and the corrected value of the zero velocity detection to a second LSTM layer to perform a second integral to determine the user's position estimate value.
[0023] In one embodiment, the zero velocity detection feature includes acceleration magnitude variance, angular velocity magnitude, orientation change rate, and window-based statistical feature.
[0024] In one embodiment, the first LSTM layer and the second LSTM layer are enhanced with a peephole connection to introduce a direct connection between the cell state and the gate.
[0025] In one embodiment, bidirectional processing is implemented in the above 2-layer LSTM model to utilize the temporal context.
[0026] In one embodiment, a skip connection is provided between the first LSTM layer and the second LSTM layer so as to bypass one of the first LSTM layer and the second LSTM layer. Brief explanation of the drawing
[0027] The accompanying drawings constitute part of the description and are used to provide further understanding of the present invention. These accompanying drawings illustrate embodiments of the present invention used to explain the principles of the present invention. Embodiments are illustrated in the drawings of the accompanying drawings as examples rather than limitations, and the same reference numerals indicate similar elements. It should be noted that in the present invention, references to “one” or “one” embodiment do not necessarily refer to the same embodiment, but mean at least one. In the drawings: FIG. 1 illustrates a block diagram of a device capable of tracking a user's three-dimensional position and recognizing gestures according to an embodiment of the present invention; FIG. 2 illustrates a block diagram of a double integral technique used to determine a position according to one embodiment of the present invention. A more complete understanding of the present invention and its embodiments can be obtained by referring to the following description and the accompanying drawings. Specific details for implementing the invention
[0028] The detailed description provided below in connection with the accompanying drawings is intended to describe various embodiments of the present invention and is not intended to represent the only embodiment in which the present invention may be practiced. Each embodiment described in this disclosure is provided merely as an example or illustration of the present invention and should not be interpreted as being more preferable or advantageous than other embodiments. The detailed description includes specific details for the purpose of providing a sufficient understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without such specific details.
[0029] As used herein, the singular forms “one (a),” “an,” and “the” are intended to include the plural forms unless otherwise explicitly stated. It will be further understood that when used herein, the terms “include,” “comprises,” “including,” and / or “comprising” specify the presence of a specified feature, integer, step, action, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof. When one element is referred to as being “connected” or “combined” with another element, it will be understood that this may be directly connected or combined with the other element, or that an intervening element may exist. Additionally, as used herein, “connection” or “combination” may include being operablely connected or combined. As used herein, the term “and / or” includes any combination and arrangement of one or more of the related enumerated items.
[0030] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as generally understood by a person skilled in the art to which this disclosure pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with that meaning in the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense unless explicitly defined so in this specification.
[0031] The terms “zero-velocity viewpoint,” “zero-velocity detector,” and “motion detection” may be used interchangeably throughout the context.
[0032] The proposed invention relates to a wearable device capable of tracking a user's three-dimensional position and recognizing gestures. The wearable device may include a smart watch, a smart band, or an electronic ring. Although details below are provided with reference to a smart ring merely for illustrative purposes, it should be understood that the invention can be sufficiently implemented in a similar manner using any other wearable device such as those listed above.
[0033] FIG. 1 illustrates a block diagram of a device (100), such as a smart ring (102), capable of three-dimensional position tracking and gesture recognition of a user, according to an embodiment of the present invention. The device (100) includes a wearable device such as a smart ring (102) and a user device (104). The smart ring (102) may be manufactured using a hypoallergenic material so that the user can wear it comfortably and continuously. The smart ring (102) may include an inertial measurement unit (IMU) (106), a Bluetooth (BLE) communication unit (108), a central processing unit (CPU) (110), a battery and charging unit / circuit (112), an ML data preprocessing algorithm module (118), an ML-based position tracking and gesture recognition module (120), an ML model (122), and a pressure touch part (114). The IMU (106) may include a gyroscope and an accelerometer. Additionally, the CPU (110) can exist and operate with a machine learning (ML) coprocessor.
[0034] The user device (104) may include a software database management system (116), an ML data preprocessing algorithm module (128), an ML-based location tracking and gesture recognition module (130), an ML model (132), a Bluetooth (BLE) communication module (124), and a raw location tracking module (126). The user device (104) may be a smartphone, tablet, PC, online or offline server, or any other device within a spatial computing environment, a virtual environment, or an augmented reality environment, but is not limited thereto. The ML data preprocessing algorithm module (128), the ML-based location tracking and gesture recognition module (130), and the ML model (132) of the user device (104) may be similar to the ML data preprocessing algorithm module (118), the ML-based location tracking and gesture recognition module (120), and the ML model (122) of the smart ring (102).
[0035] A method for tracking a user's three-dimensional position and recognizing gestures is described below. The user wears one smart ring (102) on each hand to increase the flexibility and range of motion of the gestures. Each smart ring (102) includes an IMU (106) that can be used to continuously record six-axis motion data and provide the motion data to the CPU (110). The motion data may include accelerometer data (ax, ay, az) in m / s², gyroscope data (gx, gy, gz) in rad / s, and timestamps (t) in milliseconds. The motion data may also be stored in the memory of the smart ring (102). Additionally, two smart rings (102) are connected to each other via a BLE communication unit (108) so that motion data can be monitored together or independently of each other based on requirements and user preferences. Both smart rings (102) are similar to the smart ring (102) of the device (100) and are used interchangeably in the following description. The smart ring (102) transmits motion data to the user device (104) in real time, at a predefined frequency, or when motion data is requested by the user device (104). Additionally, the raw position tracking module (126) can determine the user's position within a three-dimensional environment based on 6-axis motion data measured by the IMU (106). The raw position tracking module (126) tracks the user's position in real time to achieve precise interaction in spatial computing applications. A portion of the 6-axis motion data collected by the IMU (106) is provided below.
[0036] timestamp,ax,ay,az,gx,gy,gz
[0037] 1000,-0.245,9.812,-0.131,0.012,-0.008,0.003
[0038] 1010,-0.251,9.808,-0.128,0.015,-0.007,0.002
[0039] 1020,-0.892,9.621,-0.445,1.234,0.892,-0.341
[0040] Additionally, the CPU (110) of the smart ring (102) processes motion data using the ML data preprocessing algorithm module (118) of the smart ring (102). Specifically, the ML data preprocessing algorithm module (118) can format the motion data, refine it, and arrange it into a suitable format for further processing. Since the smart ring (102) has limited processing power, the motion data can be processed by the smart ring (102) alone or in cooperation with the user device (104). Data refinement may involve removing sensor noise using a low-pass filter, processing missing values through interpolation, and removing outliers that exceed the physical limits of human movement. Data formatting may include synchronizing the timestamps of the accelerometer and gyroscope, normalizing acceleration and angular velocity values, and creating a fixed-size sliding window for gesture recognition. Additionally, the ML data preprocessing algorithm module (118) can perform feature extraction such as calculating the magnitude of acceleration and angular velocity, calculating hand orientation using gyroscope data, and extracting statistical features, e.g., mean, variance and peak-to-peak values. Below is provided a portion of the preprocessed data generated by the ML data preprocessing algorithm module (118).
[0041] timestamp, acc_mag, gyro_mag, orientation_x, orientation_y, orientation_z, window_features
[0042] 1000,9.815,0.015,0.025,0.989,-0.013,[0.245,0.892,0.341]
[0043] 1010,9.812,0.017,0.026,0.988,-0.012,[0.251,0.895,0.344]
[0044] 1020,9.678,1.563,0.092,0.982,-0.046,[0.892,1.234,0.892]
[0045] Window features may include measures such as average acceleration across the window, peak-to-peak variation, number of changes in direction, variance of motion, and rate of change of acceleration or rotation.
[0046] Next, the ML data preprocessing algorithm module (118) can provide motion data obtained through preprocessing to the ML-based position tracking and gesture recognition module (120). The ML-based position tracking and gesture recognition module (120) may include an ML model (122). The ML model (122) may be a neural network classifier capable of predicting the user's 3D position and recognizing gestures by processing motion data. Additional details regarding the ML model (122) are provided below.
[0047] Now, details regarding the ML model (122) are provided. First, smart rings (102) worn on both hands of the user are used to record and store motion data. Motion data can be stored and relayed to a user device (104), such as a server, upon receiving a request via a Bluetooth (BLE) communication unit (108). The smart rings (102) can be worn continuously by the user for several days to record motion data frames for multiple daily activities. The user's stopping point serves as the basis for determining the user's "zero-velocity" point. This process aims to encompass all types of zero-velocity and motion movements representing the user, such as being in a stationary state, stopping movement, or interrupting movement midway. Once a sufficient amount of data regarding the "zero-velocity" point has been recorded, multiple people wear the smart rings (102) for the purpose of collecting labeled gesture data. For each gesture class supported by the ML model (122), the gesture is repeated multiple times by the wearer while data for that class is recorded. Once sufficient data is collected for one class, the same process is repeated for the next gesture class. In one implementation, 100 samples may be collected per gesture class. Gestures may include, but are not limited to, snaps, waves, shakes, upward flicks, downward flicks, and rotations. Essentially, any common gesture for UI manipulation may be recognized.
[0048] Data augmentation through rotation and time warping may be performed to generate training data used to train an ML model (122). Data augmentation through time warping is used to increase the diversity of the data by applying transformations while maintaining the original structure of the information. Transformations are typically based on changing the timing or speed of motion data collected by a smart ring (102). Time warping involves shifting, stretching, or compressing the time axis of a time series, thereby creating a transformation of the temporal features. Time warping helps the ML model (122) to be more robust against timing shifts in gesture analysis, where the speed at which an event occurs may change but the event itself remains the same.
[0049] The ML model (122) is trained using a curriculum learning method. Curriculum learning involves introducing samples of training data in order of increasing complexity. By providing training data in this manner, the ML model (122) can learn and generalize well. Additionally, early termination is performed based on validation loss, that is, the training of the ML model (122) is stopped at the inflection point where the validation loss is lowest. Furthermore, learning rate scheduling using warm-up is implemented, that is, a warm-up period is provided in which the learning rate of the ML model (122) gradually increases from an initial value to a target value over a specific number of epochs.
[0050] Additionally, the ML model (122) determines when the IMU (106) operates to predict translational and rotational motions performed by the user. The user's motion is acquired by the gyroscope and accelerometer of the IMU (106). The ML model (122) may implement a fixed threshold zero velocity detector that determines whether the IMU (106) is undergoing movement or is in a stationary state, and evaluates the presence of movement using the angular velocity measured by the gyroscope of the IMU (106), the acceleration measured by the accelerometer, and the hand orientation. When movement is detected, the movement data is classified by a classifier into one of the set of supported gesture classes, along with a confidence score for prediction / estimation. If the predicted class has a confidence score greater than the set threshold, the movement that occurred is predicted to belong to the gesture class predicted by the ML model (122). It should be noted that the zero velocity point means when the hand is not moving and nothing needs to be predicted. The ML model (122) may be a logistic regression (LR), support vector machine (SVM), random forest (RF), simple LSTM, or 6-layer LSTM (6-LSTM), but is not limited thereto.
[0051] Simple integration of acceleration to derive velocity does not work well for actual IMU data due to sensor noise and bias errors that accumulate over time. For example, even a small bias error of 0.01 m / s² can result in a velocity drift of 0.6 m / s in just one minute. Therefore, the ML model (122) uses a double integration technique to calculate the position change that occurs during the period of movement. Specifically, the ML model (122) includes a two-layer LSTM that performs an integration in which each layer considers the error present in the linear acceleration signal and compensates for said error to predict a better displacement estimate. This approach not only mitigates the effects of sensor noise and bias but also improves accuracy and robustness in position tracking and gesture recognition. In another embodiment, the ML model (122) can use an LSTM-based neural network to simultaneously predict zero velocity detection and displacement estimates for the period of movement.
[0052] FIG. 2 illustrates a block diagram of a double integral technique used to determine position according to an embodiment of the present invention. Raw acceleration data (202) is provided to an ML model (122) having a two-layer LSTM architecture. Specifically, the raw acceleration data (202) is provided to a first LSTM layer (204). The first LSTM layer (204) performs a first integral (206) of the raw acceleration data (202) to determine a velocity estimate (208). The velocity estimate (208) is provided to an error compensation layer (214) along with gyroscope data (210) and zero velocity detection features (212). The zero velocity detection features (212) include acceleration magnitude variance, angular velocity magnitude, orientation change rate, and window-based statistical features. The acceleration magnitude variance indicates the stability of the motion. The angular velocity magnitude identifies a rotational stop state. The orientation change rate helps monitor minute movements of the hand orientation. Window-based statistical features include measures such as variance, mean, and the number of changes in direction.
[0053] The error compensation layer (214) provides the corrected values of the velocity estimate (208), gyroscope data (210), and zero velocity detection feature (212) to the second LSTM layer (216). The second LSTM layer (216) determines the position estimate (220) by performing a second integral (218) using the corrected values of the velocity estimate (208), gyroscope data (210), and zero velocity detection feature (212).
[0054] In some embodiments, specific customizations are made to the ML model (122). One customization made to the ML model (122) involves enhancing the LSTM layers (204, 216) with peephole connections. Peephole connections introduce a direct connection between the cell state and the gates (forget gate, input gate, and output gate), allowing the gates to directly examine and use the cell state while making decisions. This allows the forget gate, input gate, and output gate to [remove] the hidden state (h) at the current time step. t -1 ) and input(x t ) as well as the cell state from the previous time step (c t -1 It means that it can also be accessed.
[0055] Another customization made to the ML model (122) includes implementing bidirectional processing to utilize temporal context. Bidirectional processing helps to utilize temporal context from both past and future time steps within the sequence. This approach enhances the ability of the ML model (122) to obtain information from the entire sequence, not just from the past.
[0056] Another customization made to the ML model (122) includes implementing an attention mechanism to focus on relevant motion segments. The attention mechanism enables the LSTM layers (204, 216) to focus on the most relevant parts of the input data. Based on relevance, the parts of the input data that should be given more weight (or focus) are dynamically adjusted. Another customization made to the ML model (122) includes providing residual connections between the LSTM layers (204, 216). Residual connections (also called skip connections) between the LSTM layers (204, 216) allow one or more layers to be bypassed and, by adding the original input of the layer directly to the output (location estimate (220)), solve problems such as vanishing gradients and promote better information flow.
[0057] An ML model (122) implemented using the methodology described above was also tested. For gesture recognition, 95% accuracy was observed, and for classification, a latency of less than 50 ms and a false positive rate of less than 1% were observed. Additionally, for position tracking, a displacement error of less than 2 cm over 10 seconds, an angle error of less than 2° over 10 seconds, and a drift of less than 1 cm / min were observed.
[0058] In some embodiments, the ML model (122) is quantized before being loaded into the smart ring (102). Quantization involves preparing a less complex or smaller version of the ML model (122). Quantization significantly reduces memory usage and improves the inference speed of the ML model (122) without causing a significant decrease in accuracy. Additionally, pruning of the ML model (122) may be performed before being loaded into the smart ring (102). Pruning involves removing non-important parameters from the ML model (122) to reduce size and enable more efficient model inference. Additionally, layer fusion may be performed for optimized inference, that is, multiple layers within the ML model (122) may be fused to save inference time and reduce memory consumption.
[0059] In some embodiments, the smart ring (102) may include a pressure touch area (114) which may be an area capable of distinguishing different levels of force applied to its surface. In some embodiments, the pressure touch area (114) allows additional functions or commands to be executed depending on how hard the user presses the pressure touch area (114). The user may use the pressure touch area (114) for quick actions, shortcut access, or enhanced game control in a three-dimensional spatial environment. When activated, the pressure touch area (114) may allow two types of input, primarily gestures and force, for spatial computing applications. Additionally, the pressure touch area (114) may be customized to provide precise inputs tailored to the user device (104) (for example, pressing the pressure touch area (114) for 2 seconds may be customized to provide an input corresponding to pausing a video that is playing). In one embodiment, the user may use the pressure touch area (114) in combination with gestures for better control over spatial computing applications.
[0060] One of the technical advantages of the proposed invention is to provide a wearable device capable of predicting a user's gestures in real time. The proposed invention can provide interpreted gestures as input to the user device (104) for various actions within a virtual reality or augmented reality environment. By predicting gestures and movements using a machine learning algorithm / model, the proposed invention enables intuitive and precise control in a spatial computing environment. Additionally, the proposed invention provides a pressure touch area (114) for direct interaction in a virtual or augmented reality environment.
[0061] One application of the proposed invention is realized by providing real-time gestures as input for performing various actions in spatial computing. The proposed invention can be implemented in video games to enable more natural interaction with characters using minimal hardware. The proposed invention can be implemented in the defense sector to minimize direct intervention by soldiers in combat by controlling unmanned aerial vehicles (UAVs) and unmanned artillery using gestures.
[0062] A method for tracking a user's three-dimensional position and recognizing gestures is described. The method includes the step of collecting motion data using a 6-axis IMU (106). The 6-axis IMU (106) provides acceleration and angular velocity along three axes (x, y, z). The IMU (106) consists of a 3-axis accelerometer and a 3-axis gyroscope, which provide two different measurements along each of the three axes, for a total of six measurements. Additionally, the IMU (106) is embedded in a smart ring (102), and the user typically wears the smart ring (102) on both hands to independently and simultaneously collect motion data from both hands accurately. The IMU (106) measures and reports the raw or filtered angular velocity and specific force or acceleration experienced by the user. Additionally, the measured data is transmitted in real time to the user device (104).
[0063] Additionally, the method includes the step of predicting the user's gestures and movements by analyzing motion data from the IMU (106) using a machine learning algorithm. The ML model (122) determines multiple zero-velocity points by determining whether the velocity of the IMU (106) is zero or whether the user is performing any motion. Additionally, when motion is detected by the smart ring (102), the ML model (122) predicts the user's translational motion and rotational motion across all axes into various gesture classes. The ML model (122) may use a double integral technique to minimize the error in predicting the user's movement estimates. The method includes the step of receiving input from the pressure touch area (114) of the smart ring (102). The pressure touch area (114) acts as a touchscreen and provides input in the form of pressure applied to the area and the duration of the touch. Input from the pressure touch area (114) can act as additional input provided directly to a spatial computing application.
[0064] In one embodiment, an ML model (132) may be implemented in the user device (104) to process motion data to interpret the user's gestures. The user device (104) may predict the user's gestures based on motion data independently of the smart ring (102). The user device (104) may use motion data from the smart ring (102) to implement its own gesture classes tailored to broader and more complex applications or specific use cases. In another embodiment, the user device (104) may predict the user's gestures and then verify the prediction made by the smart ring (102) by comparing them with the prediction of the smart ring (102). Additionally, the user device (104) may implement a prediction system that may not include an ML model or an ML model.
[0065] In an exemplary embodiment, a user wears a smart ring (102) on each hand that includes an IMU (106) that monitors hand motion using an accelerometer and a gyroscope. Subsequently, the smart ring (102) supplies motion data in real time to an ML model (122) and a user device (104). Subsequently, the ML model (122) determines the user's zero-velocity time points. Assuming the user raises their right hand, the ML model (122) determines, along with timestamps and orientations, the moment immediately after the user begins raising the hand and the moment immediately after the user stops raising the hand. Additionally, the ML model (122) determines the gesture class of the motion event using the hand's rotational motion and translational motion, timestamps, and orientations. In another embodiment, the user device (104) may implement an ML model (132) to process the user's motion data corresponding to a new gesture class or to verify the accuracy of the gesture predicted by the smart ring (102). The user can also interact with the pressure touch area (114) of the smart ring.
[0066] The above method further includes the step of transmitting predicted gesture data for a spatial computing application. The predicted gesture can be used in various applications, such as interacting in a mixed reality environment, providing input to a computer, or playing an interactive game.
Claims
Claim 1 A wearable device comprising: an inertial measurement unit (IMU) (106) for recording motion data of a user wearing the wearable device; and a central processing unit (CPU) (110), wherein the central processing unit (CPU) (110) is configured to process the motion data to perform at least one of: removing outliers exceeding the physical limits of human movement; synchronizing timestamps of an accelerometer and a gyroscope; normalizing acceleration and angular velocity values; performing feature extraction on the motion data; and creating a fixed-size sliding window for gesture recognition; and a machine learning (ML) data preprocessing algorithm module (118) for processing the motion data to perform at least one of the following: removing outliers exceeding the physical limits of human movement; synchronizing timestamps of an accelerometer and a gyroscope; normalizing acceleration and angular velocity values; performing feature extraction on the motion data; and creating a fixed-size sliding window for gesture recognition; and an ML-based position tracking and gesture recognition module (120) for estimating the user's gesture along with a confidence score of estimation based on the classification of the motion data. Claim 2 In claim 1, the ML-based position tracking and gesture recognition module (120) includes an ML model (122) trained using training data generated through time warping, and the training data is provided using a curriculum learning method, a wearable device. Claim 3 In paragraph 2, the ML model (122) determines the user's zero velocity point from the motion data, a wearable device. Claim 4 In paragraph 2, the ML model (122) implements a fixed threshold zero velocity detector for collecting motion data based on the identification of the user's movement, and the user's movement is identified through the processing of angular velocity measured by the gyroscope of the IMU (106), acceleration measured by the accelerometer, and hand orientation. Claim 5 In paragraph 2, the ML model (122) is one of logistic regression (LR), support vector machine (SVM), random forest (RF), simple LSTM, or 6-layer LSTM (6-LSTM), a wearable device. Claim 6 In paragraph 2, the ML model (122) is a 2-layer LSTM in which each layer performs integration to account for errors present in the linear acceleration signal and compensates for said errors to predict the displacement estimate, in a wearable device. Claim 7 A wearable device according to claim 6, wherein the two-layer LSTM comprises: a first LSTM layer (204) that determines a velocity estimate (208) by performing a first integral (206) of raw acceleration data (202); an error compensation layer (214) that processes the velocity estimate (208), gyroscope data (210), and zero velocity detection feature (212) to generate an error-free value of the velocity estimate (208), the gyroscope data (210), and the zero velocity detection (212); and a second LSTM layer (216) that determines a user position estimate (220) by performing a second integral (218) using the error-free value of the velocity estimate (208), the gyroscope data (210), and the zero velocity detection (212). Claim 8 In claim 1, the wearable device is a smart ring (102). Claim 9 A wearable device according to claim 8, further comprising a Bluetooth (BLE) communication unit (108) for communicating with another wearable device that records motion data of the user. Claim 10 A method for estimating the position of a user, comprising: providing raw acceleration data (202) acquired by an inertial measurement unit (IMU) (106) of a wearable device to a first LSTM layer (204) of a two-layer LSTM model; determining a velocity estimate value (208) by performing a first integral (206) of the raw acceleration data (202) by the first LSTM layer (204); and providing the velocity estimate value (208), gyroscope data (210), and zero velocity detection feature (212) to an error compensation layer (214) to obtain an error-free value of the velocity estimate value (208), the gyroscope data (210), and the zero velocity detection (212). A method for estimating a user's position, comprising the step of providing a second LSTM layer (216) with an error-free value of the velocity estimate (208), the gyroscope data (210), and the zero velocity detection (212) to perform a second integral (218) for determining the user's position estimate (220). Claim 11 In claim 10, the above-mentioned zero velocity detection feature (212) includes acceleration magnitude variance, angular velocity magnitude, orientation change rate and window-based statistical feature, method. Claim 12 A method according to claim 10, further comprising the step of enhancing the first LSTM layer (204) and the second LSTM layer (216) into peephole connections to introduce a direct connection between the cell state and the gate. Claim 13 A method according to claim 10, further comprising the step of implementing bidirectional processing to utilize temporal context. Claim 14 A method according to claim 10, further comprising the step of providing a skip connection between the first LSTM layer (204) and the second LSTM layer (216) so as to bypass one of the first LSTM layer (204) and the second LSTM layer (216).
