A finger tracking system and method based on a smart ring

Abstract

The application discloses a finger tracking system and method based on a smart ring, at least one proximal phalanx of a finger is provided with a smart ring, the smart ring is provided with an inertial measurement unit; the IMU signal collected by the smart ring is preprocessed; the spatial posture of the smart ring is calculated based on the preprocessed IMU signal; local time sequence features are extracted from the preprocessed IMU signal by using a convolutional neural network; a long-range dependency relationship model is established on the local time sequence features by using a Transformer model, and a finger feature sequence is obtained; the finger feature sequence or a plurality of finger feature sequences are fused and input into an LSTM network, and then pass through a full connection layer, so that predicted joint angles of fingers are obtained. The application improves the tracking accuracy and stability of fingers.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction and wearable computing technology, specifically relating to a finger tracking system and method based on a smart ring. Background Technology

[0002] Finger tracking, as a core technology in the field of human-computer interaction, has wide applications in scenarios such as virtual reality (VR), augmented reality (AR), and smart home control. Currently, the mainstream technical approaches mainly include solutions based on visual perception and solutions based on wearable sensors, but both have significant technical limitations.

[0003] In visual perception solutions, researchers typically use cameras or depth sensors to capture images of hands, and then achieve gesture recognition through image processing or deep learning models. However, their performance is heavily dependent on ambient lighting conditions, and recognition stability drops significantly when there is hand self-occlusion or object occlusion. In addition, this type of solution requires continuous collection of user video data, raising concerns about user privacy and limiting its application in privacy-sensitive scenarios such as homes and healthcare.

[0004] To overcome the limitations of vision-based solutions, data gloves (such as the CyberGlove series) achieve high-precision hand posture capture by placing bend sensors, fiber optic sensors, or inertial measurement units at key finger joints. They demonstrate good performance in virtual reality training; however, these devices are typically bulky, inconvenient to wear, and suffer from cable entanglement, severely impacting user experience and natural interaction capabilities.

[0005] Another representative solution is the wristband or armband device, such as the Myo Armband from Thalmic Labs, which indirectly infers gestures through surface electromyography (sEMG) signals and inertial sensors. While these devices offer good portability, their limited accuracy stems from the fact that the sensors are located far from the finger joints, making it difficult to capture the subtle coordinated movements between the thumb and forefinger. This results in poor performance, particularly in applications requiring the differentiation of subtle gesture differences.

[0006] In summary, the following unresolved technical problems exist in the existing technology:

[0007] Visual solutions are limited by ambient lighting and occlusion, and pose privacy risks.

[0008] Data gloves and other full-hand wearable devices are bulky and inconvenient, hindering natural interaction;

[0009] Wristbands, armbands, and other devices have long sensing distances, making it difficult to capture fine finger movements;

[0010] Single-finger ring devices have limited sensing information and cannot accurately model multi-finger coordinated movements. Summary of the Invention

[0011] The purpose of this invention is to address the shortcomings of existing technologies by providing a finger tracking system and method based on smart rings. This system can accurately capture the coordinated movement information of the thumb and index finger through a dual-smart ring configuration, and utilizes a deep learning model that incorporates hand anatomical and physical constraints to achieve high-precision, high-stability, and physiologically sound finger tracking in various postures and non-visual environments.

[0012] The objective of this invention is achieved through the following technical solution: a finger tracking method based on a smart ring, comprising the following steps:

[0013] (1) At least one finger has a smart ring attached to the proximal phalanx, and the smart ring is equipped with an inertial measurement unit;

[0014] (2) Preprocess the IMU signals acquired by the smart ring;

[0015] (3) Calculate the spatial attitude of the smart ring based on the preprocessed IMU signal;

[0016] (4) Using a convolutional neural network, extract local temporal features from the preprocessed IMU signal;

[0017] (5) The Transformer model is used to model the long-range dependency of local temporal features to obtain the finger feature sequence;

[0018] (6) The finger feature sequence or multiple finger feature sequences are fused and input into the LSTM network, and then passed through a fully connected layer to obtain the predicted angles of each finger joint.

[0019] Furthermore, the proximal phalanges of the thumb and other fingers are fitted with smart rings.

[0020] Furthermore, the preprocessing includes filtering out signal components and removing gravitational acceleration.

[0021] Furthermore, it also includes step (7): based on the physical constraints of hand anatomy, the predicted joint angles are verified and optimized, and the final finger angle sequence that conforms to physiology is output.

[0022] Furthermore, by designing the physical constraints of hand anatomy as a custom loss function, it guides model optimization together with the main loss function during the model training phase; the model includes a convolutional neural network, a Transformer model, an LSTM network, and a fully connected layer, which are connected in sequence.

[0023] Furthermore, the main loss function is the mean squared error loss. Custom loss functions include motion trend loss and smoothness constraints.

[0024] Loss of Motion Trend : Introduce a direction consistency loss based on cosine similarity to constrain the consistency between the predicted sequence and the real sequence in the overall motion trend;

[0025] Smoothness constraint Construct a smoothness loss function term based on the time dimension to fine-tune the rate of change of joint angles between adjacent time steps;

[0026] The total loss function is: ;

[0027] Here, λ1 and λ2 are hyperparameters in the user-defined loss function.

[0028] Furthermore, the following three physical constraints are set based on λ1 and λ2:

[0029] Physical constraints (1): The predicted direction of joint angle change is consistent with the actual anatomical movement direction;

[0030] Physical constraint (2): The rate of change of angle of any finger joint in adjacent time steps must not exceed its physiological limit;

[0031] Physical constraints (3): All predicted joint angles must fall within their known physiological range;

[0032] If the physical constraint (1) is violated, that is, the direction of the coordinated movement trajectory of the thumb and index finger does not conform to the anatomical knowledge, the gradient of the direction consistency loss term is enhanced by the hyperparameter λ1 to guide the model to correct the movement direction;

[0033] If physical constraints (2) are violated, that is, if the joint angle changes result in high-frequency jitter that is not physiologically allowed, then the gradient of the smoothness loss term is enhanced by the hyperparameter λ2 to force the model to generate a smooth transition sequence.

[0034] If the physical constraint (3) is violated, that is, the predicted angle exceeds the range of joint physiological activity, then a differentiable mapping is directly applied to the model output layer to project it into a reasonable range to ensure the anatomical correctness of the posture.

[0035] The present invention also provides a finger tracking system based on a smart ring, including a hardware system and a data processing system;

[0036] The hardware system includes several smart rings, which include:

[0037] The main control module is configured to control the exchange of data between the various modules of the smart ring and to issue commands to each module.

[0038] The motion sensor module, including an inertial measurement unit, is configured to: acquire IMU signals generated by finger movements in real time, wherein the IMU signals include at least three-axis acceleration and three-axis angular velocity;

[0039] The edge computing module is configured to: receive and preprocess IMU signals from the motion sensor module under the scheduling of the main control module; or may also include performing spatial attitude calculation tasks for the smart ring.

[0040] The wireless communication module is configured to: establish a data link between the system and external devices, and transmit the data collected by the motion sensor module and the data processed by the edge computing module to the external devices; the external devices include the data processing system;

[0041] The power management module is used to supply power to the various modules of the smart ring;

[0042] The data processing system includes:

[0043] The attitude estimation unit is configured to calculate the spatial attitude of the smart ring based on IMU signals.

[0044] The feature extraction unit is configured to: use a convolutional neural network to extract local temporal features from the preprocessed IMU signal and map the extracted features to a high-dimensional embedding space;

[0045] The finger tracking unit is configured to: adopt a model architecture based on a Transformer encoder to model the long-range dependencies of local temporal features to obtain a finger feature sequence; input the finger feature sequence or multiple finger feature sequences into an LSTM network and pass through a fully connected layer to obtain the predicted angles of each finger joint;

[0046] The physical constraint module is configured to apply physiological constraints to the predicted angles of each finger joint at the output of the finger tracking unit or during training.

[0047] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described finger tracking method based on a smart ring.

[0048] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described finger tracking method based on a smart ring.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] 1. Significantly Improved Tracking Accuracy and Stability: By directly capturing the coordinated movement information of the thumb and index finger through a smart ring configuration and combining it with an innovative hybrid model, high-precision tracking at the fingertip level is achieved. Experiments show that its average angle error is as low as 7.75° and the average joint position error (MPJPE) is as low as 4.88 mm, representing a 39% improvement in prediction accuracy compared to the best existing model.

[0051] 2. Excellent environmental robustness and privacy protection: Completely eliminates the dependence on external cameras, providing stable and uninterrupted finger tracking in low-light, occluded, and privacy-sensitive environments, fundamentally solving the inherent shortcomings of visual methods.

[0052] 3. Comfortable to wear and natural interaction: The ring device is lightweight and portable. The design of wearing it near the knuckle ensures signal quality while minimizing interference with daily hand activities, supporting a long-term, natural and seamless interactive experience.

[0053] 4. Algorithm Innovation and Physiological Rationality: The algorithm creatively combines data-driven deep learning models with knowledge-driven anatomical and physical constraints, ensuring that the predicted finger movements are not only numerically accurate but also physiologically realistic and reasonable, avoiding jitter or unreasonable postures caused by data. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A flowchart illustrating a finger tracking method based on a smart ring, provided in an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the tracking algorithm flow of the smart ring provided in an embodiment of the present invention;

[0057] Figure 3 This is a structural block diagram of a smart ring provided in an embodiment of the present invention. Detailed Implementation

[0058] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0059] As can be seen from the background technology, when users control AR / VR head-mounted display devices, they typically obtain the posture of their hands through physical devices such as controllers, cameras, or gloves. However, these physical devices are dependent on the environment and are inconvenient to carry, which prevents users from carrying the devices with them anytime and anywhere, resulting in a poor user experience.

[0060] To enable users to visualize the details of their hand movements anytime and anywhere, in scenarios such as walking, wearing AR / VR head-mounted displays, and interacting with distant computer devices, this solution proposes a finger interaction system and method based on smart rings. At least one smart ring is worn on the user's index finger, or two smart rings are worn on the thumb and index finger respectively (one smart ring is easier to carry, while two smart rings can recognize a wider range of movements and provide higher control precision). Each smart ring is equipped with at least one inertial measurement unit (IMU). After wearing the smart rings, the system captures the movement state of the thumb and index finger in real time, achieving an interactive input method similar to a visual camera. This allows users to visualize the movement trajectory of their fingers anytime and anywhere, improving the user experience. The following detailed description of the invention is provided through various embodiments.

[0061] This invention provides a finger tracking method based on a smart ring, comprising the following steps:

[0062] S101: At least one finger has a smart ring attached to the proximal phalanx, the smart ring being equipped with an inertial measurement unit.

[0063] For example, wearing option one: When wearing one smart ring, place it on the proximal index finger joint of the index finger. Wearing option two: When wearing two smart rings, place them on the proximal thumb joint of the thumb and the proximal index finger joint of the index finger, respectively.

[0064] Preferably, two smart rings are worn on the proximal thumb joint of the thumb and the proximal index finger joint of the index finger, respectively. By simultaneously activating their built-in inertial measurement units (IMUs) on the two smart rings worn on the proximal phalanges of the thumb and index finger, the original motion signals, i.e., IMU signals, are acquired in real time at a preset sampling frequency. The IMU signals include triaxial acceleration signals and triaxial angular velocity signals.

[0065] Understandably, a smart ring contains at least one hardware device capable of reading IMU signals in real time.

[0066] S102: Preprocesses the IMU signal acquired by the smart ring to improve signal quality and provide high-quality and robust input for subsequent calculations.

[0067] The preprocessing includes signal component filtering and gravity acceleration removal. Specifically, signal components in the IMU signal are filtered out. Based on the IMU signal with these components removed, unnecessary signals caused by signal noise and physiological jitter can be suppressed. Gravity acceleration removal involves separating the linear acceleration generated by motion from the original acceleration signal to obtain the original motion acceleration signal.

[0068] S103: Calculate the spatial attitude of the smart ring based on the preprocessed IMU signal.

[0069] Based on the preprocessed IMU signals, a sensor fusion algorithm (Mahony AHRS algorithm) is used to calculate and output the precise orientation of each smart ring in three-dimensional space in real time. This orientation is represented in quaternion form, reflecting the rotational state of the smart ring itself. The orientation of the ring in three-dimensional space is estimated using the IMU signals of at least one smart ring.

[0070] It should be noted that by multiplying a reference vector in the local coordinate system of at least one smart ring with a rotation matrix, the result is transformed to the world coordinate system, thus obtaining a stable three-dimensional direction vector representing the axis of the smart ring. Similarly, by multiplying a reference vector in the local coordinate system of two smart rings with a rotation matrix, the result is transformed to the world coordinate system, thus obtaining two stable three-dimensional direction vectors representing the axis of the smart rings.

[0071] It should be noted that the physical quantity used to describe the posture of finger bone segments is a three-dimensional direction vector. For the actual finger orientation... Two smart rings worn on different fingers output direction vector estimates respectively. , This can be considered as the condition of zero-mean random noise. Two independent observations were performed. By weighted vector fusion of the two observations, a direction vector estimate can be obtained. Its covariance is strictly smaller than that of any single path. Therefore, the spatial direction vector calculated based on IMU signals acquired by multiple smart rings shows a provable improvement in both stability and accuracy compared to the case using only a single smart ring.

[0072] S104: Sliding window construction, which divides the preprocessed continuous IMU signal stream into fixed-length data samples through a sliding window mechanism for subsequent time series analysis by deep learning models.

[0073] Specifically, the sliding window covers a fixed time length and consists of data frames from N consecutive time points. The window slides forward on the time axis with a fixed step size.

[0074] Preferably, the sliding step size is set to 1 frame to achieve continuous tracking with a high overlap rate, ensuring that no finger movements are missed. Each window encapsulates all sensor channel data (including but not limited to three-dimensional linear acceleration and three-dimensional angular velocity) of the thumb and index finger rings within this time interval, forming a standardized model input tensor.

[0075] S105: Finger Movement Feature Extraction and Modeling

[0076] The sliding window data constructed in step S104 is used as input to a dedicated deep learning model for high-level motion feature extraction and finger movement state modeling. This process is unsupervised end-to-end learning, where the model learns a complex nonlinear mapping from low-level sensor sequences to high-level motion features through training data. The deep learning model includes a convolutional neural network, a Transformer model, an LSTM network, and a regression head (such as one or more cascaded fully connected layers) connected in series.

[0077] For example, when a user wears only one smart ring on a single finger, the preprocessed IMU signal acquired by that single smart ring is input into a convolutional neural network for motion feature extraction, resulting in a finger feature sequence. If two smart rings are worn on a finger, the preprocessed IMU signals acquired by both smart rings are input into convolutional neural networks for motion feature extraction, resulting in two finger feature sequences, which are then fused. In this configuration, the model not only extracts the independent motion features of the fingers, but more importantly, it models the relative motion and cooperative relationship between the thumb and index finger, thereby achieving more complex and precise two-finger interactive tracking.

[0078] In the specific implementation step S105, each sliding window encapsulates a multimodal feature vector from the specified smart ring. This feature vector includes, but is not limited to, preprocessed IMU signals (raw acceleration signals and angular velocity signals). A deep learning model is used to perform deep analysis on this complex combined signal, aiming to automatically learn and extract the effective information components that truly drive finger movement, distilling high-level motion feature components related to finger movement from the raw, combined sensor signals.

[0079] S106: Finger joint angle tracking and optimization.

[0080] Specifically, this step receives the finger feature sequence or the fused finger feature sequence output by step S105, and completes the precise mapping and optimization from features to joint angles based on this.

[0081] S106.1: Joint Angle Regression Prediction

[0082] Based on the finger feature sequence extracted or fused in step S105, the continuous angle values ​​of each finger joint are directly predicted through regression calculation. The predicted initial angle values ​​are then optimized to ensure they conform to the physical constraints of hand anatomy (such as joint range of motion limitations and the proportional relationship of movement between finger joints), ultimately outputting a smooth, stable, and physiologically reasonable finger joint angle sequence.

[0083] It should be noted that finger joint angle tracking is based on an unsupervised machine learning model. Based on the high-level motion feature components, a regression head (such as one or more fully connected layers) directly predicts the continuous angle values ​​of the finger joints at one or more future time points. The finger joints include, but are not limited to: the thumb can be divided into the following two parts: the metacarpophalangeal joint (MCP) and the interphalangeal joint (IP); the index, middle, ring, and little fingers can be divided into the following three parts: the metacarpophalangeal joint (MCP), the proximal interphalangeal joint (PIP), and the distal interphalangeal joint (DIP).

[0084] S106.2: Kinematic Smoothing and Optimization

[0085] To ensure the prediction results are temporally smooth and physiologically reasonable, the initial angle values ​​obtained from the S106.1 regression are optimized: a deep learning-based angle predictor is introduced. This predictor analyzes the historical sequence of joint angles, learns their motion dynamics, and predicts the rate of change of angles (i.e., angular velocity) in future frames. By constraining the predicted angles with the angles calculated based on angular velocity, frame-to-frame jitter is effectively suppressed, generating a smooth and continuous motion trajectory. Furthermore, the predicted joint angles are mapped into a constraint space that conforms to hand anatomy, ensuring the physiological reasonableness of the output results.

[0086] It should be noted that in implementing this solution, the signal sensing part (reading and sending IMU signals) can be implemented through the sensors and wireless communication module transmission devices built into the smart ring, and the calculation part can be completed in the computer receiving the signal. If the smart ring has a built-in computing unit, the calculation part can also be completed on the chip inside the smart ring.

[0087] In practical applications, the system of this invention operates according to the following workflow: First, the smart ring worn on the user's finger continuously reads the raw signal from its built-in IMU; then, the signal is sent to the computing unit (which can be a computer or mobile device separate from the smart ring, or a main control chip integrated inside the ring); after receiving the signal, the computing unit performs preprocessing steps including filtering and sliding window construction, and runs the finger motion tracking algorithm as described above (covering pose estimation, feature extraction, and joint angle regression optimization); finally, the high-precision finger joint data and pose estimation output by the system can be rendered in real time in three dimensions through an external display device (such as a VR / AR headset, tablet computer, or mobile phone screen), thereby achieving intuitive visualization of finger movement.

[0088] The following is combined Figure 2 The detailed algorithm flow for finger motion feature extraction and modeling (corresponding to step S105) and joint angle regression (corresponding to step S106) in this invention is described below. The algorithm is a deep learning model specifically designed for dual-finger ring time-series data, and its core processing flow is as follows:

[0089] S201: Input Layer and Data Preparation

[0090] The deep learning model receives the output from step S104 (sliding window construction). For a sample with a window size of W, its input consists of two parallel time-series data streams (IMU signal streams): a sequence representing thumb features and a sequence representing index finger features, denoted as follows: and Where W is the sliding window size and d is the feature dimension, the feature vector includes, but is not limited to, preprocessed IMU sensor data (raw acceleration, angular velocity). The model uses this dual-channel temporal input as the basis for subsequent deep feature extraction and modeling.

[0091] S202: Local Temporal Feature Extraction

[0092] Input sequence of thumb and index finger Two independent one-dimensional convolutional neural network (1D-CNN) branches are input respectively. The 1D-CNN extracts features from local correlation patterns in the sequence by sliding the convolutional kernel along the time dimension, thereby automatically learning the local temporal features of each finger motion signal. This process not only captures subtle motion changes but also suppresses high-frequency noise in the sensor signal. The two convolutional branches output corresponding high-dimensional feature sequences. The outputs of the two 1D-CNN branches are feature sequences that encompass more motion information, denoted as... .

[0093] It should be noted that this formula describes how a CNN layer extracts local temporal features from the input sequence.

[0094]

[0095] in, It is the output feature of the l-th layer at time t. It is the weight of the k-th convolutional kernel. It is the kernel size. It is an activation function. It is a bias term.

[0096] S203: Long-range dependency modeling (Transformer encoder module)

[0097] To capture the dependencies between distant frames within the entire time window, the feature sequences extracted by the CNN are... Two parallel Transformer encoders are input separately. Embedding and positional encoding: The feature sequence is first linearly mapped to a higher dimension and sinusoidal positional encoding is added to provide temporal information to the model. The Transformer encoder achieves global dependency modeling of temporal data through its self-attention mechanism. In this mechanism, the feature vector at each time step is linearly mapped to a query vector (Q), a key vector (K), and a value vector (V). Q is used to find cooperating or complementary motion information from the entire sequence; V is used to match Q to calculate attention weights and determine its relevance to the current motion state; K carries the specific feature content and is weighted and converged according to the attention weights. Through this process, the model can selectively focus on interrelated time segments within the entire window, thereby capturing long-range dependencies and global motion patterns in complex gestures.

[0098] After processing by the Transformer encoder, feature sequences of the thumb and index finger containing global context information are obtained, where the feature representation at each time step has been fused with overall temporal information within the window range. These two feature sequences are denoted as... and .

[0099] S204: Feature Fusion and Fine-grained Temporal Modeling (Feature Fusion and LSTM Module)

[0100] The two features output by the Transformer and At each corresponding time point, the data is spliced ​​together to form a unified fusion feature sequence rich in inter-finger collaboration information. Subsequently, Input a Long Short-Term Memory (LSTM) network. LSTM, through its internal gating mechanisms (input gate, forget gate, output gate), performs refined temporal dynamic modeling of the fused feature sequence, effectively extracting smooth transition characteristics between consecutive frames and suppressing instantaneous fluctuations caused by sensor noise or sudden changes in local actions. Therefore, the temporal representation output after LSTM processing is more continuous and stable, providing a reliable dynamic feature foundation for subsequent joint angle regression.

[0101] S205: Joint Angle Regression Output (Regression Head)

[0102] The hidden state sequence output by the LSTM network is passed to a regression head consisting of fully connected layers. This regression head maps the high-dimensional features to the target joint space, directly generating predicted joint angle values. , where J is the number of joints to be predicted (e.g., thumb MCP, thumb IP, index finger MCP, index finger PIP, index finger DIP). Through this regression structure, simultaneous regression prediction of multiple joint angles can be achieved.

[0103] S206: Physical Constraint Integration

[0104] Model prediction results The data will be fed into the physics constraint module for optimization. This module can serve as a post-processing unit, or it can embed physical knowledge into the model by designing anatomical constraints as a custom loss function term, which, together with the main loss function (mean squared error MSE), guides model optimization during the model training phase.

[0105] It should be noted that the deep learning model is trained through unsupervised learning. To simultaneously ensure the numerical accuracy of the predicted angle, the correctness of the motion direction, and the smoothness of the temporal sequence, this invention defines a custom loss function that combines multi-objective optimization for model training.

[0106] Numerical precision loss Mean squared error (MSE) is used to measure the predicted joint angle. From a realistic perspective Direct numerical deviation between:

[0107]

[0108] Loss of Motion Trend We introduce a directional consistency loss based on cosine similarity to constrain the consistency between the predicted sequence and the real sequence in the overall motion trend, thus avoiding directional errors caused by scale differences.

[0109]

[0110] Smoothness constraint :

[0111] A smoothness loss function based on the time dimension is constructed to fine-tune the rate of change of joint angles between adjacent time steps. Specifically, the upper limit of the angular velocity of joint movement is limited by minimizing the first-order difference norm. This constraint forces the time series output by the model to remain continuous in the time domain, thereby effectively filtering out high-frequency noise and non-physiological joint jitter, and ensuring that the generated posture sequence meets temporal consistency.

[0112]

[0113] Let be the joint angle vector of the t-th frame; It is the joint angle vector of the previous frame (frame t-1).

[0114] The total loss function is the weighted sum of the above terms:

[0115]

[0116] Here, λ1 and λ2 are hyperparameters in the custom loss function, used to adjust the weights of different optimization objectives. By minimizing the total loss function, the model can simultaneously achieve a balance between numerical accuracy, consistency of motion trends, and temporal smoothness.

[0117] In this invention, λ1 and λ2 are hyperparameters for calculating a custom loss function, ranging from 0 to 1. The determination of λ1 aims to enhance the model's capture of the coordinated movement trajectory of the thumb and index finger, ensuring that the predicted finger joint angles are consistent with the actual physiological movement pattern in the direction of movement. The selection of λ2 is to embed the inertial characteristics and physiological range of motion limitations of human hand joint movements into the model. By limiting unnatural angular abrupt changes between adjacent frames, it effectively filters out high-frequency jitter that is physiologically impossible, thereby outputting a smooth finger movement sequence that conforms to biomechanical principles.

[0118] Based on the above information regarding λ1 and λ2, this invention establishes the following three physical constraints based on λ1 and λ2:

[0119] Physical constraints (1): The predicted direction of joint angle change is consistent with the actual anatomical movement direction.

[0120] Physical constraint (2): The rate of change (angular velocity) of any finger joint in adjacent time steps must not exceed its physiological limit.

[0121] Physical constraints (3): All predicted joint angles must fall within their known physiological range.

[0122] After obtaining the initial joint angle prediction sequence through deep learning model regression, the system will perform a consistency check with the physical constraints established in this invention. If the prediction result shows:

[0123] If the physical constraint (1) is violated, that is, the direction of the coordinated movement trajectory of the thumb and index finger does not conform to the anatomical knowledge, the gradient of the direction consistency loss term is enhanced by the hyperparameter λ1 to guide the model to correct the movement direction;

[0124] If physical constraints (2) are violated, that is, if the joint angle changes result in high-frequency jitter that is not physiologically permissible, then the gradient of the smoothness loss term is enhanced by the hyperparameter λ2 to force the model to generate a smooth transition sequence.

[0125] If the physical constraint (3) is violated, that is, the predicted angle exceeds the range of joint physiological activity, then a differentiable mapping is directly applied to the model output layer to project it into a reasonable range to ensure the anatomical correctness of the posture.

[0126] This mechanism, which deeply integrates data-driven prediction with hierarchical physical constraints, forms a fine self-correcting optimization closed loop through the synergistic effect of λ1, λ2 and output layer mapping. This fundamentally guarantees and significantly improves the synergy, smoothness, physiological rationality and overall accuracy of the final prediction results of finger joint angles.

[0127] The above embodiments of the present invention Figure 1 and Figure 2 This is a detailed description of the present invention, through Figure 1 and Figure 2 As can be seen from the content, this solution has the following advantages: This solution provides a ring-type finger tracking and interaction solution that is significantly superior to existing technologies in terms of accuracy, robustness, naturalness, and practicality.

[0128] Corresponding to the finger tracking method based on a smart ring provided in the above embodiments of the present invention, see also... Figure 3 The present invention also provides a finger tracking system based on a smart ring, including a hardware system and a data processing system;

[0129] The hardware system includes several smart rings, which include:

[0130] The main control module is configured to: control the exchange of data between the various modules of the smart ring, issue commands to each module, and encapsulate the IMU signals collected by the motion sensor module;

[0131] Specifically, the main control module is responsible for the core computing and data management of the entire ring, and is electrically connected to the motion sensor module, edge computing module, wireless communication module, and power management module. It sends start / stop sampling and sampling frequency setting commands to the motion sensor module, and receives sampled data for processing by the edge computing module. It sends the processed data signals from the edge computing module to the wireless communication module, and adjusts the sampling and communication strategies based on the power status feedback from the power management module.

[0132] The motion sensor module, including an inertial measurement unit, is configured to: acquire IMU signals generated by finger movements in real time, wherein the IMU signals include at least three-axis acceleration and three-axis angular velocity;

[0133] The edge computing module is configured to: receive and preprocess IMU signals from the motion sensor module under the scheduling of the main control module; or may also include performing spatial attitude calculation tasks for the smart ring.

[0134] The edge computing module, a key component for achieving low-latency system response and user privacy protection, receives raw data streams from the motion sensor module under the scheduling of the main control module and performs data preprocessing and computation tasks. Specific tasks include, but are not limited to: real-time filtering and calibration of sensor data, and, when local computing resources permit, performing gravity separation and coordinate transformation on the filtered and calibrated data, and obtaining the quaternion representation of each ring in real time using attitude calculation algorithms. This function effectively reduces the burden on backend computing devices and improves the overall system response speed and efficiency.

[0135] The wireless communication module is configured to: establish a data link between the system and external devices, and transmit the data collected by the motion sensor module and the data processed by the edge computing module to the external devices (such as personal computers, smartphones, or VR / AR headsets); the external devices include the data processing system;

[0136] This module preferably uses the Bluetooth Low Energy (BLE) protocol to transmit feature data processed by the edge computing module or IMU signals directly encapsulated by the main control module to the data processing system in an external device in a stable, efficient and low-power manner, thereby realizing real-time interaction and remote processing of finger movement information.

[0137] The power management module is used to supply power to the various modules of the smart ring;

[0138] As the energy supply and management unit of the entire system, it integrates a rechargeable lithium battery and a set of dedicated power management integrated circuits. It is responsible for providing power to all electronic modules in the ring and is fully responsible for the battery charging process management, dynamic voltage conversion, real-time power consumption monitoring of the system, and low power protection, aiming to maximize the device's single-charge battery life.

[0139] The battery management module provides power to the main control module, which in turn connects to and powers the other modules.

[0140] The data processing system includes:

[0141] The attitude estimation unit is configured to calculate the spatial attitude of the smart ring based on IMU signals.

[0142] Specifically, a sensor fusion algorithm based on nonlinear complementary filtering (Mahony AHRS algorithm) is used to fuse acceleration and angular velocity data, estimate the orientation of each ring in three-dimensional space in real time, and output attitude information in quaternion form.

[0143] The attitude estimation unit receives the quaternion result calculated by the edge computing module and outputs it directly.

[0144] The feature extraction unit is configured to: use a convolutional neural network to extract local temporal features from the preprocessed IMU signal and map the extracted features to a high-dimensional embedding space;

[0145] The finger tracking unit is configured to: adopt a model architecture based on a Transformer encoder to model the long-range dependencies of local temporal features to obtain a finger feature sequence; input the finger feature sequence or multiple finger feature sequences into an LSTM network and pass through a fully connected layer to obtain the predicted angles of each finger joint;

[0146] The physical constraint module is configured to apply physiological constraints to the predicted angles of each finger joint at the output of the finger tracking unit or during training; the physiological constraints are described in the above-mentioned physical constraints (1) to physical constraints (3).

[0147] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described finger tracking method based on a smart ring.

[0148] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described finger tracking method based on a smart ring.

[0149] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A finger tracking method based on a smart ring, characterized in that, Includes the following steps: (1) The proximal phalanges of the thumb and index finger are fitted with smart rings, and the smart rings are equipped with inertial measurement units; (2) The IMU signal collected by the smart ring is preprocessed, including signal component filtering and gravity acceleration removal. (3) Calculate the spatial attitude of the smart ring based on the preprocessed IMU signal; (4) Using a convolutional neural network, extract local temporal features from the preprocessed IMU signal; (5) The Transformer model is used to model the long-range dependency of local temporal features to obtain the finger feature sequence; (6) After fusing the feature sequences of multiple fingers, input them into the LSTM network and pass through the fully connected layer to obtain the predicted angles of each finger joint; (7) Based on the physical constraints of hand anatomy, the predicted joint angles are verified and optimized, and the final finger angle sequence that conforms to physiology is output; by designing the physical constraints of hand anatomy as a custom loss function, it guides the model optimization together with the main loss function during the model training stage; the model includes a convolutional neural network, a Transformer model, an LSTM network and a fully connected layer, which are connected in series. The main loss function is the mean squared error loss. Custom loss functions include motion trend loss and smoothness constraints. Loss of Motion Trend : Introduce a direction consistency loss based on cosine similarity to constrain the consistency between the predicted sequence and the real sequence in the overall motion trend; Smoothness constraint Construct a smoothness loss function term based on the time dimension to fine-tune the rate of change of joint angles between adjacent time steps; The total loss function is: ; Where λ1 and λ2 are hyperparameters in the user-defined loss function; Based on λ1 and λ2, the following three physical constraints are set: Physical constraints (1): The predicted direction of joint angle change is consistent with the actual anatomical movement direction; Physical constraint (2): The rate of change of angle of any finger joint in adjacent time steps must not exceed its physiological limit; Physical constraints (3): All predicted joint angles must fall within their known physiological range; If the physical constraint (1) is violated, that is, the direction of the coordinated movement trajectory of the thumb and index finger does not conform to the anatomical knowledge, the gradient of the direction consistency loss term is enhanced by the hyperparameter λ1 to guide the model to correct the movement direction; If physical constraints (2) are violated, that is, if the joint angle changes result in high-frequency jitter that is not physiologically allowed, then the gradient of the smoothness loss term is enhanced by the hyperparameter λ2 to force the model to generate a smooth transition sequence. If the physical constraint (3) is violated, that is, the predicted angle exceeds the range of joint physiological activity, then a differentiable mapping is directly applied to the model output layer to project it into a reasonable range to ensure the anatomical correctness of the posture.

2. A finger tracking system based on a smart ring, characterized in that, This includes hardware systems and data processing systems; Used to implement the method of claim 1; The hardware system includes several smart rings, which include: The main control module is configured to control the exchange of data between the various modules of the smart ring and to issue commands to each module. The motion sensor module, including an inertial measurement unit, is configured to: acquire IMU signals generated by finger movements in real time, wherein the IMU signals include at least three-axis acceleration and three-axis angular velocity; The edge computing module is configured to: receive and preprocess IMU signals from the motion sensor module under the scheduling of the main control module; or may also include performing spatial attitude calculation tasks for the smart ring. The wireless communication module is configured to: establish a data link between the system and external devices, and transmit the data collected by the motion sensor module and the data processed by the edge computing module to the external devices; the external devices include the data processing system; The power management module is used to supply power to the various modules of the smart ring; The data processing system includes: The attitude estimation unit is configured to calculate the spatial attitude of the smart ring based on IMU signals. The feature extraction unit is configured to: use a convolutional neural network to extract local temporal features from the preprocessed IMU signal and map the extracted features to a high-dimensional embedding space; The finger tracking unit is configured to: adopt a model architecture based on a Transformer encoder to model the long-range dependencies of local temporal features to obtain a finger feature sequence; fuse multiple finger feature sequences and input them into an LSTM network, and then pass through a fully connected layer to obtain the predicted angles of each finger joint; The physical constraint module is configured to apply physiological constraints to the predicted angles of each finger joint at the output of the finger tracking unit or during training.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a finger tracking method based on a smart ring as described in claim 1.

4. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the finger tracking method based on a smart ring as described in claim 1.

Images