Flexible glove based on sparse tensile sensing and three-dimensional hand fine-grained reconstruction method

Abstract

The invention provides a flexible glove based on sparse tensile sensing and a three-dimensional hand fine-grained reconstruction method, and relates to the technical field of man-machine interaction. The method comprises the following steps: constructing a biological constraint sensing sensor array layout; according to the sensor array layout, a wearable glove integrated with six-channel sparse stretching sensing is designed; constructing a three-dimensional multi-modal dynamic hand data set according to the wearable glove; designing a data flow processing pipeline; inputting the data set into a data stream processing pipeline to obtain a preprocessed data set; constructing a multi-stage graph attention network; inputting the preprocessed data set into a multi-stage graph attention network to obtain space mapping data from a 6-channel signal to a 15-joint signal; the mapping data are input into a multi-scale Transform model, and motion feature data of the 15 joints on the time sequence are obtained; according to the motion feature data, a forward kinematics model is adopted for decoding, and the rotation quaternion of the 20 skeletons is output. According to the invention, the stability of the wearable interactive glove can be improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a flexible glove and three-dimensional hand fine-grained reconstruction method and device based on sparse stretch sensing. Background Art

[0002] Human-computer interaction technology is a core foundation for augmented reality, virtual reality, and robotic control. High-degree-of-freedom hand pose estimation is crucial for achieving natural interaction. Traditional interactive systems rely on gesture recognition to accomplish tasks such as command transmission and virtual manipulation. Its accuracy and real-time performance directly impact the user experience. Existing technologies primarily rely on visual sensors, such as depth cameras, or wearable sensors, such as inertial measurement units and electromyography sensors.

[0003] However, existing gesture estimation schemes face numerous challenges in practical applications. First, vision-based methods are susceptible to interference from lighting changes, occlusions, and complex backgrounds. In particular, insufficient image resolution can lead to the loss of critical joint information when capturing fast, small gestures. Second, while wearable devices based on multi-sensor fusion are robust to the environment, they require densely deployed sensors, resulting in high device costs, poor wearing comfort, and high signal decoupling complexity. Furthermore, existing gesture modeling methods often rely on simplified kinematic models, which make it difficult to cover the detailed movements of the hand, limiting the ability to recognize complex gestures.

[0004] The limitations of data resources further exacerbate technical bottlenecks. Current public gesture datasets focus on static gestures or low-degree-of-freedom movements, lacking comprehensive annotation of dynamic continuous gestures, subtle joint angle changes, and personalized hand features. While existing neural network models excel in general object detection, they lack dedicated optimization strategies for the temporal dependencies, joint constraints, and sensor noise inherent in gesture estimation, resulting in insufficient generalization capabilities in real-world scenarios.

[0005] Recent advances in flexible sensing technology have provided new insights into gesture estimation. However, existing solutions based on stretchable sensors still suffer from significant drawbacks, such as redundant sensor layouts, significant signal drift, and a lack of effective decoupling of multi-degree-of-freedom wrist motion. These shortcomings make it difficult for existing systems to balance accuracy, cost, and user experience, hindering their widespread adoption in consumer interactive devices. Therefore, a low-cost, highly robust gesture estimation method that covers all degrees of freedom of the hand is urgently needed to overcome current technical bottlenecks. Summary of the Invention

[0006] To address the technical issues of poor stability, low sensitivity, and severe noise interference in existing technologies, the present invention provides a flexible glove and three-dimensional hand fine-grained reconstruction method and device based on sparse stretch sensing. The technical solution is as follows:

[0007] On the one hand, a flexible glove and a three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing are provided. The method is implemented by a flexible glove and a three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing. The method includes:

[0008] S1. Constructing sensor array layout for biological constraint sensing;

[0009] S2. Design a wearable glove with integrated 6-channel sparse stretch sensing based on the sensor array layout;

[0010] S3. Based on the wearable glove, construct a three-dimensional multimodal dynamic hand dataset covering 22 degrees of freedom and a total duration of 2400 minutes;

[0011] S4. Designing a data stream processing pipeline; inputting the three-dimensional multimodal dynamic hand dataset into the data stream processing pipeline for preprocessing to obtain a preprocessed dataset;

[0012] S5. Construct a multi-stage graph attention network; input the preprocessed dataset into the multi-stage graph attention network for preliminary spatial modeling to obtain spatial mapping data from 6-channel signals to 15 joints;

[0013] S6. Input the spatial mapping data of the 6-channel signal to the 15 joints into the multi-scale Transformer model for temporal modeling to obtain the motion feature data of the 15 joints in time series;

[0014] S7. Based on the temporal motion feature data of the 15 joints, the forward kinematics model is used for decoding, the skeletal quaternion inference is completed, and the rotation quaternions of the 20 bones are output.

[0015] Optionally, the process of integrating a 6-channel sparse stretch sensing wearable glove includes:

[0016] A resistive strain sensor is selected, with a specification of 15mm×80mm, an initial resistance of 30-40Ω, a stretchability of 0.2%-100%, and a resistance of 70-80Ω after 100% deformation.

[0017] An Arduino development board is used to connect a 6-channel flexible stretch sensor, a Bluetooth module is used to transmit data, and a 200Ω resistor is connected in series with a single sensor for voltage division to ensure that the sensor reading after voltage division remains between 250 and 308.

[0018] Optionally, the process of designing a data stream processing pipeline in S4 includes:

[0019] S41. Design a signal data acquisition system. Use the signal data acquisition system to record the sensor divider resistance values ​​under each subject's extreme conditions through a sensor signal self-calibration mechanism, including the extreme conditions of the fingers in full extension and full flexion, and the extreme conditions of the wrist in left and right lateral positions, front and back flexion and extension, pronation and supination, and neutral position.

[0020] S42. Based on the voltage divider resistance value of the sensor in the extreme state, independently normalize each finger and wrist to obtain a tensile deformation that can reflect the one-to-one mapping between the signal and the hand movement;

[0021] S43. Using the Leap Motion2 truth value acquisition system to obtain the true value of the hand skeleton; using a cross-correlation analysis method to calculate the global time offset between the signal data acquisition system and the Leap Motion2 truth value acquisition system; using a dynamic time warping algorithm to align the sensor divider resistance value in the extreme state with the true value of the hand skeleton based on the global time offset; by constructing a cumulative distance matrix, extracting the optimal path for time alignment, and establishing a nonlinear time mapping relationship; obtaining aligned data based on the nonlinear time mapping relationship;

[0022] S44. Resample the aligned data to a uniform 20 Hz time grid through cubic spline interpolation, completing the data stream processing pipeline design process.

[0023] Optionally, the S5 inputs the preprocessed data set into a multi-stage graph attention network for preliminary spatial modeling to obtain spatial mapping data from 6-channel signals to 15 joints, including:

[0024] S51, performing standardization processing on the preprocessed data set to obtain a standardized data set;

[0025] S52. Based on the standardized data set, a multi-layer perceptron is used for mapping to obtain preliminary features;

[0026] S53. Based on the preliminary features, a graph attention network is used to perform two-stage graph structure processing to obtain spatial mapping data from 6-channel signals to 15 joints.

[0027] Optionally, the step S6 inputs the spatial mapping data of the 6-channel signals to the 15 joints into a multi-scale Transformer model for time series modeling to obtain the motion feature data of the 15 joints in time series, including:

[0028] S61, input the spatial mapping data of the 6-channel signal to the 15 joints into the multi-scale Transformer model, and process it through the four-layer encoder of the multi-scale Transformer model to obtain features of different scales;

[0029] S62. Information fusion of features of different scales is performed through the cross-attention mechanism to obtain the motion feature data of 15 joints in time series.

[0030] Optionally, the S7 decodes the motion feature data of the 15 joints in time sequence using a forward kinematics model to complete skeletal quaternion reasoning and output the rotation quaternions of the 20 bones, including:

[0031] S71. Use a two-layer multi-layer perceptron to process the time-series motion feature data of 15 joints to obtain scalar joint angles.

[0032] S72, inputting the scalar joint angle into the forward kinematics model to generate a skeletal quaternion;

[0033] S73. Based on the bone quaternion and the standard hand bone length and joint rotation axis definition, traverse from the palm root node to the end node by node, convert the scalar joint angle into a global rotation, and generate the rotation bone quaternion of 20 bones.

[0034] On the other hand, a flexible glove and three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing is provided. The device is applied to a flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing. The device includes:

[0035] A first building block for constructing a sensor array layout for biological constraint sensing;

[0036] A design unit, configured to design a wearable glove integrating 6-channel sparse stretch sensing according to the sensor array layout;

[0037] A second construction unit is configured to construct a three-dimensional multimodal dynamic hand dataset covering 22 degrees of freedom and a total duration of 2400 minutes based on the wearable glove;

[0038] A data processing unit is configured to design a data stream processing pipeline; input the three-dimensional multimodal dynamic hand data set into the data stream processing pipeline for preprocessing to obtain a preprocessed data set;

[0039] The first acquisition unit is used to build a multi-stage graph attention network; the preprocessed dataset is input into the multi-stage graph attention network for preliminary spatial modeling, and the spatial mapping data of 6-channel signals to 15 joints is obtained;

[0040] The second acquisition unit is used to input the spatial mapping data of the 6-channel signal to the 15 joints into the multi-scale Transformer model for time series modeling, and obtain the motion feature data of the 15 joints in time series;

[0041] The output unit is used to decode the motion feature data of 15 joints in time sequence using a forward kinematics model, complete skeletal quaternion inference, and output the rotation quaternions of 20 bones.

[0042] Optionally, the process of integrating a 6-channel sparse stretch sensing wearable glove includes:

[0043] A resistive strain sensor is selected, with a specification of 15mm×80mm, an initial resistance of 30-40Ω, a stretchability of 0.2%-100%, and a resistance of 70-80Ω after 100% deformation.

[0044] An Arduino development board is used to connect a 6-channel flexible stretch sensor, a Bluetooth module is used to transmit data, and a 200Ω resistor is connected in series with a single sensor for voltage division to ensure that the sensor reading after voltage division remains between 250 and 308.

[0045] Optionally, the process of designing a data stream processing pipeline includes:

[0046] Design a signal data acquisition system; use the system's sensor signal self-calibration mechanism to record the sensor divider resistance values ​​under each subject's extreme conditions, including the finger extremes of full extension and full flexion, and the wrist extremes of left and right lateral positions, front and back flexion and extension, pronation and supination, and neutral position.

[0047] Based on the sensor's voltage divider resistance value at the limit state, each finger and wrist is independently normalized to obtain a tensile deformation that can reflect the one-to-one mapping between the signal and the hand movement.

[0048] The Leap Motion2 truth acquisition system was used to obtain the true value of the hand skeleton. A cross-correlation analysis method was used to calculate the global time offset between the signal data acquisition system and the Leap Motion2 truth acquisition system. Based on the global time offset, a dynamic time warping algorithm was used to align the sensor divider resistance values ​​under extreme conditions with the true value of the hand skeleton. By constructing a cumulative distance matrix, the optimal path for time alignment was extracted and a nonlinear time mapping relationship was established. Aligned data was obtained based on this nonlinear time mapping relationship.

[0049] The aligned data are resampled to a uniform 20 Hz time grid through cubic spline interpolation to complete the data stream processing pipeline design process.

[0050] Optionally, the first acquiring unit is configured to:

[0051] Standardize the preprocessed data set to obtain a standardized data set;

[0052] Based on the standardized data set, a multi-layer perceptron is used for mapping to obtain preliminary features;

[0053] Based on the preliminary features, a graph attention network is used to perform two-stage graph structure processing to obtain spatial mapping data from 6-channel signals to 15 joints.

[0054] Optionally, the second acquiring unit is configured to:

[0055] The spatial mapping data of the 6-channel signal to 15 joints is input into the multi-scale Transformer model and processed by the four-layer encoder of the multi-scale Transformer model to obtain features of different scales;

[0056] The information of features at different scales is fused through the cross-attention mechanism to obtain the motion feature data of 15 joints in time series.

[0057] Optionally, the output unit is used to:

[0058] A two-layer multi-layer perceptron is used to process the time-series motion feature data of 15 joints to obtain scalar joint angles.

[0059] The scalar joint angles are input into the forward kinematics model to generate the bone quaternion;

[0060] According to the bone quaternion, based on the standard hand bone length and joint rotation axis definition, traverse from the palm root node to the end, convert the scalar joint angle into a global rotation, and generate the rotation bone quaternion of 20 bones.

[0061] On the other hand, a flexible glove and a three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing are provided. The flexible glove and three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned flexible glove and three-dimensional hand fine-grained reconstruction methods based on sparse stretch sensing is implemented.

[0062] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any of the above-mentioned flexible gloves and three-dimensional hand fine-grained reconstruction methods based on sparse stretch sensing.

[0063] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0064] The embodiment of the present invention first constructs a sensor array layout for biological constraint perception; based on the sensor array layout, a wearable glove with integrated 6-channel sparse stretch sensing is designed; based on the wearable glove, a three-dimensional multimodal dynamic hand dataset covering 22 degrees of freedom and a total duration of 2400 minutes is constructed; a data stream processing pipeline is designed; secondly, the three-dimensional multimodal dynamic hand dataset is input into the data stream processing pipeline for preprocessing to obtain a preprocessed dataset; a multi-stage graph attention network is constructed; the preprocessed dataset is input into the multi-stage graph attention network for preliminary spatial modeling to obtain spatial mapping data from 6-channel signals to 15 joints; the spatial mapping data from 6-channel signals to 15 joints is input into a multi-scale Transformer model for time series modeling to obtain motion feature data of 15 joints in time series; finally, based on the motion feature data of 15 joints in time series, a forward kinematics model is used for decoding, skeletal quaternion reasoning is completed, and rotation quaternions of 20 bones are output.

[0065] The embodiments of this invention are widely applicable in fields such as virtual reality, augmented reality, and smart healthcare, and have promising innovative applications. They can address the common issues of poor stability, low sensitivity, and severe noise interference in existing sensor gloves, creating highly stable, wearable, and highly flexible wearable interactive gloves and improving fine-grained operational capabilities in diverse application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 This is a flow chart of a flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing provided by an embodiment of the present invention;

[0068] Figure 2 1 is a flow chart of a flexible glove and a three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing provided by an embodiment of the present invention;

[0069] Figure 3 1 is a schematic diagram of a sensor array layout for biomechanical constraint sensing provided by an embodiment of the present invention;

[0070] Figure 4 1 is a schematic diagram of a prototype of a flexible glove based on sparse stretch sensing provided by an embodiment of the present invention;

[0071] Figure 520 dynamic gesture diagrams provided by the embodiment of the present invention;

[0072] Figure 6 1 is a schematic diagram of an algorithm framework of a three-dimensional hand fine-grained reconstruction method provided by an embodiment of the present invention;

[0073] Figure 7 This is a block diagram of a flexible glove and a three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing provided by an embodiment of the present invention;

[0074] Figure 8 This is a structural schematic diagram of a flexible glove based on sparse stretch sensing and a three-dimensional hand fine-grained reconstruction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0076] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0077] In the embodiments of the present invention, the terms "image" and "picture" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same. The terms "of," "corresponding," and "corresponding" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same.

[0078] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0079] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0080] The embodiment of the present invention provides a flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing, which can be implemented by a flexible glove and three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing, which can be a terminal or a server. Figure 1The flowchart of the flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing is shown. The processing flow of the method may include the following steps:

[0081] S1. Construct a sensor array layout for biological constraint perception.

[0082] Among them, based on the constructed biological constraint perception sensor array layout, the precise capture of complex gestures can be achieved under the condition of using 6-channel sensors.

[0083] Among them, Figure 2 The figure shows a flow chart of a flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing provided by an embodiment of the present invention. The flow mainly includes: hardware sensor path construction, data processing pipeline design, and hand posture reconstruction. In a feasible implementation, a resistive strain sensor is used to collect a true hand image, integrated with a wearable glove device, and 6-channel sensor data is obtained through the wearable glove device and Bluetooth transmission. Based on the 6-channel sensor data, a preprocessing strategy is used to process the 6-channel sensor data to obtain processed 6-channel sensor data. The processed 6-channel sensor data is input into the constructed graph convolutional neural network to reconstruct the hand skeleton and obtain the hand skeleton reconstruction result. The preprocessing strategy includes: a self-calibration mechanism, a cross-correlation analysis method, a dynamic time warping algorithm, and a cubic spline interpolation method.

[0084] The hand has a highly flexible kinematic model, and the relative rotational relationships between the phalanges and joints are usually described through anatomical or kinematic modeling. Taking the four fingers and the thumb as an example, if no constraints are applied, each finger has four degrees of freedom: the MCP joint has two degrees of freedom: flexion and extension and adduction and abduction, and the PIP and DIP joints each have one degree of freedom for flexion and extension; the thumb has five degrees of freedom: MCP (2 degrees of freedom), TM (2 degrees of freedom), and IP (1 degree of freedom). This embodiment of the present invention takes into account the palm-to-palm movement of the thumb, which is the 22nd degree of freedom. Therefore, the entire hand can have up to 22 degrees of freedom; to fully track all joints, more than 20 bones need to be identified.

[0085] The hand has 22 degrees of freedom, involving complex motions of the joints of the four fingers and the thumb. These include movements of the metacarpophalangeal and phalangeal joints of the thumb, as well as palmar opposition. To simplify the system, this embodiment introduces 16 physiological constraints, including physiological coupling of finger joints, muscle involvement constraints, thumb subdivision coupling, and coordinated abduction / adduction of the index and middle fingers, reducing the effective degrees of freedom to six. When the number of sensors satisfies mk ≥ 6, where m represents the number of degrees of freedom and k represents the number of successful constraints, an observable kinematic solution can be achieved.

[0086] Among them, the physiological coupling of finger joints refers to the existence of linkage physiological constraints on the MCP-PIP, PIP-DIP and MCP-IP joints of the four fingers, totaling 9; the muscle involvement constraint refers to the fact that the extensor digitorum can extend the four fingers, and the extensor digitorum mainly extends the little finger; there is also a certain degree of coupling between the index finger, middle finger, ring finger and little finger during flexion and extension, especially during large-scale flexion, totaling 4 constraints; thumb subdivision coupling: in the initial stage of functional tasks, flexion and abduction, as well as extension and adduction of the thumb often occur simultaneously; in addition, during palmar confrontation, the adduction and flexion of the TM joint also show a certain coupling relationship with the movement of the MCP joint, totaling 2 constraints.

[0087] Among them, the sensor array layout for biological constraint perception meets the 33-degree-of-freedom requirement. The layout includes: using a 6-channel resistive strain sensor, configuring a resistive strain sensor in the MCP-PIP-DIP direction of each of the five fingers, and adding a sensor laterally at the MCP joint of the fourth finger to achieve observation of all independent degrees of freedom, thereby reconstructing the rotational posture of 20 skeletal nodes through the forward kinematic model.

[0088] In a feasible implementation, Figure 3 Figure 1 shows a schematic diagram of a sensor array layout for biomechanical constraint sensing, provided by an embodiment of the present invention. To reduce system complexity and ease the wearer's burden, this embodiment of the present invention proposes a sensor array layout for biomechanical constraint sensing based on observability theory. The layout includes: After 16 physiological coupling constraints are applied to the hand's original 22 degrees of freedom, only 6 independent degrees of freedom remain. Therefore, kinematic observability is maintained when the number of sensors satisfies mk = 22 - 16 = 6. A resistive strain sensor is placed in the MCP-PIP-DIP direction on each of the five fingers, and an additional sensor is added laterally at the MCP joints of four fingers. This ensures that all independent degrees of freedom are covered, ensuring that changes in each degree of freedom are reflected in the sensor observations. The rotations of the 20 skeletal nodes are calculated using a forward kinematic model.

[0089] S2. Based on the sensor array layout, a wearable glove with integrated 6-channel sparse stretch sensing is designed.

[0090] Optionally, the process of integrating a 6-channel sparse stretch sensing wearable glove includes:

[0091] A resistive strain sensor is selected, with a specification of 15mm×80mm, an initial resistance of 30-40Ω, a stretchability of 0.2%-100%, and a resistance of 70-80Ω after 100% deformation.

[0092] An Arduino development board is used to connect a 6-channel flexible stretch sensor, a Bluetooth module is used to transmit data, and a 200Ω resistor is connected in series with a single sensor for voltage division to ensure that the sensor reading after voltage division remains between 250 and 308.

[0093] In a feasible implementation, Figure 4 The figure shows a prototype diagram of a flexible glove based on sparse stretch sensing provided by an embodiment of the present invention. The embodiment of the present invention uses six relatively sensitive resistive strain sensors with specifications of 15mm*80mm, an initial resistance of 30-40Ω, a stretchability of 0.2%-100%, and a resistance of 70-80Ω after 100% deformation, ensuring stable resistance changes during the acquisition process. Figure 4 (a) Connect the Arduino development board, set the transmission rate to 20Hz, and use the Bluetooth module to transmit data packets. Considering that the initial resistance and resistance after deformation of the sensor are small, a 200Ω resistor is connected in series with each sensor to divide the voltage. After voltage division, the sensor reading is kept between 250 and 308. Figure 4 (a) Connect the circuit design diagram, and the connected physical diagram is as follows Figure 4 (b) Figure 4 As shown in (c), 4(d) and 4(e), medical PE excipients are used to place 5 sensors on 5 fingers respectively, and another sensor is arranged horizontally at the MCP joints of the index finger, middle finger, ring finger and little finger. Figure 4 (c) is the arrangement scheme, Figure 4 (d) is the front view of the gloves. Figure 4 (e) is the reverse side of the gloves.

[0094] S3. Based on the wearable gloves, a three-dimensional multimodal dynamic hand dataset covering 22 degrees of freedom and a total duration of 2400 minutes is constructed.

[0095] Among them, the three-dimensional multimodal dynamic hand dataset includes 6-channel input signals and the true values ​​of the rotation quaternions of 20 bones collected by the true value acquisition system.

[0096] The 3D multimodal dynamic hand dataset was provided by three subjects, totaling 40 hours of hand motion data. The subjects performed 20 different gestures with their right hands, including 10 degree-of-freedom verification gestures and 10 daily interaction gestures. The degree-of-freedom verification gestures cover some special cases that cannot be achieved in daily interaction gestures, totaling 22 degrees of freedom. The 3D multimodal dynamic hand dataset was recorded simultaneously by the signal acquisition platform and the Leap Motion2 controller device. The Leap Motion2 controller device provides ground truth images and hand motion information including hand skeletal rotation and finger joint movement.

[0097] In one feasible implementation, an integrated sparse stretch sensing glove device is connected to a laptop computer, and a Python interface is written to obtain data packets from the Arduino development board to obtain six-channel sensor data. The acquisition period for each gesture is set to 240 seconds, and the participant is required to repeat the gesture once per second. The acquisition rate of the sensor and Leap Motion2 are both initially set to 20Hz, that is, 20 data points are recorded per second. To eliminate the possible impact of hardware differences on subsequent data alignment, the system synchronously records the system timestamp when outputting the file. The structure of the obtained sensor voltage divider resistance data is 2400*6, where 2400 represents the time point, the first column is the timestamp, and the remaining six columns represent the channel data.

[0098] In a feasible implementation, Figure 5 Shown are schematic diagrams of 20 dynamic gestures provided in an embodiment of the present invention; the first line is a degree of freedom gesture, the third line is a daily interaction gesture, the second line is a joint rotation involved in the degree of freedom gesture, and the fourth line is a joint rotation involved in the daily interaction gesture.

[0099] S4. Design a data stream processing pipeline; input the three-dimensional multimodal dynamic hand dataset into the data stream processing pipeline for preprocessing to obtain a preprocessed dataset.

[0100] Optionally, the process of designing a data stream processing pipeline in S4 includes S41-S44:

[0101] S41. Design a signal data acquisition system. Use the signal data acquisition system to record the sensor divider resistance values ​​under each subject's extreme conditions through a sensor signal self-calibration mechanism, including the extreme conditions of the fingers in full extension and full flexion, and the extreme conditions of the wrist in left and right lateral positions, front and back flexion and extension, pronation and supination, and neutral position.

[0102] S42. Based on the voltage divider resistance value of the sensor in the extreme state, independently normalize each finger and wrist to obtain a tensile deformation that can reflect the one-to-one mapping between the signal and the hand movement;

[0103] S43. Using the Leap Motion2 truth value acquisition system to obtain the true value of the hand skeleton; using a cross-correlation analysis method to calculate the global time offset between the signal data acquisition system and the Leap Motion2 truth value acquisition system; using a dynamic time warping algorithm to align the sensor divider resistance value in the extreme state with the true value of the hand skeleton based on the global time offset; by constructing a cumulative distance matrix, extracting the optimal path for time alignment, and establishing a nonlinear time mapping relationship; obtaining aligned data based on the nonlinear time mapping relationship;

[0104] S44. Resample the aligned data to a uniform 20 Hz time grid through cubic spline interpolation, completing the data stream processing pipeline design process.

[0105] The 20 Hz time grid has a time interval of 0.05 seconds. The embodiment of the present invention uses cubic interpolation to eliminate sampling unevenness.

[0106] In a feasible implementation, Figure 6 The figure shows a schematic diagram of the algorithm framework of a 3D hand fine-grained reconstruction method provided by an embodiment of the present invention; the 3D hand fine-grained reconstruction method aims to predict the 3D rotation of 20 hand bones from 6-channel input signals, and the output is represented in the form of quaternions. The overall process is as follows Figure 5 As shown in the figure, it includes: data preprocessing, initial feature extraction, spatial modeling based on a multi-stage graph attention network, multi-scale temporal modeling using a multi-scale Transformer model, and decoding through a forward kinematics model to obtain the final skeletal rotation.

[0107] S5. Construct a multi-stage graph attention network; input the preprocessed dataset into the multi-stage graph attention network for preliminary spatial modeling to obtain spatial mapping data from 6-channel signals to 15 joints.

[0108] Among them, the multi-stage graph attention network models the spatial structure of the hand through a two-stage graph attention network. First, a coarse-grained six-node graph is constructed to capture the influence of MCP on each finger; then the spatial mapping graph data of 6 joints is expanded to the spatial mapping graph data of 15 joints to model the hierarchical structure of the hand and the coupling between phalanges.

[0109] Optionally, the specific implementation of S5 includes S51-S53:

[0110] S51, performing standardization processing on the preprocessed data set to obtain a standardized data set;

[0111] In one feasible implementation, the original 6-channel time series signal is recorded as Where T represents the number of time steps, is a real number; each channel corresponds to the motion component of different parts of the hand: the first five channels represent the thumb, index finger, middle finger, ring finger and little finger respectively, and the sixth channel provides additional information about the metacarpophalangeal joint. In order to generate training targets, the embodiment of the present invention uses the hand skeleton truth value acquired by the truth value acquisition system LeapMotion2, which is in the format of quaternion tensor representation as follows Each quaternion is normalized to unit length to obtain the true quaternion.

[0112] In one feasible implementation, Each channel is normalized along the time axis to achieve a mean of 0 and a variance of 1. This normalization reduces the differences between different measurement levels. The resulting true quaternion retains its original shape, facilitating direct integration with the subsequent forward kinematics model and end-to-end training.

[0113] S52. Based on the standardized data set, a multi-layer perceptron is used for mapping to obtain preliminary features;

[0114] In one feasible implementation, the 6-channel input is mapped to the feature vectors of 6 nodes. Among them, the 6 nodes are used to represent the thumb, index finger, middle finger, ring finger, and little finger respectively, and a composite node for multiple metacarpophalangeal joints. The specific implementation process includes: for the thumb, index finger, middle finger, ring finger and little finger channels, let the scalar input be Among them, i∈{1, 2, 3, 4, 5} represents the single channel value at time step t; each channel is mapped through a multi-layer perceptron to obtain a d-dimensional feature vector represented as For multiple metacarpophalangeal joints, let the channel input be Through a multi-layer perceptron mapping, four groups of metacarpophalangeal joint feature vectors are obtained; the four groups of metacarpophalangeal joint feature vectors are merged into one node and expressed as At each time step t, the feature vectors of all nodes are merged to obtain the feature tensor Among them, in the feature tensor, node 1 corresponds to the eigenvector of the thumb, node 2 corresponds to the eigenvector of the index finger, node 3 corresponds to the eigenvector of the middle finger, node 4 corresponds to the eigenvector of the ring finger, node 5 corresponds to the eigenvector of the little finger, and node 6 corresponds to the eigenvector of the metacarpophalangeal joint.

[0115] S53. Based on the preliminary features, a graph attention network is used to perform two-stage graph structure processing to obtain spatial mapping data from 6-channel signals to 15 joints.

[0116] In a feasible implementation, the implementation process of the first stage includes: based on the feature tensor Construct a graph and connect node 6 to nodes 2, 3, 4, and 5 to obtain corresponding edges. Use the GAT self-attention mechanism to assign a learnable weight to each edge. After stacking two layers of GAT, the node features of the first stage can be expressed as Among them, the node features of the first stage contain coarse-grained inter-finger coupling information.

[0117] In a feasible implementation, the implementation process of the second stage includes: expanding the 6-node graph of the first stage to 15 joints, including the TM, MCP and IP of the thumb, and the MCP, PIP and DIP of the index finger to the little finger; for the thumb node feature, expanding one node into three nodes through the interpolation multi-layer perceptron; for the other four fingers, extracting the metacarpophalangeal joint feature from node 6, and then mapping nodes 2, 3, 4 and 5 to PIP and DIP; redefining the edges between the joints, using a layer of GAT for deep update, and outputting a 64-dimensional node feature representation as The 64-dimensional node features are spatial mapping data from 6-channel signals to 15 joints.

[0118] S6. Input the spatial mapping data of the 6-channel signal to the 15 joints into the multi-scale Transformer model for temporal modeling to obtain the motion feature data of the 15 joints in time series.

[0119] In a feasible implementation, in order to model the dynamic dependence of hand movements on the time axis, a multi-scale Transformer model is used to transform the 64-dimensional node features. We process the 15 joints as tokens, perform multi-head attention operations on T time steps, and capture short-term and long-term gesture evolution through block division at multiple time scales.

[0120] Among them, a multi-scale Transformer model is used to complete the hierarchical temporal modeling from 6-channel signals to 15 joints, and the 15 nodes are self-attention modeled at three scales of 10, 20 and 40 frames. Through cross-attention fusion, temporal features with both short-term and long-term dependencies are formed.

[0121] Optionally, the specific implementation process of S6 includes S61-S62:

[0122] S61, input the spatial mapping data of the 6-channel signal to the 15 joints into the multi-scale Transformer model, and process it through the four-layer encoder of the multi-scale Transformer model to obtain features of different scales;

[0123] In one feasible implementation, superimposed sinusoidal coding is performed on the spatial mapping data of the 6-channel signals to the 15 joints to obtain the input sequence. The use of superimposed sinusoidal coding can preserve the time sequence information.

[0124] The multi-scale Transformer model consists of a four-layer encoder, each containing eight self-attention heads and an FFN structure. To achieve multi-scale temporal representation, the input sequence is divided into different patches according to three granularities: small (10 steps), medium (20 steps), and large (40 steps). Each granularity is linearly projected and then self-attention is performed. By cross-attention, information is fused between features at different scales, and the output motion feature data of 15 joints in temporal sequence is represented as in, Multi-scale motion cues are preserved for each time step and each joint node.

[0125] S62. Information fusion of features of different scales is performed through the cross-attention mechanism to obtain the motion feature data of 15 joints in time series.

[0126] S7. Based on the temporal motion feature data of the 15 joints, the forward kinematics model is used for decoding, the skeletal quaternion inference is completed, and the rotation quaternions of the 20 bones are output.

[0127] Optionally, the specific implementation process of S7 includes S71-S73:

[0128] S71. Use a two-layer multi-layer perceptron to process the time-series motion feature data of 15 joints to obtain scalar joint angles.

[0129] Among them, Each joint node is processed by a two-layer multilayer perceptron, and the output scalar joint angle is expressed as

[0130] S72, inputting the scalar joint angle into the forward kinematics model to generate a skeletal quaternion;

[0131] S73. Based on the bone quaternion and the standard hand bone length and joint rotation axis definition, traverse from the palm root node to the end node by node, convert the scalar joint angle into a global rotation, and generate the rotation bone quaternion of 20 bones.

[0132] The above process of generating the rotation quaternions of the 20 bones directly realizes the mapping of the joint angle output from the network to the real three-dimensional space rotation.

[0133] The entire model is trained end-to-end using a composite loss function consisting of four items: quaternion loss function, temporal smoothness loss function, anatomical constraint loss function, and quaternion normalization loss function.

[0134] Among them, the quaternion loss function measures the difference between the prediction and the true value through the quaternion dot product, which is expressed by the following formula (1):

[0135]

[0136] in, represents the quaternion loss function; T represents the total number of time frames; t represents the t-th time frame; j represents the j-th joint; Represents the predicted quaternion of the jth joint in the tth frame, unit quaternion; Represents the true quaternion of the j-th joint in the t-th frame.

[0137] Among them, the temporal smoothness loss function constrains the smooth transition of rotations between adjacent time steps and is expressed by the following formula (2):

[0138]

[0139] in, represents the temporal smoothness loss, which is used to measure the smoothness of joint rotation changes between consecutive frames; q t+1,i represents the predicted quaternion of the i+1th joint in the tth frame; q t,i represents the predicted quaternion of the i-th joint in the t-th frame.

[0140] The anatomical constraint loss function limits the deviation of joint coupling and is expressed by the following formula (3):

[0141]

[0142] Among them, L anat represents the anatomical constraint loss function, which indicates the degree of joint coupling deviation; θ DIP Indicates the active angle of the DIP joint; θ PIP Indicates the active angle of the PIP joint; θ IP_thumb Indicates the active angle of the thumb IP joint; θ MCP_thumb Indicates the range of motion of the thumb MCP joint.

[0143] Among them, the quaternion normalization loss function ensures that the predicted quaternion maintains the unit length, which is expressed by the following formula (4):

[0144]

[0145] Among them, L norm Represents the quaternion normalization loss function, ensuring that the quaternion maintains unit length.

[0146] In a feasible implementation, based on the above four loss functions, a total loss function is constructed, which is expressed by the following formula (5):

[0147]

[0148] in, represents the total loss; λ1 represents the first coefficient; represents the quaternion loss function; λ2 represents the second coefficient; represents the temporal smoothness loss function; λ3 represents the third coefficient; L anat represents the anatomical constraint loss function; λ4 represents the fourth coefficient; L norm Represents the quaternion normalization loss function.

[0149] Among them, joint optimization based on the above four loss functions can ensure that the prediction results have high accuracy, temporal continuity and physiological rationality.

[0150] The embodiment of the present invention first constructs a sensor array layout for biological constraint perception; based on the sensor array layout, a wearable glove with integrated 6-channel sparse stretch sensing is designed; based on the wearable glove, a three-dimensional multimodal dynamic hand dataset covering 22 degrees of freedom and a total duration of 2400 minutes is constructed; a data stream processing pipeline is designed; secondly, the three-dimensional multimodal dynamic hand dataset is input into the data stream processing pipeline for preprocessing to obtain a preprocessed dataset; a multi-stage graph attention network is constructed; the preprocessed dataset is input into the multi-stage graph attention network for preliminary spatial modeling to obtain spatial mapping data from 6-channel signals to 15 joints; the spatial mapping data from 6-channel signals to 15 joints is input into a multi-scale Transformer model for time series modeling to obtain motion feature data of 15 joints in time series; finally, based on the motion feature data of 15 joints in time series, a forward kinematics model is used for decoding, skeletal quaternion reasoning is completed, and rotation quaternions of 20 bones are output.

[0151] The embodiments of this invention are widely applicable in fields such as virtual reality, augmented reality, and smart healthcare, and have promising innovative applications. They can address the common issues of poor stability, low sensitivity, and severe noise interference in existing sensor gloves, creating highly stable, wearable, and highly flexible wearable interactive gloves and improving fine-grained operational capabilities in diverse application scenarios.

[0152] Figure 7 This is a block diagram of a flexible glove and three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing according to an exemplary embodiment. The device is used in a flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing. Figure 7 The device includes a first construction unit 710, a design unit 720, a second construction unit 730, a data processing unit 740, a first acquisition unit 750, a second acquisition unit 760 and an output unit 770.

[0153] A first construction unit 710 is used to construct a sensor array layout for biological constraint sensing;

[0154] A design unit 720 is configured to design a wearable glove with integrated 6-channel sparse stretch sensing according to the sensor array layout;

[0155] A second construction unit 730 is configured to construct a three-dimensional multimodal dynamic hand dataset covering 22 degrees of freedom and a total duration of 2400 minutes based on the wearable glove;

[0156] The data processing unit 740 is configured to design a data stream processing pipeline; input the three-dimensional multimodal dynamic hand dataset into the data stream processing pipeline for preprocessing to obtain a preprocessed dataset;

[0157] The first acquisition unit 750 is used to build a multi-stage graph attention network; input the preprocessed data set into the multi-stage graph attention network for preliminary spatial modeling, and obtain spatial mapping data from 6-channel signals to 15 joints;

[0158] The second acquisition unit 760 is used to input the spatial mapping data of the 6-channel signal to the 15 joints into the multi-scale Transformer model for time series modeling to obtain the motion feature data of the 15 joints in time series;

[0159] The output unit 770 is used to decode the motion feature data of the 15 joints in time sequence using a forward kinematics model, complete the skeletal quaternion reasoning, and output the rotation quaternions of the 20 bones.

[0160] Optionally, the process of integrating a 6-channel sparse stretch sensing wearable glove includes:

[0161] A resistive strain sensor is selected, with a specification of 15mm×80mm, an initial resistance of 30-40Ω, a stretchability of 0.2%-100%, and a resistance of 70-80Ω after 100% deformation.

[0162] An Arduino development board is used to connect a 6-channel flexible stretch sensor, a Bluetooth module is used to transmit data, and a 200Ω resistor is connected in series with a single sensor for voltage division to ensure that the sensor reading after voltage division remains between 250 and 308.

[0163] Optionally, the process of designing a data stream processing pipeline includes:

[0164] Design a signal data acquisition system; use the system's sensor signal self-calibration mechanism to record the sensor divider resistance values ​​under each subject's extreme conditions, including the finger extremes of full extension and full flexion, and the wrist extremes of left and right lateral positions, front and back flexion and extension, pronation and supination, and neutral position.

[0165] Based on the sensor's voltage divider resistance value at the limit state, each finger and wrist is independently normalized to obtain a tensile deformation that can reflect the one-to-one mapping between the signal and the hand movement.

[0166] The Leap Motion2 truth acquisition system was used to obtain the true value of the hand skeleton. A cross-correlation analysis method was used to calculate the global time offset between the signal data acquisition system and the Leap Motion2 truth acquisition system. Based on the global time offset, a dynamic time warping algorithm was used to align the sensor divider resistance values ​​under extreme conditions with the true value of the hand skeleton. By constructing a cumulative distance matrix, the optimal path for time alignment was extracted and a nonlinear time mapping relationship was established. Aligned data was obtained based on this nonlinear time mapping relationship.

[0167] The aligned data are resampled to a uniform 20 Hz time grid through cubic spline interpolation to complete the data stream processing pipeline design process.

[0168] Optionally, the first acquiring unit 750 is configured to:

[0169] Standardize the preprocessed data set to obtain a standardized data set;

[0170] Based on the standardized data set, a multi-layer perceptron is used for mapping to obtain preliminary features;

[0171] Based on the preliminary features, a graph attention network is used to perform two-stage graph structure processing to obtain spatial mapping data from 6-channel signals to 15 joints.

[0172] Optionally, the second acquiring unit 760 is configured to:

[0173] The spatial mapping data of the 6-channel signal to 15 joints is input into the multi-scale Transformer model and processed by the four-layer encoder of the multi-scale Transformer model to obtain features of different scales;

[0174] The information of features at different scales is fused through the cross-attention mechanism to obtain the motion feature data of 15 joints in time series.

[0175] Optionally, the output unit 770 is configured to:

[0176] A two-layer multi-layer perceptron is used to process the time-series motion feature data of 15 joints to obtain scalar joint angles.

[0177] The scalar joint angles are input into the forward kinematics model to generate the bone quaternion;

[0178] According to the bone quaternion, based on the standard hand bone length and joint rotation axis definition, traverse from the palm root node to the end, convert the scalar joint angle into a global rotation, and generate the rotation bone quaternion of 20 bones.

[0179] The embodiment of the present invention first constructs a sensor array layout for biological constraint perception; based on the sensor array layout, a wearable glove with integrated 6-channel sparse stretch sensing is designed; based on the wearable glove, a three-dimensional multimodal dynamic hand dataset covering 22 degrees of freedom and a total duration of 2400 minutes is constructed; a data stream processing pipeline is designed; secondly, the three-dimensional multimodal dynamic hand dataset is input into the data stream processing pipeline for preprocessing to obtain a preprocessed dataset; a multi-stage graph attention network is constructed; the preprocessed dataset is input into the multi-stage graph attention network for preliminary spatial modeling to obtain spatial mapping data from 6-channel signals to 15 joints; the spatial mapping data from 6-channel signals to 15 joints is input into a multi-scale Transformer model for time series modeling to obtain motion feature data of 15 joints in time series; finally, based on the motion feature data of 15 joints in time series, a forward kinematics model is used for decoding, skeletal quaternion reasoning is completed, and rotation quaternions of 20 bones are output.

[0180] The embodiments of this invention are widely applicable in fields such as virtual reality, augmented reality, and smart healthcare, and have promising innovative applications. They can address the common issues of poor stability, low sensitivity, and severe noise interference in existing sensor gloves, creating highly stable, wearable, and highly flexible wearable interactive gloves and improving fine-grained operational capabilities in diverse application scenarios.

[0181] Figure 8 is a structural diagram of a flexible glove and a three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing provided by an embodiment of the present invention, such as Figure 8 As shown, the flexible glove and three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing can include the above Figure 7 The flexible glove and 3D hand fine-grained reconstruction device based on sparse stretch sensing is shown. Optionally, the flexible glove and 3D hand fine-grained reconstruction device 810 based on sparse stretch sensing may include a first processor 2001 .

[0182] Optionally, the flexible glove and three-dimensional hand fine-grained reconstruction device 810 based on sparse stretch sensing may further include a memory 2002 and a transceiver 2003 .

[0183] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0184] The following combination Figure 8 The components of the flexible glove and the three-dimensional hand fine-grained reconstruction device 810 based on sparse stretch sensing are described in detail.

[0185] The first processor 2001 is the control center of the flexible glove and three-dimensional hand fine-grained reconstruction device 810 based on sparse stretch sensing, and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0186] Optionally, the first processor 2001 can perform various functions of the flexible glove based on sparse stretch sensing and the three-dimensional hand fine-grained reconstruction device 810 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0187] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 are shown in FIG.

[0188] In a specific implementation, as an embodiment, the flexible glove and three-dimensional hand fine-grained reconstruction device 810 based on sparse stretch sensing may also include multiple processors, such as Figure 8 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0189] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0190] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and communicate with the three-dimensional hand fine-grained reconstruction device 810 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0191] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0192] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 8 (not shown separately in the figure). The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0193] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or can exist independently and communicate with the three-dimensional hand fine-grained reconstruction device 810 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0194] It should be noted that Figure 8 The structure of the flexible glove based on sparse stretch sensing and the three-dimensional hand fine-grained reconstruction device 810 shown in the figure does not constitute a limitation of the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0195] In addition, the technical effects of the flexible glove and three-dimensional hand fine-grained reconstruction device 810 based on sparse stretch sensing can refer to the technical effects of the flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing described in the above method embodiment, and will not be repeated here.

[0196] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0197] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0198] The above embodiments can be implemented in whole or in part through software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0199] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0200] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0201] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0202] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0203] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0204] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0205] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0206] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0207] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0208] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing, characterized by: The method comprises: S1. Constructing sensor array layout for biological constraint sensing; S2. Design a wearable glove with integrated 6-channel sparse stretch sensing based on the sensor array layout; S3. Based on the wearable glove, construct a three-dimensional multimodal dynamic hand dataset covering 22 degrees of freedom and a total duration of 2400 minutes; S4. Designing a data stream processing pipeline; inputting the three-dimensional multimodal dynamic hand dataset into the data stream processing pipeline for preprocessing to obtain a preprocessed dataset; S5. Construct a multi-stage graph attention network; input the preprocessed dataset into the multi-stage graph attention network for preliminary spatial modeling to obtain spatial mapping data from 6-channel signals to 15 joints; S6. Input the spatial mapping data of the 6-channel signal to the 15 joints into the multi-scale Transformer model for temporal modeling to obtain the motion feature data of the 15 joints in time series; S7. Based on the temporal motion feature data of the 15 joints, the forward kinematics model is used for decoding, the skeletal quaternion inference is completed, and the rotation quaternions of the 20 bones are output.

2. The flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing according to claim 1 is characterized in that: The process of integrating a 6-channel sparse stretch sensing wearable glove includes: A resistive strain sensor is selected, with a specification of 15mm×80mm, an initial resistance of 30-40Ω, a stretchability of 0.2%-100%, and a resistance of 70-80Ω after 100% deformation. An Arduino development board is used to connect a 6-channel flexible stretch sensor, a Bluetooth module is used to transmit data, and a 200Ω resistor is connected in series with a single sensor for voltage division to ensure that the sensor reading after voltage division remains between 250 and 308.

3. The flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing according to claim 1 is characterized in that: The process of designing the S4 data stream processing pipeline includes: S41. Design a signal data acquisition system. Use the signal data acquisition system to record the sensor divider resistance values ​​under each subject's extreme conditions through a sensor signal self-calibration mechanism, including the extreme conditions of the fingers in full extension and full flexion, and the extreme conditions of the wrist in left and right lateral positions, front and back flexion and extension, pronation and supination, and neutral position. S42. Based on the voltage divider resistance value of the sensor in the extreme state, independently normalize each finger and wrist to obtain a tensile deformation that can reflect the one-to-one mapping between the signal and the hand movement; S43. Using the Leap Motion2 truth value acquisition system to obtain the true value of the hand skeleton; using a cross-correlation analysis method to calculate the global time offset between the signal data acquisition system and the Leap Motion2 truth value acquisition system; using a dynamic time warping algorithm to align the sensor divider resistance value in the extreme state with the true value of the hand skeleton based on the global time offset; by constructing a cumulative distance matrix, extracting the optimal path for time alignment, and establishing a nonlinear time mapping relationship; obtaining aligned data based on the nonlinear time mapping relationship; S44. Resample the aligned data to a uniform 20 Hz time grid through cubic spline interpolation, completing the data stream processing pipeline design process.

4. The flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing according to claim 1 is characterized in that: The S5 inputs the preprocessed dataset into the multi-stage graph attention network for preliminary spatial modeling, and obtains spatial mapping data from 6-channel signals to 15 joints, including: S51, performing standardization processing on the preprocessed data set to obtain a standardized data set; S52. Based on the standardized data set, a multi-layer perceptron is used for mapping to obtain preliminary features; S53. Based on the preliminary features, a graph attention network is used to perform two-stage graph structure processing to obtain spatial mapping data from 6-channel signals to 15 joints.

5. The flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing according to claim 1 is characterized in that: The S6 inputs the spatial mapping data of the 6-channel signal to the 15 joints into the multi-scale Transformer model for time series modeling, and obtains the motion feature data of the 15 joints in time series, including: S61, input the spatial mapping data of the 6-channel signal to the 15 joints into the multi-scale Transformer model, and process it through the four-layer encoder of the multi-scale Transformer model to obtain features of different scales; S62. Information fusion of features of different scales is performed through the cross-attention mechanism to obtain the motion feature data of 15 joints in time series.

6. The flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing according to claim 1 is characterized in that: The S7 decodes the motion feature data of 15 joints in time sequence using a forward kinematics model, completes skeletal quaternion inference, and outputs the rotation quaternions of 20 bones, including: S71. Use a two-layer multi-layer perceptron to process the time-series motion feature data of 15 joints to obtain scalar joint angles. S72, inputting the scalar joint angle into the forward kinematics model to generate a skeletal quaternion; S73. Based on the bone quaternion and the standard hand bone length and joint rotation axis definition, traverse from the palm root node to the end node by node, convert the scalar joint angle into a global rotation, and generate the rotation bone quaternion of 20 bones.

7. A flexible glove and three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing, wherein the flexible glove and three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing is used to implement the flexible glove and three-dimensional hand fine-grained reconstruction method based on sparse stretch sensing as described in any one of claims 1-6, characterized in that: The device comprises: A first building block for constructing a sensor array layout for biological constraint sensing; A design unit, configured to design a wearable glove integrating 6-channel sparse stretch sensing according to the sensor array layout; A second construction unit is configured to construct a three-dimensional multimodal dynamic hand dataset covering 22 degrees of freedom and a total duration of 2400 minutes based on the wearable glove; A data processing unit is configured to design a data stream processing pipeline; input the three-dimensional multimodal dynamic hand data set into the data stream processing pipeline for preprocessing to obtain a preprocessed data set; The first acquisition unit is used to build a multi-stage graph attention network; the preprocessed dataset is input into the multi-stage graph attention network for preliminary spatial modeling, and the spatial mapping data of 6-channel signals to 15 joints is obtained; The second acquisition unit is used to input the spatial mapping data of the 6-channel signal to the 15 joints into the multi-scale Transformer model for time series modeling, and obtain the motion feature data of the 15 joints in time series; The output unit is used to decode the motion feature data of 15 joints in time sequence using a forward kinematics model, complete skeletal quaternion inference, and output the rotation quaternions of 20 bones.

8. The flexible glove and three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing according to claim 1 is characterized in that: The process of integrating a 6-channel sparse stretch sensing wearable glove includes: A resistive strain sensor is selected, with a specification of 15mm×80mm, an initial resistance of 30-40Ω, a stretchability of 0.2%-100%, and a resistance of 70-80Ω after 100% deformation. An Arduino development board is used to connect a 6-channel flexible stretch sensor, a Bluetooth module is used to transmit data, and a 200Ω resistor is connected in series with a single sensor for voltage division to ensure that the sensor reading after voltage division remains between 250 and 308.

9. A flexible glove and three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing, characterized in that: The flexible glove and three-dimensional hand fine-grained reconstruction device based on sparse stretch sensing includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images