Glove-based human-machine interface

Abstract

A wearable human-machine interface (HMI) includes a plurality of triboelectric finger bend sensors configured to sense finger bend, deflection or curling; a triboelectric palm sensor configured to detect applied force and its direction; and at least one mechanical stimulator configured to provide tactile feedback. The human-machine interface is configured to be worn on a human hand and may take the form of a glove.

Description

Technical Field

[0001] The present invention relates to a human-machine interface (HMI) configured to be worn on a human hand. Additionally, but not exclusively, the HMI may be provided in the form of a glove. Background Art

[0002] Currently, the human-machine interface (HMI) between a user and a piece of equipment or a virtual world is a key element to achieve effective, intuitive and seamless operation to complete tasks. As technology advances, HMI is evolving from traditional controls such as keyboards, mice and joysticks to alternatives such as voice control (e.g., for mobile phones and smart home devices) and visual recognition (e.g., for facial recognition and human motion capture). However, some of these technologies have limitations in the application of virtual reality (VR) and augmented reality (AR). In this regard, HMIs in the form of wearable gloves have advantages because they provide high precision and multi-degree-of-freedom (DOFS) control, and can also be used to supplement visual and voice recognition.

[0003] Several parameters can be used to capture information from gestures. For example, finger bending motions as well as lateral direction movements can be detected and measured. Inertial sensors such as accelerometers and gyroscopes are often employed because they provide highly sensitive motion detection, and the available DOF can be increased by increasing the number of sensors. However, additional sensors are required to detect the applied force, which leads to complex and expensive circuits, and requires additional processing power to process different types of signals. Resistive sensors are also often used, enabling the measurement of strain, normal force, and shear force. However, resistive sensors may encounter problems such as temperature effects, zero offset, and creep. Optical sensors have been used to measure finger bending via detecting incident light from diodes, but these also have their limitations. All of these mentioned sensors will consume additional power to generate specific signals. Especially for large-scale distributed sensor networks, power saving becomes a major concern for the long-term sustainability of wearable HMIs.

[0004] Wearable glove-based HMIs can also include feedback mechanisms to provide an immersive experience and enhance sensing capabilities. These mechanisms are often based on technologies such as vibration motors, microfluidic channels, pneumatic chambers, and wire actuators. However, the sheer size of feedback systems utilizing these technologies, as well as the huge power requirements, often limit situations where portability and extended usage time are issues.

[0005] It would therefore be desirable to provide an HMI in the form of a wearable glove that alleviates some or all of the above-mentioned issues and problems. Summary of the invention

[0006] In one aspect, a wearable human-machine interface (HMI) is provided, comprising: a plurality of triboelectric finger bend sensors configured to sense finger bend, deflection or curling; a triboelectric palm sensor configured to detect applied force and its direction; and at least one mechanical stimulator configured to provide tactile feedback. The human-machine interface is configured to be worn on a human hand.

[0007] In a specific example, the interface may further include a wearable housing for supporting the sensor and the stimulator. Preferably, the wearable housing may include a plurality of housing segments, including a palm segment and a plurality of finger segments.

[0008] Preferably, a plurality of triboelectric finger bend sensors may include at least one triboelectric sensor respectively associated with each phalanx of one of the fingers other than the thumb during use, and at least one triboelectric sensor respectively associated with each phalanx of the thumb during use, each of the sensors being configured to detect one or more of bending, curling, and deflection of the associated phalanx.

[0009] Specifically, each of the plurality of triboelectric finger bend sensors may include a contact portion including a raised contact surface and a flat bottom, and an electrode portion embedded in or otherwise attached to the flat bottom, wherein the raised contact surface is preferably hemispherical.

[0010] Preferably, the contact portion of each sensor in the plurality of triboelectric finger bend sensors may include an elastomeric material, and wherein the interface is configured such that in use the elastomeric material can be directly contacted by one of the wearer's fingers other than the thumb or the thumb, the contact generating a triboelectric output. It is envisaged that the plurality of triboelectric finger bend sensors may include sixteen triboelectric finger bend sensors.

[0011] The triboelectric palm sensor may include an upper contact portion, a lower contact portion within the upper contact portion, and an electrode portion configured to be directly contactable by the lower contact portion, the contact generating a triboelectric output. It is possible that the upper contact portion may include a dome, the lower contact portion includes four hemispheres located within the dome, and the electrode portion includes four contact pads, each hemisphere being configured to act as a corresponding contact point for a corresponding contact pad of the electrode portion when an external force is applied to the dome.

[0012] It is also contemplated that the upper and lower contact portions of the palm sensor may include an elastomeric material.

[0013] In a specific example, at least one mechanical stimulator comprises a piezoelectric (PZT) chip. In another example, the interface may comprise five PZT chips, each of which is respectively associated with one of the user's remaining fingers or the thumb in use.

[0014] The interface may include a wiring accessory configured to bias the interface toward an initial position.If the wearable housing includes a plurality of housing segments, including a palm segment and a plurality of finger segments, the wiring accessory may be configured to interconnect the plurality of housing segments.

[0015] It is contemplated that the sensor may be configured to be manufactured at least in part using 3D printing techniques.

[0016] In another aspect, a human-machine interface system is provided, comprising the wearable human-machine interface (HMI) of the first aspect above; a microprocessor configured to receive triboelectric outputs from a plurality of triboelectric finger bend sensors and / or triboelectric palm sensors, and determine the degree of bend of one or more phalanges of the wearer's hand based on the outputs. The microprocessor is also configured to output a signal to actuate vibration of at least one mechanical stimulator.

[0017] In certain embodiments, the microprocessor may be configured to output a signal to at least one mechanical stimulator based on the triboelectric output received from the palm sensor. The microprocessor may be configured to modify the amplitude of the signal to the at least one mechanical stimulator based on the triboelectric output received from the palm sensor.

[0018] Preferably, the microprocessor can be configured to determine the direction of the applied force based on the triboelectric output received from the palm sensor. In a specific example, the microprocessor can be configured to output the triboelectric output from a plurality of triboelectric finger bend sensors and / or triboelectric palm sensors to the neural network.

[0019] In another aspect, a method of object recognition is provided, comprising: receiving, through a neural network, outputs of a plurality of triboelectric finger bend sensors and / or triboelectric palm sensors of the HMI interface of the first aspect described above, the outputs being generated during grasping of an object by a wearer of the HMI interface.

[0020] It should be appreciated that features described in relation to one aspect may also be applicable to other aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 An exemplary pair of glove-based HMIs is shown;

[0022] Figure 2 Figure shows waveform output from an exemplary triboelectric sensor and corresponding finger flexion and return;

[0023] Figure 3 Figure shows an exemplary finger housing (segment) for each phalanx, the location of the triboelectric sensor within the finger housing, and the proximal phalanx;

[0024] Figure 4aFigure 2 shows a comparison of the triboelectric output of exemplary sensors made of different materials (left); Figure 4b Graph showing the relationship between varying loading force and triboelectric output of an exemplary sensor at 2 Hz frequency (right);

[0025] Figure 5 The figure shows the variation of triboelectric output for different sensor sizes for finger sizes of (i) 60 mm and (ii) 50 mm;

[0026] Figure 6 Figure shows the triboelectric output from eight exemplary sensors (DU, DD, MU, MD, PU, ​​PD, PR, PL) with (i) distal phalanx, (ii) middle phalanx, (iii) proximal phalanx, and (iv) all three phalanges (curled) bent, with dashed boxes indicating functional signals;

[0027] Figure 7 Figure shows the triboelectric output from four exemplary sensors (PU, PD, PR, PL) when the finger is bent in eight directions: (i) up, down, left, and right, and (ii) up+left, up+right, down+left, and down+right;

[0028] Figure 8 The figure shows (i) the motion of the proximal phalanx in virtual space and the corresponding positive triboelectric outputs from eight exemplary sensors (DD, MD, PD, DU, MU, PU, ​​PL and PR) on the index finger with three distinguishable bending angles (30°, 60° and 90°), and the subsequent return motion detected by the positive signal of PU for each bending, (ii) the motion of the index finger with three phalanges under DP bending, DP+MP bending and DP+MP+PP bending, and the subsequent return motion detected by the positive signals of DU, MU and PU for each bending, and left / right deflection of the proximal phalanx, with dashed lines and boxes indicating functional signals;

[0029] Fig. 9 The figure shows a schematic diagram of letter writing, as well as photographs of finger movements and corresponding exemplary triboelectric output signals for the following: (1) activating dragging, (2) writing upward and stopping, (3) writing downward + rightward and stopping, (4) writing upward and stopping, (5) activating moving, (6) moving rightward and stopping, (7) activating dragging and (8) writing downward and stopping;

[0030] Fig.10 A schematic diagram illustrating an exemplary fabrication process of an elastomer-based triboelectric finger sensor;

[0031] Fig.11Figures illustrate waveform output from an exemplary triboelectric sensor and corresponding deformation of an exemplary triboelectric palm sensor;

[0032] Fig.12 Figure illustrates components of an exemplary triboelectric palm sensor;

[0033] Fig.13 Graph showing the relationship between varying loading force and triboelectric output from a single exemplary electrode of a palm sensor at a frequency of 2 Hz;

[0034] Fig.14 Figure illustrates the movement of a hand in a virtual space during interaction with an external object in eight directions, and the corresponding output from an exemplary triboelectric palm sensor, with the dashed boxes indicating functional signals;

[0035] Fig.15 Figure shows the triboelectric output from four electrodes (up, down, forward, backward) when the top dome of an exemplary triboelectric palm sensor is pushed in eight directions - (i) up, down, forward, and backward and (ii) down+forward, up+backward, down+backward, and up+forward, with negative signals in dashed boxes indicating activated sensors;

[0036] Fig.16 Schematic diagram showing an exemplary fabrication process of an elastomer-based triboelectric palm sensor: (top) top contact and (bottom) back view of the top contact, and top view of the bottom aluminum electrode;

[0037] Fig.17 An exemplary piezoelectric (PZT) tactile stimulator is illustrated;

[0038] Fig.18 The figure shows stimulation at a frequency of 270 Hz for adjustable input AC power (6 V, 8 V and 10 V) using the second PZT chip;

[0039] Fig.19 Figure 1 shows a demonstration of real-time haptic feedback in response to interactive events in Unity, and measured output of mechanical stimulation.

[0040] Fig. 20 A schematic diagram illustrating an exemplary manufacturing process of a PZT actuator for haptic feedback is shown;

[0041] Fig.21 A schematic diagram illustrating an exemplary process and parameters for constructing a CNN machine learning technique;

[0042] Fig. 22The figure shows the triboelectric output from the following 16 exemplary triboelectric sensors during grasping of six objects including a rod, a cube, a disk, a curved object, a pyramid, and a small cube: thumb (DD and DU), index finger (DD, MD, PD, and DU), middle finger (DD, MD, and PD), ring finger (DD, MD, and PD), little finger (DD, MD, and PD), and palm (single electrode), with the solid line box indicating the feature pattern used for recognition training;

[0043] Fig.23 The figure shows confusion plots for object recognition derived from two models made by (i) CNN and (ii) SVM, with 100 tests for each object and 40 tests for the baseline, the output class refers to the recognition result, and the target class refers to the real object.

[0044] FIG. 24 illustrates a comprehensive demonstration of an exemplary glove-based HMI in a baseball game program, including grabbing a bat ( Fig.24a (i)), tap ( Figure 24b (i)) and heavy attack ( Fig.24c (i)) and the output voltage measured from the tactile stimulation induced by the piezoelectric stimulator of the glove via the second piezoelectric chip ( Fig.24a (ii), b(ii), c(ii)), triboelectric output measured from the four electrodes of the palm sensor of the glove ( Fig.24a (iii), b(iii), c(iii)), and schematic diagrams of the corresponding strength of the interaction force applied to each palm sensor electrode ( Fig.24a (iv), b(iv), c(iv)), Figure 24b (iv), c(iv) The arrow marks indicate the direction of the shear force defined by the distribution of the non-uniform force;

[0045] Fig.25 A flow chart illustrating an exemplary operation of the baseball program of FIG. 24;

[0046] Fig.26 The figure shows a comprehensive demonstration of an exemplary glove-based HMI in a VR surgical training program, including illustrations of the functions of the glove in control mode and recognition mode; illustrations of finger movements for implementing the following: (I) mode switching, (II) (III) movement of the right hand, (IV) display of the left hand, (V) recognition of scissors, (VI) recognition of the knife, (VII) recognition of gauze, (VIII) operation of scissors (cut), (IX) operation of the knife (cut) and (X) operation of gauze (brush);

[0047] Fig. 27 The figure shows that in (i) control mode Fig.26triboelectric outputs of the following eight exemplary sensors used in the demonstration of (i) the thumb (DD and DU), index finger (DD and DU), middle finger (DD and DU), and ring finger (DD and DU), and triboelectric outputs from the following 16 exemplary sensors in the recognition mode for the motion of grasping scissors, knife, and gauze: thumb (DD and DU), index finger (DD, MD, PD, and DU), middle finger (DD, MD, and PD), ring finger (DD, MD, and PD), little finger (DD, MD, and PD), and palm (single electrode);

[0048] Fig.28 The graphs show (A) output changes of an exemplary triboelectric sensor under changing humidity, and (B) output changes of an exemplary triboelectric sensor under long-term use and after flushing;

[0049] Fig.29 Figure shows individual signal inputs of an exemplary triboelectric sensor under multi-channel reading, (A) without the triboelectric regulator PCB attached to the Arduino microprocessor, (B) with the triboelectric regulator PCB;

[0050] Fig.30 and Fig.31 The diagrams are respectively used to implement ( Fig.30 ) Example of virtual hand manipulation based on triboelectric sensors and ( Fig.31 )Core functionality of the Arduino PCB and Unity software for haptic feedback from virtual events such as collision or touch;

[0051] Fig.32 The figure shows the change in accuracy relative to the training epoch. The upper training acc curve is the training accuracy, and the lower val acc curve is the validation accuracy.

[0052] Fig.33 Figure 1 shows a table of parameters used to construct a convolutional neural network (CNN); and

[0053] Fig.34 The figure shows the SVM classification accuracy under different parameters and dimensions (Note: the accuracy data is the average of 10 runs for each parameter). DETAILED DESCRIPTION

[0054] References Figures 1 to 34 A simple, cost-effective, and intuitive HMI in the form of a wearable "smart" glove is described. The glove-based HMI (hereafter referred to as the glove) can provide multi-dimensional motion detection and real-time tactile feedback.

[0055] In one example, machine learning techniques can be used to expand the functionality of gloves with a very small number of sensors, enabling complex tasks such as object recognition. Alternatively or additionally, for AR applications, gloves can act as a supplementary control interface for traditional visual and voice control terminals, providing enhanced interaction. Gloves can also have applications in fields such as industrial production, education and training, entertainment, and home care.

[0056] like Figure 1 As shown, the glove 10 includes a triboelectric sensor, which may be elastomer-based, and one or more piezoelectric (PZT) stimulators, which may include lead zirconate titanate. In one example, these components form three main functional units of the glove: triboelectric finger bend sensors 20a, b, c; triboelectric palm sensor 30; and at least one piezoelectric mechanical stimulator 40. These three units will be discussed in more detail below. The sensors are supported by (e.g., attached to) a glove shell 50, which typically includes four shells for one of the remaining fingers other than the thumb, a thumb shell, and front and rear (palm) shells. In one example, each shell for one of the remaining fingers other than the thumb and the thumb shell are formed by a plurality of connected shell segments, typically three segments for each of the remaining fingers other than the thumb (one for each phalanx) and two segments for the thumb. The glove shell may be integrally formed, or may include multiple shell portions that may be combined to form the glove shell. In one example, the glove shell may be suitable for being 3D printed. A microcontroller for receiving, sending, and processing signals from / to the sensors of the glove may communicate data with the glove (e.g., wired, wireless). The microcontroller may be integrated into the glove or may be provided separately, or a combination thereof. A wiring fitting, preferably comprising a metal alloy, connects the glove shell(s) / segments and provides a restoring force.

[0057] In the following discussion, it will be understood that the gloves can be configured so that they can be worn on the right or left hand, and can be provided in pairs, such as Figure 1 In this specification, reference to a glove applies equally to a pair of gloves (ie, a right-hand glove and a left-hand glove).

[0058] The features of the embodiments and examples described herein may be combined or substituted without restriction.

[0059] Triboelectric Finger Bend Sensor

[0060] like Figure 2 As shown, the triboelectric finger bend sensor of the glove-based HMI can detect the movement of each phalanx with multiple DOF. In this illustrated example, each finger bend sensor includes a contact portion and an electrode portion, the contact portion including a raised contact surface and a flat bottom, the electrode portion being embedded in or otherwise attached to the flat bottom.

[0061] For example, each contact surface may be hemispherical in shape. Although other shapes such as pyramidal, cubic, trapezoidal, etc. may be envisioned, hemispherical or dome shapes are used in the examples described herein. Each contact portion may be made of a soft elastomer (e.g., Eco-flex TM (00-20), including platinum-catalyzed silicones (www.smooth-on.com / products / ecoflex-00-20 / ). The electrode portion may be in the form of a thin pad, preferably made of aluminum.

[0062] The finger sensors may be bonded or otherwise attached to the inner surface of at least one finger shell of the glove shell. The finger sensors are arranged as follows: Figure 3 The available DOF determinations for the following three parts of a human finger are shown: the distal phalanx (DP) closest to the nail, the middle phalanx (MP), and the proximal phalanx (PP) closest to the hand. As shown, both the DP and MP can only move up and down (DU, DD, MU, MD), while the PP can deflect in other directions, such as up, down, left, and right (PU, PL, PR, PD). In one example, a finger sensor arrangement (i.e., a finger sensor assembly) can include 2 DP, 2 MP, and 4 PP finger sensors in at least one finger housing. Of course, depending on the intended application, it may be desirable to: have a greater or lesser number of sensors in each finger housing; have the same number of sensors in each finger housing; have a different number of sensors in two or more finger housings; or have at least one finger housing without a sensor. In other words, the distribution of sensors is highly customizable, depending on the sensing requirements and the degrees of freedom.

[0063] Typically, the glove has four housings for one of the fingers other than the thumb, as follows: index finger, middle finger, ring finger, and pinky finger. The glove may also have a thumb housing. As previously described, each housing may be formed from multiple segments. For certain applications, alternative arrangements of one of the fingers other than the thumb and the thumb housing may be provided. In this specification, references to a bend sensor for one of the fingers other than the thumb may also apply to a triboelectric sensor configured to measure / detect the bend of the thumb. Typically, the finger bend sensor described herein may detect the bend, curl, deflection, or other movement of the fingers other than the thumb or the thumb.

[0064] The triboelectric output is generated by contact electrification during the surface interaction between two materials with different electronegativity. For example, it is possible to utilize two external materials to perform the contact and separation of an arched structure, or the sliding of a grid-style strip. In contrast, for the finger bend sensor of the glove-based HMI described herein, the skin of the glove wearer is used as a positive triboelectric material, which interacts with the triboelectric finger bend sensor forming a negative material. The triboelectric finger bend sensor described herein is capable of outputting a self-generated signal, thereby significantly reducing the power consumption of the glove-based HMI.

[0065] Return to reference Figure 1 , triboelectric sensors typically output one of two opposite pulse waveforms corresponding to contact or separation cycles, depending on which side the output signal is extracted from. In the gloves described herein, the triboelectric finger sensor relies on the interaction between the sensor contact surface (e.g., elastomer) and the skin of the finger. In the non-contact state ( Figure 1 Detail 1), the finger sensor is in a neutral state, where the negative charge of the contact surface is balanced by the positive charge from the embedded electrode pads. When the skin of the finger contacts the sensor surface (e.g., the curved part of the hemisphere) ( Figure 1 2), the positive charge from the skin surface neutralizes the negative potential at the sensor contact surface and causes the previous positive charge on the electrode pad to be repelled to ground. This is because the skin is generally much more positively charged than the sensor contact surface (e.g., elastomer). As a result, when the output signal is measured from the electrode pad, a positive pulse waveform is observed due to the direction of the current flow. In contrast, the re-separation between the finger skin and the sensor contact surface ( Figure 1 Details 3 and 4) in will cause positive charge to flow back to the electrode pad to neutralize the negative sensor contact surface, thereby generating a negative pulse waveform.

[0066] In other words, when the user of the glove bends their fingers and / or thumb, the triboelectric finger bend sensors at different locations supported by the glove shell come into contact and out of contact with the skin of the user's fingers / thumbs. As a result, the pulse waveform output by the sensor changes from positive to negative (or vice versa). This change in output can be used to determine which one of the fingers / thumbs is being bent and in which direction.

[0067] The glove shell can be made of different types of elastomers. Figure 4a As shown in the material comparison test in , for the same 4 mm diameter hemisphere, the MP finger bending triboelectric sensor output in millivolts (mV) of Eco-flex elastomer (about 200 mV) is much greater than the output of the sensor made of polydimethylsiloxane (PDMS) elastomer. Both are biocompatible materials, which ensures signal sensitivity and comfortable long-term use. Figure 4a The test results shown indicate that forming the finger bend sensor contact portion from Eco-flex elastomer provides more accurate sensing of finger bend. According to the triboelectric theorem, the output generated is primarily determined by the contact area for the same material due to the amount of interacting surface charges.

[0068] Characterizes the finger sensor output response to applied force at 2 Hz ( Figure 4b ). For a hemispherical finger bend sensor with a diameter of 3.5 mm, a typical sensing range from 0.1 N to 3.5 N (from left to right) at 1.5 Hz compression was obtained. This range can also be considered as a reference for choosing the stiffness of the connecting wires (wiring accessories) that exert a restoring force during finger bending, i.e. biasing the housing(s) towards the initial (unbent) position. Figure 5 As shown, the effect of the hemispherical sensor contact surface diameter is studied using samples with three different hemispherical diameters: 3mm (left), 3.5mm (middle), and 4mm (right) for finger sizes with circumferences of (i) 60mm and (ii) 50mm. The output shows an increase in ( Figure 5 ).

[0069] It will be appreciated that users may have a range of finger sizes. For a particular size finger housing and user, such as Figure 5 (i) and Figure 5 (ii), there is an optimal configuration for the highest output. In other words, the diameter of the finger bend sensor can be adjusted according to the finger size. The limited internal space of the glove shell will set constraints on the contact and separation process, that is, larger fingers will experience less contact and separation motions, resulting in less surface charge interactions. In the case where the glove described in this article is composed of separate finger shells, each joint of the glove can be replaced and customized to achieve the desired performance. Typically, for the experiments described in this article, a finger bend sensor hemisphere diameter of 3.5 mm was used.

[0070] like Figure 6 As shown, the index finger was tested with a series of common movements to investigate the sensing capabilities of the triboelectric finger bend sensor. Based on the three segments of the index finger, finger bend sensors corresponding to the upper and lower bends of the distal phalanx (DP), middle phalanx (MP), and proximal phalanx (PP) were provided. These finger sensors are labeled DU (i.e., upper distal phalanx), DD (i.e., lower distal phalanx), MU (i.e., upper middle phalanx), MD (i.e., lower middle phalanx), PU (upper proximal phalanx), and PD (lower proximal phalanx), respectively. In addition, left and right bend finger sensors PL and PR were attached for the proximal phalanx. For the test of a single phalanx ( Figure 6(i), (ii) and (iii)), the signals of the main functional sensors are highlighted in dotted boxes. These functional signals refer to the pulse-shaped output voltage peaks (positive / upward or negative / downward) marked by the dotted boxes. These signals are used to identify whether a specific sensor is triggered and what the triggering status is. For example, a positive peak can indicate contact / pressing, and a negative peak can indicate separation / release.

[0071] exist Figure 6 In each of (i), (ii), (iii) and (iv), the sensor outputs are illustrated from top to bottom as follows: PD, PU, ​​MD, MU, DD, DU.

[0072] To ensure that the distinction between different bend angles (30°, 60°, and 90°) can be determined using the triboelectric finger bend sensor, the primary resolution (i.e., the difference between bend angles) for each phalanx was set to 30° based on the acquired data with a reasonable data error tolerance. The actual range of motion of the distal phalanx for a human finger has only two angles (30° and 60°). Figure 6 As shown in the data, for the middle phalanx and the proximal phalanx, both have three bending angles (30°, 60° and 90°), during bending, smaller signals are detected from the DD (distal phalanx) and MD (middle phalanx) sensors, and larger signals are detected from the main functional sensor (PD (proximal phalanx) sensor), especially for the larger bending angle (90°).

[0073] This phenomenon is caused by the controllability of the actual movement of a single phalanx of a particular user, and it is often observed that during the flexion of the middle or proximal phalanx, the front phalanx (PD) will also bend unintentionally. This unintentional flexion is detected by the finger flexion sensor, generating a small signal output.

[0074] like Figure 6 As shown in (iv), as an advantage of using multiple finger bending sensors, the curling movement of the finger can be distinguished from the bending movement of the proximal phalanx. Figure 6 From the data of the three dashed boxes in (iv), it is apparent that both DD and MD outputs for finger curling show signals above the activity threshold, and only insignificant outputs are observed for DD and MD in bending motion.

[0075] Generally, the joints of the proximal phalanges have more DOF than the other phalanges. Figure 7There are eight directions shown in FIG, including (i) four standard directions of up, down, left, and right, and (ii) four diagonal directions of upper right, lower left, upper left, and lower right. In this example, the outputs from all four finger bending triboelectric sensors are measured simultaneously, so that the signals from theoretically inactive (i.e., non-contact) sensors can be monitored for potential interference because the finger is almost not in contact with all sensors in the original state.

[0076] Figure 7 The measurement data in indicate that there is almost no interference between the two pairs of counter sensors PR / PL and PU / PD. In other words, even though the finger may slide across the surface of the PU / PD sensor when performing a left-right motion, the contact surface area of ​​the PU / PD sensor actually results in very low triboelectric charge transfer. For the four diagonal directions ( Figure 7 (ii)), two adjacent sensors can be used to detect each direction. Since the contact force is distributed on two sensors, it is also visible that the sensor output drops from ∼150 mV to less than 100 mV.

[0077] Figure 8 The figure shows a typical example of index finger bending. In this example, the real-time signal waveforms of eight finger bending triboelectric sensors are monitored. As mentioned earlier, the thresholds for specific bending angles (30°, 60°, 90°) are calibrated for the sensors of each phalanx, as shown by the black dashed lines on the signal of the PD. At 30°, when the proximal phalanx is bent and the PD is pressed, a positive triboelectric signal from the PD is generated, and a negative signal is also recorded on the PU due to the separation event. The return movement of the proximal phalanx causes the pressing of the PU and induces a positive signal, while the separation of the PD produces a negative signal. In the exemplary demonstration program, once the sensor output signal reaches the threshold ( Figure 8 (i)), a program run by a microprocessor communicating with the sensor will output the corresponding code, for example, “0,0,1,0,0,0,0,0” and “0,0,2,0,0,0,0,0,0” for 30° and 60° bending of the PD, respectively, where the position of each number represents a different sensor.

[0078] Processing software such as the Unity program can receive the code output as a command by the microprocessor, for example, via a communication protocol of a serial port, and convert it into the movement of a virtual hand in a virtual space. Alternatively, after the sensing signal is processed by the microprocessor, the command code can be used for any kind of control terminal, Unity is just a game / demo development software, other similar engines include Unreal, etc. On the other hand, the code can be sent to a servo motor to control a robotic hand, alternatively or additionally.

[0079] For verification of multi-DOF manipulation ( Figure 8(ii)), the functional signals are marked with dotted boxes, and all upper finger bend sensors DU, MU and PU are responsible for the return motion. Based on the real-time sensor signal waveform, each bending activity is performed in a cycle of bending and returning. For example, in the case of a curling motion (bending of DP+MP+PP), all three finger bend sensors are triggered, the DP and MP sensors reach a threshold of 60°, and the PP reaches 30°. This curling motion is projected onto the virtual hand. Then, all three return signals from the DU, MU and PU sensors are activated to set the finger back to its original position. A unified low threshold is set for these upper sensors to simplify the return process.

[0080] The glove described herein can also provide "mouse control" by converting the index finger into a virtual pointer to mimic the movement of a cursor. This is demonstrated using online shopping and letter writing (see Fig. 9 ). In addition to the eight directions of motion determined using the four finger bend sensors of the proximal phalanx of the index finger, the functions of scrolling, dragging, and clicking are assigned to the middle finger and thumb, respectively. Fig. 9 As shown, the drag mode is initiated by the positive peak generated by the bending of the thumb detected by DD (step 1: activate drag). The index finger is then used to write letters (step 2: write upward and stop). For the letter writing process, the positive and negative peaks output from the finger bending sensor corresponding to the bending and releasing actions are responsible for the start and stop commands respectively, and diagonal writing (step 3: write down + right and stop; step 4 write upward and stop) is achieved by pressing both the PD and PR sensors based on the writing movement of the finger. After completing the writing of the first letter (N), the thumb then returns and presses DU to start the movement mode of the cursor (step 5: activate movement), where the movement is again controlled by the index finger (step 6: move right and stop; step 7: activate drag; step 8: write down and stop).

[0081] Fabrication of an Exemplary Elastomer-Based Triboelectric Finger Sensor

[0082] In a non-limiting and exemplary method of manufacture, Fig.10As shown, three 3D printed hemispherical molds (e.g., made using Anycubic 4Max Pro printer) are prepared, each with a different hemispherical diameter (i.e., 3 mm, 3.5 mm, and 4 mm). A mixture of solutions A and B, both of which are Eco-flex (ratio: 1:1, model: 00-20) is poured into each mold and cured at room temperature for 90 minutes to form a hemispherical contact portion of the finger bending triboelectric sensor. Aluminum foil (e.g., with a thickness of about 50 to 100 μm) is cut into a circle to accommodate the size of each hemisphere. The foil is joined with a lead (e.g., with a diameter of about 0.2-0.5 mm) to form an embedded electrode portion of an exemplary finger bending sensor. The electrode is then attached to the flat bottom of the cured Eco-flex hemisphere and covered with an additional mixture of Eco-flex. After the second cure, the manufactured elastomeric finger bending sensor is released from the mold and bonded to the inner surface of the glove finger shell using an adhesive epoxy resin or the like. Finger bend sensors made of PDMS (e.g. Sylgard 184, Dow Corning) were fabricated using the same approach as described above for comparison purposes, using a 10:1 ratio for the base and curing agent mixture and a curing temperature of 70°C for 60 minutes as described above.

[0083] Triboelectric Palm Sensor

[0084] Another triboelectric sensor is located approximately in the center of the palm portion of the glove-based HMI, which can be used to sense normal forces and shear (sliding) forces in eight directions caused by contact with an external object. Unlike the finger bend sensor described above, the electrode portion for the palm sensor is provided with four quadratic segments or pads. These segments are separated so as to clearly identify the direction of the external force. In further examples, more than four segments may be provided, thereby increasing the number of detectable directions.

[0085] exist Fig.11 In one example shown, the upper contact portion of the palm sensor (also referred to herein as a membrane or upper dome) is made of a polydimethylsiloxane (PDMS) elastomer. The contact portion has a large dome shape of about 15 mm in diameter. In addition, four smaller lower hemispheres (each having a diameter of about 3 mm) are provided within the upper dome, which form the lower contact portion, i.e., the bottom layer, of the palm sensor. The four smaller lower hemispheres serve as corresponding contact points ( Fig.12 ). The four electrode pads are separated from the upper contact portion (i.e., upper dome) of the palm sensor by about 3 mm.

[0086] The triboelectric palm sensor utilizes an elastomer and an electrode instead of the skin and electrode of the finger as the negative triboelectric material and the positive triboelectric material, respectively. During the contact between the electrode and the elastomer, the electrode will extract positive charge from the ground to balance the negative charge of the elastomer from the smaller hemisphere. Therefore, a negative output is generated instead of the positive output caused during contact of the above-mentioned triboelectric finger sensor. This difference in the signals output from the triboelectric palm sensor and the triboelectric finger sensor may be useful for various applications.

[0087] For palm sensors, such as Fig.11 As shown, an external impact to the top of the large upper dome will cause deformation of the dome, and thus of at least one smaller lower hemisphere within the dome. Fig.12 As shown, typically the dome portion of the sensor faces outward from the wearer's palm, while the electrode portion is located closest to the user's palm.

[0088] In the case where the external impact is a shear force (i.e., a forward sliding motion), the direction of deformation of the upper dome will cause the forward lower hemisphere to deform more than the other three lower hemispheres, and will result in a larger triboelectric output from the corresponding electrode (i.e., the electrode in contact with the deformed lower hemisphere), providing an indication of the direction of the shear force.

[0089] The normal force (i.e., downward motion) can be determined from the magnitude of the negative triboelectric output, as Fig.13 As shown, it illustrates the relationship between the loading force (N) changing at a frequency of 2Hz and the triboelectric output from a single electrode. Since most "normal" forces do not contact the sensor in a purely normal direction (i.e., exactly ninety degrees), the four-quadratic design of the exemplary triboelectric palm sensor can be used for simultaneous measurement of all four outputs to identify the lateral direction of the applied force, i.e., the shear force as a decomposed portion of the "normal" force. Similarly, a typical forward push (shear) force will generate not only a signal from the forward sensor (electrode), but also some smaller signals from the other three sensors (electrodes). To this end, four standard directions are defined based on straight fingers pointing forward and the center of the palm pointing to the left or right: up, down, forward, and backward. Fig.15 The output data in indicates that a total of 8 directions ((i) up, down, forward and backward; (ii) down + forward, up + backward; down + backward and up + forward) can be detected, including the four standard directions sensed individually by each quadrant of the electrode portion, and the other four diagonal directions measured by two adjacent quadrants. When processing the output signal, a comparison code for the triboelectric outputs from the four electrodes is implemented to help identify the direction of the force, that is, if one or two outputs show a much larger amplitude than the other outputs, the direction judgment is performed by the microprocessor and / or processing software.

[0090] exist Fig.14 In, a demonstration using a virtual hand in a virtual space shows the functionality for defining an entire hand motion by sliding the palm sensor across an external object. For example, a downward slide of the palm sensor causes contact of the electrode responsible for downward sensing and separation of the electrode for upward sensing on the opposite side, and results in one negative signal and one positive signal, respectively. For diagonal motion, such as up+backward, the slide causes contact of two adjacent electrode portions for sensing up and backward motion, and separation from two opposite electrodes, and thus generates two positive signals and two negative signals. The characterization data is tested with a pure lateral force applied on the top of the dome in order to obtain a characteristic output.

[0091] In addition to identifying hand finger movements, detection of external impacts helps enhance the user experience of VR / AR applications.

[0092] Fabrication of an Exemplary Elastomer-Based Palm Sensor

[0093] In one exemplary and non-limiting method of manufacture, see Fig.16 , two 3D printed molds (e.g., made using Anycubic 4Max Pro printer) are prepared for the upper contact portion (i.e., upper dome or button) of the palm sensor and for the four triboelectric contact points (hemispheres) forming the lower contact portion. A mixture of solutions A and B of Eco-flex elastomer (ratio: 1:1, model: 00-20) is poured into the mold for the four hemispherical contact points, and a mixture of PDMS (ratio: 10:1, Sylgard184, Dow Corning) is poured into the mold for the upper dome. After curing, the upper dome is released and placed on top of the four cured hemispherical contact points. An additional mixture of PDMS is then poured on the two parts (i.e., the dome and the four contact hemispherical points) to bond them together. After the second curing, the entire unit is released from the mold. The bracket for the four bottom electrode pads is 3D printed (e.g., using Anycubic 4Max Pro printer). The circular aluminum foil is cut into four quarter circles and bonded with wires and attached to the bracket.

[0094] Piezoelectric Mechanical Stimulator

[0095] At least one piezoelectric (PZT) tactile mechanical stimulator is also included in the glove-based HMI described herein. In a non-limiting example, five PZT stimulators are preferably included in the glove in the form of thin PZT chips. The PZT chip can be placed at the root of each finger shell / shell segment and the root of the thumb shell / shell segment to provide tactile feedback to the wearer of the glove. It will be understood that, depending on the specific application, additional or fewer PZT stimulators may be included in the glove. For example, the thumb shell / segment and / or one or more finger shells / segments other than the thumb may have more than one PZT stimulator, or there may be no PZT stimulator.

[0096] To perform tactile mechanical stimulation, the inverse piezoelectric effect is applied to the PZT stimulator. Once an interaction event is triggered (i.e., a normal force or shear force is applied to the glove, and / or a finger bends / curls / deflects), a microcontroller in data communication with the glove will send a pulse width modulation (PWM) signal with a resonant frequency to actuate the vibration of one or more PZT stimulators. In one example, the intensity of the vibration (i.e., mechanical stimulation) can be adjusted to reflect the degree or intensity of the interaction. The interaction event can be customized according to the specific application.

[0097] The intensity of the mechanical stimulation provided by the PZT stimulator can be adjusted by controlling the input voltage from 6V to 12V (e.g., providing voltages of 6V, 8V, and 10V). The resonant frequency is determined to be 270Hz based on the size of the PZT chip. Fig.17 As shown in the example of , an exemplary PZT chip can be attached to a portion of the glove shell (e.g., one of the remaining fingers other than the thumb and the palm portion to which the thumb shell can be attached, as shown). The exemplary chip can have a length of about 1.5 cm, a width of about 0.8 cm, and can have a thickness of about 100 μm. Of course, the size and position of each PZT chip can vary depending on the specific application. In the case where more than one PZT chip is provided in the glove shell, the multiple PZT chips do not have to be of the same size.

[0098] In one example, if Fig.18 As shown, three levels of mechanical stimulation intensity (6V, 8V, 10V) can be set for activation of mechanical tactile stimulation. In this example, two PZT chips are bonded together, and one PZT chip is used to convert the mechanical stimulation generated by the other PZT chip back to electricity via the piezoelectric effect. This enables real-time stimulation to be visualized and also provides a check on the stimulation intensity.

[0099] like Fig.19As shown, processing software (in this case, the Unity software crash reactor module) running or communicating with a microprocessor linked to the glove enables delivery of virtual collision events back to the microcontroller via a serial communication port and activation of mechanical stimulation via a programmed PWM input. As a result, contact and release events between the glove and an external object (real or virtual) are generated (see Fig.19 ) can be immediately fed back to the user of the glove via vibrations of a PZT mechanical stimulator with adjustable intensity to distinguish between events of varying degrees (e.g., light, medium, and strong events). In one example, feedback can be delivered to the user (i.e., the wearer of the glove) through one or more of the multiple PZT mechanical stimulators.

[0100] Fabrication of an exemplary PZT chip

[0101] In one exemplary and non-limiting method of manufacture, see Fig. 20 , 1.5cm×1.5cm lead zirconate titanate (PZT) ceramic (e.g., Fuji Ceramics Incorporation, C-591 6) and 50μm beryllium copper foil are polished. After sputtering chromium / gold (chromium: 20nm, 100W, 2mins; gold: 200nm; 100W; 5mins) to form a bottom electrode on one side (bottom surface) of the PZT ceramic, the electrode is bonded to the copper foil with a conductive silver paste and baked in a vacuum oven (175°C, 3.5 hours). Then, the bonded PZT chip is thinned to 20μm by chemical mechanical polishing. The top Au electrode on the other side (top surface) of the PZT ceramic is then sputtered using the same method. Afterwards, the thinned PZT chip thus made is further cut into a size of 8mm×5mm by laser cutting. The leads are then connected to both the top and bottom electrodes using silver paste, and the PZT chip is subsequently encapsulated using polyimide tape (e.g., 3M).

[0102] Exemplary Glove Shell Fabrication and Assembly

[0103] In one non-limiting example, one of the remaining fingers of the glove other than the thumb / thumb shell and the palm shell (e.g., see Fig.17) are manufactured separately, for example, using 3D printing, optionally using an Anycubic 4Max Pro printer, preferably using polylactic acid (PLA). The printed shell is then polished in preparation for the engagement of the various sensors. In order to assemble the separate one of the fingers other than the thumb / thumb shell with the palm shell, a metal alloy wire with a certain restoring force is used as a connector so that the entire assembled glove shell can be supported during the bending of the one of the fingers other than the thumb / thumb, thereby enabling the corresponding finger bending and palm sensor deformation. It will be understood that the various sensors and feedback devices can be attached or engaged to the glove shell in various ways. Although the feedback device is usually attached to the inside of the glove shell element, this is not necessarily the case. In a further example, the glove shell can be formed as a whole. Additional shells or coverings can be added to the gloves to provide protection for the sensors and / or feedback devices, to provide comfort to the user or for aesthetics. The sensors and / or feedback devices can be assembled into one or more accessories before or after being attached to the (multiple) glove shells. While various elastomers have been described for making the sensor / glove shell(s), it will be appreciated that this is not intended to be limiting and other suitable materials or combinations of materials may be used.

[0104] In the examples described herein, the output voltages of the triboelectric sensors (finger and palm) and the PZT chip are typically measured by an oscilloscope (e.g. Agilent, InfiniVision, DSO-X 3034A). The calibration of the output voltage versus force for the triboelectric sensor is performed by a dynamometer (e.g. Mecmesin, MultiTest 2.5-i) at a speed of 900 mm / min.

[0105] Object recognition based on machine learning

[0106] As mentioned earlier, machine learning techniques can be used to expand the functionality of the glove, enabling complex tasks such as object recognition.

[0107] In one example, sixteen triboelectric sensors of a right hand glove were monitored simultaneously for machine learning-based object recognition using triboelectric output. Based on typical finger usage for general tasks, the sixteen sensors included the thumb (DD and DU), index finger (DD, MD, PD, and DU), middle finger (DD, MD, and PD), ring finger (DD, MD, and PD), pinky finger (DD, MD, and PD), and palm, with the thumb and index finger being the most commonly used fingers. Six objects with various shapes (i.e., rod, cube, disk, curved structure / moon, pyramid, and small cube) were selected for grasping, while recording the triboelectric sensor outputs from the palm and finger bend sensors.

[0108] The combined data sets from the sixteen sensors (channels) can form a spectrum to provide sufficient features to be automatically extracted using machine learning, that is, the raw data containing various important information related to the dynamic grasping process (e.g., finger bending speed, contact force between the finger and the sensor array, the trigger sequence of the sensor, and the operation mode of the specific user) can be directly fed into the training model (neural network) without the need for pre-processing steps (e.g., normalization and segmentation, etc.). In other words, the general features of the signal patterns across all sixteen sensors are used as direct input to the machine learning technique for object recognition.

[0109] Convolutional Neural Networks (CNNs) are a very effective way to derive interesting features from shorter (fixed-length) segments. Fig.21 The process and parameters used to construct the CNN model are shown in Figure 1. Further details can be found in Fig.33 The constructed CNN model includes three convolutional layers with 32 / 64 / 128 filters and three pooling layers. For the operation of each layer, the parameter k refers to the kernel size and s refers to the stride. For training sample collection, each of the sixteen triboelectric sensors mentioned above has 200 data points recorded by sample to train the model for object recognition. For each object, 500 samples were collected for training (80%) and testing (20%).

[0110] Fig. 22 Typical triboelectric outputs from sixteen sensors (thumb (DD and DU), index finger (DD, MD, PD and DU), middle finger (DD, MD and PD), ring finger (DD, MD and PD), little finger (DD, MD and PD) and palm (single electrode) for grasping all six objects) are shown, where the characteristic patterns are marked with black dashed boxes. The characteristic pattern refers to the entire waveform marked in the dashed box, which can be used for feature extraction in machine learning algorithms. These patterns are very unique between different objects and can therefore be regarded as identities for recognition. Due to the limitation of the number of sensors, the basic working mechanism of object recognition is not completely defined by the detection of object shape, and anyway the human hand cannot usually cover the entire object. In other words, the output from the glove actually indicates the grasping habits of a specific user for a specific object. For example, a participant can use five fingers to grasp a cube, but use three fingers to grasp a small cube, which will result in additional outputs from the triboelectric finger bending sensors of the ring finger and little finger for the larger cube.

[0111] As another commonly used machine learning method, support vector machine (SVM) has been widely used as a classifier in various applications of pattern recognition. Based on optimization experiments using linear kernel and radial basis kernel for classification, as well as computational power issues, the data dimension was set to 300 to achieve sufficient recognition accuracy (see Fig.34 ). The data dimension refers to the number of valid data points for each sample, which is set to 300 in this example. For each sample, there are 16 input channels and 200 data points per channel. Therefore, the original data dimension is 3200. To reduce the data dimension, the principal component analysis (PCA) method is used to extract features and reduce the dimension of the data to 300.

[0112] Fig.23 The figure shows confusion plots for object recognition derived from two models made by (i) CNN and (ii) SVM, where the output class refers to the recognition result and the target class refers to the real object for 100 tests per object and 40 tests for the baseline. The output class (output object) is the output result (identified identity) of the object recognition algorithm (CNN or SVM) based on the sensory data of the grasped real object, and the target class (target object) is the identity of the real / real object. If the recognition result is the same as the real object, then this result can be counted as a true recognition. Otherwise, the result is a false recognition. For Fig.23 In the example in (i), for the real cube object, there are 99 output results that correctly show the recognition of the cube, and only 1 incorrect result gives the identity of the small cube. Therefore, the accuracy of cube recognition is 99%. In this figure, the accuracies of all correct recognitions are aligned in the diagonal direction. The average accuracy is just the overall accuracy of this machine learning algorithm. For each object / class (a total of 6 objects / classes, except for the baseline / no object), 100 grasps are performed, and the sensing data of these 100 grasps is then fed to the CNN or SVM algorithm for recognition. The output result of each object (100 grasps) refers to the recognition result. The target class / object refers to the true identity of the object / class. A confusion map can be drawn to see how many of the recognition results of each object / class match the real object / class.

[0113] Fig.23 The confusion plots for the models in Figure 3 show that both approaches (CNN and SVM) can help the glove-based HMI achieve object recognition with over 96% accuracy using training samples for each object. A few objects showed slightly lower accuracy, such as 94% accuracy for poles and pyramids from the CNN model ( Fig.23 (i)), four outputs fall into the cube, and three outputs fall into the curved structure. Fig. 22In the triboelectric output patterns of the graspers, certain similarities can be found between them. For the rod and cube, the main difference is the signal from the palm sensor, which seems to lead to erroneous object recognition if this signal is too small during grasping. Similarly, for the curved and pyramid structures, the signals from the ring and pinky fingers are the main features, and the abnormal amplitude of the signal from the pyramid grasp may also lead to erroneous object recognition. In the SVM model ( Fig.23 (ii)), the pyramid recognition achieved an accuracy of 91% because six samples were identified as curved for the same reason. Therefore, in order to ensure the stability of the performance for irregular human motion, in addition to increasing the number of samples and adjusting the grasping force, it is also helpful to change the position of the signal in the time domain to cover the entire segment of the collection window.

[0114] Fig.32 Involving a CNN model, and plotting the training accuracy change relative to the training epoch (where the number of epochs controls the number of complete passes through the training data set), the upper curve represents the training accuracy, and the lower curve represents the validation accuracy. As shown in the figure, the accuracy generally improves with the increase of epochs.

[0115] The comparison results indicate that for a small amount of sample data with sufficiently distinguishable features, SVM can provide good recognition accuracy. For a large amount of data with similar patterns, the CNN model generally provides better performance because CNN can automatically extract features instead of using manual selection. In general, both trained recognition models developed according to the CNN and SVM methods performed well in the experiments of the gloves described in this article. The classification of the SVM-based model was sufficient by manually extracting features using principal component analysis (PCA). For the CNN method with more advanced automatic extraction functions, the described glove only requires a relatively simple network to achieve high performance.

[0116] Example: Comprehensive demonstration in baseball game program

[0117] FIG24 illustrates a comprehensive demonstration of a glove-based HMI using a baseball game program. Screenshots provided by Unity software. A comprehensive demonstration was prepared using a baseball game program to perform manipulation of a baseball bat using a glove in addition to providing tactile stimulation from a hitting event ( Fig.24a , 24b and 24c illustrate grabbing the bat, a light hit, and a heavy hit, respectively). Fig.25 A flowchart showing the operation of a baseball program is shown. Fig.25As shown, the logic cycle for executing the baseball game program includes several main steps. First, the program enters the preparation mode for the operation check of the triboelectric finger bending and palm sensors. Next, once the real bat is grabbed, both the finger bending sensor and the palm sensor detect the grabbing action. This detection switches the program to the play mode, where the automatic pitching machine starts the pitching cycle. The triboelectric output from the palm sensor is detected as having the same amplitude for all four electrodes, which indicates the basic normal force applied to the palm sensor during the grabbing action. It is worth noting that in addition to the negative peak used as the sensing signal, there is also a positive peak, which is mainly caused by the first contact between the bat and the glove, and it will not be obvious after grabbing the bat. Then, in the automatic pitching stage, the user can swing the bat at different speeds to produce a hit, and the reaction force applied to the palm sensor causes different triboelectric outputs accordingly.

[0118] Figure 24b (iv) and Fig.24c (iv) The figure shows that the swinging motion of the bat causes a non-uniform distribution of the contact-related triboelectric output on the four electrodes of the palm sensor due to the presence of shear forces. As an example of a hard hit (see Fig.24c (iv)), the forward and upward electrodes of the palm sensor both show the largest triboelectric output compared to the downward and backward electrodes. Fig.24c (iv) Schematic diagram of the pressure mapping shown, by examining the trend of the output change, the direction of the shear force caused by the relative motion between the bat and the hand is defined, because the upper dome of the palm sensor is pushed forward + upward, causing the membrane (upper dome) to deflect more towards the contact of the forward and upward electrodes. The resulting shear force direction is shown in Fig.24c As indicated by the arrow in (iv).

[0119] As output from the triboelectric sensor, the corresponding hitting action is virtually executed in the baseball training program, i.e., a ground ball or a fly ball. As mentioned before, with the help of the Unity collision reactor module, the collision event between the ball and the bat is fed back to the microprocessor in communication with the glove to activate the feedback in the form of mechanical stimulation via the PZT chip of the glove. In order to distinguish the various hits of the user, the input voltage for actuating the PZT stimulator is adjusted depending on the measured triboelectric output of the palm sensor during the swing.

[0120] Example: Comprehensive demonstration in VR surgery training using object recognition

[0121] Fig.26 and Fig. 27 Figure 1 shows a comprehensive demonstration of a glove-based HMI using a VR surgical training program. Fig.26As depicted, in this case, the left glove is primarily assigned to control motion and manipulation modes, while the right glove is responsible for object recognition and manipulation. Once the glove is switched to recognition mode using the left index finger, the left glove is disabled and the right glove begins recognizing grasping gestures. In terms of recognition of the three selected tools (scissors, gauze, and scalpel), the specific triboelectric signals from the grasping action are Fig. 27 (ii). For example, the user should use the thumb and index finger to operate the scissors, which results in signals from the triboelectric sensors of the thumb and index finger only. However, for grabbing gauze, an additional signal from the middle finger is detected because three fingers are used.

[0122] After the specific tool is successfully identified and displayed, the glove is then switched back to control mode, in which the left thumb and middle finger can control the movement of the right virtual hand to locate the operation area, and the right glove can perform the surgical operation accordingly. Fig. 27 The corresponding triboelectric output is shown in , together with an image of the functional fingers of the glove.

[0123] Fig.26 The diagram illustrates the finger movements used to implement the following: (I) mode switching, (II) (III) movement of the right hand, (IV) display of the left hand, (V) recognition of scissors, (VI) recognition of a knife, (VII) recognition of gauze, (VIII) operation of scissors (cut), (IX) operation of a knife (cut) and (X) operation of gauze (brush). Fig. 27 The figure shows the measured triboelectric output from the following eight triboelectric sensors in (i) control mode: thumb (DD and DU), index finger (DD and DU), middle finger (DD and DU) and ring finger (DD and DU), and the triboelectric output from the following 16 sensors in (ii) recognition mode for the motion of grasping scissors, knife and gauze: thumb (DD and DU), index finger (DD, MD, PD and DU), middle finger (DD, MD and PD), ring finger (DD, MD and PD), little finger (DD, MD and PD) and palm (single electrode).

[0124] As shown in the figure, during scalpel operation, the left ring finger is first bent and then straightened to show and hide the left hand ( Fig.26 (IV) and Fig. 27 (IV)). Then bend the left index finger to trigger the recognition mode via the index finger sensor (DD) ( Fig.26 (I) and Fig. 27 (i)(I)). During the collection of recognition data from the right glove, the right hand imitated grabbing a scalpel, and the scalpel was successfully recognized and presented in the VR space ( Fig.26 (VI) and Fig. 27(ii)(VI)). Next, the left index finger returns to trigger the control mode via the index finger sensor (DU) ( Fig.26 (I) and Fig. 27 (i)(I)). Next, the left thumb and middle finger are used to move and rotate the right hand to the operating area ( Fig.26 (II)(III) and Fig. 27 (i)(II)(III)). Finally, the right middle finger triggered the cutting operation of the scalpel ( Fig.26 (IX) and Fig. 27 (i)(IX)).

[0125] In general, by utilizing AR technology, the gloves described herein can be used to enable more intuitive interactions regarding communication with virtual characters or objects, such as (for example) waving, shaking hands, and raising a glass ("cheers"). The described glove-based HMI can be used, for example, as a terminal for a virtual social network, and may be particularly useful in providing more interactive communications for the elderly during remote home care from a doctor or family member.

[0126] Output changes under changing humidity and long-term use

[0127] In general, triboelectric sensors may encounter problems from external influences such as humidity and contamination in certain situations. Fig.28 As shown in Figure A, a humidifier can be used to control the relative humidity (RH) to simulate the increase of water content on the surface of the triboelectric sensor, which mainly comes from ambient water vapor and sweat. In the case where the glove is formed of an elastomeric material and has an open structure, i.e., a separate one of the remaining fingers / thumb / palm shell except the thumb, rapid evaporation of water content occurs and minimizes the output drop, even at very high humidity levels (e.g., 80%, 90% RH), the measured output drop is around 15%, which is unlikely to occur under normal circumstances.

[0128] After long-term use, oil and dirt on the user's skin may also have a negative impact on the triboelectric sensor output. Fig.28 As shown in Figure 2B, the change in output relative to overall contamination over a certain period of use (i.e., 10,000 and 20,000 uses) was investigated. The glove itself (except for the removable signal processing circuitry, i.e., the leads connecting the various sensors / chips to the microprocessor) is completely washable, and therefore the triboelectric output was retested after rinsing, washing, and drying the gloves. The triboelectric sensor is completely washable, and typically the PZT chip may require further packaging to achieve the same level of washability. The data indicates that while the triboelectric output decays with long-term use, it can return to its original state after a simple wash (e.g., Fig.28B). Therefore, the deterioration of triboelectric output can be avoided by washing the gloves regularly.

[0129] Custom Triboelectric Regulator PCB for Optimized Multi-Channel Sensing

[0130] Analog-to-digital converters (ADCs) are commonly used in various sensing applications. In most cases, multichannel sensing is performed using a single ADC and applying sequential scanning of a series of input channels, rather than using an ADC for each channel, in order to reduce cost. However, in terms of simultaneous multichannel sensing, especially for triboelectric sensors, there are several constraints that need to be overcome to achieve real-time control of the entire hand, as described below and in general. Fig.29 As shown in A:

[0131] (1) The mismatch between the small triboelectric output (~100 mV) and the much larger measurement range (0~5V) of the ADC (e.g., Arduino), which results in high noise levels and low effective sensitivity.

[0132] (2) Crosstalk issues, especially those caused by channels near the input channel currently being scanned, which may be due to the coupling effect of adjacent channels via the ADC's sampling capacitors.

[0133] In order to solve the above problems, Fig.29 An exemplary method is shown in Figure B. This method uses a customized triboelectric regulator PCB (TENG - triboelectric nanogenerator regulator) on which the signal readout is optimized via an operational amplifier to enhance sensitivity, and all input channels are grounded to avoid crosstalk while maintaining a floating signal to obtain the entire triboelectric waveform. In one example, for an Arduino MEGA 2560 model development board, a total of 8 integrated amplifier circuits are designed for 16 analog input channels.

[0134] Core functions in the implementation of virtual hand control and virtual event feedback

[0135] In one example, for Fig.30In the virtual hand control part depicted in FIG, all signals from the triboelectric sensors are pre-processed by the regulator PCB, as described above, and then received by the microprocessor PCB (such as the Arduino MEGA 2560 PCB). In one example, the PCB can be designed as a detachable module of the glove together with the wireless module so that the signals can be sent wirelessly to the computer. Each Arduino PCB contains 16 analog input channels. After the threshold calibration of each triboelectric sensor, the identification code is responsible for identifying the trigger signal and issuing a digital string, such as "1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0" (16 numbers represent 16 channels), which indicates that the bending of the finger with bending level 1 is detected by the first sensor. On the processing software (e.g., Unity) side, the protocol of the serial communication allows the digital string to be received and assigned to a specific name. Therefore, by defining the corresponding finger joints and using the control code, the bending movement of the virtual hand can be realized as a projection of the real hand in the virtual space.

[0136] In one example, for Fig.31 The tactile feedback part shown can perform the reverse process of event projection by applying an open source module called Ardunity in the processing software (e.g., Unity), which is specially designed to enable mutual communication between Arduino and Unity. The collision reactor function in Ardunity provides the possibility of transmitting virtual events to the microprocessor to achieve various responses (such as light, vibration, and voice, etc.). Therefore, once the collision reactor recognizes the interaction event, by defining the PWM operating frequency and matching it with the PZT resonant frequency in the Arduino IDE, vibratory tactile feedback to the user can be triggered via serial communication.

[0137] The multifunctional glove-based HMI described in this article has several advantages over traditional HMIs. For example, the HMI is cheaper to manufacture, consumes less energy, is more user-friendly, and is highly customizable and signal stable for a variety of human hand sizes.

[0138] Overall, the smart gloves with simply designed triboelectric sensors and piezoelectric tactile mechanical stimulators indicate the potential for building low-cost and low-power HMIs that can achieve multi-dimensional and multi-purpose manipulation in VR / AR. The high customizability of the gloves ensures user-friendliness and signal stability for various human hands. In the future of 5G communication and IoT, applications under human-computer interaction will significantly affect human lifestyles. In terms of social networks, such devices can improve the intelligence of machines based on big data obtained from AI technology. In terms of VR training programs, enhanced two-way interaction can be achieved through the above gloves to improve training effects. Overall, the gloves reveal new possibilities as HMI solutions, and they are comparable to current inertial and resistance-based gloves for applications in VR training, entertainment, social networking, and robot control.

Claims

1. A wearable human-machine interface (HMI) configured to be worn on a human hand, the interface comprising: a plurality of triboelectric finger sensors configured to detect finger bending, deflection and / or curling, each triboelectric finger sensor comprising a contact portion and an electrode portion; a triboelectric palm sensor configured to detect an applied force and its direction from a wearer's hand; as well as at least one mechanical stimulator configured to provide tactile feedback; wherein each triboelectric finger sensor is associated with one of the fingers of the wearer's hand other than the thumb or a corresponding phalanx of the thumb and is configured to detect movement of the corresponding phalanx and the direction of the movement independently of the other phalanges; wherein the contact portion of each triboelectric finger sensor is arranged to be directly contactable by the skin of the corresponding phalanx; wherein the plurality of triboelectric finger sensors include at least an upper triboelectric finger sensor and a lower triboelectric finger sensor associated with each phalanx, the upper triboelectric finger sensor and the lower triboelectric finger sensor being configured to detect the movement of the corresponding phalanx in an upper direction and a lower direction and the direction of the movement; wherein for each respective phalanx, the respective upper triboelectric finger sensor and the lower triboelectric finger sensor are arranged such that when the skin of the respective phalanx is in direct contact with one of the upper triboelectric finger sensor and the lower triboelectric finger sensor, the skin of the respective phalanx is separated from the other of the upper triboelectric finger sensor and the lower triboelectric finger sensor; and The triboelectric finger sensor is configured to generate a triboelectric output at the electrode portion in response to changes in direct contact and separation between the contact portion and the skin of the phalanx, and the movement and direction of the movement of the phalanx can be detected based on the changes in the triboelectric output.

2. The interface of claim 1, further comprising a wearable housing for supporting the sensor and stimulator, wherein the sensor is attached to an inner surface of the wearable housing.

3. The interface of claim 2, wherein the wearable housing comprises a plurality of housing segments, including a palm segment and a plurality of finger segments.

4. An interface as described in any one of claims 1 to 3, wherein the plurality of triboelectric finger sensors include a left triboelectric finger sensor and a right triboelectric finger sensor, the left triboelectric finger sensor and the right triboelectric finger sensor are associated with each phalanx and are configured to detect the movement of the corresponding phalanx in the left direction and the right direction and the direction of the movement.

5. The interface of any one of claims 1 to 3, wherein the contact portion of each of the plurality of triboelectric finger sensors comprises a raised contact surface and a flat bottom, wherein the electrode portion is embedded in or otherwise attached to the flat bottom.

6. An interface as described in any one of claims 1 to 3, wherein the contact portion of each of the plurality of triboelectric finger sensors comprises an elastomeric material that can be directly contacted by the skin of the corresponding phalanx to generate the triboelectric output.

7. The interface of any one of claims 1 to 3, wherein the plurality of triboelectric finger sensors comprises sixteen triboelectric finger sensors.

8. An interface as described in any one of claims 1 to 3, wherein the triboelectric palm sensor includes an upper contact portion, a lower contact portion within the upper contact portion, and an electrode portion, wherein the electrode portion is configured to be directly contacted by the lower contact portion, and the contact generates a triboelectric output.

9. An interface as described in claim 8, wherein the upper contact portion includes a dome, the lower contact portion includes four hemispheres located within the dome, and the electrode portion includes four contact pads, each hemisphere being configured to serve as a corresponding contact point for a corresponding contact pad of the electrode portion when an external force is applied to the dome.

10. The interface of claim 8, wherein the upper contact portion and the lower contact portion of the palm sensor comprise an elastomeric material.

11. The interface of any one of claims 1 to 3, wherein the at least one mechanical stimulator comprises a piezoelectric PZT chip.

12. An interface as claimed in claim 11, wherein the interface comprises five PZT chips, each PZT chip being associated with one of the wearer's fingers other than the thumb or the thumb in use.

13. The interface of claim 3, comprising a wiring accessory configured to bias the interface toward an initial position.

14. The interface of claim 13, the wiring assembly being configured to interconnect the plurality of housing segments.

15. The interface of any one of claims 1 to 3, wherein the sensor is configured to be manufactured at least in part using 3D printing technology.

16. A human-machine interface system, comprising: A wearable human-machine interface HMI as claimed in any one of claims 1 to 15; a microprocessor configured to receive triboelectric outputs from the plurality of triboelectric finger sensors and / or the triboelectric palm sensor and determine the extent of the movement of the phalanges based on the triboelectric outputs; The microprocessor is also configured to output a signal to actuate vibration of the at least one mechanical stimulator.

17. The system of claim 16, wherein the microprocessor is configured to output the signal to the at least one mechanical stimulator based on the triboelectric output received from the palm sensor.

18. The system of claim 17, wherein the microprocessor is configured to modify the amplitude of the signal to the at least one mechanical stimulator based on the triboelectric output received from the palm sensor.

19. The system of claim 17 or 18, wherein the microprocessor is configured to determine the direction of the applied force based on a triboelectric output received from the palm sensor.

20. The system of claim 17 or 18, wherein the microprocessor is configured to output triboelectric outputs from the plurality of triboelectric finger sensors and / or the triboelectric palm sensor to a neural network.

21. A method for object recognition, comprising: The triboelectric outputs of the plurality of triboelectric finger sensors and / or the triboelectric palm sensor of the HMI interface of any one of claims 1 to 15 are received by a neural network, the triboelectric outputs being generated during grasping of an object by the wearer of the HMI interface.

Images