System and method for triboelectric sensor

Abstract

The stereognosis sensor system (100) includes a triboelectric proprioception sensor (102), a triboelectric tactile sensor (104), and a processor (106) having an artificial neural network. The stereognosis sensor system (100) may combine proprioception and touch sense with the textile-integrated triboelectric sensory units. The triboelectric tactile sensor (104) may include a plurality of triboelectric tactile sensors, such as one for each fingertip of a user's hand. The triboelectric tactile sensor(s) (104) and the triboelectric proprioception sensor (102) may be coupled with a textile glove that may be worn by a user. The triboelectric proprioception sensor (102) may capture tendon motion for global shape recognition, and the triboelectric tactile sensor(s) (104) may perceive the surface material information of objects. The artificial neural network may process and learn the stereognosis information encoded in the sensor data received from the triboelectric proprioception sensor (102) and the triboelectric tactile sensor(s) (104) for recognizing versatile objects.

Description

Attorney Docket No. 70328-02SYSTEM AND METHOD FOR TRIBOELECTRIC SENSORCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 693,881 filed September 12, 2024, the entirety of which is hereby incorporated by reference.GOVERNMENT RIGHTS

[0002] This invention was made with government support under 1925194 awarded by the National Science Foundation. The government has certain rights in the invention.FIELD

[0003] The disclosure generally relates to sensor systems and, more particularly, to wearable sensors.INTRODUCTION

[0004] This section provides background information related to the present disclosure which is not necessarily prior art.

[0005] The integrated proprioception and touch sense of human hands endow us with the stereognosis ability for perceiving, recognizing, and identifying complex three-dimensional objects in dexterous manipulation, social interactions, and other tasks. Without visual or auditory clues, human hands can detect the object's shape through the proprioceptive fibers embedded in muscles and tendons, and derive the material information of the object surface through cutaneous mechanoreceptors. Emulating such stereognosis capability with wearable devices would enable unprecedented accuracy, precision, and safety in robotics, mixed realities, human-machine teaming, and many societally pervasive fields. Numerous reports explored various types of wearable sensors for object recognition. However, these demonstrations can only detect a single attribute, such as the shape or surface properties, without the stereognosis capability necessary for practical applications. Massively distributed sensors are also typically required to sense the local geometry for contact shape recognition, significantly increasing the manufacture and computational complexity. A recent work reports shape recognition by monitoring single finger movement with a nanomesh sensor, but it is expected to face challenges in detecting the global shape of complex objects. While numerousAttorney Docket No. 70328-02 wearable sensors have been explored for object recognition, most sense only limited attributes, such as shape or surface properties alone. Electronic mimicry of human-like stereognosis has been hindered by two prevailing paradigms. The ‘brute-force’ approach employs high-density arrays, incurring prohibitive wiring, bandwidth, and computational overheads. The ‘incomplete’ approach uses minimalist, single-modality sensors, which, even with advanced machine learning, fail to provide a holistic perception required to distinguish complex objects robustly. Consequently, a wearable system that achieves true dual-modality stereognosis through informationally efficient architecture remains a critical unmet goal. In other words, the human-like stereognosis capability of simultaneous proprioception and touch sense with frugally engineered sensory units in a single wearable has yet to be demonstrated.

[0006] Accordingly, there is a continuing need for a wearable sensor that can recognize a wide range of objects with high precision through grasping and learning from a small training set.SUMMARY

[0007] In concordance with the instant disclosure, a wearable sensor that can recognize a wide range of objects with high precision through grasping and learning from a small training set, has surprisingly been discovered.

[0008] The stereognosis sensor system of the present disclosure includes a triboelectric proprioception sensor, a triboelectric tactile sensor, and a processor having an artificial neural network. The stereognosis sensor system emulates human stereognosis for combined proprioception and touch sense with the frugally engineered textile-integrated triboelectric sensory units. In a specific example, the triboelectric tactile sensor may include a plurality of triboelectric tactile sensors, such as one for each fingertip of a user’s hand. Each of the triboelectric tactile sensor(s) and the triboelectric proprioception sensor may be coupled with a textile glove that may be worn by a user. During a grasp event, the single-channel triboelectric proprioception sensor may capture the subtle tendon motion for global shape recognition, and the five-channel triboelectric tactile sensors may perceive the surface material information of objects. The artificial neural network may process and learn the stereognosis information encoded in the sensor data received from the triboelectric proprioception sensor and the triboelectric tactile sensor(s) for recognizing versatile objects.Attorney Docket No. 70328-02

[0009] Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.DRAWINGS

[0010] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations and are not intended to limit the scope of the present disclosure.

[0011] FIG. 1 is a bottom plan view of an exterior surface of the stereognosis sensor system having a glove, a triboelectric proprioception sensor, a triboelectric tactile sensor , and a processor having an artificial neural network, according to one embodiment of the present disclosure;

[0012] FIG. 2 is a bottom plan view of an interior surface of the stereognosis sensor system, as shown in FIG. 1, having the glove substrate, the triboelectric proprioception sensor, the triboelectric tactile sensor, and the processor having an artificial neural network, according to one embodiment of the present disclosure;

[0013] FIG. 3 is an enlarged cross-sectional view of the triboelectric proprioception sensor and the triboelectric tactile sensor , further depicting the sensors including a polydimethylsiloxane (PDMS) dielectric layer, a conductive electrode layer, and an adhesive layer, according to one embodiment of the present disclosure;

[0014] FIG. 4 is an enlarged top plan view of the textile glove, further depicting the textile glove having a weaving pattern, according to one embodiment of the present disclosure;

[0015] FIG. 5 is an enlarged top plan view of the conductive electrode, further depicting the conductive electrode having a weaving pattern, according to one embodiment of the present disclosure;

[0016] FIG. 6A is an electron-cloud-potential-well model illustrating the charge transfer between triboelectric sensors and the objects, according to one embodiment of the present disclosure;

[0017] FIG. 6B is a bar graph illustrating the unique triboelectric signals of different materials when interacting with the PDMS sensor since the surface electron cloud states are distinct for different materials, thus the unique output signals of five materials interacting with the tactile sensor show distinct output amplitudes, according to one embodiment of the present disclosure;Attorney Docket No. 70328-02

[0018] FIG. 6C is a cluster diagram of a t-distributed stochastic neighbor embedding (t- SNE) algorithm applied to unique signal of three typical human grasps; according to one embodiment of the present disclosure;

[0019] FIG. 6D is a line graph of illustrating the robustness of the textile-base triboelectric sensors by cyclic testing over 10,000 contacts, where the stable output of the textile-based triboelectric sensor exhibits its good reliability for long-term use in object recognition, according to one embodiment of the present disclosure;

[0020] FIG. 7A illustrates the evaluation of the recognition performance of the stereognosis sensor system of the present disclosure, further depicting two groups of objects (labeled as 1 to 9) were designed and 3D printed, where Group 1 includes five objects (objects 1-5) with the same surface material (polylactide, PLA) but different shapes, including sphere, cube, hexagonal prism, cone, and cylinder and Group 2 includes five objects (objects 5-9) with the same shape (cylinder) but different surface materials, including PLA, paper, polyethylene terephthalate (PET), cloth, and aluminum (Al);

[0021] FIG. 7B illustrates a machine learning model that was constructed for data feature extraction and object recognition, according to one embodiment of the present disclosure;

[0022] FIG. 7C illustrates the data set obtained by the stereognosis sensor system for Group1, as shown in FIG. 7A;

[0023] FIG. 7D illustrates the data set obtained by the stereognosis sensor system for Group2, as shown in FIG. 7A;

[0024] FIG. 7E illustrates the data set obtained by the stereognosis sensor system for Groups 1 and 2, as shown in FIG. 7A;

[0025] FIG. 8 is a top plan view of the interior surface of the stereognosis sensor system, further depicting a shielding layer disposed over conductive wires coupling the triboelectric tactile sensor , the triboelectric proprioception sensor, and the processor, according to one embodiment of the present disclosure;

[0026] FIG. 9 is a table listing the thirty items tested with the stereognosis sensor system;

[0027] FIG. 10 is a table comparing the accuracy of the stereognosis sensor system to other sensor systems, according to one embodiment of the present disclosure;

[0028] FIG. 11 is a flow chart of a first method for using the stereognosis sensor system, according to one embodiment of the present disclosure.Attorney Docket No. 70328-02

[0029] FIG. 12 is a flow chart of a second method for manufacturing the stereognosis sensor system, according to one embodiment of the present disclosure.DETAILED DESCRIPTION

[0030] The following description of technology is merely exemplary in nature of the subject matter, manufacture, and use of one or more inventions, and is not intended to limit the scope, application, or uses of any specific invention claimed in this application or in such other applications as may be filed claiming priority to this application, or patents issuing therefrom. Regarding methods disclosed, the order of the steps presented is exemplary in nature, and thus, the order of the steps can be different in various embodiments, including where certain steps can be simultaneously performed. “A” and “an” as used herein indicate “at least one” of the item is present; a plurality of such items may be present, when possible. Except where otherwise expressly indicated, all numerical quantities in this description are to be understood as modified by the word “about” and all geometric and spatial descriptors are to be understood as modified by the word “substantially” in describing the broadest scope of the technology. “About” when applied to numerical values indicates that the calculation or the measurement allows some slight imprecision in the value (with some approach to exactness in the value; approximately or reasonably close to the value; nearly). If, for some reason, the imprecision provided by “about” and / or “substantially” is not otherwise understood in the art with this ordinary meaning, then “about” and / or “substantially” as used herein indicates at least variations that may arise from ordinary methods of measuring or using such parameters.

[0031] Although the open-ended term “comprising,” as a synonym of non-restrictive terms such as including, containing, or having, is used herein to describe and claim embodiments of the present technology, embodiments may alternatively be described using more limiting terms such as “consisting of’ or “consisting essentially of.” Thus, for any given embodiment reciting materials, components, or process steps, the present technology also specifically includes embodiments consisting of, or consisting essentially of, such materials, components, or process steps excluding additional materials, components or processes (for consisting of) and excluding additional materials, components or processes affecting the significant properties of the embodiment (for consisting essentially of), even though such additional materials, components or processes are not explicitly recited in this application. For example, recitation of a composition or process recitingAttorney Docket No. 70328-02 elements A, B and C specifically envisions embodiments consisting of, and consisting essentially of, A, B and C, excluding an element D that may be recited in the art, even though element D is not explicitly described as being excluded herein.

[0032] As referred to herein, disclosures of ranges are, unless specified otherwise, inclusive of endpoints and include all distinct values and further divided ranges within the entire range. Thus, for example, a range of “from A to B” or “from about A to about B” is inclusive of A and of B. Disclosure of values and ranges of values for specific parameters (such as amounts, weight percentages, etc.) are not exclusive of other values and ranges of values useful herein. It is envisioned that two or more specific exemplified values for a given parameter may define endpoints for a range of values that may be claimed for the parameter. For example, if Parameter X is exemplified herein to have value A and also exemplified to have value Z, it is envisioned that Parameter X may have a range of values from about A to about Z. Similarly, it is envisioned that disclosure of two or more ranges of values for a parameter (whether such ranges are nested, overlapping, or distinct) subsume all possible combination of ranges for the value that might be claimed using endpoints of the disclosed ranges. For example, if Parameter X is exemplified herein to have values in the range of 1-10, or 2-9, or 3-8, it is also envisioned that Parameter X may have other ranges of values including 1-9, 1-8, 1-3, 1-2, 2-10, 2-8, 2-3, 3-10, 3-9, and so on.

[0033] When an element or layer is referred to as being “on,” “engaged to,” “connected to,” or “coupled to” another element or layer, it may be directly on, engaged, connected, or coupled to the other element or layer, or intervening elements or layers may be present. In contrast, when an element is referred to as being “directly on,” “directly engaged to,” “directly connected to” or “directly coupled to” another element or layer, there may be no intervening elements or layers present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., “between” versus “directly between,” “adjacent” versus “directly adjacent,” etc.). As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0034] Although the terms first, second, third, etc. may be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms may be only used to distinguish one element, component, region, layer or section from another region, layer, or section. Terms such as “first,” “second,” and other numerical terms when used herein do not imply a sequence or orderAttorney Docket No. 70328-02 unless clearly indicated by the context. Thus, a first element, component, region, layer, or section discussed below could be termed a second element, component, region, layer, or section without departing from the teachings of the example embodiments.

[0035] Spatially relative terms, such as “inner,” “outer,” “beneath,” “below,” “lower,” “above,” “upper,” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. Spatially relative terms may be intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the FIG. is turned over, elements described as “below”, or “beneath” other elements or features would then be oriented “above” the other elements or features. Thus, the example term “below” can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

[0036] Endowing machines with haptic intelligence that reasons about the physical world requires generalizable object perception beyond fixed-class recognition. Prior approaches are hindered by dense sensor arrays, which increase manufacturing and computational costs. Here, the present disclosure may provide a textile-integrated triboelectric wearable sensor system that overcomes this barrier, learning abstract shapes and material primitives to identify previously unseen shape-material combinations with a tested accuracy of 94.06%. This capability for deductive inference arises from a frugal, bioinspired architecture that emulates human stereognosis by decoupling proprioceptive signals from wrist tendon motion and tactile signals from fingertip contact using only six triboelectric channels, provided as a non-limiting example. Such compositional generalization is built upon sensors’ exceptional classification, achieving -99% accuracy across 30 real-world objects, including visually indistinguishable cases. Achieving robust perception with an -99% reduction in sensor count compared to state-of-the-art, our work establishes a new paradigm for informationally efficient electronics, paving the way for autonomous, edge-intelligent systems in robotics, prosthetics, and human-machine interaction. The dual-modality design of the present disclosure, processed by a computationally lean neural network trained on small datasets, not only achieves state-of-the-art accuracy on known objects, including visually ambiguous ones, but also enables a more profound capability; it learns abstract primitives of shape and material to identify previously unseen objects through deductive inference. This represents a paradigm shift in wearable touch sensors from simple pattern matching to generalizable human-like haptic perception.Attorney Docket No. 70328-02

[0037] The textile-integrated wearable system of the present disclosure moves beyond simple classification to achieve a form of haptic reasoning. By emulating the dual-modality nature of human stereognosis with a frugal six-sensor architecture, once again, provided as a non-limiting example, the system of the present disclosure learns abstract primitives of shape and material, enabling it to identify novel objects through deductive inference. This capability, validated by the successful identification of previously unseen objects, represents a paradigm shift from conventional pattern matching to generalizable, human-like perception. Furthermore, the ability of the system of the present disclosure to distinguish visually identical objects by sensing subtle mechanical properties highlights the fundamental advantages of a haptic-centric approach, particularly in scenarios where vision is unreliable or insufficient. The findings of the present disclosure offer a new design philosophy for robotics and a tangible pathway toward true edge intelligence. While many advanced robotic systems pursue multimodal perception by integrating complex and power-intensive sensors, the present disclosure demonstrates a paradigm of 'intelligent efficiency,’ extracting maximal information from minimal data. This is critical for the next generation of dexterous robotics, prosthetics, and edge-intelligent human-machine interfaces operating on limited power budgets. Moreover, the low data bandwidth and efficacy of a computationally lean ID CNN make it an ideal platform for edge Al. This may open the door to fully untethered smart machines that can perceive, reason, and act on devices without relying on cloud connectivity. It is contemplated that the system of the present disclosure may include fewshot learning techniques to enable rapid personalization for new users, and extending the meta- leaming concepts used for single-modality sensors to the dual-modality system of the present disclosure. Integrating on-board processing and wireless communication is also contemplated to be utilized with the system of the present disclosure. By demonstrating that a wearable system can learn, abstract, and reason about the physical world, the present disclosure may open new avenues for creating more seamless, scalable, and vision-independent intelligent interactions between humans and machines.

[0038] As shown in FIGS. 1-2, the stereognosis sensor system 100 of the present disclosure includes a triboelectric proprioception sensor 102, a triboelectric tactile sensor 104, and a processor 106 having an artificial neural network. The triboelectric tactile sensor 104 may be disposed at a distal contact region of a substrate 108. The triboelectric tactile sensor 104 may include a conductive electrode 110 and a dielectric layer 112 which generates an electrical signal in response to contactAttorney Docket No. 70328-02 with a surface of an object. The triboelectric proprioception sensor 102 may be disposed at a proximal region of the substrate 108. The triboelectric proprioception sensor 102 may be configured to generate an electrical signal in response to movement of a joint and / or a tendon during object manipulation. The processor 106 may be configured to receive and process the electrical signals to determine at least one of a shape, a surface material, and a mechanical property of the object. The stereognosis sensor system 100 may include a signal transmission network 114 electrically coupled to the triboelectric tactile sensor 104 and the triboelectric proprioception sensor 102. The stereognosis sensor system 100 may also include an electrically shielding conductive fabric layer 116 which may cover the signal transmission network 114. In a specific example, the substrate 108 may include a textile glove that may be worn by a user. The triboelectric proprioception sensor 102 may be disposed on an interior surface of a cuff portion of the glove. Accordingly, the triboelectric proprioception sensor 102 may also be known as a wrist sensor. The triboelectric tactile sensor 104 may be disposed on a finger portion of the glove, such as at the fingertips of the glove. Accordingly, the triboelectric tactile sensor 104 may also be known as a fingertip sensor. In an alternative example, the substrate 108 may be integrated with and / or mountable on a robotic hand, a prosthetic hand, a robotic gripper, and / or a manipulator. The stereognosis sensor system 100 may be a wearable sensor that emulates human stereognosis for combined proprioception and touch sense with the frugally engineered textile- integrated triboelectric sensory units. In a specific example, the triboelectric tactile sensor 104 may include a plurality of triboelectric tactile sensors 104. The triboelectric tactile sensors 104 maybe arranged to conformally contact surfaces having one of a flat, a curved, and an edged local geometry. For instance, the triboelectric tactile sensors 104 may be positioned at each fingertip of a user’s hand. During a grasp event, the triboelectric proprioception sensor 102 may capture the subtle tendon motion for global shape recognition, and the triboelectric tactile sensor(s) 104 may perceive the surface material information of objects. The triboelectric proprioception sensor 102 may be positioned to detect multiple grasp modes for the same object shape, which may enhance recognition robustness. The artificial neural network may process and learn the stereognosis information encoded in the sensor data received from the triboelectric proprioception sensor 102 and the triboelectric tactile sensor(s) 104 for recognizing versatile objects. In a more specific example, the artificial neural network may include a neural network trained to identify objects based on combined tactile and proprioceptive data.Attorney Docket No. 70328-02

[0039] The triboelectric proprioception sensor 102 and the triboelectric tactile sensor(s) 104 may be provided in various ways. For instance, as shown in FIG. 3, the triboelectric proprioception sensor 102 and / or the triboelectric tactile sensor(s) 104 may be manufactured by integrating conductive textile electrodes 110, an adhesive layer 111, and poly dimethyl siloxane (PDMS) dielectric layers 112 with adhesives. The PDMS contact layer 112 is biocompatible with good stretchability and suitable for wearable devices. PDMS has also been reported as a tribo-negative material, which facilitates the efficient contact triboelectrification with the tribo-positive human skin for detecting subtle skin deformation. The textile-based sensors can be cut into designer shapes and coupled to glove fingertips as tactile sensors and on a glove cuff as the wrist sensor(s). The fingertip tactile sensors may perceive the surface information of objects, and the wrist sensor may detect subtle tendon motion induced by human grasp. The fingertip signals and wrist signals from grasp actions may be transmitted through electrically shielded signal wires disposed inside the glove. The collected data may then be sent to the artificial neural network for classification and learning. It is also contemplated that the triboelectric proprioception sensor 102 may be provided as a plurality of triboelectric proprioception sensors.

[0040] The stereognosis sensor system 100 of the present disclosure may be used in various ways. For instance, the stereognosis sensor system 100 may be used according to a first method 200. The first method 200 may include a step 202 of contacting a surface of an object with a triboelectric tactile sensor 104. Surface charge data may then be acquired via the triboelectric tactile sensor 104. Tendon motion signal data may also be acquired via the triboelectric proprioception sensor 102. In some instances, surface curvature data may also be acquired via the triboelectric tactile sensor 104. Then, the surface charge data, the tendon motion signal data, and / or the surface curvature data may be transmitted to the processor 106. Next, a shape, a surface material, and / or a mechanical property of the object may be determined. The shape, the surface material, and / or the mechanical property of the object may be determined by utilizing a machine learning algorithm trained on a combined tactile and proprioceptive signal dataset. One skilled in the art may select other suitable ways for using the stereognosis sensor system 100, within the scope of the present disclosure.

[0041] As shown in FIGS. 4-5, the textile substrate 108 and the conductive electrodes 110 may include a weaving structure to enhance the wearability of the triboelectric sensors. The triboelectric sensors may withstand various large deformations which may enable conformalAttorney Docket No. 70328-02 sensory interfaces on different parts of the human skin and ensure the intimate contact with local geometry of versatile objects during grasp. In certain circumstances, the materials used to fabricate the stereognosis sensor system 100 may be low-cost and feasible for large-scale applications.

[0042] Without being bound to any particular theory, the working principle of materials recognition by the stereognosis sensor system 100 may be based on energy level difference of surface states between different materials. During grasping, the contact electrification between fingertip tactile sensors and the objects induces unique triboelectric signals. The electron-cloudpotential-well model illustrates the charge transfer between triboelectric sensors and the objects, as shown in FIG. 6A. More specifically, the electron clouds of two materials (e.g., PDMS sensor and object) before the atomic-scale contact remain separated without overlap. The potential well (Ei) may bind the electrons tightly and militate against them from escaping, where the electron-occupied energy level of material A (AA) is lower than the potential well (AA<A’I). When the object is grasped, the electron clouds may overlap with primary bonds formed. The initial single potential wells become an asymmetric double-well potential, and the energy barrier for charge transfer between the sensor and object may be lowered because of strong electron cloud overlap. Surface charges then transfer from the surface of the tribo-positive object (i.e., aluminum) to the sensor, or verse when sensor interacts with more tribo-negative materials. The triboelectric signals of different materials are unique when interacting with the PDMS sensor, since the surface electron cloud states are distinct for different materials. Based on the above principle, the unique output signals of five materials interacting with the tactile sensor show distinct output amplitudes, as shown in FIG. 6B. In addition to surface material property, these fingertip tactile signals may contain richer information about the objects, such as surface curvature (contact shape), which could be analyzed and decoded by machine learning algorithms.

[0043] The shape recognition by the stereognosis sensor system 100 may rely on decoding proprioceptive information from tendon motion by the wrist sensor. Wrist proprioceptive signals with unique features are generated when grasping objects with different shapes. The proprioceptive information encoded in the wrist signals can be extracted using ML algorithms. There are three typical human grasps which have unique wrist signals. At first glance, these signals show subtle differences in the amplitude and waveform. To test the discriminability of these signals, a / -distributed stochastic neighbor embedding (z-SNE) algorithm was applied to visualize the clustered results, as shown in FIG. 6C. LSNE is a nonlinear dimensionality reduction method widely used for visualizing high-Attorney Docket No. 70328-02 dimensional data in a low-dimensional space. Data distributions exhibit good cluster for samples within the same grasp, while samples from different grasps spatially separate from the others. The results indicate that wrist signals generated from the tendon motion during different human grasps contain critical information unique to the object shapes. As shown in FIG. 6D, the robustness of the textile-base triboelectric sensors was also verified by cyclic testing over 10,000 contacts. The stable output of the textile-based triboelectric sensor exhibits its good reliability for long-term use in object recognition.

[0044] Leveraging the integrated sensing capabilities of fingertip tactile and wrist sensors, the performance of the stereognosis sensor system 100 was evaluated for recognizing obj ects with various shapes and surface properties, as shown in FIGS. 7A-7E The wearable sensor’s self-powered proprioceptive capability was first demonstrated to recognize the object global shape from a single wrist sensor, which is much simpler and more efficient than the tactile mapping strategies dominantly adopted in previous literature. Traditional analysis of triboelectric signals, such as frequency and peak amplitude extraction from a single waveform, cannot suffice complex features recognition. The extracted features such as amplitudes in previous reports have subtle variations and are susceptible to environmental changes. To evaluate the recognition performance of the stereognosis sensor system 100 of the present disclosure, two groups of objects (labeled as 1 to 9) were designed and 3D printed, as shown in FIG. 7A. Group 1 includes five objects (objects 1-5) with the same surface material (polylactide, PLA) but different shapes, including sphere, cube, hexagonal prism, cone, and cylinder. Group 2 includes five objects (objects 5-9) with the same shape (cylinder) but different surface materials, including PLA, paper, polyethylene terephthalate (PET), cloth, and aluminum (Al). Each object is grasped using the stereognosis sensor system 100 for 100 times to obtain a reliable data set. The typical grasp signal for each object is shown at the bottom in. A machine learning model was constructed for data feature extraction and object recognition, as shown in FIG. 6B. The 100 samples of each data set are randomly split into two groups at a ratio of 8:2, with 80 samples for training, and 20 samples for testing. The whole data set is built from 9 objects and has 900 samples. The signal length of each channel is set as 1000 sampled data to ensure information completeness during the grasp action, so there are 1000 x 6 = 6000 features from the six channels (1 wrist channel and 5 fingertip channels) for each sample.

[0045] The data set of these 9 objects are trained separately as group 1 (FIG. 7C) and group 2 (FIG. 7D), and finally trained as a whole group (FIG. 7E). Also, for each group, three modes haveAttorney Docket No. 70328-02 been applied for the model training, including the training with the pure wrist signals, fingertip signals, and mixed wrist-fingertip signals. The accuracies of the trained model to recognize objects in group 1 are shown in confusion maps, as shown in FIG. 7C. In group 1, the accuracy of the trained models using wrist-fingertip signals and pure wrist signal reach 99%, higher than the model trained using pure fingertip signals (86%). Objects 1-5 in group 1 share the same surface material (PLA) but different shapes, so it is easier to recognize these objects using pure wrist signals (accuracy of 99%) than using the fingertip signals (accuracy of 86%). In addition, the model trained using pure fingertip signals with fewer channels, either 1 finger or 3 fingers, achieved lower accuracies of 74% and 82% respectively. In ideal conditions, the training results with different number of fingertip channels should be similar since each fingertip channel provide the same information about the surface material. However, the training results of 1, 3 and 5 fingertip channels are 74%, 82%, 86%, respectively. This observation indicates that the fingertip signals contain richer information (e.g., contact shape) in addition to the surface material. Such information is critical for accurate object recognition when the surface materials in group 1 are the same. With more fingertip channels, such information beyond surface materials becomes more comprehensive, resulting in a higher training accuracy, as shown in FIG. 7C. However, the highest accuracy of 86% (with 5 fingertip channels) is still much lower than that achieved with the mixed wrist-fingertip signals (99%). It is also worth noting that our wrist signals could recognize group 1 objects with an accuracy of 99%, which is much higher than previous reports, as shown in FIG. 10, e.g., a recent demonstration using proprioceptive signals from index finger to recognize similar object shape with an accuracy of 82.1%. The much improved accuracy of the stereognosis sensor system 100 of the present disclosure may be attributed to the proprioceptive signal in the stereognosis sensor system 100 contains more comprehensive information from the tendons that relate to the entire hand, while previous work’s proprioceptive signal was derived from one finger. Such results indicate that comprehensive proprioceptive signals collected from grasp action by the stereognosis sensor system 100 may significantly enhance the recognition accuracy at much lower integration and computation expenses.

[0046] When recognizing the objects in group 2, models trained using pure wrist signals achieve a low accuracy of 68%, as shown in FIG. 7D. Objects 5-9 in group 2 share the same shape but have different surface materials, it is therefore difficult to recognize these five objects only by shape. However, the models trained using wrist-fingertip signals and pure fingertip signals in group 2 achieve similar recognition accuracies of 99% and 98%, with continued reference to FIG. 7D.Attorney Docket No. 70328-02Models trained using pure fingertip signals with 1- or 3-finger channels achieve accuracies of 96% and 97%, respectively. The similar accuracies in recognizing group 2 objects trained from 1, 3 and 5 fingertip channels indicate that tactile information of each fingertip is similar, and the information beyond material (e.g., contact shape) is not significant for recognizing objects in group 2 since they share the same shape. Finally, we considered the data set of the entire 9 objects to train the model using wrist-fingertip signals and achieved a high accuracy of 99.44%, as shown in FIG. 7E. The global shape differences lead to distinct wrist signals. The surface materials and local geometries contribute to the unique features in the fingertip tactile signals. The accuracy of machine learning models decreases when trained using less channel data. We also trained the model with pure wrist signals and pure fingertip signals, which achieved accuracies of 80.56% and 93.89% for recognizing all 9 objects, with continued reference to FIG. 7E. Such results indicate that either lacking shape or material information will decrease the recognition accuracy significantly.

[0047] To evaluate the capability of the stereognosis sensor system 100 for more versatile recognition, 30 common objects, including fruits, vegetables, boxes, balls, and cups are included for the demonstration, as shown in FIG. 9. To probe the limit of the stereognosis sensor system’s 100 capability, we intentionally chose objects with subtle differences in shapes and / or surface materials (e.g., objects 1-9 in FIG. 7A). Triboelectric signals are generated when grasping these objects using the stereognosis sensor system 100. The excellent deformability of the sensors of the stereognosis sensor system 100 not only provides good wearability to the human hand, but also enables good conformal interaction with various geometries during grasping. The sample length of the recorded signals for each channel is set as 1000. 100 samples are collected for each object (80% for training and 20% for testing). The whole dataset totals 3000 samples. The object recognition accuracy of the stereognosis sensor system 100 was found to be 98.67%, exhibiting a good potential for high- accuracy object recognition in daily life. The data distribution of a t-SNE analysis visualizing the discriminability exhibits good cluster among samples from the same object, while samples from different objects well separate from each other. The data of objects with similar shapes, especially those wrapped with the same materials, are selected from these 30 objects for further evaluation. For instance, the images of aluminum-wrapped-apple, orange, and tomato show hard-to-distinguish differences in shape and surface materials. Such objects with highly similar visual features are challenging to recognize using human or machine vision. The stereognosis sensor system 100 of the present disclosure was able to recognize all these objects with an accuracy of 98.33%. AlthoughAttorney Docket No. 70328-02 these three objects have similar shapes and same surface material, their different mechanical properties result in interaction variations at the contact area of each fingertip. Such imperceivable variations lead to distinct features in both the fingertip tactile signals and the wrist proprioceptive signals. Such capability of the stereognosis sensor system 100 could be useful for object recognition in scenarios where visual recognition is obstructed or not available.

[0048] To evaluate the capability of the stereognosis sensor system 100 for recognizing unknown objects, libraries for shapes and materials were constructed to train the machine learning model. After learning from the shapes and materials in the libraries, the stereognosis sensor system 100 can recognize versatile unknown objects beyond the libraries. Specifically, we built our shape library from four representative shapes, including sphere (apple), cuboid (box), cylinder (cup) and cone. In real applications, objects may be grasped randomly with different wrist signals generated. Therefore, different grasp modes were also included in the library. For each shape, 100 wrist signals of each grasp mode were collected, and totally 100 x 2 = 200 samples of each shape were used for model training (80% for training and 20% for testing).

[0049] The material library is built from four commonly used packaging materials, including polyvinyl chloride (PVC), PET, paper and aluminum foil. Since the contact shapes for fingertips may be different in practical applications, we collected data from three local geometries (curved surface, flat surface and edge) for these four materials. For each material, 100 contacts of each local geometry were collected, and totally 100 x 3 = 300 samples of each material were obtained for the model training (80% for training and 20% for testing).

[0050] The accuracies of the stereognosis sensor system 100 for recognizing the library shapes and materials are 98.75% and 97.92%, respectively. The high recognition accuracies of these two libraries are necessary for detecting unknown objects. The t-SNE cluster results of wrist signal (different grasp modes of four shapes) from the convolutional neural network (CNN) input and output layers indicate that our model can extract features from different grasp modes for shape recognition. Although the wrist-signal clusters of two grasp modes exhibit clear boundaries in the input layer, their corresponding clusters coincide perfectly in the output layer. Such results show that different grasp modes of the same shape can be correctly recognized as the same group by our sensor system, which is essential for practical application. The t-SNE cluster results of the fingertip tactile signals (for different contact shapes of the four materials) from the CNN input and output layers also indicate that the stereognosis sensor system 100 can perceive material information fromAttorney Docket No. 70328-02 different contact shapes. The excellent capability to eliminate distractions, such as different grasp modes and contact shapes, enable the stereognosis sensor system 100 to learn, explore, and recognize unknown objects. To evaluate such capability, we constructed 16 unknown objects based on shapes and materials from the libraries, where objects of four different shapes were wrapped with four materials, respectively. The stereognosis sensor system 100 was used to grasp these objects and collect 20 samples for each object. Totally 20 x 16 = 320 samples were collected for evaluating the recognition performance. The resulting accuracy for recognizing these 16 unknown objects is 94.06%. Advantageously, the high recognition accuracy suggests that the stereognosis sensor system 100 can recognize unknown objects based on the knowledge learned from the shape and material libraries, unlike known sensor systems.

[0051] The stereognosis sensor system 100 may be constructed in various ways. For instance, the stereognosis sensor system 100 may be constructed according to a second method 300. The second method 300 may include a step 304 of coupling a triboelectric tactile sensor 104 at a distal contact region of a substrate 108. Next, a triboelectric proprioception sensor 102 may be coupled at a proximal region of the substrate 108. Each of the triboelectric tactile sensor 104 and the triboelectric proprioception sensor 102 may include a conductive textile electrode 110 and a polydimethylsiloxane (PDMS) dielectric layer 112. Afterwards, a signal transmission network 114 may be electrically coupled between the triboelectric tactile sensor 104 and the triboelectric proprioception sensor 102. Then, the triboelectric tactile sensor 104 and the triboelectric proprioception sensor 102 may be coupled to the processor 106. A conductive fabric layer 116 may be applied over the signal transmission network 114 that provides electromagnetic shielding. The method 300 may also include a step 302 of fabricating the triboelectric tactile sensor 104 by applying a PDMS dielectric layer 112 onto a conductive electrode 110, curing the dielectric layer 112, and cutting the triboelectric tactile sensor 104 into a predetermined dimension before the step 304 of coupling the triboelectric tactile sensor 104 at the distal contact region of the substrate 108. One skilled in the art may select other suitable ways for providing the stereognosis sensor system 100, within the scope of the present disclosure.

[0052] Providing as a non-limiting example, the following details describe how one nonlimiting embodiment of the stereognosis sensor system 100 was made. The stereognosis sensor system 100 may include a textile glove with five tactile sensors and a wrist sensor. The five fingertip sensors were mounted on the textile glove’s outside layer, which would be propped up byAttorney Docket No. 70328-02 human fingertips. These five fingertip sensors may collect tactile information when interacting with objects during grasp actions. A wrist sensor was mounted on inside layer of glove cuff, and it may interface with the inner wrist of a user to collect tendon motion signals during grasp actions. The triboelectric sensor may include a textile electrode layer and a polydimethylsiloxane (PDMS) dielectric layer 112. PDMS was blade coated on the textile electrode layer with a thickness of -100 um and cured in an oven at -60 °C for about one hour. The textile electrode covered with cured PDMS was cut into specific dimensions to make the fingertip tip sensors and the wrist sensor, which would then be attached to the textile glove substrate 108 with wiring out for collecting the signals. Sensing signals of fingertip and wrist sensors were wired out using 30 AWG wire. To avoid interference during grasp motions, all signal wires in the inside layer were designed to interface the dorsal side of hand when worn. A conductive fabric was applied to cover all signal wires and form a shielding layer, as shown in FIG. 8. All signal wires were assembled at the glove cuff of inside layer, and then wired out to the processor 106, which may include an Arduino system. For instance, the Arduino micro with an ATmega32U4 microcontroller was used for signal readout. The obtained signals from the stereognosis sensor system 100 were amplified and processed by the microcontroller. The amplified and processed data may then undergo characterization using the machine learning algorithm. It is contemplated that the characterization may occur at a separate computer with enhanced processing capabilities.

[0053] The t-distributed Stochastic Neighbor Embedding (t-SNE) algorithm is used for nonlinear dimensionality reduction and visualize high-dimension data in low dimension space (2D or 3D). The data used for model training in this study have 1000 features (a large feature list) for each channel, so it is effective to use the t-SNE algorithm to check if there are clusters in our training data.

[0054] The algorithm process data set in three steps:

[0055] (1) Calculate a joint probability distribution of high dimension data set, which represents similarities between data points (each data point represents a sample in our data set). Euclidian distances of each point from all the other points are calculated and transformed into conditional probability that represent similarity between every two points. Gaussian distribution is used to represent conditional probability in high dimension space, and the probability of point Xi to have xj as its neighbor:Attorney Docket No. 70328-02

[0056] where cr£is the standard deviation of gaussian distribution.

[0057] The joint probability distribution could be calculated from the conditional distribution:

[0058] (2) Create a random data set of points in low dimension space (2D space in this study), and calculate the joint probability distribution for them using t distribution.

[0059] The joint probability distribution in low dimension space created using t-distribution is: n (i + l|y; — yjlf)-1Mi + lly>. - y.ll2)’1

[0060] (3) Using gradient descent to change the data set in the 2D space to make its joint probability distribution (Q) as similar as possible to that of the high dimension data set (P). Kullback- Leiber divergence (KL divergence) is used as gradient descent function. In probability space / , KL divergence between the probability distribution P and Q is defined by:

[0061] The value of KL divergence is getting smaller as probability distribution P and Q becoming more similar to each other.

[0062] The cost function for the gradient descent is the KL divergence between P and Q:

[0063] Once the gradient descent process finished, values of the data set in 2D space could be obtained for visualization.

[0064] Example embodiments are provided so that this disclosure will be thorough and will fully convey the scope to those who are skilled in the art. Numerous specific details are set forth such as examples of specific components, devices, and methods, to provide a thorough understanding of embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that example embodiments may be embodied in many different forms, and that neither should be construed to limit the scope of the disclosure. InAttorney Docket No. 70328-02 some example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail. Equivalent changes, modifications and variations of some embodiments, materials, compositions, and methods can be made within the scope of the present technology, with substantially similar results.

Claims

Attorney Docket No. 70328-02CLAIMSWHAT IS CLAIMED IS:

1. A stereognosis sensor system configured to characterize data of a surface of an object upon contact, the sensor system comprising: a substrate; a triboelectric tactile sensor disposed at a distal contact region of the substrate; the triboelectric tactile sensor includes a conductive electrode and a dielectric layer which generates an electrical signal in response to contact with the surface; a triboelectric proprioception sensor disposed at a proximal region of the substrate, the proprioception sensor is configured to generate an electrical signal in response to movement of one of a joint and a tendon during object manipulation; a signal transmission network electrically coupled to the triboelectric tactile sensor and the triboelectric proprioception sensor; and a processor configured to receive and process the electrical signals to determine at least one of a shape, a surface material, and a mechanical property of the object.

2. The system of Claim 1, wherein the substrate is a glove.

3. The system of Claim 1, wherein the substrate is a mountable substrate configured to be coupled to one of a robotic hand, a prosthetic hand, a robotic gripper, and a manipulator.

4. The system of Claim 2, wherein the triboelectric proprioception sensor is disposed on an interior surface of a cuff portion of the glove.

5. The system of Claim 2, wherein the triboelectric tactile sensor is disposed on a finger portion of the glove.Attorney Docket No. 70328-026. The system of Claim 5, wherein triboelectric tactile sensor includes a plurality of triboelectric tactile sensors each disposed on different finger portions of the glove.

7. The system of Claim 1, wherein the triboelectric tactile sensors are arranged to conformally contact surfaces having one of a flat, a curved, and an edged local geometry.

8. The system of Claim 1, wherein the dielectric layer includes poly dimethylsiloxane (PDMS).

9. The system of claim 1, wherein the triboelectric proprioception sensor is positioned to detect multiple grasp modes for the same object shape.

10. The system of claim 1, further comprising an electrically shielding conductive fabric layer which covers the signal transmission network.

11. A method of using a stereognosis sensor system configured to characterize data of a surface of an object upon contact, the method comprising the steps of: contacting the surface with a triboelectric tactile sensor; acquiring surface charge data via the triboelectric tactile sensor; acquiring tendon motion signal data via the triboelectric proprioception sensor; transmitting the surface charge data and the tendon motion signal data to the processor; and determining at least one of a shape, a surface material, and a mechanical property of the object.Attorney Docket No. 70328-0212. The method of Claim 11, further comprising a step of acquiring surface curvature data via the triboelectric tactile sensor.

13. The method of Claim 11, wherein the step of determining at least one of a shape, a surface material, and a mechanical property of the object includes utilizing a machine learning algorithm trained on a combined tactile and proprioceptive signal dataset.

14. A method of manufacturing a stereognosis sensor system configured to characterize data of a surface upon contact, the method comprising the steps of: coupling a triboelectric tactile sensor at a distal contact region of a substrate; coupling a triboelectric proprioception sensor at a proximal region of the substrate; electrically coupling a signal transmission network between the triboelectric tactile sensor and the triboelectric proprioception sensor; and coupling the triboelectric tactile sensor and the triboelectric proprioception sensor to the processor.

15. The method of Claim 14, wherein each of the triboelectric tactile sensor and the triboelectric proprioception sensor include a conductive textile electrode and a polydimethylsiloxane (PDMS) dielectric layer.

16. The method of Claim 14, further comprising a step of applying a conductive fabric layer over the signal transmission network that provides electromagnetic shielding.

17. The method of Claim 14, furthering comprising a step of fabricating the triboelectric tactile sensor by applying a PDMS dielectric layer onto a conductive electrode, curing the dielectric layer, and cutting the triboelectric tactile sensor into a predetermined dimensionAttorney Docket No. 70328-02 before the step of coupling the triboelectric tactile sensor at the distal contact region of the substrate.

Images