Methods and apparatuses to track hand configurations based on ultrasound signals

Abstract

Systems, apparatuses, and methods for providing a human-computer interface include and / or implement a wearable imaging probe configured to image an internal structure of a body of a wearer; and a memory storing a machine learning algorithm, wherein the machine learning algorithm is configured to receive an imaging data from the wearable imaging probe and determine a pose of the wearer based on the imaging data.

Description

METHODS AND APPARATUSES TO TRACK HAND CONFIGURATIONS BASED ONULTRASOUND SIGNALSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 552,543, filed February 12, 2024, the entire contents of which are herein incorporated by reference for all purposes.STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0002] This invention was made with government support under 1R01HL153857-01 and 1R01HL167947-01 awarded by the National Institutes of Health, EFMA-1935291 awarded by the National Science Foundation, and PR200524P1 awarded by the Department of Defense Congressionally Directed Medical Research Programs. The government has certain rights in the invention.TECHNICAL FIELD

[0003] This disclosure relates to the field of human-computer interface devices, and more particularly to the field of camera-free and hands-free devices for tracking the pose and motion of a user’s hands in real-time.SUMMARY

[0004] The hand is a human’s most dexterous and versatile manipulative organ that interacts with both physical and virtual environments. Hand tracking during daily activities may be relevant to a wide range of technologies, including spatial computing, virtual and augmented reality, simulated training, robotics, prosthetics, biomechanics, and biomedicine. Comparative techniques based on cameras, strain and inertial sensors, and electromyography sensors suffer from severe limitations, including limited view angles and hand positions, constrained hand activities and sensations, and tracking only discrete hand gestures, respectively.

[0005] According to one example of the present disclosure, a human-computer interface system is provided. The interface system comprises a wearable imaging probe configured to image an internal structure of a body of a wearer; and a memory storing a machine learning algorithm, wherein the machine learning algorithm is configured to receive an imaging data from the wearable imaging probe and determine a pose of the wearer based on the imaging data.

[0006] According to another example of the present disclosure, a method of determining a pose of a portion of a user is provided. The method comprises imaging an internal structure of a body of the user using a wearable imaging probe; inputting an imaging data from the wearable imaging probe to a machine learning algorithm; determining the pose of the portion of the user by the machine learning algorithm, based on the imaging data.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. la illustrates a demonstration of inherent limitations of a tracking strategy according to a comparative example.

[0008] FIG. lb illustrates a demonstration of inherent limitations of a tracking strategy according to a comparative example.

[0009] FIG. 2a illustrates a comparison between a comparative example and an example of the present disclosure.

[0010] FIG. 2b illustrates a comparison between a comparative example and an example of the present disclosure.

[0011] FIG. 2c illustrates a comparison between a comparative example and an example of the present disclosure.

[0012] FIG. 2d illustrates a comparison between a comparative example and an example of the present disclosure.

[0013] FIG. 2e illustrates a comparison between a comparative example and an example of the present disclosure.

[0014] FIG. 2f illustrates a comparison between a comparative example and an example of the present disclosure.

[0015] FIG. 2g illustrates a comparison between a comparative example and an example of the present disclosure.

[0016] FIG. 3a illustrates an example design of a wearable high-resolution ultrasound probe in accordance with the present disclosure.

[0017] FIG. 3b illustrates an example design of a wearable high-resolution ultrasound probe in accordance with the present disclosure.

[0018] FIG. 3c illustrates an example design of a wearable high-resolution ultrasound probe in accordance with the present disclosure.

[0019] FIG. 4a illustrates an example of a wearable high-resolution ultrasound probe in accordance with the present disclosure, coupled to the wrist via an ultrasound couplant.

[0020] FIG. 4b illustrates an example of a wearable high-resolution ultrasound probe in accordance with the present disclosure, coupled to the wrist via an ultrasound couplant.

[0021] FIG. 4c illustrates an example of a wearable high-resolution ultrasound probe in accordance with the present disclosure, coupled to the wrist via an ultrasound couplant.

[0022] FIG. 5a illustrates an imaging performance characterization of a high-resolution ultrasound probe in accordance with the present disclosure.

[0023] FIG. 5b illustrates an imaging performance characterization of a high-resolution ultrasound probe in accordance with the present disclosure.

[0024] FIG. 5c illustrates an imaging performance characterization of a high-resolution ultrasound probe in accordance with the present disclosure.

[0025] FIG. 5d illustrates an imaging performance characterization of a high-resolution ultrasound probe in accordance with the present disclosure.

[0026] FIG. 5e illustrates an imaging performance characterization of a high-resolution ultrasound probe in accordance with the present disclosure.

[0027] FIG. 5f illustrates an imaging performance characterization of a high-resolution ultrasound probe in accordance with the present disclosure.

[0028] FIG. 6 illustrates Krimholtz-Leedom-Matthaei simulation results of the ultrasound probe performance, in accordance with the present disclosure.

[0029] FIG. 7 illustrates measured results of the ultrasound probe performance of one element in an array, in accordance with the present disclosure.

[0030] FIG. 8 illustrates measured results of the ultrasound probe performance for all array elements, in accordance with the present disclosure.

[0031] FIG. 9a illustrates the specifications of an ultrasound phantom in accordance with the present disclosure.

[0032] FIG. 9b illustrates the specifications of an ultrasound phantom in accordance with the present disclosure.

[0033] FIG. 10 illustrates an example of the manner in which ultrasound regions characterize the configurations of the five fingers and the palm, in accordance with the present disclosure.

[0034] FIG. I la illustrates the principle and performance of an example of an ultra- dexterous virtual hand in accordance with the present disclosure.

[0035] FIG. 11b illustrates the principle and performance of an example of an ultra- dexterous virtual hand in accordance with the present disclosure.

[0036] FIG. 11c illustrates the principle and performance of an example of an ultra- dexterous virtual hand in accordance with the present disclosure.

[0037] FIG. l id illustrates the principle and performance of an example of an ultra- dexterous virtual hand in accordance with the present disclosure.

[0038] FIG. l ie illustrates the principle and performance of an example of an ultra- dexterous virtual hand in accordance with the present disclosure.

[0039] FIG. I lf illustrates the principle and performance of an example of an ultra- dexterous virtual hand in accordance with the present disclosure.

[0040] FIG. 12 illustrates the architecture of an artificial intelligence model in accordance with the present disclosure.

[0041] FIG. 13 illustrates a displacement analysis to visualize ultrasound regions, in accordance with the present disclosure.

[0042] FIG. 14a illustrates an ultrasound region, in accordance with the present disclosure.

[0043] FIG. 14b illustrates an ultrasound region, in accordance with the present disclosure.

[0044] FIG. 14c illustrates an ultrasound region, in accordance with the present disclosure,

[0045] FIG. 15a illustrates an ultrasound region, in accordance with the present disclosure,

[0046] FIG. 15b illustrates an ultrasound region, in accordance with the present disclosure,

[0047] FIG. 15c illustrates an ultrasound region, in accordance with the present disclosure,

[0048] FIG. 15d illustrates an ultrasound region, in accordance with the present disclosure,

[0049] FIG. 15e illustrates an ultrasound region, in accordance with the present disclosure,

[0050] FIG. 16a illustrates an ultrasound region, in accordance with the present disclosure,

[0051] FIG. 16b illustrates an ultrasound region, in accordance with the present disclosure,

[0052] FIG. 16c illustrates an ultrasound region, in accordance with the present disclosure,

[0053] FIG. 16d illustrates an ultrasound region, in accordance with the present disclosure,

[0054] FIG. 16e illustrates an ultrasound region, in accordance with the present disclosure,

[0055] FIG. 17a illustrates an ultrasound region, in accordance with the present disclosure,

[0056] FIG. 17b illustrates an ultrasound region, in accordance with the present disclosure,

[0057] FIG. 17c illustrates an ultrasound region, in accordance with the present disclosure,

[0058] FIG. 17d illustrates an ultrasound region, in accordance with the present disclosure,

[0059] FIG. 17e illustrates an ultrasound region, in accordance with the present disclosure,

[0060] FIG. 18a illustrates an ultrasound region, in accordance with the present disclosure,

[0061] FIG. 18b illustrates an ultrasound region, in accordance with the present disclosure,

[0062] FIG. 18c illustrates an ultrasound region, in accordance with the present disclosure,

[0063] FIG. 18d illustrates an ultrasound region, in accordance with the present disclosure,

[0064] FIG. 18e illustrates an ultrasound region, in accordance with the present disclosure,

[0065] FIG. 19a illustrates an ultrasound region, in accordance with the present disclosure,

[0066] FIG. 19b illustrates an ultrasound region, in accordance with the present disclosure,

[0067] FIG. 19c illustrates an ultrasound region, in accordance with the present disclosure,

[0068] FIG. 19d illustrates an ultrasound region, in accordance with the present disclosure,

[0069] FIG. 19e illustrates an ultrasound region, in accordance with the present disclosure.

[0070] FIG. 20a illustrates an evaluation of noise sensitivity, in accordance with the present disclosure.

[0071] FIG. 20b illustrates an evaluation of noise sensitivity, in accordance with the present disclosure.

[0072] FIG. 20c illustrates an evaluation of noise sensitivity, in accordance with the present disclosure.

[0073] FIG. 21a illustrates an evaluation of the hysteresis effect, in accordance with the present disclosure.

[0074] FIG. 21b illustrates an evaluation of the hysteresis effect, in accordance with the present disclosure.

[0075] FIG. 22a illustrates an evaluation of the drifting effect, in accordance with the present disclosure.

[0076] FIG. 22b illustrates an evaluation of the drifting effect, in accordance with the present disclosure.

[0077] FIG. 22c illustrates an evaluation of the drifting effect, in accordance with the present disclosure.

[0078] FIG. 23a illustrates an application of the present disclosure.

[0079] FIG. 23b illustrates an application of the present disclosure.

[0080] FIG. 23c illustrates an application of the present disclosure.

[0081] FIG. 23d illustrates an application of the present disclosure.

[0082] FIG. 23e illustrates an application of the present disclosure.

[0083] FIG. 23f illustrates an application of the present disclosure.

[0084] FIG. 23g illustrates an application of the present disclosure.

[0085] FIG. 24a illustrates example gestures, in accordance with the present disclosure.

[0086] FIG. 24b illustrates example signals associated with gestures, in accordance with the present disclosure.

[0087] FIG. 25a illustrates an application of the present disclosure.

[0088] FIG. 25b illustrates an application of the present disclosure.

[0089] FIG. 25c illustrates an application of the present disclosure.

[0090] FIG. 25d illustrates an application of the present disclosure.

[0091] FIG. 25e illustrates an application of the present disclosure.

[0092] FIG. 25f illustrates an application of the present disclosure.

[0093] FIG. 25g illustrates an application of the present disclosure.

[0094] FIG. 26a illustrates a comparison between ultrasound and magnetic resonance imaging in accordance with the present disclosure.

[0095] FIG. 26b illustrates a comparison between ultrasound and magnetic resonance imaging in accordance with the present disclosure.

[0096] FIG. 27a illustrates a comparison between ultrasound and magnetic resonance imaging in accordance with the present disclosure.

[0097] FIG. 27b illustrates a comparison between ultrasound and magnetic resonance imaging in accordance with the present disclosure.

[0098] FIG. 28 illustrates an example human-computer interface system in accordance with the present disclosure.

[0099] FIG. 29 illustrates an example method of determining a pose of a portion of a user in accordance with the present disclosure.DETAILED DESCRIPTION

[0100] The detailed description set forth below in connection with the appended drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the subject matter described herein may be practiced. The detailed description includes specific details to provide a thorough understanding of various embodiments of the present disclosure. However, it will be apparent to those skilled in the art that the various features, concepts, and embodiments described herein may be implemented and practiced without these specific details.

[0101] The present disclosure may be implemented on or with the use of computing devices including control units, processors, and / or memory elements in some examples. As used herein, a “control unit” may be any computing device configured to send and / or receiveinformation (e.g., including instructions) to / from various systems and / or devices. A control unit may comprise processing circuitry configured to execute operating routine(s) stored in a memory. The control unit may comprise, for example, a processor, microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), and the like, any other digital and / or analog components, as well as combinations of the foregoing, and may further comprise inputs and outputs for processing control instructions, control signals, drive signals, power signals, sensor signals, and the like. All such computing devices and environments are intended to fall within the meaning of the term “controller,” “control unit,” “processor,” or “processing circuitry” as used herein unless a different meaning is explicitly provided or otherwise clear from the context. The term “control unit” is not limited to a single device with a single processor, but may encompass multiple devices (e.g., computers) linked in a system, devices with multiple processors, special purpose devices, devices with various peripherals and input and output devices, software acting as a computer or server, and combinations of the above. In some implementations, the control unit may be configured to implement cloud processing, for example by invoking a remote processor.

[0102] Moreover, as used herein, the term “processor” may include one or more individual electronic processors, each of which may include one or more processing cores, and / or one or more programmable hardware elements. The processor may be or include any type of electronic processing device, including but not limited to central processing units (CPUs), graphics processing units (GPUs), ASICs, FPGAs, microcontrollers, digital signal processors (DSPs), or other devices capable of executing software instructions. When a device is referred to as “including a processor,” one or all of the individual electronic processors may be external to the device (e.g., to implement cloud or distributed computing). In implementations where a device has multiple processors and / or multiple processing cores, individual operations described herein may be performed by any one or more of the microprocessors or processing cores, in series or parallel, in any combination.

[0103] As used herein, the term “memory” may be any storage medium, including a nonvolatile medium, e.g., a magnetic media or hard disk, optical storage, or flash memory; a volatile medium, such as system memory, e.g., random access memory (RAM) such as dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), static RAM (SRAM), extended data out (EDO) DRAM, extreme data rate dynamic (XDR) RAM, double data rate (DDR) SDRAM, etc.; on-chip memory; and / or an installation medium where appropriate, such as software media, e.g., a CD-ROM, or floppy disks, on which programs may be stored and / or data communications may be buffered. The term “memory” may also include other types of memory or combinations thereof. For the avoidance of doubt, cloud storage is contemplated in the definition of memory.

[0104] Before any aspects of the invention are explained in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the following drawings. The invention is capable of other aspects and of being practiced or of being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless specified or limited otherwise, the terms “mounted,” “connected,” “supported,” and “coupled” and variations thereof are used broadly and encompass both direct and indirect mountings, connections, supports, and couplings. Further, “connected” and “coupled” are not restricted to physical or mechanical connections or couplings.

[0105] It is also to be understood that any reference to an element herein using a designation such as “first,” “second,” and so forth does not limit the quantity or order of those elements, unless such limitation is explicitly stated. Rather, these designations may be used herein as a convenient method of distinguishing between two or more elements or instances of an element. Thus, a reference to first and second elements does not mean that only two elements may be employed or that the first element must precede the second element in some manner.

[0106] Also as used herein, unless otherwise limited or defined, “or” indicates a nonexclusive list of components or operations that can be present in any variety of combinations, rather than an exclusive list of components that can be present only as alternatives to each other. For example, a list of “A, B, or C” indicates options of: A; B; C; A and B; A and C; B and C; and A, B, and C. Correspondingly, the term “or” as used herein is intended to indicate exclusive alternatives only when preceded by terms of exclusivity, such as, e.g., “either,” “one of,” “only one of,” or “exactly one of.” Further, a list preceded by “one or more” (and variations thereon) and including “or” to separate listed elements indicates options of one or more of any or all of the listed elements. For example, the phrases “one or more of A, B, or C” and “at least one of A, B, or C” indicate options of: one or more A; one or more B; one or more C; one or more A and one or more B; one or more B and one or more C; one or more A and one or more C; and one or more of eachof A, B, and C. Similarly, a list preceded by “a plurality of’ (and variations thereon) and including “or” to separate listed elements indicates options of multiple instances of any or all of the listed elements. For example, the phrases “a plurality of A, B, or C” and “two or more of A, B, or C” indicate options of A and B; B and C; A and C; and A, B, and C. In general, the term “or” as used herein only indicates exclusive alternatives (e.g., “one or the other but not both”) when preceded by terms of exclusivity, such as, e.g., “either,” “one of,” “only one of,” or “exactly one of.”

[0107] The following discussion is presented to enable a person skilled in the art to make and use embodiments of the invention. Various modifications to the illustrated embodiments will be readily apparent to those skilled in the art, and the generic principles herein can be applied to other embodiments and applications without departing from embodiments of the invention. Thus, embodiments of the invention are not intended to be limited to embodiments shown but are to be accorded the widest scope consistent with the principles and features disclosed herein. The following detailed description is to be read with reference to the figures, in which like elements in different figures have like reference numerals. The figures, which are not necessarily to scale, depict selected embodiments and are not intended to limit the scope of embodiments of the invention. Skilled artisans will recognize that the examples provided herein have many useful alternatives and fall within the scope of embodiments of the invention.

[0108] The present disclosure describes a high-resolution wristband (e.g., an ultrasound wristband), integrated with an artificial intelligence (Al) algorithm, that addresses limitations present in comparative examples. The wristband can continuously track all 22 degrees of freedom of a hand’s five fingers and palm in real-time during daily activities with high tracking accuracy (<1.5° root mean square error) and low processing latency (<1 ms). The algorithm may be, for example, a machine learning (ML) model. The algorithm receives image data and predicts hand configurations based on the received image data. The resultant ultra-dexterous virtual hands enabled by the systems and methods set forth herein can provide previously inaccessible, intuitive, and versatile controls for virtual reality and robotic hands. While the following description presents examples of the present disclosure based on high-resolution ultrasound, the present disclosure is not so limited. In practice, the present disclosure may be implemented using any imaging modality that is capable of accurately imaging the interior structure of the wrist. In some examples, the present disclosure may be practiced in implementations using photoacoustic imaging, MRI, optical computed tomography (OCT) imaging, and / or infrared (IR) imaging so long as suchimplementations have sufficient imaging resolution, imaging depth, device size, etc. to operate in accordance with the methods set forth herein.

[0109] The present disclosure provides human-computer interfaces and methods therefor, which may find applicability in a wide range of technical fields. For example, the methods and apparatuses set forth herein may be applied to the fields of virtual reality (VR) (e.g., including augmented reality (AR) and mixed reality (MR)), robotics control, video gaming, telecommunications (e g., interactions in the metaverse), special training for dangerous applications (e.g., bomb squad training), professional training (e.g., DIY or mechanical training), skills training (e.g., virtual piano lessons), biomechanical study and medical applications (e.g., rehabilitation) and the like.

[0110] Dexterous virtual hands, capable of accurately and continuously tracking the configurations and motions of individual fingers and palms in real-time during daily activities, may be used to increase the effectiveness of the interactions among humans, virtual reality, and machines such as robots. However, widely adopted and easily accessible strategies for dexterous virtual hands are still unavailable using systems and methods according to comparative examples. Camera-based systems are used in comparative examples for hand tracking, particularly due to their compatibility with VR headsets. Additionally, quantitatively evaluating the performance of these tracking systems can be challenging because they are highly susceptible to uncontrollable factors, such as the viewing angle, visual obstacles, environmental light, and the like, and these systems only provide a limited field of view. These inherent issues limit the overall performance of these systems. To illustrate the intrinsic limitations of camera-based strategies, FIGS, la and lb show a demonstration using a Leap Motion controller 2 (Ultraleap, Mountain View, CA, USA), a hand tracking system equipped with dual cameras and a depth sensor, in common scenarios (i.e., different viewing angles and a small tool as the visual obstacle). Hand tracking results significantly vary in different scenarios, as shown in FIGS, la and lb, showing the limited robustness of the camera-based tracking strategy.[0U1] In FIG. la, the hand conducts the ‘Pinch’ gesture. The tracking system generates different results when it has different viewing angles. Tracking accuracy decreases when the viewing angle is tilted. In the worst case, the ‘Pinch’ is not trackable because the thumb and index finger are blocked by the palm. In FIG. lb, the hand holds scissors. Tracking accuracy decreases when the hand holds a tool, and sometimes the system loses track of the hand. Even ifsupplemented with algorithms (e.g., machine learning algorithms) in an attempt to track detailed finger and palm motions, the comparative example devices only offer the tracking of a limited number of specific hand gestures. Therefore, comparative camera-based head-mounted devices are constrained to tracking specific hand positions and gestures, and they can be burdensome during prolonged daily wear. Comparative examples of camera-free alternatives for dexterous virtual hands include strain sensors, inertial sensors, and electromyography (EMG) sensors. However, strain and inertial sensors need to be attached to finger and palm joints, thereby constraining normal hand activities and sensations. While EMG sensors can provide camera-free and hand-free armbands for hand tracking, the EMG signals generated from internal muscles are detected from the skin surface with significantly compromised spatial and temporal accuracy. Therefore, comparative examples of EMG sensors can only recognize a few predefined discrete hand gestures, but not continuous degrees of freedom (DOFs).

[0112] Comparative examples exist of armbands based on ultrasound sensors for hand motion tracking. However, these (e.g., EMG-based) armbands are not wearable probes, rely on low-resolution ultrasound sensing or imaging, and are not positioned to directly apply the ultrasound to the wrist; therefore, they remain confined to recognizing predefined discrete hand gestures or a limited number (< 5) of DOFs. As used herein, “low-resolution” ultrasound may be ultrasound imaging with a center frequency lower than 10 MHz. If harmonic imaging technology is used, imaging using a harmonic frequency of greater than 10 MHz may be considered “high- resolution” ultrasound.

[0113] The present disclosure provides continuous high-resolution imaging of a human hand’s wrist assisted by an artificial intelligence (Al) algorithm that can accurately and continuously track all 22 DOFs of the hand’s five fingers and palm in real-time. FIGS. 2a-2g illustrate a comparison between dexterous virtual hands according to the present disclosure and comparative examples. As shown in FIGS. 2a-c, comparative examples of electromyographybased (EMG-based) hand tracking systems only recognize a limited number of predefined discrete hand gestures. In particular, as shown in FIG. 2a, EMG sensors are usually placed on the forearm skin to detect neural signals from muscles. FIG. 2b is an example of signals acquired by the EMG sensors. FIG. 2c shows representative predefined discrete hand gestures recognized by the EMG sensors. In contrast, as shown in FIGS. 2d-g, the wearable high-resolution ultrasound wristband according to the present disclosure can continuously and quantitatively track all 22 DOFs of thehand’s five fingers and palm. As shown in FIG. 2d, the wearable high-resolution ultrasound probe is stably placed on the wrist to capture images of the tendons and muscles. FIG. 2e shows schematics of the 22 DOFs of the hand’s five fingers and palm. FIG. 2f shows a time-series of high-resolution ultrasound images of the wrist tendon-muscle anatomy as acquired by the ultrasound wristband. The scale bar is 1 cm. In FIG. 2g, the ultrasound images are processed by a 22-output regression machine learning model to continuously quantify the 22 DOFs of the hand in real-time.

[0114] The present disclosure may be embodied as a high-resolution ultrasound wristband, integrated with an Al algorithm, that can continuously track all 22 DOFs of a human hand’s five fingers and palm in real-time during daily activities with high tracking accuracy (<1.5° root mean square error) and low processing latency (<1 ms). The Al algorithm may be included in the wearable device itself, or may be stored on an external device (e.g., on a VR headset of the wearer). In implementations where the algorithm is external to the wristband, the wristband may be configured to communicate data with the external device. The high-resolution ultrasound wristband enables previously inaccessible and important applications, including camera-free and hand-free tracking of various continuous and subtle hand motions for intuitive and versatile controls of virtual reality and robotic hands. The performances of the ultra-dexterous virtual hands are superior to those of all comparative virtual hands based on camera-free hand-free armbands (e g., the comparative EMG armbands discussed above). Table 1 is a comparison of comparative example devices (EMG and low-resolution ultrasound) with devices according to the present disclosure.TABLE 1

[0115] In one example, the wristband of the present disclosure comprises a high-resolution wearable ultrasound probe, as illustrated in FIGS. 3a-3c, and an ultrasound couplant, as illustrated in FIGS. 4a-c. The ultrasound couplant can be made of water, water-based gel, hydrogel, oil, oilbased gel, Glycerin, etc. and other compound materials or layered structures. FIG. 3a shows the layered ultrasound probe. An example of design specifications is shown in Table 2. FIG. 3b shows a photo of 1-3 composite piezoelectric material from the top view. FIG. 3c shows a photo of 1-3 composite lead zirconate titanate (PZT)-5 from the side view. Scale bars indicate 100 pm. As shown in FIG. 3a, an example of a wearable imaging probe comprises a backing layer; a circuit layer; a piezoelectric layer including a first electrode, a piezoelectric material, and a second electrode; a first matching layer; and a second matching layer. In some examples, the circuit layer may include a processor and a memory, but in other examples the memory may be disposed on an external device (e.g., a VR headset of the wearer).

[0116] While FIGS. 3a-3c show one example implementation of a wearable ultrasound probe in accordance with the present disclosure, other implementations of a wearable ultrasound probe in accordance with the present disclosure may implement a piezoelectric micromachine ultrasound transducer (PMUT), a capacitive micromachined ultrasound transducer (CMUT), and the like, including combinations of different types of probes and / or transducers. Moreover, while certain examples of the wearable ultrasound probe described herein provide information regarding 22 DOFs of the hand, the ultrasound probe may be combined with or complemented by another sensing modality (e.g., an optical tracking system, an inertial measurement unit (IMU), and the like) to realize full hand motion tracking, including information regarding 27 DOFs of freedom of the hand.

[0117] FIG. 4a shows an example illustration of the wearable high-resolution ultrasound probe of FIGS. 3a-3c imaging the wrist. As illustrated in FIG. 4b, the couplant may include hydrogel, an elastic solid, which is resistant to shear and torsion deformation. As shown in FIG.4c, the hydrogel maintains its shape during large-scale hand motions, ensuring a stable ultrasound imaging window.TABLE 2

[0118] Thus, in one example, a high-resolution ultrasound probe in accordance with the present disclosure comprises an array of 256 high-performance piezoelectric components with a center frequency of 18 MHz. However, in various implementations, a probe in accordance with the present disclosure is not limited to this center frequency and may be usable with any desire ultrasound frequency (including, in certain implementations, “low-frequency” ultrasound). In this example, the ultrasound probe has an overall size of 5.5 cm in length, 4 mm in width, 5 mm in thickness, and a weight of 15 g. The ultrasound probe is connected to a control unit (e.g., Verasonics Vantage system) with two flexible flat cables. The ultrasound probe provides a field of view measuring 5.2 cm in width and 3 cm in depth, covering the muscles and tendons in the wrist that control five fingers and the palm. The axial resolution of the wristband is 0.67 mm, and the lateral resolution is 1.86 mm at 4 cm depth. FIGS. 5a-9b illustrate a characterization of the ultrasound probe of FIG. 3.

[0119] FIG. 5a is an illustration of the phantom resolution and contrast regions at 4 cm depth, indicated by the green, red and blue dotted boxes respectively; FIG. 5b shows zoomed-in views of the resolution and contrast regions, where the scale bar is 5 mm; FIG. 5c shows zoomedin ultrasound imaging results of the phantom resolution region, where the scale bar is 5 mm; FIG. 5d shows data on the two dashed lines for determination of the lateral and axial resolution, respectively, represented as line graphs to determine the lateral and axial resolution in which resolution values indicate the ‘Mean ± standard deviation’ (n = 3 independent measurements); FIG. 5e shows zoomed-in imaging results of the hyperechoic contrast region, in which the imaging contrast resolutions of each reflector are labeled and the scale bar is 5 mm; FIG. 5f shows zoomedin results of the anechoic contrast region, in which the imaging contrast resolutions of each reflector are labeled, the scale bar is 5 mm, and contrast values indicate the ‘Mean ± standard deviation’ (n = 3 independent measurements).

[0120] FIG. 6 illustrates Krimholtz-Leedom-Matthaei simulation results of the ultrasound probe performance. The upper graph shows the electrical impedance spectrum. The lower graph shows the pulse-echo signals and the frequency spectrum. FIG. 7 illustrates measured results of the ultrasound probe performance of the 128th element in the array. The upper graph shows the electrical impedance spectrum. The lower graph shows the pulse-echo signals and the frequency spectrum. FIG. 8 illustrates measured results of the ultrasound probe performance of all 256 elements. The upper graph shows the center frequency distribution. The middle graph of FIG. 8shows the -6 dB proportional bandwidth distribution. The lower graph shows the signal amplitude distribution. Values in the respective titles indicate the ‘Mean ± standard deviation’ (n = 256 elements). FIGS. 9a-b illustrate the specifications of the ultrasound phantom. FIG. 9a shows an illustration of the phantom. FIG. 9b shows the scatter distributions of the phantom. The dotted box indicates the imaging region of the ultrasound probe shown in this disclosure.

[0121] The imaging contrasts of hyperechoic and anechoic reflectors range from 18.63 dB to 2.69 dB. The image resolution and quality guarantee accurate tracking of the 22 DOFs of the hand. In one particular non-limiting example, the ultrasound couplant may be a bioadhesive couplant. In this example, the bioadhesive couplant consists of a soft yet tough hydrogel encapsulated by a thin elastomer membrane. The elastomer membrane prevents dehydration of the hydrogel and provides comfortable skin contact with a dry couplant surface. The elastomer membrane is further coated by a thin bioadhesive layer to robustly adhere the wearable probe to the wrist. The robust adhesion ensures stable and continuous hand tracking over time.

[0122] In one example, during use the wristband is positioned 2 cm away from the carpus and perpendicular to the ulna. The wristband provides continuous high-resolution imaging of the muscles and tendons in the wrist at a frequency of 30 images per second. On each image, six regions characterize the configurations of the five fingers and the palm, respectively, as shown in FIG. 10. In particular, as shown in FIG. 10, on the high-resolution ultrasound image of the wrist, six regions characterize the configurations of the five fingers and the palm, respectively, indicated by red rectangles. Sub-regions in these six regions further characterize the DOFs of individual joints in the corresponding fingers or palm, indicated by green circles. The scale bar is 5 mm.

[0123] FIGS. 1 la-1 If illustrate the principle and performance of an example of an ultra- dexterous virtual hand. FIG. I la shows how the 22 DOFs of the hand are correlated to the features of the corresponding sub-regions in the high-resolution ultrasound image, as will also be described in more detail below with regard to FIGS. 14a-19e. Note that the number of the 22 DOFs is shown in FIGS. 2a and 2g. FIG. 11b shows a sequence of ultrasound images of the sub-region corresponding to the metacarpophalangeal (MCP) joint angle of the index finger in the flexion and extension motion. The index MCP joint angle increases monotonically with the decrease of a landmark angle in the sub-region. FIG. 11c shows the index MCP joint angles predicted by the wristband compared with the ground truth. FIG. l id shows a sequence of ultrasound images of the sub-region corresponding to the proximal interphalangeal (PIP) joint angle of the index fingerin the flexion motion. The index PIP joint angle increases monotonically with the decrease of the distance between two landmarks in the sub-region. FIG. l ie shows the index PIP joint angles predicted by the wristband compared with the ground truth. FIG. 1 If shows the root mean square errors (RMSE) of the 22 DOFs of the hand predicted by the wristband compared with the ground truth. All scale bars indicate 5 mm. Error bars indicate the standard deviation of RMSE (n = 5 independent measurements). Sub-regions in the six regions shown in FIG. 10 further characterize the DOFs of individual joints in the corresponding fingers or palm, as shown in FIG. I la.

[0124] Because a joint’s DOF is characterized by a unique feature in the corresponding sub-region (e.g., the shapes of the muscles and tendons), the interference among different joints’ DOFs is minimal in the high-resolution ultrasound image. The present disclosure implements a compact and efficient Al algorithm for a 22-dimensional output regression model based on the convolutional neural network (CNN), as illustrated in FIG. 12. In FIG. 12, ReLU refers to Rectified Linear Units. To train the CNN, a marker tracking system with multi -view cameras was used to measure the DOFs of the hand as the ground truth values, as will be described with regard to the example detailed below. The trained Al algorithm can quantify all 22 DOFs of the hand’s five fingers and palm by analyzing the high-resolution ultrasound images, which have not been used for training. In some examples, the algorithm may be trained on a user-by-user basis. However, in other examples the algorithm may be trained using a larger data set so as to have the ability to predict hand configurations based on imaging data for a wide range of users. In still other examples, the algorithm may be coarsely trained for all users, and subsequently fine-tuned for an individual user.

[0125] To validate that the wristband according to the present disclosure can accurately and continuously track the 22 DOFs of the hand, a validation test was performed. In the test, a human subject varied only one DOF of the joints each time while using the wristband to track the 22 DOFs. The change of each DOF is correlated to a distinct shape change in the corresponding sub-region in the ultrasound image, as shown in FIG. 1 Ib-e and FIGS. 13 - 19e. To further quantify and visualize the relationship between characterizing regions in ultrasound images and degrees of freedom (DOFs), displacement analysis was performed on 22 pairs of ultrasound images. Each pair of images consisted of the start and end frame of one DOF action. Displacement maps (see FIG. 12) visualized changes in ultrasound images during hand motions and can highlight minor displacements that are imperceptible to humans yet noticeable to the computer. Whiledisplacement maps were not used as inputs to machine learning models, they can help to see ultrasound images from a view that is closer to computers and enhance the interpretability of the machine learning model.

[0126] For example, FIG. 11b presents a sub-region that corresponds to the metacarpophalangeal (MCP) joint angle of the index finger (i.e., DOF 7) during the flexion and extension motion. The index MCP joint angle (i.e., DOF 7) increases monotonically with the decrease of a landmark angle in the sub-region. The Al algorithm can capture this correlation and accurately predict the index MCP angle by analyzing the ultrasound image (see FIG. 2c). As another example, FIG. l id presents a sub-region of the ultrasound image that corresponds to the proximal interphalangeal (PIP) joint angle of the index finger (i.e., DOF 9) in the flexion motion. The index PIP joint angle (i.e., DOF 9) increases monotonically with the decrease of the distance between two landmarks in the sub-region. Similarly, the Al algorithm can accurately predict the index PIP angle (see FIG. l ie). In addition, the Al algorithm can accurately quantify the changing DOF (see FIG. 2g), while recognizing that the other 21 DOFs remain in their unchanged baseline states.

[0127] FIG. 13 shows a displacement analysis to visualize the regions in ultrasound images that characterize 22 DOFs . The displacement maps show the changes in the ultrasound images during the actions of the 22 DOFs . FIGS. 14a, 15a, 16a, 17a, 18a, and 19a shows the relationship between the 22 DOFs of the hand and the corresponding sub-regions in the high-resolution ultrasound image. FIGS. 14a- 14c illustrate ultrasound image sequences showing palm motions, in which FIG. 14b shows the sub-region on the ultrasound image correlated to DOF 1 of the hand and FIG. 14c shows the sub-region on the ultrasound image correlated to DOF 2 of the hand. FIGS. 15a-15e illustrate ultrasound image sequences showing thumb motions, in which FIG. 15b shows the sub-region on the ultrasound image correlated to DOF 3 of the hand; FIG. 15c shows the subregion on the ultrasound image correlated to DOF 4 of the hand; FIG. 15d shows the sub-region on the ultrasound image correlated to DOF 5 of the hand; and FIG. 15e shows the sub-region on the ultrasound image correlated to DOF 6 of the hand. FIGS. 16a-16e illustrate ultrasound image sequences showing index finger motions, in which FIG. 16b shows the sub-region on the ultrasound image correlated to DOF 7 of the hand; FIG. 16c shows the sub-region on the ultrasound image correlated to DOF 8 of the hand; FIG. 16d shows the sub-region on the ultrasound image correlated to DOF 9 of the hand; and FIG. 16e shows the sub-region on the ultrasound imagecorrelated to DOF 10 of the hand. FIGS. 17a-17e illustrate ultrasound image sequences showing middle finger motions, in which FIG. 17b shows the sub-region on the ultrasound image correlated to DOF 11 of the hand; FIG. 17c shows the sub-region on the ultrasound image correlated to DOF 12 of the hand; FIG. 17d shows the sub-region on the ultrasound image correlated to DOF 13 of the hand; and FIG. 17e shows the sub-region on the ultrasound image correlated to DOF 14 of the hand. FIGS. 18a-18e illustrate ultrasound image sequences showing ring finger motions, in which FIG. 18b shows the sub-region on the ultrasound image correlated to DOF 15 of the hand; FIG. 18c shows the sub-region on the ultrasound image correlated to DOF 16 of the hand; FIG. 18d shows the sub-region on the ultrasound image correlated to DOF 17 of the hand; and FIG. 18e shows the sub-region on the ultrasound image correlated to DOF 18 of the hand. FIGS. 19a-19e illustrate ultrasound image sequences showing thumb motions, in which FIG. 19b shows the subregion on the ultrasound image correlated to DOF 19 of the hand; FIG. 19c shows the sub-region on the ultrasound image correlated to DOF 20 of the hand; FIG. 19d shows the sub-region on the ultrasound image correlated to DOF 21 of the hand; and FIG. 19e shows the sub-region on the ultrasound image correlated to DOF 22 of the hand.

[0128] To further validate that the 22 DOFs tracked by the wristband are quantitatively accurate, the root mean square error between the tracked DOF and the corresponding ground truth measured by the multi-view camera system was calculated. The root mean square errors of the 22 DOFs tracked by the wristband range from 0.53° to 1.37°, indicating a high tracking accuracy (see FIG. I lf). In comparison, the root mean square errors of the DOFs tracked by comparative examples of camera-free hand-free armbands (e.g., the EMG armbands discussed above) range from 7.35° to 22.58°, and these armbands can only track a limited number (< 5) of DOFs. Furthermore, the processing time of each ultrasound image by the Al algorithm ranges from 0.67 ms to 0.93 ms. This low processing latency enables real-time tracking of the 22 DOFs of the hand motion using the wristband. To validate the general applicability of the wristband, another human subject repeated the test to track the 22 DOFs of the hand.

[0129] In addition to the real-time, high-accuracy, and low-latency tracking of the hand’s 22 DOFs, the high-resolution ultrasound wristband of the present disclosure also demonstrates exceptional robustness against noise, hysteresis, and drifting effects during hand tracking. These effects constitute limitations for the robustness and performance of comparative examples of camera-free hand-tracking techniques, including strain sensors and EMG sensors.

[0130] To evaluate the sensitivity of the wristband to noises, ultrasound image datasets were generated with varying signal-to-noise ratios (i.e., 14 dB, 8 dB, 2 dB, -6 dB) by introducing random Gaussian noises with different standard deviations. FIGS. 20a-20c illustrate this evaluation of the noise sensitivity of the ultrasound wristband. FIG. 20a illustrates ultrasound images with different levels of white-gaussian noise. FIG. 20b illustrates the tracking results of all 22 DOFs on data with different levels of noise. The model was trained only on the data without add-on noise. Error bars show the standard deviation of RMSE (n = 5 independent measurements). FIG. 20c illustrates the overall tracking accuracy changes along with the signal-to-noise ratio (SNR). Error bars show the standard error of the mean ( / / = 22 DOFs). 5 is the standard deviation of the added white gaussian noises. While the Al algorithm is only trained with a noiseless image dataset, the algorithm may be used to quantify the 22 DOFs by analyzing ultrasound image datasets with varying SNRs. The RMSE of the DOF is used to evaluate the sensitivity to noise. The RMSE is smaller than 3.3° when the SNR is higher than 8 dB, and smaller than 11° when the SNR is as low as -6 dB. This low noise sensitivity is because the wristband relies on images and patterns to quantify the DOFs. In contrast, strain sensors and EMG sensors according to comparative examples are known to be sensitive to noise because they rely on linear data curves for hand tracking, which are sensitive to noise.

[0131] To evaluate the hysteresis effect of the wristband, a human subject performed a pair of actions of “palm close” and “palm open” because these actions involve 20 out of the 22 DOFs and induce substantial changes in the ultrasound images. Both actions are executed for one second consecutively, resulting in two groups of 30 images. The hysteresis effect is quantified by the similarity between a pair of ultrasound images from the “palm close” and “palm open” groups, respectively. This evaluation is illustrated in FIGS. 21a and 21b. FIG. 21a shows photos of the hand performing palm open and close actions, and the corresponding ultrasound images. FIG. 21b shows a comparison of ultrasound images between the palm open action and the palm close action. High similarity between corresponding images indicates a low hysteresis effect. While all similarities between the paired images are larger than 0.96, the similarities among unpaired images are generally lower than 0.4. This indicates a low hysteresis effect of the wristband.

[0132] The drifting effect is often due to the fatigue of hand-related neurons and muscles after long-term usage. To evaluate the drifting effect of the wristband, a human subject exercised the hand for one hour using a hand gripper and a wrist curl. Following the exercise, the ultrasonicwristband was employed to capture the wrist images. This evaluation is illustrated in FIGS. 22a- 22c. FIG. 22a shows photos of hand and wrist training to cause nerve and muscle fatigue. FIG. 22b shows two representative photos and corresponding ultrasound images before and after the 1- hour exercise. FIG. 22c shows a comparison of tracking results on the data acquired before and after the 1-hour exercise. The model was trained only on the data acquired before the exercise. Error bars show the standard deviation of RMSE (ri = 5 independent measurements). No discernible difference can be observed between ultrasound images for the same hand configuration before and after the exercise. Furthermore, the Al algorithm was used to quantify the DOFs of the hand before and after the exercise. The RMSEs of the DOFs are consistently lower than 1.5°, affirming the low drifting effect of the wristband.

[0133] Systems, methods, and apparatuses according to the present disclosure allow previously inaccessible applications of the ultra-dexterous virtual hands enabled by the high- resolution ultrasound wristband in virtual reality and robotics. In virtual reality, the wristband can provide a hand-free camera-free human-computer interface that manipulates objects using an ultra- dexterous virtual hand with the 22 DOFs. This is illustrated in FIGS. 23a-23g. As shown in FIG. 23a, the ultrasound wristband provides a hand-free camera-free human-computer interface that manipulates objects using an ultra-dexterous virtual hand with the 22 DOFs in virtual reality. FIG. 23b shows sub-regions of an ultrasound image corresponding to the metacarpophalangeal (MCP) joint angle and proximal interphalangeal (PIP) joint angle of the index finger and the MCP joint angle of the thumb to control the pinch action. FIG. 23c shows the pinch distance D, calculated in real-time to control the size of the photo in virtual reality. FIG. 23d shows the quantified pinch distance D over 35 seconds. Both ‘agile move’ and ‘hold’ of the pinch action are accurately tracked. As shown in FIG. 23e, the ultra-dexterous virtual hand controls the photo with three other actions, including rotating the photo around two axes by rotating the palm in two ways and moving the photo forward and backward by bending the four fingers. FIG. 23f shows the quantified relevant DOFs of the virtual hand that is sequentially performing the four actions over 120 seconds. FIG. 23g shows the root mean square errors (RMSE) of the relevant DOFs of the virtual hand that is sequentially performing the four actions over 120 seconds, compared with the ground truth. Error bars indicate the standard deviation of RMSE (w = 5 independent measurements).

[0134] To demonstrate that the wristband can continuously reconstruct naturalistic and complex hand gestures and motions by tracking combinations of the 22 DOFs, the human subjectswere required to continuously perform 69 multi-DOF hand gestures that represent the 10 numbers, 26 letters, and all 33 human grasp types, respectively. These are represented in FIG. 24a. The wristband can continuously reconstruct the 69 hand gestures as well as the undefined hand gestures and motions during the transitions between the 69 hand gestures. The signals used for the reconstruction are shown in FIG. 24b. The ultrasound wristband continuously reconstructs the hand gestures and motions by tracking combinations of the 22 DOFs of the hand in real time. To quantify the tracking accuracy of the 22 DOFs for performing the hand gestures and motions, the root mean square error between the tracked DOFs and the corresponding ground truth measured by the multi-view camera system was calculated. The root mean square errors of the 22 DOFs tracked by the wristband ranged from 1.39° to 5.92°, indicating a high tracking accuracy. In comparison, the root mean square errors of the DOFs tracked by comparative camera-free hand- free armbands range from 7.35° to 22.58°, and these armbands can only track a limited number (<6) of DOFs. Furthermore, the processing time of each ultrasound image by the Al algorithm ranged from 0.67 ms to 0.93 ms. This low processing latency enables real-time tracking of combinations of the 22 DOFs of the hand to continuously reconstruct naturalistic and complex hand gestures and motions.

[0135] FIGS. 23a-23d show that the high-resolution ultrasound wristband of the present disclosure permits additional movements. For example, a human subject can pinch the thumb and index finger to various degrees to accurately manipulate the size of a photo in virtual reality. The wristband can continuously quantify the distance between the two fingertips by tracking the corresponding DOFs, and adjust the photo size accordingly. In contrast, comparative example hand-free camera-free human-computer interfaces such as EMG sensors can only qualitatively detect the pinch gesture and use it as a click command. In addition, the ultra-dexterous virtual hand can stably hold the photo at any specific size, which is also unachievable with comparative example hand-free camera-free human-computer interfaces.

[0136] FIGS. 23e-23g further show that the human subject can rotate the photo around two axes and perform translational movements of the photo by naturally rotating the palm and bending the four fingers, respectively. Section f gives the quantified DOFs of the relevant joints when the hand is sequentially performing the four actions, including pinching, rotating the palm in two ways, and bending the four fingers. The root mean square errors of the DOFs are consistently lower than 0.95 cm, 0.60°, 1.12°, and 0.85° throughout the four actions (see FIG. 23g). Furthermore, thewristband can also track the hand when performing these actions simultaneously, further demonstrating the intuitive and versatile controls in virtual reality.

[0137] In robotics, the wristband can provide a hand-free camera-free human-computer interface that can control the 22 DOFs of a robotic hand. This is illustrated in FIGS. 25a-25g, in which, to demonstrate the intuitive and dexterous control of the robotic hand, a human subject controlled a robotic hand to play a desktop basketball game while wearing the wristband. FIG. 25a shows the intuitive and versatile controls of a robotic hand with the ultra-dexterous virtual hand enabled by the high-resolution ultrasound wristband. In FIG. 25b, a robotic hand is controlled by the virtual hand in real-time to play a desktop basketball game. FIG. 25c is a comparison of the real hand and the robotic hand that is playing the desktop basketball game. FIG. 25d shows the quantified relevant DOF (DOF 7) of the virtual hand that controls the robotic hand while playing the desktop basketball game, measured over a 12-second period. FIG. 25e is a comparison of the real hand and robotic hand that is playing a piano. FIG. 25g shows the quantified relevant DOFs (DOFs 7, 11, 15, and 19) of the virtual hand that controls the robotic hand while playing the piano, measured over a 6-second period.

[0138] In the demonstration, the human subject bent the MCP joint of the index finger to various angles to quantitatively control the robotic hand. The index finger of the robotic hand bent to the corresponding degrees to press down the ball-shooting pad in real-time. After a few trials of the robotic hand controlled by the human subject, an optimal bending angle of the index finger has been determined to shoot the ball into the hoop (see FIG. 25d). It was further demonstrated that the human subject can control multiple fingers of the robotic hand to perform more complicated tasks, such as playing a piano. To demonstrate the general applicability of the ultra-dexterous virtual hand in robotics, another human subject repeated the control of the robotic hand to play the piano.

[0139] Example

[0140] One example ultrasound array encompassed 256 channels with a central frequency of 18 MHz and a pitch size of 0.2 mm. Design and material selection were optimized using the Krimholtz-Leedom-Matthaei (KLM) model simulation tool (Biosono Inc., Fremont, CA) and summarized in Table 2 above.

[0141] The ultrasound array was constructed using high-performance 1-3 composite piezoelectric material. Soft lead zirconate titanate (PZT-5, Del-Piezo Specialties, West Palm Beach, FL) was chosen as the piezoelectric layer, due to its high electromechanical coupling factor and high dielectric constant, which increased the power efficiency of the designed array. The kerf between elements was filled with EPO-TEK 301 (Epoxy Technology, Billerica, MA).

[0142] The array’s backend was connected to a thin flexible printed circuit board (F-PCB) and a high-attenuation acoustic backing layer, mitigating the ringdown noise in ultrasound signals. The backing layer for the ultrasound probe provided mechanical support to the elements inside the probe that will generate high-frequency vibrations. The backing layer also had strong attenuation of the ultrasound wave to effectively shorten the pulse duration and thus increase the imaging resolution. When the acoustic impedance match between the backing and the piezoelectric material becomes better, the generated pulse will be shorter while the amplitude of the pulse will be lower. Thus, the backing material was carefully in view of this trade-off. In the example design, the KLM model was used to select the design of the backing layer. A 3-mm-thick backing layer made of 3022 E-Solder conductive adhesive (Von Roll, Breitenbach, Switzerland), approximately 24 times the wavelength, was added to the back of the piezoelectric layer. This layer provided mechanical adhesion (adhesive strength: 2030 psi) and electrical connection to the printed circuit board. Additionally, due to the relatively low longitudinal sound velocity (1920 m / s) and high acoustic attenuation coefficient (-3.67 dB / mm / MHz), this backing layer improved the axial resolution without significantly increasing the size of the array.

[0143] To further increase the pressure amplitude and axial resolution of the array, a duallayer acoustic impedance matching was implemented. The matching layers smoothed the acoustic impedance mismatch between the piezoelectric material and the skin, allowing the acoustic wave from the transducer to smoothly penetrate the skin and the reflected acoustic waves (the returning echo) to smoothly return to the transducer for imaging. The theoretical acoustic impedances of the dual-layer matching are given by the following Equations (1) and (2):where Zmis the acoustic impedance of matching layer z (z = 1 or 2), and ZPand Zware the acoustic impedance of the piezoelectric material and the water, respectively. The 1-3 composite piezoelectric material adopted in this example had an acoustic impedance of 17.1 MRayl and the water had an acoustic impedance of 1.54 MRayl. Based on Equations (1) and (2), Zmi= 6.09 MRayl and Zm2 = 2.17 MRayl. In the example, a quarter-wavelength 2-3-pm silver epoxy composite with a thickness of quarter wavelength was attached to the surface of the piezoelectric layer as the first layer of the matching. The silver epoxy included 2-3-pm silver powders (Fisher Scientific, Hampton, NH) and the epoxy mixture included Insulcast 501 and Insulcure 9 (American Safety Technologies, Roseland, NJ). The acoustic impedance of the 2-3-pm silver epoxy is adjustable by changing the ratio of the silver powder. It was adjusted to 6.1 MRayl in this example. Afterwards, a quarter- wavelength parylene C was coated on the array surface as the second layer of the matching. It also provided insulation and protection.

[0144] The PZT composite with electrodes in both sides was coated with the 2-3-pm silver epoxy on the top side as the first matching layer. After curing at room temperature for 24 hours, the whole stack was lapped to the designed thickness. To prepare the PZT element of each channel, the composite piezoelectric material was further cut by a sub-scratch-dicing process (Tear 864-1, Thermocarbon, Casselberry, FL). Subsequently, the prepared PZT composite with 256 separated channels was bonded and glued to the designed F-PCB. The top side of the material and the ground pads of the F-PCB were then sputtered with a Cr / Au (50 / 100 nm) electrode by a sputtering system (NSC-3000 Sputter Coater, Nano-Master, Inc., Austin, TX) for ground connection. The E-solder 3022 backing was glued to the other side of the F-PCB as the backing layer. Finally, the amount of Parylene C film was coated on the surface of the materials as the second matching layer.

[0145] To characterize the performance of the fabricated ultrasound array, KLM modeling, impedance spectrum measurements, and pulse-echo tests were conducted. The electric impedance of the piezoelectric element was 211 at 17.7 MHz with a proportional -6-dB bandwidth of 61.6% (see FIGS. 6 and 7). Modeling results and experiment results matched well. The measured center frequency of all elements was 16.9 MHz ± 1.1 MHz, bandwidth was 63.4% ± 3.2%, and a signal amplitude difference was less than -6 dB (see FIG. 8).

[0146] The performance of each element in the fabricated array (center frequency, bandwidth, signal amplitude, and level of crosstalk) was characterized by connecting the array to the 256-channel Vantage system (Verasonics, Inc., Kirkland, WA). The pulse-echo signal (1 -cyclepulse, 20 Vpp) of each element was acquired and analyzed to determine the center frequency, bandwidth, and signal amplitude. The electrical impedance of the fabricated array was assessed in air using an impedance analyzer (E4990A, Keysight, Santa Rosa, CA).

[0147] The 256-channel Vantage system (Verasonics, Inc, Kirkland, WA) was used for imaging. Radio-frequency signals received by the ultrasound probe were acquired, digitized, and post-processed in real-time by the Vantage system. Ultrasound imaging was performed using a line-focused beamforming mode with an electrical focus at 1 cm depth. The imaging speed was 90 frames per second. All imaging algorithms and image post-processing were adapted from the algorithm packages of the Verasonics Vantage system. The ultrasound imaging speed can be improved to over 500 frame rates by using the plane wave beamforming mode. Thus, imaging speed does not limit the tracking rate of the ultrasonic wristband.

[0148] A multi-purpose multi-tissue phantom (MODEL 040GSE, CIRS Inc., Norfolk, VA, USA) was used in the phantom imaging test. The resolution section of the phantom was made of 80 pm diameter nylon monofilament wires. Their axial separations were 4, 3, 2, 1, 0.5 and 0.25 mm, and lateral separations were 4, 3, 2, 1, 0.5 and 0.25 mm. The resolution of the ultrasound imaging probe was determined by measuring the full beam width at half maximum of single wire. The measured resolutions were also confirmed by the minimum distance between distinguishable phantom wires. The contrasts of hyperechoic reflectors were 3, 6, and 15 dB, from left to right. The contrasts of anechoic reflectors were -9, -6, and -3 dB, from left to right. To quantify the image contrast, signals from the reflector-free regions at the same depth (4 cm) were taken as the background signals (Sb). Signals from each reflector were taken as reflector signals (Sr). The imaging contrasts were calculated according to the following Equation (3).

[0149] An 8-camera hand motion tracking system (Motion Analysis, Rohnert Park, CA) with 25 markers was used to capture hand motions in real time at a frame rate of 100 Hz (up to 810 Hz). 3D positions of each marker were acquired and used to calculate the bending angles of 22 DOFs. To synchronize the hand tracking data and ultrasound images acquired from the Verasonics Vantage system, a linear interpolation was performed on the hand tracking data to match the time stamp of each ultrasound image. All experiments were approved by theMassachusetts Institute of Technology Committee on the Use of Humans as Experimental Subjects. Two intact human participants (1 male and 1 female, aged 28 and 23 years) with no reported neurological disorders were recruited in the experiment.

[0150] Before being fed into the machine learning model, all ultrasound images were pre- processed. All images were sampled to have a size of 250 by 150 pixels and normalized to have a value from 0 to 1. FIG. 1 lb illustrates the detailed architecture of the CNN-based 22-dimensional output regression model according to the present disclosure. Feature maps of each image were first extracted by two layers of convolutional neural network without pre-trained weights followed by a 2-by-2 Max pool layer. Next, two fully connected layers with ReLU as the activation functions were used to achieve 22 outputs. The learning objective was to minimize the mean square error of all 22 outputs. Optimization of the model was performed using an adaptive moment estimation (ADAM) optimizer in a batch size of 64 and 50 epochs. Max pooling is used to reduce the dimensionality of the data by passing only the locally highest activations.

[0151] The models in this example were implemented on a Windows 11 computer equipped with an Intel i9- 13900k processing unit and NVIDIA RTX 4090 graphics processing unit. The model development platform was Python 3.10 with the PyTorch (version 2.10; pytorch.org) and CUDA (version 12.1) deep-learning framework. For analyzing the effects of hysteresis, the similarities between ultrasound images were computed using the structural similarity index function from MATLAB R2022a (Mathworks, Natick, MA).

[0152] Virtual reality demonstration was developed using Unity 2021.3.29. The ultrasound images acquired from the Verasonics Vantage system were processed in Python to predict the value of 22 DOFs. The outputs from the Python script were returned to Unity C# to control the object in virtual space in real time. For robotic hand control, a programmable 6-DOF (5 for the fingers and 1 for the wrist) robotic hand (uHandPi, Hiwonder, Shenzhen) was used. The communication between the robotic hand and the computer was via the STM32 controller and the applicable programing interface provided by the manufacturer. The controlling scripts were developed based on the script packages from the manufacturer and with the help from its customer service.

[0153] To better analyze the anatomical meaning of each ultrasound region, magnetic resonance imaging (MRI) was performed to capture cross-sectional images at the same wrist position and aligned MRI images with ultrasound images. The analysis is illustrated in FIGS. 26a-26b and 27a-27b, which show the consistency between characterizing regions of 22 DOFs in the ultrasound image and the anatomy locations in the magnetic resonance image (MRI). FIGS. 26a and 27a show the distribution of regions that characterize all DOFs in the high-resolution ultrasound image. The scale bar is 5 mm. FIGS. 26b and 27b show MRI imaging of the same crosssection of the wrist. The darker color shows the tissue with fewer water molecules. Tendons that control the motion of the hand are labeled as follows: FDS - Flexor Digitorum Superficialis; FDP - Flexor Digitorum Profundus; FCU - Flexor Carpi Ulnaris; FCR - Flexor Carpi Radialis; FPL - Flexor Pollicis Longus; PL - Palm Longus. The scale bar is 5 mm.

[0154] MRI of the wrist were obtained on a Three Tesla, Siemens Magnetom, Prisma MRI scanner (Siemens Healthcare GmbH). A home-built, 100mm diameter, receive-only, circular surface coil was used for improving the signal noise ratio over the available RF coils and to eliminate signal from the contra-lateral wrist and other anatomy. Seventeen 3 mm slices were acquired with a fast spin echo (FSE) pulse sequence with TR = 951 ms, TE = 15 ms, a turbo factor of 6, and a refocusing angle of 150 degrees. In-plane pixel size was 0.22 mm by 0.22 mm with a slice thickness of 3 mm. Total scan time was just over 6 minutes. Slices were taken orthogonally to the axis of the ulna and radius of a volunteer. The distribution of characterizing regions in the ultrasound images was consistent with the anatomical distribution of tendons and muscles responsible for controlling each DOF. For example, the FDS tendon of the middle finger, which is used in flexing both middle and proximal phalanges, is slightly above the FDS tendon of the index finger. FDP tendons of the index to the little fingers are deeper, distributed from left to right.

[0155] EMG signals in FIG. 2b were generated from using an open-access online dataset and were plotted using MATLAB. No additional signal processing was performed on the data.

[0156] To compute the displacement maps between two ultrasound images, a customized script based on 2D cross-correlation was developed in MATLAB. Ultrasound images were firstly resized to be 200 by 100 pixels and normalized to 0 to 1. To compute the displacement, the crosscorrelation was performed using a 15 -by- 15 -pixel window shifting with a 1-pixel step. The distance between the maximum value in correlation results and the center is the displacement. Only the displacement with a correlation value larger than 0.5 is plotted. While both lateral and axial displacements are available, only axial displacement maps were selected as representative results.

[0157] FIG. 28 illustrates an example of a human-computer interface system 100 in accordance with various aspects of the present disclosure. In the illustrated example, the system 100 includes a wearable probe 110 configured to sense an internal structure of a body of a wearer of the wearable probe 110, and a processing unit 120 in communication with the wearable probe 110. The processing unit 120 may be configured to operate a signal processing algorithm (e.g., an image processing algorithm) that may be configured to receive signal data (e.g., image data) from the wearable probe and output a pose of the wearer based on the signal data.

[0158] The wearable probe 110 and the processing unit 120 may be operatively connected by a communication interface between the wearable probe 110 and the processing unit 120, illustrated as a dashed line in FIG. 28. The communication interface may be a wired interface (e.g., a universal serial bus (USB) interface, an Ethernet interface, and / or any other interface that enables communication over a wire), a wireless interface (e.g., a Wi-Fi interface, a Bluetooth interface, a near-field communication (NFC) interface, an optical interface, and / or any other interface that enables communication using electromagnetic radiation), or combinations thereof.

[0159] The wearable probe 110 may include at least one of a single ultrasonic transducer, multiple ultrasonic transducers, an array of ultrasonic transducers, one or more PMUTs, or one or more CMUTs, in any combination. In one example, the wearable probe 110 may include a backing layer, a circuit layer, a piezoelectric layer, and a matching layer (e.g., as illustrated in FIG. 3). The backing layer may be configured to provide mechanical support to the wearable probe. The circuit layer may include circuitry configured to extract a plurality of data channels from the signal data. The piezoelectric layer may include a piezoelectric material and first and second electrode layers. The matching layer may be a layer of a material configured to reduce an acoustic impedance mismatch between the piezoelectric layer and a skin of the wearer. The wearable probe 110 may be or include a wristband and / or a patch. In some examples, an ultrasound couplant may be provided to couple the wearable probe 110 to a surface of the body. In such examples, the ultrasound couplant may be an elastic solid gel, such as a hydrogel.

[0160] The processing unit 120 may include at least one electronic processor and a memory. In some examples, the algorithm may be stored in the memory and directly invoked by the processing unit 120. In other examples, the algorithm may be stored in a remote memory (e.g., in a cloud-based memory structure), and the processing unit 120 may transmit inputs to and receiveoutputs from the remote algorithm. The algorithm may be, in examples, a machine learning algorithm, such as any one or more of the ML algorithms set forth above.

[0161] In the illustrated example, the wearable probe 110 is located at or near the wearer’s wrist, and thus the internal structure sensed by the wearable probe 110 may be an internal structure of the wearer’s wrist. The signal data may, in such examples, be used to determine (as the pose of the wearer) a hand configuration of the wearer. The hand configuration may include information regarding seven or more DOFs of the hand, and in some implementations may include information regarding twenty -two DOFs of the hand.

[0162] FIG. 29 illustrates an example of a method 200 of determining a pose of a portion of a user. For purposes of explanation, the method 200 will be described as being performed by or in conjunction with the system 100 of FIG. 28. However, it should be understood that the method 200 may be performed by or in conjunction with any of the systems described in the present disclosure.

[0163] Method 200 includes an operation 210 of sensing an internal structure of a body of the user using a wearable probe (e.g., the wearable probe 110). In examples, the wearable probe may be an ultrasound imaging probe, for example disposed at or near the user’s wrist. Method 200 further includes an operation 220 of inputting a sensing data from the wearable probe to a signal processing algorithm (e.g., an ML algorithm). The sensing data may be received from a sensor of the wearable probe, for example via a wired and / or wireless communication interface. Method 200 further includes an operation 230 of determining the pose of the portion of the user by the algorithm, based on the sensing data. In examples, the pose of the portion of the user may be a hand configuration of the user.

[0164] The hand is the most dexterous and versatile manipulative organ in the human body. As shown in the present disclosure, high-resolution ultrasound imaging of tendons and muscles in the wrist can continuously and accurately track all 22 DOFs of the hand’s five fingers and palm in real-time. Similarly, wearable ultrasound patches may image tendons, muscles, and ligaments in other parts of the body to track other DOFs of the full human body. A set of such ultrasound bands and patches could provide a camera-free and wearable strategy for continuously and accurately tracking the configurations and motions of the full human body in real-time during daily activities.

[0165] Other examples and uses of the disclosed technology will be apparent to those having ordinary skill in the art upon consideration of the specification and practice of the invention disclosed herein. The specification and examples given should be considered exemplary only, and it is contemplated that the appended claims will cover any other such embodiments or modifications as fall within the true scope of the invention.

[0166] The Abstract accompanying this specification is provided to enable the United States Patent and Trademark Office and the public generally to determine quickly from a cursory inspection the nature and gist of the technical disclosure and in no way intended for defining, determining, or limiting the present invention or any of its embodiments.

Claims

CLAIMSWhat is claimed is:

1. A human-computer interface system, comprising: a wearable probe configured to sense an internal structure of a body of a wearer; and a processing unit in communication with the wearable probe and configured to operate a signal processing algorithm, wherein the algorithm is configured to receive signal data from the wearable probe and output a pose of the wearer based on the signal data.

2. The system of claim 1, wherein the wearable probe includes at least one of a single ultrasonic transducer, multiple ultrasound transducers, an array of ultrasound transducers, one or more Piezoelectric Micromachined Ultrasonic Transducers (PMUT), or one or more Capacitive Micromachined Ultrasonic Transducers (CMUT).

3. The system of claim 1 or claim 2, wherein the internal structure of the body of the wearer is an internal structure of a wrist of the wearer.

4. The system of any one of claims 1 to 3, wherein the pose of the wearer is a hand configuration of the wearer.

5. The system of claim 4, wherein the hand configuration of the wearer includes information regarding more than or equal to seven (> 7) degrees of freedom of the hand.

6. The system of claim 4 or claim 5, wherein the hand configuration of the wearer includes information regarding twenty-two degrees of freedom of the hand.

7. The system of any one of claims 1 to 6, wherein the algorithm is stored in a memory of the wearable probe.

8. The system of any one of claims 1 to 7, further comprising a communication interface between the wearable probe and the processing unit, wherein the processing unit includes at least one electronic processor and a memory.

9. The system of any one of claims 1 to 8, further comprising an ultrasound couplant configured to couple the wearable probe to a surface of the body of the wearer.

10. The system of claim 9, wherein the ultrasound couplant includes an elastic solid gel.

11. The system of any one of claims 1 to 10, wherein the wearable probe includes: a backing layer; a circuit layer; a piezoelectric layer; and a matching layer.

12. The system of claim 11, wherein the backing layer is configured to provide mechanical support to the wearable probe.

13. The system of claim 11 or claim 12, wherein the circuit layer includes circuitry configured to extract a plurality of data channels from the signal data.

14. The system of any one of claims 11 to 13, wherein the piezoelectric layer includes a first electrode layer, a piezoelectric material, and a second electrode layer.

15. The system of any one of claims 11 to 14, wherein the matching layer is configured to reduce an acoustic impedance mismatch between the piezoelectric layer and a skin of the wearer.

16. The system of any one of claims 1 to 15, wherein the wearable probe includes a wristband.

17. The system of any one of claims 1 to 16, wherein the wearable probe includes a patch.

18. A method of determining a pose of a portion of a user, comprising: sensing an internal structure of a body of the user using a wearable probe; inputting a sensing data from the wearable probe to a signal processing algorithm; and determining the pose of the portion of the user by the algorithm, based on the sensing data.

19. The method of claim 18, wherein the wearable probe is an ultrasound imaging probe.

20. The method of claim 18 or claim 19, wherein the pose of the portion of the user is a hand configuration of the user.

Images