A haptic hand controller system for mixed reality

Abstract

The technology disclosed herein includes a controller or device that provides multi-dimensional hand interaction with the digital world by providing physical sensations to the palm and fingertips. The device translates motions from the hand and fingers into control of a computing device while simultaneously receiving signals that display kinesthetic sensations. The device is "controller-held" around the user's hand and held over the hand anatomy at key locations. In some embodiments, the device has one-handed engagement and disengagement. In some embodiments, the device may be used as a game controller incorporating WebVR electronics and software, wireless communication, power collection electronics, inertial measurement unit electronics including additional inputs for camera-based IMU supplementation, battery recharging electronics, and internal communication protocol support electronics. In some embodiments, the device may be used in non-gaming environments and may include additional electronics supporting universal remote controller components, IoT compatibility, and compatibility for wireless charging.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 183,602, filed May 3, 2021, entitled "HAPTIC HAND CONTROLLER SYSTEM FOR MIXED REALITY," which is specifically incorporated by reference into this specification for all that it discloses or teaches. Summary of the Invention

[0002] The technology disclosed herein includes a computer peripheral, game, and / or Internet of Things (IoT) controller that provides multi-dimensional hand interaction with the digital world while providing physical sensations to the hand and fingertips through wireless or Internet-based services. The controller translates user intent, typically measured as movements or forces from the hand and fingers, to control a graphical user interface of a computer or virtual reality (VR) headset. In contrast to a hand-held computer mouse or joystick, the controller grasps the user's hand and is held over the anatomy of the hand of choice. In some embodiments, the controller has one-handed engagement and disengagement. In some embodiments, the controller may be used as a game controller incorporating WebVR electronics and software, wireless communication, power collection electronics, inertial measurement unit electronics including additional inputs for camera-based IMU supplementation, battery recharging electronics, and internal communication protocol support electronics. In some embodiments, the controller may be used in non-gaming environments and may include additional electronics supporting universal remote controller components, IoT compatibility, and compatibility for wireless charging.

[0003] This Summary is provided to introduce in a simplified form some of the concepts that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Other features, details, utilities, and advantages of the claimed subject matter will become apparent from the following more fully written detailed description of various embodiments, as further illustrated in the accompanying drawings and defined in the appended claims.

[0004] These and various other features and advantages will become apparent from a reading of the following detailed description. [Brief description of the drawings]

[0005] [Figure 1] FIG. 2 is a diagram of an exemplary controller in a user's hand. [Figure 2A] FIG. 1 illustrates an exemplary dual hinge fit controller. [Figure 2B] FIG. 1 illustrates an exemplary dual hinge fit controller. [Figure 3A] FIG. 13 is a schematic diagram of an exemplary controller engage / disengage operation. [Figure 3B] FIG. 13 is a schematic diagram of an exemplary controller engage / disengage operation. [Figure 4] 1 is a set of schematic diagrams of a finger mechanism substrate in an example controller. [Diagram 5] FIG. 2 is a perspective view of a finger brake solenoid in an exemplary controller. [Figure 6] FIG. 2 is a schematic diagram of an exemplary finger and thumb mechanism in an exemplary controller. [Figure 7] FIG. 13 illustrates a hand clasp action of an example controller. [Figure 8] FIG. 1 is a set of schematic diagrams of fingertip and thumb components in an example controller. [Figure 9] FIG. 13 is a schematic diagram of a cross section of a fingertip component in an exemplary controller. [Figure 10] FIG. 2 is a block diagram of an electronic circuit board within an exemplary controller. [Figure 11] FIG. 2 is a block diagram illustrating signal connectivity within an exemplary controller. [Figure 12] 11 is a flow chart of an example operation for engaging, stowing, and disengaging an example controller. [Figure 13] FIG. 2 shows a block diagram of a computer system within the disclosed system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] In the following description, for the purpose of explanation, many specific details are described to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without some of these specific details. For example, although various features are attributed to specific embodiments, it should be understood that a feature described with respect to one embodiment may also be incorporated in several other embodiments. Similarly, any single feature or features of the described embodiments should not be considered essential to the present invention, and some embodiments of the present invention may omit such features.

[0007] In this disclosure, the term "user intent" refers to any measurement of hand or finger force applied by a user, the user's finger or hand posture, finger or hand position, or finger or hand movement, and may also include supplemental signals such as voice commands or other digital inputs collected via a wireless or wired computer interface on the controller.

[0008] In this disclosure, the term "hand clasp" refers to a grasping action performed by a controller on a user's hand and / or wrist.

[0009] In some embodiments, hand clasp refers to a simultaneous grasping action performed by the controller on both the ulnar and radial sides of the palm. In some embodiments, hand clasp refers to grasping on both the ulnar and radial sides at the wrist. In some embodiments, hand clasp refers to a complete envelopment of the hypothenar eminence on the palmar and dorsal surfaces. In some embodiments, hand clasp refers to a palmar fixation point at the proximal knuckle between the fourth and fifth fingers.

[0010] Examples of hand clasps are described in more detail below with respect to the "hypothenar clasp," the "proximal index finger base clasp," the "proximal fourth finger clasp," and the "radial wrist clasp."

[0011] In this disclosure, the terms "finger mechanism" or "thumb mechanism" refer to all mechanical or electromechanical components that connect the distal fingertip or thumb components to the transverse structures that extend across the palm and connect the hand clasp to the ulnar and radial sides of the hand clasp mechanism, which may include one or more portions hinged to each other or fixed as mechanical flexures.

[0012] In this disclosure, the terms "fingertip component" or "thumb component" refer to an electromechanical sensor and actuator unit that encapsulates a user's finger or thumb pad, which may be connected to the distal end of the finger mechanism as a mechanical or electrical interface, and which is a means for displaying tactile sensations on the finger or thumb pad as well as measuring user intent at their location.

[0013] In this disclosure, the term "lateral structure" refers to the mechanical support between and connecting the ulnar and radial hand clasps on the palmar side, but also serving as the base connection point for the finger mechanisms. In some embodiments, the "lateral structure" is mechanically connected to, or is sometimes the same part that follows the lateral crease of the user's palm and extends on the palmar side of the hypothenar eminence clasp down to the radial side of the wrist proximal to the thenar eminence.

[0014] In this disclosure, the term "hint" is an abbreviation of the term "haptic interaction" and is a code and / or set or sequence of commands to and from a human interface device (HID). A hint, along with sensor signals and software algorithms, can simultaneously display finger position and movement, as well as tactile and kinesthetic haptics to the user's hand and fingers. In some embodiments, a hint can include sending or receiving a short or encrypted code that triggers a set or sequence of existing sensor signals, software algorithms, and actuator commands to display haptics on the HID. In some embodiments, a hint can include a set or sequence of commands and sensory signals that cause the user to experience haptics built into or inherent to the HID, such as passive haptic detents inherent to a mechanical button press. In some embodiments, a hint includes a sensory signal from a remote device with no associated command, allowing the receiving HID full freedom to interpret and translate the sensory signal into any command for position, movement, and / or haptic display.

[0015] In some embodiments, a hint is a sensory and / or control signal intended to convey a haptic sensation to a user of a HID, such as the disclosed device, for the purpose of controlling, discovering, exploring, manipulating, or interpreting the nature or state or mode or operation of one or more products on another, possibly remote device or object. In some embodiments, the content of a hint may originate from or be displayed on a device or object in physical contact with the sending or receiving HID.

[0016] In some embodiments, hints may include direct representations of aspects of existing products that attempt to replicate or establish sensations available to a user in direct physical contact with the device, such as, but not limited to, an on / off switch or a volume knob. Hints may include entirely new haptics associated with product interaction and control, such as sensory substitution. In one example, a hint may include a "wake up" sensation, or a reminder (an event that does not have an inherent physical tactile sensation). In some embodiments, hints may directly represent the physical position or movement of the hands using sign language, and thus represent linguistic letters, words, or phrases for communication purposes.

[0017] Haptics of Things, or HoT, is a communications network service that facilitates any digital, wireless exchange of hints between devices or objects and HIDs, or between virtual or augmented representations of devices or objects and HIDs, including hint exchange between HIDs, specifically including the disclosed devices.

[0018] In some embodiments, the HID may stream hints in real time via another device or wireless communication based on the HID. When an HID such as the disclosed device receives or transmits hints to another similar device or devices, or exchanges hints with a third party electronic product, the disclosed device becomes a node on the HoT.

[0019] In some embodiments, the content of a hint specifically associated with an object or product or event may reside remotely on a computer server also wirelessly connected as a node on the HoT and be accessed by another node on the HoT.

[0020] In some embodiments, the disclosed technology includes a computer interface device. The computer interface device performs computer input functions (e.g., a human interface device, or HID) and computer output functions in the form of physical sensations to the user's palm and fingertips. However, in contrast to sliding a mouse across a tabletop, or equivalently, sliding one's fingers across a trackpad or screen (both essentially two-dimensional interactions), the disclosed technology allows for multi-dimensional hand and finger interaction with the digital world. The ability to interact in a natural way contributes to an expansion of "hand presence" in virtual and augmented environments.

[0021] Additionally, the controller provides physical sensations to the user's hand and fingertips in a way not previously incorporated into a computer interface. The controller can simultaneously display haptic sensations to the user's hand and individual fingertips in coordination with the movement or position of the user's hand, fingers, and thumb. These additional dimensions of haptic display are provided by the controller in an easy to engage and disengage manner due to the instant hand clasp capability with one hand. Going beyond the typical user input manner of a push button or mini-joystick, the controller enables natural open-hand gestures and individual finger interaction with any computer interface.

[0022] In some embodiments, to measure and stimulate the user's hand and fingers in an effective and unconstrained manner, the controller operates similarly to the framework of a full hand glove, held over the hand anatomy in a selected position while still allowing the user to manipulate and grasp real-world objects.

[0023] FIG. 1 is a diagram of an example controller on a user's hand 100, illustrating the ulnar and radial conventions and anatomical structure of choice where the controller contacts or grasps the user's hand.

[0024] Exemplary controller components on the hand include a hypothenar eminence clasp 102, a proximal index finger base clasp 104, a lateral structure 106, a proximal fourth finger clasp 108, a radial wrist clasp 110, an index finger mechanism 112, a middle finger mechanism 114, a middle finger tip component 116, an index finger tip component 118, a thumb mechanism 120, and a thumb component 122.

[0025] The controller has a hypothenar eminence clasp 102 that grasps the palm of the user on the hypothenar eminence of the hand (both palmar and dorsal, i.e., top and bottom), and a proximal index finger clasp 104 that sits on the opposite side of the user's palm at the proximal base of the index finger. The transverse structure 106 is a thin curved piece of structure that extends along the transverse of the palmar hand crease. In some cases, the transverse structure 106 may be rigid, as may other components of the controller. The transverse structure 106 connects the hypothenar eminence clasp 102 and the proximal index finger clasp 104, and also supports the base of the finger features (e.g., index finger feature 112, middle finger feature 114, etc.). Some embodiments may include two additional protrusions (small curved posts) that help secure the hand, a first protrusion 108 extending from the transverse structural piece 106 between the pinky and ring fingers (fourth and fifth fingers) and a second protrusion 110 extending from the hypothenar clasp above the wrist crease on the radial side. The extension of the hypothenar eminence clasp to the protrusion 110 is the wrist seat. At the end of the wrist seat is the protrusion 110 that may also function as a base connection for the thumb mechanism 120. Some embodiments may have the thumb mechanism 120 support the thumb component from the dorsal or radial side of the thenar eminence. The fingertip components 116 and 118 mechanically and electrically connect to the distal ends of their respective finger mechanisms via attachment posts and are removable. The thumb component 122 correspondingly connects to the thumb mechanism 120 and is also removable.

[0026] The arrows in Figure 1 illustrate a minimum of seven exemplary adjustable dimensions of the exemplary controller. The minimum adjustable dimensions include distal thumb segment 124, proximal thumb segment 126, lateral structure length 128, wrist seat distance from lateral structure 130, index finger length 132, middle finger length 134, and wrist seat width 136, with the relative angle between the thenar eminence clasp and the lateral structure, and angle A being fixed (as well as the finger mechanism angle from the lateral structure). In other embodiments, additional adjustable dimensions may be included in the controller.

[0027] The finger and thumb mechanisms (described herein) may be hinged (with or without tension) at both the transverse structure connections, i.e., the proximal base of the mechanism, the middle knuckle. In some cases, flexion (e.g., the ability to bend or curve or the state of being bent or curved) or hinges may provide movement to portions of the exemplary controller. The finger mechanisms 112 and 114 between the fingers act as mechanical support and electrical conduits for the fingertip components. In some embodiments, measurement electronics and or digital control electronics (such as light emitters for remote control) may be present on the finger mechanisms. The thumb mechanism 120, located radially to the thenar eminence or dorsally to the thenar eminence, acts as mechanical and electrical support for the thumb component. At the distal ends of the fingertip and thumb mechanisms 112, 114 and 120 are posts that extend across the corresponding finger or thumb, nominally located at the distal crease of the finger. This post serves as a potential sensing and actuation interface (force display) for the finger or thumb (such as a trigger), as well as a potential mechanical mount and electrical joint for the fingertip and thumb components.

[0028] In some embodiments, the finger mechanism may be constructed from two or more printed circuit boards that act as mechanical structural members, and the relative rotational actuation is an electromagnetic interaction with a permanent magnet surrounded by a coil on one printed circuit board that may enhance or cancel the magnetic attraction with other magnets or coils on the other board, thus allowing for separate electric torque. Torque-controlled actuation at the joint generates a lateral force against the fingertip, either at the mounting post or at the fingertip component if attached. (Impedance) control. In some embodiments, the finger mechanism may be passively pulled against the user's finger. For example, in some embodiments, the finger mechanism may be passively pulled against the user's finger with a constant force spring steel coil. The magnets and corresponding coils in the boards also provide a means of collecting power from the movement of the user's finger. As the user bends the finger, the rotational motion at the joint generates a current that can be captured and stored in the electronic components as the permanent magnet passes from one board over the electric coil on the other board. As an additional power collection means, the board supports a hard stop mount on which resides a piezoelectric element that generates a voltage difference when the user grips tightly. The changing voltage is then used to collect the current.

[0029] 2A and 2B are diagrams of an exemplary dual hinge fit controller 200. The finger mechanism shows prototypes of both the primary and alternate mounting methods, with the hypothenar clasp 102 constructed from circuit board material to house most of the controller's electrical components. The thumb mechanism, fourth finger protrusion, radial wrist protrusion, and fingertip components shown in FIG. 1 are not shown in FIGS. 2A and 2B (although the fingertip component mounting posts are shown).

[0030] In some embodiments, the user's hand with the exemplary controller holds (grasp) the palmar side of the user's hand during use. The hypothenar eminence clasp 102 and proximal index finger base clasp 104 connect to the lateral structure 106 with one or two hinges, allowing hand clasping action when the user presses down to engage the controller resting on a surface (or in some embodiments, when the controller is hooked onto another object). In some embodiments, a one or two finger (fourth and fifth finger) release mechanism (not shown) may simultaneously disengage both clasps (in two hinge embodiments) or one hand clasp (in one hinge embodiments). In some embodiments, the ball of the user's foot, thenar eminence, and palm, distal to the lateral crease movement, remain unobstructed by the disclosed controller.

[0031] 3A and 3B are schematic diagrams of an exemplary controller engagement / disengagement action in the disclosed technology. When a user presses their hand downward into the controller (as indicated by the downward arrow), the back side of each clasp rotates around one or more hinges, palm down, and closes around the respective side of the user's hand, eventually mechanically locking into place. The resulting clasp action effectively secures the user's hand with a single, one-handed motion. 3A and 3B show two embodiments in which one or two spring-loaded hinges comprise the hand clasp.

[0032] In the dual hinge embodiment of FIG. 3A, the proximal index knuckle clasp may operate independently, possibly at a different tension or speed than the hypothenar clasp on the ulnar side of the palm. As an alternative embodiment, the mechanically simple single hinge clasp action of FIG. 3B closes both hand clasps at the same speed and tension. Disengagement may be immediate with a fourth or fifth finger release switch or button (not shown) that simultaneously releases one or more hinges, opens the clasp, and allows the hand to be removed. This feature allows the disclosed controller to be effectively "held" on the user's hand even when open, without requiring the user to hold the controller.

[0033] In some embodiments, the clasp opens when not engaged, allowing top-down access to the device without the need to first remove anything, such as a securing strap. When engaged (user pushes down and clasp closes), the device is secured to the hand, while still allowing the user to open and close the grip and move thumb and fingers freely and naturally and unconstrained.

[0034] FIG. 4 is a set of schematic diagrams of a finger mechanism board 400 in an exemplary controller. The diagram illustrates the board 400 connected by a bushing (hinge) to include a finger mechanism (constant force spring not shown). At the proximal end of the board is an adjustable sliding contact printed circuit board that provides a means to adjust the length of the finger or thumb mechanism for fit. The sliding contact allows for electrical connections for both power and signals. At the distal end of the fingertip mechanism is an adjustable position slot intended to support a post that extends across the corresponding finger. Finally, the finger or thumb mechanism mounting board is mechanically hinged and electrically connected to the transverse structure.

[0035] In some embodiments, one of the controller's modes of operation is the ability to "stow" the controller on another object, such as a user's arm, by locking the finger mechanisms into a claw shape. To enable this mode, the finger mechanism board optionally has a brake mechanism on the hinge consisting of a solenoid with an eccentric post that presses itself into a slot in the other board to lock relative movement.

[0036] FIG. 5 is a perspective view of a finger brake solenoid 500 in an exemplary controller. Specifically, a possible location of the finger brake mechanism at the proximal end of the distal finger board is shown. When the solenoid is actuated, the solenoid actuated eccentric post twists from 0 degrees to 90 degrees. The resulting friction between the solenoid actuated eccentric post on the distal board and the brake slot on the proximal finger printed circuit board (or vice versa) locks the boards from moving relative to each other, allowing the controller to hold any relative angular position without the use of power thereafter. The user may then disengage the hand clasp, effectively hooking the controller onto an object for temporary stowage.

[0037] FIG. 6 is a schematic diagram of a second embodiment finger and thumb mechanism 600 in an exemplary controller. In the embodiment of FIG. 6, the method of mechanically supporting and actuating the finger mechanism uses a permanent magnet attached to the end of a piston inserted into a casing wound with an electric coil as part of a four-bar linkage. When the user bends the finger, the magnet on the piston moves within the encased electric coil, generating an electric current that can be captured. Conversely, the transmission of the electric current through the coil interacts with the permanent magnet, forcing a movement within the piston, thus exerting a force on the user's finger via the mechanical lever action of the four-bar linkage. Actuation and power collection from the movement of the finger or thumb are electromagnetically controlled using controller electronics. In some embodiments of the fingertip mechanism, brakes on the joints can be used to provide force sensations to the finger with pulse width modulation (PWM). PWM control of the brakes in combination with the finger mechanism under spring tension allows independent control of the frequency and amplitude of the periodic signal, resulting in a wide variety of kinesthetic and vibrotactile force sensations displayed at the fingertip.

[0038] In some embodiments, permanent magnets may be attached to the ends of the solenoid casing where the internal moving magnets reach their maximum extent. These magnets interact with the moving magnets on the pistons in the coils to generate an attractive / repulsive force when the user bends his finger to provide a kinematic detent at the interaction point. One magnet may provide the attractive / repulsive force, while another magnet at the base of the movement provides a repulsive force to prevent the moving magnets from jamming. An electric coil may surround one or more of the magnets such that when a current is passed through the magnets, a counteracting magnetic field cancels the magnetic effect of the magnets under the control of the hand controller electronics. The counteracting magnetic field allows the controller to eliminate the kinematic detent induced by the permanent magnets upon command from the controller electronics.

[0039] Some embodiments of the electric coil surrounding the piston have non-linear (e.g., non-uniform) windings that create stronger or weaker electromagnetic force or power collection capability based on the non-uniformity of the coil windings. The available force is directly proportional to the number of turns in the solenoid according to the following formula: F~(n×i) where F is force, i is current, and n is the number of turns in a given cross-sectional area. For example, by wrapping the wire with an exponentially increasing number of turns toward the proximal end of the throw, a stronger electromagnetic force is generated when the finger mechanism is at its maximum range (straight line). The increased number of turns can compensate for the mechanically weaker four-bar linkage angle and allow for greater authority to display kinesthetic detents along the entire range of motion.

[0040] 7A and 7B are diagrams of the hand clasp operation of an exemplary controller 700. As shown in FIG. 7A, the controller is in an open position and rests on a surface. A user may use it in its open state as a computer peripheral device, sliding the device across the surface similar to a traditional computer mouse. The controller is backward compatible as a computer peripheral device. When a user desires more advanced interaction, as shown in FIG. 7B, the user presses down on the controller with their palm, resulting in a hand clasp around their hand in one motion, securing the controller to the user.

[0041] FIG. 8 is a set of schematic diagrams of an exemplary controller fingertip or thumb component 800, including an exemplary fingertip mechanism that connects to the controller fingertip component via a mounting post at the distal end of the finger / thumb mechanism.

[0042] The fingertip component connects to a post at the distal end of the finger mechanism and may obtain computer signals and power from the connection. The fingerpad component displays force sensations to the user's finger or thumb while providing an opposing rubber contact surface carrying a force sensor. In the absence of the finger component, the disclosed controller still transmits force and sensations to the user's finger via a post attached to the fingertip or thumb mechanism, but as a trigger-like structure for the distal knuckle joint.

[0043] FIG. 9 is a schematic diagram of a cross section of a fingertip component 900 in an exemplary controller. The fingertip component includes force sense display and force sensing mechanisms. In some embodiments, vibrotactile force feedback is achieved using piezoelectric elements and a foam-bearing finger pad surface attached to one side of a printed circuit board, with control electrical components and force sensors attached to the other side of the board. Some embodiments of force sense actuators in the fingertip component use shape memory alloy membranes or filaments to pull on the finger pad under the control of the hand controller electronics to create a temporary force sensation. In some examples, the fingertip component may be about 1.5 cm long and about 0.5 cm high, although other similar sizes and ranges are contemplated.

[0044] FIG. 10 is a block diagram of an electronic circuit board 1000 in an exemplary controller. In some embodiments, a printed circuit board (PCB) may be part of the controller structure itself or may be attached to the controller. The PCB may include electronic circuitry for measuring the controller's attitude, movement, and applied forces, as well as for actuating the finger and thumb mechanisms or (finger / thumb) tip component actuators, as well as anything attached to the clasp or lateral structure or wrist seat separately therefrom. A central processing unit (CPU) may implement some or all of the functionality included in the electronic circuit board components and controller mechanisms. In some embodiments, the controller may be used as a game controller. In some embodiments, the hand controller electronic circuit board may include standardized USB-HID or WebVR electronic circuitry and software, as well as wireless communications such as Wi-Fi or Bluetooth, power collection electronic circuitry, inertial measurement unit (IMU) electronic circuitry including additional inputs for camera-based IMU supplementation, battery recharging electronic circuitry, and internal communication protocol support electronic circuitry. To collect analog signal measurements, the hand controller electronic circuit board may also include analog-to-digital (ADC) and corresponding filter electronic components in some embodiments. Additional electronics supporting a universal remote controller component, IoT compatibility, and wireless charging may also be included.

[0045] In some embodiments, a key feature of the controller is the modularity of the fingertip components: To support development, including by third-party haptic device developers, the board-mounted electronics include standardized haptic ICs such as piezoelectric drivers and servo motor controllers to support Linear Resonant Actuators (LRAs) or Eccentric Rotary Motors (ERMs).

[0046] Some embodiments of the controller may provide interactions as typically done with a traditional computer mouse, in some cases replacing interactions that may be done with a 6-DOF mouse (common in the engineering and gaming communities), trackballs, presentation mice, and any other commercially available HID. Other embodiments create new user interactions with any digital controller (connected by Bluetooth, Wi-Fi, Internet of Things (IoT), etc.). This may include, but is not limited to, a TV remote, a projector remote, or any remote for a toy.

[0047] Some embodiments of the controller provide game controller functionality for the new worlds of mixed, virtual, and augmented reality. The controller enables open-hand gestures with explicit finger tracking and interaction to provide virtual or augmented interactions while still allowing the user to hold or manipulate real-world objects.

[0048] In some embodiments, there are both left and right handed versions. The thumb mechanism may support the thumb component on the radial side of the thenar eminence. Some embodiments may support the thumb component from the dorsal side of the thenar eminence. The controller may wirelessly communicate with a VR headset (or MR glasses, for example) in a VR setting.

[0049] In some embodiments, the disclosed invention includes a fastener for attaching a superficial cosmetic accessory for user personalization, for example, to the dorsal side of the ulnar (hypothenar) clasp.

[0050] In some embodiments, the controller may house LED features on the volar and dorsal sides for use as illumination sources, to indicate the status of the device or operation, or to serve as a personalization feature. The LEDs may also serve as an external indication that the controller is specifically an electronic device and distinct from other objects such as weapons.

[0051] In some embodiments, the controller may be used in conjunction with VR glasses to allow the user to interact with the digital world (e.g., IoT) in a convenient and unobtrusive way. For example, the user may wear MR glasses that can project an image that appears in the user's field of view, in this case a computer screen. When the user wears the disclosed invention on both hands, the user may interact with the projected image as if the user was physically in front of the screen and touching it. The user feels a sense of touch from the interaction via the fingertip haptic display on the controller. Larger hand gestures are measured and transmitted by the hand controller so that the user can control the interaction with the virtual object in an immersive and satisfying way, such as by touching, swiping, pushing, waving, pointing, sliding, or pinching. Similar to the illustrated scenario, the disclosed invention replaces the need for a traditional mouse or keyboard for computer interaction.

[0052] FIG. 11 is a block diagram illustrating the signal connectivity 1100 of an exemplary controller. The main electrical components of the controller and how the electrical components connect to each other are shown. The box on the left labeled "Generic (Third Party) Device" represents an exemplary external electronic device capable of wireless communication, such as those associated with a game. Some embodiments of the controller electronics include chipsets for wireless connectivity, including Wi-Fi, Bluetooth, RF, or optical wireless communication protocols. Additional wired connections between the game controller and the external electronic device serve as test, debug, hardwired power connections, and backup communication connections, such as USB ports.

[0053] The abbreviation "HoT" stands for "Haptics of Things," i.e., a communication network including interactions enabled by the disclosed invention, whereby a user can feel a unique sensory or haptic signature associated with a particular brand of electronic device or digital device IoT.

[0054] FIG. 11 illustrates the signal flow within the disclosed controller for sensing and control, which may support all industry interfaces and communication protocols associated with HID and WebVR input devices. All connections within the dotted box labeled "Gaming / IoT Controller" are wired with the exception of connections to the finger / thumb tip module electronics, which may also be wireless. Major electrical components include, but are not limited to, power collection, battery power and recharging, any force actuators attached to the lateral structure or hypothenar or dorsal clasp mechanism, hand clasp or lateral structure sensors, finger mechanism actuators and harvesters, and any finger mechanism sensors. Additional hand controller base electronics may include LED displays and solar panel power acquisition.

[0055] The types of sensor signals between components in the disclosed controller may include joint angle encoders or Hall effect sensors, clasp force or tension sensors, inertial measurement unit sensors (IMU), camera-based sensors for IMU supplementation, temperature, thermal conductivity, moisture, battery, capacitive, electromagnetic, power flux to and from associated wireless charging electronics, and vibration sensors from the base structure or finger mechanism. Connections to the fingertip components are detachable and optional and may be hardwired or wirelessly connected via mounting post connections. Some embodiments include up to three finger or thumb components. The electrical components in the disclosed controller together enable sensing, interaction, and control of any external electronic device with wireless external input interface capabilities, such as (but not limited to) gaming headsets or consoles, any remote controlled appliances such as televisions, remote controlled toys, or smart home appliances that communicate via a wireless hub.

[0056] 12 is a flow chart of an example operation 1200 for engaging, stowing, and disengaging an example controller utilizing the hand clasp functionality of the disclosed invention, independent of the position of the finger mechanism, so that a user may engage or disengage the hand controller from the finger mechanism in any state. Thus, the sequence of events in FIG. 12 illustrates the steps of "hooking" or "grabbing" another object when a user wishes to temporarily disengage the hand controller, but does not have a flat surface to place the controller on. Perhaps the most convenient object available is the user's other arm.

[0057] The steps of operation include first engaging the device by pressing down with the palm of the hand and grasping it. Once engaged, the user curls the controller's fingers and thumb around (for example) their other arm to grab the other arm to temporarily stow the controller. The user then locks the finger and thumb mechanisms into place by verbally, digitally, or physically pressing a button on the controller. Then, issuing an "unengage" command using a button release or other digital command opens the hand clasp and temporarily stows the controller with the finger mechanism locking function in place.

[0058] The disclosed controller is in constant contact with the skin of both the palm and fingertips of the user, allowing for temporary and sustained haptic sensations between two or more sensing areas of the user's hand, for example between the index finger pad and the palm of the hypothenar eminence. The disclosed controller may display coordinated skin stretching on both the palm and fingers to create a greater sense of shape or weight when holding a virtual object, for example. The controller may display transient vibrations that "traverse" the hand from palm to fingers and vice versa. Temporary sensations may be displayed periodically, like waves, from finger to finger and then to palm to create a sense of movement. If the fingertip and palm haptic components (displays) are capable of displaying hot and cold sensations, the controller may display, for example, the sensation of gradually dipping the hand into water. The modular functionality of the controller's fingertip components provides a means to select and customize nearly infinite combinations of haptics between the user's palm and fingers.

[0059] In some embodiments of the disclosed technology, the controller uses hand movements and measured finger postures and movements to convert sign language into digital input. Electronic circuitry detects when the user forms a recognizable character in sign language, or if any corresponding movements match a known sign language phrase, and digitally transmits the conversion. Additionally, the controller can simply transmit the hand and finger movements of one user to another with another controller, relaying the hand and finger posture and movement activity in real time. When one controller follows the movements of another controller (whether or not it conforms to a given sign language protocol), any form of haptic communication is possible, such as mimicry.

[0060] In some embodiments of the disclosed technology, the finger mechanism combines a printed circuit board and an electromagnetic solenoid approach, for example, when both embodiments are used simultaneously, one embodiment may be used only for power collection and the other only for the force display, or any combination thereof.

[0061] The disclosed device may be used as a haptic communication device, transmitting haptic sensations between two or more devices, either for the same user or different users. The content of the communication is a display sensation using hand and finger movements and signals associated therewith, including but not limited to pressure, vibrotactile, or skin stretching stimulation on the fingers, palm, or any combination thereof. In some embodiments, in implementations where the device is targeted to other parts of the body (e.g., the user's back or neck), the sensations may be transmitted to other parts of the user's body. Additional haptic content includes indirect kinesthetic haptics, such as sensations derived from the finger posture and hand movements when signals are sent or received. Some embodiments of haptic communication receive sensory signals from one device, such as force or pressure sensors in the fingers, and convert and display those signals as haptics on another device, via wireless communication, in an equivalent form (such as applying pressure when force is sensed) or as an alternative display, including but not limited to displaying vibrations at one user based on another user's force signal on another device.

[0062] Some embodiments of haptic communication with the disclosed devices include sensing or receiving haptic signals from broadcasts when many devices are connected to a single device (which may be the disclosed device itself, or may include any electronic device wirelessly connected to the IoT, for example). The content of such haptic communication may be sensations associated with interactions, temporal events, states, or any other aspect of the electronic device as desired by the manufacturer of said electronic device.

[0063] Any haptic or indirect haptic content, i.e., hints, received, transmitted, or displayed between an IoT-connected device and a disclosed device, or between two or more disclosed devices, based on user interaction or state (e.g., presence), is defined and described herein as Haptics of Things (HoT).

[0064] Specific uses of HoT with the disclosed devices may include, but are not limited to, personal haptic communication between users' devices, as computer peripheral device interactions (e.g., HID) or as any hint with IoT devices, sign language display and interpretation between two or more users, VR / AR / MR haptic content display and interaction control. Some embodiments of HoT include using the haptic display capabilities of the disclosed devices on other human anatomical structures other than hands and fingers. The disclosed devices may display direct or indirect haptics resulting from hints received or sent via HoT on any body part as far as the user can reach with their hands. This implementation includes receiving and displaying signals from all parts of the head and face, all erogenous zones on the body including genitals, or body orifices.

[0065] Some embodiments of HoT interaction via the disclosed devices include, for example, one user sending a hint over HoT signal to control another user's device while the other user is in contact with a third person's body. Thus, the disclosed devices extend HoT between all users of the disclosed devices and to any other users in contact with the device on both ends of the communication path. For example, a doctor wearing the disclosed device may remotely receive HoT information (e.g., pulses) from a patient via the disclosed device worn by a attending nurse by pressing his or her finger against the patient's neck.

[0066] Referring to FIG. 13, a block diagram of a computer system 1300 suitable for implementing one or more aspects of the HoT system and the disclosed controller is shown. The computer system 1300 can execute a computer program product embodied in a tangible computer-readable storage medium to execute a computer process. Data and program files may be input to the computer system 1300, and the computer system reads the files and executes the programs in the files using one or more processors. Some of the elements of the computer system 1300 are shown in FIG. 13, where a processor 1302 is shown having an input / output (I / O) section 1304, a central processing unit (CPU) 1306, and a memory section 1308. There may be one or more processors 1302, such that the processor 1302 of the computing system 1300 comprises a single central processing unit CPU 1306 or multiple processing units. The processor may be a single-core processor or a multi-core processor. The computing system 1300 may be a conventional computer, a distributed computer, or any other type of computer. The described techniques are optionally implemented in software loaded into memory section 1308, disk storage 1312, and / or communicated over a carrier wave signal via a wired or wireless network link 1314 (e.g., Ethernet, 3G wireless, 5G wireless, LTE (Long Term Evolution)), transforming the computing system 1300 of FIG. 13 into a dedicated machine for performing the operations described.

[0067] The I / O section 1304 may be connected to one or more user interface devices (e.g., a keyboard, a touch screen display 1318, a VR headset 1320, any device connected to the IoT, the disclosed controller, etc.) or disk storage 1312. A computer program product containing mechanisms for implementing systems and methods according to the described technology may reside in the memory section 1308 or storage 1312 of such a system 1300.

[0068] The communication interface 1324 can connect the computer system 1300 to an enterprise network via a network link 1314, through which the computer system can receive instructions and data embodied in or transmitted on a carrier wave. When used in a local area network (LAN) environment, the computing system 1300 is connected (wired or wirelessly) to the local network via the communication interface 1324, which is a type of communication device. When used in a wide area network (WAN) environment, the computing system 1300 typically includes a modem, a network adapter, or any other type of communication device for establishing communications over the wide area network. In a networked environment, the program modules depicted with respect to the computing system 1300, or portions thereof, may be stored in a remote memory storage device. It is understood that the illustrated network connections are examples of communication devices for establishing a communication link between the computers, and other means may be used.

[0069] In an exemplary embodiment, the user interface software modules, communication interfaces, input / output interface modules, and other modules may be embodied by instructions stored in the memory unit 1308 and / or storage device 1312 and executed by the processor 1302. Additionally, local computing systems, remote data sources and / or services, and other associated logic represent firmware, hardware, and / or software that may be configured to assist in obtaining and processing hints. The HoT network service may be implemented using a general-purpose computer and dedicated software (such as a server running service software to a user), a dedicated computing system and dedicated software (such as a mobile device or network appliance running service software), or other computing configurations. In addition, hint parameters may be stored in the memory unit 1308 and / or storage device 1312 and executed by the processor 1302.

[0070] Computer system 1300 may include a variety of tangible processor-readable storage media and intangible processor-readable communication signals. Tangible processor-readable storage media can be embodied by any available medium that can be accessed by computer system 1300 and includes both volatile and nonvolatile storage media, removable and non-removable storage media. Tangible processor-readable storage media excludes intangible communication signals and includes volatile and nonvolatile, removable and non-removable storage media implemented in any method or technology for storing information such as processor-readable instructions, data structures, program modules or other data. Tangible processor-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other tangible medium that can be used to store the desired information and that can be accessed by computer system 1300.

[0071] In contrast to tangible processor-readable storage media, intangible processor-readable communication signals may embodied computer-readable instructions, data structures, program modules, or other data present in a modulated data signal, such as a carrier wave or other signal-carrying mechanism. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, intangible communication signals include signals that travel over wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media.

[0072] The embodiments of the invention described herein are implemented as logical steps in one or more computer systems. The logical operations of the invention are implemented (1) as a series of processor-implemented steps executed in one or more computer systems, and (2) as interconnected machine or circuit modules within one or more computer systems. The implementation is a matter of choice determined by the performance requirements of the computer system implementing the invention. Thus, the logical operations making up the embodiments of the invention described herein are variously referred to as operations, steps, objects, or modules. Moreover, it should be understood that logical operations may be performed in any order, with additions and omissions as necessary, unless expressly claimed otherwise or a particular order is inherently required by the language of the claims.

[0073] The above specification, examples, and data provide a complete description of the structure and use of exemplary embodiments of the disclosed technology. Since many embodiments of the disclosed technology can be made without departing from the spirit and scope of the disclosed technology, the disclosed technology resides in the claims appended below. Moreover, structural features of different embodiments may be combined in yet other embodiments without departing from the scope of the described claims.

[0074] The following provides a summary of non-limiting aspects of the disclosure.

[0075] Aspect 1: A controller system includes a computer controller that grips a user's hand with a controller gripping force, the computer controller including a surface that contacts the hand and each finger to display a force sense, a wireless computer interface for communicating with the outside world, and a sensor that tracks all hand and finger movements and user intent (force, pressure, etc.). In some cases, the controller is configured to provide a gripping force for removably attaching to the user's hand. In some cases, the gripping force may be mechanically actuated by a force provided by the user's hand. In some cases, the gripping force may be augmented by an electronically controlled clasp mechanism (e.g., one or more solenoids actuated by pressure against a portion of the electronic controller). In some cases, the gripping force is mechanically actuated by a force provided by the user's hand. In some cases, the user intent may include a pressure exerted on a surface of the electronic controller applied by the user's hand. In some cases, the user intent may include a position or orientation of the user's hand (e.g., an okay sign, thumbs up, thumbs down, etc.).

[0076] Embodiment 2. In the controller system of embodiment 1, the computer controller measures the position and orientation of the hand, and the position and movement of the fingers.

[0077] Embodiment 3. The controller system according to embodiment 1 further comprises one or two hinged clasps having user adjustable spring tension for actuating the controller retention clasps.

[0078] Example 4. The controller system of example 1 further comprises an actuator for displaying force feedback on the palm and / or top of the hand of the user.

[0079] Aspect 5. The controller system of aspect 1, wherein the computer controller is attached to the user's hand in approximately 3-5 selectable positions.

[0080] Example 6. The controller system of example 1, wherein the computer controller measures wrist angle and movement.

[0081] Embodiment 7. The controller system according to embodiment 1, wherein the computer controller is configured for use on a tabletop, similar to a conventional PC controller (mouse).

[0082] Example 8. The controller system of example 1, wherein the computer controller is configured to grasp an external object or detach the controller while hooked to the external object.

[0083] Example 9. The controller system described in example 1 further includes a mechanical brake for locking the knuckles of the controller in position.

[0084] Embodiment 10. The controller system of embodiment 1, wherein the computer controller employs 2-5 finger mechanisms extending to each finger and thumb, each having 1-3 1D or 2D joints and housing a mechanical fingertip component port at a distal end.

[0085] Example 11. In the controller system of Example 10, the computer controller may include a modular finger pad haptic display component.

[0086] Example 12. In the controller system described in Example 11, a modular finger pad haptic display component attaches and detaches from a mechanical port on the distal knuckle.

[0087] Example 13. In the controller system of example 12, a modular finger pad haptic display component includes electronic circuitry and actuators that convey haptic sensations such as vibration, surface orientation, temperature, shape, or other haptic features to the finger pad.

[0088] Example 14. The controller system of example 12, wherein the modular finger pad force sense display component has a concave surface for a user's finger pad and nominally includes a three-axis force sensor and a rubber pad on the distal side.

[0089] Example 15. In the controller system described in Example 12, a modular finger pad haptic display component interacts with other finger pad components (via sensors and actuators) to generate haptics such as finger snapping.

[0090] Example 16. The controller system of example 10, wherein the finger mechanism is a bent or hinged segment between the user's fingers.

[0091] Example 17. The controller system of example 10, wherein each finger mechanism has a mechanical port or interface near a distal finger joint to which a separate component can optionally be attached.

[0092] Example 18. The controller system of example 10, wherein the finger mechanism includes an electrical connection (signal and power) between the palm and the distal finger joint port.

[0093] Example 19. The controller system of example 10, wherein the finger mechanism includes electronic circuitry for harvesting power from the movement or force of a user's finger.

[0094] Example 20. The controller system of example 10, wherein the finger mechanism can apply active or passive forces to the distal finger joints via one or more of the finger mechanism components, e.g., fingertip component mounting posts.

[0095] Aspect 21. Devices that enabled standard HID interaction in the "real world" such as virtual reality, augmented reality, and fancy computer mice will enable digital and haptic interaction with all products connected via the IoT.

[0096] Example 22. In the device described in example 21, enabling broadcasting of haptics from one to many users or vice versa in a virtual or augmented setting.

[0097] Embodiment 23. The device according to embodiment 22, specifically eliminating the use of a dorsal strap to allow for one-handed engagement and disengagement.

[0098] Aspect 24. The method includes grasping a user's hand with a computer controller at a strategic location in the user's palm using a holding force, performing a closed hand movement or an open hand movement, contacting a surface of the computer controller with each finger to impart a temporary or sustained force sensation, and providing tracking of hand and finger movement using a wireless computer interface.

[0099] Example 25 The method of example 24, wherein the retention force is one of active and passive.

[0100] Aspect 26. The method of aspect 24 further includes measuring the orientation and position of the user's hand and measuring the position and force of the fingers.

[0101] Example 27. The method of example 24 further includes engaging or disengaging a grip of the controller with a single user interaction.

[0102] Example 28. The method of example 27, wherein the single user interaction is a downward palm press or a button.

[0103] Example 29. The method of example 24 further includes using one or two hinges with user-adjustable spring tension to actuate the retention force.

[0104] Example 30. The method of example 24 further includes holding the user's hands on the controller at approximately 3-5 selection positions.

[0105] Example 31. In the method described in example 24, a specific geometric shape of contact to the palm and top of the user's hand is used to maximize natural hand movements.

[0106] Example 32. The method of example 1 further includes a step of displaying force sensations to the palm and / or top of the user's hand using an actuator.

[0107] Aspect 33. A system includes a communication network and a hint transmitted by the communication network to at least one human interface device, the hint indicating a haptic sensation.

[0108] Example 34. The system of example 33, wherein the hint indicates at least one of finger position, finger movement, sensory information, tactile tactile sensation, and kinesthetic tactile sensation.

[0109] Aspect 35. One or more tangible computer-readable storage media encoding computer-executable instructions for executing a computer process on a computer system, the computer process including transmitting sensor signals, providing tactile sensations, and displaying hand and finger position and / or movement on a human interface device.

[0110] It should be noted that the techniques and aspects described herein describe possible implementations, and that the operations and steps may be rearranged or otherwise modified, and other implementations are possible. Additionally, aspects from two or more of the techniques or methods may be combined.

Claims

1. An assembly configured for one-handed engagement with a user's hand; an electronic module operably coupled to the assembly; An electronic controller comprising: The assembly comprises: A cross member; a hand clasp and a base clasp coupled to the cross member at at least one hinge and configured to engage and disengage the assembly with the user's hand; a first finger portion coupled to the lateral member and configured to express haptic sensations, including both tactile and kinesthetic sensations; Equipped with The electronic module comprises: A processor; A memory coupled to the processor for storing instructions; Equipped with The instructions direct the electronic controller to: providing a signal to the first finger portion representative of the force sensation; and Electronic controller.

2. Further comprising a thumb portion configured to represent the force sense, The instructions direct the electronic controller to: providing a signal to the thumb portion to present the force sensation; and further operable by the processor to: The electronic controller of claim 1 .

3. A first horizontal portion; A second lateral portion; and a first elongate member coupled to the first transverse portion; a second elongate member coupled to the second transverse portion; Further equipped with the thumb portion is coupled to the first elongated member; the first finger portion is coupled to the second elongate member; The electronic controller of claim 2 .

4. The electronic controller of claim 3, wherein the first lateral portion is configured to extend across at least a portion of the user's hand proximal to the palm of the user's hand.

5. The electronic controller of claim 3, wherein the second lateral portion is configured to extend across at least a portion of the user's hand and configured to avoid contact with or interference with the ball of the user's hand, the thenar eminence portion of the user's hand, and the pinkie eminence portion of the user's hand.

6. The instructions are sent to the electronic controller: tracking user intent of the user with a sensor on the electronic controller; Measure tracked user intent, transforming the user intent in response to measuring the tracked user intent. and further operable by the processor to: The electronic controller of claim 1 .

7. The instructions are sent to the electronic controller: Convert user intent into digital input, performing computer input functions and computer output functions in response to converting the user intent into digital input; and further operable by the processor to: The electronic controller of claim 1 .

8. The instructions are sent to the electronic controller: communicating hints to and from at least one human interface device over a communications network; The method is further executable by the processor to: the hints represent at least one of finger position, finger movement, sensor information, tactile haptics, and kinesthetic haptics; The electronic controller of claim 1 .

9. The method of claim 8, further comprising: providing a dorsal portion configured to engage at least a portion of the user's hand proximal to a side of the user's hand opposite the thumb; The instructions direct the electronic controller to: providing a signal to the dorsal portion to present the force sensation to the user's hand; and further operable by the processor to: The electronic controller of claim 1 .

10. The electronic controller of claim 1, wherein the electronic controller is configured to simultaneously present the force sensations to the user's hand and individual fingertips in accordance with the movement or position of the user's fingers.

11. A fastening assembly configured to be engaged and disengaged with one hand of a user; an electronic module operably coupled to the fastening assembly; An electronic controller comprising: The fastening assembly includes: A cross member; a hand clasp and a base clasp coupled to the cross member at at least one hinge and configured to engage and disengage the fastening assembly with a hand of the user; a finger portion coupled to the cross member, the finger portion including a finger mechanism configured to generate a user intent; Equipped with The electronic module comprises: A processor; a memory coupled to the processor for storing instructions; Equipped with The instructions direct the electronic controller to: Converting the user intent into a digital input; performing computer input functions and computer output functions in response to converting the user intent; and Electronic controller.

12. The electronic controller of claim 11, wherein the electronic controller grasps the palm side of a user's hand during use.

13. An electronic controller as described in claim 11, further comprising a clasp mechanism configurable to mechanically engage with the user's hand in a first fixably movable orientation to hold the electronic controller in the user's hand, the clasp mechanism being further configurable to mechanically disengage from the user's hand in a second fixably movable orientation to remove the electronic controller from the user's hand.

14. The instructions to the electronic controller: tracking user intent of said user with a sensor; Measure tracked user intent, Transforming the user intent in response to measuring the user intent to control a graphical user interface. and 14. The electronic controller of claim 13.

15. The instructions to the electronic controller: communicating hints to and from at least one human interface device over a communications network; The method is further executable by the processor to: the hints represent at least one of finger position, finger movement, sensor information, tactile haptics, and kinesthetic haptics; 15. The electronic controller of claim 14.

16. A method of operating the electronic controller of claim 11, comprising the steps of: providing signals to a first finger portion of the electronic controller to present force sensations, including both tactile and kinesthetic sensations, to the dorsal and palmar sides of a user's hand; A method comprising:

17. Tracking user intent of the user with a sensor on the electronic controller; and Measuring tracked user intent; and transforming the user intent in response to measuring the tracked user intent; 20. The method of claim 16, further comprising:

18. The method of claim 17, further comprising: converting the user intent into a digital input; performing computer input and output functions in response to converting the user intent into digital input; 20. The method of claim 17, further comprising:

19. Communicating hints between at least one human interface device over a communications network; Further comprising: the hints represent at least one of finger position, finger movement, sensor information, tactile haptics, and kinesthetic haptics; 20. The method of claim 18.

Images