Computer system with finger device

Abstract

A system may include a finger device. A touch sensor may be mounted in a housing of the finger device to collect input from an external object as the object moves along an external surface of the housing. The touch sensor may include capacitive sensor electrodes. Sensors such as force sensors, ultrasonic sensors, inertial measurement units, optical sensors, and other components may be used to collect finger input from a user. The finger input from the user may be used to process virtual objects in a mixed reality environment or a virtual reality environment, and a tactile output device in the finger device provides associated tactile output. The user may interact with real world objects, and computer-generated content may be superimposed on some or all of the objects. Force measurements or other sensor measurements made with the finger device may be used to convert object rotations and other motions into input for a mixed reality or virtual reality system.

Description

[0001] This patent application claims priority to U.S. Patent Application No. 16 / 395,806, filed on April 26, 2019, and U.S. Provisional Patent Application No. 62 / 699,642, filed on July 17, 2018, which are hereby incorporated by reference in their entirety. Technical Field

[0002] The present invention relates generally to electronic devices and, more particularly, to sensors for finger-mounted electronic devices. Background Art

[0003] Electronic devices such as computers can be controlled using computer mice and other input accessories. In virtual reality systems, force feedback gloves can be used to control virtual objects. Mobile phones can have touch screen displays and vibrators for creating tactile feedback in response to touch input.

[0004] Devices such as these may be inconvenient, cumbersome or uncomfortable for the user, or may provide inadequate feedback. Summary of the invention

[0005] A system may include a finger device. A head mounted device or other device with a display may display virtual content superimposed on real world content.

[0006] The touch sensor may be mounted in the finger device housing to collect input from an external object when the object moves along the external surface of the housing. For example, the external object may be the tip of a finger other than the finger contained in the finger device.

[0007] The touch sensor on the finger device may include a one-dimensional array of sensor elements or a two-dimensional array of sensor elements. The sensor elements may be capacitive sensor electrodes or touch sensor elements based on optical sensing, ultrasonic sensing, or other types of sensing. In some arrangements, the optical sensor, ultrasonic sensor, or other sensor mounted in the housing of the finger device may be configured to collect touch input from an area of ​​the user's finger that is adjacent to the finger device and not overlapped by the finger device.

[0008] When a user is viewing virtual content presented using a head-mounted device or other electronic device, sensors such as force sensors, ultrasonic sensors, inertial measurement units, optical sensors, and other components in a finger device can be used to collect finger input from the user.

[0009] Finger input from the user may be used to manipulate virtual objects. For example, the user moves a virtual object corresponding to a movable control based on finger input collected using a finger device or a displayed menu option that can be selected. In some arrangements, the user may interact with real-world objects while computer-generated content is superimposed on some or all of the objects. Rotation of real-world objects and other interactions with real-world objects and virtual content superimposed on real-world objects may be used as input for controlling the operation of a mixed reality system or other electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of an illustrative system with a finger device according to an embodiment.

[0011] Figure 2 is a top view of an illustrative finger of a user having placed a finger device according to an embodiment.

[0012] Figure 3 is a cross-sectional side view of an illustrative finger device on a user's finger according to an embodiment.

[0013] Figure 4 is a side view of an illustrative finger device collecting input from an adjacent area on a finger not covered by the finger device according to an embodiment.

[0014] Figure 5 is a side view of an illustrative finger device showing how virtual content may be superimposed on a surface of the device and an adjacent finger surface, according to an embodiment.

[0015] Figure 6 is a side view of an illustrative electronic device showing how a touch sensor may be incorporated into the device according to an embodiment.

[0016] Figure 7 is a cross-sectional top view of a portion of an exemplary finger device having a touch sensor array and strain gauge circuitry and having a tactile output circuit mounted on a flexible elongated arm portion of a housing for the finger device in accordance with an embodiment.

[0017] Figure 8 is a front view of an illustrative finger device with proximity sensing circuitry according to an embodiment.

[0018] Fig. 9 is a perspective view of an illustrative finger device having a sensor, such as a depth sensor based on multiple cameras with corresponding digital image sensors, according to an embodiment.

[0019] Fig.10 is a perspective view of an illustrative finger device having sensors oriented along three orthogonal axes to monitor finger activity according to an embodiment.

[0020] Fig.11 is a top view on an illustrative finger device having a sensor mounted on a housing portion that protrudes beyond the tip of a user's finger in accordance with an embodiment.

[0021] Fig.12 is a diagram showing how a user may use multiple finger devices and may use the devices to interact with body parts or other objects, according to an embodiment.

[0022] Fig.13 is a side view of an illustrative system having a computer housing (with a display or other device with a display mounted therein) and a computer with a gaze tracker according to an embodiment.

[0023] Fig.14 is a top view of an illustrative head-mounted device according to an embodiment, the head-mounted device having a support structure configured to support a display and sensors such as a front-facing camera and a gaze tracker.

[0024] Fig.15 Graphs illustrating exemplary tactile output signals that may be generated when interacting with virtual objects and / or real-world objects to provide a user with different textures, according to an embodiment.

[0025] Fig.16 is a side view of an illustrative object (such as an electronic device or a non-electronic inanimate object) showing how a user with one or more finger devices may interact with the object in accordance with an embodiment.

[0026] Fig.17 is a perspective view of an illustrative inanimate object, such as a pencil, that a user may interact with while wearing one or more finger devices in accordance with an embodiment.

[0027] Fig.18 2 is a perspective view of an illustrative electronic device and associated computer-generated visual content (virtual content) with which a user may interact using one or more finger devices in accordance with an embodiment.

[0028] Fig.19 is a perspective view of an illustrative pair of finger devices for use with each other when worn on two fingers of the users according to an embodiment.

[0029] Fig. 20 is a perspective view of a three-dimensional workspace showing how a user with a finger device may create three-dimensional graphics or otherwise interact with three-dimensional computer-generated images according to an embodiment.

[0030] Fig.21is a perspective view of another illustrative three-dimensional workspace according to an embodiment, which shows how a user can interact with a virtual object using a finger device.

[0031] Fig. 22 is a schematic diagram of an exemplary system according to one embodiment. DETAILED DESCRIPTION

[0032] Electronic devices configured to be mounted on a user's body may be used to collect user input and provide output to the user. For example, electronic devices configured to be worn on one or more fingers of a user (which are sometimes referred to as finger devices or finger-mounted devices) may be used to collect user input and provide output. As an example, a finger device may include an inertial measurement unit having an accelerometer for collecting information about finger motions (such as finger taps or free-space finger gestures); and may include force sensors for collecting information about normal and shear forces in the finger device and the user's fingers; and may include other sensors for collecting information about the interaction between the finger device (and the user's fingers on which the device is mounted) and the surrounding environment. The finger device may include a tactile output device to provide tactile output to the user's fingers, and may include other output components.

[0033] One or more finger devices can collect user input through the user. The user can use the finger device to operate a virtual reality or mixed reality device (e.g., a head-mounted device such as glasses, goggles, a helmet, or other device with a display). During operation, the finger device can collect user input, such as information about the interaction between the finger device and the surrounding environment (e.g., the interaction between the user's finger and the environment, which includes finger movement and other interactions associated with the virtual content displayed for the user). The user input can be used to control the visual output on the display. The finger device can be used to provide a corresponding tactile output to the user's fingers. For example, the tactile output can be used to provide a desired texture feeling to the user's fingers when the user touches a real object or when the user touches a virtual object. The tactile output can also be used to generate stoppers and other tactile effects.

[0034] The finger device may be worn on any or all of the user's fingers (e.g., the index finger, middle finger, and thumb, three user fingers on one hand of the user, some or all fingers on both hands, etc.). In order to enhance the sensitivity of the user's touch when the user interacts with surrounding objects, the finger device may have an inverted U shape, or have other configurations that allow the finger device to be worn on the top and sides of the user's fingertips while exposing the user's finger pads. This allows the user to touch objects with the finger pad portions of the user's fingers during use. If desired, the finger device may be worn on the user's fingers, between the knuckles, and / or on other portions of the user's fingers. The use of the finger device on the user's fingertips is sometimes described herein as an example.

[0035] The user may use the finger devices to interact with any suitable electronic equipment. For example, the user may use one or more finger devices to interact with a virtual reality or mixed reality system (e.g., a head mounted device with a display screen), to provide input to a desktop computer, tablet computer, cellular phone, watch, earbuds, or other accessories, or to interact with other electronic equipment.

[0036] Figure 1 is a schematic diagram of an exemplary system of the type that may include one or more finger devices. Figure 1 As shown, system 8 may include electronic devices such as finger device 10 and other electronic devices 24. Each finger device 10 can be worn on a finger on a user's hand. Additional electronic devices in system 8 (such as device 24) may include devices such as laptop computers, computer monitors containing embedded computers, tablet computers, desktop computers (e.g., displays on stands with integrated computer processors and other computer circuits), cellular phones, media players, or other handheld or portable electronic devices, small devices such as watch devices, hanging devices, head-mounted or earphone devices, head-mounted devices such as glasses, goggles, helmets, or other equipment worn on the user's head, or other wearable or miniature devices, televisions, computer displays that do not contain embedded computers, gaming devices, remote controls, navigation devices, embedded A wearable system (such as a system in which the device is mounted on a kiosk, car, airplane, or other vehicle), or other vehicle, a removable housing for electronic equipment, a strap, wristband, or headband, a removable device cover, a case or bag with straps or other structure to hold and carry electronic equipment and other items, a necklace or armband into which electronic equipment or other items can be inserted, a purse, sleeve, pocket, or other structure, a portion of a chair, sofa, or other seating (e.g., a cushion or other seating structure), a garment or other wearable item (e.g., a hat, belt, wristband, headband, socks, gloves, shirt, pants, etc.), or equipment that performs the functions of two or more of these devices.

[0037] With one illustrative configuration (which may sometimes be described herein as an example), device 10 is a finger-mounted device having a U-shaped body that grips a user's finger or having a finger-mounted housing of another shape configured to rest on a user's finger, and device 24 is a cellular phone, a tablet computer, a laptop computer, a watch device, a head-mounted device, a device with a speaker, or other electronic device (e.g., a device with a display, audio components, and / or other output components). The finger device having a U-shaped housing may have opposing left and right sides configured to accommodate a user's finger and a top housing portion that connects the left and right sides and overlaps the user's fingernail.

[0038] Device 10 and device 24 may include control circuit 12 and control circuit 26. Control circuit 12 and control circuit 26 may include storage and processing circuits for supporting the operation of system 8. Storage and processing circuits may include memories such as non-volatile memory (e.g., flash memory or other electrically programmable read-only memory configured to form a solid-state drive), volatile memory (e.g., static or dynamic random access memory), etc. The processing circuits in control circuits 12 and 26 may be used to collect inputs from sensors and other input devices, and may be used to control output devices. The processing circuits may be based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors and other wireless communication circuits, power management units, audio chips, application-specific integrated circuits, etc.

[0039] To support communication between device 10 and device 24 and / or to support communication between equipment in system 8 and external electronic equipment, control circuit 12 may communicate using communication circuit 14 and / or control circuit 26 may communicate using communication circuit 28. Circuit 14 and / or circuit 28 may include an antenna, a radio frequency transceiver circuit, and other wireless communication circuits and / or wired communication circuits. For example, circuit 14 and / or circuit 26 (which may sometimes be referred to as control circuits and / or control and communication circuits) may support two-way wireless communication between device 10 and device 24 via wireless link 38 (e.g., a wireless local area network link, a near field communication link, or other suitable wired or wireless communication link (e.g., link, Link, 60 GHz link or other millimeter wave link, etc.)). Device 10 and device 24 may also include power circuits for transmitting and / or receiving wired and / or wireless power, and may include batteries. In a configuration that supports wireless power transfer between device 10 and device 24, an inductive power transfer coil (as an example) may be used to support in-band wireless communication.

[0040] Device 10 and device 24 may include input-output devices, such as device 16 and device 30. Input-output device 16 and / or input-output device 30 may be used to collect user input, to collect information about the user's surroundings and / or to provide output to the user. Device 16 may include sensor 18, and device 24 may include sensor 32. Sensors 18 and / or 32 may include force sensors (e.g., strain gauges, capacitive force sensors, resistance sensors, etc.), audio sensors (such as microphones), touch sensors and / or proximity sensors (such as capacitive sensors), optical sensors (such as optical sensors that emit and detect light), ultrasonic sensors, and / or other touch sensors and / or proximity sensors, monochrome and color ambient light sensors, image sensors, sensors for detecting position, orientation and / or motion (e.g., accelerometers, magnetic sensors (such as compass sensors, gyroscopes and / or inertial measurement units that include some or all of these sensors)), muscle activity sensors (EMG) for detecting finger movements, radio frequency sensors, depth sensors (e.g., structured light sensors and / or depth sensors based on stereo imaging devices), optical sensors (such as self-mixing sensors that collect time-of-flight measurements and light detection and ranging (lidar) sensors), optical sensors (such as visual ranging sensors that use images collected by digital image sensors in cameras to collect position and / or direction information), gaze tracking sensors, visible light and / or infrared cameras with digital image sensors, humidity sensors, moisture content sensors and / or other sensors. In some arrangements, device 10 and / or device 24 may use sensor 18 and / or sensor 32 and / or other input-output devices 16 and / or input-output devices 30 to collect user input (e.g., buttons may be used to collect button press inputs, touch sensors overlaid with displays may be used to collect touch screen inputs, touch pads may be used to collect touch inputs, microphones may be used to collect audio inputs, accelerometers may be used to monitor when a finger contacts an input surface and may therefore be used to collect finger press inputs, etc.). If desired, device 10 and / or device 24 may include a spin button (e.g., a crown mechanism on a watch or finger device or other suitable spin button that rotates and optionally may be pressed to select an item of interest). Alphanumeric keys and / or other buttons may be included in device 16 and / or device 30.

[0041] Device 16 and / or device 30 may include haptic output device 20 and haptic output device 34. Haptic output device 20 and / or haptic output device 34 may generate motion sensed by a user (e.g., motion sensed by a fingertip of a user). Haptic output device 20 and / or haptic output device 34 may include the following actuators: such as electromagnetic actuators, motors, piezoelectric actuators, electroactive polymer actuators, vibrators, linear actuators, rotary actuators, actuators that bend members, actuator devices that generate and / or control repulsive and / or attractive forces between device 10 and / or device 24 (e.g., components for generating electrostatic repulsive and / or attractive forces, such as electrodes; components for generating ultrasonic outputs, such as ultrasonic transducers; components for generating magnetic interactions, such as electromagnets, permanent magnets, magnetic materials such as iron or ferrites for generating DC and / or AC magnetic fields; and / or other circuits for generating repulsive and / or attractive forces between device 10 and / or device 24). In some cases, the actuators used to generate forces in device 10 can be used to squeeze the user's fingers and / or otherwise interact directly with the user's fingertips. In other cases, these components can be used to interact with each other (e.g., by using electromagnets to generate dynamically adjustable electromagnetic repulsive and / or attractive forces between a pair of devices 10 and / or between device 10 and device 24).

[0042] If desired, input-output device 16 and / or input-output device 30 may include other devices 22 and / or device 36, such as a display (e.g., displaying a user image in device 24), status indicators (e.g., light-emitting diodes in device 10 and / or device 24 used as power indicators, and other light-based output devices), speakers and other audio output devices, electromagnets, permanent magnets, structures formed from magnetic materials (e.g., iron rods or other ferromagnetic members attracted to magnets, such as electromagnets and / or permanent magnets), batteries, etc. Device 10 and / or device 24 may also include power transmission and / or reception circuitry configured to transmit and / or receive wired and / or wireless power signals.

[0043] Figure 2 4 is a top view of a user's finger (finger 40) and an exemplary finger mounting device 10. Figure 2 As shown, device 10 may be formed of a finger-mounted unit mounted on or near the tip of finger 40 (e.g., partially or completely overlapping nail 42). If desired, device 10 may be worn at other locations on a user's finger, such as on a joint, between joints, etc. Configurations in which a device such as device 10 is worn between fingers 40 may also be used.

[0044] The user may wear one or more devices 10 simultaneously. For example, the user may wear a single device 10 on the user's ring finger or index finger. As another example, the user may wear a first device 10 on the user's thumb, a second device 10 on the user's index finger, and an optional third device 10 on the user's middle finger. Arrangements in which devices 10 are worn on other fingers and / or all fingers of one or both hands of the user may also be used.

[0045] Control circuitry 12 (and, if desired, communication circuitry 14 and / or input-output devices 16) may be contained entirely within device 10 (e.g., in a housing for a fingertip-mounted unit) and / or may include circuitry coupled to the fingertip structure (e.g., via wires from an associated wristband, glove, mitt, etc.). Configurations in which device 10 has a body mounted on a single user's fingertip are sometimes described herein as examples.

[0046] Figure 3 A cross-sectional side view of an exemplary finger device (finger-mounted device) 10 is shown showing exemplary mounting locations 46 for electronic components (e.g., control circuitry 12, communication circuitry 14, and / or input-output devices 16) within and / or on a surface of the finger device housing 44. These components may be incorporated into other portions of the housing 44, if desired.

[0047] like Figure 3 As shown, housing 44 can have a U-shape (e.g., housing 44 can be a U-shaped housing structure that faces downward and covers the upper surface of the tip of the user's finger 40 and nail 42). During operation, a user can press on a structure such as structure 50. When the bottom of the finger 40 (e.g., finger pad 40P) is against the surface 48 of structure 50, the user's finger can press and force a portion of the finger outward against the side wall portion of housing 44 (e.g., for sensing by a force sensor or other sensor mounted on the side of housing 44). Lateral movement of finger 40 in the XY plane can also be sensed using a force sensor or other sensor on the side wall of housing 44 or other portion of housing 44 (e.g., because lateral movement will tend to press portions of finger 40 against some sensors rather than other sensors and / or will generate shear forces that are measured by a force sensor configured to sense shear forces).

[0048] Ultrasonic sensors, optical sensors, inertial measurement units, strain gauges and other force sensors, radio frequency sensors, and / or other sensors may be used to collect sensor measurements indicative of the activity of finger 40. If desired, these sensors may also be used to map the contours of three-dimensional objects (e.g., through time-of-flight measurements and / or other measurements). For example, an ultrasonic sensor (such as a two-dimensional image sensor or an ultrasonic sensor with a single ultrasonic transducer element) may emit a free-space ultrasonic sound signal that is received and processed after reflecting off an external object. This allows the generation of a three-dimensional ultrasound map indicating the shape and position of the external object.

[0049] In some configurations, finger activity information (position, movement, orientation, etc.) is collected using sensors installed in external electronic equipment (e.g., in a computer or other desktop device, in a head-mounted device or other wearable device, and / or in other electronic devices 24 separate from the device 10). For example, optical sensors (such as image sensors separate from the device 10) can be used to monitor the devices 10 to determine their position, movement, and / or orientation. If desired, the device 10 may include passive optical alignment features and / or active optical alignment features to help the image sensor in the device 24 track the position, orientation, and / or motion of the device 10. For example, the device 10 may include a light emitting device (such as a light emitting diode and / or a laser). The light emitting device may be arranged in an asymmetric pattern on the housing 44 and may emit light detected by the image sensor, depth sensor, and / or other light-based tracking sensor circuit in the device 24. By processing the received pattern-shaped emitted light, the device 24 is able to determine the position, orientation, and / or motion of the device 10.

[0050] Tracking involving extrapolation from known body part orientations (e.g., finger orientations) to generate orientation information at other body parts (e.g., wrist and / or arm orientations estimated using inverse kinematics) may also be performed. If desired, visual ranging sensors may be included in device 10. These sensors may include image sensors that collect frames of image data of the environment of device 10, and may be used to measure the position, orientation, and / or motion of frames from the image data. If desired, lidar, ultrasonic sensors oriented in multiple directions, radio frequency tracking sensors, and / or other finger device tracking arrangements may be used. In some arrangements, user input for controlling system 8 may include both user finger input and other user input (e.g., user eye gaze input, user voice input, etc.). For example, when controlling device 10 and / or device 24 in system 8, gaze tracking information (such as the user's gaze point measured with a gaze tracker) may be fused with finger input. Finger input may include information about finger orientation, position, and / or motion, and may include information about how the finger is forced to press the surface (e.g., force information).

[0051] For example, sensors in device 10 may measure how firmly a user presses device 10 (and finger 40) against surface 48 (e.g., in a direction parallel to a surface normal n of surface 48, such as Figure 3 The direction of movement of device 10 in the XY plane and / or the Z direction may also be measured by force sensors and / or other sensors 18 at locations 46.

[0052] Structure 50 may be part of a housing of device 24, may be part of another device 10 (e.g., another housing 44), may be part of a user's finger 40 or other body part, may be the surface of a real-world object (such as a table), a movable real-world object (such as a bottle or pen), or may be other inanimate objects external to device 10, and / or may be any other structure that a user may contact with finger 40 while moving finger 40 in a desired direction with a desired force. Because motions such as these may be sensed by device 10, device 10 may be used to capture pointing input (e.g., input that moves a cursor or other virtual object on a display such as the display in device 36), may be used to capture click input, swipe input, pinch-to-zoom input (e.g., when a pair of devices 10 are used), or other gesture input (e.g., finger gestures, hand gestures, arm movements, etc.), and / or may be used to capture other user input.

[0053] Figure 4 is a side view of an illustrative finger device on a user's finger. Figure 4 In the exemplary configuration of , device 10 includes touch sensor 56. Touch sensor 56 may be formed from an array of capacitive touch sensor electrodes, such as electrodes 52 that overlap side surfaces and / or top surfaces of housing 44. Touch sensor 56 may be used to capture touch inputs, such as inputs from direct contact and / or close proximity with different fingers of a user or other external objects. Figure 4In the example of , touch sensor 56 may overlap touch input area 54 on the side of device 10. If desired, additional touch input may be collected in an adjacent area (such as touch input area 62 on the exposed side of finger 40 adjacent to device 10). Touch input may be collected from area 62 using sensors in device 10 that face area 62. For example, these sensors may be capacitive sensors, optical sensors, ultrasonic sensors, and / or other sensors that may monitor area 62 (see, e.g., sensor 58). With one exemplary configuration, sensor 58 is an optical sensor having a light emitting diode (e.g., a light emitting diode or a laser) that emits light 60 that overlaps area 62, and has a light detector (e.g., a photodiode) that measures reflected light from a user's finger or other external object in area 62. In this way, the area covered by the touch sensor circuitry of device 10 may extend across two portions of housing 44 and portions of an adjacent body part (such as area 62 on finger 40).

[0054] System 8 may have a head-mounted display device or other device that presents mixed reality content to the user. In a mixed reality environment, virtual content (e.g., computer-generated content from control circuitry in system 8) may be superimposed on real-world content (e.g., real-world images obtained by directly viewing the real world through an optical coupler and / or real-world images obtained using a front-facing camera or other image sensor that operates in a pass-through mode to provide the user with an image of the real world on a display). As an example, the mixed reality content of system 8 may include icons and other computer-generated visual content (sometimes referred to as virtual content) overlaid on areas where user input (e.g., touch input) is captured. As an example, consider Figure 5 In this example, a user wears finger device 10 on finger 40. Touch sensor 56 is formed in device 10 (e.g., in housing 44) and collects touch sensor input in area 54 on the outer surface of housing 44. Optical sensor 58 collects touch input from area 62 on finger 40 adjacent to device 10. The user can use a mixed reality head mounted device (e.g., see Figure 1 The mixed reality device may present computer-generated content (virtual content generated by device 24) that overlaps the real-world content in virtual content area 66.

[0055] For example, the virtual content in area 66 may include selectable icons corresponding to functions (e.g., functions performed by application software and / or operating system software) that a user may invoke by providing corresponding touch input. For example, device 24 may be a mixed reality head-mounted device that presents selectable calendar icons in area 66. When a user touches a calendar icon in area 66 using a free finger other than the finger on which device 10 is worn, touch sensor 56 or optical sensor 58 will detect the touch input, depending on the portion of area 66 in which the calendar items are presented. In response to a user touch input selection of a calendar icon, control circuitry in device 24 may take appropriate actions, such as launching a calendar application using a display in device 24 and visually displaying the contents of the launched calendar application (e.g., as an overlay on a real-world image). If desired, free-form touch input may be collected (e.g., a user may trace the shape of alphanumeric characters in area 66 used as an input command). Device 10 may provide tactile output in response to the received touch input.

[0056] As another example, a volume slider or other sliding control icon may be displayed in region 66. When the user's finger points to and overlaps the sliding control icon, the sliding control icon is selected. The user can then slide the user's finger back and forth in region 66 to adjust the slider. If desired, a tactile output (e.g., a click) may be provided in response to the selection and / or movement of the control icon. For example, a tactile output stopper (a vibration provided when the user's finger position coincides with a predetermined position 64) may be provided during user finger interaction in region 66. In some arrangements, icons (e.g., dots, glowing dots, flashing dots, finger icons, movable lines, etc.) may be used to depict the finger position in region 66. In general, any suitable content may be displayed in an area such as region 66 (e.g., an area overlapping all or part of device 10, an area overlapping all or part of finger 40, and / or other areas overlapping real-world objects). The content may include still and / or moving images containing text, graphics, photos, real-world videos, moving animations, icons, and / or other content).

[0057] Figure 6is a front view of device 10, which shows how device 10 has a touch sensor (illustrative touch sensor 68) that overlaps both sidewalls (sides) 44SW and top 44T of housing 44. Touch sensor 68 may have electrodes formed on a flexible printed circuit, on an inner surface of a housing sidewall in housing 44, on an outer surface of housing 44, and / or on other suitable portions of device 10. In one exemplary configuration, circuitry such as circuitry 72 in housing 44 (e.g., a printed circuit, an integrated circuit, a touch sensor control circuit, etc.) may communicate with touch sensor 70 on sidewall 44SW of housing 44 (e.g., using a signal path formed by a flexible printed circuit extending between circuitry 72 and sensor 70, using a flexible printed circuit substrate containing a signal path for coupling to circuitry 72 and a capacitive touch sensor electrode array, etc.).

[0058] The flexible printed circuit containing the capacitive touch sensor electrodes can be formed of a conductive material such as silver (e.g., a silver paint / ink including a silver filler in a polymer binder), silver nanowires, copper, or other metal traces on other polymer substrate materials (such as a polyethylene terephthalate (PET) substrate or a polyimide substrate). Constructions in which the capacitor sensor substrate includes a fabric (e.g., a fabric with conductive traces deposited thereon and / or a fabric with strands of conductive material forming electrodes, etc.) can also be used. The capacitive sensor can use a mutual capacitance sensing arrangement or a self-capacitance sensing arrangement. The touch sensor for device 10 can be formed on a single side wall 44SW, one or two side walls 44SW (e.g., on the left and right housing portions of finger 40), on one or both side walls 44SW and top 44T, on top 44T alone, and / or on any other suitable area of ​​device 10.

[0059] If desired, non-capacitive touch sensing, such as touch / proximity sensors based on optical sensing, ultrasonic sensing, force detection, etc., may be used in addition to and / or in lieu of capacitive touch sensing. Figure 6 Touch sensors such as exemplary touch sensor 68 and / or touch sensor 70 may be formed using a single touch sensor element (e.g., a single capacitive touch sensor electrode, a single ultrasonic sensor element, a single optical sensor element, etc.) and / or may be formed using one-dimensional (linear) and / or two-dimensional arrays of sensor elements (e.g., a sensor having rows and columns of capacitive sensing electrodes, optical sensor devices, ultrasonic sensor devices, etc.).

[0060] Figure 7 , a cross-sectional top view of device 10 in an exemplary configuration is shown, wherein device 10 includes force sensor circuitry, such as strain gauge circuitry, and a tactile output component. Figure 7As shown, electrodes 52 may form touch sensor 56. For example, touch sensor 56 may be a two-dimensional touch sensor that extends on the sides and / or top of housing 44 (e.g., on an interior surface of housing 44 and / or other suitable portion of device 10). Housing 44 may have an elongated portion, such as Figure 7 44' (also sometimes referred to as an arm) that extends along a longitudinal housing axis 81 (e.g., an axis parallel to the length of finger 40). Arm 44' can be configured to bend in direction 84 under pressure from finger 40 (e.g., lateral finger movement and / or other finger input). Strain gauge circuit 80 can be formed near the base of arm 44' (e.g., where arm 44' is attached to the rest of housing 44) so ​​that strain gauge circuit 80 can detect bending of arm 44'. Strain gauge circuit 80 can include one or more strain gauges. If desired, other force sensing components can be used to detect bending of arm 44' (e.g., piezoelectric force sensors, etc.).

[0061] like Figure 7 As shown, additional components such as component 82 can be mounted on arm 44'. As an example, component 82 can be a tactile output device that generates a tactile output for finger 40 or other suitable electronic components. In some arrangements, component 82 can be a piezoelectric device that serves as both a tactile output component and a force sensor. Figure 7 With circuitry of the type shown, device 10 can monitor finger forces applied to the sidewalls of device 10, thereby measuring shear and / or normal forces as a user manipulates device 10 and interacts with the surrounding environment. Touch input can be acquired from touch sensor 56 and / or can be inferred from force sensor measurements made using strain gauge circuitry 80.

[0062] Figure 8 4 shows how housing 44 is provided with one or more sensors 86 for sensing separation between device 10 (e.g., housing 44 and finger 40) and surface 48 of structure 50. Sensor 86 may be an optical sensor, a capacitive sensor, an ultrasonic sensor, and / or other sensor (e.g., a proximity sensor, a distance sensor, etc.). Figure 8 With the sensors 86 , device 10 may be capable of monitoring the distance between device 10 and surface 48 , and / or may collect information regarding device orientation and / or device movement.

[0063] Fig. 994. The housing 44 is shown as being provided with a depth sensor or other sensor having a plurality of sensor components 92. For example, the sensor components 92 may be a pair of digital image sensors configured to form a stereo depth sensor. With one exemplary configuration, portion 90 of the housing 44 may be coupled to a hinge aligned with hinge axis 88. When it is desired to use component 92 (e.g., to capture an image of an object in direction 96), component 90 may be rotated (pivoted) about hinge axis 88 in direction 94. Fig. 9 With a depth sensor, device 10 may collect information about the orientation, position and / or movement of device 10 relative to other structures, and / or may be able to collect information about the environment surrounding device 10 (e.g., images of real-world objects, three-dimensional maps of real-world objects, etc.).

[0064] Fig.10 is a perspective view of the device 10 in an exemplary configuration, wherein the device 10 has a set of sensors 98 oriented to collect data in orthogonal directions. For example, there may be three sensors 98, each sensor along Fig.10 The sensor 98 can be used to collect information about the movement, orientation, and position of the device 10 while the device 10 is moved while being worn on the finger 40 by the user. The sensor 98 can be, for example, a radio frequency sensor (e.g., a transmitter and a receiver) such as a radio frequency transceiver connected to a directional antenna. During operation, each sensor 98 can transmit a radio frequency signal and can detect a corresponding reflected signal from an external object. In another exemplary arrangement, a base station and / or other external electronic device (e.g., device 24, etc.) can transmit a reference (beacon) radio frequency signal measured by a receiver in the sensor 98. If desired, other types of directional sensors (e.g., optical sensors such as lidar sensors, directional ultrasonic sensors, etc.) can be included in the device 10. The use of three orthogonally oriented radio frequency sensors is exemplary.

[0065] Fig.11 is a top view of device 10 in an exemplary configuration, wherein housing 44 has a forwardly protruding portion, such as portion 100, that extends beyond the tip of finger 40 when device 10 is worn by a user. Portion 100 may serve as a support for electrical components, such as component 102. Component 102 may be, for example, a proximity sensor and / or distance sensor, such as an optical sensor, a capacitive sensor, an ultrasonic sensor, and / or may be configured to collect information about the distance between device 10 and a surrounding surface. This allows device 10 to collect information about the position, orientation, and / or movement of device 10 and / or information about nearby objects.

[0066] like Fig.12As shown, a user may have more than one device 10. Fig.12 In the example, the user wears a first finger device 10 on a first finger 40 and a second finger device 10 on a second finger 40. When the first and second fingers 40 interact with a third finger (e.g., the underside surface of the user's thumb in this example), each of these finger devices 10 can collect user input (e.g., input measured by a force sensor, an ultrasonic sensor, an optical sensor, a capacitive sensor, etc.). The input collected in this manner may include information about the interaction between the finger device 10 and surfaces associated with other fingers 40 and / or other surfaces (e.g., surfaces associated with the user's legs, the back of the user's hand, the user's palm, or other body parts; surfaces associated with inanimate external objects such as pencils, bottles, desktops, etc.; surfaces associated with electronic devices, such as display surfaces, keyboard surfaces, button surfaces in housings and accessories, etc.). The input may include touch input (e.g., input indicating contact between a finger 40 and an external surface), shear force input (e.g., information indicating that the user pushes and / or drags the finger 40 to one side while contacting the external surface with the finger pad 40P) and / or normal force input (e.g., information about how strongly the finger 40 presses on the external surface).

[0067] In some arrangements, information may be collected regarding relative motion between devices 10. For example, sensors in devices 10 may be used to collect information indicating that devices 10 are moving toward or away from each other and / or information regarding device position, orientation, etc. This allows a user to perform multi-finger gestures with multiple devices 10 (e.g., pinch-to-zoom gestures, gestures to select an item by grabbing the item with opposite fingers, etc.).

[0068] exist Fig.12In a configuration of the type shown, in which a user's finger 40 with device 10 makes contact and interacts with another finger 40 or other surface of the user's body, the user need not be near an external inanimate object, such as a desktop, in order to provide input. Instead, the user can provide touch gestures and other gestures by creating finger interactions between fingers 40 wearing device 10 and / or fingers not wearing device 10 and / or other objects. If desired, device 10 can be used to collect finger motion through the air and / or other finger activities associated with changes in finger position, orientation, motion, and / or finger force as finger input. Touch sensor input can also be collected using touch sensor circuitry in device 10. Finger input can include finger taps, finger swipes (e.g., speed-dependent and / or direction-dependent swipe gestures), finger pinch-to-zoom gestures, gestures in which the fingers are squeezed together, gestures in which the fingers press against a surface with varying forces, three-dimensional free-space gestures such as finger flicks and / or up-and-down finger motions (e.g., downward motion over a particular tap location within a predetermined time and upward motion over a particular tap location to select an item associated with the tap location), gestures such as thumb scrolling, fine-tuning gestures, inertial swipes, tap and swipe gestures, dwell-time-dependent force input, and / or other finger activities in which the finger 40 moves in the air, moves along a surface, and / or presses a surface in a predetermined pattern.

[0069] System 8 may include optical sensors, such as gaze detection sensors (sometimes referred to as gaze detectors, gaze trackers, gaze tracking systems, or eye monitoring systems). A gaze tracking system for system 8 may, for example, include an image sensor, a light source, and / or other devices for monitoring the user's eyes. The system may include one or more visible and / or infrared cameras that face the viewer's eyes and capture images of the viewer's (user's) eyes. During operation of system 8, control circuitry in system 8 (e.g., control circuitry coupled to a housing in device 24) may use a gaze tracking system to track the viewer's gaze. Cameras and / or other sensors in device 24 may, for example, determine the location of the user's eyes (e.g., the center of the user's pupil), and may determine the direction in which the user's eyes are oriented.

[0070] The orientation of the user's gaze can be used to determine the location in the virtual and / or real world environment to which the user's eyes are directed (sometimes referred to as the user's gaze point). If desired, device 24 and / or other devices in system 8 can use gaze tracking information (such as information about the user's gaze point) to determine which actions to take in system 8. For example, a gaze tracking system can determine that the user's gaze point is directed to a first object rather than a second object, and can respond by assuming that the user visually selected the first object rather than the second object. Finger input and / or other user input can be used in conjunction with input (such as gaze point information in determining which actions to take in system 8).

[0071] As an example, consider Fig.13 . In this example, device 24 has a housing in which a gaze tracker 112 is mounted for monitoring a user's eyes 104. Device 24 may include a component such as component 111. For example, component 111 may be a display configured to display an image to a user. The image may include one or more objects (e.g., visual items), such as object 110. Control circuitry in device 24 may use gaze tracker 112 to determine the direction 106 in which the user is looking at component 111 or other objects. Using direction 106 and / or other information from gaze tracker 112 and / or other sensors (e.g., depth sensors and / or other sensors that determine the distance of the user from device 24), device 24 may determine the location of the user's gaze point 108 on component 111. For example, device 24 may determine whether the user is currently looking at a virtual object (such as Fig.13 object 110 on the display).

[0072] Another exemplary system with gaze tracking is Fig.14 In Fig.14 In the example of FIG. 1 , device 24 is a head-mounted device having a head-mounted support structure 116 (sometimes referred to as a housing) configured to be worn on a user's head. A rear-facing gaze tracking system 112 may monitor the user's eyes 104 to determine the direction 106 of the user's gaze. Additional sensors (e.g., a depth sensor 114) may be used to determine the location and / or other properties of objects in the user's field of view, such as the direction of the user's gaze. Fig.14 104. The object 110 may be a real-world object (e.g., a body part of the user or another person, an inanimate object having circuitry such as one or more devices 24, a non-electronic inanimate object such as a pencil, ball, bottle, cup, table, wall, etc.) or may be a computer-generated (virtual) object presented to the user's eye 104 by a display in the device 24 (e.g., a see-through display system or a display system in which virtual content is superimposed on a real-world image on the display captured with the camera 114). Using information about the user's gaze direction 106 and information about the relative position between the user and the object 110 (e.g., information from a depth sensor in the device 24 and / or information about a virtual object presented to the user), the device 24 may determine when the user's gaze point 108 coincides with the object 110.

[0073] Fig.13 and Fig.14Arrangements of the type shown in allow a user to interact with real-world content and computer-generated (virtual) content. For example, a user may select an object of interest by directing gaze point 108 toward the object (e.g., for a predetermined dwell time and / or until receiving relevant user input such as finger input to confirm the selection). Using finger device 10 and / or other devices in system 8, the user may perform an operation on the selected object.

[0074] As a first example, consider a scenario where object 110 is a computer-generated icon. In this case, after aligning gaze point 108 to overlap with the computer-generated icon and thereby selecting the icon for further operation, the user may provide a command with finger device 10 and / or other input means in system 8 that instructs system 8 to begin the relevant operation in system 8. As an example, if the icon is an email icon, system 8 may launch an email program on device 24 upon receiving the finger input from the user.

[0075] In a second example, object 110 is a real-world object, such as a non-electronic inanimate object (e.g., a Fig.14 In response to detecting that the user's gaze point 108 is directed toward object 110 and in response to receiving a user finger input or other suitable input from the user, device 24 may generate a virtual object that overlaps all or part of object 110 in the user's field of view. Other operations may include enlarging a portion of object 110, changing the color or texture of object 110, adding an outline around object 110, adding a graphical element aligned with object 110, and / or taking other suitable actions.

[0076] In a third example, object 110 is a real-world object including a circuit. Object 110 may be, for example, a wireless speaker or other electronic device 24. In response to detecting that the user's gaze point 108 is directed to object 110 and in response to receiving a user's finger input or other suitable input from the user, device 24 may adjust the output volume of the speaker. If the object coincident with gaze point 108 is a device such as a television, the channel of the television may be changed in response to the user's finger input or other input. In this way, by simply gazing at the object and coordinating with this gaze point selection to provide additional user input, the user may interact with electronic devices around the user's home or office or other electronic devices in an environment. Additional user input may be made using finger device 10 and / or other input devices (e.g., a mouse or other pointing device, a device for processing voice commands, a keyboard, a touch screen display in a cellular phone or tablet device, etc.). Gaze point dwell time, blinking, and other eye activities may also be used as user input.

[0077] The user may touch items in the user's environment while wearing the finger device 10. When the user touches the surface of these items, measurements taken using sensors in the device 10 may be used to determine the outline of the items. This information may then be combined with optional additional sensor data (e.g., depth sensor data, camera images, etc.) to determine properties of the items (such as item size, item shape, item location, etc.). Examples of sensors that may be used in the device 10 to measure item outlines include an inertial measurement unit that may track the orientation, position, and / or movement of the device 10 in three dimensions, and force and / or touch sensors in the device 10 (which may sense when the user touches the surface of an object). Depth sensors in the device 10 and / or device 24 may also be used to collect three-dimensional surface maps (surface contour information) of objects in the user's environment.

[0078] Once system 8 has acquired information about the outlines of objects in the user's environment, virtual content can be created that partially or completely overlaps the surfaces of the objects and / or tactile outputs associated with the surfaces of the objects can be generated. As an example, knowing the locations of the cube surfaces, system 8 can overlay virtual content on one or more cube surfaces and can use a tactile output device in finger device 10 to apply tactile outputs to the user's fingers 40 whenever system 8 determines that the user's fingers are touching those surfaces. If desired, the tactile output device can be used to provide different virtual textures for different surfaces of a cube or other object. As an example, whenever a user's finger touches a first side of the cube, the control circuit in device 10 can provide a first drive signal D having a relatively high amplitude and frequency (see, e.g., Fig.15 , where the drive signal D is plotted as a function of time t). Whenever it is determined that the user's finger is touching the second side of the cube, a different texture may be provided to the user's finger using the tactile output component in the finger device 10. As an example, if the user is Fig.15 During the time period 120 of the second side of the cube is touched, the lower frequency and lower amplitude drive signal D can be used to control the tactile output component in the finger device 10. As a result, when the first side of the cube is touched, the user will feel the first texture, and when the second side of the cube is touched, the user will feel the second texture. If desired, tactile effects can be provided to the cube and other objects along the edges of the objects, such as the tactile output can produce shapes and / or rounded edges, can produce effects associated with compliant structures, and / or can produce stops, force feedback simulating resistance to movement, clicking simulates depression and / or release of buttons with a physical click feel, and / or other tactile effects. Corresponding visual effects can be provided.

[0079] Fig.16A simplified diagram is provided to illustrate how a user may press fingers 40 onto an exposed surface of an object 110 with one or more finger devices 10 to measure in three dimensions the contours of the object 110. This allows system 8 to determine the shape of all sides of the object 110, including the far ends of the object 110 that may not be visible to a camera or other sensor in device 24 (e.g., where device 24 is a head-mounted device in which a front-facing camera captures images of the real world).

[0080] The object 110 may be an inanimate object without circuitry (e.g., an object other than an electronic device such as a pencil, etc.) or may be an electronic device. Fig.16 In the example of , object 110 is an electronic device having an input component (such as component 110' that can collect user input). For example, object 110 can be a touch screen display, and component 110' can be a touch sensor in the display. In another exemplary configuration, object 110 can be a computer mouse that collects pointing input by the user, and component 110' can be a depressible button (switch) on the computer mouse. In other cases, object 110 can be a cellular phone, a tablet computer, a desktop computer, or a keyboard, and component 110' can be a touch screen, a touch-sensitive button, a keyboard key, or other input device on object 110.

[0081] During operation, system 8 may optionally use device 24 (e.g., a head mounted display device) to display an image overlaid on top of some or all of object 110. If desired, real-world objects (e.g., a stone or common household item) may be transformed into control knobs (e.g., a rotatable knob with tactile stops and / or other tactile features, such as a tactile output button area, and having an appearance controlled by the color, texture, and shape of the computer-generated image superimposed on the stone) using the superimposed virtual content. In this way, powered-off electronic devices, objects without circuits, and / or other various objects in the user's environment may be used as input devices for controlling the operation of system 8. As an example, a virtual controller may be constructed from a pencil, an eraser, a basket, a cup, a plate, furniture, a shoe or other clothing, a box or other housing, a paper notebook, and / or other items that can be touched, rotated, moved, and / or otherwise manipulated by a user.

[0082] In some arrangements, when a user wears the finger device 10 during interaction with an object overlaid with virtual content, finger input from the device 10 may be collected from the user and / or tactile output may be provided by the device 10. If desired, a gaze tracking system, a depth sensor, a camera, and / or other sensors may be used to collect user input. If desired, virtual content may be superimposed on some portions of an object while leaving other portions uncovered. For example, a page in a notebook may contain a handwritten note. Virtual content (such as a computer-generated text annotation) may be superimposed on a portion of a page adjacent to the note. Pages in a notebook may have barcodes (e.g., QR codes) and / or other identifying information to help identify which pages are associated with corresponding annotation information (as an example).

[0083] exist Fig.17 In the exemplary configuration of , the object 110 is a pencil and does not contain any circuitry. A user wearing one or more finger devices 10 can rotate the object 110 around the longitudinal axis 112 of the object 110. The user can also move the object 110 (e.g., the tip and / or end of the object 110) laterally, such as Fig.17 During the movement of the object 110 , the finger device 10 may collect information about the interaction between the finger device 10 and the object 110 , and thus may be used to determine the position, orientation, and movement of the object 110 .

[0084] Different interactions with object 110 may be used to control different corresponding functions in system 8. For example, rotation of object 110 about axis 122 may adjust volume or other simulated properties in system 8, may rotate virtual objects, and / or may produce other changes in the operation of system 8. Lateral movement of object 110, changes in the angular orientation of object 110 (e.g., tilting of object 110), and / or other manipulation of object 110 may be used to produce other adjustments to the operation of system 8 (e.g., virtual object movement, adjustment of virtual object properties, adjustment of hardware and / or software properties in system 8, etc.). Object 110 may be used as a staff, sword, joystick, and / or item in a virtual world.

[0085] In some arrangements, some or all of the objects 110 may be superimposed with virtual content. As an example, icons, movable controls, and / or other interactive visual elements may be displayed on all objects 110 or in a selected area of ​​an object such as the illustrative area 126. As an example, the area 126 may include an array of virtual buttons that can be selected when the user presses the finger 40 on the desired virtual button while wearing the finger device 10. The user may also provide input to the area 126 or other parts of the object 110, such as force input (e.g., when the user squeezes the object 110 in the user's finger, the pressure on the side of the object 110 in the area 126 is different), sliding input (e.g., sliding gestures), etc. In some arrangements, the position of the tip of the object 110 may be tracked by analyzing the position of the finger 40 and / or an image of the object 110 captured by an image sensor in the tracking device 24. To help calibrate visual measurements of the shape and position of the surface of object 110, the user may touch the entire surface of object 110 while taking corresponding measurements with finger device 10 to determine the surface contour of object 110, and the user may cause object 110 to interact with other system components (e.g., by resting the top of object 110 against a force sensor in device 24 while object 110 is imaged with a camera, etc.).

[0086] If desired, the user may interact with the virtual object that has been scanned using the depth sensor, finger device, or other sensing arrangement (e.g., to change the object's color, size, shape, appearance, etc.). In some arrangements, the device 24 may use pattern recognition to identify the object 110, and may take action accordingly (e.g., by creating a wand skin for the identified pencil).

[0087] exist Fig.18 In the configuration, system 8 includes multiple electronic devices. Fig.18 Device 24A in the example of is an electronic device, such as an Internet-connected voice-controlled speaker or other device that does not include a display capable of displaying extensive text. Device 24B is a head-mounted device having a head-mounted support structure (housing) worn by a user so that the user's eyes 104 can view content generated by device 24B and the real-world environment surrounding device 24A.

[0088] When interacting with device 24A, the user can view supplementary computer-generated content, such as text-based content, on device 24B. For example, the user may wear one or more finger devices 10. The user may wish to view a list of available music tracks. In response to finger commands or other user inputs from finger device 10, device 24B may determine (directly or with device 24A via wireless communication) which tracks are available. The user may provide input commands with finger device 10 or other input devices to navigate between virtual menu options and / or other options, thereby selecting a desired music type to view. The user may then display a computer-generated music track list 130 via device 24B.

[0089] As an example, list 130 may be displayed adjacent to device 24A and / or overlaid with some or all of device 24A. List 130 may include selectable music tracks. A user may select a desired track by gesturing toward the track with finger device 10 (e.g., by placing the user's finger tip on the desired track for more than a predetermined amount of time, etc.), such as in Fig.18 , as illustrated by the selected trace 132 in . In response to the selection of the trace 132, the finger device 10 may provide a tactile output (e.g., clicking to confirm the selected trace 132), the device 24B may take an appropriate action (e.g., by highlighting the trace, by displaying information on the selected trace, etc.), and the device 24A may take an appropriate action (e.g., by playing the selected trace in response to receiving the user's finger input of the selection of the trace 132). Wireless communication and / or wired communication between the devices 24A, 24B and the finger device 10 may be used to coordinate the operation of the system. In this type of environment, if desired, the device 24A may be used as a host device and the device 24B may be used as an input-output accessor connected to the device 24A (e.g., in addition to the input-output finger device 10 being worn on the user's finger) and / or other networked device arrangements may be used.

[0090] If desired, in addition to and / or in lieu of using finger input, selections of music tracks and other items in a computer-generated list or other virtual content may be made based on eye tracking information. For example, list 130 may be displayed in response to eye tracking information. For example, if the user's gaze point is directed toward device 24A for more than a predetermined amount of time, and / or is directed toward device 10 when the user performs a particular finger gesture using device 24A, an interactive list (such as list 130) may be automatically displayed to allow the user to make a music track selection or other suitable selection.

[0091] Any device 24 can be controlled in this manner. For example, Fig.18Device 24A may be a television, tablet computer, laptop computer, cellular phone, and / or other adjustable devices in a home or office (e.g., lights, thermostats, fans, speakers, lampshades, etc.). Fig.18 The gaze tracker in system 8 may be installed on device 24A and / or device 24B. If desired, device 24A may include additional components (e.g., a display, etc.).

[0092] like Fig.19 As shown, devices 10 can be used to interact with each other (e.g., to collect tap, swipe, pinch-to-zoom gesture input, etc.). If desired, device 10 can measure the amount of finger pinch force generated when a user presses a pair of fingers 40 together. In addition to other user finger inputs from device 10, the user can dynamically change the amount of finger pinch force to control the operation of system 8. As an example, a user can adjust the line width of a line drawn in a virtual three-dimensional space through a drawing application running on system 8 (e.g., on device 24, such as a head-mounted display or other device 24 with a display).

[0093] An exemplary three-dimensional line graph is shown in Fig. 20 As shown in Fig. 20 As shown, line 136 may have a line width W that varies as a function of distance along line 136. The shape of line 136 may be determined by a gesture (e.g., a three-dimensional gesture made by a user's hand through the air). These movements (which may sometimes be referred to as three-dimensional finger gestures) may be captured using an inertial measurement unit or other sensor in device 10. Fig. 20 While drawing line 136 on the three-dimensional workspace of the user, the user can pinch fingers 40 together with variable force. The amount of clamping force provided by the user can be measured by finger device 10 (e.g., using strain gauge circuitry) and can be used to dynamically adjust line width W. For example, if the user wishes to create a wide line width portion of line 136, the user can temporarily increase the clamping force applied to finger device 10. In a pinch gesture, both devices 10 receive increased force measurements simultaneously, so that the pinch gesture can be distinguished from gestures involving force applied to only a single finger device 10 (such as a single finger tap).

[0094] In general, the pinch gesture and / or other finger input may be used to adjust any suitable properties of a drawn line or other on-screen content (e.g., brush texture and other brush properties, line color, object size, etc.). The use of the pinch gesture input to adjust the line width W is exemplary. The pinch gesture and other finger input from device 10 may also be used to control zoom functions (e.g., to zoom in on an item of interest by performing a three-dimensional pinch-to-zoom operation, etc.), rotation operations (e.g., to rotate a three-dimensional object), moving an object along a selected axis, and / or other operations in system 8.

[0095] Fig.21 It is a simplified diagram of a mixed reality environment that can be presented to a user of device 24 when the user interacts with device 24 using finger device 10. The surface of real-world desktop 154 ​​can be superimposed with computer-generated objects, such as virtual horizontal surface 140. Virtual vertical surface 142 can be displayed adjacent to virtual horizontal surface 140 (e.g., to form a virtual three-dimensional desktop). If desired, surface 140 and surface 142 can have colors, textures, and other visual attributes that help distinguish surface 140 and surface 142 from objects in the real world. Icons, lists, menu options, text, static and / or moving images, and / or other virtual objects that can be displayed on surface 140 and / or surface 142. These virtual objects can be manipulated by the user using finger device 10.

[0096] In some configurations, surface 140 and surface 142 are not associated with any real-world object (e.g., surface 140 and surface 142 may float). In other configurations, surface 140 and / or surface 142 may be aligned with a wall, table, or other non-electronic device surface and / or may be aligned with a surface of a display or other component in an electronic device. For example, surface 142 may be aligned with a desktop so that the movement of a user's fingers and device 10 on the desktop can be used to drag and drop, and otherwise move and manipulate virtual objects visually displayed on the desktop. At the same time, due to the physical presence of the desktop, the user's fingers will contact the desktop (and if necessary, the finger device 10 will detect the contact and provide tactile feedback), so that it will be natural for the user to select a virtual object on the desktop. If desired, a touchpad or mouse input or other input of device 24 can be used to manipulate virtual objects on a surface such as surface 142 (e.g., move objects, etc.).

[0097] If desired, surface 142 can be aligned with a display in a desktop computer (see, e.g., Fig.21 160, which may be a desktop computer having a housing and a display in the housing), surface 140 may be aligned with a display in a tablet computer located on a tabletop 154 ​​(see, for example, Fig.21 162, which may be a tablet computer having a housing and a display in the housing, etc. These arrangements may be used to allow device 10 to collect input and provide tactile output when a user's finger touches an object on a desktop computer display or a display on a tablet computer. Fig.21 In the example of FIG. 1 , surface 140 is aligned with surface 154. This is illustrative.

[0098] Users can Fig.218 to interact with virtual content in the environment to control one or more related devices 24 in the system 8. As an example, consider a virtual object such as virtual object 138, which may be an email icon. A user who wishes to view an email inbox on a device 24 in the system 8 may select object 138 by moving the finger device 10 inwardly in a direction 150 toward object 138 with a finger 40. When the user's finger reaches the surface of object 138, a tactile output device in the device 10 may provide appropriate tactile feedback. After selecting the email icon, the user may move the email icon from surface 140 to surface 142 (e.g., using a drag and drop command, using a flick gesture, by picking up and dropping the icon, etc.). This instructs the user's system (e.g., a computer and / or a head mounted device 24 that wirelessly communicates with the computer) to display an email inbox 144 on surface 142. After the email inbox 144 has been displayed, the user may place a finger 40 on an email message subject line 146 listed in the email list of the inbox 144 to select and open the desired email.

[0099] If desired, the user need not directly interact with virtual object 138. For example, a virtual pointer (e.g., a pair of virtual fingers) may appear on surface 140 and may be remotely manipulated by the user as the user's finger device 10 is moved, for example, in direction 152 to grasp object manipulation location 154 on real-world desktop 156. In this way, the virtual fingers may be directed to manipulate virtual object 138 (e.g., to move object 138 from surface 140 to surface 142 or otherwise manipulate the virtual object in a virtual world presented to the user).

[0100] When interacting with virtual objects (such as virtual object 138 and list 144), in addition to and / or in lieu of using finger input from finger device 10, the user's gaze point may be used as user input. As an example, when the user's gaze point is directed to object 138, the appearance of object 138 may be automatically enlarged or otherwise highlighted. The user may then grab the object using finger device 10, or may provide other finger input so that the object may be moved to surface 142. In this way, virtual objects such as virtual object 138 that are superimposed on the real world (and / or a real-time image of the real world) may be manipulated using gaze point information and finger input from device 10.

[0101] Fig.21 The image sensor or other circuitry in the device can be used as a hand tracking system. For example, Fig.21 The structure 160 may be a desktop computer with an image sensor, and / or Fig.21The structure 162 can be a tablet computer or other electronic device with an image sensor (e.g., see device 24). The image sensor can be used to capture an image of a user's hand (e.g., a hand including finger 4). By processing this image data, system 8 can track the position of the user's hand (and finger 40). System 8 can be used, for example, to determine the distance of the user's hand from the display and monitor the horizontal and vertical movement of the hand relative to the display. By using a hand tracking system to determine the current position of the user's hand, the accuracy with which the computer or other device 24 can determine the position of the finger device 10 and finger 40 can be enhanced.

[0102] For example, image-based hand tracking can be used alone or in combination with finger position data from device 10 to determine the position of device 10 and a user's finger relative to a device containing an image sensor and / or other circuitry of a hand tracking system (e.g., device 24). Device 10 can then be used to collect force measurements or other user inputs indicative of a touch event in which a user's finger contacts the surface of a display in structure 160 and / or structure 162 (e.g., to select a virtual object of interest, etc.). Tactile output can be provided in response to a touch contact detected between an exposed surface of a user's finger and a display surface. In this way, gross movements such as movement of device 10 due to hand movement can be accurately tracked (e.g., using only an image-based hand tracking system or other hand tracker or in combination with finger and hand movement data from device 10), while finger device 10 is able to use force sensors or other sensors to sensitively detect touch events (e.g., tapping the surface of a display and other fine movements, such as those associated with touch, to select a virtual object, so that hand movement or other movement of finger 40 can be used to move the selected virtual object on the display). When manipulating displayed objects using hand position measurements from image sensors or other sensors in a hand tracking system and / or using finger device position data, finger device 10 can use a tactile output device to provide associated tactile output, such as tactile output associated with a touch event.

[0103] As demonstrated in this example, a user can manipulate virtual objects (e.g., the user can select and move objects, draw lines, etc.) by moving the finger device 10 (whether using a hand tracking system in the device 24, a position sensor in the device 10, and / or both hand tracking and position sensor measurements to detect the position of the finger device 10). For example, the user can select, drag, and drop virtual objects on the display of the device 24.

[0104] As the user interacts with system 8, user input from the sensors may be wirelessly transmitted from device 10 to device 24 in real time, and control signals for haptic output devices in device 10 may be received from control circuitry within device 10. The control signals for the haptic output devices can originate from within device 10 and / or be provided to the haptic output devices by control circuitry within device 10 based on wirelessly received signals from device 24.

[0105] Fig. 22 is a schematic diagram of an exemplary configuration of system 8. Fig. 22 In the example of FIG. 1 , system 8 includes a first device, such as finger device 10, which wirelessly communicates with external electronic device (host) 24 via path 38. Device 10 may have storage and processing circuits (such as controller 170 and memory 172) and / or other control circuits for controlling the operation of device 10. During operation, the control circuits of device 10 (see, e.g., Figure 1 The control circuit 12) can use the communication circuit (see for example Figure 1 The controller 170 may use a communication circuit 14) to communicate with the electronic device 24. For example, the controller 170 may use a communication circuit such as The wireless transceiver circuitry of the transceiver circuit 194 and the antenna 196 is used to wirelessly communicate with the electronic device 24.

[0106] Power for device 10 may be provided using an energy storage device such as battery 178 or a supercapacitor (as an example). Battery 178 may be charged using power provided through a connector in port 182 and / or power wirelessly received from a wireless power transmitter using wireless power receiving circuitry in port 182. Charging circuitry 180 may provide the received power to battery 178 to charge battery 178 and / or direct the received power to a load circuit in device 10.

[0107] To enhance the ability of cameras and other tracking devices to track the position and orientation of device 10, controller 170 may use LED drive circuit 184 to turn on a group of LEDs 186 (e.g., a group of four LEDs, each located at a corresponding corner of the housing of device 10). Controller 170 may provide tactile output to a user's finger by driving piezoelectric actuator 176 using piezoelectric driver circuit 174, or may use other tactile output devices 20 to provide tactile output.

[0108] The device 10 may collect data from sensors (see, e.g. Figure 1The controller 170 may also use input from a plurality of strain gauges 190 and other sensors 192 (e.g., force sensing resistors). The input may provide information about the orientation and movement of the device 10 to the controller 170. The input from one or more strain gauges 190 and other sensors 192 (e.g., force sensing resistors) may be collected using an analog-to-digital converter circuit 188. The information from the strain gauges 190 and other sensors may indicate the force on the housing 44 and may therefore be used to measure lateral movement of the finger 40 (e.g., movement that causes the housing 44 to bend, such that the strain gauges may be used to detect such bending, movement that causes the finger 40 to press against the force sensing resistor, etc.). The use of multiple strain gauges 190 and / or other groups of multiple sensors may help make the system 8 more robust to failures. The input from multiple strain gauges 190 may also be used by the controller 170 to determine the touch location and / or touch direction. If desired, touch input from a capacitive touch sensor or a force sensor-based touch sensor may be collected. User input (eg, finger input acquired using motion and orientation sensors and / or other sensors such as strain gauge circuits and / or other force sensors) may be processed locally and / or may be wirelessly transmitted to the device 24 for processing.

[0109] The Bluetooth circuit 194 may conform to a standard such as the Universal Serial Bus (USB) Human Interface Device (HID) standard. During operation, the controller 170 may process raw sensor data to detect taps and other gestures and detect pointing movements of the device 10. Detected user inputs can be wirelessly transmitted to the electronic device 24. With another exemplary arrangement, the controller 170 sends the measured sensor data to the electronic device 24 for processing (e.g., for tap recognition and other gesture processing). Other arrangements may be used if desired.

[0110] When controller 170 detects taps and other user input events, controller 170 can use driver 174 and actuator 176 to provide tactile output to the user. As an example, for example, such tactile output may be suitable for situations where it is desired to provide corresponding tactile feedback to a user who makes a tap input or other input with device 10. In scenarios where processing circuitry at device 24 is used to process taps and other user input devices, device 24 can wirelessly direct controller 170 to use driver 174 and piezoelectric actuator 176 to provide tactile output to the user's finger. Tactile output waveform (see, e.g., Fig.15 ) can be stored in memory 172.

[0111] If desired, input may be collected from multiple users using the finger device 10 and / or other devices in the system 8. Output may also be provided to multiple users using the finger device 10 and / or other devices in the system 8. In some arrangements, user input from the finger device 10 and / or other devices in the system 8 may be collected from multiple users interacting with each other in a virtual environment (e.g., a game, a social application, or other application in an online environment, where users communicate via local wireless communications (e.g., a game, a social application, or other application in an online environment, a local environment (where users interact via local wireless communications, such as a communication arrangement where devices in the system 8 are connected via a wireless local area network and / or peer-to-peer wireless), or other collaborative environments)). During these interactions between users, tactile output and / or other output may be provided to the users using the finger device 10 and / or other devices in the system 8.

[0112] The control circuitry 12 in device 10 and / or the control circuitry 26 in device 24 may take any suitable action in response to input from multiple users. For example, if multiple users are playing a game and / or interacting in a social application, the control circuitry in system 8 may collect game and / or social application input from each user, and may provide tactile output or other output to one or more appropriate users as appropriate (e.g., as a form of communication, as feedback related to game content, etc.). As another example, if two users are interacting socially (e.g., by shaking hands), the control circuitry in system 8 may collect user input indicating the handshake of the users shaking hands, and in response, may take actions such as providing tactile output to confirm to each of these users that business card information or other personal information has been wirelessly exchanged (e.g., a tactile output of a first type indicating the contact information that has been transmitted and / or a tactile output indicating the type of contact information that has been received). If desired, handshakes and other social interactions may be conducted online, and the tactile output or other output is performed accordingly at device 10 and / or other devices provided in system 8 (e.g., as feedback during an online handshake or other social interaction).

[0113] In some arrangements, the device 10 and / or other devices in the system 8 can be used by a disabled person. As an example, consider a user with limited vision. The user can run an application on the system 8 (e.g., on the control circuit 12 and / or the control circuit 26). The application can use the sensor 18 and / or the sensor 32 to sense when the user is near an object (e.g., a physical element in the user's environment, such as a wall, door, or step). Time-of-flight optical sensors, radio frequency sensors, and / or other sensors for range detection (e.g., long-range proximity sensors) can be used to detect external objects when the user is interacting with the user's environment. In response to detecting an external object, the finger device 10 and / or other devices in the system 8 can provide the user with corresponding tactile outputs, audio outputs, and / or other outputs. As an example, the system 8 can be used to implement a virtual cane for a non-signing person of any length. When the system 8 detects a step or other obstacle, tactile feedback or other output can notify the user of the presence of the step, so that the user can take appropriate action.

[0114] If desired, a head mounted device (e.g., the device 24 may include a head mounted device) may be used to provide tactile output, audio output, and / or visual output to a user with limited vision or other users, with or without accompanying output from the finger device 10 in the system 8. When an obstacle or person outside the user's field of view is detected, for example, the system 8 may provide a tactile vibration to warn the user or may provide other outputs that indicate to the wearer of the head mounted device and / or finger device whether an obstacle or person exists. Tactile output and / or other outputs may also be provided to hearing impaired users in this manner to indicate the presence of external objects, people present to the user.

[0115] According to one embodiment, a system is provided, the system including a display, a control circuit, the control circuit being configured to receive finger input collected by a finger device worn on a user's finger while exposing the finger pad at the tip of the finger; and a gaze tracker, the gaze tracker being configured to generate gaze point information associated with the user's eyes, the control circuit being configured to use the display to display virtual content superimposed on real-world content, and the control circuit being configured to use the display to display the virtual content based on the finger input and the gaze point information.

[0116] According to another embodiment, the finger input includes strain gauge circuit input collected by a strain gauge circuit in the finger device and includes inertial measurement unit input collected by an inertial measurement unit in the finger device, the virtual content is associated with real-world objects in the real-world content, and the control circuit is configured to display selectable items in the virtual content using the display.

[0117] According to another embodiment, the real-world object comprises an electronic device, and the control circuitry is configured to display the selectable items in the list using the display.

[0118] According to another embodiment, the finger device includes a tactile output device, and the control circuit is configured to capture finger input when the finger device is used to select a selectable item and to provide corresponding tactile output to the finger using the tactile output device.

[0119] According to another embodiment, the system includes a head-mounted support structure having a display mounted thereon, and the gaze tracker is mounted to the head-mounted support structure.

[0120] According to another embodiment, the control circuit is configured to receive additional finger input from an additional finger device, and the control circuit is configured to display virtual content based on the finger input from the finger device and the additional finger input from the additional finger device.

[0121] According to one embodiment, a head-mounted device operable by a finger device is provided, the head-mounted device being configured to collect finger input from a user's fingers, the head-mounted device comprising a control circuit configured to receive the finger input from the finger device; a head-mounted support structure; a display, which can be supported by the head-mounted support structure; and a gaze tracker, the gaze tracker being configured to collect gaze point information, the control circuit being configured to use the display to display virtual content superimposed on real-world content and based on the gaze point information and the finger input.

[0122] According to another embodiment, the user has eyes, the gaze point information is associated with the user's eyes, and the control circuit includes wireless communication circuitry configured to wirelessly receive finger input from a finger device.

[0123] According to another embodiment, the finger device includes an inertial measurement unit configured to collect finger input when the finger interacts with virtual content, and the control circuit is configured to display the virtual content using the display based on the finger input collected by the inertial measurement unit.

[0124] According to another embodiment, the head-mounted support structure is configured to support a gaze tracker that collects gaze point information when a user views an electronic device, and the virtual content includes a list associated with the electronic device.

[0125] According to one embodiment, a finger device operated by a head-mounted device is provided, the finger device being configured to display virtual content on an object in the real world, the finger device comprising a shell configured to be coupled to a finger at the tip of the finger while exposing a finger pad at the tip of the finger; a tactile output device coupled to the shell; a sensor circuit configured to collect finger position information when the finger interacts with the displayed virtual content; and a control circuit configured to provide tactile output to the finger using the tactile output device based on the finger position information.

[0126] According to another embodiment, the sensor circuit includes an inertial measurement unit configured to collect position information when a user is interacting with a real-world object overlaid with virtual content.

[0127] According to another embodiment, the sensor circuit includes a strain gauge circuit configured to collect force measurements associated with forces generated by a finger when the finger contacts a surface of a real-world object while interacting with displayed virtual content.

[0128] According to another embodiment, the sensor circuit includes an ultrasonic sensor.

[0129] According to another embodiment, the sensor circuit includes an image sensor.

[0130] According to another embodiment, the sensor circuit includes a depth sensor.

[0131] In another embodiment, the control circuit is configured to provide tactile output to the tactile output device while a finger selects an item in an interactive list in the virtual content.

[0132] According to another embodiment, the real world object does not contain circuitry, and the sensor circuitry is configured to collect finger position information when the finger interacts with the displayed virtual content while moving the real object.

[0133] According to another embodiment, the control circuit is configured to provide different textures in different areas of the real world to the finger by using a tactile output device to provide different tactile outputs to the finger in different areas based on the finger position information.

[0134] In another embodiment, the virtual content includes a virtual object and a control circuit configured to move the virtual object in response to the finger position information.

[0135] According to one embodiment, a finger device configured to be worn on a user's finger is provided, the finger device comprising a shell, the shell being configured to be coupled to the tip of the finger while exposing a finger pad at the tip of the finger; a tactile output device coupled to the shell; a touch sensor configured to collect touch input along an outer surface of the shell; and a control circuit configured to provide a tactile output to the finger using the tactile output device based on the collected touch input.

[0136] According to another embodiment, the touch sensor includes a capacitive touch sensor electrode array extending along the housing, the touch sensor is configured to collect touch input from an additional finger touching the touch sensor, and the control circuit is configured to create a tactile stop by providing tactile output when the additional finger moves along the capacitive touch sensor electrode array.

[0137] According to another embodiment, the housing is configured to accommodate a finger without covering a finger pad at the tip of the finger, and the touch sensor has an array of sensor elements extending along the housing.

[0138] According to another embodiment, the sensor element includes a capacitive touch sensor electrode.

[0139] According to another embodiment, a finger device includes a sensor coupled to a housing that collects information about a finger's interaction with an external object.

[0140] According to another embodiment, the sensor coupled to the housing includes a sensor selected from the group consisting of an inertial measurement unit and a strain gauge.

[0141] According to another embodiment, the sensor coupled to the housing includes an image sensor.

[0142] According to another embodiment, the sensor coupled to the housing includes a depth sensor.

[0143] According to another embodiment, a sensor coupled to the housing includes a strain gauge coupled to an elongated arm in the housing.

[0144] According to another embodiment, a finger device includes a sensor element configured to capture touch input from an area of ​​the finger that is adjacent to the housing and does not overlap the housing.

[0145] According to another embodiment, a finger device includes a force sensor configured to capture a finger pinch input from a finger when the finger presses against another finger.

[0146] According to one embodiment, a finger device that can be operated by a wireless electronic device is provided, and is configured to be worn on a user's first finger while receiving touch input from the user's second finger. The finger device includes a shell, which is configured to be connected to the first finger; a touch sensor, which is configured to collect touch input from the second finger along the outer surface of the shell; and a control circuit, which is configured to wirelessly transmit the collected touch input to the wireless electronic device.

[0147] According to another embodiment, a finger device includes a haptic output device configured to provide a haptic output.

[0148] According to another embodiment, the control circuit is configured to create a stop using a haptic output device when the second finger is moved along the touch sensor.

[0149] According to another embodiment, a finger device includes a sensor configured to capture a finger input associated with a first finger.

[0150] According to another embodiment, the control circuit is configured to provide a tactile output to a first finger using a tactile output device based on the finger input.

[0151] According to another embodiment, the sensor includes an accelerometer and the finger input includes finger movement sensed using the accelerometer.

[0152] According to another embodiment, the sensor includes an inertial measurement unit.

[0153] According to another embodiment, the sensor includes a force sensor.

[0154] According to another embodiment, the sensor comprises an ultrasonic sensor.

[0155] According to another embodiment, the sensor comprises an optical sensor.

[0156] According to another embodiment, the housing is configured to couple to a tip of a first finger while leaving a finger pad at the tip of the first finger exposed.

[0157] According to another embodiment, the housing includes a U-shaped housing having a first side and a second side connected by a top, and the touch sensor is configured to capture touch input on the first side, the second side, and the top.

[0158] According to another embodiment, a wireless electronic device includes a head-mounted device configured to display virtual content superimposed on an external surface when a touch sensor captures touch input.

[0159] According to another embodiment, the housing has a bendable arm with a strain gauge, and the touch sensor is superimposed on the bendable arm.

[0160] According to one embodiment, a finger device that can be operated by a wireless electronic device is provided, the finger device is configured to be worn on a first finger of a user and simultaneously receive touch input from a second finger of the user, the finger device comprising a shell configured to be connected to the first finger; a sensor configured to collect finger input from the second finger, an area on the first finger not overlapping with the shell; and a control circuit configured to wirelessly transmit the collected finger input to the wireless electronic device.

[0161] According to another embodiment, the finger device includes orthogonal first, second and third direction sensors, and the control circuit is configured to collect information about the orientation of the first finger from the first, second and third direction sensors.

[0162] According to another embodiment, the first sensor, the second sensor, and the third sensor include radio frequency sensors.

[0163] According to another embodiment, the housing has a portion protruding in front of a finger tip portion of a first finger and includes a sensor circuit configured to collect information about movement of the first finger.

[0164] In accordance with one embodiment, a finger device operable with a computer having a display is provided, the finger device comprising a housing configured to be worn on a user's finger; a tactile output device; a sensor configured to collect user input when the user's finger touches the display; a wireless communication circuit configured to transmit the user input to the computer; and a control circuit configured to provide tactile output to the user's finger using the tactile output device when an exposed surface of the finger contacts the display.

[0165] According to another embodiment, the computer includes a tablet computer, the display is configured to display an object, and the control circuit is configured to use the sensor to collect user input when a finger touches the object.

[0166] According to another embodiment, a finger device includes a touch sensor configured to collect touch input along an exterior surface of a housing, and a control circuit configured to provide a tactile output to the finger using a tactile output device based on the collected touch input.

[0167] According to another embodiment, the computer comprises a desktop computer, the display is configured to display an object, and the control circuit is configured to collect user input using the sensor when a finger touches the object.

[0168] According to another embodiment, a finger device includes a touch sensor configured to collect touch input along an exterior surface of a housing, and a control circuit configured to provide a tactile output to the finger using a tactile output device based on the collected touch input.

[0169] According to one embodiment, an electronic device operable by a finger device is provided, the finger device being configured to be wearable on a finger of a user's hand and having a sensor configured to collect user input, the finger device comprising a display; a hand tracking system configured to measure movement of the user's hand; a wireless communication circuit configured to receive user input collected by the sensor in the finger device; and a control circuit configured to move an object on the display in response to the measured movement of the hand and in response to the user input.

[0170] According to one embodiment, a system is provided, the system including a display, a control circuit, the control circuit being configured to receive finger input collected by a finger device worn on a user's finger while exposing the finger pad at the tip of the finger; and a gaze tracker, the gaze tracker being configured to generate gaze point information associated with the user's eyes, the control circuit being configured to use the display to display virtual content superimposed on real-world content, and the control circuit being configured to use the display to display the virtual content based on the finger input and the gaze point information.

[0171] According to another embodiment, the finger input includes strain gauge circuit input collected by a strain gauge circuit in the finger device and includes inertial measurement unit input collected by an inertial measurement unit in the finger device, the virtual content is associated with real-world objects in the real-world content, and the control circuit is configured to display selectable items in the virtual content using the display.

[0172] According to another embodiment, the real-world object comprises an electronic device, and the control circuitry is configured to display the selectable items in the list using the display.

[0173] According to another embodiment, the finger device includes a tactile output device, and the control circuit is configured to capture finger input when the finger device is used to select a selectable item and to provide corresponding tactile output to the finger using the tactile output device.

[0174] According to another embodiment, the system includes a head-mounted support structure having a display mounted thereon, and the gaze tracker is mounted to the head-mounted support structure.

[0175] According to another embodiment, the control circuit is configured to receive additional finger input from an additional finger device, and the control circuit is configured to display virtual content based on the finger input from the finger device and the additional finger input from the additional finger device.

[0176] According to one embodiment, a head-mounted device operable by a finger device is provided, the head-mounted device being configured to collect finger input from a user's fingers, the head-mounted device comprising a control circuit configured to receive the finger input from the finger device; a head-mounted support structure; a display supported by the head-mounted support structure; and a gaze tracker configured to collect gaze point information; the control circuit being configured to use the display to display virtual content, the virtual content being superimposed on real-world content and being based on the gaze point information and the finger input.

[0177] According to another embodiment, the user has eyes, the gaze point information is associated with the user's eyes, and the control circuit includes wireless communication circuitry configured to wirelessly receive finger input from a finger device.

[0178] According to another embodiment, the finger device includes an inertial measurement unit configured to collect finger input when the finger interacts with virtual content, and the control circuit is configured to display the virtual content using the display based on the finger input collected by the inertial measurement unit.

[0179] According to another embodiment, the head-mounted support structure is configured to support a gaze tracker that collects gaze point information when a user views an electronic device, and the virtual content includes a list associated with the electronic device.

[0180] According to one embodiment, a finger device operable with a head-mounted device, the finger device being configured to display virtual content on an object in the real world, is provided, the finger device comprising a housing configured to be coupled to a finger at the tip of the finger while exposing a finger pad at the tip of the finger; a tactile output device coupled to the housing; a sensor circuit configured to collect finger position information when the finger interacts with the displayed virtual content; and a control circuit configured to provide tactile output to the finger using the tactile output device based on the finger position information.

[0181] According to another embodiment, the sensor circuit includes an inertial measurement unit configured to collect position information when a user is interacting with a real-world object overlaid with virtual content.

[0182] According to another embodiment, the sensor circuit includes a strain gauge circuit configured to collect force measurements associated with forces generated by a finger when the finger contacts a surface of a real-world object while interacting with displayed virtual content.

[0183] According to another embodiment, the sensor circuit includes an ultrasonic sensor.

[0184] According to another embodiment, the sensor circuit includes an image sensor.

[0185] According to another embodiment, the sensor circuit includes a depth sensor.

[0186] In another embodiment, the control circuit is configured to provide tactile output to the tactile output device when the finger is selecting an item in the interactive list in the virtual content.

[0187] According to another embodiment, the real world object does not contain circuitry, and the sensor circuitry is configured to collect finger position information when the finger interacts with the displayed virtual content while moving the real object.

[0188] According to another embodiment, the control circuit is configured to provide different textures in different areas of the real world to the finger by using a tactile output device to provide different tactile outputs to the finger in different areas based on the finger position information.

[0189] In another embodiment, the virtual content includes a virtual object and a control circuit configured to move the virtual object in response to the finger position information.

[0190] According to one embodiment, a finger device configured to be worn on a user's finger is provided, the finger device comprising a shell, the shell being configured to be coupled to the tip of the finger while exposing a finger pad at the tip of the finger; a tactile output device coupled to the shell; a touch sensor configured to collect touch input along an outer surface of the shell; and a control circuit configured to provide a tactile output to the finger using the tactile output device based on the collected touch input.

[0191] According to another embodiment, the touch sensor includes a capacitive touch sensor electrode array extending along the housing, the touch sensor is configured to collect touch input from an additional finger touching the touch sensor, and the control circuit is configured to create a tactile stop by providing tactile output when the additional finger moves along the capacitive touch sensor electrode array.

[0192] According to another embodiment, the housing is configured to accommodate a finger without covering a finger pad at the tip of the finger, and the touch sensor has an array of sensor elements extending along the housing.

[0193] According to another embodiment, the sensor element includes a capacitive touch sensor electrode.

[0194] According to another embodiment, a finger device includes a sensor coupled to a housing that collects information about a finger's interaction with an external object.

[0195] According to another embodiment, the sensor coupled to the housing includes a sensor selected from the group consisting of an inertial measurement unit and a strain gauge.

[0196] According to another embodiment, the sensor coupled to the housing includes an image sensor.

[0197] According to another embodiment, the sensor coupled to the housing includes a depth sensor.

[0198] According to another embodiment, a sensor coupled to the housing includes a strain gauge coupled to an elongated arm in the housing.

[0199] According to another embodiment, a finger device is defined that includes a sensor element configured to capture touch input from an area of ​​the finger that is adjacent to the housing and not overlapped by the housing.

[0200] According to another embodiment, a finger device includes a force sensor configured to capture a finger pinch input from a finger when the finger presses against another finger.

[0201] According to one embodiment, a finger device that can be operated by a wireless electronic device is provided, and is configured to be worn on a user's first finger while receiving touch input from the user's second finger. The finger device includes a shell, which is configured to be connected to the first finger; a touch sensor, which is configured to collect touch input from the second finger along the outer surface of the shell; and a control circuit, which is configured to wirelessly transmit the collected touch input to the wireless electronic device.

[0202] According to another embodiment, a finger device includes a haptic output device configured to provide a haptic output.

[0203] According to another embodiment, the control circuit is configured to create a stop using a haptic output device when the second finger is moved along the touch sensor.

[0204] According to another embodiment, a finger device includes a sensor configured to capture a finger input associated with a first finger.

[0205] According to another embodiment, the control circuit is configured to provide a tactile output to a first finger using a tactile output device based on the finger input.

[0206] According to another embodiment, the sensor includes an accelerometer and the finger input includes finger movement sensed using the accelerometer.

[0207] According to another embodiment, the sensor includes an inertial measurement unit.

[0208] According to another embodiment, the sensor includes a force sensor.

[0209] According to another embodiment, the sensor comprises an ultrasonic sensor.

[0210] According to another embodiment, the sensor comprises an optical sensor.

[0211] According to another embodiment, the housing is configured to couple to a tip of a first finger while leaving a finger pad at the tip of the first finger exposed.

[0212] According to another embodiment, the housing includes a U-shaped housing having a first side and a second side connected by a top, and the touch sensor is configured to capture touch input on the first side, the second side, and the top.

[0213] According to another embodiment, a wireless electronic device includes a head-mounted device configured to display virtual content superimposed on an external surface when a touch sensor captures touch input.

[0214] According to another embodiment, the housing has a bendable arm with a strain gauge, and the touch sensor is superimposed on the bendable arm.

[0215] According to one embodiment, a finger device that can be operated by a wireless electronic device and is configured to be worn on a user's first finger while receiving touch input from a second finger of the user, the finger device includes a shell, which is configured to be connected to the first finger; a sensor, which is configured to collect finger input from the second finger in an area on the first finger that does not overlap with the shell; and a control circuit, which is configured to wirelessly transmit the collected finger input to the wireless electronic device.

[0216] According to another embodiment, the finger device includes orthogonal first, second and third direction sensors, and the control circuit is configured to collect information about the orientation of the first finger from the first, second and third direction sensors.

[0217] According to another embodiment, the first sensor, the second sensor, and the third sensor include radio frequency sensors.

[0218] According to another embodiment, the housing has a portion protruding in front of a finger tip portion of a first finger and includes a sensor circuit configured to collect information about movement of the first finger.

[0219] According to one embodiment, a finger device operable with a computer having a display is provided, the finger device comprising a housing configured to be worn on a user's finger; a tactile output device; a sensor configured to collect user input when the user's finger touches the display; a wireless communication circuit configured to transmit the user input to the computer; and a control circuit configured to provide tactile output to the user's finger using the tactile output device when an exposed surface of the finger contacts the display.

[0220] According to another embodiment, the computer includes a tablet computer, the display is configured to display an object, and the control circuit is configured to use the sensor to collect user input when a finger touches the object.

[0221] According to another embodiment, a finger device includes a touch sensor configured to collect touch input along an exterior surface of a housing, and a control circuit configured to provide a tactile output to the finger using a tactile output device based on the collected touch input.

[0222] According to another embodiment, the computer comprises a desktop computer, the display is configured to display an object, and the control circuit is configured to collect user input using the sensor when a finger touches the object.

[0223] According to another embodiment, a finger device includes a touch sensor configured to collect touch input along an exterior surface of a housing, and a control circuit configured to provide a tactile output to the finger using a tactile output device based on the collected touch input.

[0224] According to one embodiment, an electronic device operable by a finger device is provided, the electronic device being configured to be worn on a finger of a user's hand, the electronic device having a sensor configured to collect user input; the provided electronic device includes a display, a hand tracking system configured to measure movement of the user's hand; a wireless communication circuit configured to receive user input collected by the sensor in the finger device, and a control circuit configured to move an object on the display in response to the measured movement of the hand and in response to the user input.

[0225] The foregoing is merely illustrative and various modifications may be made to the embodiments described. The foregoing embodiments may be implemented independently or in any combination.

Claims

1. A system capable of being operated using a finger device, the system comprising: case; monitor; a sensor circuit coupled to the housing, the sensor circuit being configured to collect finger position information; a control circuit configured to receive finger input collected by a finger device worn on a finger of a user while leaving a finger pad at a tip of the finger exposed, and determine a finger position relative to a real-world object based on the finger position information; and A gaze tracker configured to generate gaze point information associated with an eye of the user, wherein the control circuit is configured to use the display to display virtual content superimposed on the real-world object, and wherein the control circuit is configured to use the display to display the virtual content based on the finger input and the gaze point information, and wherein the sensor circuit is configured to collect the finger position information when the finger interacts with the virtual content and determine the movement of the real-world object based on the finger position information.

2. A system according to claim 1, wherein the finger input includes strain gauge circuit input collected by a strain gauge circuit in the finger device and includes inertial measurement unit input collected by an inertial measurement unit in the finger device, wherein the virtual content is associated with real-world objects in real-world content, and wherein the control circuit is configured to display selectable items in the virtual content using the display.

3. The system of claim 1 , wherein the control circuit is configured to receive additional finger input from an additional finger device, and wherein the control circuit is configured to display the virtual content based on the finger input from the finger device and the additional finger input from the additional finger device. 4 . The system of claim 1 , wherein the finger device comprises a sensor configured to collect a finger pinch input, and wherein the control circuit is further configured to display the virtual content using the display based on the finger pinch input.

5. The system of claim 1, wherein the finger device includes a sensor configured to collect finger device orientation information, and wherein the control circuit is further configured to display the virtual content using the display based on the finger device orientation information.

6. A head mounted device capable of being operated using a finger device, the head mounted device being configured to collect finger input from a user, the head mounted device comprising: A sensor circuit configured to collect finger position information; a control circuit configured to receive the finger input from the finger device and determine a finger position relative to a real-world object based on the finger position information; Head-mountable support structure; a display supported by the head-mountable support structure; and A gaze tracker configured to collect gaze point information, wherein the control circuit is configured to use the display to display virtual content superimposed on the real-world object and based on the gaze point information and the finger input, and wherein the sensor circuit is configured to collect the finger position information when the finger interacts with the virtual content and determine the movement of the real-world object based on the finger position information.

7. A head-mounted device according to claim 6, wherein the user has eyes, wherein the gaze point information is associated with the eyes of the user, and wherein the control circuit includes a wireless communication circuit, and the wireless communication circuit is configured to wirelessly receive the finger input from the finger device.

8. A head-mounted device according to claim 6, wherein the head-mountable support structure is configured to support the gaze tracker, wherein the gaze point information is collected by the gaze tracker when the user is viewing the electronic device, and wherein the virtual content includes a list associated with the electronic device.

9. A finger device operable with a head mounted device, the head mounted device being configured to display virtual content on a real world object, the finger device comprising: a housing configured to be coupled to a finger at a tip of the finger while leaving a finger pad at the tip of the finger exposed; a haptic output device coupled to the housing; a sensor circuit configured to collect finger position information when the finger interacts with the displayed virtual content; and and a control circuit configured to determine a position of a finger relative to the real-world object based on the finger position information, provide a tactile output to the finger using the tactile output device based on the finger position information, and determine movement of the real-world object based on the finger position information. 10 . The finger device of claim 9 , wherein the sensor circuit comprises an inertial measurement unit configured to collect position information when a user is interacting with the real-world object overlaid by the virtual content.

11. The finger device of claim 9, wherein the sensor circuit comprises a strain gauge circuit configured to capture force measurements associated with a force generated by the finger when the finger contacts a surface of the real-world object while interacting with displayed virtual content.

Images