Mid-air haptic system and human interaction

Abstract

To provide a method for delivering haptic sensations that remove the need for a device in / on the hand or on the body without a peripheral device such as a haptic glove or armband.SOLUTION: Ultrasound transducer arrays may be mounted on a person (such as on a head mounted display or other wearable accessory). This array may utilize some form of 6 degree-of-freedom tracking for both the body and hands of the user. The arrays may be combined to apply focused acoustic pressure at specific locations on moving hands such that a touch sensation is simulated. Using wearable microphones, ultrasonic signal reflected and transmitted into the body can be used for hand and gesture tracking.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] Related Applications This application claims the benefit of the following two U.S. provisional patent applications, all of which are incorporated herein by reference in their entireties:

[0002] 1) No. 62 / 609,621, Application date: December 22, 2017

[0003] 2) No. 62 / 655,795, Application date: April 10, 2018

[0004] The present disclosure relates generally to human interaction with acoustically driven air-to-air haptic systems. [Background technology]

[0005] A continuous distribution of sound energy, called an "acoustic field," can be used for a range of applications, including haptic feedback in air.

[0006] In this acoustic field, one or more control points can be defined that are amplitude modulated with a signal, thereby generating vibrotactile feedback in the air. Another way to generate feedback is to generate control points that are not amplitude modulated and move them in space to generate a spatio-temporal modulation that can be felt.

[0007] With sufficient actuation surface area, mid-air haptic technology can be scaled up. Rather than being directed to regions of space where it can be actuated, mid-air haptic effects can be brought on-demand to all of the local space the user occupies. In this scenario, much of the existing knowledge about haptic effects is not applicable. A viable approach to generating human-machine interfaces in this scenario would be valuable.

[0008] Furthermore, as mobile technology progresses toward augmented, virtual, and mixed reality head-mounted displays, there is no way to continue the established gesture and productivity related haptics developed for the mobile phone and tablet technology space. Also, once the primary functionality of the phone or tablet is transferred to a head-mounted device, there will be no handheld device capable of generating standard, mechanically coupled haptics. Graphics can be displayed / projected onto a virtual device in the air (possibly held in the hand or on the body), but without a peripheral device such as a haptic glove or haptic armband, there is no good way to deliver haptic sensations that eliminates the need for a device in the hand or on the body.

[0009] These gestures need to be recognized in a fast, efficient manner to provide timely visual and tactile feedback. Camera technology has been widely adopted but has limited capture rates, requires extensive computing, and often has high latency. Wearables can be inconvenient and bulky. Wearable ultrasound arrays can provide high-speed signals that can be used to provide independent or supplemental hand tracking. Summary of the Invention

[0010] Interacting with a haptic system in a volume, any part of which may be enhanced with interactive haptic feedback, is unknown. In this application, several different strategies are described to achieve an 'always on' solution and their implications.

[0011] Additionally, multiple ultrasound transducer arrays may be provided on a person (e.g., on the person's head mounted display or other wearable accessory). Each array would utilize some form of 6 degree of freedom tracking of both the user's body and hand. Using this tracking data, the arrays may be combined to apply acoustic pressure focused to specific locations on the moving hand such that a sense of touch is simulated. A person holding a palm like tablet or phone would experience a sense of touch as would a person holding a physical device and making the same gestures and interactions. With a wearable microphone, ultrasound signals reflected and transmitted to the body can be used to track hands and gestures. [Brief description of the drawings]

[0012] In the accompanying drawings, in which like reference numerals refer to identical or functionally similar elements throughout the various drawings, together with the following detailed description, the drawings are incorporated in and form a part of this specification, and serve to further illustrate embodiments of the concepts that comprise the claimed invention and to explain various principles and advantages of those embodiments.

[0013] [Figure 1] FIG. 1 shows the holding of a portable device.

[0014] [Diagram 2] FIG. 2 shows examples of handheld gestures for emulating with air haptics.

[0015] [Diagram 3] FIG. 3 illustrates hand-based interaction with touch.

[0016] [Figure 4] FIG. 4 illustrates hand-based interaction without touch.

[0017] [Diagram 5] 5A and 5B show a two-handed configuration for measuring finger gestures.

[0018] [Figure 6] FIG. 6 shows tracking of fingers on a palm.

[0019] [Figure 7] FIG. 7 shows the closed palm gesture.

[0020] Those skilled in the art will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to facilitate an understanding of embodiments of the present invention.

[0021] The components of the apparatus and methods are represented, where appropriate, by conventional symbols in the drawings, and only those specific details relevant to understanding the embodiments of the invention are shown, so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of this description.

[0022] I. Ubiquitous mid-air haptic feedback

[0023] A. Asserting Control

[0024] When a user in a haptically activated space requests a connection to a machine, the user must perform a specific gesture to make the machine "follow" the gesture command. This gesture is necessary to provide a gateway for the user to access services and can be customized by the user. This gesture can be unique to each user and can indicate to the machine which user is interacting with it, acting according to the user's preferences. Once this is achieved, a "ready" signal in the form of an air haptic effect is sent to the hand, informing the system that it is now in gesture control mode. This air haptic effect can be configured and reassigned by each user. Further interactions with the machine can be performed using hand gestures. The gesture can be used to transfer control to another device where more precise commands can be obtained, in which case the "ready" signal is turned off as the system is taken out of gesture control mode.

[0025] In gesture control mode, each user should be able to "pin" gestures to a specific device so that the system can reduce the amount of cognitive bandwidth and energy the user needs to expend to perform the action corresponding to the command. A predefined volume of space may be bound to nearby objects. For passive objects, this allows the system to provide location services and haptics to users who have misplaced an object to teach them how to best use the item, or simply hint at the object in an eye-free manner. For objects that actively engage with the system, such as third-party electronic devices and systems, they may have haptic effects attached to them and associated with them to send and receive information and events to and from the user. These air haptic effects may be configurable and reassigned by each user. Each device may have a "pinned" interaction, where each command has a gesture and an air haptic associated with the gesture that captures the task. This task may be determined by considering the most common tasks associated with a given third-party system or device, and may be automatically mapped to cognitively light gestures. This can be seen as an automatic construction of a mapping between tasks and gestures, similar to a Huffman code, where common tasks are assigned simple gestures, while rare tasks are assigned more complex gestures.

[0026] A selection gesture may be performed by a user to assert control over a particular device or appliance. This gesture may include a form of remote selection, such as pointing, or another method of gesturally selecting an area or volume tied to the application, device, or appliance in question. Recognizing the selection elicits a haptic response so that the user is aware that a selection has been made. This may be differentiated from other nearby "active" areas by a difference in haptic effect. If a nearby area is inactive, there may be no haptic effect there, or there may be a general active effect corresponding to the machine being ready to accept gesture input.

[0027] B. Event Notifications

[0028] Additionally, when a user is included within a space, airborne haptic feedback can be used to generate event notifications. These may be configured to convey different information using different airborne haptic effects. Periodically modulated modulations on airborne haptic effects can be used to generate "alarm" notifications. Airborne haptic notifications conveying warnings may interrupt other sensory modalities after a period of time or when other conditions are met. Event notifications can be configured by the user to be applicable only to identified and authenticated individual users in an unobtrusive manner, or to be sent to everyone in the space.

[0029] After an event notification has occurred, a "dismiss" gesture can be used to dismiss the event notification. Events conveyed by the air haptic system may be configured to stop without intervention or to continue until explicitly dismissed in this way or another way. The dismiss action may be configured to request dismissal of all users in the space, for example due to some change notification, or may be satisfied by the action of a single user. In the case of a single user, the notification may be a request for action. This may change the mode of dismissal to either a new event notification with an "accept" gesture and requesting further action of the user before dismissal occurs, transition to another sensory modality, activation of another system, or planning of a further event. The "dismiss" action may be configured to provide a postponement of the air haptic event notification until a later time, as in the case of a timer alarm notification. Such a notification system is not limited to communication by purely haptic stimuli. Additional senses, including the application of parametric audio through the system, may be used to further enhance and stimulate the user to become aware and responsive.

[0030] C. External influences

[0031] When the user is contained within a space where haptics can be activated, air-to-air haptic feedback may be used to communicate the state of external variables. By modifying the haptic effect using the external variables as inputs to a function that generates the haptic feedback pattern, the user can determine, for example, whether the front door is locked, the time remaining in a wash cycle, or the price of a product.

[0032] This may be designed in the form of a haptic template that takes such variables as parameters. These variable types include, but are not limited to, Boolean variables, members of option sets, and actual values. Each template may generate feedback based on one or more of these variables, varying over a range of variables. The way in which these templates interpret input values ​​may be modified by the user, allowing customization of the device. Templates are defined in a coordinate space that may be global, relative to haptic "fixed" positions for the entire hand, palm surface, any finger bone or finger surface, or combinations thereof, facilitating warping of the template to fit the shape of the hand.

[0033] Haptic feedback may be modified by environmental factors in order to optimize the feedback. For example, as the speed of sound increases due to temperature and humidity, the haptic points increase the further away from the device. This is because the wavelengths get longer, with the same number of wavelengths traveled. By modifying the relative distance from the device with the wavelength, the haptic points may be normalized to a constant measurable distance. Similarly, temperature affects the sensitivity of a person's hand. By modifying the strength of the feedback, the numbing effect of low temperatures can be countered.

[0034] D. Additional Disclosures

[0035] Further descriptions of these embodiments include the following. (1) an acoustic field including a distribution of sound energy and generated by a plurality of transducers; A tracking system for tracking human gestures; a control system having a monitoring mode and an action mode, including the use of haptics; A human-machine interface including: A system, characterized in that when the tracking system detects a pre-specified human gesture, the control system switches from a monitoring mode to an action mode. (2) The system of (1) above, wherein the pre-specified human gestures are defined by a user of the human-machine interface. (3) The system of (1) above, wherein when the tracking system detects a pre-specified human gesture, the acoustic field generates a pre-specified haptic effect perceivable by a user of the human-machine interface. (4) A system as described above in (3), in which the pre-specified haptic effect is fixed to the hand of a user of the human-machine interface. (5) The system of (1) above, wherein upon the occurrence of an external event, the control system puts the acoustic field into an action mode and generates a pre-specified haptic effect perceivable by a user of the human-machine interface.

[0036] II. Body-worn ultrasonic haptic solution

[0037] A. Shortcomings of traditional solutions

[0038] 1. Actuated gloves have hygiene issues and limited precision.

[0039] 2. Fixed-location air haptics has the limitations of a "desktop" experience where the location of the area is fixed, presenting a limited zone of haptic interaction.

[0040] 3. Compressed air vortices ("smoke rings") are thrown into the air and felt at a distance. These projectors are bulky machine devices and require mechanical actuators to operate. Besides their size, their biggest drawback in a shifting environment is the high latency. Vortices only move through the air at a few meters / second. The user can easily move in the time it takes for the vortex to move from the source to the user. As a result, this is only really suitable for position-fixed experiences with predictable interactions (e.g. a slow moving ball with a user moving towards it, not reactive interactions). If placed on the user, the effect is compromised, as the interaction is fixed to the user and the haptic location needs to be determined as much as half a second in advance. This latency would result in off-target or unintended feedback with only minimal movement. Wearable ultrasound arrays generate haptics that move at the speed of sound (300+ m / s), virtually eliminating any latency on human length scales. Furthermore, the theoretical size of the transducer is very small (mm) and can be easily integrated into a variety of wearable form factors.

[0041] 4. The following paper ( http: / / www.damngeeky.com / 2013 / 05 / 30 / 11576 / virtual-projection-keyboard-technology-with-haptic-feedback-on-palm-of-your-hand.html ) states the following:

[0042] "3D projection systems and interfaces will be the future of technology contained in the gizmos and gadgets that are already an integral part of our lives. Similar to 3dim Tech's gesture control technology, a design team led by Professors Masatoshi Ishikawa and Hiroyuki Shinoda of the University of Tokyo has developed an interface system that projects interactive displays, such as a virtual keyboard, onto random surfaces such as your hand or paper to provide input to gadgets. Equipped with around 2,000 ultrasonic emitters that provide haptic feedback to the movement and pressure of the keyboard keys virtually projected onto your hand, this new technology has great potential for the future. This means that you will be able to operate your computer, smartphone, tablet, and other gadgets from anywhere in your company office or at home. The technology is still a long way from being perfected, as it is still five years away from seeing commercial applications available to the public, but for now it looks good enough."

[0043] This paper details a user interface projected onto the palm of the hand and a fixed ultrasound array that provides haptic feedback. Differences between this paper and the approach described herein include:

[0044] a. The paper states that the user interface is generated by a projector. The approach described here proposes to use graphics provided by a head-mounted display (AR or VR).

[0045] b. It is not clear from this paper that the projected interface changes with user input. The approach described here changes dynamically with such user input (buttons change, text scrolls, etc.).

[0046] c. The interface presented in this paper appears to only provide "button" gestures. The approach described herein proposes to provide feedback for a wide variety of contact-based gestures.

[0047] d. The approach described herein proposes to provide feedback for gestures intended to control a screen / interface that is not necessarily projected onto the palm of the hand.

[0048] e. The approach described herein proposes a way to use a wearable array to update the tracking information it uses to provide this feedback.

[0049] f. The ultrasound device shown in this paper is likely only capable of generating one focal point at a time. The approach described herein is capable of generating and controlling any number and / or complete acoustic fields.

[0050] g. The paper does not discuss the use of ultrasound as a means to track contact using wearable microphones.

[0051] B. Body-Based Haptic Devices

[0052] The approach described here establishes a robust method for operating ultrasonic phased arrays. By focusing acoustic energy, precise and dynamic haptics can be generated for stationary and moving targets. The goal of the proposed method is to extend this airborne haptic capability to include targeting mobile users, possibly users wearing head-mounted displays. In addition to fixed-position arrays, the method also allows arrays to be placed on users and in areas to target mobile users. Body parts can be used as reflective and interactive haptic surfaces. Additionally, acoustic signals reflected and / or transmitted to the body can be used for responsive and accurate hand tracking, including motion and hand-to-hand contact.

[0053] Figure 1 shows a schematic diagram 100 of a traditional portrait orientation hold of a portable device 110 actuated by a hand 120. In this configuration, one-handed swipes update AR / VR graphics in response to screen scrolling and possibly slider movement. Figure 2 shows a schematic diagram 200 of alternative portable gestures to emulate with air haptics. These include swipe 202, tap 204, pinch 206, multi-touch 208, 210, 212, 218, tilt 214, shake 216, and zoom 220, 222.

[0054] The palmar side of a person's hand is highly sensitive and an ideal target for ultrasound-based haptics. As such, they are the primary interactive zone and vehicle. In addition to not requiring peripheral clothing such as haptic gloves to enable haptic sensation, there is a valuable privacy element to future AR interactions in public spaces that this method will facilitate. Gestures made on the hand maintain the user's privacy, as the surface of the hand serves to prevent revealing the specific gesture or type of content being explored.

[0055] Nevertheless, the individual's arms, legs and other body parts may also be considered as surfaces and means of interaction: for example, some interface elements may be projected onto the user's legs to adjust the radio or air conditioning while driving.

[0056] Haptics can be provided from one or more ultrasound arrays either located on the user (e.g., built into or attached to an HMD, or on some other worn, tracked location accessory), or from fixed locations in the environment, allowing haptics to target active interaction zones. Also, arrays in the environment may be mobile, mounted on actuated arms / surfaces, or even operated by drones or robots.

[0057] One key to achieving focusing is for all ultrasound arrays in the body or environment to know the respective position and orientation of their transducers at all times. HMDs typically operate at ~100Hz in "frames" relative to the display. For each frame, all ultrasound arrays in the system need updated position / orientation information for both their position in the system and the desired focal position (e.g., hand). This can be done using standard optical methods common to HMDs (IR LEDs, fiducials, SLAM, etc.) fixed to the array or with a known relative position to the array. Alternatively, this can be done with tracking body-suits where the array is in a known relative position. The focal position to generate a proper haptic sensation needs to be updated faster than the inter-frame time (typically 5kHz+). This can be done as a pre-programmed route that is updated every frame. Alternatively, the array can update the focal position based on a separate, faster-operating tracker.

[0058] In another configuration, the array can simultaneously send acoustic signals to microphone pickups at known positions that feed back to the array controller. By measuring the amplitude and / or phase changes at the microphones, this can be used as an inter-frame focus location adjustment while waiting for a new frame with tracking updates. In another configuration, an external microphone can be used to measure the reflected sound rather than a signal precisely aimed at the microphone. Hand shape, orientation and velocity information from the last frame can be used to model the reflected signal and how it changes with movement. In another configuration, the reflected signal, along with multiple pickup microphones, can acoustically image the environment, eliminating the need for separate tracking information. Any of these configurations can have the signal coded to include time of transmission and / or array position information.

[0059] 3 and 4 show potential two-handed configurations and how a user may hold their hands with the palm facing the user. This inward-facing palm is referred to as the "surface hand." The user's other hand is referred to as the "interacting hand" and is responsible for two-handed interaction, including any interface gestures such as swipes, clicks, and pinches. FIG. 3 (taken from U.S. Patent Application Publication No. 2015 / 0248787) shows a haptic-based schematic 300 of an interacting hand 310 touching a surface hand 320 via palm 330 or fingertips 340. FIG. 4 (taken from Sixth Sense wearable gestural interface) shows a non-haptic-based schematic 400 of an interacting hand 402 touching a surface hand 404 onto which a numeric pad 405 is projected. The hand includes trackers 406, 408, 410 to determine its position with the projected keyboard and to calculate its interaction with the projected keyboard.

[0060] This surface hand allows any ultrasound array within range or viable angle to impart haptics to the hand in coordination with the graphics shown on the AR / VR HMD. The result is an interface that is externally projected onto the user's hand and that can be interacted with. This is similar to a phone or tablet interface, except with haptics imparted to the surface of the hand to correspond to gestures performed by the user.

[0061] One of the key elements of this method is that haptics are applied towards the surface hand, but it is intended to enable haptics on the interacting hand as needed. Thus, for example, pressing a button will involve the sensation of the surface hand that the button is being pressed inwards, while haptics are also applied to the fingertips of the interacting hand that is performing the interaction. Some of these haptics are directed towards the interacting hand, and some are intended to bounce or bounce off the surface hand, so that they are experienced by parts of both hands involved in the interaction (such as the fingertips of one hand and the palm of the other).

[0062] In a mobile haptic experience, the method of graphics presentation is irrelevant; hands and other body parts are simply used to track and anchor those graphics, whether it be a headset display or projected onto the surface of the body part. Also, such haptics and interfaces do not require graphics to function. Graphics presentation may be provided by an AR headset or other AR device, and may include holograms. Graphics presentation may or may not be present along with haptics.

[0063] One embodiment of this method can include a touchpad like navigation interface that a user can use effectively without graphics being projected directly onto the surface being interacted with. This is similar to moving a mouse cursor on a computer screen while the user moves the mouse across a desk. In another configuration, the hand on the surface can function as a trackpad and the interactive hand can behave as it would a trackpad on a laptop. This includes simple tap and drag gestures as well as 2-5 finger interactions such as two finger drags, pinch-to-zoom, etc.

[0064] To perform these interfaces, hand tracking is required. This is typically achieved by optical tracking. Depth sensitivity while maintaining good xy (image plane) sensitivity is very challenging. This method has difficulty distinguishing whether a finger is hovering just above the palm or actually touching it. This lack of sensitivity limits the precision of palm-to-finger contact and requires exaggerated, high-energy gestures, limiting their appeal. This problem can be addressed with ultrasound by including skin-coupled microphones. These can be located in the headset, on an array, or on any number of wearable devices.

[0065] The most basic configuration of this configuration is shown in Figures 5A and 5B. Figure 5A shows a schematic 500 where an ultrasound array sends a signal in the form of a focal point 510 to one hand 506. The focal point 510 (or multiple points) can be targeted at a point perpendicular to the normal of the array to maximize coupling to the area. Also, folded skin or touching fingers can provide sufficient coupling. As a result of the ultrasound array sending a signal at the focal point 510, a large amplitude sine wave 508 is present on one hand 506 while a small amplitude sine wave 504 is present on the other hand 502.

[0066] FIG. 5B shows a schematic diagram 550 in which an ultrasound array sends a signal in the form of a focal point 560 to one hand 556. The focal point 560 (or points) can be targeted at a point perpendicular to the normal of the array to maximize coupling to the area. Also, folded skin or touching fingers can provide sufficient coupling. One hand 556 is touching another hand 552 at a contact point 562. As a result of the ultrasound array sending a signal at the focal point 510 and at the contact point 562, a large amplitude sine wave 508 is present at one hand 556 and a large amplitude sine wave 554 is present at the other hand 552. This is because the ultrasound is coupled through the skin at the contact point 562, likely by bulk transmission.

[0067] Alternatively, the sound field can be formed into a beam or any shape to maximize the transmitted signal to the desired area. This does not have to be at a level that produces a tactile sensation. The skin-coupled microphone is positioned to be more sensitive to acoustic energy on the receiving side of the body (left side in this case). This can be achieved by placing it near the receiving hand (e.g., on a watch) or by placing a directional microphone in the headset. It should be insensitive to airborne ultrasound (e.g., encased in foam) to avoid collisions of reflected signals. In this configuration, due to the large acoustic impedance mismatch between air and skin, little sound will be transmitted between the hands until there is mechanical contact. Once contact occurs, the skin-coupled microphone receives a clear signal. The signal can employ methods to encode the time of flight, such as phase-keying, to avoid various internal or external reflections.

[0068] In another configuration, multiple skin-coupled microphones are used and combined so that the relative signals can be used to create a more robust touch detection signal. Also, the degree of mechanical contact (force of contact) will increase the signal coupling, which will provide a measure of force. In this way, ultrasound can be used to improve the touch sensitivity of hand tracking.

[0069] In the absence of optical tracking, or if its accuracy or latency is not sufficient for reliable gesture recognition, acoustics can be used to enable (or improve) finger-to-palm tracking. One configuration is in FIG. 6, where a schematic diagram 600 of a hand 610 with three focal points 612, 614, 616 projected onto the palm 611 is shown. A skin-coupled microphone is sensitive to the receiver (in this case the fingers 620 of the interactive hand 619). Each focal point 612, 614, 616 is coded in some way (phase key, frequency, etc.). When the fingers 620 contact the palm 611 at the contact point 622, they couple different amounts of acoustic signal depending on their distance from each focal point 612, 614, 616. By isolating and evaluating the relative strength and / or timing of the acoustic signals 618 received by the interactive hand 619, it is possible to estimate the position of the fingers 620 contacting the palm 611. This allows for both high sensitivity for touch vs. hovering, and low latency xy detection. Using the combined signals from each of the focal points, position can be determined by looking at the relative magnitude and phase.

[0070] More focal points can be used to improve accuracy and minimize the possibility of shadowing by the user with their interactive hands. More redundant signals will reduce possible errors from reflections or misalignment. The focal points can also be adjusted while the user is interacting to maximize resolution and accuracy.

[0071] The method of projecting a focal point onto the hand and then using the interactive hand / fingers to pick up a mechanically coupled signal and measure that signal with a skin-coupled microphone to determine contact can also be applied to inanimate objects. For example, one can consider a flat table with a projected AR or VR interactive user interface. In one configuration, a single focal field is projected onto the surface. This excites acoustic waves (both bulk and surface) in the table similar to the description above for hand binding. When the interactive hand contacts the table, that acoustic signal is coupled to the hand, picked up by the skin-coupled microphone, and analyzed. This works in conjunction with the hand tracking system to give high-speed, high-fidelity contact vs. non-contact tracking. In another configuration, a multi-point field can be projected onto the interactive surface, very similar to the description above describing multiple points on the palm of the hand (Figure 6). When the user contacts the surface, the body combines signals related to the distance to each projected focal point, and relative to those focal points, the contact location is determined.

[0072] There is no need to utilize the palm for control. Figure 7 shows a compilation 700 of some additional gestures possible with a closed hand: buttons 710, dials 720, and sliders 730 (Figure 7 taken from Google's Project Soli). The fingertips (including the thumb) are highly sensitive to ultrasonic haptics and are the subject of feedback in these gestures. In one embodiment, taps or bumps can be added to dials or scrolls, similar to bumps on a mouse scroll wheel. For buttons, two different contact pressures can be recognized (similar to 3D Touch on Apple devices) by using one (or both) haptic taps.

[0073] C. Exemplary Features

[0074] The following illustrates exemplary features of the preceding description.

[0075] 1. It allows for the delivery of haptics to moving targets from mobile locations.

[0076] 2. Using the user's own appendages and other body parts to act as surfaces for haptic interaction.

[0077] 3. Development of haptics designed to combine simultaneous surface and interaction hand haptics to generate responsive and intuitive haptic sensations.

[0078] 4. A skin-coupled microphone is used to measure the transmitted signal.

[0079] Further descriptions of these embodiments include the following. (6) generating an acoustic field from a transducer array attached to a user and having a known relative position and orientation; defining a plurality of control fields, each of the plurality of control fields having a known spatial relationship to the transducer array; dynamically updating the position and orientation of the transducer array as the user moves; dynamically updating the position and orientation of the control field as the user moves; The method includes: (7) The method of (6) above, wherein the position and orientation information is provided by at least one of an optical tracking system, an accelerometer tracking system, and a user-worn tracking system. (8) The method of (6) above, wherein the acoustic field is generated by an airborne haptic feedback system. (9) The method of (8) above, wherein the mid-air haptic feedback system is combined with at least one of graphics provided by a head-mounted display and gestures performed by a user. (10) The method of (9) above, wherein the graphics comprise an interactive user interface. (11) The method of (10) above, in which the graphics are projected near the user's hand. (12) The method of (9) above, in which the gesture uses the palm of the user's hand as a trackpad control interface. (13) The method of (6) above, wherein the user is wearing one or more skin-coupled microphones. (14) The method of (13) above, wherein the acoustic field is directed to couple to a specific body region of the user. (15) The method of (14) above, wherein the acoustic field is measured by a body-coupled microphone to provide tracking information. (16) The method of (13) above, wherein the acoustic field is directed to couple to the object. (17) The method of (16) above, wherein the acoustic field is measured by a body-coupled microphone to provide tracking information. (18) The method of (15) above, in which a body coupled microphone is most sensitive to a particular body part of the user. (19) The method of (15) above, wherein the acoustic field is directed to couple to a specific area of ​​the body where the body coupled microphone is not sensitive, and when the user touches the specific area, the body coupled microphone receives a signal. (20) a) generating an acoustic field from a transducer array having a known relative position and orientation; b) defining a plurality of control fields, where each of the plurality of control fields has a known spatial relationship to the transducer array; c) positioning the control field on the user's hand; d) generating a user interface including visual means provided by a head mounted display; an airborne haptic feedback generating step including: the user interface comprises a virtual screen for displaying graphics information; A method in which a user's hand movements manipulate a virtual screen. (21) The method of (20), wherein the user interface emulates at least one of a smartphone, a touchpad, a tablet, a GUI interface, a trackpad, a keyboard, and a number pad. 22. The method of (20) above, in which the palm of one hand acts as a trackpad for operating a virtual screen for the other hand. (23) The method of (22) above, further comprising the step of projecting the virtual screen onto the palm of the hand. (24) The method of (20) above, wherein the control field on the user's hand provides haptic feedback.

[0080] III. conclusion

[0081] Although the above description discloses specific values, any other specific values ​​may be used to achieve similar results. Furthermore, the various features of the above-described embodiments may be selected and combined to produce numerous variations of improved haptic systems.

[0082] In the foregoing specification, specific embodiments have been described. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of the invention as set forth in the claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present teachings.

[0083] Furthermore, in this specification, relational terms such as first and second, above and below, etc. may be used only to distinguish one entity or action from another entity or action, and do not necessarily require or imply the necessity of such an actual relationship or sequence between such entities or actions. The words "comprises," "includes," "has," "has," "comprises," "comprises," "contains," "containing," or any other variation thereof are intended to include a non-exclusive inclusion. As a result, a process, method, article, or apparatus that includes a list of elements does not include only those elements, but may include other elements that are not expressly listed or that are inherent to such process, method, article, or apparatus. An element that begins with "comprises...a," "has...a," "includes...a," or "contains...a" does not exclude, without limitation, the presence of additional identical elements in the process, method, article, or apparatus that includes the element. "a" and "an" are defined herein as one or more, unless otherwise specified. "Substantially," "essentially," "approximately," "about," or any other version thereof, are defined as approximating what would be understood by one of ordinary skill in the art. The term "coupled," as used herein, is defined as connected, not necessarily directly or mechanically. A device or structure that is "configured" in a certain way is at least so configured, but may be configured in other ways not recited.

[0084] The Abstract of the Disclosure is provided to enable the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the foregoing Detailed Description, it will be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

Claims

1. generating an acoustic field from a transducer array attached to a user and having a relative position and orientation; transmitting signals in the form of focal points targeted at control points in a body region with an ultrasound array to generate sine waves in said body region; dynamically updating the relative positions and orientations of the transducer arrays as the user moves; and dynamically updating the position of the focal point as the user moves; 11. A method for providing tracking information of a body area, comprising: the user wearing one or more skin-coupled microphones; and an amplitude of the sine wave is measured by the one or more skin-coupled microphones to provide tracking information of the body area.

2. The method of claim 1 , wherein the position and orientation information is provided by at least one of an optical tracking system, an accelerometer tracking system, and a tracking system worn by the user.

3. The method of claim 1 , wherein the acoustic field is generated by an air-to-air haptic feedback system.

4. The method of claim 3 , wherein the air-to-air haptic feedback system is combined with at least one of graphics provided by a head-mounted display and gestures performed by the user.

5. The method of claim 4 , wherein the graphics comprise an interactive user interface.

6. The method of claim 4 , wherein the gesture uses the palm of the user's hand as a trackpad control interface.

7. The method of claim 1 , wherein the user is wearing two or more skin-coupled microphones.

Images