Three-dimensional perceptions in haptic systems

Abstract

An acoustic field may be produced from a transducer array having known relative positions and orientations. In this acoustic field, one or more control points may be defined. An amplitude may be assigned to the control point. Mid-air haptic effect for a virtual object (120) on a human hand (110) may be generated by moving the control point in a single closed curve comprising a plurality of curve segments. The single closed curve traverses at least one location where the human hand intersects with the virtual object. Additionally, a user may interact with virtual three-dimensional content using the user's hands while a tracking system monitoring the user's hands, a physics engine updates the properties of the virtual three-dimensional content and a haptic feedback system provides haptic information to the user.

Description

<DP=1>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²THREE-DIMENSIONAL PERCEPTIONS IN HAPTIC SYSTEMS²RELATED APPLICATION²

[0001] This application claims the benefit of the following four U.S. ²Provisional ²Patent Applications²

[0002] 1) Serial No. 62 / 370,522, filed on August 3, 2016; ²

[0003] 2) Serial No. 62 / 370,786, filed on August 4, 2016; ²

[0004] 3) Serial No. 62 / 370,865, filed on August 4, 2016; and ²

[0005] 4) Serial No. 62 / 397,419, filed on September 21, 2016.²FIELD OF THE DISCLOSURE²

[0006] The present disclosure relates generally to improved techniques for ²perception of simulated three-dimensional objects and generation of three-²dimensional sound in haptic-based systems.²BACKGROUND²

[0007] A continuous distribution of sound energy, referred to as an "acoustic ²field" may be used for a range of applications including haptic feedback in ²mid-²air. Such an acoustic field may be produced from a transducer array having ²known relative positions and orientations In this acoustic field, one or more ²control points may be defined. Such control points may have a known spatial ²relationship relative to the transducer array.²

[0008] These control points may be assigned an amplitude and then amplitude-²modulated with a signal and as a result produce vibro-tactile feedback in mid-²air. ²An alternative method to produce feedback is to create control points that are ²not²- -²Date Recue / Date Received 2023-12-04²<DP=2>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²modulated in amplitude and move them around spatially to create "spatio-²temporal" modulation that can be felt.²

[0009] These control points are effectively concentrations of ultrasonic ²energy ²and moving them around generates disturbances in the air. By spatially ²modulating these disturbances created by ultrasonic foci, simply moving them ²backwards and forwards, it is possible to generate low frequency sound through ²²the principle of acoustic radiation force as the focus pushes on the air or ²other ²materials around it.²

[0010] Furthermore, very sudden disturbances are created when such focus ²points ²are quickly created or destroyed without slowly increasing or decreasing the ²amplitude. This creates pops or clicks which are often an unwanted side ²effect. ²Moving the control point can be used to achieve similar effects to changing ²the ²amplitude, but without the sharp changes in pressure that cause pops and ²clicks. ²This means that moving the control point is much more preferable than changing ²²its amplitude. Due to the speed of sound being generally faster than the ²control ²point motion, this enables any generated pressure imbalances to dissipate, so ²the ²control point may be moved quickly without creating powerful air disturbances.²

[0011] When considering the intersections between human hands and virtual ²objects, each finger and the palm in many instances feel a very small part of ²the ²overall shape that go towards the appreciation of the whole. Thus, when the ²hand ²feels a small part of the shape intersection the whole shape intersection must ²be ²created in order to not create and destroy the necessary control points. If ²the ²control points were created and destroyed such that they provide only the ²small ²part of the intersection, this would create unwanted noise. A more economical ²method of describing a shape therefore would save power and time and would ²then enable more of the power expended by the device to be used in creating ²haptic feedback in places which are touched by a hand.²

[0012] Due to cross-modal and other perceptual effects, the induction of touch ²²sensations through mid-air haptics remains surprisingly effective at ²communicating the existence, geometry and surface properties of objects.²- 2 -²Date Recue / Date Received 2023-12-04²<DP=3>²CA 03033789 2019-02-13²WO 2018 / 024788 PCT / EP2017 / 069569²Fundamentally, although much research concentrates on exploration through ²touch alone, the effective use of these system is primarily used and driven ²through ²systems that cross multiple sensory modalities to provide an experience. For ²this ²reason, there exist in practice simple but conceptually complicated effects ²that can ²only be achieved through the realization of these underlying principles.²

[0013] Furthermore, with mid-air haptic devices, virtual objects may be ²theoretically recreated in mid-air for the user to touch. But due to the ²nature of ²mid-air feedback, some fundamental physical limitations exist. This is because ²²there are some parts of the somatosensory system that cannot be directly ²manipulated by the device, for example thermoreceptors for temperature ²sensing.²But finding an optimal method to map output nom the device which ²necessarily ²has strengths and weaknesses peculiar to mid-air haptics¨onto simulated ²physical interactions with virtual objects has been difficult. It is therefore ²²valuable to develop an approach that is readily accepted by a human as ²corresponding to a plausible interaction with the object. Similarly, it is ²useful to ²manipulate focus point spinning in the air to generate a sound wave each time ²it ²moves along a path.²BRIEF DESCRIPTION OF THE FIGURES²

[0014] The accompanying figures, where like reference numerals refer to ²identical ²or functionally similar elements throughout the separate views, together with ²the ²detailed description below, are incorporated in and form part of the ²specification, ²and serve to further illustrate embodiments of concepts that include the ²claimed ²invention, and explain various principles and advantages of those embodiments.²

[0015] Figure 1 is a representation of a hand interacting with a virtual cube ²in ²accordance with some embodiments.²

[0016] Figure 2 is a representation of a control point moving in air that ²generates ²sound waves in accordance with some embodiments.²- 3 -²Date Recue / Date Received 2023-12-04²<DP=4>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²

[0017] Skilled artisans will appreciate that elements in the figures are ²illustrated ²for simplicity and clarity and have not necessarily been drawn to scale. For ²example, the dimensions of some of the elements in the figures may be ²exaggerated relative to other elements to help to improve understanding of ²embodiments of the present invention.²

[0018] The apparatus and method components have been represented where ²appropriate by conventional symbols in the drawings, showing only those ²specific ²details that are pertinent to understanding the embodiments of the present ²invention so as not to obscure the disclosure with details that will be ²readily ²apparent to those of ordinary skill in the art having the benefit of the ²description ²herein.²DETAILED DESCRIPTION²

[0019] The solution described herein involving touch and aural interactions ²with²virtual objects has properties that may be used in combination or separately.²

[0020] I. CURVE MANIPULATION FOR 3D-SHAPE REPRODUCTION ²

[0021] When a hand intersects the shell of a three dimensional simulated ²shape, ²the current solution is to find a closed curve that best represents the ²intersection ²that is to be described to the user and actuate that path. Since it is a ²closed path, ²the system may use a single control point moving along the path with a given ²speed (and thus frequency) in order to produce it in mid-air haptics without ²much ²noise being created. But this suffers from two important limitations. First, ²only ²shapes containing paths that can be conveniently interpreted as closed curved ²contours may be represented. A shape containing open contours is not as ²suitable ²for haptic representation: although the path can be bent back upon itself, ²this ²changes the haptic effect. Second, if the constraint of a single control point ²is ²imposed (since each control point requires computation, it is prudent to use ²them ²sparingly) only one contour may be used. If one contour is effective at ²providing ²feedback for one finger and another contour is effective for the palm, both ²cannot²- 4 -²Date Recue / Date Received 2023-12-04²<DP=5>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²be used without recourse to multiple control points, which would split the ²capacity for output from a given device to provide multiple points of ²feedback. ²These drawbacks are in addition to the power limitations of the single contour ²²approach. As the length of the contour increases, the speed that the point ²moves ²increases the point become spread more thinly, eventually decreasing its ²effective ²haptic response.²

[0022] Instead of taking a single contour, finding the set of all ²intersections with ²the parts of the user on which the haptic effect is to be elicited, as line or ²curve ²segments or surfaces is the best solution. But this would involve creating ²many ²open curves, and thus entail creating and destroying the point many times at ²each ²beginning and end point respectively. Alternatively, this would require using ²many control points in order to create separate haptic effects for each ²intersection ²so that each region of haptic feedback may stay spatially locked. For example, ²in ²the specific case of a hand intersection, each finger and potentially also the ²palm ²must have separate haptic effects. This requires many points of contact, but ²so that ²the each control point may stay local to the point of contact or intersection, ²the ²existing solution is either loud (creation and destruction of a single point) ²or ²costly in device power and compute (many points). To overcome this requirement ²²for multiple disconnected paths, taking the set of all intersections and then ²optimizing for the single shortest path that traverses each of them is much ²more ²effective. But this action will move across the shape in a linear fashion with ²a ²beginning and an end, which will create a control point at the beginning, ²traverse ²the shape and destroy the control point at the end, before recreating it at ²the ²beginning of the shape. This continuous creation and destruction, although an ²improvement, remains incompatible with a quiet system.²

[0023] By connecting the beginning and end of the curves into a single closed ²curve and traversing it at the appropriate speed the desired haptic effect can ²be ²generated with significantly reduced noise. Further, this does not preclude ²amplitude modulation of the points as they traverse the given curves. This ²path is ²illustrated in Figure 1.²- 5 -²Date Recue / Date Received 2023-12-04²<DP=6>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²

[0024] Shown in figure 1 is a model 100 of a hand 110 interacting with a ²haptically generated virtual object 120. The points 132, 134, 136, 138 where ²the ²hand touches the virtual shape are connected into a path 140, and this path ²140 is ²actuated, creating the feeling while moving the focus smoothly. Smooth ²movement of the focus along the path 140 without interruption minimizes any ²unwanted audible noise, resulting in correct, strong haptic sensation across ²multiple fingers while operating quietly. In addition, although figure 1 shows ²a ²hand 110, any body part may be used.²

[0025] Because the curve connects each curve segment, the system determines ²whether to traverse them in a preset clockwise or counterclockwise motion. ²This ²motion is relative to a normal vector which may be defined relative to the ²user ²(such as a local coordinate space relative to the geometry of their hand), ²intersections or the geometry. Alternatively, it may be useful to use points ²traversing multiple curves in order to provide feedback for more than one user ²or ²hand.²

[0026] One drawback of this approach is that in order to close the curve the ²user ²himself / herself must be avoided, in order to prevent spurious hapfic effects ²being ²created on the user at locations where intersections did not occur. To achieve ²this ²in the context of the users' hand, the curve should be either made to go over ²or ²under the hand (and thus defocusing at the location of the hand) or around the ²²hand. Alternatively, the state space of the transducers may be interpolated ²between each endpoint, removing the need to find a spatially coherent approach ²to ²closing the curve. The state of the ultrasonic waves in the acoustic field is ²driven ²linearly by the complex valued activation coefficients of each transducer. ²This ²requires, grouping all of these complex values for all transducers into a high-²²dimensional phase space and linearly interpolating between a first state in ²the ²space (that represents, for instance, an end point on one curve) and a second ²state ²(that represents an initial point at a different location). The linear ²interpolation ²must also apply to the acoustic fields resulting from the actuation of the ²newly ²created states in the intermediate stages. It should be noted that in these²- 6 -²Date Recue / Date Received 2023-12-04²<DP=7>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²intermediate stages between the two states, the control point does not ²necessarily ²have any particular location. But because the linear interpolation applies to ²the ²produced acoustic field, the field must smoothly vary in time and therefore ²also ²could serve to also greatly reduce audible output while achieving the movement ²²between points.²

[0027] Another potentially more compatible approach is to change the speed at ²which the control point is moving along the curve. To start, the solution ²computes ²the length of the curve required to complete the closed path, traversing each ²intersection curve segment in turn creating haptic effects at each ²intersection with ²the user. Next, first considering only the parts of the path that do not ²contribute to ²the haptic effects, the solution traverses these haptically unnecessary curves ²at the ²fastest possible speed to avoid wasting power on them. This reduces the haptic ²²actuation of these curves and is beneficial in two ways. First, the amount of ²time ²taken to traverse these parts of the path is reduced; second, if the user is ²intersected in error during this time the amount of haptic effect that he / she ²would ²receive would be minimal. Nevertheless, in the limit of this technique, it ²will ²behave as creation and destruction of the control point, creating pops and ²clicks as ²before. For this reason, it is useful to consider a speed limit that lessens ²the effects ²generated by discontinuities between the states of the acoustic field. The ²result of ²this technique is to create a control point that accelerates and decelerates ²between ²haptic regions. By considering a curve harboring haptic effects in this way, ²there ²may be the requirement that the curve must cycle at a haptically active ²frequency. ²This has the effect of creating a time "budget" of a closed path cycle period, ²part ²of which may be spent on the haptically active parts of the path and the ²remainder ²used to reduce audible noise. In this light, the remaining time in the ²haptically-²actuated curve length is split between the parts of the path that require ²haptics. ²This time "budget" must ensure that these intersections receive as much of the ²²energy from the array as possible so that haptic effects have the time (and ²thus ²power) to be realized at the intersections with the user. In this way, the ²device²- 7 -²Date Recue / Date Received 2023-12-04²<DP=8>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²may be quiet and behave in a fashion that maximizes the energy deposited on ²the ²parts of the user to be haptically actuated such as the fingers.²

[0028] Another issue is that the virtual shape or user position may ²dynamically ²change while haptic effects are being produced. For example, at the current ²time ²to a closed curve containing haptic effects is in current use. At the next ²time step ²in the future t1, a new curve containing haptic effects is synthesized to ²reflect the ²changes in the virtual shape. In this case, a further curve must be plotted ²that ²allows the point to traverse from the old curve representing the state of the ²virtual ²and physical world at time to to the new curve representing the state at time ²t1 ²without discontinuity. Selecting the closest point on each curve, and ²constructing ²a path between them that allows the point to move from the to curve to the t1 ²curve may be used to achieve this. This may also benefit from the introduction ²of ²a speed limit similarly to the unactuated portion of the haptic path described ²²previously to limit the unusable traversal time from one curve to the other.²

[0029] II. TOUCHING VIRTUAL THREE-DIMENSIONAL OBJECTS ²

[0030] A. Introduction ²

[0031] This disclosure is related to a system where a user interacts with ²virtual ²3D-content by a tracking system monitoring his / her hands, using a physics ²engine ²to update the position and properties of the virtual content, and a haptic ²feedback ²system providing haptic information to the user. The haptic system may be mid-²air haptic feedback. A haptic impulse may be applied to the user when his or ²her ²hand contacts a virtual object. This impulse may be applied at the exact ²location ²of the contact. Or the user's hand is formed into a skeletal model and the ²impulse ²is applied to each bone in the skeleton that contacts the object.²

[0032] The strength and duration of the impulse is adjusted based on output ²from ²the physics engine, including momentum and physical properties of the hand and ²²virtual object. The waveform of the haptic impulse may be adjusted based on ²the ²above physics engine outputs. A weaker haptic sensation may be applied to ²parts²- 8 -²Date Recue / Date Received 2023-12-04²<DP=9>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²of the hand when the object is released or when contact with it ceases. In ²contrast, ²there is no haptic feedback applied during continuous holding.²

[0033] Haptic feedback may be applied to the opposite side of the hand to ²which ²the contact is occurring. And haptic feedback follows the hand for the ²decaying ²amplitude experienced after contact.²

[0034] B. Interacting with 3D Objects ²

[0035] A system may have a user who interacts with virtual 3D-content by a ²tracking system monitoring his / her hands, using a physics engine to update the ²²position and properties of the virtual content, and a haptic feedback system ²providing haptic information to the user. The haptic system may use mid-air ²haptic feedback. A haptic impulse may be applied to the user when his or her ²hand contacts a virtual object. This impulse may be applied at the exact ²location ²of the contact. The user's hand may be formed into a skeletal model and the ²impulse is applied to each bone in the skeleton that contacts the object. The ²strength and duration of the impulse may be adjusted based on output from the ²physics engine, including momentum and physical properties of the hand and ²virtual object. The waveform of the haptic impulse may be adjusted based on ²the ²above physics engine outputs.²

[0036] A weaker haptic sensation may be applied to parts of the hand when the ²object is release or when contact with it ceases. Furthermore, there may be no ²²haptic feedback applied during continuous holding. Haptic feedback may be ²applied to the opposite side of the hand to which the contact is occurring. In ²²addition, haptic feedback follows the hand for the decaying amplitude ²experienced after contact.²

[0037] Specifically, when a user is engaged in virtual reality that has been ²enhanced by the addition of a mid-air haptic device, a hand tracking system ²may ²be employed to track their hands in order to determine the position and ²orientation ²of each hand, finger and joint. This information can be input into the virtual ²world ²in the form of a virtual hand object that is capable of interactions with ²virtual²- 9 -²Date Recue / Date Received 2023-12-04²<DP=10>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²objects, albeit only one way. This is because the physical hand has one-to-one ²²correspondence with, and puppets, the virtual hand. The virtual hand has the ²ability to interact with virtual objects, generally through some form of ²physics ²engine. (A physics engine is understood as one of the components of a game ²engine.)²

[0038] For rigid bodies that do not change shape, a physics engine will ²generally ²be responsible for managing the simulated forces and torques applied to the ²objects in the virtual world and the resulting changes in linear, and angular, ²²velocity and acceleration, before generating the corresponding changes in ²spatial ²transformation for each object or object part represented. Other object types ²may ²imply additional management or information, and may also include more complex ²simulation of physical phenomena.²

[0039] Nonetheless, fundamentally, interactions between virtual objects are ²mediated by their simulated forces and torques, which must be found before ²these ²interactions can be simulated. In order to find these simulated forces and ²torques, ²the physics engine will ensure virtual objects are tested for "collisions", ²which is ²effectively testing for future situations in which objects in the virtual ²scene might ²intersect. This generates collision information which feeds the management ²portion of the physics engine with positions, velocities, forces and torques. ²This ²information can also be used to inform a haptics engine about what a user ²might ²feel if parts of the user extended into the virtual world. But this machine is ²unable ²to affect the physical user due to the virtual objects not being able to ²physically ²collide with the user. In the case of the users' hand, the virtual objects do ²not ²affect the virtual hand and thus physical hand. This disconnect causes a loss ²of ²immersion in the virtual world.²

[0040] A satisfying experience results if the mid-air haptic device is ²configured to ²provide feedback to the user at the point at which an impulse (a short sharp ²force, ²during which the force changes in time sharply), contact or similar ²interaction in ²which the force changes sharply in time is applied to the virtual object by ²the user, ²or on the virtual representation of the user by the virtual object. A haptic ²impulse²- to -²Date Recue / Date Received 2023-12-04²<DP=11>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²that mirrors this may be created by generating a similarly short haptic ²effect. This ²is effective because of the occurrence of an illusion of two-way interaction ²due to ²the cross-modal effects in play at this setting. This haptic impulse may then ²be ²adjusted using the trajectory of the object and the hand, along with the ²simulated ²momentum, restitution and other physical properties of the object and hand.²

[0041] The feedback may then be applied with an intensity and waveform²appropriate and proportional to the impulse applied. Due to the proprioceptive ²and ²cross-modal phenomena in effect in this situation, in the case of the users' ²hand, ²the closest addressable part of the hand may be used (including through the ²hand). ²The user will still perceive this haptic effect to be originating at the ²location of the ²collision. Thus, if the impulse occurs on the back of the hand, due to the ²relative ²vibrotactile receptiveness of glabrous skin on the opposite side of the hand, ²this ²other side would be actuated instead of the real location of the contact. ²Perceptual ²effects then cause the user to believe that the impulse was stimulated in the ²correct ²location.²

[0042] In addition, the user in a virtual world may want to pick up virtual ²objects. ²Since picking up and dropping objects is a common activity, it is important to ²²preserve the user experience through this process. When picking up an object, ²the ²user receives impulses corresponding to the grip placed on the object. During ²the ²grip time, no feedback is applied. When the object is released or dropped from ²the ²users' virtual grip, a reduced impulse is used to indicate that the object has ²been ²released.²

[0043] In many cases, the feedback follows the part of the hand to which the ²impulse was directed since it is modelling the vibration of and subsequent ²haptic ²effect on the skin at the point of contact. The feeling on contact can be ²accentuated, potentially using a decaying amplitude on the point of contact on ²the ²hand, to create a more persuasive effect and overcome the short duration of ²each ²impulse. The duration of the impulse may also be driven by data from the ²physics ²engine.²

[0044] III. SPATIO-TEMPORALLY MODULATED SOUND ²- ii-²Date Recue / Date Received 2023-12-04²<DP=12>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²

[0045] A. Sound from ultrasound via an alternate mechanism²

[0046] The noise generated from certain haptic-based interactions is often an ²unwanted side effect of the haptic activity. But it is possible to use the ²property of ²noise generation to produce desired sound effects. Precisely controlling ²disturbances in haptic systems that generate low frequency noise provides the ²capacity to generate broadband signals by spatially moving the focusing ²points. ²This enables the creation of sound from ultrasound using an entirely different ²²mechanism from existing parametric audio applications. Rather than modulating ²a point or beam in amplitude, a control point may instead be moved in a ²trajectory ²that encodes the waveform. This turns the control point into an audio source.²

[0047] B. Plotting spatial trajectories ²

[0048] Turning to Figure 2, shown is a representation 200 of a focus point 210 ²²spinning in the air via path 220, which generates a sound wave 230 each time ²it ²moves along the path 220. The motion can be modified to reproduce a sound that ²²is audible to the human ear in a localized way without amplitude modulating ²the ²focusing system.²

[0049] The easiest way to generate these trajectories is to take an existing ²sound ²source and reinterpret it as a linear path along which a focus point moves to ²create ²a sound wave. At each point in time, the level in a digitally sampled sound ²channel may be interpreted as a parameterized position along this path that ²the ²focus is to be created. By creating foci at these positions at the sampled ²time ²offsets, an audible sound can be created along a 1D open curve path.²

[0050] Alternatively, such an existing digitally sampled sound source may be ²decomposed into in-phase and quadrature components, that is into the real and ²imaginary parts of a complex signal that describes the audio. This is not ²trivial for ²an audio source since it is wideband. Thus, the most effective way to achieve ²the ²desired result is to apply a Fourier transform to the signal and apply a n / 2 ²phase ²offset. The standard approaches concerning windowing, blocking of the signal ²and ²transitions in phase may also be used. This generates the imaginary component ²of²- 12 -²Date Recue / Date Received 2023-12-04²<DP=13>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²the sampled audio signal. Since the signal may be represented as a complex ²value, ²the signal may be mapped onto a 1D closed curve, or a 2D circle, plane or ²surface. ²Moving the point around or remapping these 2D coordinates into the space will ²produce sound waves, thereby recreating this audio signal. In this way, the ²audio ²signal may be generated in a 2D area.²LOOM] This effect may also be used in a third dimension to provide the phase ²cues ²that govern stereo audio (potentially with a single sound source) for systems ²that ²cannot create strong foci in disparate positions. Similarly, for multiple ²focus ²points, multiple sound sources may be synthesized. This enables multi-channel ²audio that can be created from focus points in the air. These focus points may ²also ²provide haptic feedback.²

[0052] IV. GENERATIVE HAPTICS FROM AUDIO EXEMPLARS ²

[0053] Control points may be changed in amplitude many thousands of times a ²second, although only low frequencies can be detected as haptics. Through non-²linearity effects, higher frequencies can be reproduced. Updating the control ²points quickly yields the ability to produce many frequencies at once, giving ²the ²ability to reproduce audio in some sense. However, audio does not have ²information in frequency ranges that are haptically effective.²

[0054] Designing haptic effects in the air is challenging as due to the ²sensitivity of ²the skin only some bands of frequencies are available for haptic use. Due to ²this, ²effects can be created that create pleasurable sounds from the array that do ²not ²convey haptics and conversely haptic effects that do not sound pleasing. Due ²to ²these reasons, an easier approach to designing mid-air haptic effects would be ²²valuable.²

[0055] Creating a haptic sensation using a sound as input may be achieved in a ²²number of ways. The frequencies of the sound that are in the haptic range may ²be ²amplified until a haptic threshold is reached. Alternatively, the initial ²sound may ²have frequencies that are haptically active added to it. The original sound ²may be ²analyzed and a profile constructed. This profile may then be used to generate ²a²- 13 -²Date Recue / Date Received 2023-12-04²<DP=14>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²haptic effect. This may be constructed using the qualities present in the ²sound, ²allowing the created haptic texture to take on aspects of the exemplar such as ²²speed, rhythm or texture. The sound may be processed into a haptic effect ²using a ²variety of processing mechanisms. These mechanisms may be configured to ²emphasize certain aspects in the resultant haptic effect while potentially ²ignoring ²others. This can also be achieved by extracting a feature set from the sound ²before ²processing the sound into a haptic effect. A feature set could also contain ²discontinuities inferred from the sound, such as for example a plosive phoneme ²²being interpreted in this way.²

[0056] Specifically, this can be implemented by (1) extracting a feature set ²from a ²sound profile before assigning the plurality of amplitudes to the control ²point over ²a preset time period; and (2) applying a feature set to the plurality of ²amplitudes ²assigned to the control point over a preset time period. Alternatively, it can ²²implemented by (1) extracting a feature set from the plurality of amplitudes ²to the ²control point over a preset time period; and (2) applying the feature set to ²the ²sound profile.²

[0057] A library of haptic effects may be searched and sorted based on the ²extracted feature set alongside similarity metrics in order to provide ²feedback to a ²haptic author. This haptic author may use the sound interface to describe ²basic ²components of a haptic effect, which can then be edited and mixed further. The ²²haptic effect may also be mixed back into the waveform, resulting in a ²combination of audio and haptic effects. This may occur automatically or via a ²²range of processing styles. These may involve accentuating or augmenting ²particular haptic effects in order to stylize the effects. The processing ²styles may ²be concatenated or remixed to create new processing schemes. Labels can be ²attached to these processing mechanisms and haptic effects, such as "buzzy" or ²²"warm". These may be used to adjust existing haptic effects, adjust audio ²inputs ²to be synthesized into haptic effects or search for existing effects that are ²examples of the style described.²

[0058] V. CONCLUSION ²- 14 -²Date Recue / Date Received 2023-12-04²<DP=15>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²

[0059] The various features of the foregoing embodiments may be selected and ²combined to produce numerous variations of improved haptic systems.²

[0060] In the foregoing specification, specific embodiments have been ²described. ²However, one of ordinary skill in the art appreciates that various ²modifications ²and changes can be made without departing from the scope of the invention as ²set ²forth in the claims below. Accordingly, the specification and figures are to ²be ²regarded in an illustrative rather than a restrictive sense, and all such ²modifications are intended to be included within the scope of present ²teachings.²

[0061] The benefits, advantages, solutions to problems, and any element(s) ²that ²may cause any benefit, advantage, or solution to occur or become more ²pronounced are not to be construed as a critical, required, or essential ²features or ²elements of any or all the claims. The invention is defined solely by the ²appended ²claims including any amendments made during the pendency of this application ²and all equivalents of those claims as issued.²

[0062] Moreover, in this document, relational terms such as first and second, ²top ²and bottom, and the like may be used solely to distinguish one entity or ²action ²from another entity or action without necessarily requiring or implying any ²actual ²such relationship or order between such entities or actions. The terms²"comprises," "comprising," "has", "having," "includes", "including," ²"contains", ²"containing" or any other variation thereof, are intended to cover a non-²exclusive ²inclusion, such that a process, method, article, or apparatus that comprises, ²has, ²includes, contains a list of elements does not include only those elements but ²may ²include other elements not expressly listed or inherent to such process, ²method, ²article, or apparatus. An element proceeded by "comprises ...a", "has ...a", ²"includes ...a", "contains ...a" does not, without more constraints, preclude ²the ²existence of additional identical elements in the process, method, article, or ²²apparatus that comprises, has, includes, contains the element. The terms "a" ²and ²"an" are defined as one or more unless explicitly stated otherwise herein. The ²²terms "substantially", "essentially", "approximately", "about" or any other ²version thereof, are defined as being close to as understood by one of ²ordinary²- 15 -²Date Recue / Date Received 2023-12-04²<DP=16>²CA 03033789 2019-02-13²WO 2018 / 024788 ²PCT / EP2017 / 069569²skill in the art. The term "coupled" as used herein is defined as connected, ²although not necessarily directly and not necessarily mechanically. A device ²or ²structure that is "configured" in a certain way is configured in at least that ²way, ²but may also be configured in ways that are not listed.²

[0063] The Abstract of the Disclosure is provided to allow the reader to ²quickly ²ascertain the nature of the technical disclosure. It is submitted with the ²understanding that it will not be used to interpret or limit the scope or ²meaning of ²the claims. In addition, in the foregoing Detailed Description, it can be seen ²that ²various features are grouped together in various embodiments for the purpose ²of ²streamlining the disclosure. This method of disclosure is not to be ²interpreted as ²reflecting an intention that the claimed embodiments require more features ²than ²are expressly recited in each claim. Rather, as the following claims reflect, ²inventive subject matter lies in less than all features of a single disclosed ²embodiment.²- 16 -²Date Recue / Date Received 2023-12-04²

Claims

<DP=1>²CLAIMS²1. A method comprising:²producing an acoustic field from a ultrasonic transducer array having known ²relative positions²and orientations;²defining an ultrasonic control point, the ultrasonic control point ²representing a spatio-temporally ²modulated focus of ultrasonic energy with an assigned amplitude, wherein the ²ultrasonic control ²point has a known spatial relationship relative to the transducer array;²moving the ultrasonic control point in a trajectory to generate a plurality of ²sound waves, ²wherein the plurality of sounds waves define a sound waveform, wherein moving ²the ultrasonic ²control point generates a haptic effect, wherein moving the ultrasonic control ²point results in the ²sound waveform being audible to a human ear.²2. The method as in claim 1, wherein moving the ultrasonic control point ²results in the sound ²waveform being a localized audio source.²3. The method as in claim 1, further comprising:²receiving a signal, wherein the signal represents a sound source, wherein the ²signal is interpreted²as a linear path along which the ultrasonic control point moves to create the ²sound waveform.²4. The method as in claim 3, further comprising:²digitally sampling and interpreting the signal as a parameterized position ²along the linear path.²5. The method as in claim 3, further comprising:²digitally sampling the signal and decomposing the signal into in-phase and ²quadrature²components.²6. The method as in claim 5, wherein decomposing the signal into in-phase and ²quadrature ²components comprises applying a Fourier transform to the signal and applying ²an / 2 phase ²offset.²7. The method as in claim 5, wherein the signal is mapped onto a one-²dimensional closed curve.²8. The method as in claim 5, wherein the signal is mapped onto a two-²dimensional surface.²9. The method as in claim 1, wherein the trajectory comprises a three-²dimensional path.²10. The method as in claim 1, wherein the trajectory is adjusted to provide ²phase cues that ²govern stereo audio.²11. The method as in claim 10, wherein the phase cues that govern stereo audio ²use a single ²sound source.²12. The method as in claim 1, further comprising:²17²Date Recue / Date Received 2023-12-04²<DP=2>²producing, by the ultrasonic control point, a mid-air haptic effect.²13. The method as in claim 10, further comprising:²defining a second ultrasonic control point, wherein the second ultrasolUc ²control point has ²a known spatial relationship relative to the transducer array;²moving the second ultrasonic control point in a trajectory to generate a ²second plurality of sound ²waves, wherein the second plurality of sounds waves define a second sound ²waveform,²wherein moving the second ultrasonic control point results in the second sound ²waveform being ²audible to a human ear, wherein the second trajectory is adjusted to provide ²second phase cues ²that govern stereo audio.²18²Date Recue / Date Received 2023-12-04²