Hand sensation mapping

Abstract

An improved low-fidelity single-plane-based haptic rendering that affect the sensation designed by a sensation editor (sketch-based UI) tool is described by generating a dynamic mapping to redirect the haptic rendering over a 3D hand model in real-time. The four solutions comprise (1) the generation of a bijective mapping between the template (2D) and hand spaces (3D), (2) a direct skinning approach for sensation relocation in 3D, (3) a direct 3D to 3D mesh mapping, and (4) a smooth-blend skinning directly to the sensation points. This allows a playful action which continuously provides feedback to the user as they progress along their actions - in this case the object becoming "squished" as they select it, and "unsquished" as they summon it.

Description

Docket Number 1133.0498 [137-US]HANDSENSATIONMAPPINGPRIOR APPLICATIONS

[0001] This application claims the benefit of the following application, which is incorporatedby references in its entirety:

[0002] U.S. Provisional Patent Application No. 63 / 669,402, filed on July 10, 2024,FIELD OF THEDISCLOSURE

[0003] The present disclosure relates for improving low-fidelity single-plane-based hapticrendering issues that affect the sensation designed by a sensation editor (sketch-based UI) tool BACKGROUND

[0004] Currently, designing mid-air haptic sensations can be done in two main ways:

[0005] 1. By manually adjusting the settings to generate a haptic pattern.

[0006] 2. By drawing a 2D sketch inside a UI in which the user would define not only the setof positions but intensity variations along the pattern in a simple visual manner.

[0007] The haptic sensation design using manually generated settings is highly complex as itrequires the user to master different skills from programming to physics. The UI sketch-basedapproach is an easier way for naïve users to design haptic patterns [1], and therefore, the bestoption among these two for the customers of this technology.

[0008] However, this type of tool commonly uses 2D templates limiting the rendering tocustom haptic sensations in 2D space, and therefore, the design of 3D sensations is not possible. This means that a sensation that is rendered on the user’s hand, assumes a 2D plane (which contains the designed sensation in 2D), and anchors the target location on the hand through a fixation point (e.g., palm’s center).

[0009] For example, assuming a circular haptic sensation that targets the fingers with thecenter on the palm, the sensation will be perfectly displayed when the user has an open handpose. However, performing a fist hand pose during the sensation rendering will move all the fingers out of the sensation plane, causing the haptic rendering to miss the target fingers.Similarly, assuming a sensation anchored on the palm center that moves up towards thefingers, the abduction / adduction deviation of the fingers is not supported. This means that asimple haptic line linking the palm center to the index fingertip will often miss the finger.

[0010] Current method issues include:

[0011] 1. The current 2D rendering method is constraining the sensation size to local areas(finger’s phalanges or palm area). That is, it does not allow the correct display of sensationsmoving across fingers or across fingers and palm when the hand is not fully open or when thefingers are moving.

[0012] 2. There is a rendering mismatch between the designed and perceived sensation due tothe unrealistic representation of the rendering plane using the 2D template. The current 2Dperspective of the hand does not accurately represent how the sensation will be displayed onthe real hand increasing pattern re-design and increasing overall design time (probably impacting the user’s experience).

[0013] 3. There is a lack of support for designing haptics feedback on body parts beyond thepalm (e.g. back of the hand, knuckle, forearm or face).

[0014] The current method to overcome the low-fidelity plane-based haptic rendering issuesincludes:

[0015] 1. Anchoring the sensation design to a specific location on the hand using fixationpoints (e.g. fingertip, palm etc.).

[0016] 2. Limiting the sensation design complexity and size. This is to keep the sensation asclose as possible to the anchor point and inside the anchored area.

[0017] This solution can work for sensation designs whose sizes fit into the anchored area,for instance:

[0018] 1. Display a sequence of pulses on the index fingertip.

[0019] 2. Display haptic patterns on the palm. This is the area most commonly used as it is arelatively big flat area that can handle relatively large patterns.

[0020] However, this solution does not cover interdigital sensation designs, nor supportdesigns on knuckles or the back of the hand. This solution also constrains the exploration of more complex haptic patterns.

[0021] This issue limits designers' ability to design what they want to transmit through thehaptic sensation and limits it to what the UI allows them to do. SUMMARY

[0022] This application proposes methods to improve the low-fidelity single-plane-basedhaptic rendering issues that affect the sensation designed by a sensation editor (sketch-basedUI) tool. By generating a dynamic mapping to redirect the haptic rendering over a 3D hand model in real-time. The four solutions comprise (1) the generation of a bijective mapping between the template (2D) and hand spaces (3D), (2) a direct skinning approach for sensation relocation in 3D, (3) a direct 3D to 3D mesh mapping, and (4) a smooth-blend skinning directly to the sensation points.

[0023] This invention differs from previous attempted solutions in that it provides aprogressive way to both select, and summon an object, through the metaphor of Squishing, rather than a discrete action.

[0024] This allows a playful action which continuously provides feedback to the user as theyprogress along their actions - in this case the object becoming “squished” as they select it,and “unsquished” as they summon it, although other visualizations are possible as discussedabove.

[0025] This progressive feedback is especially powerful as it can be paired with posedetection, which, if the user understands the pose, they need to make in order to interact withthe object, intuitively suggests to the user when the object they are interacting with will beselected or summoned. This also allows the entire action to be performed in one fluid motion- moving into, and out of, a pose.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying figures, where like reference numerals refer to identical orfunctionally similar elements throughout the separate views, together with the detailed description below, are incorporated in and form part of the specification, serve to further illustrate embodiments of concepts that include the claimed invention and explain various principles and advantages of those embodiments.

[0027] Figure 1 shows prior examples of haptic rendering method based on a singlerendering plane anchored to the palm center.

[0028] Figure 2 shows sequences of positions.

[0029] Figure 3 shows the areas / planes defined across spaces.

[0030] Figure 4 shows a cylindrical coordinate system and projection.

[0031] Figure 5 shows a 2D texture of a hand wrapped on the 3D hand model

[0032] Figure 6 shows a first 3D model as a haptic template.

[0033] Figure 7 shows a second 3D model as a haptic template.

[0034] Skilled artisans will appreciate that elements in the figures are illustrated forsimplicity and clarity and have not necessarily been drawn to scale. For example, thedimensions of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of embodiments of the present invention.

[0035] The apparatus and method components have been represented where appropriate byconventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein. DETAILED DESCRIPTION

[0036] The methods proposed in this document target to improve the transition / mappingfrom the sensation designed by the sensation editor (sketch-based UI) tool to generate a dynamic haptic rendering over a 3D hand model in real time. Improving the rendering fidelity and extending the available areas used to display haptic sensation on the hand.

[0037] This document considers the following solutions:

[0038] 1. The generation of a bijective mapping between the template (2D) and hand spaces(3D).

[0039] 1.a The generation of the 2D to 3D mapping based on a multi-space approach(Cuboid approximation).

[0040] 1.b Cylinder approximation for sensation relocation in 3D based on multi-spacemapping.

[0041] 2. A direct skinning approach for sensation relocation in 3D.

[0042] 3. A direct 3D to 3D mesh mapping.

[0043] 4. Apply a smooth-blend skinning directly to the sensation points.

[0044] Turning to Figure. 1, shown is a prior art schematic 100 of examples of the currenthaptic rendering method based on a single rendering plane anchored to the palm center. Thefigures on the left show a haptic circle rotating across the fingers in an open hand pose 110and missing the fingers when in a fist hand pose is done 120. The figures on the right shows ahaptic point moving from the palm center to the index fingertip in open 130 and fist-hand 140 poses.

[0045] 1. Solution 1: Generation of a bijective mapping between the template (2D) andhand spaces (3D)

[0046] This mapping is based on a tree of transformation matrices of both spaces. Thisalready allows the transition from a single 2D plane (current default in the sensation editor tool) to a dynamically located multi-space haptic rendering.

[0047] Turning to Figure 2, shown is a schematic 200 of a top sequence of positions of acircle moving from the palm 210, through the proximal 220 and intermediate 230 phalangesof the middle finger using the current sketch-based sensation editor approach (single plane).On the bottom, the same sequence of position using the approach proposed in this document(2D to 3D sensation mapping) to relocate the position of the sensation on the correct targetphalange 240250260.

[0048] 1.a The generation of the 2D to 3D mapping based on a multi-space approach(Cuboid approximation).

[0049] This section describes the method to generate dynamic motion retargeting using themulti-space concept.

[0050] This mathematical description uses the right-hand systems of reference, homogeneouscoordinates (i.e., 3D points in A’s coordinates as p^(^, ^, ^, 1) ∈ ℝ4) and homogeneoustransformation matrices (M^^∈ℝ4^4, to convert coordinates from A to B).

[0051] In the first solution, the haptic sensation rendering on the hand assumes a directcorrespondence between the template space (Template), where the sensation player replicates a sensation pattern from a sensation file in JSON format (from the sensation editor tool); and the Leap Motion space (LeapM), local to the haptic device. All points are mapped from one space to another directly through a transformation matrix like:

[0052] This approach uses a set of volume pairs, one defined in each space, Template andLeapM. Let VTemplate={^Template, ^Template, ^Template, ^Template}⊆Template and VLeapM={^LeapM,^LeapM, ^LeapM, ^LeapM}⊆ LeapM be a volume pair described by the template coordinates andretargeted leap motion coordinates, relative to the Haptic device’s center.

[0053] The transformation matrices allow us to directly map any template point ^Templateinside VTemplateto its analogous Volume VLeapM, by computing its local coordinates in VTemplate and mapping the point to the same coordinates in the equivalent volume VLeapM:

[0054] By using this mapping strategy, the pair {VTemplate, VLeapM} now identifies twoequivalent volumes in Template and LeapM spaces, even if their shape is different. Thus, not only physical vertices { ^Template, ^Template, ^Template, ^Template} are mapped to their equivalent retargeted vertices {^LeapM, ^LeapM, ^LeapM, ^LeapM}. Any other point inside VTemplatecan also be mapped to its equivalent in VLeapM.

[0055] 1.1 Bounding the space: template and Leap Motion trees

[0056] Turning to Figure 3 shown is a schematic 300 of how the areas / planes are definedacross spaces, and how muti-planes and 3D hand areas match. On the left 310 shown is asingle plane template from sketch-based sensation editor approach. In the center 320 shown isa multi-plane approach as proposed in our first solution. On the right 330 shown is a 3Dhand in Leap Motion coordinates.

[0057] Having a proper delimitation of the spaces allows for avoiding distortions whentransitioning across volumes (e.g., phalanges, these distortions can potentially introduceaudible artefacts during the haptic rendering). Then, to build the space partitioning trees(referred to them as tree TemplateTree and tree LeapMTree). Then the boundary of the template and the Leap Motion spaces are defined. Specifically approximated these ascuboids, (15 cuboids to represent phalanges and palm volumes, as in Figure 3).

[0058] This geometry provides a basic structure of the trees.

[0059] LetTemplateTree and VLM^={ p0LM, p1 LM, p2 LM, p3LMdescribe each of the 15 equivalent cuboids in bothspaces. The mapping between boundary points is computed as p^^^^^^^^ ^^^^^^ = ^^^^^^∗ ^^^^^^^^^^, ^∈{0,1,2} (i.e. vertices to define the axis vectors).15 cuboids are used to produce the two basic tree structures for TemplateTree and LeapMTree, with each tree containing15 nodes and each cuboid node VTemplate^ in TemplateTree having an analogous cuboidnode VLeapM^ in LeapMTree.

[0060] Any point ^Template from the current sensation will be inside a unique leaf cuboidnode VTemplate^ in TemplateTree. Thus, point ^Template can be mapped to spaceLeapM using VLeapM^, as in Equation 2.

[0061] 1.b. Cylinder approximation for sensation relocation in 3D based on multi-spacemapping

[0062] This approach makes the sensation move over the 3D hand model surface. It relocatesthe position of the sensation from the multi-space approach to the 3D hand at any given time in world coordinates. This means this method will work with continuous coordinates for the hand model.

[0063] Turning to Figure 4, shown is a schematic 400 with a cylindrical coordinate system410 with an origin O, polar axis A and a longitudinal axis L. The dot is the point, forexample, with the radial distance ρ =4, angular coordinates φ =130 and high z=4 [2].

[0064] This approach takes the local spaces 450 defined for each phalange and palm andcomputes a cylindrical projection 460420430400 for the sensation plane (located at bone level in the 3D model) over the 3D hand model. For instance, it takes the proximal phalange of the index finger and assumes a cylinder and using a cylindrical projection generates a mapbetween the haptic rendering plane (at bone level) and the cylinder surface (the phalangesurface, using the equations 3, 4, 5 and 6).

[0065] In this cylindrical projection method, a generic 2D pixel of an acquired image [u, v],can be projected to a 3D point x = [x, y, z] using a camera’s intrinsic projection parameters -focal length, f, and optical center [cx, cy] like:Where K represents the internal calibration matrix of the camera and refers to the pixel’s depth. This 3D point is projected onto a unit cylinder as follows:

[0066] Similar to the 2D to 3D mapping proposed in 1 (multi-space approach), a bijectivemapping is generated between the haptic rendering plane and the 3D cylinders composing the finger model surface, allowing for smooth haptic displacement over the 3D hand model, extending the limits of the current sensation rendering by allowing wrap the sensation around the finger reaching the sides of the fingers from the same sensation template in JSON formatwhich is not possible with the current single-plane or multi-space approaches.

[0067] 2. Solution 2: Direct skinning approach for sensation relocation (Sensation as atexture)[2]

[0068] Turning to Figure 5, shown is a schematic 500 of a 2D Texture of a hand will begenerated including the sensation path 510 that will be wrapped on the 3D hand modelallowing it to transform from the texture coordinates to the hand model coordinates 520. On the right, the image 530 represents the texture-to-mesh approach.

[0069] This approach generates a 2D texture out of the haptic sensation from the sketch-based sensation editor tool (storing the sensation path data in texture coordinates). These texture coordinates can be used to compute the 3D location of the sensation over time by binding the texture to the 3D hand model interpolating the sensation position over time and directly retrieving 3D hand coordinates based on the UV coordinates of the sensation textureat any given time t (see Figure 5). This means that the solution works with continuouscoordinates for the hand model (local-to-hand). This approach does not require the mapping proposed in the solution 1 (multi-space).

[0070] This approach uses a single 3D point-to-pixel correspondence, which means that eachpixel will be mapped to a single position on a triangle on the hand mesh (avoiding the one-to- many approach commonly used in texture mapping).

[0071] The approach takes a given UV coordinates of a texture point and iterates through thetriangles in the hand mesh until it finds the one that contains the target point, the triangle vertices are interpolated to find the exact 3D point in the mesh. Since mesh coordinates are in local space, it is important to make sure to convert the points to world space by a local-to- world transformation matrix.

[0072] This approach can be embedded into a shader to optimize the per-vertex computationin the 3D position retrieve method.

[0073] Noise consideration with skinning-like approaches (1.a, 2 & 3).

[0074] One concern that may arise from skinning is that the distance between two subsequentpoints of interest in the model space could be stretched greatly in the real space.

[0075] Per prior studies in [3], the maximum hand spread is measured from the outer borderof the tip of the little finger to the outer border tip of the thumb. The fingers and thumb are stretched as widely apart as the person finds comfortable.

[0076] Table 1 shows hand spread data classified by country and sex.TABLE 1: Country / Sex Mean mm SD 5%ile mm 95%ile mm SourceUK M 212.9 18.5 182.4 243.4 PeopleSize 1998UK F 200.2 15.6 174.6 226.9 PeopleSize 1998Japan F 186.5 11.1 168.2 204.7 PeopleSize 1998Sri Lanka M 206 15.19 185 222 Abeysakara &Shahnauvaz 1997 Sri Lanka F 184 15.82 160 210 Abeysakara &Shahnauvaz 1997 US M 213.6 18.8 182.6 244.5 PeopleSize 1998US F 201.1 17.0 173.1 229.0 PeopleSize 1998

[0077] Per prior studies in [3], hand breadth (including the thumb) is measured across thepalm of the hand at the level of the base of the thumb and including the joint at the base of the thumb.

[0078] Table 2 shows hand breadth data classified by country and sex.TABLE 2: Country / Sex Mean mm SD 5%ile mm 95%ile mm SourceUK M 106.8 5.7 97.4 116.2 PeopleSize 1998UK F 91.9 5.6 82.7 101.1 PeopleSize 1998China M 102.8 6.1 92.8 112.9 PeopleSize 1998China F 89.5 5.6 80.2 98.7 PeopleSize 1998Germany M 107 98 116 DNN 1986Germany F 92 82 101 DNN 1986Japan M 105.6 4.3 98.5 112.7 PeopleSize 1998Japan F 89.8 4.9 81.8 97.8 PeopleSize 1998Poland M 95 114 PKN 1988Poland F 82 100 PKN 1988Sri Lanka M 99 6.53 90 110 Abeysakara &Shahnauvaz 1997 Sri Lanka F 89 5.59 80 99 Abeysakara &Shahnauvaz 1997 US M 107.1 5.8 97.6 116.7 PeopleSize 1998US F 92.3 6.1 82.3 102.3 PeopleSize 1998

[0079] Let’s assume the case where a haptic line is drawn from the tip of the thumb to the tipof the pinky finger. The distance of interest thus becomes the breath of the hand in modelspace, the fingers could be close, and that distance would be equivalent to the hand breathincluding the thumb and be on average 106.8 mm for a UK male. The worst-case scenariowould then be if in the real space, the fingers could be open, and that distance would beequivalent to the hand maximum spread and be on average 212.9 mm for a UK male. Theratio between the two distances in model space and real space is slightly greater than 2. Thisratio remains approximately the same across gender and nationality.

[0080] Our first proposition is to limit this ratio by using a model space that is midwaythrough the closed fist and open hand / open finger. Thus, the distance could only beshrink / increase by a factor sqrt(2) = 0.41. To illustrate this with the previous example, this represents rendering a haptic line at 8m / s in model space at f = 8 / 0.1598 = 50Hz (159.8mm would be the midpoint between 106.8mm and 212.9mm) and the two opposite worst case of skinning drawing a line at f = 8 / 0.1068 = 74.9Hz or f = 8 / 0.2129 = 37.47Hz.

[0081] However, this is assuming skinning at the node level (using hap-e notation). Wherethe only two coordinates that are modified are the two extremities of the line. Our secondproposition is to apply skinning on the sample path of the haptics, where the sampling rate isthe one of the array (i.e.40kHz or above).

[0082] Still, with the example of our line, the space between fingers in the model space isabout 53 mm (i.e., (maximum hand spread – hand breath) / 2). Thus approximately 13.25mmbetween each finger. In comparison, the distance between one finger edge to the other isabout 21.36 mm (i.e., hand breadth / 5 fingers). Assuming again an 8m / s line (i.e. 50Hz),there should be a control point every 0.2 mm. If skinning is then applied, to all these points,only the point between the fingers gets stretched out / reduced.

[0083] In the case of a fully open hand, the space between each finger grows from 13.25mmto 26.5mm (max spread – hand breadth) / 4). This represents a factor 2. Thus, the gap betweeneach control point would also grow by a factor of 2, from 0.2 mm to 0.4 mm. Because thewavelength of ultrasound is about 8 mm (for a 40khz ultrasound) the jump betweenconsecutive positions is still far less than that of the focal point size itself and poses little impact on audio noise.

[0084] In the case of a closed hand, the space between each finger would actually shrinkfrom 13.25 to 0. Similarly, the gap between each control point would also shrink to 0. Thus, no impact on noise as the point would be static.

[0085] Thus, using a fully open hand as model space is proposed to remove any audiblenoise.

[0086] 3. Solution 3: Direct 3D to 3D mesh mapping

[0087] Turning to Figure 6, shown is a schematic 600 using a 3D model as a template todesign haptic sensations. This method can store the sensation pattern in model coordinates (directly in 3D) 610 and automatically use them inside a game engine (e.g. Unity) as local toobject coordinates 620.

[0088] The skinning-like approaches proposed in this document translate sensation pointsfrom 2D to 3D coordinates local to the hand model used by the leap motion tracking system to represent the real hand position. A simple method to directly access the 3D model coordinates of any sensation point is by storing the sensation data in mesh coordinates straight from the design stage inside the sketch-based sensation editor tool. By using a 3D model of the hand as a template in the sensation editing tool instead of the 2D template currently used, will allow not only a direct design of the sensation in a 3-dimensional perspective (a more realistic design pattern) but also it allows to store the sensation’s pathdata in model coordinates which can be directly mapped to a 3D hand model in a virtualreality scenario as a local to model coordinates. See Figure 6.

[0089] Turning to Figure 7, shown is a schematic 700 of two methods to handle haptics pathsgoing outside the hand. On the left 710: The lines that go outside the 3D template will bewrapped on the visible side of the 3D model template, where the last intersection point between the haptic line and the 3D template is set as an ending node for the current hapticline. On the right 720: The sections of the haptic lines that go outside the 3D model that wereintentionally defined by users to be part of the haptic pattern are dimmed down on the final haptic rendering to avoid unnecessary ultrasound waves being spread out to the environment.

[0090] The pattern's sections drawn outside the 3D model are not desired in the final hapticrendering. As shown in Figure 7, this document considers 2 main mechanisms in this case. 1)Automatic line endings: the haptic lines that go outside the 3 model will be wrapped on the visible side of the 3D model template, the last intersection point between the line and the 3D model template is set as an ending node for the current haptic line.2) The sections of the haptic lines that go outside the 3D model that were intentionally defined by users to be part of the pattern (e.g. defocusing, hairy skin simulation, passing through the palm / back of the hand) are dimmed down on the final haptic rendering to avoid unnecessary ultrasound waves being spread out to the environment.

[0091] This method does not require any additional mapping but only the translation frommodel coordinates to world coordinates using a standard local-to-world transformation matrix.

[0092] 4. Solution 4: Apply smooth-blend skinning directly to the sensation points

[0093] A further approach to the problem is by using the smooth blend skinning algorithmfrom character animation in computer graphics. This is algorithm is used in computer graphics to bind a mesh-based skin onto a skeleton with discrete bones and joints. The skeleton is represented as a hierarchy of bones, with each discrete joint being considered as the origin of a local transformation at the base of its respective bone. The skin is then created by the artist in the bind pose of the skeleton and in the domain-specific parlance, joint weighting coefficients (or simply weights) are `painted` onto the skin which informs each skin element which joints may modify the skin position. Alternatively, an algorithm may be used to initially create the assignment of weights to the skin, which the artist may then tweak.

[0094] Such an algorithm may involve taking the weights to be a normalized distancefunction either from the joint or bone whose joint represents the base of the hierarchy. Then afunction ^^(^bind) = ^(^) may be used where ^ is the shortest distance to the ^thbase joint (point) or base bone (line segment) in the skeleton to map a general position vector ^bindrepresenting a location in the bind pose space to a function of distance from an element in the skeletal hierarchy. This then creates a bijective mapping between arbitrary points from the 3D space of the bind pose of a skeletal model to the 3D space of the skeletal model when the skeleton of the 3D model is posed. For the general point, this may be represented as:where a ^thpoint is mapped to a weight associated with the ^thhierarchical element. In this disclosure, it is proposed that this algorithm may be repurposed to instead of assigning initial weight values to a mesh skin, instead assign final weighting value to sensation points. Thegeneral position vector representing each sensation point created in the bind pose of theskeletal model ^bindmay then be taken to the final vertex in the 3D space of the posed model ^poseusing the equation:where ^^is the pose transformation associated with the ^thhierarchical element. Analogies may be at this stage made between the character design bind space and a sensation design “bind” space, with the posed space of the skeletally modelled character being equivalent to a skeletally tracked body part onto which a sensation may be projected by applying thisbijective mapping to move from the space in which the sensation was designed to the space in which the sensation is applied to a posed skeletally tracked body part.

[0095] Generation of a bijective mapping between the template (2D) and hand spaces (3D) ->Multi-plane approach: This will allow users of the sensation editor tools (sketch-based UI), to generate sensations for the whole hand (volar and dorsal sides of the hand) while improving accuracy on the haptic sensation presentation over the 3D hand, allowing also, more dynamic / complex sensation designs and overcoming the current rendering size / accuracy. The bounding approach will take care of the possible artefacts when transitioning across volumes (e.g., phalanges) by using continuous coordinates in world space.

[0096] Generation of a bijective mapping between the template (2D) and hand spaces (3D)->Cylinder approximation: This will allow users of the sketch-based sensation editor tool to generate sensations covering not only the hand palm side when using the standard 2D template from the UI, but also reaching the sides of the fingers and the palm side areas as well (which is not possible with the current single-plane or multi-space approaches). Improving also the presentation accuracy of the mid-air haptic stimulation on the hand. This method also uses continuous coordinates in world and local-to-hand spaces.

[0097] Points of Novelty include:

[0098] 1. Direct skinning approach for sensation relocation (Sensation as a texture): Thismethod provides all the benefits of the other solutions proposed in this document without the need for complex mapping computation inside the game engine (e.g., Unity).

[0099] 2. Direct 3D to 3D mesh mapping: This approach can gather the benefits frompreviously proposed methods (multi-space and cylindrical projections) but takes a different path (2D sensation to texture -> texture to hand model). It can also potentially reduce the audible artefacts when rendering sensations that move across areas in the hand by using hand model coordinates directly.

[0100] 3. A smooth-blend skinning directly to the sensation points: This approach combinesthe benefits of “bijective mapping between the template (2D) and hand spaces (3D)” and“Direct skinning approach for sensation relocation” by directly working the sensation patternon the template skin coordinates, but local to the bone structure. Improving not only congruency and accuracy between the stimulus designed and the one displayed, but also reducing computation time by precomputing part of the required transformation for it to work in real-time.

[0101] References

[0102] [1] Hasti Seifi, Sean Chew, Antony James Nascè, William Edward Lowther, WilliamFrier, and Kasper Hornbæk.2023. Feellustrator: A Design Tool for Ultrasound Mid-Air Haptics. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 266, 1–16. https: / / doi.org / 10.1145 / 3544548.3580728

[0103] [2] Pahwa, Ramanpreet Singh, Wei Kiat Leong, Shaohui Foong, Karianto Leman, andMinh N. Do. "Feature-less stitching of cylindrical tunnel." arXiv preprint arXiv:1806.10278 (2018).

[0104] [3] Laura' Peebles and Beverley . “Handbook of adult anthropometric and strengthmeasurements” Norris Institute for Occupational Ergonomics, Department of Manufacturing Engineering and Operations Management, University of Nottingham, University Park, Nottingham, NG72RD.

[0105] Conclusion

[0106] In the foregoing specification, specific embodiments have been described. However,one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the invention as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of present teachings.

[0107] Moreover, in this document, relational terms such as first and second, top and bottom,and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “comprises,” “comprising,” “has”, “having,” “includes”, “including,” “contains”, “containing” or any other variation thereof, are intendedto cover a non-exclusive inclusion, such that a process, method, article, or apparatus thatcomprises, has, includes, contains a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by “comprises …a”, “has …a”, “includes …a”, “contains …a” does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises, has, includes,contains the element. The terms “a” and “an” are defined as one or more unless explicitlystated otherwise herein. The terms “substantially”, “essentially”, “approximately”, “about” or any other version thereof, are defined as being close to as understood by one of ordinary skill in the art. The term “coupled” as used herein is defined as connected, although notnecessarily directly and not necessarily mechanically. A device or structure that is “configured” in a certain way is configured in at least that way but may also be configured in ways that are not listed.

[0108] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain thenature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

Claims

CLAIMS We claim:1 A method comprising:scanning a hand having a palm and a plurality of phalanges;defining a plurality of local spaces for the palm and the plurality of phalanges;for each of the plurality of local spaces, computing a cylindrical projection for a sensationplane over a 3D hand model.

Images