Humptick Rendering

The haptic rendering device with a patterned auxetic layer and controllable actuators addresses the challenge of providing realistic tactile feedback in VR by smoothly mimicking virtual object shapes, improving user experience through continuous tactile representation.

JP7596314B2Active Publication Date: 2024-12-09MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2021572330
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-06-07
Filing Date
2020-04-29
Publication Date
2024-12-09
Estimated Expiration
2040-04-29

AI Technical Summary

Technical Problem

Existing VR display systems struggle to provide realistic haptic feedback that accurately represents physical interactions with virtual images, often resulting in discontinuities and inaccuracies due to the use of discrete elements like pins or actuators, which can obscure the shape of virtual objects.

Method used

A haptic rendering device utilizing a patterned layer with auxetic behavior, controlled by individually controllable actuators, to smoothly change curvature in two dimensions, mimicking the shape of virtual objects, thereby providing a continuous tactile experience.

Benefits of technology

The device offers a continuous and accurate tactile representation of virtual shapes, reducing surface discontinuities and enhancing the realism of haptic feedback, as demonstrated by user experiments showing high accuracy in shape recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007596314000002
    Figure 0007596314000002
  • Figure 0007596314000003
    Figure 0007596314000003
  • Figure 0007596314000004
    Figure 0007596314000004
Patent Text Reader

Abstract

Examples related to haptic rendering are disclosed. One disclosed example provides a haptic rendering device that includes a patterned layer exhibiting auxetic behavior and a plurality of actuators configured to move the patterned layer, each actuator individually controllable to coordinately vary the curvature of the patterned layer in two dimensions.
Need to check novelty before this filing date? Find Prior Art

Description

[Background technology]

[0001] Virtual reality (VR) display systems immerse a user in a virtual image that occupies the entire field of view. Such display systems can represent the visual and auditory aspects of the virtual experience in a lifelike manner. Summary of the Invention

[0002] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Moreover, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of the present disclosure.

[0003] Examples related to haptic rendering are disclosed. One disclosed example provides a haptic rendering device having a patterned layer exhibiting auxetic behavior and a plurality of actuators configured to move the patterned layer, each actuator being individually controllable to coordinately change the curvature of the patterned layer in two dimensions. [Brief description of the drawings]

[0004] [Figure 1] 1 illustrates aspects of an example display system in which a haptic rendering device simulates haptic interaction with a displayed virtual object.

[0005] [Diagram 2] 1 illustrates an example haptic rendering device in which a patterned layer has been moved to exhibit a convex hill shape.

[0006] [Diagram 3] 3A and 3B illustrate schematic diagrams of an exemplary patterned layer for the haptic rendering device of FIG. 2.

[0007] [Figure 4] 3 illustrates an example actuator assembly for the haptic rendering device of FIG. 2.

[0008] [Diagram 5] 3 illustrates an exemplary socket connection by which the patterned layer of the haptic rendering device of FIG. 2 can be connected to an actuator.

[0009] [Figure 6] 5 illustrates the example actuator assembly of FIG. 4 with an optional constraining plate.

[0010] [Figure 7] 1 illustrates diagrammatically a geometric representation of the relative positions of two actuators.

[0011] [Figure 8] 3 illustrates the example haptic feedback system of FIG. 2, showing a patterned layer configured into a concave bowl shape.

[0012] [Figure 9] 3 illustrates the example haptic feedback system of FIG. 2, showing a patterned layer configured in a saddle shape.

[0013] [Figure 10] 3 illustrates the example haptic feedback system of FIG. 2, showing a patterned layer configured in a concave cylindrical shape.

[0014] [Figure 11] 1 is a flow diagram illustrating an example method for operating a haptic rendering device.

[0015] [Figure 12] We present experimental results depicting participants' classification responses when presented with specific shapes of different curvatures.

[0016] [Figure 13] We present experimental results depicting participants' perceived curvature responses when presented with specific shapes of different curvatures.

[0017] [Figure 14] FIG. 1 is a block diagram illustrating an exemplary computing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Although VR display systems can present visual and auditory stimuli with high fidelity, providing haptic feedback that realistically represents physical interactions with the virtual images is challenging.

[0019] Some display systems may be configured to present force feedback via a grounded robot with which the user interacts. As one example, the display system may include a handle-shaped robot configured to be grasped by the user, controlled by a linkage that moves the robot in six degrees of freedom. As another example, the display system may include a hand-mounted robot, such as a glove device, configured to selectively restrict finger movement and / or actuate fingers to present normal or textured haptics of a surface. However, such mechanisms provide haptic feedback to the user only when the user performs a particular posture (e.g., gripping a handle) or wears another device.

[0020] In a VR experience, the user's view of the surrounding real-world environment is obscured by an opaque display. Thus, the user may not perceive the haptic device during use. Given this, rather than hiding the haptic feedback mechanism within the surface of a wearable device or grounded robot, VR display systems may utilize more complex and / or larger devices to present the haptic experience. This has led to the development of encounter-style haptics, where a user may reach out to touch an object or surface that moves or changes shape.

[0021] Shape-changing devices are robotic systems that attempt to create shapes and surfaces for humans to encounter, touch, and manipulate, and thus may be used to provide encounter-style haptic experiences in VR experiences. Shape-changing devices may also be used for human-computer interaction in AR (Augmented Reality) applications (including MR (Mixed Reality)), as well as various other haptic feedback applications. To represent general forms, shape-changing devices may have a reasonably high number of degrees of freedom and mechanical structures that translate actuation into some form of constrained surface. Such shape-changing devices may be configured to support full-hand interaction, or may be configured in other form factors (e.g., for surface interaction on the thumb or fingers).

[0022] One type of shape-changing device takes the form of an articulated surface, where two-dimensional elements are connected via hinges that fold the articulated surface into different shapes when actuated. Another type of shape-changing device includes stretchable surfaces and in-surface actuators that, when actuated, deform the surface by stretching. Another type of shape-changing device utilizes pin arrays where the pins are actuated to form an approximation of a three-dimensional surface. Each of these shape-changing devices represents a shape through a discrete set of elements. To achieve a suitably high resolution, such devices may include a dense arrangement of discrete elements (hinges, in-surface actuators, pins). However, each discrete element may be controlled by a separate motor, which increases the cost, complexity, and maintenance of the system as it grows in size. Although such devices may adequately convey the overall shape, a user may experience discontinuities in the surface when touching such a device. For example, the user may feel the relief between the pins of a pin array, the hinges between the elements of an articular surface, or the spacing between intra-surface actuators.

[0023] Furthermore, if the shape-changing device represents the shape of a virtual display object (e.g., an elephant, a car, etc.) that is larger than the shape-changing device, a user touching the shape-changing device will be haptically blinded to any portion of the virtual display object that is not being touched. The user may experience other portions of the virtual display object by moving their hand along the shape-changing device. However, discrete elements such as pins that move up / down may create poor cues for sliding the hand along the shape-changing device, as the user feels a stepped transition between the edges of the pins.

[0024] To create the impression of a smoother surface, such a shape-changing device may include an elastic membrane stretched over the discrete elements (pins, hinges, actuators, etc.) as described above. However, the elastic membrane does not change the frequency of the discrete elements that a user may perceive through tactile exploration. Furthermore, the elastic membrane may not be able to accurately interpolate the shape. Instead, the elastic membrane may change direction quickly at each discrete element and stretch linearly to the next discrete element. Furthermore, the elasticity may cause the membrane to deform to the user's finger when the user touches the membrane at a position between the pins (or other discrete elements), which may obscure the shape represented by the shape-changing device.

[0025] In addition, a variety of other shape-changing devices may manipulate the developable surface, which cannot interpolate between a wide variety of surface types.

[0026] Thus, examples disclosed herein relate to haptic rendering devices that can address the above-mentioned problems with shape-changing devices. Briefly, the disclosed examples utilize a shape-shifting layer constructed from a material that exhibits auxetic behavior. This layer may also be referred to herein as an auxetic layer. The auxetic layer bends independently in two directions to provide a variety of different Gaussian curvatures, including both positive and negative Gaussian curvatures, that are not necessarily flexible under forces perpendicular to its surface. While some materials thin in the direction perpendicular to an applied stretching force, auxetic materials retain their width or thicken in the direction perpendicular to an applied stretching force, and as such have a negative or zero Poisson's ratio.

[0027] Instead of using natural auxetic materials, the disclosed examples utilize structural engineering approaches to form patterned layers that exhibit auxetic behavior from non-auxetic materials. The patterned structures can be designed to bend and stretch along two dimensions and animated to create a wide variety of surface shapes. Multiple individually controllable actuators can be arranged to move the patterned layer to coordinately change the curvature of the patterned layer in two dimensions to generate a wide variety of surface shapes. The patterned layer can be further configured to be rigid to touch while being movable by the actuators to change shape. In some examples, the haptic rendering device can include a tactile layer on a surface of the patterned layer to generate a desired touch experience.

[0028] FIG. 1 illustrates aspects of an exemplary display system 100 configured to present a virtual or augmented reality (e.g., mixed reality) experience to a user 102. The display system 100 includes a head-mounted display (HMD) device 104 and a haptic rendering device 106. The HMD device 104 has a near-eye display 108 configured to display a virtual image in the user's field of view (indicated by dotted line 110). In some examples, the near-eye display 108 is opaque, providing a fully immersive virtual reality. In other examples, the near-eye display 108 has a see-through display, allowing real-world and virtual images to be mixed within the user's field of view 110 for an augmented or mixed reality experience. In the HMD device 104, a signal encoding the virtual display image is transmitted to the near-eye display 108 via an on-board computing device 112. The on-board computing device has at least one processor 114 and memory 116, examples of which are described below with reference to FIG. 14. The HMD device 104 further includes speakers that enable the user 102 to experience immersive audio, and may also include other input and / or output devices.

[0029] Utilizing communications components located within the HMD device 104, the computing device 112 may be communicatively coupled to one or more off-board computing devices over a network. Thus, the virtual display image seen by the user 102 may, in some examples, be constructed and / or rendered by an off-board computing device and transmitted wirelessly to the computing device 112. In other examples, the virtual display image may be constructed and rendered on-board.

[0030] The haptic rendering device 106 is configured to provide a haptic experience that can be encountered by the user's hand 118. The haptic rendering device 106 moves the auxetic layer to simulate the shape of the displayed virtual object, thereby mapping the haptic output to the displayed virtual image. In this example, the HMD device 104 displays a virtual dog 120 and also displays a virtual hand 122 that the user 102 controls by the movement of his or her real hand 118. As the user 102 moves his or her hand 118 across the haptic rendering device 106, the haptic rendering device 106 changes shape based on the contour of the virtual dog 120 at the location where the user's hand 118 interacts with the virtual dog 120, thereby "rendering" the shape of the surface of the virtual dog 120. Because the surface that the user 102 interacts with the haptic rendering device 106 is smooth, the user 102 does not feel any discontinuity between the actuators of the haptic rendering device 106 while moving the hand 118 across the device. In some examples, haptic rendering device 106 is configured as a stationary device that a user moves across their hand to experience different surface shapes. In other examples, haptic rendering device 106 can be configured as a movable device, similar to a computer mouse, where a user moves across a real-world surface (e.g., table surface 124) and the surface of haptic rendering device 106 adapts according to the movement. In either case, haptic rendering device 106 can provide a continuous sensation of an object such that the surface sensed by hand 118 is perceived to be a continuation of the same virtual surface.

[0031] The display system 100 may use the coordinate system of the HMD device 104 to calculate the registration between the visual model and the physical model, for example, by measuring points on the surface of the haptic rendering device 106 with corresponding virtual points on the surface. From this, a rigid affine transformation can be calculated to register the displayed visual shape to the rendered physical shape.

[0032] In some examples, the display system 100 may also obtain information regarding the pose (position and / or orientation) and / or movement of a body part interacting with the haptic rendering device 106, which may be used to control the pose and / or movement of the virtual hand 122. In some examples, one or more image sensors (depth and / or two-dimensional) integrated with the HMD device 104 or external to the HMD device 104 may be configured to obtain image data of the haptic rendering device 106 and a body part, such as a hand or fingers, interacting with the haptic rendering device 106. Various classification techniques may be utilized to identify the hand 118 (or other body part) in the image data. Alternatively or additionally, the user may wear an optical and / or inertial tracking device communicatively coupled to the HMD device 104, which may be used to track the pose of the body part. In some examples, the image data may be compared to one or more filters that define known poses and gestures to determine whether the body part is performing a known pose or gesture. Additionally, pixel or voxel locations corresponding to identified body parts in the image data may be compared between image frames and compared to a movement threshold to determine if the body part has moved between image frames. Additionally, haptic rendering device 106 may optionally include an integrated sensor (e.g., one or more capacitive touch sensors) that senses interaction between the body part and haptic rendering device 106. Additionally, haptic rendering device 106 may optionally include an inertial motion sensor that tracks movement of haptic rendering device 106 by a user.

[0033] To modify the shape presented by haptic rendering device 106, object information for one or more virtual objects in the user environment may be maintained by on-board computing device 112, haptic rendering device 106, and / or a remote computer in communication with HMD device 104 or haptic rendering device 106. The object information may include information regarding the position (absolute and / or relative), orientation, and shape of each virtual object in the virtual environment. Based on the pose of body part 118 relative to the virtual display object, haptic rendering device 106 may be controlled to match the curvature of the virtual display object at locations where body part 118 (or virtual body part 122) is predicted to intersect with virtual display object 120 or is likely to intersect with virtual display object 120.

[0034] Haptic rendering device 106 can additionally or alternatively be configured to modify the rendered shape based on movement of haptic rendering device 106 within the environment of use, such as along the surface of table 124 in FIGURE 1. For example, the surface of table 124 can be registered with the displayed terrain, and as user 102 moves haptic rendering device 106 along table surface 124, haptic rendering device 106 can change shape to match the displayed terrain.

[0035] Figure 2 shows an example haptic feedback system 200 having a haptic rendering device 201 that can be used as haptic rendering device 106 of Figure 1. Haptic rendering device 201 has a patterned layer 202 that exhibits auxetic behavior and an electromechanical actuation system 204 configured to impart curvature to patterned layer 202. Patterned layer 202 is designed, via a pattern imparted to the material, to bend and stretch in two directions without conforming to a user's touch.

[0036] The patterned layer 202 is formed from a non-auxetic material that behaves similarly to natural auxetic materials and is therefore structurally designed to exhibit auxetic behavior. Examples of materials suitable for forming the patterned layer 202 include polymeric, metallic, and composite materials that have the toughness and rigidity to withstand pattern-imparting manufacturing processes (e.g., laser jet cutting and / or water jet cutting). Such materials may also be selected to resist sagging when not supported by an actuator. In one specific example, the patterned layer 202 comprises a polycarbonate sheet. Additionally, in some examples, the patterned layer 202 may include a conductive material, which may enable capacitive touch sensing and / or other embedded sensing applications (e.g., pressure sensing for manipulating virtual objects by pressing the patterned layer 202).

[0037] In addition to or in lieu of subtractive processes (machining via water jet, laser cutter, etc.), additive processes may be used to form patterned layer 202. Exemplary additive processes suitable for forming patterned layer 202 include molding, casting, and / or 3D printing. In either example, the pattern is formed through the full thickness of the material forming patterned layer 202 such that the material forming patterned layer 202 remains continuous. This allows patterned layer 202 to bend into nondevelopable shapes. As described in more detail below, the spacing of the mechanical fixtures of electromechanical actuation system 204 may influence the material, size, and pattern selected for patterned layer 202.

[0038] FIG. 3 depicts an example pattern 300 that can be used to provide auxetic behavior to non-auxetic materials (as viewed from the surface 206 where a user interacts with the patterned layer 202 of FIG. 2). The pattern 300 is a repeating spiral pattern, in this example repeating Archimedean spirals 302, which can be used to provide a double curvature to a flat sheet. The space-filling curves of the pattern 300 include connections 304 between the repeating spirals 302, which can provide flexibility to a strong, somewhat inflexible structure (e.g., a 5-10 mm thick polycarbonate sheet). Each spiral 302 occupies a 10 mm square in this example, but can have another size or a range of sizes in other examples. The pattern 300 also preserves most of the surface area through which the pattern is cut, and thus can provide support for an optional tactile layer, which is described in more detail below.

[0039] A variety of patterns other than pattern 300 may be used to form patterned layer 202. Since the pattern provides the auxetic behavior exhibited by patterned layer 202, any pattern or shape that allows for such auxetic behavior may be used. Exemplary patterns include patterns with a significant amount of void space, such as 40-60% void space by volume, which allows patterned layer 202 to bend in two dimensions. Additionally, the pattern includes symmetry in some instances. In any case, through the pattern, the material from which patterned layer 202 is formed acquires flexibility and extensibility not found in unpatterned materials.

[0040] When bent, the patterned layer 202 may raise surface features at the interface between the material forming the patterned layer 202 and the void spaces of the pattern. To prevent the perception of these surface features, the haptic rendering device 201 may have an optional tactile layer on the surface 206 where the user interacts with the haptic rendering device 201, as described above. The tactile layer may have a stretchable and flexible structure that may be continuous or discontinuous. Materials suitable for use as a tactile layer include soft elastomeric materials, cloth materials, and metallic materials (e.g., multiple metal strips disposed on the surface 206). Additionally, in some examples, the tactile layer may include different materials at different locations of the surface 206 to provide different haptic experiences. For example, in a VR / MR / AR application, the user may be instructed to touch different portions of the surface 206 to experience different textures, hardness, etc.

[0041] In some examples, the tactile layer includes a layer of material that is adhered or otherwise attached to the surface 206. In other examples, the tactile layer may be molded, cast, injected, or otherwise formed around the patterned layer 202, thereby filling or partially filling the void spaces. In some examples, forming the tactile layer may include forming the tactile layer on multiple surfaces of the patterned layer. In one specific example, forming the tactile layer includes molding 30A Shore hardness silicone rubber on the surface 206, partially curing the material, and then molding 30A Shore hardness silicone rubber on the surface 208 of the patterned layer 202 opposite the surface 206. Returning to the example shown in FIG. 3, the patterned material 306 may be optically clear, while the filler 308 (e.g., the tactile layer that fills the void spaces) may be diffuse. In other examples, both the patterned material and the optional filler may be optically clear, opaque, and / or have any other suitable appearance. The distinctiveness of the pattern 300 is optionally obfuscated by painting the surfaces (206 and / or 208) of the patterned layer or by an index matching material forming the patterned layer 202 and the tactile layer.

[0042] The type and thickness of material forming the patterned layer 202, as well as the pattern of the patterned layer 202, can be selected to obtain a stiffness that allows the patterned layer 202 to form a variety of shapes while having sufficient stiffness between the points where the actuators contact the patterned layer 202 to interpolate between the areas between the actuators. When a user touches the surface 206, the user can perceive various types of single and double curvatures depending on the movement of the electromechanical actuation system 204.

[0043] FIG. 4 illustrates an exemplary electromechanical actuation system 400 suitable for use as electromechanical actuation system 204. The number k of actuators contacting the underside of the patterned layer (208 in FIG. 2) at k points may be selected to generate a sufficient vocabulary of surfaces for a given application, such that areas of the patterned layer between the k points can physically approximate the surface defined by the actuators. In the illustrated example, electromechanical actuation system 400 includes nine actuators 402a-402i, which allows the haptic rendering device to approximate both convex and concave surfaces in two dimensions. The use of nine actuators may give the palm sufficient expressiveness to distinguish between concave and convex surfaces without moving the hand along the curvature of the surface. In other examples, any other suitable number of actuators may be used.

[0044] Each actuator may have any suitable structure that, when actuated, applies a force to displace the underside (208 in FIG. 2) of the patterned layer. In the illustrated example, the actuation system includes nine linear actuators 402a-i. Although shown fully extended, the linear actuators 402a-i may be controlled to telescope to different heights relative to the frame 404. A central actuator 402e controls the elevation of the patterned layer relative to the frame 404, while the eight peripheral actuators 402a-d and 402f-i allow the patterned layer to be curved convexly and / or concavely. The extension range, speed, maximum force, and / or other parameters of the linear actuators may be selected to help prevent backdrive when the actuation system is stationary. An exemplary linear actuator suitable for use includes the L12-R micro linear servo available from Actuonix of Victoria, BC Canada, which has an extension range of 50 mm, a speed of 6.8 m / sec, and a maximum force of 80 Newtons.

[0045] Each actuator 402a-402i is configured to move in one dimension, for example, by varying the extension length of the actuator. When two neighboring actuators move in opposite directions (e.g., up and down), the actuation stretches an area of ​​the patterned layer between the points where the end 408 of each actuator contacts the patterned layer. In some examples, to reduce stress in the patterned layer, one or more actuators may be connected at a first end to a frame via a mechanical coupling 406 (e.g., a bracket) that allows the actuator to move relative to the frame 404. Although each linear actuator 402a-402i is controlled to move in one dimension by varying the extension height, the illustrated mechanical coupling 406 allows the actuator to pivot. In other examples, the actuators may be connected to the frame 404 in any other manner. In some examples, different actuators may be connected via different types of mechanisms.

[0046] In addition to the pivot table connection to the frame 404, one or more actuators may be connected to the patterned layer in a non-rigid manner that allows rotation of the actuator about the connection point, which may help to further reduce stress in the patterned layer. 5 shows an example ball-and-socket joint 500 that can be used to connect an end of an actuator connected to the underside 208 of the patterned layer 202. The ball-and-socket joint 500 has a housing 502 attached to the underside 208 of the patterned layer 202 and a ball bearing 504 with internal threads 506 disposed within the housing 502 for attachment to a corresponding threaded end of the actuator. When an actuator is connected to the patterned layer via the ball-and-socket joint 500, the ball-and-socket joint 500 allows the actuator to move relative to one or more other actuators, which may help to reduce stress on the patterned layer as the patterned layer changes shape. In another example, the actuator may be configured to move the patterned layer without being coupled to the patterned layer. For example, the underside of the patterned layer (208 in FIG. 2) may include an indentation configured to grip the opposing end 408 of the actuator at a contact point without being mechanically coupled to the actuator.

[0047] The spacing between actuators of a haptic rendering device can affect the type and amount of shapes that the patterned layer interpolates. In some examples, the spacing between actuator connections to the frame 404 can be narrower than the spacing between actuator connections or contacts to the underside of the patterned layer. This can provide a closer-to-orthogonal angle between each actuator and the patterned layer over a wider range of shapes compared to equal spacing between actuators in the frame and the patterned layer.

[0048] The haptic rendering device may optionally include a mechanical assembly that constrains the movement of the actuators. Such an assembly may also help protect the actuators from damage. FIG. 6 shows an example mechanical assembly in the form of a constraint plate 600 connected to a frame 404 via a support 602. The constraint plate 600 may allow the actuators to actuate the patterned layer within certain ranges (e.g., two degrees of freedom for the peripheral actuators 402a-402d and 402f-402i, and one degree of freedom for the central actuator 402e) while preventing other free motions. In this example, the constraint plate 600 allows the central actuator 402e to move strictly in a straight line and vertical direction and prevents unintended lateral movement. Additionally, the illustrated constraint plate 600 includes slots 604 that allow each peripheral actuator 402a-402d and 402f-402i to rotate in a plane defined by the mechanical couplings 406 that connect the actuators to the frame 404. In this example, the rotation plane of each of the peripheral actuators 402a-402d and 402f-402i passes through a vertical line passing through the center of the central actuator and is rotationally symmetric.

[0049] Simulations for rendering three-dimensional surfaces via a haptic rendering device as disclosed may be based, for example, on geometric proxies that mirror physical structures. In some examples, such simulations may be developed using a game engine. An initial simplifying assumption may be made - i.e., the surface is not extensible along lines that radiate from the center to the actuators - and the control system starts with a (3x3) matrix (or other size matrix, depending on the number of actuators used) of target heights of the points of the surface. In one virtual prototype, these heights are controlled by blending a target set of heights representing simple shapes such as bowls, hills, saddles, and cylinders. The target actuator length may then be determined, for example, as follows: First, the central actuator 402e may be configured to move purely vertically, so the actuator length of the central actuator 402e may simply be the target height. Next, four edge actuators 402b, 402d, 402f, 402h positioned along the edges of the frame can be configured according to the structure shown in FIG. 7, where:

number

[0050] The Gaussian function above is based on the central actuator length C, which is first determined as described above. The lengths of the four corner actuators 402a, 402c, 402g, and 402i can then be determined in a similar manner, where H and A are multiplied by √2. In the example shown in Figure 4, the linear actuators each have a range of 50 mm, and therefore a range of 130 mm to 180 mm in coupling length on the patterned layer.

[0051] In some examples, the haptic rendering device may receive instructions for each actuator to render a particular shape from a remote source. In other examples, the haptic rendering device may receive instructions regarding a shape to be rendered and convert the particular shape into actuator commands to control the multiple actuators to cooperatively render the particular shape. Any suitable control circuitry may be used to control the multiple actuators 402a-402i. In one particular example, the multiple actuators may be individually controlled by a multi-channel servo controller communicatively coupled to a host computer (e.g., an HMD device or other computing device).

[0052] During use, the controller 210 may send commands to update the shape represented by the haptic rendering device within the frame rate (e.g., 90 Hz) of the associated VR, AR, or MR application. The controller may receive and execute control instructions in a trivial timeframe, but the actuators may take longer to move to the next position. Thus, in some cases, a display system (e.g., display system 100) may be configured to predictively set the initial state of the haptic rendering device when a user is observed or predicted to "touch" a new virtual surface. This prediction may be based on the user's gaze direction, for example, as determined via an eye-tracking camera. Once the user makes contact with the virtual object, the haptic rendering device may gradually change shape in response to the user's hand / fingers moving continuously over the virtual object until the user breaks contact. The controller 210 may also be configured to predict changes to the displayed shape based on hand motion vectors, eye tracking, and / or other information.

[0053] In FIG. 2, patterned layer 202 is actuated to represent a hill shape having a positive Gaussian curvature. Examples of other shapes that can be achieved by haptic feedback system 200 are shown in FIGS. 8-10. In FIG. 8, haptic feedback system 200 actuates patterned layer 202 to represent a bowl shape 800, which also has a positive Gaussian curvature. Saddle shape 900, represented in FIG. 9, has a negative Gaussian curvature. In FIG. 10, haptic feedback system 200 actuates patterned layer 202 to represent a cylindrical shape 1000 - a shape that can also be formed by non-auxetic behavior materials. Each of the shapes shown in FIG. 2 and FIGS. 8-10 are achieved with a smaller number of actuator elements (e.g., 9 actuators for approximately 1000 pins) than would be used to achieve shapes of suitably similar resolution using pin arrays. In addition to the cost and reliability improvements offered by reducing the number of actuated elements, the example haptic feedback systems disclosed herein provide a continuous sensation of a surface, and therefore a potentially more realistic touch and feel, than discrete surfaces.

[0054] 11 is a flow diagram illustrating an example method 1100 of operating a haptic rendering device, such as haptic rendering device 201. Method 1100 can be implemented as stored instructions executable by a controller (e.g., controller 210) of the haptic rendering device.

[0055] At 1102, the method 1100 includes receiving information regarding a shape to be rendered by a haptic rendering device. In some examples, receiving the information may include receiving information regarding a shape of a virtual object to be displayed from an HMD device, as shown at 1104. Receiving the information may also include receiving information regarding a hand posture relative to the virtual object to be displayed from the HMD device, as shown at 1106. In other examples, receiving the information may include receiving actuator commands specifying adjustments to be made to one or more actuators.

[0056] Based on the received information, method 1100 includes controlling one or more actuators of the haptic rendering device to move a patterned layer of the haptic rendering device, thereby coordinately varying a curvature of the patterned layer in two dimensions, at 1108. In some examples, the actuators of the haptic rendering device can include linear actuators, and controlling the one or more actuators can include varying an extension length of the linear actuators, as shown at 1110.

[0057] The aspects of the foregoing description should not be understood in a limiting sense, as many variations are possible as well. While the illustrated patterned layer 202 is sized to support full-hand interaction, the haptic rendering device may have a larger or smaller form factor in various examples. In some examples, the haptic rendering device may optionally include a hierarchy of actuators to provide more detailed tactile feedback in one or more portions of the patterned layer, e.g., to provide finer interaction for the fingertips than for the palm. Additionally, the haptic rendering device may optionally include an integrated touch and / or pressure sensor, where one or more actuators may be controlled to change shape in response to sensed touch and / or pressure input.

[0058] Although the illustrated example shows the haptic rendering device as having one changeable surface (patterned layer 202), the haptic rendering device may also be configured as a three-dimensional object, such as a cube or sphere, with multiple moveable surfaces and actuators disposed within the object. As another example, the haptic rendering device may include two opposing patterned layers configured to be held between both hands to simulate the haptic experience of holding or moving both hands around a virtual object. Additionally, the haptic rendering devices described herein may be components of another construct, such as a robotic arm that follows the movements of a user's hands within an environment of use.

[0059] Experimental Results

[0060] The haptic feedback system 200 was tested by examining a user's ability to explore and perceive the curvature of an object.

[0061] Experiment 1

[0062] To verify the ability of the haptic rendering device 201 to render different types of objects, a study was conducted in which 17 participants (n=17) selected one of four shapes: a convex hill as shown in FIG. 2, a convex tube, a concave bowl as shown in FIG. 8, and a concave cylinder as shown in FIG. 10. Three different curvature levels of the patterned layer 202 were tested: radius of curvature (r)=25 mm, 10 mm, and 5 mm. Participants had an overall accuracy of 80% when the haptic rendering device 201 presented a curvature of r=25 mm or greater, with accuracy dropping to 42% when the curvature was r=5 mm.

[0063] Paired Friedman analysis showed that not all tested shapes were equally recognized (χ 2 =13.6, df= 3, p=0.033). Referring to FIG. 12, the most recognizable surface was the convex hill, with 75% accuracy across all curvatures, reaching 80% accuracy for curvatures greater than 10 mm. The convex hill shape was significantly more recognizable than the two concave shapes, as shown by a Conover test with Bonferroni adjusted p<0.001. No significant difference in recognition was found between the convex hill and the convex cylinder (p=0.09). The convex cylinder also showed a high accuracy score of 66%, maximizing at a curvature of 25 mm with 92% recognition. These results confirm the ability of the haptic rendering device as a proxy for the haptic perception of shapes of different curvatures. Shapes exhibiting convexity were particularly well recognized, which may in part be related to the anatomical shape of the human hand.

[0064] Experiment 2

[0065] Experiment 2 sought to better understand the resolution of the haptic rendering device 201. Participants underwent forced-pair trials within a VR environment, where each user selected one of two shapes that they perceived to exhibit a higher curvature. To avoid other types of errors unrelated to resolution, a convex hill shape (the shape most accurately identified in Experiment 1) was selected for this experiment. Participants were presented with convex hilltop shapes with different heights. The lowest height experienced was a radius of curvature of 5.5 mm, and the most significant height was a radius of curvature of 25 mm. Completely flat shapes were not presented to avoid bias in participants' evaluations.

[0066] In each iteration of the paired comparison, one of the curvatures was always the lowest height (5.5 mm). The assignment of which curve was presented the first or second time was randomized. If the participant correctly selected the shape with the greater curvature, the experiment decreased the curve level for the next paired comparison. If the participant made a mistake, the experiment increased the curvature by three levels. Participants could make three mistakes in total. The level at which the participant made an error was considered the participant's perceptual threshold.

[0067] The experiment included three different conditions to determine which shape exhibited higher curvature. Given the size of the haptic rendering device 201, participants could decide to touch the surface with a single finger, multiple fingers, or the entire palm. To test this variability, participants performed curvature assessments under the following exploration conditions, counterbalanced with a Latin square: one finger - participants performed the assessment after touching the surface with only the index finger; two fingers - participants used the index and middle fingers to touch and assess the curvature of the surface; whole hand - participants explored the surface with the entire palm.

[0068] Hand position was tracked using a wireless tracking device attached to the wrist and tracked by the virtual reality HMD device. Finger-level tracking was not included to reduce tracking mismatch errors and to allow hand exploration without interference from the haptic rendering device. Participants viewed a virtual hand that matched the wrist-level movement condition.

[0069] The average threshold reached by each participant over the three trials was calculated. The results are shown in Figure 13. A non-parametric paired Conover test was also performed. Searching with two fingers resulted in poorer detection of curvature than using one finger or the whole hand (paired Conover test, p=0.022). No significant difference in accuracy was found between one finger and one hand (p=1). Threshold accuracy was combined with a study of confidence. Given the experimental design, it is assumed that the longer it takes participants to reach threshold, the more confident they are that the threshold is the actual threshold. Using a non-parametric paired Conover test, we found that the whole hand was the most confident search form (vs. single finger p=0.04, vs. two fingers p=0.002), but participants were significantly less confident in their threshold with one or two fingers.

[0070] Visual control experiment

[0071] A control experiment paired with Experiment 2 was conducted to ensure that participants were unable to establish the curvature or shape of the shapes presented in Experiments 1 and 2 based on the visual shapes depicted in virtual reality.

[0072] In this control experiment, participants were not allowed to touch the surface and were required to make the assessment based only on visual cues. The VR setup and rendering of the shape display was designed through several iterations and pilots to visually obscure the shape curvature as much as possible. Therefore, the assessment during the task needs to be based on haptic input, not visual. This control condition explains this assumption and validates that in previous experiments, the assessment was based only on the haptic experience of touching the haptic rendering device.

[0073] Considering the nonparametric nature of paired tests, paired Friedman analyses were performed. Significant differences were observed between conditions (χ 2 = 12.9, df = 3, p = 0.004). Results showed that visual assessment was not sufficient to complete the curvature task (Experiment 2). When using only visual cues, thresholds were reached at only 15 ± 5 sd (standard deviation) mm of curvature, far from the results obtained when participants were allowed to explore the curvature on the whole hand (9 ± 2 sd mm), a single finger (8.5 ± 1 sd mm) and two fingers (9.5 ± 3 sd mm) (paired Conover test with Bonferroni adjustment p < 0.002). Participants reached an average of 1.5 counts mean threshold.

[0074] Unlike other shape-changing devices, the examples described herein showed smooth representation of shapes and transitions between shapes, rather than high-frequency displacements or abrupt changes in curvature. Furthermore, surface orientation affects curvature perception. The disclosed examples present curvatures locally accurately because the patterned layer smoothly interpolates between actuation positions. The above experiments may show that bi-directional curvature of the haptic rendering device is detectable even for small curvatures. These results may also show that the eye and hand are better at determining smaller shape changes than the eye alone, indicating potential use in various simulation applications that utilize haptic feedback.

[0075] In some embodiments, the methods and processes described herein may be coupled to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer application program or service, an application programming interface (API), a library, and / or other computer program product.

[0076] 14 diagrammatically illustrates a non-limiting embodiment of a computing system 1400 capable of implementing one or more of the methods and processes described above. The computing system 1400 is illustrated in simplified form. The computing system 1400 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices.

[0077] The computing system 1400 includes a logic machine 1402 and a memory machine 1404. The computing system 1400 may optionally include a display subsystem 1406, an input subsystem 1408, a communication subsystem 1410, and / or other components not shown in FIG.

[0078] The logical machine 1402 includes one or more physical devices configured to execute instructions. For example, the logical machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical structures. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.

[0079] The logical machine 1402 may include one or more processors configured to execute software instructions. Additionally or alternatively, the logical machine may include one or more hardware or firmware logical machines configured to execute hardware or firmware instructions. The processors of the logical machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Individual components of the logical machine may optionally be distributed among two or more separate devices, which may be remotely located and / or configured for cooperative processing. Aspects of the logical machine may be virtualized and executed by remotely accessible networked computing devices arranged in a cloud computing arrangement.

[0080] The storage machine 1404 includes one or more physical devices configured to hold instructions executable by a logical machine to implement the methods and processes described herein. When such methods and processes are implemented, the state of the storage machine 1404 may be transformed, for example, to hold different data.

[0081] The storage machine 1404 may include removable and / or built-in devices. The storage machine 1404 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic memory (e.g., hard disk drives, floppy disk drives, tape drives, MRAM, etc.), among others. The storage machine 1404 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, position addressable, file addressable, and / or content addressable devices.

[0082] It will be appreciated that storage machine 1404 may include one or more physical devices, however, aspects of the instructions described herein may alternatively be propagated by a communication medium (e.g., electromagnetic signals, optical signals, etc.) that is not retained by a physical device for a finite period of time.

[0083] Aspects of the logic machine 1402 and the storage machine 1404 may be integrated together into one or more hardware logic components, which may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASICs / ASICs), program and application specific standard products (PSSPs / ASSPs), systems on a chip (SOCs), and complex programmable logic devices (CPLDs).

[0084] The terms "program" and "engine" may be used to describe aspects of the computing system 1400 that are implemented to perform a particular function. In some cases, a program or engine may be instantiated via the logic machine 1402 executing instructions held by the storage machine 1404. It will be understood that different programs and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same program and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "program" and "engine" may include individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

[0085] As used herein, a "service" will be understood to be an application program capable of running across multiple user sessions. A service may be available to one or more system components, programs, and / or other services. In some implementations, a service may run on one or more server computing devices.

[0086] If included, the display subsystem 1406 may be used to present a visual representation of the data maintained by the storage machine 1404. This visual representation may take the form of a graphical user interface (GUI). As the methods and processes described herein change the data maintained by the storage machine and thus transform the state of the storage machine, the state of the display subsystem 1406 may likewise be transformed to visually represent the changes in the underlying data. The display subsystem 1406 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the logic machine 1402 and / or the storage machine 1404 in a shared enclosure or such display devices may be peripheral display devices.

[0087] If included, the input subsystem 1408 may have or interface with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may have or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and conversion and / or processing of input actions may be handled on-board or off-board. Exemplary NUI components may include microphones for speech and / or voice recognition; infrared, color, stereoscopic, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field detection components for assessing brain activity.

[0088] If included, the communications subsystem 1410 may be configured to communicatively couple the computing system 1400 with one or more other computing devices. The communications subsystem 1410 may include wired and / or wireless communication devices compatible with one or more different communications protocols. As a non-limiting example, the communications subsystem may be configured to communicate over a wireless telephone network, or a wired or wireless local or wide area network. In some embodiments, the communications subsystem may enable the computing system 1400 to send and receive messages to and / or from other devices over a network, such as the Internet.

[0089] Another example provides a haptic rendering device having a patterned layer exhibiting auxetic behavior and a plurality of actuators configured to move the patterned layer, each actuator being individually controllable to cooperatively change the curvature of the patterned layer in two dimensions. In such examples, the patterned layer may additionally or alternatively have an Archimedes spiral pattern. In such examples, the patterned layer may additionally or alternatively include a metallic material, a polymeric material, or a composite material. In such examples, the patterned layer may additionally or alternatively include 40-60% void space by volume. In such examples, the haptic rendering device may additionally or alternatively include an elastomeric material that fills the void space. In such examples, the plurality of actuators may additionally or alternatively include a central actuator configured to move the patterned layer at a center of the patterned layer and may additionally or alternatively include one or more actuators configured to move the patterned layer around a periphery of the patterned layer. In such examples, the plurality of actuators may additionally or alternatively include a plurality of linear actuators. In such examples, the one or more actuators may each additionally or alternatively be attached at a first end to the patterned layer via a ball-and-socket joint. In such examples, the haptic rendering device may additionally or alternatively have a tactile layer attached to the patterned layer at a surface where a user interacts with the haptic rendering device.

[0090] Another example provides a haptic feedback system having a patterned layer exhibiting auxetic behavior, a haptic rendering device having a plurality of actuators configured to move the patterned layer, each actuator being individually controllable, and a haptic layer attached to the patterned layer, and a controller configured to selectively control the actuators to move the patterned layer and impart curvature to the patterned layer. In such examples, the controller may additionally or alternatively be configured to receive control instructions from a remote computing device and control the actuators based on the control instructions. In such examples, the control instructions may additionally or alternatively include information regarding a shape of an object to be rendered by the haptic rendering device. In such examples, the patterned layer may additionally or alternatively be formed from a non-auxetic material. In such examples, the non-auxetic material may additionally or alternatively include a metallic material, a polymeric material, or a composite material. In such examples, the patterned layer may additionally or alternatively include 40-60% void space, and the haptic layer may additionally or alternatively include an elastomeric material that fills the void space. In such an example, the tactile layer may additionally or alternatively include a fabric material attached to the patterned layer at the surface where a user interacts with a haptic rendering device.

[0091] Another example provides a method of operating a haptic rendering device including a patterned layer exhibiting auxetic behavior and having multiple actuators individually controllable to cooperatively vary a curvature of the patterned layer, the method including receiving information regarding a shape to be rendered by the haptic rendering device and controlling one or more actuators of the haptic rendering device to move the patterned layer based on the received information, thereby varying the curvature of the patterned layer in two dimensions. In such an example, receiving information may additionally or alternatively include receiving information regarding a shape of a displayed virtual object from a head mounted display device. In such an example, receiving information may additionally or alternatively include receiving information regarding a hand posture relative to a displayed virtual object from a head mounted display device. In such an example, controlling the one or more actuators may additionally or alternatively include, for each of the one or more linear actuators, varying an extension length of the linear actuator.

[0092] It will be understood that the configurations and / or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, the various acts illustrated and / or described may be performed in the sequence illustrated and / or described, in other sequences, in parallel, or omitted. Similarly, the order of the processes described above may be altered.

[0093] The subject matter of the present disclosure includes all novel and non-obvious combinations and subcombinations of the various process, system and configurations, and other features, functions, acts, and / or properties disclosed herein, and any and all equivalents thereof.

Claims

1. a patterned layer exhibiting auxetic behavior, said patterned layer being formed from a non-auxetic material, said patterned layer having a pattern formed by using at least one of a subtractive process and an additive process; a plurality of actuators configured to move the patterned layer, each actuator being individually controllable to coordinately vary a curvature of the patterned layer in two dimensions; Haptic rendering device.

2. the patterned layer having an Archimedes spiral pattern; The haptic rendering device of claim 1 .

3. The non-auxetic material comprises a metallic material, a polymeric material, or a composite material; The haptic rendering device of claim 1 .

4. the patterned layer comprises 40-60% void space by volume; The haptic rendering device of claim 1 .

5. further comprising an elastomeric material filling said void space. The haptic rendering device of claim 4 .

6. the plurality of actuators includes a central actuator configured to move the patterned layer at a center of the patterned layer and also includes one or more actuators configured to move the patterned layer around a periphery of the patterned layer. The haptic rendering device of claim 1 .

7. the plurality of actuators includes a plurality of linear actuators; The haptic rendering device of claim 1 .

8. one or more of the actuators are each attached at a first end to the patterned layer via a ball and socket joint; The haptic rendering device of claim 1 .

9. a tactile layer attached to the patterned layer at a surface where a user interacts with the haptic rendering device; The haptic rendering device of claim 1 .

10. 1. A haptic rendering device comprising: a patterned layer exhibiting auxetic behavior, said patterned layer being formed from a non-auxetic material, said patterned layer having a pattern formed by using at least one of a subtractive process and an additive process; a plurality of actuators configured to move the patterned layer, each actuator being individually controllable; a tactile layer attached to the patterned layer; a haptic rendering device; a controller configured to selectively control the actuators to move the patterned layer and impart a curvature to the patterned layer. Haptic feedback system.

11. The controller is further configured to receive a control command from a remote computing device and control the actuator based on the control command. The haptic feedback system of claim 10.

12. the control instructions include information regarding a shape of an object to be rendered by the haptic rendering device; The haptic feedback system of claim 11.

13. The non-auxetic material comprises a metallic material, a polymeric material, or a composite material; The haptic feedback system of claim 10.

14. the patterned layer comprises 40-60% void space by volume, and the tactile layer comprises an elastomeric material filling the void space. The haptic feedback system of claim 10.

15. the tactile layer comprises a fabric material attached to the patterned layer at a surface where a user interacts with the haptic rendering device; The haptic feedback system of claim 10.

16. 1. A method of operating a haptic rendering device having a patterned layer exhibiting auxetic behavior and having a plurality of actuators individually controllable to coordinately vary a curvature of the patterned layer, the method comprising: receiving information regarding a shape to be rendered by the haptic rendering device; and controlling one or more of the actuators of the haptic rendering device to move the patterned layer based on the received information, thereby varying a curvature of the patterned layer in two dimensions, wherein the patterned layer is formed from a non-auxetic material, and the patterned layer has a pattern formed by using at least one of a subtractive process and an additive process. method.

17. receiving the information includes receiving information from a head mounted display device regarding a shape of a displayed virtual object; 17. The method of claim 16.

18. receiving the information includes receiving information from a head mounted display device regarding a hand pose relative to a displayed virtual object; 17. The method of claim 16.

19. and controlling the one or more actuators includes, for each of the one or more linear actuators, varying an extension length of the linear actuator.

17. The method of claim 16.