Deployable controller

Abstract

This concept relates to devices that can employ a deployable controller. In one example, the device can include a base assembly configured to affix the device to a non-hand body part of a user. The example can also include an engagement assembly configured to receive haptic input from or deliver haptic output to a hand of the user. The device can also include a deployment assembly that extends from the base assembly to the engagement assembly and is configured to deploy the engagement assembly from a stowed orientation proximate the base assembly to a deployed orientation proximate the hand of the user.

Description

Background Technology

[0001] In real life, humans tend to interact with objects using their hands. They tend to reach out to grab, touch, grasp, manipulate, and release these objects. However, in augmented reality (AR) and / or virtual reality (VR), this fine-grained interaction with virtual objects is often impossible today. For example, AR / VR headsets can track the position of a user's hand but cannot provide haptic feedback to his / her hand.

[0002] Handheld controllers have been developed for AR and VR scenarios to simulate real-world interactions (e.g., providing positional information and / or haptic feedback for the user's hands). Handheld controllers come in various shapes and can perform a range of functions. While most track three-dimensional (3D) motion, simple controllers are designed only for movement and button-based input. More advanced controllers can include complex controls and provide outputs to the user. While most commercial devices only provide vibrational haptic feedback, researchers have demonstrated a variety of handheld controllers that can present texture, shape, grip and squeeze feedback, weight movement, and haptic behavior for two-handed use. While the functionality of these controllers can vary, an unfortunate commonality is that users essentially have to hold them constantly or interrupt the AR / VR experience to put them down when not needed and pick them up when needed.

[0003] Therefore, one problem with handheld controllers is that users must constantly grip them, hindering natural use of other objects in the physical world. This is especially true in VR, where the virtual environment replaces one's perception of the real world, requiring users to frequently use controllers for all virtual interactions. Repeatedly picking up and putting down controllers is both slow and cumbersome when the real world intrudes.

[0004] Another popular type of controller includes glove controllers, but because these are worn, users cannot easily remove them. Glove controllers typically provide dexterous feedback, including pressure and vibration on the user's fingertips. However, glove controllers still restrict movement and hinder the flexibility of using real-world tools or quickly switching to traditional input devices such as keyboards. This concept can solve any of these and / or other problems. Attached Figure Description

[0005] The accompanying drawings illustrate embodiments of this concept. The features of the illustrated embodiments can be more readily understood by referring to the following description in conjunction with the accompanying drawings. Where feasible, the same reference numerals are used in various drawings to indicate the same elements. In some cases, parentheses are used after the reference numerals to distinguish similar elements. The use of reference numerals without associated parentheses is general to the elements. The drawings are not necessarily drawn to scale. In the drawings, the leftmost number of the reference numerals indicates the drawing in which the reference numeral first appears. Using similar reference numerals in different instances in the description and drawings may indicate similar or identical items.

[0006] Figure 1A and Figure 12 An example system is shown that can employ a deployable controller concept consistent with some implementations of this concept.

[0007] Figure 1B , Figure 1C , Figure 2A , Figure 2B , Figures 3A to 3G , Figures 4A to 4B , Figures 5A to 5C , Figures 6A to 6B , Figures 7A to 7C , Figures 8A to 8B , Figures 9A to 9D , Figures 10A to 10B and Figures 11A to 11D An example expandable controller perspective view consistent with some implementations of this concept is shown.

[0008] Figure 13 A schematic diagram related to an exemplary deployable controller consistent with some implementations of this concept is shown.

[0009] Figure 14 A flowchart of an example controller deployment method consistent with some implementations of this concept is shown. Detailed Implementation

[0010] This concept relates to a device including a deployable controller that can be used by a user in various scenarios, including AR and VR scenarios. The deployable controller allows a user to tactilely interact with virtual objects using their hands. The device can be attached to a body part adjacent to the user's hand, such as the forearm. When engagement is required, the deployable controller can unfold from a stored or retracted orientation to an engaged orientation and return to its original position when engagement ceases. Attaching the device to the forearm allows the deployable controller to be fixed in place, applying forces that a strictly handheld controller cannot. Furthermore, storing the deployable controller allows the user to use their hand normally and unimpeded when not using the deployable controller.

[0011] Figure 1AA system 100 consistent with some implementations of this concept is shown. Figures 1A to 1C The aspects described in Figure 1 are collectively described. For illustrative purposes, System 100 is explained relative to a virtual reality use case scenario, but may optionally or additionally be implemented in other use case scenarios. System 100 may include Base Station 102. In some configurations, Base Station 102 may include hardware and / or software for generating and executing virtual reality worlds, including receiving and processing input from User 104, and generating feedback and outputting it to User 104. Base Station 102 may be any computing device, including personal computers (PCs), servers, game consoles, smartphones, tablets, laptops, automobiles, simulators, etc.

[0012] In some embodiments, system 100 may include a head-mounted device 106. The head-mounted device 106 may be, for example, a head-mounted display (HMD) that can receive information related to virtual reality, the real world (e.g., a scene), and / or the user. In some embodiments, the head-mounted device 106 may include one or more sensors ( Figure 1A (Not shown in the image) is used to provide input to base station 102 and / or head-mounted device 106. Sensors may include, for example, accelerometers, gyroscopes, cameras, microphones, etc. Therefore, head-mounted device 106 can detect objects around the user, the position of the user's head, the direction the user's head is facing, whether the user's eyes are open or closed, which direction the user's eyes are looking in, and the position of the user's body parts (such as hands 108). The head-mounted device may have the ability to present data (such as audio and / or visual data) to user 104.

[0013] System 100 may also include a deployable controller device 110. Device 110 may include a base assembly 112, a deployment assembly 114, and a engagement assembly 116 that function as a deployable controller 118. Consistent with this concept, device 110 can be engaged via a user's hand 108 to provide input and / or output to base station 102 and / or head-mounted device 106. The following will describe the relationship with... Figures 3A to 3E Provide a more detailed description of the example.

[0014] Figure 1A The example system configuration is just one of the system configurations considered. For example, another system configuration might require device 110 to work in conjunction with an audio device, such as headphones. For visually impaired users, headphones can provide audio input, and device 110 can provide corresponding tactile input. Systems employing other devices, either individually or in combination, are also considered.

[0015] exist Figure 1A and Figure 1BIn the illustrated example, the base assembly 112 can be configured to secure the device to a non-hand body portion 120 of the user 104. For example, the base assembly 112 can be secured to the user's forearm 122 or upper arm 124. In some cases, the base assembly can be secured to a joint above the body portion engaging the engagement assembly 116 (e.g., towards the torso). For example, in the illustrated configuration, the engagement assembly is configured to be engaged by the user's hand and the base assembly 112 can be secured to the forearm above the wrist.

[0016] Figure 1C The joining component 116 is shown to be divided in half, thus revealing its internal contents. Joining Figure 1C Watch Figure 1A and Figure 1B The engagement assembly 116 can be configured to receive tactile input from the user's hand and / or transmit tactile output to the user's hand. For example, the engagement assembly 116 may include various input devices 126 to detect user input. Examples of input devices may include pressure sensors, force sensors (such as strain gauges 128), capacitive touch sensor electrodes 130, and / or user-activatable switches 132 (e.g., triggers), etc. In this embodiment, there are four capacitive touch sensor electrodes 130 within the engagement assembly 116, which can serve to distinguish different grips. This data can be used to allow the device to predict the user's intention. In this case, one capacitive touch sensor electrode faces the area of ​​the palm that first contacts the hand, then detects when a grip is taken around the middle finger, and the two pads on the thumb can be used as coarse position input devices.

[0017] The engagement assembly 116 may include various output devices 134 (such as microphones, buzzers, voice coil actuators (VCA) 136), surface simulators (such as balloons), and / or heaters / coolers, etc.

[0018] Device 110 may also include various position sensors 138 (such as six-axis (e.g., 6-DOF) sensors), inertial measurement units (IMUs), etc. Position sensors 138 can provide data related to the device's position in 3D space (e.g., x, y, and z coordinates), the device's orientation, rotation, acceleration, etc. Position sensors 138 may be located on multiple components or a single component. For example, a six-axis sensor may be located on both the engagement component 116 and the base component 112. Note that the terms "input device" 126 and "position sensor" 138 are used for their explanatory purposes herein, but these terms may overlap. For example, the listed input devices are often sensors.

[0019] Various device implementations may include other sensors, input devices, and / or output devices. For example, various sensors may be located on the unfolding assembly 114. In another case, various sensors 140 may be located on the base assembly. Some of these sensors 140 may be configured to sense potential physiological aspects of the user. For example, sensors may sense tendons extending from the fingers to the forearm. Information from the sensors may indicate the position of each finger, the movement of the finger, the direction of that movement, forces (such as gripping force), etc. Optionally or additionally, sensor 140 may include a camera (such as an IR depth camera) to provide positional data about the hand / finger. As used herein, the term "finger" may include the thumb.

[0020] Other sensing implementations are considered. For example, device 110 can sense more user input and use that input to inform its tactile behavior. For example, some implementations may integrate finger tracking (e.g., via a self-capacitance array or wearable camera) around the engagement components and can approach the user's palm and fingers during interaction, providing a tactile response to dexterous input. This can also allow sensing torque on a lever, which helps the device simulate gravity and its resistance to heavy objects. These aspects are discussed below in relation to... Figures 7A to 8B It was discussed.

[0021] Device 110 may also include a controller 142 and a power unit 144. In this case, the power unit 144 is manifested as a servo motor 146, but other types of power units (such as other types of motors, pneumatic systems, and / or hydraulic systems) may be used. The servo motor 146 may generate a powered hinge 148 that rotates about a first axis (FA). The controller 142 may receive information from input device 126 and position sensor 138. The controller may control the device, such as the power unit 144, at least in part based on this information. An example of such a device is given below in relation to... Figure 2A and Figure 2B It is described in more detail.

[0022] In some cases, controller 142 may receive additional information, such as virtual data (e.g., data related to virtual objects). Controller 142 may use this additional information, in conjunction with data from input device 126 and touch sensor electrodes (e.g., position sensor) 130, to control device 110. The following is in contrast to... Figure 3A and Figure 3E An example of this is described in more detail. This configuration can provide information about the position of the engagement assembly 116 relative to the base assembly 112, the rotation of the engagement assembly 116 about the base assembly 112, and / or the velocity and / or acceleration of the engagement assembly and / or the base assembly.

[0023] Note that various conductors (shown but not specified) can be used to communicatively couple various components and / or power various components. Optionally, some components of device 110 can employ wireless technologies (such as Bluetooth). TM The device 110 communicates within and / or with other devices (e.g., base station 102 and head-mounted device 106) within the device (e.g., controller 142 and input device 126). The device 110 may also include a battery (shown but not specified) and / or be tethered to another device to receive power. Network sharing can also couple the device to other devices for communication, rather than using wireless technology.

[0024] Figure 2A and 2B Together they illustrate the technique for deploying the engagement assembly 116 for controlling device 110. Figure 2A The joining assembly 116 is shown deployed to the deployment orientation and almost in the deployment orientation. Figure 2B The engagement assembly 116 is shown deployed to and nearly deployed in its storage orientation. In this configuration, the controller 142 can trigger deployment upon detecting a specific user gesture indicating a user command. In this example, the user gesture is either an upward wrist roll 202 or a downward wrist roll 204. In this example, the controller 142 can obtain data from the position sensor 138 on the base assembly 112 and / or the engagement assembly 116. The controller can interpret the data to detect the gesture. The controller can then cause the deployment assembly 114 to deploy the engagement assembly 116 to fulfill the user command. Figure 2A In the example, controller 142 interprets the sensor data as an "unfolding" posture 202 and causes servo motor 146 to move engagement assembly 116 to the engagement orientation. Figure 2B In the example, controller 142 interprets the sensor data as a "store" posture 204 and causes servo motor 146 to move engagement assembly 116 to the store orientation. In other configurations, the controller may interpret the cessation of engagement as a "store" posture. Therefore, if the user completes interaction with the engagement structure and releases it, the sudden lack of pressure and / or other input on the engagement assembly can be interpreted as the user completing engagement and the controller being able to retract it.

[0025] From one perspective, Figure 2A and 2B An example is shown of how user gestures cause the engagement assembly 116 to be deployed or stored. In this example, the device sensors detect... Figure 2A Summoning gestures and Figure 2B The closing or storing gesture in the game. In this case, the summoning gesture is similar to capturing such a gesture. Figure 2AThe image shows a yo-yo. The device's sensors (such as accelerometers) can detect this motion and actively drive the coupling component into the user's hand. In some configurations, the coupling component can provide ergonomics and functionality similar to a typical VR controller (including touch sensing and trigger buttons). In some cases, the device may include a modular design that separates the device's input space from its control and actuation, facilitating different coupling components. In some configurations, the coupling component can be easily replaced to better match user needs or preferences in a specific scenario. Examples of such configurations are described below relative to... Figures 11A to 11D Described.

[0026] Figures 3A to 3G This also illustrates another technique for deploying the engagement assembly 116 for controlling device 110. In this case, Figure 3A , Figure 3E and Figure 3F It shows that it can be done, for example, by... Figure 1A The head-mounted device 106 is designed for user-generated visualizations 300. However, from the perspective of an external observer, Figures 3B to 3D as well as Figure 3G The same scenario is shown.

[0027] In this case, such as Figure 3A As shown, visualization 300 includes a virtual object 302 in the form of a virtual apple 304, which is suspended from a virtual tree 306 in a 3D position. Visualization 300 also includes a representation 308 of the user's hand in a position corresponding to the 3D position of the user's actual hand 108. In this example, as indicated by arrow 310, the user's hand 108 is approaching the virtual apple 304 (e.g., the user is reaching for the virtual apple 304).

[0028] But from the perspective of an external observer Figure 3B and Figure 3A The user's hand 108 is also shown. The user is wearing device 110. Controller 142 can use visual data about the virtual object from the head-mounted device, such as size, shape, mass, 3D position, velocity, acceleration, etc. (e.g., virtual reality data), and data from position sensor 138 as input to determine how to control device 110. Here, the data indicates that the user is reaching for the virtual apple and will likely attempt to grasp it soon (e.g., within a calculable extrapolation time between the distance between the hand and the virtual apple and the velocity of the hand). Controller 142 can use motion trajectories to predict in advance when the user's hand will contact the virtual object 302. This temporal projection combined with existing grasping models can further aid in multi-sensory integration to enhance the illusion of realistic grasping.

[0029] Based on this time prediction / calculation, the controller can make the deploying component 114... Figure 3C The appropriate time and rotational speed or rate are indicated for the engagement assembly 116 to unfold, so that the engagement assembly is in such a state as shown Figure 3D The virtual apple shown is positioned to contact the user's palm.

[0030] But from the user's perspective, Figure 3E It shows the relationship with Figure 3D At the same point in time, the user's hand contacts the engagement component 116 at the time and location at which the user expects to touch the virtual apple.

[0031] Figure 3F The visualization 300 shows what happens when the user subsequently places the virtual apple on the surface and releases it. Figure 3G The same view is shown from the perspective of an external observer. Note that when the user releases... Figure 3F When the user interacts with the virtual apple, they are actually releasing the engagement component 116, which can automatically be lowered and retracted. Therefore, the device 110 can present the visual-tactile sensation of grasping and releasing an object. Existing haptic controllers cannot achieve this.

[0032] Therefore, this series Figures 3A to 3G The illustration shows how device 110 can automatically deploy joining components, at least in part, based on data from a virtual scene, to simulate a real-world feel for the user. The illustrated joining component shape can render different object forces, and the rendering of forces provides a good sense of the object's weight and inertial forces when held in the hand. Further scenarios relating to allowing the user to "throw" or "put down" an apple are described below.

[0033] As shown above, a novel aspect of device 110 is its ability to automatically deploy the engagement component 116, allowing the engagement component to appear visually and physically in the user's hand at an appropriate time during interaction with VR content. Similarly, the engagement component 116 can "disappear" (e.g., be retracted) when the user puts down the VR object (e.g., releases a virtual object) or throws the object. This can be achieved through methods such as... Figure 2A and Figure 2B The user gesture shown can be used to invoke the conversion, and / or can be invoked by, for example, the user gesture shown. Figures 3A to 3G The application shown will be triggered automatically.

[0034] Recalling the above description, device 110 can be secured to (e.g., attached to) a body part different from the body part to which it is engaged. In the illustrated configuration, device 110 is secured to forearm 122 and engagement assembly 116 is engaged by hand 108. As described below, this aspect enables at least two additional features of device 110. First, when not in use, engagement assembly 116 can be stored unobstructed, allowing the user to interact normally with the physical environment. This feature will be discussed below in relation to... Figure 4A and Figure 4B This is under discussion. Secondly, attaching the device to a body part different from the joint allows the device to apply forces to the joint to simulate real-world forces (or at least scaled-down but believable forces), such as those experienced during lifting, throwing, catching, etc. This feature will be discussed below in relation to... Figures 5A to 5C It was discussed.

[0035] Assuming that at this time relative to Figure 3F The user places the virtual Apple 304 and temporarily completes the visualization. At this point, the joining components can automatically retract without obstructing the view, and the user can join the physical object with his / her hand.

[0036] Figure 4A and 4B A device 110 worn by a user is shown. In this case, the base assembly 112 may include a mechanism for securing the device to the user's forearm 122. In this example, the mechanism manifests as one or more straps 400 that work in conjunction with a cuff 401. The straps 400 may extend from one side of the cuff 401 and may wrap around the user's forearm to the other side of the cuff to secure the base assembly 112 to the forearm. Various strap materials can be used. Elastic materials tend to provide comfort while maintaining a secure relationship between the device and the forearm.

[0037] Figure 4A and Figure 4B A device 110 with an engagement component 116 is shown in its retracted orientation. In this configuration, the user is able to perform manual tasks using the hand associated with the device. For example, Figure 4A This illustrates a user typing on keyboard 402 using the same hand 108 that will be engaged in the directional engagement assembly 116. Similarly, Figure 4B The illustration shows a user using this hand to open a door handle 404. The user experiences normal dexterity and normal tactile sensation with this hand. In contrast, with a traditional handheld controller, the user must put the controller down, into a pocket, or take other actions to free their hand. Similarly, if the user is wearing a traditional glove-type controller, they will have to deal with the reduced or altered sensation that occurs when attempting to complete these tasks.

[0038] From one perspective, in terms of the orientation of the retracted component, the unfolded component 114 positions the engaging component 116 relative to the user's forearm, minimizing its interference with the user's freehand movements. This retracted orientation not only allows for freehand interaction but also enables the use of tangible real-world objects such as keyboards, mice, or doorknobs. This can be particularly useful in augmented reality (AR) scenarios where users frequently engage with both virtual and physical objects. This implementation allows the user to easily and quickly unfold the engaging component, use it as a controller relative to virtual content, and then easily and quickly retract it.

[0039] Recall that a key feature of this concept is the ability to render tactile sensations associated with touching, grasping, lifting, throwing, capturing, and / or releasing virtual objects. The first example is in... Figures 5A to 5C It is shown in the middle.

[0040] Figures 5A to 5C This involves scenarios where users use device 110A to pick up virtual objects. In this case... Figure 5A It shows that it can be done, for example, by... Figure 1A The head-mounted device 106 is designed for user-generated visualizations 500. However, from the perspective of an external observer, Figure 5B The same scenario is shown.

[0041] In this case, visualization 500 includes a virtual object 502 in the form of a virtual coffee cup 504 and a representation 506 of the user's hand. Figure 5B As illustrated by arc or line 510, when a user reaches for the virtual coffee cup 504, device 110A unfolds engagement component 116, bringing it synchronously with the virtual object towards the user's hand 108. At the moment the virtual hand collides with the coffee cup, the engagement component contacts the user's palm, ready to accept grasping input. At this point, controller 142 deactivates servo motor 146, allowing the user to passively move the engagement component themselves, enhancing the feeling that the object is "real" in their hand. This simulated coupling between the virtual object and physical feedback creates a convincing sense of acquiring and holding the virtual object. This type of interaction is also more natural than currently available technologies for acquiring virtual objects, which are primarily based on automatically acquiring and placing virtual objects by pressing buttons.

[0042] In some embodiments, device 110 can also simulate the weight of the virtual coffee cup 504 by generating a downward force (represented by line 512) on the engagement assembly 116 when a user attempts to “lift” the virtual coffee cup 504. For example, the unfolding assembly 114 can be driven to rotate counterclockwise relative to the base assembly 112 to generate a downward force in the user's hand that mimics the weight of the virtual coffee cup. Thus, this embodiment can provide two axes of motion between the base assembly and the engagement assembly. First, as relative to... Figure 1A and 1B The deployed assembly discussed may include a power hinge 148 (which can rotatably engage the assembly relative to a first axis (FA), in which case the first axis (FA) is illustrated as extending generally vertically). Secondly, the deployed assembly 114 may also include a rotation mechanism 514 (mostly obscured in this view) to rotate along a path 90 degrees to the first axis and... Figure 5B The second axis (SA), which is typically horizontal and extends into and out of the accompanying drawings, rotates and unfolds relative to the base assembly. Therefore, in this embodiment, the power hinge is a multi-axis power hinge.

[0043] The rotating mechanism 514 can be controlled by the controller 142 to generate a downward force corresponding to an event in the visualization 500. The rotating mechanism 514 may also include a disengagement clutch to prevent the generation of excessive force that could otherwise injure the user or the equipment 110. Note that the power hinge 148 may also include a similar disengagement clutch for similar reasons.

[0044] As described above, when the user reaches for the virtual coffee cup 504, the engagement component 116 can gradually pivot into the user's hand to match the position of the virtual coffee cup. This can create a simulated sensation of touching and grasping the virtual object.

[0045] Figure 5C Additional features introduced relative to Figure 5 are shown. In this case, the rotation mechanism 514 can allow the engagement assembly 116 to unfold along a path that is not in a plane (e.g., a complex path). For example, suppose the engagement assembly needs to engage the user's hand along the long axis (LA) of the user's forearm. However, suppose that in the storage orientation, an angle away from the long axis (such as 20 degrees away from the long axis as indicated by line 516) is determined to be comfortable for the user. In this case, the controller 142 can rotate the rotation mechanism 514 while the unfolding assembly 114 is unfolding or storing the engagement assembly to achieve the desired angle in each orientation. Another example configuration relative to... Figure 6A and Figure 6B Described.

[0046] Figures 5A to 5C An example is provided of attaching the base assembly 112 of the device 110 to a different part of the body than engaging the engagement assembly 116, and how this allows the device to apply force to the engaged body part to simulate real-world forces (such as those experienced when lifting, throwing, catching, etc.). In this case, the base assembly 112 is attached to the forearm and the engagement assembly 116 is engaged by the user's hand.

[0047] Figure 6A and Figure 6BAdditional aspects of device 110 are shown together. This embodiment can employ a single-servo pivot design. The moving component may include a deploying assembly 114, which can actively pivot the engaging assembly 116 about a generally vertical axis (e.g., a first axis (FA)) as shown in 602. Thus, the power hinge 148 of the deploying assembly is a single-axis power hinge. In this case, the deploying assembly 114 can rotate within a range of approximately 190 degrees, but other ranges are also possible. The deploying assembly 114 can actively pivot the engaging assembly 116 about the first axis, while the engaging assembly 116 can provide passive up-and-down movement as shown in 604 to accommodate hand tilt. In this case, the up-and-down movement is approximately 30 degrees, but other ranges are also possible.

[0048] The base assembly 112 and the deployment assembly 114 can be adjusted to accommodate different user physiology (e.g., users of different sizes with different long bone lengths) and / or preferences. In this case, adjustability is achieved by a slider 606 that allows the user to control the relative position of the deployment assembly 114. The position of the deployment assembly controls the position of the engagement assembly 116 in the deployment orientation for an individual user. Thus, users with larger / longer arms and hands, as well as users with shorter arms and hands, can be accommodated. Further adjustment mechanisms are considered. For example, the length of the rod 608 extending between the deployment assembly 114 and the engagement assembly 116 can be adjustable.

[0049] Note that in the illustrated configuration, the first axis is slightly tilted from a vertical orientation. In this embodiment, the choice of the angle and rotation axis of the engagement assembly 116 has been carefully considered. Figure 6A and Figure 6B As shown, the unfolding component 114 rotates the engaging component 116 not about an axis perpendicular to the hand, but about a tilted first axis. This rotation about the tilted first axis is important because it allows the engaging component 116 to retract into the palm without interfering with the thumb. Furthermore, this tilted axis allows the engaging component 116 to be folded back into a position close to the user's arm, where interference with the hand is minimized during interaction with real-world objects or when resting on a table. See also Figure 4A and Figure 4B .

[0050] From one perspective, slider 606 can provide adjustability to the position of the engaging component to accommodate different hand sizes. The three degrees of freedom of device 110 can accommodate different hand sizes and the movement of the user's hand.

[0051] The base assembly 112, the unfolding assembly 114, and the engaging assembly 116 can be formed from various materials, such as polymers. Individual polymers can be selected for these parts based on their function. For example, a slightly flexible polymer can be selected for the cuff 401 to allow the cuff to adapt to different arm diameters and shapes. After the device is worn, the spring-like behavior of the cuff can embrace the user's arm, and when in contact with the strap (400, Figure 4A and 4B When used together, this provides a comfortable yet secure grip. Other parts can be made of more rigid polymers. For example, the rod 608 can be relatively rigid, so a polymer that provides this property can be chosen. Polymers are suitable because they can be easily formed into various shapes, such as by injection molding or 3D printing. However, other materials, such as metals or composite materials, can be used optionally or additionally.

[0052] The above describes several examples of device 110 simulating feedback from interactions with ungrounded objects. The following discussion concerns simulating interactions with grounded objects, such as furniture or walls.

[0053] Figures 7A to 7C A device 110 that collectively involves simulating feedback sensations related to grounded objects. Figure 7A It is a visualization 700 of a large virtual object 702 in the form of a virtual table 704 with a flat virtual surface 706, where the user touches it. Additionally, 708 represents the user's hand. Figure 7B and Figure 7C The device 110 and the user's hand are shown from the observer's perspective.

[0054] When a user touches the flat virtual surface 706 with their palm, in some implementations of the device 110, the contact may be perceived only in the middle of the palm, but the sensation is strong enough to perceive that the virtual surface is there. Figure 5B and Figure 5C The device 110 can generate a "bumping" sensation to mimic the virtual surface 706 by continuously forcing the engagement component closer to the user's hand. In other words, as the user moves his / her hand toward the virtual surface, the continuous movement of the engagement component toward the user's hand, as indicated by arrow 710, can be proportional to the level of the virtual hand penetrating the virtual surface. In other words, continuing to actuate the unfolding component 114 to move the engagement component 116 against the user's hand as the hand penetrates the virtual object produces a sensation of "bumping" against the virtual object, such as pushing a heavy object onto a surface.

[0055] Figure 8A and Figure 8B Together they demonstrate how similar principles can be applied to rendering haptic feedback in response to larger and heavier virtual objects. Figure 8AA visualization 800 shows a user lifting a heavy virtual object 802 (e.g., a virtual box). The user's hand is represented by representations 804(1) and 804(2). Figure 8B The illustration shows a user wearing devices 110(1) and 110(2) on both hands. Devices 110(1) and 110(2) can apply corresponding forces to the user's hands, thereby generating the sensation of lifting an object against gravity. In this case, the two-handed devices can create haptic feedback for hand interactions, such as lifting a heavy object.

[0056] Furthermore, devices 110(1) and 110(2) can simulate the weight and volume of lifting a virtual object, despite their compact form and fixed close to the user's hand (e.g., the user's forearm). Therefore, the compact form factor allows device 110 to be easily stored on the user's forearm when not in use and to unfold almost instantly when needed / expected. It should also be noted that in some embodiments, the shape of the engagement assembly can be adjusted to mimic the shape of the object being picked up. For example, the engagement assembly may include multiple independently controllable chambers that can be filled to simulate shapes, such as the flat shape of a virtual box. In other embodiments, the engagement assemblies may be interchangeable, allowing for the selection of versions of the engagement assembly configured (e.g., shaped) to mimic a particular function and / or perform a particular task. This aspect will be discussed below in relation to... Figures 11A to 11D It is described in more detail.

[0057] Furthermore, the individual device 110 can adjust the force it applies to the user's hand to simulate a compliant object, such as a soft ball. In the illustrated configuration where the user uses devices 110(1) and 110(2) with both hands, the controller can collaboratively simulate the compliance of an object held between them. For example, device 110 can mimic a user squeezing a large balloon between their hands and can mimic the air added to the balloon, thus pushing their hands away. Similarly, these devices can simulate pulling scenarios, such as a user grasping a rubber band with both hands and stretching it.

[0058] The following and Figures 9A to 10B The related discussions involved using device 110 to simulate capture and throwing. Figure 9A The diagram shows a virtual ball 902 falling toward the user's hand, represented by a symbol 904, and the user positioning his / her hand to catch the virtual ball, represented by a virtualization 900. The speed and distance between the virtual ball and the hand (representation) are indicated by arrow 906.

[0059] Figure 9B The device 110 is shown deploying the engagement assembly 116 at a time and speed corresponding to the distance and speed at which the virtual ball leaves the hand. Figure 9C A subsequent visualization 908 shows the user capturing the virtual ball 902. Figure 9DThe engaging component is shown to contact the user's palm at the time and speed expected by the user, as indicated by arrow 910. Therefore, by rapidly rotating the device's engaging component into the hand and imparting a pulse function upon the predicted impact, it creates the sensation of capturing an object.

[0060] Device 110 can naturally provide haptic feedback in response to throwing and catching virtual objects. A novel aspect is that the device's processing loop can detect when the virtual object moves toward the hand and anticipate the user's intention to catch it, so as to unfold the engagement component 116 from the retraction orientation to address system latency and place it in the user's hand when the user expects the object to make contact. Because the device is fixed to the forearm, it can create realistic haptic feedback for catching objects. The device can provide this feedback by generating a "bang" pulse upon impact with the caught object and / or via an output device (134). Figure 1A It produces audible sounds to further enhance the tactile impression.

[0061] Figure 9C and Figure 9D This demonstrates how the device renders weight by pushing the engagement assembly 116 into the hand when the palm is facing upwards. The device can match the magnitude of the thrust intensity to the sine of the angle on the vertical axis to produce a plausible effect. Figure 10A and Figure 10B Together they demonstrate how device 110 applies a similar force to pull an object from the hand when the palm is facing down.

[0062] Figure 10A A visualization 1000 (e.g., 904) shows a user holding a virtual ball 902. Figure 10B A device is shown that generates a downward force, indicated by arrow 1002, which is produced by the deployment assembly 114 of the device acting on the engagement assembly 116. Arrow 1002 may represent the expected gravity on the virtual sphere 902. In this embodiment, because the deployment assembly 114 has one degree of freedom, it can generate gravity by pulling the engagement assembly 116 away from the hand when the palm is facing down.

[0063] In addition to tactile feedback, device 110 can also simulate grasped objects. For example, when a user grasps engagement component 116 in their palm to generate the sensation of acceleration or friction exerted by the grasped object, the device can continuously actuate its unfolding component 114. This force feedback can be scaled to various forces, such as gravity, inertia, and frictional resistance.

[0064] Device 110 can function as a wrist / forearm-attached VR / AR controller with an deployable haptic controller that can pivot in and out of the user's hand as needed. Compared to existing VR controllers, this device enables hands-free interaction in both VR and the real world, while also acting as a handheld controller when needed. When approaching a virtual object, the device can position the engaging component closer to the user's hand. This allows the user to grasp and / or release virtual objects consistent with the visual scene. This can generate tactile sensations of touching, holding, releasing, and capturing and throwing virtual objects. The device's active pivoting mechanism can also render static and dynamic forces acting on virtual objects, such as inertia, gravity, and / or sliding friction.

[0065] Figures 11A to 11D Another device 110B is also shown. In this case, the engagement component 116 can be easily removed and replaced with a variant that may be suitable for a particular use case scenario. In this example, the unfolding component 114 and the engagement component 116 together define the connector or interface 1102. The engagement component 116 can be positioned against the unfolding component 114 to complete the connector 1102. The connector 1102 can physically and electronically connect the engagement component and the unfolding component. The user can interchange the unfolding component 116 (e.g., remove the engagement component 116(1) or 116(2) and replace it with a different engagement component 116(1) or 116(2)). In this embodiment, device 110B also includes a storage mechanism or bracket 1104 for one or more currently unused engagement components. This embodiment allows the user to select a single engagement component that the user typically or in a particular scenario prefers, and to store other engagement components in an inconvenient location without the risk of loss.

[0066] Figure 12 Further details of system 100, consistent with some embodiments of this concept, are shown. System 100 may include one or more devices 110, a head-mounted device 106, a base station 102, and / or other devices (such as personal computers, desktop computers, laptop computers, cellular phones, smartphones, personal digital assistants, tablets, mobile computers, wearable devices, cameras, appliances, smart devices, Internet of Things devices, vehicles, etc.), and / or any of the numerous evolving or yet-to-be-developed types of computing devices. As described above, any of these devices may operate independently to perform a given function, or may operate collaboratively with other devices to perform that function.

[0067] Figure 12 Two example device configurations 1202 are shown that can be adopted by device 110, head-mounted device 106, base station 102 and / or other devices. A single device (such as device 110) may adopt either configuration 1202(1) or 1202(2), or an alternative configuration.

[0068] (Due to space limitations on the attached pages, only one instance of each device configuration is shown, rather than a device configuration associated with each device.) In short, device configuration 1202(1) represents an operating system (OS)-centric configuration. Device configuration 1202(2) represents a system-on-a-chip (SOC) configuration. Device configuration 1202(1) is organized into one or more applications 1204, an operating system 1206, and hardware 1208. Hardware 1208 may include storage devices / memory 1210 and processor 1212. Other hardware 1208 (such as base component 112, deployment component 114, and bonding component 116) have been described in detail above and will not be repeated here. Device configuration 1202(2) is organized into shared resources 1214, dedicated resources 1216, and interfaces 1218 between them.

[0069] The controller 142 may be software stored on storage device / memory 1210 and executed by processor 1212. In other cases, the controller 142 may be a dedicated hardware or firmware controller, such as a microcontroller. The controller may receive information related to a scene (such as a virtual reality scene, an augmented reality scene, a mixed reality scene, etc.). This information may include information about the properties of virtual objects (such as the object's 6 degrees of freedom (6-DOF) (e.g., x, y, z coordinates plus roll, pitch, and yaw)) and / or other information such as various positions, velocities, accelerations, masses, weights, dimensions, and / or textures. The controller may also receive information about user body parts, such as fingers, arms, or legs. For example, the controller may receive information about the user's hands from an outward-facing camera on the head-mounted device 106. This information may include 6-DOF information (e.g., x, y, z coordinates plus roll, pitch, and yaw) and / or other information such as posture, velocity, acceleration, etc. The controller may also receive some of this information from devices 110 located on the user's forearm and hand. The controller can predict the interaction between the hand and the virtual object, at least in part, based on this information. The controller can then directly control the engagement component (such as an output device on the engagement component) and / or indirectly control the engagement component by controlling the unfolding component, based on the prediction. In this way, the controller can enable device 110 to allow the user to interact with the virtual object to simulate interaction with an equivalent physical object.

[0070] As used herein, the terms “device,” “computer,” or “computing device” can refer to any type of device having a certain amount of processing power and / or storage capacity. Processing power can be provided by one or more processors that can execute data in the form of computer-readable instructions to provide functionality. Data (such as computer-readable instructions and / or user-related data) can be stored on storage devices (such as internal or external storage devices). Storage devices can include one or more of the following: volatile or non-volatile memory, hard disk drives, flash memory devices and / or optical storage devices (e.g., CDs, DVDs, etc.), and remote storage devices (e.g., cloud-based storage devices). As used herein, the term “computer-readable medium” can include signals. Conversely, the term “computer-readable storage medium” does not include signals. Computer-readable storage media includes “computer-readable storage devices.” Examples of computer-readable storage devices include volatile storage media (such as RAM) and non-volatile storage media (such as hard disk drives, optical discs, and flash memory, etc.).

[0071] As described above, device configuration 1202(2) can be considered a system-on-a-chip (SOC) type design. In this case, the functionality provided by the device can be integrated on a single SOC or multiple coupled SOCs. One or more processors 1212 can be configured to coordinate with shared resources 1214 (such as storage devices / memory 1210, etc.) and / or one or more dedicated resources 1216 (such as hardware blocks configured to perform certain specific functions). Therefore, the term "processor" as used herein can also refer to a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), controller, microcontroller, processor core, and / or other types of processing devices.

[0072] Generally, any functionality described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), or a combination of these implementations. As used herein, the term "component" generally refers to software, firmware, hardware, an entire device or network, or a combination thereof. For example, in the case of a software implementation, these can represent program code that performs a specified task when executed on a processor (e.g., a CPU or CPU). The program code can be stored in one or more computer-readable storage devices, such as computer-readable storage media. The characteristics and technologies of components are platform-independent, meaning they can be implemented on a variety of commercial computing platforms with various processing configurations.

[0073] Figure 13A schematic diagram 1300 related to the above description is shown. Schematic diagram 1300 illustrates how a controller 142 (which manifests as a Teensy microcontroller 1302) can be communicatively coupled to other components. In this configuration, the microcontroller 1302 communicates with a BLE transceiver 1304 and an IMU 1306. The microcontroller 1302 communicates with a switch 132 and touch sensor electrodes 130, and receives force data from a strain gauge 128. The microcontroller 1302 can drive a VCA 136 and a servo motor 146 via a pulse width modulation (PWM) driver, and can receive information about the servo motor position as shown in 1308. The microcontroller 1302 can also interact with a unified game engine 1310 that operates in conjunction with an operating system, such as the Windows-branded operating system from Microsoft Corp., as shown in 1312.

[0074] In this case, servo motor 146 is a modified version of a commercially available servo motor (Hitech HS-7115TH). This modification provides control over (1) torque and speed, (2) reverse drive capability, and (3) real-time position feedback. To achieve this, the original control circuitry can be removed and replaced with custom drive electronics and software running on the Teensy controller. The implemented PID loop can have a time-based protection mechanism to prevent overloading of servo motor 146. The absolute position of potentiometer 1308 is read, and the front-end software is always up-to-date with the current position of the engagement assembly—even when the motor is off. This helps to make correct actions, such as detecting whether the user is holding the engagement assembly. While a particular servo motor implementation has been described herein, other servo motor implementations are also conceivable. For example, a more powerful servo motor and / or a servo motor with a faster response time could be used.

[0075] This feature allows control over the device's engagement components, ensuring they reach the user's hand at the correct speed, apply the correct (proportional) force, and can be switched off at any time to allow for passive rotation of the handle and prevent breakage.

[0076] The device's control board can be built around a Teensy 3.6 microcontroller 1302 that interfaces with a custom I / O daughterboard 1314. This daughterboard 1314 includes a servo motor driver and VCA PWM circuitry, an inertial sensor (e.g., IMU 1306) for detecting hand movements, a BLE chip (Nordic NRF52832) 1304 for wireless communication, and operational amplifiers for processing the full-bridge output of an analog strain gauge and position data from a servo potentiometer encoder. The device can detect touch events in active-load mode by sensing the conductive coating capacitance of the electrodes inside the bonding components (e.g., touch sensor electrodes 130) using the microcontroller's built-in capacitance sensing capability. (See also...) Figure 1C The engagement assembly 116 also includes a VCA 136 for providing haptic feedback and triggering buttons (e.g., switch 132).

[0077] Some software implementations can use the 2019 version of the Unity Game Engine 1310 as a software platform. This software can run on tethered devices such as the Alienware 15R3 laptop equipped with the ViveProVR system. The Unity Game Engine 1310 can maintain a representation of all virtual objects in the interaction space per frame (90Hz frames per second), as well as the position and orientation of the user's head and a position tracker attached to the user's palm. A spherical "trigger volume" can be defined around the device 110. Each virtual object penetrating this volume is a touchable object, allowing the engagement component to be rotated to an angle closer to the palm. Once the object reaches the hand, commands can be sent to the microcontroller 1302 to rotate the engagement component accordingly, simulating the tactile sensation of touch.

[0078] Other implementations of device 110 can be standalone units (e.g., completely unconstrained). Such implementations can employ trackers such as 6-DOF inside-out tracking or fully integrated VIVE lighthouse systems.

[0079] Figure 14 A flowchart illustrating an example method 1400 relating to a simulated object is shown. In action 1402, the method may receive information relating to the virtual object. This information may include position, velocity, mass, texture and / or size, as well as other information about the virtual object.

[0080] In action 1404, the method may receive information about the user's hand. This information may include position, posture, and / or velocity, as well as other information about the user's hand.

[0081] In action 1406, the method can predict whether the user's hand will engage with a virtual object. For example, the prediction could involve predicting the engagement point where the hand and the virtual object will come together at a specific time.

[0082] In action 1408, the method can begin moving the deployable controller from the retracted orientation toward the user's hand before the predicted engagement.

[0083] In action 1410, the method can make the deployable controller contact the user's hand to simulate the feel of the virtual object and the force exerted by the virtual object on the user's hand.

[0084] In action 1412, when the user detaches from the virtual object (e.g., stops engaging), the method can move the deployable controller away from the user's hand.

[0085] Therefore, this method can provide a deployable controller that dynamically appears and disappears in the user's palm, enabling rapid switching between haptic feedback-assisted interaction with virtual content and hand-held interaction with physical objects in the real world. This ability to switch rapidly makes the deployable controller particularly suitable for AR scenarios where users can frequently switch between using virtual and physical tools.

[0086] Various examples have been described above. Other examples are described below. One example includes a device comprising: a base assembly configured to attach the device to a non-hand body part of a user; an engagement assembly 116 configured to receive tactile input from the user's hand or to transmit tactile output to the user's hand; and a deployment assembly 114 extending from the base assembly to the engagement assembly and configured to deploy the engagement assembly from a storage orientation proximal to the base assembly to a deployment orientation proximal to the user's hand.

[0087] Another example may include any of the above and / or the following examples, wherein the base component is configured to be secured to a non-hand body part, including the user's forearm or the user's upper arm.

[0088] Another example may include any of the above and / or the following examples, wherein the engagement component is configured to receive tactile input from the user's hand and deliver tactile output to the user's hand.

[0089] Another example may include any of the above and / or the following examples, where the tactile output includes applying force to the user's hand.

[0090] Another example may include any of the above and / or the following examples, wherein the unfolding component is configured to generate a force from a non-hand body part to the hand.

[0091] Another example may include any of the above and / or the following examples, wherein the unfolding component includes a single-axis powered hinge.

[0092] Another example may include any of the above and / or the following examples, wherein the unfolding component includes a multi-axis powered hinge.

[0093] Another example may include any of the above and / or the following examples, wherein the device also includes a sensor configured to detect the user’s unfolding posture.

[0094] Another example may include any of the above and / or the following examples, where the sensor is located in both the coupling assembly and the base assembly.

[0095] Another example may include any of the above and / or the following examples, where the user unfolding posture includes wrist rotation movements.

[0096] Another example may include any of the above and / or below examples, wherein the device further includes a controller 142, and wherein the controller is configured to use virtual reality data as input data to control the unfolding component to unfold or store the joining component.

[0097] Another example may include any of the above and / or the following examples, wherein the controller is configured to automatically cause the unfolding component to unfold the engaging component at a rotational rate and time to engage the user's hand, in order to mimic the user capturing a virtual object.

[0098] Another example may include any of the above and / or the following examples, wherein the controller is configured to automatically cause the unfolding component to push the engaging component toward the user's palm to mimic the speed and force of the virtual object.

[0099] Another example may include any of the above and / or the following examples, wherein the controller is also configured to automatically move the unfolded component away from the user's hand when the user's hand stops engaging the virtual object.

[0100] Another example may include any of the above and / or the following examples, wherein the joining component is removably secured to the unfolding component, and wherein the user can interchange the joining component with another joining component.

[0101] Another example may include any of the above and / or the following examples, wherein the device further includes a storage mechanism for either the engagement component or another engagement component not fixed to the deployment component.

[0102] Another example includes a device comprising: a base assembly configured to attach the device to a non-hand body part of a user; a coupling assembly configured to receive tactile input from the user's hand or to transmit tactile output to the user's hand; a deployment assembly extending from the base assembly to the coupling assembly and configured to deploy the coupling assembly from a storage orientation proximal to the base assembly to a deployment orientation proximal to the user's hand; a position sensor configured to sense 3D position data of the device; and a controller configured to receive virtual 3D position data associated with a virtual object and control the deployment assembly to induce the deployment of the coupling assembly based at least in part on the 3D position data of the device and the virtual 3D position data of the virtual object.

[0103] Another example may include any of the above and / or below examples, wherein the position sensor is a 6-DOF sensor configured to sense the x, y, and z coordinates of the device, as well as roll, pitch, and yaw.

[0104] Another example includes a device comprising: a base assembly configured to secure the device to a user's forearm; an engagement assembly configured to transmit tactile output as force to the user's hand; and a deployment assembly extending from the base assembly to the engagement assembly and configured to pivotally deploy the engagement assembly from a storage orientation proximal to the base assembly to a deployment orientation proximal to the user's hand, and generate a force against the user's hand.

[0105] Another example may include any of the above and / or the following examples, wherein the unfolding component includes a powered single-axis hinge or a powered multi-axis hinge.

[0106] Another example may include any of the above and / or the following examples, wherein the unfolding component is configured to unfold the joining component along a near-arc path, or wherein the unfolding component is configured to unfold the joining component along a complex path.

[0107] Although the subject matter has been described in language specific to structural features and / or methodological behavior, the subject matter defined in the appended claims is not necessarily limited to the specific features or behaviors described above. Rather, the specific features and behaviors described above are presented as examples of implementing the claims, while other features and behaviors that can be recognized by a person skilled in the art are intended to be within the scope of the claims.

Claims

1. An apparatus comprising: A base assembly is configured to secure the device to the user's forearm; The engagement component is configured to receive tactile input from the user's hand or to transmit tactile output to the user's hand; An unfolding assembly extends from the base assembly to the engagement assembly and is configured to unfold the engagement assembly from a storage orientation near the base assembly to an unfolding orientation near the user's hand; as well as The controller is configured to use virtual reality data as input data to control the unfolding component to unfold or store the engaging component. The controller is also configured to automatically cause the unfolding component to unfold the engaging component at a rotational rate and time to engage the user's hand, in order to mimic the user capturing a virtual object.

2. The device of claim 1, wherein the engagement component is configured to receive tactile input from the user's hand and transmit tactile output to the user's hand.

3. The device of claim 2, wherein the tactile output comprises applying force to the user's hand.

4. The device of claim 3, wherein the unfolding component is configured to generate the force from the forearm to the hand.

5. The device of claim 4, wherein the deployment assembly comprises a single-axis powered hinge.

6. The device of claim 4, wherein the deployment assembly comprises a multi-axis powered hinge.

7. The device of claim 1, wherein the device further comprises a sensor configured to detect a user unfolding posture.

8. The device of claim 7, wherein the sensor is located in both the engagement assembly and the base assembly.

9. The device of claim 7, wherein the user unfolding posture includes a wrist rotation motion.

10. The device of claim 1, wherein the controller is configured to automatically cause the unfolding component to force the engaging component toward the palm of the user's hand to simulate the speed and force of the virtual object.

11. The device of claim 1, wherein the controller is further configured to: automatically move the unfolding component away from the user's hand when the user's hand stops engaging the virtual object.

12. The device of claim 1, wherein the engaging component is removably secured to the deploying component, and wherein the user is interchangeable between the engaging component and another engaging component.

13. The device of claim 12, further comprising a storage mechanism for either the engagement assembly or the other engagement assembly not fixed to the deployment assembly.

14. An apparatus comprising: A base assembly is configured to secure the device to the user's forearm; The engagement component is configured to receive tactile input from the user's hand or to transmit tactile output to the user's hand; An unfolding assembly extends from the base assembly to the engagement assembly and is configured to unfold the engagement assembly from a storage orientation near the base assembly to an unfolding orientation near the user's hand; A position sensor is configured to sense 3D position data of the device; as well as The controller is configured to receive virtual 3D position data associated with a virtual object and control the unfolding component to cause the engagement component to unfold based at least in part on the 3D position data of the device and the virtual 3D position data of the virtual object.

15. The device of claim 14, wherein the position sensor is a 6-DOF sensor configured to sense the x, y, and z coordinates of the device, as well as roll, pitch, and yaw.

16. An apparatus comprising: A base assembly is configured to secure the device to the user's forearm; The engagement component is configured to transmit tactile output as force to the user's hand; An unfolding assembly extends from the base assembly to the engagement assembly, and the unfolding assembly is configured to pivotally unfold the engagement assembly from a storage orientation near the base assembly to an unfolding orientation near the user's hand, and generate a force against the user's hand; as well as The controller is configured to use virtual reality data as input data to control the unfolding component to unfold or store the engaging component. The controller is also configured to automatically cause the unfolding component to unfold the engaging component at a rotational rate and time to engage the user's hand, in order to mimic the user capturing a virtual object.

17. The device of claim 16, wherein the deployment assembly comprises a powered single-axis hinge or a powered multi-axis hinge.

Images