Method and device for virtual objects on a mobile computing device

Abstract

Embodiments of the present disclosure relate to virtual object movement. A method for moving a virtual object on a computing device includes displaying a virtual object; moving the virtual object based on user input; based on the user input moving the virtual object in an attempt to violate an obstacle, displaying a collision indicator and an input indicator, wherein a shape of the input indicator is derived from a shape of the virtual object; moving the collision indicator based on user input and movement constraints imposed by the obstacle; and moving the input indicator based on user input without movement constraints imposed by the obstacle.

Description

[0001] Related application citations

[0002] This application is a continuation-in-part of International Application No. PCT / US2017 / 057070, International Filing Date, October 18, 2017, entered into the National Stage in China on April 11, 2019, Chinese National Application No. 201780062997.5, entitled "Virtual Object Movement," the disclosure of which is incorporated by reference in its entirety. TECHNICAL FIELD

[0003] Embodiments of the present disclosure relate to virtual object movement. BACKGROUND

[0004] Head-mounted display devices (HMDs) can be used to provide an augmented reality (AR) experience and / or a virtual reality (VR) experience by presenting virtual images to a user. The virtual images can be manipulated by the user and / or otherwise interacted with based on user input. SUMMARY

[0005] This summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the DETAILED DESCRIPTION. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in limiting the scope of the claimed subject matter. Furthermore, subject matter claimed in this application is not limited to solving any or all disadvantages with any particular implementation described in any part of this disclosure.

[0006] A method for moving a virtual object includes displaying a virtual object and moving the virtual object based on user input. A collision indicator and an input indicator are displayed based on user input that attempts to move the virtual object in violation of an obstacle. The collision indicator is moved based on the user input and movement constraints imposed by the obstacle. The input indicator is moved based on the user input without movement constraints imposed by the obstacle. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1A and 1B A user moving a virtual object displayed by a virtual reality computing device through an environment is shown schematically.

[0008] Figure 2 An example method for moving a virtual object is shown.

[0009] Figure 3A and 3B A virtual object is shown schematically replaced with a collision indicator and an input indicator.

[0010] Figure 4 and 5 Movement of the collision and input indicators is shown schematically.

[0011] Figure 6A and 6B Replacement of collision indicators and input indicators with virtual objects is shown.

[0012] Figure 7A and 7B The presentation of a virtual image to a user of a virtual reality computing device is schematically illustrated.

[0013] Figure 8 An example virtual reality computing device is schematically illustrated.

[0014] Figure 9 An example computing system is schematically illustrated. DETAILED DESCRIPTION

[0015] A virtual or augmented reality computing device can present a virtual object to a user and allow the user to freely move the virtual object through three-dimensional space. However, this movement can cause the virtual object to be directed towards a virtual or real-world obstacle (e.g., a real-world object in a physical environment, such as a real wall or a real table, or another virtual object). In some scenarios, after reaching an obstacle, the user may expect the virtual object he or she is moving to appear as a tangible object from the real world and collide with the obstacle. However, in other scenarios, the user may intend for the virtual object to pass through the obstacle as if the obstacle did not exist. Given this ambiguity, it may be difficult to provide a user experience that allows the user to freely move virtual objects in a predictable and satisfying manner.

[0016] Therefore, the present disclosure relates to a method for moving a virtual object when the movement of the virtual object conflicts with the movement constraints of an obstacle. Specifically, when the movement constraints of an obstacle are violated (e.g., the user attempts to move the virtual object through another object), an input indicator is displayed that tracks the input without constraints, and a collision indicator that obeys the movement constraints. This scheme allows the user to carefully and intuitively move the virtual object to the desired location, regardless of whether the user intends the object to obey the constraints imposed by the obstacle.

[0017] Figure 1A A user 100 is schematically shown wearing a virtual reality computing device 102 and viewing a surrounding environment 104. The virtual reality computing device 102 includes one or more near-eye displays 106 configured to present virtual images to the user's eyes, as will be described below. Figure 1A Also shown is the user's field of view (FOV) 108 , which indicates the area of ​​the environment 104 visible to the user 100 from the shown vantage point.

[0018] Although the term "virtual reality computing device" is generally used herein to describe a head-mounted display device (HMD) that includes one or more near-eye displays, devices having other form factors may alternatively be used to view and manipulate virtual images. For example, virtual images may be presented and manipulated via a smartphone or tablet computer that facilitates an augmented reality experience, and / or other suitable computing devices may be used instead.

[0019] Virtual reality computing device 102 may be an augmented reality computing device that allows user 100 to directly view the real world environment through a partially transparent near-eye display, or virtual reality computing device 102 may be completely opaque, or present an image of the real world environment captured by a front-facing camera, or present a completely virtual surrounding environment. To avoid repetition, the experience provided by both implementations will be referred to as "virtual reality," and computing devices used to provide augmented or purely virtualized experiences will be referred to as virtual reality computing devices. Furthermore, it should be understood that regardless of whether a virtual or augmented reality experience is implemented, Figure 1A and 1B At least some virtual images are shown that are visible only to a user of the virtual reality computing device.

[0020] Figure 1A Also shown is a virtual object 110. As described above, a user of a virtual reality computing device can move virtual objects through three-dimensional space. Figure 1A , where virtual object 110 has moved away from user 100 in the direction of arrow 112. Virtual object 110 moves in response to user input provided by the user. In the figures, arrows such as arrow 112 are used to indicate user input. Such user input can be performed in various ways using a variety of different input methods supported by the virtual reality computing device. For example, the user can provide gesture input captured by one or more cameras and / or motion sensors, and / or voice commands captured by a microphone. Additionally or alternatively, the user can use one or more input interfaces, such as a computer mouse, trackpad, joystick, video game controller, touch screen interface, or any other input interface suitable for manipulating virtual objects. Such input interfaces can be part of the virtual reality computing device 102, or the input interface can be part of an external device (e.g., a smartphone or dedicated input device).

[0021] Figure 1BAn alternative view of the movement of virtual object 110 is shown. As shown, virtual object 110 has moved in the direction of arrow 112 through the surrounding environment 104 away from user 100. The dashed outline of the virtual object indicates the previous position occupied by the virtual object during the movement. While virtual object 110 is shown as moving in only a single direction - away from the user - the virtual reality computing device can be configured to move the virtual object freely in all three dimensions of space in response to user input. In other words, the movement of the virtual object can have at least three degrees of freedom (3DOF). In addition, the virtual reality computing device can be configured to change one or more of the pitch, roll, and / or yaw of the virtual object based on user input, for a total of six degrees of freedom (6DOF).

[0022] The movement of virtual objects as described herein is generally referred to as being caused by user input or being performed by the virtual reality computing device based on receiving user input. However, the virtual reality computing device can move the virtual object for various reasons that do not involve explicit user input provided by the user. For example, the virtual object can be moved according to a predetermined pattern, dynamically moved according to software instructions (e.g., artificial intelligence), etc. It should be appreciated that the present disclosure applies to scenarios in which the movement of the virtual object is caused by explicit user input or is not caused by explicit user input.

[0023] As shown, user input indicated by arrow 112 attempts to move virtual object 110 in violation of obstacle 114, which can take the form of a real-world object (e.g., a wall, a screen, a piece of furniture) or a virtual object drawn by the virtual reality computing device. The movement of virtual object 110 can be limited by a movement constraint of the obstacle. In other words, user 100 can not be able to move virtual object 110 beyond obstacle 114. In some cases, user 100 can expect the virtual object to behave as a tangible real-world object, and such a movement limitation can be desirable. However, in other cases, the user can expect the virtual object to pass through the obstacle, or the movement of the virtual object to be relatively unaffected by the presence of the obstacle. Thus, based on the user input attempting to move the virtual object in violation of the obstacle, the virtual reality computing device can replace the virtual object with a collision indicator and an input indicator, as described below.

[0024] Figure 2 An example method 200 for moving a virtual object, such as virtual object 110, is shown. At 202, method 200 includes displaying a virtual object. The virtual object can have virtually any size, shape, or appearance, and can be displayed via a display of a virtual reality computing device, such as device virtual reality computing 102.

[0025] At 204, method 200 includes moving the virtual object based on user input. This can be as described above with reference to Figure 1A and 1B The virtual reality computing device can be configured to interpret user input in various forms. In addition, the virtual object can be moved, rotated, translated, or otherwise manipulated in any suitable manner.

[0026] At 206, method 200 includes replacing the virtual object with a collision indicator and an input indicator based on user input attempting to move the virtual object in violation of an obstacle. Figure 3A and 3B Shown in. Figure 3A Schematically, user 300 is shown using virtual reality computing device 302 to view virtual object 304. Arrow 306 indicates user input performed by user 300, and the user input attempts to move the virtual object against obstacle 308. Therefore, the virtual reality computing device may replace virtual object 304 with collision indicator 310 and input indicator 312, as shown in FIG. Figure 3B As shown. As described below, the movement of the collision indicator may be limited by movement constraints imposed by obstacles. However, the movement of the input indicator may not be constrained by obstacles, but may continue to move unconstrained based on user input. In this way, the input indicator provides visual feedback to the user about the user input, while the collision indicator maintains a sense of realism in which a collision is observed. As used herein, replacing a virtual object with a collision indicator and an input indicator may include adding a collision indicator and treating the virtual object as an input indicator; adding an input indicator and treating the virtual object as a collision indicator; and removing the virtual object and adding the input indicator and the collision indicator.

[0027] exist Figure 3B , collision indicator 310 and input indicator 312 are shown as having a similar, but different, appearance than virtual object 304. However, in some implementations, the appearance of the collision indicator can mimic the appearance of the virtual object. In other words, the collision indicator and the virtual object can be substantially identical so that the user does not perceive the difference between the two. In other implementations, the appearance of the collision indicator can differ from the appearance of the virtual object in one or more ways (e.g., different color, different transparency, different size, different shape).

[0028] The appearance of the input indicator can match the appearance of the virtual object. In some implementations, the appearance of the input indicator can have an appearance that is different from the appearance of the virtual object but is derived from the appearance of the virtual object. For example, the appearance of the input indicator can differ from the appearance of the virtual object based on one or more of color, size, transparency, and other visual attributes. In other implementations, the appearance of the input indicator is not derived from the appearance of the virtual object. As an example, the input indicator can be a generic pointer.

[0029] In some implementations, the collision indicator can have an appearance that matches the appearance of the virtual object, while the input indicator has a different color, size, transparency, etc. Furthermore, the difference in appearance between the collision indicator and the input indicator can change based on the distance between the collision indicator and the input indicator. For example, as the user moves the input indicator further away from the collision indicator, the appearance of the input indicator can change to be less similar to the collision indicator. In general, the collision and input indicators described herein can have virtually any size, shape, color, and overall appearance.

[0030] return Figure 2 At 208, the method 200 includes moving the collision indicator based on the user input and the movement constraints imposed by the obstacle. At 210, the method 200 includes moving the input indicator based on the user input without the movement constraints imposed by the obstacle. Figure 4 At time T1, Figure 4 User 400 is shown viewing an environment via virtual reality computing device 402 after a virtual object has been replaced by a collision indicator 404 and an input indicator 406. Collision indicator 404 is adjacent to obstacle 408, while input indicator 406 has largely passed through the obstacle. Based on the user's attempt to push the virtual object through the obstacle, the virtual object is replaced by the collision and input indicators. As shown, input indicator 406 continues to move in the direction of the user input, as indicated by arrow 410, while the movement of collision indicator 404 is constrained by obstacle 408. As described above, the user can choose to move the virtual object, so while the movement of the collision indicator may be limited by the movement constraints of the obstacle, the collision and input indicators can be used in various suitable ways. For example, rotation of the input indicator may result in rotation of the input indicator that allows for collision.

[0031] exist Figure 4 Further movement of the input indicator and collision indicator is shown at times T2 and T3 of FIG. Specifically, at T2, input indicator 406 has moved away from collision indicator 404 and obstacle 408 in the direction of the user input indicated by arrow 410, without regard to the movement constraint of obstacle 408. Collision indicator 404 cannot move in a direction perpendicular to obstacle 408 because it is restricted by the movement constraint of obstacle 408. However, collision indicator 404 has moved in a direction parallel to obstacle 408 based on the directional component of the user input that is parallel to obstacle 408 and does not violate the movement constraint of obstacle 408. The continued movement of the input and collision indicators continues in FIG. Figure 4 4. It is shown at T3 where the input indicator has moved further away from the obstacle 408, regardless of the movement constraint, while the movement of the collision indicator 404 is still restricted by the obstacle.

[0032] As mentioned above, the difference in appearance between the input indicator and the collision indicator may optionally depend on the distance between the two indicators. Figure 4 , where the input indicator 406 decreases in size as it moves further from the collision indicator.

[0033] Obstacles can constrain the movement of the collision indicator in a variety of ways. In some implementations, the movement constraints imposed by the obstacle can prevent the collision indicator from passing through the obstacle or sharing three-dimensional space with the obstacle. Alternatively, the movement constraints can prevent the object from occupying space within a threshold distance of the obstacle. Similarly, the movement constraints can allow the collision indicator to move past the obstacle to a lesser extent, but not completely around the obstacle. The specific constraints discussed herein are not limiting, and method 200 is compatible with virtually any type of movement constraints.

[0034] In some implementations, the movement constraints imposed on the collision indicator can be relaxed in certain circumstances. For example, when the distance between the collision indicator and the input indicator increases, the constraints can be relaxed. Figure 5 At time T1, Figure 5 User 500 is shown viewing the environment via virtual reality computing device 502 after a user input attempts to push a virtual object past obstacle 508. Consequently, the virtual object has been replaced by a collision indicator and input indicator 506. As shown, the movement constraint imposed by obstacle 508 is affecting the ability of the collision indicator to move in the direction of input arrow 510. Simultaneously, input indicator 506 is moving in the direction of arrow 510, regardless of the movement constraint imposed by obstacle 508.

[0035] exist Figure 5 Further movement of the input indicator and the collision indicator is shown at times T2 and T3 of FIG. At T2, the input indicator 506 continues to move in the direction of the user input without regard to the obstacle 508. At the same time, although the movement constraints imposed by the obstacle still prevent the collision indicator from passing the obstacle and connecting with the input indicator, these constraints have been partially relaxed as the distance between the collision indicator and the input indicator increases. As a result, a portion of the collision indicator 504 has bypassed the obstacle. This can indicate to the user that further movement of the input indicator will completely overcome the movement constraints. The continued movement of the input indicator and the collision indicator in FIG. Figure 5 3, where the distance between the input indicator and the collision indicator increases. Thus, the movement constraints of the obstacle 508 have been further relaxed, allowing a larger portion of the collision indicator 504 to bypass the obstacle.

[0036] Furthermore, in some implementations, the movement constraints imposed by an obstacle may be affected by factors other than the distance between the collision indicator and the input indicator. For example, a movement constraint may be gradually relaxed as the collision indicator contacts the obstacle, a movement constraint may apply differently to collision indicators representing certain types of virtual objects than to collision indicators representing other types of virtual objects, and so on.

[0037] Movement of the collision indicator may optionally be based at least in part on a simulated attractive force pulling the collision indicator toward the input indicator. Figure 5 506. This is indicated by dashed line 512, which represents a force pulling the collision indicator 504 toward the input indicator 506. This can cause the collision indicator to move in a manner that is influenced by the movement of the input indicator. In other words, as the input indicator moves, the collision indicator can move so as to remain as close to the input indicator as possible while still complying with any full or relaxed movement constraints imposed by the obstacle. Alternatively, the collision indicator may not move with the input indicator, but may instead occupy a fixed position relative to the obstacle as the input indicator moves.

[0038] Return to Figure 2 At 212 , method 200 optionally includes replacing the collision indicator and the input indicator with a virtual object at the location of the input indicator based on a separation condition between the collision indicator and the input indicator. Figure 6A is schematically shown in Figure 6A A user 600 is shown using a virtual reality computing device 602 to view a collision indicator 604 and an input indicator 606, which are displayed on opposite sides of an obstacle 608. As shown, the user 600 is providing user input to move the collision indicator and the input indicator in the direction of arrow 610, resulting in movement of the input indicator away from the collision indicator to the opposite side of the obstacle 608. Figure 6A However, the movement constraint imposed on collision indicator 604 by obstacle 608 prevents the collision indicator from passing the obstacle and moving in the direction of the user input.

[0039] Based on a separation condition between the collision indicator and the input indicator, these indicators can be replaced by a virtual object at the location of the input indicator. In some implementations, the separation condition can be a distance between the collision indicator and the input indicator, and these indicators can be replaced by a virtual object based on the distance exceeding a threshold. Figure 6B, where virtual object 612 has replaced collision indicator 604 and input indicator 606 at the previous location of input indicator 606. It should be understood that any suitable distance threshold can be used to trigger when the indicator is replaced by the virtual object, and the threshold can depend on the type of virtual object, the type of obstacle, the time that has passed since the virtual object was replaced by the input indicator and collision indicator, etc. Such thresholds can be set to nearly any value, with shorter threshold distances making it easier to overcome movement constraints, and longer threshold distances making it easier to more accurately place the virtual object at a specific location relative to the obstacle. Furthermore, as described above, as the distance between the collision indicator and the input indicator increases, the movement constraints of the obstacle can be relaxed. This can allow the collision indicator to begin to gradually move through the obstacle, serving as a visual indicator to the user that further movement of the input indicator will push the virtual object through the obstacle. As used herein, replacing collision and input indicators with virtual objects may refer to removing the input indicator and treating the collision indicator as a virtual object; removing the collision indicator and treating the input indicator as a virtual object; and removing both the collision indicator and the input indicator and adding a virtual object in the place of the collision indicator or the input indicator.

[0040] In some implementations, a virtual reality computing device may provide auditory and / or tactile feedback to a user based on virtual object collisions and / or state changes. For example, when a virtual object comes into contact with an obstacle, the virtual reality computing device may generate an audible "impact" sound. Similar sound effects may be generated when a virtual object is replaced by a collision indicator and an input indicator, when the user continues to provide user input that violates an obstacle, when the movement constraints of an obstacle are relaxed, when the input indicator and collision indicator are replaced by a virtual object, and so on. Similarly, in scenarios where a user controls the movement of a virtual object using one or more physical input devices, such input devices may provide vibration or other tactile feedback in addition to or in lieu of the sound effects described above.

[0041] When the virtual object replaces the collision indicator and the input indicator at the position of the input indicator, the movement constraints of the obstacle may no longer restrict the movement of the virtual object. In other words, the virtual reality computing device can be configured to move the virtual object based on user input without considering the movement constraints imposed by the object. Therefore, if the user moves the virtual object toward an obstacle and the user wants the virtual object to pass through the obstacle, this can be achieved if the user continues to provide user input that violates the movement constraints of the obstacle. Such user input can cause the input indicator to continue to move away from the collision indicator until the collision indicator and the input indicator are replaced by the virtual object based on the separation condition, allowing the user to move the virtual object freely regardless of the presence of the obstacle.

[0042] In the event that a user ceases providing user input after a virtual object has been replaced by a collision indicator and an input indicator, the virtual reality computing device may be configured to replace the input indicator and collision indicator with the virtual object. In some implementations, the input indicator may move in the direction of the collision indicator at any time the user is not providing user input in a different direction. Therefore, when the user interrupts the user input, the input indicator may move back to the collision indicator and reconnect with the collision indicator, causing both indicators to be replaced by the virtual object. The virtual object may then move based on the user input and the movement constraints imposed by the obstacle. In this scenario, the separation condition may be the distance between the collision indicator and the input indicator, and these indicators may be replaced by the virtual object based on the distance falling below a threshold. Therefore, if the user moves the virtual object such that it contacts an obstacle and is replaced by the collision indicator and input indicator, even though the user does not intend for the virtual object to circumvent the obstacle, the user may interrupt the user input that attempted to violate the obstacle's movement constraints. Notably, the input indicator may also move toward the collision indicator based on user input that moves the input indicator away from the obstacle, which may additionally result in the input and collision indicators being replaced by the virtual object.

[0043] In some implementations, the virtual object may replace the input indicator and the collision indicator based on a separation condition not described above. For example, the specific separation condition that determines when to replace the collision indicator and the input indicator may vary based on the type of virtual object and the type of obstacle. The separation condition may be the length of time after the virtual object is replaced by the indicator, and after the length of time exceeds a threshold, the indicator may be replaced by the virtual object. Similarly, when the two indicators reach different sides of an obstacle, the input indicator and the collision indicator may be automatically replaced by the virtual object. It should be understood that the input indicator and the collision indicator may be replaced by the virtual object based on other separation conditions not explicitly described herein.

[0044] After the virtual object is pushed through the obstacle, the virtual object may be partially or completely obscured by the obstacle. Figure 6B , where a virtual object 612 has replaced the collision indicator 604 and the input indicator 606 at the position of the input indicator. Assuming that the user and the virtual object are on different sides of the obstacle, if the obstacle 608 is not transparent, it may be difficult or impossible for the user to see the virtual object. Therefore, the virtual reality computing device can be configured to display a virtual window to maintain the visibility of the input indicator or virtual object based on the input indicator or virtual object being blocked by the obstacle. This is in Figure 6B 6, where a virtual window 614 allows the user 600 to see the virtual object 612 through the obstacle 608. A similar virtual window may be generated in the event that the input indicator is partially or completely obscured by an obstacle.

[0045] The virtual windows described herein can take various forms. For example, if the obstacle is a virtual object, the virtual reality computing device can simply alter the virtual object's appearance to increase its transparency. Alternatively, if the obstacle is a real-world object, the virtual reality computing device can present a transparent or "ghost" image of the input indicator, sized and shaped to appear visible through the obstacle. Similarly, the virtual window can have various suitable sizes and shapes to ensure that the input indicator or virtual object remains visible to the user.

[0046] Virtual images, such as the virtual objects described above, can be generated and displayed in various suitable manners. In some implementations, a near-eye display associated with a virtual reality computing device can include two or more micro-projectors, each micro-projector configured to project light onto or within the near-eye display in order to display a virtual image to a user. Figure 7A FIG. 7 shows a portion of an exemplary near-eye display 700. The near-eye display 700 includes a left micro-projector 702L positioned in front of a user's left eye 704L. It should be understood that the near-eye display 700 also includes a right micro-projector 702R positioned in front of a user's right eye 704R. Figure 7A Not visible in.

[0047] The near-eye display includes a light source 706 and a liquid crystal on silicon (LCOS) array 708. The light source can include a collection of light emitting diodes (LEDs)—for example, white LEDs or a distribution of red, green, and blue LEDs. The light source can be positioned to direct its emission onto the LCOS array, which is configured to form a display image based on control signals received from a logic machine associated with the virtual reality computing device. The LCOS array can include a plurality of individually addressable pixels arranged in a rectangular grid or other geometric shape. In some embodiments, pixels that reflect red light can be juxtaposed in the array to pixels that reflect green and blue light, so that the LCOS array forms a color image. In other embodiments, a digital micromirror array can be used in place of the LCOS array, or an active matrix LED array can be used instead. In other embodiments, the display image can be formed using transmissive, backlit LCD, or scanning beam technology.

[0048] In some embodiments, the displayed image from the LCOS array 708 may not be suitable for direct viewing by the user of the near-eye display 700. Specifically, the displayed image may be offset from the user's eye, may have undesirable vergence and / or a very small exit pupil (i.e., the area where the display light is released, not to be confused with the user's anatomical pupil). In view of these issues, the displayed image from the LCOS array can be further adjusted on the way to the user's eye. For example, the light from the LCOS array can pass through one or more lenses, such as lens 710, or other optical components of the near-eye display 700 to reduce any offset, adjust vergence, expand the exit pupil, etc.

[0049] The light projected by each microprojector 702 can take the form of a virtual image visible to the user and occupy a specific screen space location relative to the near-eye display. As shown, light from the LCOS array 708 forms a virtual image 712 at screen space location 714. Specifically, virtual image 712 is a virtual object in the form of a banana, but any other virtual image can be displayed instead of and / or in addition to the virtual banana. A similar image can be formed by microprojector 702R and occupy a similar screen space location relative to the user's right eye. In some implementations, the two images can be offset from each other in such a way that they are interpreted as a single three-dimensional image by the user's visual cortex. Thus, the user can perceive the images projected by the microprojectors as a single virtual object, occupying a three-dimensional world space location that is behind the screen space location of the virtual image presented by the near-eye display.

[0050] This is Figure 7B As shown in Figure 7B 7. A top view of a user wearing a near-eye display 700 is shown. As shown, a left micro-projector 702L is located in front of the user's left eye 704L and a right micro-projector 702R is located in front of the user's right eye 704R. A virtual image 712 is visible to the user as a virtual object existing at a three-dimensional world space location 716. The user can provide user input to move the virtual object. Such user input can cause the screen space location at which the virtual image is presented to change, thereby giving the illusion that the virtual object is moving through three-dimensional space. As with the other figures shown above, Figure 7B Includes virtual images that are visible only to the user of the virtual reality computing device.

[0051] Figure 8 Aspects of an example virtual reality computing system 800 are shown, including a near-eye display 802 that can present virtual images via one or more micro-projectors, as described above with reference to FIG. Figure 7A and 7B The virtual reality computing system 800 is Figure 1A and1B the virtual reality computing system 102, Figure 3A , 3B the virtual reality computing device shown in FIGS. 4, 4, 6A, 6B, the virtual reality device integrated with the near-eye display 700 of FIG. 7, and / or Figure 9 the non-limiting example of the computing system 900.

[0052] The virtual reality computing system 800 can be configured to present any suitable type of virtual reality experience. In some implementations, the virtual reality experience includes a fully virtual experience in which the near-eye display 802 is opaque such that the wearer is fully immersed in the virtual reality images provided via the near-eye display 802.

[0053] In some implementations, the virtual reality experience includes an augmented reality experience in which the near-eye display 802 is wholly or partially transparent from the perspective of the wearer to give a clear view of the physical space around the wearer. In such configurations, the near-eye display 802 is configured to direct display light to the user's eyes such that the user will see augmented reality objects that do not actually exist in the physical space. In other words, the near-eye display 802 can direct display light to the user's eyes while light from the physical space reaches the user's eyes through the near-eye display 802. In this way, the user's eyes simultaneously receive light from the physical environment and display light.

[0054] In such augmented reality implementations, the virtual reality computing system 800 can be configured to visually present augmented reality objects that appear to be body-locked and / or world-locked. Body-locked augmented reality objects can appear to move with the perspective of the user as the pose (e.g., six degrees of freedom (DOF): x, y, z, yaw, pitch, roll) of the virtual reality computing system 800 changes. In this way, the body-locked augmented reality objects can appear to occupy the same portion of the near-eye display 802 and appear to be at the same distance from the user even as the user moves in the physical space. Alternatively, world-locked augmented reality objects can appear to remain at a fixed location in the physical space even as the pose of the virtual reality computing system 800 changes. When the virtual reality computing system 800 visually presents world-locked augmented reality objects, this type of virtual reality experience can be referred to as a mixed reality experience.

[0055] In some implementations, the opacity of the near-eye display 802 can be dynamically controlled via a dimming filter. Thus, a substantially see-through display can be switched to full opacity for a fully immersive virtual reality experience.

[0056] The virtual reality computing system 800 may take any other suitable form in which a transparent, translucent, and / or opaque display is supported in front of the viewer's eyes. Furthermore, the implementations described herein may be used with any other suitable computing device, including but not limited to wearable computing devices, mobile computing devices, laptop computers, desktop computers, smartphones, tablet computers, and the like.

[0057] Any suitable mechanism can be used to display images via the near-eye display 802. For example, the near-eye display 802 can include an image-generating element located within a lens 806. As another example, the near-eye display 802 can include a display device, such as a liquid crystal on silicon (LCOS) device or an OLED microdisplay located within a frame 808. In this example, the lens 806 can serve as or otherwise include a light guide for transmitting light from the display device to the wearer's eyes. Additionally or alternatively, the near-eye display 802 can present left-eye and right-eye virtual reality images via respective left-eye and right-eye displays.

[0058] The virtual reality computing system 800 includes an onboard computer 804 configured to perform various operations related to receiving user input (e.g., gesture recognition, eye gaze detection), visual presentation of virtual reality images on a near-eye display 802, and other operations described herein. In some implementations, some to all of the aforementioned computing functions may be performed off-board.

[0059] The virtual reality computing system 800 may include various sensors and related systems to provide information to the onboard computer 804. Such sensors may include, but are not limited to, one or more inward-facing image sensors 810A and 810B, one or more outward-facing image sensors 812A and 812B, an inertial measurement unit (IMU) 814, and one or more microphones 816. The one or more inward-facing image sensors 810A, 810B may be configured to obtain gaze tracking information from the wearer's eyes (e.g., sensor 810A may obtain image data from one of the wearer's eyes, and sensor 810B may obtain image data from the other of the wearer's eyes).

[0060] The onboard computer 804 can be configured to determine the gaze direction of each of the wearer's eyes in any suitable manner based on information received from the image sensors 810A, 810B. The one or more inward-facing image sensors 810A, 810B and the onboard computer 804 can collectively represent a gaze detection machine that is configured to determine the wearer's gaze target on the near-eye display 802. In other implementations, different types of gaze detectors / sensors can be employed to measure one or more gaze parameters of the user's eyes. Examples of gaze parameters measured by the one or more gaze sensors used by the onboard computer 804 to determine eye gaze samples can include eye gaze direction, head orientation, eye gaze velocity, eye gaze acceleration, angular change in eye gaze direction, and / or any other suitable tracking information. In some implementations, eye gaze tracking can be recorded independently for both eyes.

[0061] One or more outward-facing image sensors 812A, 812B can be configured to measure physical environmental attributes of the physical space. In one example, image sensor 812A can include a visible light camera configured to collect visible light images of the physical space. Additionally, image sensor 812B can include a depth camera configured to collect depth images of the physical space. More specifically, in one example, the depth camera is an infrared time-of-flight depth camera. In another example, the depth camera is an infrared structured light depth camera.

[0062] Data from outward-facing image sensors 812A, 812B can be used by onboard computer 804 to detect movement, such as gesture-based input or other movement performed by the wearer or a person or physical object in the physical space. In one example, data from outward-facing image sensors 812A, 812B can be used to detect user input, such as gestures, performed by the wearer of virtual reality computing system 800. Data from outward-facing image sensors 812A, 812B can be used by onboard computer 804 to determine orientation / position and orientation data (e.g., from imaged environment features), which can enable position / motion tracking of virtual reality computing system 800 within a real-world environment. In some implementations, data from outward-facing image sensors 812A, 812B can be used by onboard computer 804 to construct still images and / or video images of the surrounding environment from the perspective of virtual reality computing system 800.

[0063] The IMU 814 can be configured to provide position and / or orientation data of the virtual reality computing system 800 to the onboard computer 804. In one implementation, the IMU 814 can be configured as a three-axis or three-degree-of-freedom (3DOF) position sensor system. This exemplary position sensor system can, for example, include three gyroscopes to indicate or measure changes in the orientation of the virtual reality computing system 800 about three orthogonal axes (e.g., roll, pitch, and yaw) in 3D space.

[0064] In another example, the IMU 814 can be configured as a six-axis or six-degree-of-freedom (6DOF) position sensor system. Such a configuration can include three accelerometers and three gyroscopes to indicate or measure changes in the position of the virtual reality computing system 800 along three orthogonal spatial axes (e.g., x, y, and z) and changes in the device orientation about three orthogonal rotational axes (e.g., yaw, pitch, and roll). In some implementations, the position and orientation data from the outward-facing image sensors 812A, 812B and the IMU 814 can be used in combination to determine the position and orientation (or 6DOF pose) of the virtual reality computing system 800.

[0065] The virtual reality computing system 800 may also support other suitable positioning technologies, such as GPS or other global navigation systems. Furthermore, while a specific example of a position sensor system has been described, it should be understood that any other suitable sensor system may be used. For example, head pose and / or movement data may be determined based on sensor information from any combination of sensors mounted on and / or external to the wearer, including but not limited to any number of gyroscopes, accelerometers, inertial measurement units, GPS devices, barometers, magnetometers, cameras (e.g., visible light cameras, infrared cameras, time-of-flight depth cameras, structured light depth cameras, etc.), communication devices (e.g., WIFI antennas / interfaces), etc.

[0066] One or more microphones 816 can be configured to measure sounds in the physical space. Data from the one or more microphones 816 can be used by the onboard computer 804 to recognize voice commands provided by the wearer to control the virtual reality computing system 800.

[0067] The onboard computer 804 may include logic and storage machines that communicate with the near-eye display 802 and various sensors of the virtual reality computing system 800, as described below. Figure 9 Discuss in more detail.

[0068] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. In particular, these methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0069] Figure 9 A non-limiting embodiment of a computing system 900 is schematically illustrated, which can implement one or more of the methods and processes described above. The computing system 900 is shown in simplified form. The computing system 900 can take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), virtual reality computing devices, and / or other computing devices.

[0070] The computing system 900 includes a logic machine 902 and a storage machine 904. The computing system 900 may optionally include a display subsystem 906, an input subsystem 908, a communication subsystem 910, and / or Figure 9 Other components not shown.

[0071] The logic machine 902 includes one or more physical devices configured to execute instructions. For example, the logic machine can be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions can be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise achieve a desired result.

[0072] The logic machine may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic machine may optionally be distributed across two or more separate devices that may be remotely located and / or configured for collaborative processing. Aspects of the logic machine may be virtualized and executed by a remotely accessible, networked computing device configured in a cloud computing configuration.

[0073] The storage machine 904 includes one or more physical devices configured to store instructions executable by a logic machine to implement the methods and processes described herein. When implementing such methods and processes, the state of the storage machine 904 may be transformed—for example, to maintain different data.

[0074] The storage device 904 may include removable and / or built-in devices. The storage device 904 may include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray Disc, etc.), semiconductor storage (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic storage (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), etc. The storage device 904 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices.

[0075] It should be understood that storage device 904 includes one or more physical devices. However, various aspects of the instructions described herein may alternatively be propagated by a communication medium (eg, electromagnetic signals, optical signals, etc.) that is not retained by a physical device for a limited duration.

[0076] Aspects of the logic machine 902 and the storage machine 904 may be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASICs), program and application specific standard products (PSSP / ASSPs), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0077] The terms "module," "program," and "engine" may be used to describe aspects of the computing system 900 that are implemented to perform specific functions. In some cases, a module, program, or engine may be instantiated via the logic machine 902 executing instructions held by the storage machine 904. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" may encompass a single or a group of executable files, data files, libraries, drivers, scripts, database records, and the like.

[0078] It should be understood that a "service" as used herein is an application program that is executable across multiple user sessions. A service can be used for one or more system components, programs, and / or other services. In some implementations, a service can run on one or more server computing devices.

[0079] When included, the display subsystem 906 can be used to present a visual representation of the data maintained by the storage machine 904. This visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data maintained by the storage machine and therefore change the state of the storage machine, the state of the display subsystem 906 can also be transformed to visually represent the changes in the underlying data. The display subsystem 906 can include one or more display devices utilizing virtually any type of technology. Such a display device can be combined with the logic machine 902 and / or the storage machine 904 in a shared enclosure, or such a display device can be a peripheral display device.

[0080] When included, the input subsystem 908 may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for voice and / or speech recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity.

[0081] When included, the communication subsystem 910 can be configured to communicatively couple the computing system 900 with one or more other computing devices. The communication subsystem 910 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem can allow the computing system 900 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0082] In one example, a method for moving a virtual object includes: displaying a virtual object; moving the virtual object based on user input; displaying a collision indicator and an input indicator based on the user input attempting to move the virtual object against an obstacle; moving the collision indicator based on the user input and a movement constraint imposed by the obstacle; and moving the input indicator based on the user input without the movement constraint imposed by the obstacle. In this or any other example, the obstacle is a real-world object. In this or any other example, the obstacle is a virtual object. In this or any other example, the appearance of the collision indicator mimics the appearance of the virtual object. In this or any other example, the appearance of the input indicator is different from, but derived from, the appearance of the virtual object. In this or any other example, the collision indicator and the input indicator differ in appearance based on one or more of color, size, and transparency. In this or any other example, the difference in appearance between the collision indicator and the input indicator changes based on the distance between the collision indicator and the input indicator. In this or any other example, rotation of the input indicator causes a collision-permitting rotation of the collision indicator. In this or any other example, the movement of the collision indicator is also based on a simulated attractive force pulling the collision indicator toward the input indicator. In this or any other example, as the distance between the collision indicator and the input indicator increases, the movement constraints imposed by the obstacle are relaxed. In this or any other example, the collision indicator and the input indicator replace a virtual object. In this or any other example, the method further includes replacing the collision indicator and the input indicator with a virtual object at the location of the input indicator based on a separation condition between the collision indicator and the input indicator. In this or any other example, the separation condition is the distance between the collision indicator and the input indicator, and based on the distance exceeding a threshold, the collision indicator and the input indicator are replaced with a virtual object at the location of the input indicator. In this or any other example, when replacing the collision indicator and the input indicator with the virtual object, the method further includes moving the virtual object based on user input without the movement constraints imposed by the obstacle. In this or any other example, the method further includes displaying a virtual window to maintain visibility of the input indicator or the virtual object.

[0083] In one example, a computing device includes a logic machine; a storage machine holding instructions executable by the logic machine to: display a virtual object; move the virtual object based on user input; and replace the virtual object with a collision indicator and an input indicator based on user input attempting to move the virtual object in violation of an obstacle; move the collision indicator based on the user input and movement constraints imposed by the obstacle; and move the input indicator based on the user input without movement constraints imposed by the obstacle. In this or any other example, the collision indicator and the input indicator differ in appearance according to one or more of color, size, and transparency, and the difference in appearance between the collision indicator and the input indicator changes based on a distance between the collision indicator and the input indicator. In this or any other example, the instructions are further executable to replace the collision indicator and the input indicator with the virtual object at a location of the input indicator based on the distance between the collision indicator and the input indicator exceeding a threshold, and move the virtual object based on the user input without movement constraints imposed by the obstacle. In this or any other example, movement of the collision indicator is further based on a simulated attractive force pulling the collision indicator toward the input indicator.

[0084] In one example, a method for moving a virtual object includes: displaying a virtual object; moving the virtual object based on user input; and replacing the virtual object with a collision indicator and an input indicator based on user input attempting to move the virtual object in violation of an obstacle; moving the collision indicator based on the user input, movement constraints imposed by the obstacle, and a simulated attractive force pulling the collision indicator toward the input indicator; moving the input indicator based on the user input without movement constraints imposed by the obstacle; replacing the collision indicator and the input indicator with the virtual object at a location of the input indicator based on a distance between the collision indicator and the input indicator exceeding a threshold; and moving the virtual object from the location based on the user input without movement constraints imposed by the obstacle.

[0085] It should be understood that the configurations and / or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein can represent one or more of any number of processing strategies. As such, various acts illustrated and / or described can be performed in the sequence illustrated and / or described, in other sequences, in parallel, or omitted. Likewise, the order of the above-described processes can be altered.

[0086] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations described herein, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A method for moving a virtual object on a computing device, comprising: Display virtual objects; moving the virtual object based on user input; displaying a collision indicator and an input indicator based on the user input attempting to move the virtual object in violation of an obstacle, wherein a shape of the input indicator is derived from a shape of the virtual object; moving the collision indicator based on user input and movement constraints imposed by the obstacle; as well as moving the input indicator based on user input without movement constraints imposed by the obstacle; Wherein the movement of the collision indicator is further based on a simulated attractive force pulling the collision indicator toward the input indicator. The method of claim 1 , wherein the collision indicator has the same size and shape as the virtual object. The method of claim 1 , wherein the appearance of the collision indicator is different from, but derived from, the appearance of the virtual object. The method of claim 1 , wherein the appearance of the input indicator differs from the appearance of the collision indicator according to one or more of: color, size, and transparency. 5 . The method of claim 4 , wherein the difference in appearance between the collision indicator and the input indicator changes based on a distance between the collision indicator and the input indicator. The method of claim 5 , wherein one or both of a color and a transparency of the input indicator changes based on a distance between the input indicator and the collision indicator. The method of claim 1 , wherein the obstacle is a real-world object. The method of claim 1 , wherein rotation of the input indicator results in collision-enabled rotation of the collision indicator. 9 . The method of claim 1 , wherein the movement constraints imposed by the obstacle are relaxed as the distance between the collision indicator and the input indicator increases.

10. The method of claim 1, wherein the collision indicator and input indicator replace the virtual object. 11 . The method of claim 10 , further comprising replacing the collision indicator and the input indicator with the virtual object at the location of the input indicator based on a separation condition between the collision indicator and the input indicator. 12 . The method of claim 11 , wherein the separation condition is a distance between the collision indicator and the input indicator, and based on the distance exceeding a threshold, the collision indicator and the input indicator are replaced with the virtual object at the position of the input indicator.

13. The method of claim 11 , wherein when replacing the collision indicator and the input indicator with the virtual object, the method further comprises: The virtual object is moved based on user input without movement constraints imposed by the obstacle.

14. The method according to claim 1, further comprising: A virtual window is displayed to maintain visibility of the input indicator or the virtual object.

15. A computing device comprising: logical machines; as well as A storage machine holds instructions executable by the logic machine to: Display virtual objects; moving the virtual object based on user input; as well as replacing the virtual object with a collision indicator and an input indicator based on the user input attempting to move the virtual object in violation of an obstacle, wherein an appearance of the collision indicator mimics an appearance of the virtual object, a shape of the input indicator is derived from a shape of the virtual object, and the appearance of the input indicator changes based on a distance between the input indicator and the collision indicator; moving the collision indicator based on user input and movement constraints imposed by the obstacle; as well as moving the input indicator based on user input without movement constraints imposed by the obstacle; Wherein the movement of the collision indicator is further based on a simulated attractive force pulling the collision indicator toward the input indicator.

16. The computing device of claim 15, wherein the collision indicator and input indicator differ in appearance according to one or more of: color, size, and transparency.

17. A computing device according to claim 15, wherein the instructions are further executable to: based on the distance between the collision indicator and the input indicator exceeding a threshold, replace the collision indicator and the input indicator with the virtual object at the location of the input indicator, and move the virtual object based on user input without movement constraints imposed by the obstacle.

18. A method for moving a virtual object on a computing device, comprising: Display virtual objects; moving the virtual object based on user input; as well as replacing the virtual object with a collision indicator and an input indicator based on the user input attempting to move the virtual object in violation of an obstacle, wherein an appearance of the input indicator changes based on a distance between the input indicator and the collision indicator, and wherein a shape of the input indicator is derived from a shape of the virtual object; moving the collision indicator based on user input, movement constraints imposed by the obstacle, and a simulated attractive force pulling the collision indicator toward the input indicator; moving the input indicator based on user input without movement constraints imposed by the obstacle; replacing the collision indicator and the input indicator with the virtual object at a location of the input indicator based on a distance between the collision indicator and the input indicator exceeding a threshold; as well as The virtual object is moved from the position based on user input without movement constraints imposed by the obstacle.

Images