Context-sensitive user interface activation in augmented and / or virtual reality environments

Abstract

In a system for determining a detected virtual contact between a user and a virtual object in a virtual reality environment, contact can be detected between a designated selection device manipulated by the user and the virtual object. It can then be determined whether the detected contact between the designated selection device and the virtual object is intentional. It can be determined whether the detected contact is intentional based on at least one of a direction of approach of the virtual contact with the virtual object, a speed of approach of the virtual contact with the virtual object, an acceleration of approach of the virtual contact with the virtual object.

Description

[0001] Cross-references to related applications

[0002] This application is a continuation of U.S. Nonprovisional Application No. 15 / 371,886, filed December 7, 2016, and claims priority to that application, which claims priority to U.S. Provisional Application No. 62 / 266,449, filed December 11, 2015, the disclosure of which is incorporated herein by reference.

[0003] This application claims priority to U.S. Provisional Application No. 62 / 266,449, filed December 11, 2015, the disclosure of which is incorporated herein by reference. Technical Field

[0004] This article mainly deals with input processing in immersive virtual reality systems. Background Technology

[0005] Augmented reality (AR) and / or virtual reality (VR) systems can generate immersive three-dimensional (3D) virtual environments. Users can experience these 3D virtual environments by interacting with various electronic devices, such as, for example, helmets or other head-mounted devices including displays, glasses, or goggles (through which the user views while viewing the display device), sensor-equipped gloves, external handheld devices including sensors, and other such electronic devices. Once immersed in the virtual environment, the user's interaction with the virtual environment can take various forms, such as, for example, eye gaze, head gaze, physical movement and / or manipulation of electronic devices to interact with the virtual environment, personalize the virtual environment, and control the virtual environment. Summary of the Invention

[0006] In one aspect, the method may include: generating a virtual environment, including displaying a plurality of virtual objects in the virtual environment; detecting virtual contact between a selected device and one of the virtual objects; determining, based on at least one characteristic of the detected virtual contact, whether the detected virtual contact corresponds to a command to be executed in the virtual environment; and when it is determined that the detected virtual contact corresponds to a command to be executed in the virtual environment, selecting a virtual object for actuation and performing an action in the virtual environment corresponding to the selected virtual object and the command based on the detected virtual contact.

[0007] In another aspect, the system may include a computing device configured to generate a virtual environment. The computing device may include: a memory storing executable instructions; and a processor configured to execute the instructions. Executing the instructions may cause the computing device to: generate a virtual environment including displaying a plurality of virtual objects; detect virtual contact between a selection device and one of the virtual objects; determine whether the detected virtual contact corresponds to a command to be executed in the virtual environment based on at least one characteristic of the detected virtual contact; and when it is determined that the detected virtual contact corresponds to a command to be executed in the virtual environment, select the virtual object for actuation and perform an action in the virtual environment corresponding to the command and the selected virtual object.

[0008] Details of one or more embodiments are set forth in the following drawings and description. Other features will become apparent from the description, drawings, and claims. Attached Figure Description

[0009] Figure 1 These are example embodiments of augmented and / or virtual reality systems, including head-mounted display devices and handheld electronic devices, according to the embodiments described herein.

[0010] Figure 2A and 2B This is a perspective view of an example head-mounted display device according to the embodiments described herein.

[0011] Figure 3 This is a block diagram of a head-mounted electronic device and a handheld electronic device according to the embodiments described herein.

[0012] Figures 4A to 4F and Figures 5A to 5F It is a third-person view illustrating user interaction with one or more virtual objects in an augmented and / or virtual reality environment according to the embodiments described herein.

[0013] Figures 6A to 6D The illustrations depict various methods for users to interact with virtual objects in an enhanced and / or virtual reality environment according to the embodiments described herein.

[0014] Figures 7A to 7J It is a third-person view illustrating user interaction with one or more virtual objects in an augmented and / or virtual reality environment according to the embodiments described herein.

[0015] Figure 8 This is a flowchart of a method for selecting virtual objects in an enhanced and / or virtual reality environment according to the embodiments described herein.

[0016] Figure 9Examples of computer devices and mobile computer devices that can be used to implement the techniques described herein are shown. Detailed Implementation

[0017] For example, a user wearing a head-mounted display (HMD) immersed in an augmented reality and / or virtual reality environment can explore the virtual environment and interact with virtual objects, virtual elements, virtual features, etc., within the virtual environment through various types of input. These inputs can include physical interactions, such as hand / arm gestures (including virtual contact between virtual objects and various body parts of the real user), head movements and / or directional gaze of the head and / or eyes, and / or manipulation of electronic devices separate from the HMD (e.g., virtual contact between electronic devices and virtual objects, virtual rays or beams emitted through electronic devices and / or the HMD, movement of electronic devices and / or the HMD, touches applied to the touch-sensitive surfaces of electronic devices and / or the HMD, etc.). Therefore, virtual contact refers to contact in virtual space between a virtual object and a virtual counterpart of the real user, a body part of the real user, or a selected device in the real world, wherein the virtual counterpart reflects the user's movement of body parts, directional gaze, etc., and / or manipulation of electronic devices in the real world. The user's virtual counterpart can be displayed as an immersive user in the virtual space. Users can perform one or more of these different types of interactions to select virtual objects and / or specific actions in the virtual environment. The systems and methods described herein facilitate accurate interpretation of detected virtual touches and determination of whether the detected virtual touches correspond to commands to be executed in the virtual environment when a user selects a specific virtual object and / or performs a specific action in the virtual environment, thereby enhancing the user's immersive virtual experience.

[0018] exist Figure 1In the illustrated example embodiment, a user wearing HMD 100 is holding a portable handheld electronic device 102. For example, the handheld electronic device 102 may be a controller, a smartphone, or another portable handheld electronic device that can be paired with and communicate with HMD 100 to interact in a virtual environment generated by HMD 100 and displayed to the user on HMD 100's display. For example, the handheld electronic device 102 may be operatively coupled or paired with HMD 100 via a wired or wireless connection (such as, for example, WiFi or Bluetooth). This pairing or operative coupling of the handheld electronic device 102 with HMD 100 can provide communication and data exchange between the handheld electronic device 102 and HMD 100, allowing the handheld electronic device 102 to act as a controller communicating with HMD 100 to interact in an immersive virtual environment generated by HMD 100. That is, manipulation of the handheld electronic device 102 (such as, for example, a beam or ray emitted by the handheld electronic device 102 and directed to a virtual object for selection, and / or input received on the touch surface of the handheld electronic device 102, or movement of the handheld electronic device 102) can be translated into corresponding selections, movements, or other types of interactions in the immersive virtual environment generated by the HMD 100.

[0019] Figure 1 The example implementation includes a handheld electronic device 102 communicating with the HMD 100. This handheld electronic device 102 is used to exchange data with the HMD 100 and interact with virtual objects, virtual elements, virtual features, etc., in a virtual environment generated by the HMD 100. However, in some implementations, more than one handheld electronic device 102 may be operatively coupled to and communicate with the HMD 100, and may operate together or individually to interact in the virtual environment.

[0020] Figure 2A and 2B Examples of HMDs that generate augmented and / or virtual reality environments (such as, for example, generated by...) Figure 1 The image shows a perspective view of a user-worn HMD 100. The HMD 100 may include a housing 110 coupled to a frame 120 having an audio output device 130, including, for example, a speaker mounted in headphones coupled to the frame 120. Figure 2BIn this configuration, the front portion 110a of the housing 110 rotates away from the base 110b of the housing 110, making some components housed within the housing 110 visible. A display 140 may be mounted inside the front portion 110a of the housing 110. When the front portion 110a is in a closed position relative to the base 110b of the housing 110, a lens 150 may be mounted within the housing 110, positioned between the user's eye and the display 140. In some embodiments, the HMD 100 may include a sensing system 160 (including various sensors) and a control system 170 (including a processor 109 and various control system devices) to facilitate the operation of the HMD 100.

[0021] In some implementations, HMD 100 may include a camera 180 to capture still and moving images. Images captured by camera 180 may be used to help track the physical location of the user and / or handheld electronic device 102 in the real world, or in relation to the physical environment of the virtual environment, and / or may be displayed to the user on display 140 in pass-through mode, providing the user with situational awareness of various aspects of the real world and the physical environment, and / or enabling the user to temporarily leave the virtual environment and return to the physical environment without removing HMD 100, or to change the configuration of HMD 100 to remove housing 110 from the user's line of sight.

[0022] In some embodiments, HMD 100 may include a gaze tracking device 165 to detect and track a user's eye gaze. The gaze tracking device 165 may include, for example, an image sensor 165A or multiple image sensors 165A to capture images of the user's eyes (e.g., specific parts of the user's eyes, such as, for example, the pupil) to detect and track the direction and movement of the user's gaze. In some embodiments, HMD 100 may be configured such that detected gazes are processed as user input to translate them into appropriate interactions within the virtual environment.

[0023] exist Figure 3 The diagram illustrates a block diagram of a system for activating a context-aware user interface in an augmented and / or virtual reality environment. The system may include a first electronic device 300 communicating with a second electronic device 302. The first electronic device 300 may be, for example, the one described above relative to... Figure 1 , 2A The HMD described in 2B generates a virtual environment and displays it to the user, and the second electronic device 302 can be, for example, the one described above relative to... Figure 1 The handheld electronic device described communicates with a first electronic device 300 to facilitate user interaction with a virtual environment generated and displayed by the first electronic device 300.

[0024] The first electronic device 300 may include a sensing system 360 and a control system 370, the sensing system 360 and the control system 370 being respectively similar to Figure 2A and 2B The sensing system 160 and control system 170 are shown. The sensing system 360 may include many different kinds of sensors (including, for example, light sensors, audio sensors, image sensors, distance / proximity sensors, and / or other sensors) and / or different combinations of sensors, including, for example, image sensors positioned to detect and track the user's eye gaze, such as... Figure 2B The gaze tracking device 165 is shown. The control system 370 may include, for example, power / pause control devices, audio and video control devices, optical control devices, switching control devices, and / or different combinations of such devices and / or devices. Depending on a particular embodiment, the sensing system 360 and / or control system 370 may include more or fewer devices. For example, in addition to… Figure 2A and 2B Within an HMD other than the HMD 100 shown, the components included in the sensing system 360 and / or control system 370 may have different physical arrangements (e.g., different physical locations). The first electronic device 300 may also include a processor 390 communicating with the sensing system 360 and control system 370, a memory 380, and a communication module 350 enabling communication between the first electronic device 300 and another external device (such as, for example, a second electronic device 302).

[0025] The second electronic device 302 may include a communication module 306 that enables communication between the second electronic device 302 and another external device (such as, for example, the first electronic device 300). In addition to enabling data exchange between the first electronic device 300 and the second electronic device 302, the communication module 306 may also be configured to emit the virtual rays or virtual beams described above. The second electronic device 302 may include a sensing system 304, which includes, for example, image sensors and audio sensors (such as those included in, for example, cameras and microphones), an inertial measurement unit, a touch sensor (such as those included in the touch-sensitive surface of a handheld electronic device, controller, or smartphone), and other such sensors and / or different combinations of sensors. A processor 309 may communicate with the sensing system 304 and the controller 305 of the second electronic device 302, the controller 305 being able to access the memory 308 and control the overall operation of the second electronic device 302.

[0026] For example, various virtual objects, virtual manipulation devices, and / or virtual user interface controls (such as buttons, sliders, scrolling lists, etc.) can be presented to the user in three dimensions for selection, manipulation, and assignment within a virtual environment. When multiple virtual objects are displayed to the user in this manner and selected by the user through the various input methods described above, determining whether a user command (e.g., selecting a virtual object and / or performing some other action in the virtual environment) should be executed based on the virtual contact detected between the immersive user and the virtual objects can be difficult. That is, determining whether the virtual contact detected between the user and the virtual objects indicates a user command (e.g., selecting a virtual object) can be difficult, especially when more than one virtual object is displayed to the user in the virtual environment.

[0027] Figures 4A to 4F Figures 5A to 5F illustrate a third-person view of user interaction with a virtual display of virtual objects (as seen by the user) in a virtual environment generated and displayed within HMD 100. Although the user, as a third-party observer, cannot see the virtual objects, this view illustrates the interaction between the user and the virtual objects. In this example implementation, HMD 100 displays multiple virtual objects A, B, C, D, and E to the user in the virtual environment, and the user can select and / or manipulate them using, for example, touching and grasping actions of the user's hand and / or handheld electronic device 102, and / or a virtual beam generated by the handheld electronic device 102 pointing at the virtual object to be selected (for ease of discussion and illustration only). However, as mentioned above, other types of input can be received to facilitate user interaction with virtual objects in the virtual environment, such as user hand and / or arm gestures (including touch, grasp, and swipe-type movements), user manipulation of the handheld electronic device 102, and / or HMD 100. HMD 100 can display multiple virtual objects A to F to the user for selection, manipulation, and interaction, making them appear to be arranged or interwoven along different planes to enhance the three-dimensional effect of the virtual environment.

[0028] Depending on the specific implementation, the arrangement of virtual objects A, B, C, D, and E can be virtually displayed in different ways. For example, in some implementations, multiple virtual objects A through E can be arranged in a single virtual plane. In some implementations, multiple virtual objects A through E can be arranged in different planes. In some implementations, the virtual objects can be arranged or grouped based on various factors (such as suitability, user location, and others). Figures 4A to 4EIn the example embodiments shown in 5A to 5F, virtual objects A, C, and E are arranged in one virtual plane, and virtual objects B, D, and F are arranged in another virtual plane, for the purpose of discussion and illustration only.

[0029] like Figures 4A to 4B As shown in 5A to 5B, for the purpose of selecting a virtual object D, a user can guide a designated selection device (e.g., the real user's hand and / or physical handheld electronic device 102, another part of the user's body whose position is detected and tracked by the system, a virtual beam generated by the handheld electronic device 102 and / or HMD 100, etc., as described above) toward virtual object D among multiple virtual objects A to F. Because virtual object D does not provide physical resistance or boundaries to user movement toward virtual object D, the immersive user can break (i.e., penetrate) the virtual plane of virtual object D in the virtual space or pass through virtual object D, such as... Figure 4C and Figure 5C As shown, and facing the virtual object E. Specifically, the user can pass through the front side D1 and through the rear side D2 of the virtual object D, and approach the front side E1 of the virtual object E placed behind object D according to the arrangement of virtual objects A to F.

[0030] In some cases, the user may realize that he has passed directly through virtual object D and is approaching the front surface E1 of virtual object E, and may consciously stop the forward movement of the selection device before touching the front side E1 of virtual object E. In other cases, the user may not be aware of the physical range of the selection movement relative to virtual objects D and E until the object selector has passed through the front side E1 and rear side E2 of virtual object E, passing directly through virtual object E. Because the user's purpose is to select virtual object D, not object E, the user can reverse the direction, such as... Figure 4D and 5D As shown, and returns through virtual object D, through the back side D2 and the front side D1 of virtual object D, as... Figures 4E to 4F and Figures 5E to 5F As shown.

[0031] In the above context Figures 4A to 4F In the examples discussed in 5A to 5F, the virtual objects displayed to the user are essentially flat objects, for ease of discussion and illustration only. However, the principles described above can be applied to other virtual objects with other three-dimensional, planar, or non-planar shapes and contours, where the front side of the virtual object can be, for example, the part of the virtual object closest to the user, and the back side of the virtual object can be, for example, the part of the virtual object furthest from the user.

[0032] The lack of physical resistance and / or physical boundaries and / or physical feedback associated with the 3D virtual display of multiple virtual objects A through F in a virtual environment can make it difficult to determine whether to execute a user command in response to detected virtual contact (and to determine the user's purpose based on the detected virtual contact), a difficulty typically not encountered in 2D interactive spaces. For example, when using a mouse to select objects from a 2D display, the physical boundaries of the display limit the movement of the mouse cursor. Similarly, when using a finger to select objects displayed on a touchscreen, the physical edges defining the size of the touchscreen limit the movement of the user's finger on the touchscreen. Conversely, in Figures 4A to 4F In the examples shown in 5A to 5F, because user movement in the physical space and corresponding movement in the virtual environment are not physically restricted in this manner, and because the user and / or selected device can continue to move after contacting one of the virtual objects A to F, the system may have difficulty determining whether to execute a user command based on the detected virtual contact. For example, in this example, the system may have difficulty determining whether the user wants to select virtual object D, or virtual object E, or vice versa, and therefore may have difficulty determining whether to select or actuate a particular virtual object and / or whether to perform a specific action in the virtual environment.

[0033] In some cases, as discussed above, movement of virtual objects within a virtual reality environment can be addressed by, for example, moving the virtual display of a virtual object once virtual contact or interaction with it is detected (such as the virtual contact described above), or by deleting the virtual display of another virtual object once virtual interaction / contact with it is detected. However, these relatively abrupt changes in the virtual environment can detach the user from the virtual environment and / or disrupt the user's virtual presence, thereby impairing the user's immersive virtual experience.

[0034] In the systems and methods according to the embodiments described herein, the system can determine what to perform, if any, an action and / or what to perform in the virtual environment, if any, a command (i.e., a user purpose based on detected virtual contact relative to the selection and / or manipulation of virtual objects, or other actions in the virtual environment), even if the user may pass through one or more virtual objects. Specifically, the system can be configured to allow the user to pass through virtual objects (e.g., virtual objects corresponding to virtual selection buttons used for user interaction in the virtual environment) and to compensate so that other virtual objects / selection buttons are not accidentally triggered when the user passes through the physical environment and moves accordingly in the virtual environment, and so that the user's movement in the physical environment does not trigger unintentional consequences in the virtual environment. In other words, the embodiments described herein achieve accuracy in recognizing the correspondence between virtual contact with virtual objects in the virtual environment and user commands (e.g., selecting a virtual object), especially when more than one virtual object is displayed to the user in the virtual environment. Therefore, the accuracy of the user's interaction with virtual objects in the virtual space can be improved.

[0035] In some implementations, intentional virtual contact between a user (e.g., the user's hand and / or arm, or other body part, or a user-controlled designated selection device) and a virtual object can be distinguished from unintentional virtual contact based on, for example, the proximity velocity at the virtual contact point between the user and the virtual object. In some implementations, intentional virtual contact between a user (e.g., the user's hand and / or arm, or other body part, or a user-controlled designated selection device) and a virtual object can be distinguished from unintentional virtual contact based on, for example, the proximity acceleration at the virtual contact point between the user and the virtual object. In some implementations, intentional virtual contact between a user (e.g., the user's hand and / or arm, or other body part, or a user-controlled designated selection device) and a virtual object can be distinguished from unintentional virtual contact based on, for example, the proximity direction at the virtual contact point between the user and the virtual object. In some implementations, this may include tracking six degrees of freedom of a designated selection device in a three-dimensional physical environment to translate that movement into a corresponding movement in the virtual environment, enabling the determination of approach velocity and / or acceleration and / or approach direction at a virtual point of contact between the user and the virtual object. In some implementations, the system may be configured to detect and / or track the user's body and / or designated portions of the user's body in physical space, enabling the system to track the user's orientation relative to the virtual environment approximately in real time and to translate the detected and / or tracked movements into the virtual environment relative to a virtual object displayed to the user.

[0036] exist Figures 6A to 6D Example virtual object 700 is shown. For ease of discussion and illustration only, example virtual object 700 has a generally rectangular three-dimensional shape, having a front side 701 (the side of virtual object 700 intended to face the user for interaction), a rear side 702 (the side of virtual object 700 intended to be away from the user), sides 703, and top and bottom sides 704. However, depending on system performance, specific virtual immersive experience, options to be displayed, and other such factors, the virtual object to be displayed can take many other forms.

[0037] In some implementations, a threshold for the proximity velocity and / or acceleration at the virtual contact point between the user and the virtual object 700 can be defined, and in some implementations, it can be considered together with the proximity direction or the virtual contact area to determine whether a specific selection should be made based on the detected virtual contact. The threshold for the proximity velocity and / or acceleration at the virtual contact point can be based on, for example, a specific virtual environment, a context associated with a specific scenario within the virtual environment being selected, characteristics associated with the designated selection device in the virtual environment, user history and / or preferences and / or profiles, and other variations of such factors. Similarly, defining the proximity direction and / or virtual contact area based on virtual contact within the proximity direction range relative to the virtual object and / or the designated contact area of ​​the virtual object, and the threshold for selecting / activating a specific virtual object can be based on, for example, a specific virtual environment, a context associated with a specific scenario within the virtual environment being selected, characteristics associated with the designated selection device in the virtual environment, user history and / or preferences and / or profiles, and / or other variations of such factors. In this document, the terms activation and selection may be used interchangeably.

[0038] exist Figure 6B In the example shown, by Figure 6B The various solid-line arrows shown correspond to different approaches to the virtual object 700 and to corresponding instances of virtual contact between the user and the virtual object 700. In some embodiments, by Figure 6B Each of the different approach directions illustrated by the solid-line arrows can represent one or more of these factors: approach direction, approach velocity, approach acceleration, approach area, and / or other factors associated with virtual contact between the user (i.e., a user-controlled designated selection device, including, for example, the user's hand or other body part, handheld electronic device 102, etc.) and the virtual object 700. In some embodiments, the approach can be based on specified characteristics of the virtual object (such as, for example, ...). Figure 6BThe approach direction is detected by the front side 701 of the virtual object 700 in the example shown. In some embodiments, the approach can be measured, for example, based on an angle relative to a specified feature or other such measuring unit. In some embodiments, the approach can be detected and measured based on a portion or area of ​​the virtual object 700 at which virtual contact is first detected. In some embodiments, by... Figure 6B The solid arrow shown identifies a virtual contact pointing to the front side 701 of the virtual object 700, which can be identified as an intentional contact corresponding to a command to select, actuate, or manipulate the virtual object 700.

[0039] In some implementations, for example, by Figure 6C The dashed arrow shown identifies virtual contact pointing to the rear side 702 of the virtual object 700, which can be identified as accidental or unintentional contact. In some embodiments, the system can determine the selection of the virtual object 700 and the... Figure 6C The virtual contact identified by the dashed arrows shown is unrelated to the purpose of the contact. In this case, the system can therefore determine that a virtual contact detected at the rear side 702 of the virtual object 700 will not trigger the execution of the corresponding command in the virtual environment without other user input / commands. In some embodiments, virtual contacts detected at the upper and / or lower side 704, and / or the side 703 of the virtual object 700 can be identified as intentional virtual contacts corresponding to commands to select, actuate, or manipulate the virtual object 700. In some embodiments, virtual contacts detected at the upper and / or lower side 704, and / or the side 703 can be identified as unintentional, accidental, or accidental virtual contacts because the selection of the virtual object 700 was not intentional. Intentional virtual contacts are those, according to the above classification, that are highly likely to match the user's actual purpose of selecting the virtual object through user, body movement, directional gaze, etc., and / or manipulation of electronic devices in the real world. Unintentional virtual contact (also known as accidental or unintended virtual contact) is a virtual contact that, according to the classification above, is unlikely to match the user's actual purpose of selecting the virtual object through the user's movement of body parts, directional gaze, and / or manipulation of electronic devices in the real world.

[0040] As mentioned above, such as Figure 6B The solid arrow shown indicates whether the approach to the virtual object 700 from a specific direction is intentional (and indicates the user command to be executed in the virtual environment), or as... Figure 6CThe dashed arrows shown indicate that determining whether the approach was unintentional / accidental can be based on many factors, such as, for example, the specific virtual environment, the context associated with the specific scenario in the virtual environment being selected, the characteristics associated with the specified selected device in the virtual environment, user history and / or preferences and / or profiles, and other such factors.

[0041] exist Figure 6D In the example shown, various virtual contacts between the user and the virtual object 700 can be identified by solid arrows. In this example, different virtual contacts between the user and the virtual object 700 can have different approach velocities, for example, measured by detecting and tracking a designated selection device in the physical environment (the user's hand or other body part, a user holding the handheld electronic device 102, a virtual beam emitted by the handheld electronic device 102, etc., as described above) and translating the detected physical movement into the virtual environment. In some implementations, a velocity threshold can be set for a specific virtual environment and a specific virtual object 700 in a specific context, and the approach velocity of a detected virtual contact can be compared to the threshold to determine whether the detected virtual contact corresponds to an intentional contact that corresponds to a command to select, actuate, and / or otherwise manipulate the virtual object 700. For example, a detected virtual contact exceeding the set approach velocity threshold might indicate that the user is passing by the virtual object with the intention of selecting another virtual object placed beyond the virtual object 700 in three-dimensional space. In this case, the system can identify the detected virtual contact as unintentional contact, not intended for selecting the virtual object 700. In some cases, a detected virtual contact below a set proximity speed threshold may indicate that the user's intention is to select the virtual object 700, and the system may identify the detected virtual contact as an intentional virtual contact corresponding to a command to select, actuate, and / or otherwise manipulate the virtual object 700.

[0042] In some implementations, the proximity speed threshold can be represented by a range, for example, defined by a lower threshold and an upper threshold. The system can identify detected virtual contact with a proximity speed less than the lower threshold as unintentional contact, not intended to select virtual object 700. The system can identify detected virtual contact with a proximity speed greater than the upper threshold as unintentional virtual contact, for example, indicating that the user's intention is to select another virtual object placed beyond virtual object 700 in the virtual environment. Detected virtual contact with a proximity speed greater than or equal to the lower threshold and less than or equal to the upper threshold, or within the proximity speed threshold range defined by the upper and lower thresholds, can be identified as intentional virtual contact corresponding to a command to select, actuate, and / or otherwise manipulate virtual object 700, and the corresponding action is performed in the virtual environment.

[0043] In addition to Figure 6D The solid arrows shown identify virtual contacts with different approach velocities, or alternatively, by... Figure 6D The virtual contacts identified by the solid arrows shown can have different proximity accelerations. For example, the proximity acceleration of a user contacting a virtual object 700 can be measured by detecting and tracking a designated selection device in the physical environment (a user's hand and / or other body parts, a user holding a handheld electronic device 102, a virtual beam emitted by the handheld electronic device 102, etc., as described above) and translating the detected physical movement into the virtual environment. In some implementations, an acceleration threshold can be set for a specific virtual environment and a specific virtual object 700 in a specific context, and the proximity acceleration of a detected virtual contact can be compared to the threshold to determine whether the detected virtual contact corresponds to a command to select, actuate, and / or otherwise manipulate the virtual object 700. For example, a detected virtual contact exceeding the proximity acceleration threshold may indicate that the user's intention is to select another virtual object placed beyond the virtual object 700 in the virtual environment, and the system may identify the detected virtual contact as unintentional contact not intended for selection of the virtual object 700. In some cases, a detected virtual contact below the near-acceleration threshold may indicate that the user's intention is to select virtual object 700, and the system may identify the detected virtual contact as an intentional virtual contact corresponding to a command to select, actuate, or otherwise manipulate virtual object 700.

[0044] In some implementations, the proximity acceleration threshold can be represented by a range, for example, defined by a lower threshold and an upper threshold. The system can identify detected virtual contact with a proximity acceleration less than the lower threshold as unintentional contact, not intended for selecting virtual object 700. The system can identify detected virtual contact with a proximity acceleration greater than the upper threshold as unintentional virtual contact, for example, indicating that the user's intention is to select another virtual object placed beyond virtual object 700 in three-dimensional space. Detected virtual contact with a proximity acceleration greater than or equal to the lower threshold and less than or equal to the upper threshold, or within the proximity acceleration threshold range defined by the upper and lower thresholds, can be identified as intentional virtual contact corresponding to a command to select, actuate, and / or otherwise manipulate virtual object 700, and the corresponding action is performed in the virtual environment.

[0045] In some embodiments, one of the approach direction, approach velocity, or approach acceleration of the detected virtual contact can be used to determine whether the detected virtual contact is intentional and corresponds to a command to select, actuate, or manipulate the virtual object 700. In some embodiments, various combinations of approach direction and / or approach velocity and / or approach acceleration can be used to determine whether the detected virtual contact is intentional and correspond to a command to select, actuate, or manipulate the virtual object 700. In some embodiments, the selection of the components of the detected virtual contact (approach direction, approach velocity, and / or approach acceleration) can depend on various factors, such as, for example, the specific virtual environment, the context associated with a specific scenario in the virtual environment being selected, characteristics associated with the specified selected device in the virtual environment, user history and / or preferences and / or profiles, and other such factors.

[0046] Figures 7A to 7J It is a third-person view (similar to other third-person views) through which the user interacts with virtual objects in a virtual environment generated and displayed by the HMD 100. Figure 7A and 7B As shown, when a user walks through the physical environment of an immersive virtual environment generated and displayed to the user by the HMD100, the user can sometimes configure virtual objects that can be selected and actuated by the user to be placed behind the user, such as... Figure 7A and 7B As shown, instead of placing it in front of the user, such as Figures 4A to 4F As shown above, a user's position in the physical environment can be detected and tracked, and the detected movement can be translated into a corresponding movement in the virtual environment. For example, this tracking can be achieved via six degrees of freedom tracking of a specified selected device in a three-dimensional physical environment to translate that movement into the virtual environment, as described above. In some embodiments, this tracking may include tracking of the specified selected device. In some embodiments, this tracking may include full-body tracking of the user, enabling the system to maintain a generally real-time awareness of the user's position and / or orientation relative to virtual objects in the virtual environment. In addition to the approach velocity and / or approach acceleration and / or direction / area discussed above in determining whether detected virtual contact with a virtual object is intentional, or alternatively, the user's position and orientation relative to the virtual object, and which corresponds to the user's commands, can be taken into account.

[0047] When the virtual object is placed behind the user, such as Figure 7A As shown, the user can move backward and make unintentional virtual contact with one of the virtual objects, such as... Figure 7B As shown. In Figure 7BIn the example shown, virtual contact can be detected between the user and virtual object D, either in front of or in the active area of ​​virtual object D. Virtual contact detected in this way (where the virtual object is placed behind the user and outside the user's field of vision) can have an approach direction and / or approach velocity and / or approach acceleration that identifies the detected virtual contact as an intentional virtual contact indicating the execution of a command in the virtual environment. However, in this example, since the positions of virtual objects A through F relative to the user (and relative to the specified selected device) are known to be behind the user (based on six degrees of freedom tracking of the specified selected device within the three-dimensional physical environment), the user can... Figure 7B The detected virtual contact shown is identified as unintentional contact that will not trigger the execution of the corresponding command in the virtual environment.

[0048] As mentioned above, in some implementations, the system can perform near real-time full-body tracking of a user in physical space, translating the user's full-body movements in the physical space into corresponding movements in the virtual environment, tracking the user's position and / or orientation relative to virtual objects in the virtual environment. For example, in some implementations, this may include tracking the position and / or orientation of the user's head (e.g., the front and back of the head). For example, in some implementations, this may include tracking the position and / or orientation of the user's shoulders (e.g., the left and right shoulders). In some implementations, this may include tracking the position and / or orientation of the user's arms / elbows / hands (e.g., the left arm / elbow / hand and the right arm / elbow / hand). In some implementations, this may include tracking the position and / or orientation of the user's torso (e.g., the chest and back). In some implementations, this may include tracking the position and / or orientation of the user's knees (e.g., the left and right knees). In some implementations, this may include tracking the position and / or orientation of the user's feet (e.g., the left and right feet). In some implementations, this may include tracking the position and / or orientation of more or fewer parts of the user's body. In some implementations, body parts can be tracked so that the system can detect the position and orientation of the body part when it makes virtual contact with a virtual object to determine whether the detected virtual contact corresponds to a user command to perform an action in the virtual environment.

[0049] For example, in some implementations, the system can distinguish between a first virtual contact between a first part of the user's hand (e.g., the user's fingertips) and a virtual object, and a second virtual contact between a second part of the user's hand (e.g., the palm or back of the user's hand) and a virtual object. In this case, the system can determine that the first virtual contact is intentional, corresponding to a user command to perform an action in the virtual environment, while the second contact is accidental, with no action in the virtual environment intended as a result of that virtual contact. In some implementations, the system can distinguish between a first virtual contact between a first body part (e.g., the user's right hand) and a virtual object, and a second virtual contact between a second body part (e.g., the user's left hand) and a virtual object. In this case, the system can be configured to be more sensitive to virtual contact with the user's dominant hand (e.g., the user's right hand) compared to virtual contact with the user's non-dominant hand (which is generally easier to perform accidental movements). Therefore, in this case, the system can determine that the first virtual contact is intentional, corresponding to a user command to perform an action in the virtual environment, while the second contact is accidental, with no action in the virtual environment intended as a result of that virtual contact. These types can be determined based on, for example, the context of the virtual environment and virtual objects associated with the detected virtual contact, user preferences, system preferences, etc.

[0050] exist Figure 7C In the example shown, virtual objects A through F are placed in front of the user in a virtual environment, and the system can be configured to: for example, track the user's position relative to the virtual objects A through F in the virtual environment via body tracking (including a set of preset body parts of the user) as described above, and also track the position and orientation of the HMD 100 and / or the handheld electronic device 102, as described above. Figure 7D In this process, the user has shifted their attention away from virtual objects A through F, for example, to interact with another user, or to familiarize themselves with other virtual elements in the virtual environment. This is because the user has turned their head, such as... Figure 7D As shown in the example, virtual objects A through F are no longer within the user's field of vision. This can be determined, for example, by the position and / or orientation of the user's head, based on, for example, tracking of the user's head, position data from the HMD100, etc. Figure 7D As shown, during the process of turning the head, the user may also have changed, for example, the position of his / her left arm / hand, so that the user's left hand makes virtual contact with the virtual object D.

[0051] In some implementations, removing virtual objects A through F out of the user's field of vision in this manner may render the virtual objects as inactive (e.g., virtual objects no longer in the user's field of vision). Therefore, in this example, even if virtual contact has been detected between the user's hand and virtual object D (e.g., at a proximity velocity and / or acceleration and / or contact area as described above, typically corresponding to an intentional user command), virtual object D is effectively disabled because it is no longer in the user's field of vision, and takes no action as a result of the detected virtual contact. In some implementations, for example, one or more virtual objects removed from the user's field of vision (e.g., for a preset time, or before the virtual object returns to the user's field of vision, etc.) may be rendered as inactive (e.g., disabled).

[0052] In some implementations, virtual contact detected between a specific body part of an immersive user and a specific virtual object in the virtual environment can trigger an action in the virtual environment; however, virtual contact between a specific body part and another virtual object, or between another body part and a specific virtual object, cannot trigger an action in the virtual environment. In this case, determining whether a specific virtual contact will trigger an action in the virtual environment can be based on, for example, a specific application and associated settings, a specific virtual scene and the context of virtual elements included in the virtual scene, user preferences, and profiles.

[0053] exist Figure 7E In the example shown, a user participating in a virtual soccer game might encounter a virtual object 750 representing a ball. In some cases, the system can be configured to ignore virtual contact detected between the user's foot and the virtual object. However, in this example, based on the context of the specific application, the specific virtual object 750 involved, and the specific body part involved, virtual contact between the user's foot and the virtual soccer ball 750 can be recognized as a command to trigger an action in the virtual environment. In some implementations, in addition to requiring the detected virtual contact between a specific body part (in this example, the user's foot) and a specific virtual object (in this example, the virtual soccer ball 750) to trigger an action in the virtual environment, the system may also consider approach velocity and / or approach acceleration and / or contact area / direction when determining whether the detected virtual contact is intended to trigger an action in the virtual environment. For example, in Figure 7EIn the example shown, virtual contact between the user's foot and the virtual soccer ball with a proximity velocity and / or acceleration less than a set threshold can indicate that the virtual contact may not be intentional and is intended to elicit a corresponding action in the virtual environment. For example, detected contact below the set velocity and / or acceleration threshold can indicate that the user accidentally or unknowingly walked into or past the virtual soccer ball 750. Conversely, detected contact at or above the set velocity and / or acceleration threshold, and / or within the set contact area / direction, can indicate that the detected virtual contact corresponds to an intentional movement or kicking of the virtual soccer ball 750, and the system can use the corresponding force in the virtual environment to virtually move the virtual soccer ball 750 in the corresponding direction.

[0054] In some implementations, the thresholds used to trigger actions in the virtual environment can be different when virtual contact is detected between a virtual object (such as, for example, a virtual soccer ball 750) and different body parts. For example, the action of picking up or flicking the virtual soccer ball 750 can be triggered by relatively simple contact between the user's hand and the virtual soccer ball 750 (applying a relatively wide threshold or no threshold when determining whether to trigger the corresponding action), such as... Figure 7F As shown. In this example, because the user must bend or stoop to reach the virtual soccer ball 750, the system can determine that any contact between the user's hand and the virtual soccer ball 750 in this particular application is intentional and triggers the corresponding action.

[0055] In some implementations, users can choose to teleport to a new virtual location or further explore virtual objects displayed in the virtual environment. Figure 7G In the example shown, for instance, a user viewing a virtual scene 400 on the display 140 of HMD 100 directs a virtual beam of light towards a virtual object 450 or a virtual car 450 within the virtual scene 400 to teleport to the virtual car 450. The user can reach the teleportation location, such as the interior of the virtual car 450, using a designated control device in a location corresponding to the active control area of ​​the teleportation location, such as a body part (e.g., a hand), a handheld electronic device 102, etc. Accidental user movement upon reaching the teleportation location may cause accidental movement of the designated control device in the active control area, and may generate accidental user commands to be executed.

[0056] For example, as shown in 7H(a), during transmission, a user can reach the interior of the virtual car 450 using their hand within a virtual contact area of ​​the control area (e.g., a virtual steering wheel 420 inside the virtual car 450). In this example, accidental movement of the user's hand while reaching the interior of the virtual car 450 using their hand placed on the virtual steering wheel 420 may cause accidental and unwanted movement of the virtual steering wheel 420, such as... Figure 7H (b) is shown. In some implementations, as described above, to avoid such accidental and unnecessary movement of control mechanisms when arriving at a new transport location or other points in the virtual environment where the user can transition, a grace period, inactivity period, or pause period can be incorporated into the transport action. During this inactivity period, various control mechanisms at the new transport location can be temporarily disabled so that virtual contact with these control mechanisms does not trigger actions in the virtual environment. This allows the user to adapt to the new virtual environment within a set timeframe before continuing to participate in it.

[0057] As mentioned above, in contrast to Figure 7C and 7D As discussed, in some implementations, movement of virtual objects outside the user's field of vision may sometimes disable the virtual object, thus preventing accidental activation of the virtual object by movement not intended by the user. In some implementations, for example, movement of virtual objects outside the user's field of vision may be disabled or kept active, depending on the application context, the specific virtual object being activated, settings associated with the viewing and / or focusing time of the specific virtual object, etc. For example, such as Figure 7I As shown, when in the virtual car 450, the user can scan downwards at the virtual transmission 430, allowing the user to accurately and virtually grasp the virtual transmission 430. While maintaining a virtual grip on the virtual transmission 430, for example, the user can then move their gaze from the virtual transmission 430 to the virtual steering wheel 420 and / or the forward trajectory or road in the virtual environment to engage with the virtual steering wheel 420 and control the virtual car 450 as it travels along the virtual trajectory, etc. In this example, the context of the specific virtual scene 400 viewed by the user and the characteristics of the specific virtual object (virtual transmission 430) enable the virtual transmission 430 to remain active while the user maintains virtual contact with it, even if the user has moved their gaze away from the virtual transmission 430 and the virtual transmission 430 is no longer in the user's field of vision, such as... Figure 7J As shown.

[0058] The above text is relative to Figures 7A to 7J The characteristics discussed enable the effective application of the above-mentioned features based on the specific characteristics of a particular virtual scene and the specific characteristics of the virtual objects that can be interacted with by the user within that virtual scene. Figures 4A to 6D The activation rules for discussion.

[0059] exist Figure 8 The document illustrates a method for actuating a virtual object selected from a user interface in an enhanced and / or virtual reality environment according to embodiments described herein. Upon detecting virtual contact between a user (e.g., a user-controlled designated selection device) and a virtual object displayed in three dimensions within the virtual environment (box 810), it can be determined whether the detected virtual contact corresponds to a command instructing the system to perform a specific action (820). For example, upon detecting virtual contact, the system can determine whether the detected virtual contact is intentional based on, for example, proximity velocity and / or acceleration and / or direction relative to the virtual object, the scene or specific virtual environment, user settings, and user preferences, as described above. Figures 4A to 4F , Figures 5A to 5F , Figures 6A to 6D and Figures 7A to 7B In detail, as described above, the system can determine, based on these factors, the likelihood or probability that the detected virtual contact between a user immersed in the virtual space and a virtual object is related to the user's intention to perform further actions. When it is determined that the detected virtual contact is intentional, the system can determine that the user has selected, actuated, or manipulated the virtual object, and can perform an action associated with the selected virtual object in the virtual reality environment (box 830). This process can be performed before determining that the virtual reality experience has been terminated (box 840).

[0060] Figure 9 Examples of a general-purpose computer device 900 and a general-purpose mobile computer device 950 are shown, which can be used with the techniques described herein. The computing device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, tablet computers, workstations, personal digital assistants, televisions, servers, blade servers, mainframe computers, and other suitable computing devices. The computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0061] The computing device 900 includes a processor 902, a memory 904, a storage device 906, a high-speed interface 908 connecting the memory 904 and a high-speed expansion port 910, and a low-speed interface 912 connecting a low-speed expansion port 914 and a storage device 906. The processor 902 may be a semiconductor-based processor. The memory 904 may be a semiconductor-based memory. Each component 902, 904, 906, 908, 910, and 912 is interconnected via various buses, and some components may be mounted on a common motherboard or otherwise installed as needed. The processor 902 can process instructions executed within the computing device 900, including instructions stored in the memory 904 or storage device 906 to display graphical information of a GUI on an external input / output device (such as a display 916 coupled to the high-speed interface 908). In other embodiments, multiple processors and / or multiple buses may be used with multiple memories and various types of memory, if desired. Similarly, multiple computing devices 900 can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system).

[0062] The memory 904 stores information within the computing device 900. In one embodiment, the memory 904 is one or more volatile memory cells. In another embodiment, the memory 904 is one or more non-volatile memory cells. The memory 904 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0063] Storage device 906 provides massive storage for computing device 900. In one embodiment, storage device 906 may be or may include computer-readable media, such as floppy disk devices, hard disk devices, optical disk devices, magnetic tape devices, flash memory, or other similar solid-state storage devices, or arrays of devices (including storage area networks or other configured devices). The computer program product may be tangibly embodied as an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 904, storage device 906, or memory on processor 902.

[0064] High-speed controller 908 manages bandwidth-intensive operations of computing device 900, while low-speed controller 912 manages less bandwidth-intensive operations. This functional allocation is merely exemplary. In one embodiment, high-speed controller 908 is coupled to memory 904, display 916 (e.g., via a graphics processor or accelerator), and high-speed expansion port 910, which can accept various expansion cards (not shown). In another embodiment, low-speed controller 912 is coupled to storage device 906 and low-speed expansion port 914. Low-speed expansion port 914 may include various communication ports (e.g., USB, Bluetooth, Ethernet, and wireless Ethernet), and may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices (e.g., switches or routers), for example, via a network adapter.

[0065] As shown in the figure, the computing device 900 can be implemented in various different forms. For example, the computing device 900 can be implemented as a standard server 920, or multiple times in a group of such servers. The computing device 900 can also be implemented as part of a rack-mount server system 924. Alternatively, the computing device 900 can be implemented in a personal computer (such as a laptop computer 922). Alternatively, components from the computing device 900 can be combined with other components in a mobile device (not shown) (such as device 950). Each such device can contain one or more computing devices 900, 950, and the entire system can consist of multiple computing devices 900, 950 communicating with each other.

[0066] In addition to other components, computing device 950 includes processor 952, memory 964, input / output devices (such as a display 954), communication interface 966, and transceiver 968. Device 950 may also be provided with storage devices providing additional storage, such as micro hard disks or other devices. Each component 950, 952, 964, 954, 966, and 968 is interconnected via various buses, and some components may be mounted on a common motherboard or otherwise installed as needed.

[0067] Processor 952 can execute instructions within computing device 950, including instructions stored in memory 964. The processor can be implemented as a chip set including individual and multiple analog and digital processor chips. The processor can provide, for example, coordination with other components of device 950, such as control of the user interface, applications running by device 950, and wireless communication via device 950.

[0068] Processor 952 can communicate with the user via control interface 958 and display interface 956 coupled to display 954. For example, display 954 can be a TFT LCD (Thin Film Transistor Liquid Crystal Display) or OLED (Organic Light Emitting Diode) display, or other suitable display technologies. Display interface 956 may include suitable circuitry for driving display 954 to present graphics and other information to the user. Control interface 958 can receive commands from the user and translate them for submission to processor 952. Additionally, external interface 962 can provide communication with processor 952, enabling device 950 to communicate with other devices in the vicinity. In some embodiments, external interface 962 can provide, for example, wired communication, or in some embodiments, wireless communication, and multiple interfaces may be used.

[0069] Memory 964 stores information within computing device 950. Memory 964 can be implemented as one or more computer-readable media, one or more volatile memory cells, or one or more non-volatile memory cells. Extended memory 974 can also be provided and connected to device 950 via extended interface 972, which may include, for example, a SIMM (Single Wire Memory Module) card interface. Such extended memory 974 can provide additional storage space for device 950, or it can also store applications or other information of device 950. Specifically, extended memory 974 may include instructions for performing or supplementing the processes described above, and may also include security information. Therefore, for example, extended memory 974 can be provided as a security module of device 950, and can be programmed with instructions that allow secure use of device 950. Additionally, secure applications can be provided via a SIMM card along with additional information (such as placing identification information on the SIMM card in a non-intrusive manner).

[0070] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, the computer program product is tangibly embodied as an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 964, extended memory 974, or memory on processor 952, which may be received, for example, via transceiver 968 or external interface 962.

[0071] Device 950 can communicate wirelessly via communication interface 966, which may include a digital signal processing circuitry system if needed. Communication interface 966 can provide communication in various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS text messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS. This communication can occur, for example, via radio frequency transceiver 968. Additionally, short-range communication can occur using transceivers such as Bluetooth, WiFi, or others (not shown). Furthermore, GPS (Global Positioning System) receiver module 970 can provide additional navigation or location-related wireless data to device 950, which, if appropriate, can be used by applications running on device 950.

[0072] Device 950 can also perform audible communication using audio codec 960, which can receive verbal information from a user and convert the verbal information into usable digital information. Audio codec 960 can also generate audible sounds for the user, such as through a speaker, for example, the speaker in the earpiece of device 950. Such sounds can include sounds from voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications operating on device 950.

[0073] As shown in the figure, the computing device 950 can be implemented in many different forms. For example, the computing device 950 can be implemented as a cellular phone 980. It can also be implemented as a smartphone 982, a personal digital assistant, or part of other similar mobile devices.

[0074] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, a storage system, at least one input device, and at least one output device, the programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from and transmit data and instructions to the programmable processor.

[0075] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0076] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0077] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a client computer with a graphical user interface or web browser through which a user can interact with the implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), and the Internet.

[0078] A computing system may include clients and servers. Clients and servers are typically located far apart and usually interact through a communication network. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0079] Several implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure.

[0080] Furthermore, the logical flow depicted in the accompanying drawings does not require the specific order or sequence shown to achieve the desired result. Additionally, other steps may be provided, or steps may be removed from the described flow, and other components may be added to or removed from the described system. Therefore, other embodiments are within the scope of the following claims.

[0081] Implementations of the various techniques described herein can be implemented in digital electronic circuit systems, or in computer hardware, firmware, software, or combinations thereof. Implementations can be carried out as computer program products, i.e., computer programs tangibly embodied as information carriers (e.g., tangibly embodied as machine-readable storage devices (computer-readable media)) for processing by or controlling the operation of data processing apparatuses (e.g., programmable processors, computers, or multiple computers). Therefore, computer-readable storage media can be configured to store instructions that, when executed, cause a processor (e.g., a processor at a host device, a processor at a client device) to perform processes.

[0082] Computer programs (such as those described above) can be written in any programming language (including compiled or interpreted languages) and can be deployed in any form (including as standalone programs or modules, components, subroutines, or other units suitable for a computing environment). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communication network.

[0083] The method steps can be executed by one or more programmable processors, which execute computer programs to perform functions by manipulating input data and generating outputs. The method steps can also be executed by a dedicated logic circuit system (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit)), and the device can be implemented as such an FPGA or ASIC.

[0084] Processors suitable for executing computer programs include, for example, general-purpose microprocessors, special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The components of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer may also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or the computer may be operatively coupled to receive data from or transfer data to or both of such mass storage devices. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including, for example, example semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a special-purpose logic circuit system.

[0085] To provide interaction with the user, an implementation can be carried out on a computer having: a display device for displaying information to the user, such as a cathode ray tube (CRT), light-emitting diode (LED), or liquid crystal display (LCD) monitor; and a keyboard and pointing device, such as a mouse or trackball, through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0086] Implementations can be carried out in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a client computer with a graphical user interface or web browser through which a user can interact with the implementation), or any combination of such backend, middleware, or frontend components. These components can be interconnected via any form of digital data communication medium (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0087] Further implementation methods are summarized in the following examples:

[0088] Example 1: A method comprising: generating a virtual environment, including displaying a plurality of virtual objects in the virtual environment; detecting virtual contact between a selected device and one of the virtual objects; determining, based on at least one characteristic of the detected virtual contact, whether the detected virtual contact corresponds to a command to be executed in the virtual environment; and when it is determined that the detected virtual contact corresponds to a command to be executed in the virtual environment, selecting the virtual object for actuation and performing an action in the virtual environment corresponding to the selected virtual object and the command based on the detected virtual contact.

[0089] Example 2: The method of Example 1, wherein determining whether a detected virtual contact corresponds to a command to be executed in a virtual environment based on at least one characteristic of the detected virtual contact includes: detecting at least one characteristic of the detected virtual contact, the at least one characteristic including: selecting a device approach direction relative to a virtual object at the virtual contact point, selecting a device approach speed relative to a virtual object at the virtual contact point, or selecting a device approach acceleration relative to a virtual object at the virtual contact point; and determining whether a detected virtual contact corresponds to a command to be executed in a virtual environment based on the detected at least one characteristic of the detected virtual contact.

[0090] Example 3: The method of Example 2, wherein detecting at least one characteristic of the detected virtual contact includes: detecting the approach direction of the selected device relative to the virtual object at the virtual contact point, and determining whether the detected virtual contact corresponds to a command to be executed in the virtual environment includes: comparing the detected approach direction of the detected virtual contact with a preset approach direction threshold.

[0091] Example 4: The method of Example 3, wherein a preset proximity direction threshold limits the range of proximity angles relative to the actuating surface of the virtual object, and wherein determining whether the detected virtual contact corresponds to a command to be executed in the virtual environment includes: executing the command in the virtual environment when the detected proximity direction is within the preset proximity angle range.

[0092] Example 5: The method of Example 3 or 4, wherein the preset proximity direction threshold is defined relative to one or more actuation regions of the virtual object, and wherein determining whether a detected virtual contact corresponds to a command to be executed in the virtual environment includes: executing the command in the virtual environment when the detected proximity direction is detected within one of the one or more actuation regions of the virtual object.

[0093] Example 6: A method of one of Examples 2 to 5, wherein detecting at least one characteristic of a detected virtual contact includes: detecting the approach speed of the detected virtual contact relative to a virtual object, and determining whether the detected virtual contact corresponds to a command to be executed in the virtual environment includes: comparing the detected approach speed of the detected virtual contact with a preset approach speed threshold.

[0094] Example 7: The method of Example 6, wherein a preset approach speed threshold limits the range of approach speeds relative to a virtual object, and wherein determining whether a detected virtual contact corresponds to a command to be executed in the virtual environment includes: executing the command in the virtual environment when the detected approach speed is within the preset approach speed range.

[0095] Example 8: A method of one of Examples 2 to 7, wherein detecting at least one characteristic of a detected virtual contact includes: detecting the proximity acceleration of the detected virtual contact relative to a virtual object, and determining whether the detected virtual contact corresponds to a command to be executed in a virtual environment includes: comparing the detected proximity acceleration of the detected virtual contact with a preset proximity acceleration threshold.

[0096] Example 9: The method of Example 8, wherein a preset proximity acceleration threshold limits the range of proximity accelerations relative to a virtual object, and wherein determining whether a detected virtual contact corresponds to a command to be executed in the virtual environment includes: executing the command in the virtual environment when the detected proximity acceleration is within the preset proximity acceleration range.

[0097] Example 10: A method of one of Examples 2 to 9, wherein the detected virtual contact is detected at a first surface of a virtual object, the first surface of the virtual object defining an actuating surface of the virtual object, the method further comprising: detecting subsequent virtual contact on a second surface of the virtual object, the second surface defining an inactive surface of the virtual object; determining that the subsequent virtual contact does not correspond to a command to be executed in the virtual environment; and ignoring the detection of the subsequent virtual contact.

[0098] Example 11: A method of one of Examples 1 to 10, wherein detecting virtual contact between the selected device and one of a plurality of virtual objects includes: detecting that the virtual object is outside the user's field of view; determining that the detected virtual contact does not correspond to a command to be executed in the virtual environment; and ignoring the detection of the virtual contact.

[0099] Example 12: A method of one of Examples 1 to 11, wherein detecting virtual contact between a selected device and one of a plurality of virtual objects includes: detecting a change in user field of view from a first field of view to a second field of view, wherein virtual contact with the virtual object is detected within the first field of view and the virtual object is outside the second field of view; determining that the detected virtual contact corresponds to a command to be executed relative to the virtual object, the virtual object being outside the second field of view; and executing the command in response to the detected virtual contact.

[0100] Example 13: A method of one of Examples 1 to 12, wherein the specified selection device is a user's body part, detecting virtual contact includes: tracking the position and orientation of the user's body part relative to a plurality of virtual objects in a virtual environment, the detected virtual contact being detected between the user's body part and the virtual objects, and determining whether the detected virtual contact corresponds to a command to be executed in the virtual environment includes: determining whether the detected virtual contact corresponds to a command to be executed based on the position and orientation of the body part when the virtual contact is detected.

[0101] Example 14: A method of one of Examples 1 to 12, wherein the specified selection device is at least one of a plurality of body parts of a user, detecting virtual contact includes: tracking the position and orientation of the plurality of body parts of the user relative to a plurality of virtual objects in a virtual environment, the detected virtual contact being detected between one of the plurality of body parts of the user and a virtual object, and determining whether the detected virtual contact corresponds to a command to be executed in the virtual environment includes: determining the context of the virtual environment when the virtual contact is detected; determining whether the one of the plurality of body parts matches a virtual object to actuate the virtual object within the determined context of the virtual environment; if the one of the plurality of body parts matches a virtual object, executing a command in the virtual environment in response to the detected virtual contact; and if the one of the plurality of body parts does not match a virtual object, ignoring the detected virtual contact.

[0102] Example 15: A method of one of Examples 1 to 12, wherein the specified selection device is at least one of a plurality of body parts of a user, detecting virtual contact includes: tracking the position and orientation of the plurality of body parts of the user relative to a plurality of virtual objects in a virtual environment, the detected virtual contact being detected between one of the plurality of body parts of the user and a virtual object, and determining whether the detected virtual contact corresponds to a command to be executed in the virtual environment includes: determining whether the detected virtual contact is detected between an actuating portion of the one of the plurality of body parts and an actuating portion of the virtual object based on the tracked position and orientation of the one of the plurality of body parts when the virtual contact is detected; if the detected virtual contact is between the actuating portion of the one of the plurality of body parts and the virtual object, then executing a command in the virtual environment in response to the detected virtual contact; and if the detected virtual contact is not between the actuating portion of the one of the plurality of body parts and the virtual object, then ignoring the detected virtual contact.

[0103] Example 16: A system comprising: a computing device configured to generate a virtual environment, the computing device including: a memory storing executable instructions; and a processor configured to execute the instructions to cause the computing device to: generate a virtual environment comprising a plurality of virtual objects; detect virtual contact between a selection device and one of the plurality of virtual objects; determine whether the detected virtual contact corresponds to a command to be executed in the virtual environment based on at least one characteristic of the detected virtual contact; and when it is determined that the detected virtual contact corresponds to a command to be executed in the virtual environment, select the virtual object for actuation and perform an action in the virtual environment corresponding to the command and the selected virtual object.

[0104] Example 17: The system of Example 16, wherein, upon determining whether a detected virtual contact corresponds to a command to be executed in a virtual environment, the processor is further configured to execute instructions to cause the computing device to: detect at least one characteristic of the detected virtual contact, the at least one characteristic including: selecting an approach direction of the device relative to a virtual object at the virtual contact point, selecting an approach velocity of the device relative to a virtual object at the virtual contact point, or selecting an approach acceleration of the device relative to a virtual object at the virtual contact point; and determine whether the detected virtual contact corresponds to a command to be executed in a virtual environment based on the detected at least one characteristic of the detected virtual contact.

[0105] Example 18: The system of Example 17, wherein, when determining whether a detected virtual contact corresponds to a command to be executed in a virtual environment based on at least one characteristic of the detected virtual contact, the processor is further configured to execute instructions to cause the computing device to perform at least one of the following operations: comparing the detected approach direction of the detected virtual contact with a preset approach direction range, and executing the command in the virtual environment if the detected approach direction of the detected virtual contact is within the preset approach direction range; comparing the detected approach velocity of the detected virtual contact with a preset approach velocity range, and executing the command in the virtual environment if the detected approach velocity of the detected virtual contact is within the preset approach velocity range; or comparing the detected approach acceleration of the detected virtual contact with a preset approach acceleration range, and executing the command in the virtual environment if the detected approach acceleration of the detected virtual contact is within the preset approach acceleration range.

[0106] Example 19: A system of one of the examples 16 to 18, wherein the detected virtual contact is detected at a first surface of a virtual object, the first surface defining an actuating surface, and the processor is further configured to execute instructions to cause the computing device to: detect subsequent virtual contact on a second surface of the virtual object, the second surface defining an inactive surface of the virtual object; determine that the subsequent virtual contact does not correspond to a command to be executed in the virtual environment; and ignore the detection of the subsequent virtual contact.

[0107] Example 20: A system of one of the examples in Examples 16 to 19, wherein, when detecting virtual contact between a selected device and one of a plurality of virtual objects, the processor is further configured to execute instructions to cause the computing device to: detect that the virtual object is out of the user's field of view; determine that the detected virtual contact does not correspond to a command to be executed in the virtual environment; and ignore the detection of the virtual contact.

[0108] While certain features of the described embodiments have been illustrated as described herein, many modifications, substitutions, alterations, and equivalents will appear to those skilled in the art. Therefore, it is to be understood that the appended claims are intended to cover all such modifications and alterations that fall within the scope of the embodiments. It should be understood that they are presented by way of example only and not as limitations, and various changes in form and detail may be made. Any part of the apparatus and / or method described herein can be combined in any combination, except for mutually exclusive combinations. The embodiments described herein may include various combinations and / or sub-combinations of the functions, components, and / or features of the different embodiments described.

Claims

1. A method for processing input in a virtual environment, the method comprising: A virtual environment is generated for interaction, the virtual environment including the display of multiple virtual objects within the virtual environment; The detection of virtual contact characteristics between a selection device and one of the plurality of virtual objects includes: tracking the position and orientation of the selection device relative to the plurality of virtual objects, the characteristics including one or more of the following: the approach direction of the selection device relative to the virtual object at the virtual contact point, the approach velocity of the selection device relative to the virtual object at the virtual contact point, or the approach acceleration of the selection device relative to the virtual object at the virtual contact point; Determining whether a detected virtual contact corresponds to intentional or unintentional virtual contact in the virtual environment is based on the characteristics of the detected virtual contact and the context of the virtual environment when the virtual contact is detected; and When it is determined that the detected virtual contact corresponds to an intentional virtual contact in the virtual environment Select the virtual object to actuate; and Execute actions within the virtual environment corresponding to the selected virtual object and the commands corresponding to the detected virtual contact; and When it is determined that the detected virtual contact corresponds to an unintentional virtual contact in the virtual environment, the detection of the virtual contact is ignored.

2. The method according to claim 1, wherein, Detecting the characteristics of the detected virtual contact includes: detecting the approach direction of the selection device relative to the virtual object at the virtual contact point, and Determining whether the detected virtual contact corresponds to an intentional or unintentional virtual contact in the virtual environment includes comparing the detected approach direction of the virtual contact with a preset approach direction threshold.

3. The method according to claim 2, wherein, The preset approach direction threshold defines the range of approach angles relative to the actuation surface of the virtual object, and wherein determining whether the detected virtual contact corresponds to intentional or unintentional virtual contact in the virtual environment includes: executing the command in the virtual environment when the detected approach direction is within the preset approach angle range.

4. The method according to claim 2, wherein, The preset approach direction threshold is defined relative to one or more actuation regions of the virtual object, and wherein determining whether the detected virtual contact corresponds to intentional or unintentional virtual contact in the virtual environment includes: executing the command in the virtual environment when the detected approach direction is detected within one of the one or more actuation regions of the virtual object.

5. The method according to claim 1, wherein, Detecting the characteristics of the detected virtual contact includes: detecting the approach speed of the detected virtual contact relative to the virtual object, and Determining whether the detected virtual contact corresponds to intentional or unintentional virtual contact in the virtual environment includes comparing the detected approach speed of the virtual contact with a preset approach speed threshold.

6. The method according to claim 5, wherein, The preset approach speed threshold defines a range of approach speeds relative to the virtual object, and wherein determining whether the detected virtual contact corresponds to intentional or unintentional virtual contact in the virtual environment includes: executing the command in the virtual environment when the detected approach speed is within the preset approach speed range.

7. The method according to claim 1, wherein, The characteristic of detecting the detected virtual contact includes: detecting the approach acceleration of the detected virtual contact relative to the virtual object, and Determining whether the detected virtual contact corresponds to intentional or unintentional virtual contact in the virtual environment includes comparing the detected proximity acceleration of the detected virtual contact with a preset proximity acceleration threshold.

8. The method according to claim 7, wherein, The preset proximity acceleration threshold defines a range of proximity accelerations relative to the virtual object, and wherein determining whether the detected virtual contact corresponds to intentional or unintentional virtual contact in the virtual environment includes: executing the command in the virtual environment when the detected proximity acceleration is within the preset proximity acceleration range.

9. The method according to claim 1, wherein, The detected virtual contact is detected at a first surface of the virtual object, the first surface of the virtual object defining an actuating surface of the virtual object, and the method further includes: Detect subsequent virtual contact on a second surface of the virtual object, the second surface defining an inactive surface of the virtual object; Determining that the subsequent virtual contact does not correspond to an intentional virtual contact in the virtual environment; and Ignore the detection of subsequent virtual contacts.

10. The method according to claim 1, wherein, The detection of virtual contact between the selected device and one of the plurality of virtual objects includes: The virtual object was detected to be outside the user's field of view; Determining that the detected virtual contact does not correspond to intentional virtual contact in the virtual environment; and The detection of the virtual contact is ignored.

11. The method according to claim 1, wherein, The detection of virtual contact between the selected device and one of the plurality of virtual objects includes: A change in user field of view from a first field of view to a second field of view is detected, wherein virtual contact with the virtual object is detected within the first field of view, and the virtual object is outside the second field of view; Determining that the detected virtual contact corresponds to an intentional virtual contact with respect to the virtual object, which is outside the second field of view; and The command is executed in response to the detected virtual contact.

12. The method according to claim 1, wherein, The selection device is at least one of a plurality of body parts of the user. Detecting virtual contact includes: tracking the position and orientation of the user's plurality of body parts relative to the plurality of virtual objects in the virtual environment, wherein the detected virtual contact is detected between one of the user's plurality of body parts and the virtual object, and Determining whether the detected virtual contact corresponds to intentional or unintentional virtual contact in the virtual environment includes: Determine the context of the virtual environment when the virtual contact is detected; Determine whether one of the plurality of body parts matches the virtual object to actuate the virtual object within the defined virtual environment; If one of the plurality of body parts matches the virtual object, the command is executed in the virtual environment in response to the detected virtual contact; and If one of the multiple body parts does not match the virtual object, the detected virtual contact is ignored.

13. The method according to claim 1, wherein, The selection device is at least one of a plurality of body parts of the user. Detecting virtual contact includes: tracking the position and orientation of the user's plurality of body parts relative to the plurality of virtual objects in the virtual environment, wherein the detected virtual contact is detected between one of the user's plurality of body parts and the virtual object, and Determining whether the detected virtual contact corresponds to intentional or unintentional virtual contact in the virtual environment includes: The detection of the virtual contact is determined based on the tracked position and orientation of one of the multiple body parts when the virtual contact is detected, and whether the detected virtual contact is detected between the actuating part of the one of the multiple body parts and the actuating part of the virtual object. If the detected virtual contact occurs between the actuating portion of one of the plurality of body parts and the virtual object, the command is executed in the virtual environment in response to the detected virtual contact; and If the detected virtual contact is not between the actuating portion of one of the plurality of body parts and the virtual object, then the detected virtual contact is ignored.

14. A system for processing input in a virtual environment, the system comprising: A computing device configured to generate a virtual environment, the computing device comprising: a memory storing executable instructions; and a processor configured to execute the instructions to cause the computing device to: A virtual environment is generated for interaction, the virtual environment including the display of multiple virtual objects; The detection of virtual contact characteristics between a selection device and one of the plurality of virtual objects includes: tracking the position and orientation of the selection device relative to the plurality of virtual objects, the characteristics including one or more of the following: the approach direction of the selection device relative to the virtual object at the virtual contact point, the approach velocity of the selection device relative to the virtual object at the virtual contact point, or the approach acceleration of the selection device relative to the virtual object at the virtual contact point; Determining whether a detected virtual contact corresponds to intentional or unintentional virtual contact in the virtual environment is based on the characteristics of the detected virtual contact and the context associated with the specific situation in the virtual environment at the time the virtual contact was detected; and When it is determined that the detected virtual contact corresponds to an intentional virtual contact in the virtual environment Select the virtual object to actuate, and Execute actions within the virtual environment corresponding to the selected virtual object and the commands corresponding to the detected virtual contact; and When it is determined that the detected virtual contact corresponds to an unintentional virtual contact in the virtual environment, the detection of the virtual contact is ignored.

15. The system according to claim 14, wherein, In determining whether the detected virtual contact corresponds to an intentional or unintentional virtual contact in the virtual environment, the processor is further configured to execute the instructions to cause the computing device to: The characteristics of the detected virtual contact are detected, including one or more of the following: the approach direction of the selection device relative to the virtual object at the virtual contact point, the approach velocity of the selection device relative to the virtual object at the virtual contact point, and the approach acceleration of the selection device relative to the virtual object at the virtual contact point.

16. The system according to claim 15, wherein, When determining whether the detected virtual contact corresponds to an intentional or unintentional virtual contact in the virtual environment based on the detected characteristics of the virtual contact, the processor is further configured to execute the instructions to cause the computing device to perform at least one of the following operations: The detected approach direction of the detected virtual contact is compared with a preset approach direction range, and if the detected approach direction of the detected virtual contact is within the preset approach direction range, the command is executed in the virtual environment. The detected approach speed of the detected virtual contact is compared with a preset approach speed range, and if the detected approach speed of the detected virtual contact is within the preset approach speed range, the command is executed in the virtual environment. Alternatively, the detected approach acceleration of the detected virtual contact can be compared with a preset approach acceleration range, and if the detected approach acceleration of the detected virtual contact is within the preset approach acceleration range, then the command is executed in the virtual environment.

17. The system according to claim 14, wherein, The detected virtual contact is detected at a first surface of the virtual object, the first surface defining an actuating surface, and the processor is further configured to execute the instructions to cause the computing device to: Detect subsequent virtual contact on a second surface of the virtual object, the second surface defining an inactive surface of the virtual object; Determining that the subsequent virtual contact does not correspond to an intentional virtual contact in the virtual environment; and Ignore the detection of subsequent virtual contacts.

18. The system according to claim 14, wherein, When detecting virtual contact between the selected device and one of the plurality of virtual objects, the processor is further configured to execute the instructions to cause the computing device to: The virtual object was detected to be outside the user's field of view; Determining that the detected virtual contact does not correspond to intentional virtual contact in the virtual environment; and The detection of the virtual contact is ignored.

Images