Systems and methods for modifying security boundaries of virtual reality systems

By receiving reference elevation and user direction indicator data from the real world environment and establishing and modifying virtual boundaries, it solves the problem that users find it difficult to recognize real world obstacles in the virtual reality system, and achieves a safe virtual reality experience.

CN119024995APending Publication Date: 2024-11-26CTRL-LABS CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411065545.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-10-31
Filing Date
2018-12-20
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In virtual reality systems, users find it difficult to realize obstacles in real-world environments when wearing head-mounted monitors, resulting in possible collisions.

Method used

By receiving the reference elevation of the real world environment, establishing a virtual boundary, and modifying and correcting the virtual boundary based on the user's direction indicator data, ensuring that the virtual boundary maintains a safe distance from obstacles in the real world environment.

Benefits of technology

Effectively prevent users from colliding with obstacles in real-world environments, improves security in the virtual reality experience, and does not require additional sensors or hardware resources, and will not affect computing performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119024995A_ABST
    Figure CN119024995A_ABST
Patent Text Reader

Abstract

The invention relates to a system and method for modifying security boundaries of a virtual reality system. A disclosed computer-implemented method may include receiving an indication of a reference elevation representing a plane of a real-world environment, and establishing a virtual boundary of a virtual-world environment relative to the reference elevation. The method may include receiving a request from a user to modify a virtual boundary, and in response to the request from the user, monitoring an orientation of the direction indicator to generate orientation data. The method may also include modifying the virtual boundary based on the reference elevation and orientation data. Various other methods, systems, and computer readable media are also disclosed.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of an application with a filing date of December 20, 2018, application number 201880100579.5, and invention name “System and method for modifying the safety boundary of a virtual reality system”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Nonprovisional Application No. 16 / 177,424, filed on October 31, 2018, the disclosure of which is incorporated by reference in its entirety.

[0004] background

[0005] Virtual reality (VR) systems and augmented reality (AR) systems can enable users to have a more immersive experience than what ordinary television or video games can provide. When wearing a head-mounted display (HMD), a user can view different parts of a captured scene or an artificially generated scene simply by orienting his or her head as the user would naturally do to view a real-world environment. The scene can be presented to the user in the HMD based on the position and orientation of the user's head, so that the scene changes based on changes in the position and orientation of the user's head. Mobile VR systems can also take into account the movement of the user as he or she walks around in the real-world environment, so that the user feels that he or she is moving in the virtual environment.

[0006] While immersive, these features may allow the user to engage with the virtual environment in a manner that causes the user to be oblivious to important aspects of the user's real-world environment. For example, a user attempting to walk from one location to another in a virtual environment may fail to account for (or see) real-world obstacles, such as tables, couches, or walls, due to the user's lack of awareness of the real-world environment. This may result in collisions with the real-world environment or features in the real-world environment.

[0007] Overview

[0008] As will be described in more detail below, the present disclosure describes systems and methods that may enable a user wearing an HMD device to modify a virtual boundary in a real-world environment, which may be used to prevent collisions with features or obstacles in the real-world environment.

[0009] In one example, a computer-implemented method for modifying a virtual boundary may include: (1) receiving an indication of a reference elevation of a plane representing a real world environment, (2) establishing a virtual boundary of a virtual world environment relative to the reference elevation, (3) receiving a request from a user to modify the virtual boundary, (4) in response to the request from the user, monitoring an orientation of a direction indicator to generate orientation data, and (5) modifying the virtual boundary based on the reference elevation and the orientation data.

[0010] In some examples, the orientation data may include elevation data of the elevation of the direction indicator relative to a reference elevation. Modifying the virtual boundary may include determining an intersection between the plane and a virtual line extending from the user device at the elevation indicated by the elevation data and at the orientation indicated by the orientation data.

[0011] In some examples, the method may include capturing a view of a real-world environment with an imaging system of a head-mounted display system. The captured view may have lens-induced distortion. The method may also include correcting the lens-induced distortion in the captured view to produce a compensated view of the real-world environment, superimposing a virtual boundary on the compensated view of the real-world environment, and displaying the compensated view of the real-world environment in a display of the head-mounted display system during the boundary modification state.

[0012] In some examples, when the virtual boundary meets the minimum area threshold, the virtual boundary can be displayed in a success color, and when the virtual boundary does not meet the minimum area threshold, the virtual boundary can be displayed in a warning color. In some examples, the virtual boundary can be displayed as a filled-in shape defined by the virtual boundary.

[0013] In some examples, modifying the virtual boundary may include adding a portion to the virtual boundary or subtracting a portion from the virtual boundary. In some examples, the method may include receiving a confirmation of the reference elevation. In some examples, the method may include receiving an indication to reset the virtual boundary.

[0014] In some examples, the above methods may be encoded as computer-readable instructions on a computer-readable medium. For example, the computer-readable medium may include one or more computer-executable instructions that, when executed by at least one processor of a computing device, may cause the computing device to (1) receive an indication of a reference elevation of a plane representing a real-world environment, (2) establish a virtual boundary of a virtual-world environment relative to the reference elevation, (3) monitor an orientation of a direction indicator to generate orientation data, and (4) modify the virtual boundary based on the reference elevation and the orientation data.

[0015] In some examples, the orientation data may include elevation data of the elevation of the direction indicator relative to a reference elevation. Modifying the virtual boundary may include determining an intersection between the plane and a virtual line extending from the direction indicator at the elevation indicated by the elevation data and at the orientation indicated by the orientation data.

[0016] In some examples, the instructions may include instructions for capturing a view of a real-world environment using an imaging system of a head-mounted display system. The captured view may have lens-induced distortion. The instructions may include instructions for correcting the lens-induced distortion in the captured view to produce a compensated view of the real-world environment, overlaying a virtual boundary as a filled shape defined by the virtual boundary on the compensated view of the real-world environment, and displaying the compensated view of the real-world environment in a display of the head-mounted display system during the boundary modification state.

[0017] In some examples, when the virtual boundary meets the minimum area threshold, the virtual boundary may be displayed in a success color, and when the virtual boundary does not meet the minimum area threshold, the virtual boundary may be displayed in a warning color.

[0018] In some examples, modifying the virtual boundary may include adding a portion to the virtual boundary or subtracting a portion from the virtual boundary. In some examples, the instructions may include instructions for receiving confirmation of the reference elevation. In some examples, the instructions may include instructions for receiving an indication to reset the virtual boundary.

[0019] In addition, the head mounted display system may include a display secured to a user attachment system, a directional indicator, and a processing system. The processing system may be configured to (1) identify a reference elevation representing a plane of a real world environment, (2) establish a virtual boundary of a virtual world environment relative to the reference elevation, and (3) modify the virtual boundary based on the reference elevation and orientation data representing an orientation of the directional indicator.

[0020] In some examples, the processing system may be further configured to capture a view of the real-world environment using an imaging system of the head-mounted display system. The captured view may have lens-induced distortion. The processing system may also be configured to correct the lens-induced distortion in the captured view to generate a compensated view of the real-world environment, overlay the virtual boundary as a filled shape defined by the virtual boundary on the compensated view of the real-world environment, and display the compensated view of the real-world environment in a display of the head-mounted display system during the boundary modification state.

[0021] In some examples, when the virtual boundary meets the minimum area threshold, the virtual boundary may be displayed in a success color, and when the virtual boundary does not meet the minimum area threshold, the virtual boundary may be displayed in a warning color.

[0022] In some examples, modifying the virtual boundary may include adding a portion to the virtual boundary or subtracting a portion from the virtual boundary. In some examples, the orientation data may include elevation data of an elevation of the direction indicator. Modifying the virtual boundary may include determining an intersection between the plane and a virtual line extending from the user device at the elevation indicated by the elevation data and at the orientation indicated by the orientation data.

[0023] Features from any of the above-mentioned embodiments may be used in combination with each other according to the general principles described herein. These and other embodiments, features and advantages will be more fully understood when the following detailed description is read in conjunction with the accompanying drawings and claims.

[0024] In an embodiment according to the present invention, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to perform a method according to the present invention or any of the above-mentioned embodiments.

[0025] In an embodiment according to the present invention, a system may include: one or more processors; and at least one memory coupled to the processor and including instructions executable by the processor, and the processor is operable to perform a method according to the present invention or any of the above-mentioned embodiments when executing the instructions.

[0026] In an embodiment according to the present invention, a computer program product, preferably comprising a computer readable non-transitory storage medium, is operable to perform a method according to the present invention or any of the above mentioned embodiments when executed on a data processing system.

[0027] Embodiments according to the invention are specifically disclosed in the attached claims, which relate to a method, a storage medium and a head mounted display (HMD), wherein any feature mentioned in one claim category (e.g., a method) may also be claimed in another claim category (e.g., an HMD, a system and a computer program product). The dependencies or back-references in the attached claims are selected for formal reasons only. However, any subject matter resulting from an intentional back-reference (particularly multiple references) to any preceding claim may also be claimed, so that any combination of the claims and their features is disclosed and may be claimed, regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features as set forth in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. In addition, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any feature of the attached claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings illustrate many exemplary embodiments and are a part of the specification. Together with the following description, these drawings demonstrate and explain various principles of the present disclosure.

[0030] Figure 1 is a flow chart of an exemplary method for modifying a virtual boundary relative to a real-world environment in accordance with aspects of the present disclosure.

[0031] Figure 2 is a block diagram of an exemplary head mounted display (HMD) system according to aspects of the present disclosure.

[0032] Figure 3 According to various aspects of the present disclosure, it may include Figure 2 A perspective view of an exemplary HMD device in an HMD system of FIG.

[0033] Figure 4 According to various aspects of the present disclosure, it may include Figure 2 A perspective view of an exemplary handheld controller in an HMD system.

[0034] Figure 5A and Figure 5B The user wearing the device in a real-world environment according to various aspects of the present disclosure is respectively presented. Figure 3 HMD device and hold it Figure 4 Perspective and top views of a handheld controller.

[0035] Fig. 6Aand Figure 6B Perspective and top views are respectively presented of a user interacting with a reproduction of a real-world environment to generate a virtual safety perimeter in accordance with aspects of the present disclosure.

[0036] Figure 6C is a diagram depicting a system for interacting with a real-world environment to define a virtual boundary in accordance with aspects of the present disclosure.

[0037] Fig. 7A and Figure 7B A perspective view and a top view are respectively presented of a user continuing to interact with a rendering of a real-world environment to generate a virtual boundary in accordance with aspects of the present disclosure.

[0038] Fig. 8A and Figure 8B Perspective and top views are presented, respectively, of a user interacting with a defined virtual boundary in accordance with aspects of the present disclosure.

[0039] Fig. 9 is a diagram containing a set of physically defined memory devices corresponding to a set of predefined virtual boundaries in accordance with aspects of the present disclosure.

[0040] In all drawings, the same reference symbols and descriptions represent similar but not necessarily identical elements. Although the exemplary embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been illustrated by way of example in the drawings and will be described in detail herein. However, the exemplary embodiments described herein are not intended to be limited to the specific forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives that fall within the scope of the appended claims.

[0041] Detailed Description of Exemplary Embodiments

[0042] The present disclosure generally relates to systems and methods for allowing a user to modify a virtual boundary relative to a user's real world environment and / or virtual environment. As will be explained in more detail below, embodiments of the present disclosure may allow a user to modify a virtual boundary on a floor of a real world environment by using, for example, a handheld controller, a finger, an eye gaze, or other directional indicator. The user may "draw" (e.g., virtually draw using a directional indicator) modifications to the virtual boundary. When the user wears an HMD device, a view of the real world environment may be provided by one or more cameras disposed on the HMD device. These cameras may allow a pass-through view of the real world environment to be displayed as if the user were not wearing the HMD device. In this way, the user can see features of the environment and obstacles to avoid, and may define and modify virtual boundaries that are a safe distance from these features.

[0043] In some embodiments, a user may use a directional indicator to point to a location on the floor of a real-world environment. For example, the HMD system may include a handheld controller that can be used to point to the floor. The processing subsystem may use the position and orientation information of the handheld controller to identify a specific point on the floor, such as by using the height of the handheld controller above the floor and the orientation of the handheld controller. During the boundary modification state, a virtual boundary may be displayed on the floor, and when the user modifies the virtual boundary on the floor, a virtual line may appear to the user to extend from the handheld controller to the floor, thereby providing visual feedback to the user.

[0044] After modifying the virtual boundary, whenever the user enters within a threshold distance of the virtual boundary, an indication of the virtual boundary and / or a boundary wall derived from the virtual boundary can be presented to the user in the HMD device to make the user aware of the real world environment and / or prevent the user from tripping, falling, or knocking over objects. Relying on the virtual boundary, the user can keep the HMD device turned on and move safely in the real world environment to better participate in the virtual environment presented to the user during the operating state. The aspects described herein can improve VR and AR technology by providing safety features without the need for additional sensors or dedicated hardware. In addition, the aspects described herein can improve the functionality of a computer by providing safety features that can be implemented without the need for additional resources. For example, the safety features can be implemented without requiring a large amount of processing and memory resources from the computer, and therefore will not negatively affect computing performance.

[0045] refer to Figure 1-Figure 9 , a detailed description of systems and methods is provided below that allow a user to modify his or her own virtual boundaries relative to a real-world environment and use the virtual boundaries to provide visual and / or audible indications to the user to make the user aware of the real-world environment when the risk of accidental collision becomes too high. Figure 1 An exemplary process for modifying a virtual boundary is shown. Figure 2 An exemplary VR system is shown. Figure 3 An exemplary HMD device is shown. Figure 4 An exemplary direction indicator is shown. Figure 5A-Figure 5B A user using an exemplary AR system is shown. Figure 6A-6C It shows how a user can interact with a reproduction of a real-world environment. Figure 7A-7B Shows how the user can modify the virtual boundary. Figure 8A-B Shown is how a user can interact with the virtual boundary. Fig. 9 An exemplary memory device for a VR system is shown.

[0046] Figure 1 is a flow chart of an exemplary computer-implemented method 100 for modifying a virtual boundary. Figure 1 The steps shown may be implemented by any suitable computer executable code and / or computing system (including Figure 2-Figure 4 In one example, Figure 1 Each step shown may represent an algorithm whose structure includes and / or is represented by multiple sub-steps, examples of which are provided in more detail below.

[0047] like Figure 1 As shown, at step 110, one or more systems described herein may receive an indication of a reference elevation of a plane representing a real-world environment. Figure 2 As shown, the processing subsystem 210 of the HMD system 200 may receive an indication of a reference elevation.

[0048] In some embodiments, the term "reference elevation" may refer to an elevation corresponding to a reference plane in a real-world environment, which may be used to determine the point / position at which a direction indicator is pointing. In some embodiments, the reference elevation may be a baseline elevation corresponding to a base plane of the real-world environment. The term "base plane" may refer to the lowest elevation that a user may encounter in the real-world environment. Examples of base planes include, but are not limited to, floors, tables, ground, etc. The baseline elevation may represent the elevation of the base plane and may be defined as being perpendicular to the floor. The virtual boundaries of the VR system may be defined and / or modified relative to the reference elevation and / or corresponding plane. Figure 2 An exemplary VR system that may utilize a virtual boundary is shown.

[0049] Figure 2 1 is a block diagram of one embodiment of an HMD system 200 that presents a scene (e.g., a captured scene, an artificially generated scene, or a combination thereof) to a user. The HMD system 200 may operate in a virtual reality (VR) system environment, an augmented reality (AR) system environment, a mixed reality (MR) system environment, or some combination thereof. Figure 2 The illustrated HMD system 200 may include an HMD device 205 that includes or is in communication with a processing subsystem 210 and an input / output (I / O) interface 215. In some embodiments, the HMD device 205 may completely block a user's view of a real-world environment. Other embodiments may only partially block a user's view of a real-world environment and / or may block a user's view, depending on what is displayed in a display of the HMD device 205.

[0050] Although Figure 2An exemplary HMD system 200 is shown that includes at least one HMD device 205 and at least one I / O interface 215, but in other embodiments, any number of these components may be included in the HMD system 200. For example, there may be multiple HMDs 205, each HMD 205 having an associated I / O interface 215, and each HMD device 205 and I / O interface 215 communicating with the processing subsystem 210. In embodiments where the processing subsystem 210 is not included within or integrated with the HMD device 205, the HMD device 205 may communicate with the processing subsystem 210 via a wired connection or a wireless connection. In alternative configurations, different and / or additional components may be included in the HMD system 200. Furthermore, in some embodiments, the processing subsystem 210 may be integrated with the HMD device 205. Figure 2 The functionality described by one or more components shown may be different in combination. Figure 2 The described approach is distributed across components.

[0051] The HMD device 205 can present various content to the user, including virtual views of artificially reproduced virtual world environments and / or augmented views of physical, real-world environments augmented with computer-generated elements (e.g., two-dimensional (2D) or three-dimensional (3D) images, 2D or 3D video, sound, etc.). In some embodiments, the presented content includes audio presented via an internal or external device (e.g., a speaker and / or a headset) that receives audio information from the HMD device 205, the processing subsystem 210, or both, and presents audio data based on the audio information. In some embodiments, such speakers and / or headsets can be integrated into the HMD device 205 or releasably coupled or attached to the HMD device 205. The HMD device 205 may include one or more bodies that may be rigidly or non-rigidly coupled together. Rigid coupling between rigid bodies may allow the coupled rigid bodies to act as a single rigid entity. Conversely, non-rigid coupling between rigid bodies may allow the rigid bodies to move relative to each other. One embodiment of the HMD device 205 is Figure 3 HMD device 300 is shown and described in further detail below.

[0052] In some examples, the HMD device 205 may include a depth sensing subsystem 220 (or depth camera system), an electronic display 225, an image capture subsystem 230 including one or more cameras, one or more position sensors 235, and / or an inertial measurement unit (IMU) 240. Other embodiments of the HMD device 205 may include an optional eye tracking or gaze estimation system configured to track the eyes of a user of the HMD device 205 to estimate the user's gaze. An optional zoom module may be configured to adjust the focus of one or more images displayed on the electronic display 225 based on determined eye tracking information obtained from the eye tracking system and other components. Some embodiments of the HMD device 205 may have different features than those incorporating the eye tracking system. Figure 2 Describes the components.

[0053] The depth sensing subsystem 220 may capture data describing depth information that characterizes a local real-world area or environment surrounding some or all of the HMD device 205, and / or characterizes the position, velocity, or location of the depth sensing subsystem 220 (and thereby the HMD device 205) within the local area. The depth sensing subsystem 220 may use the collected data (e.g., based on the captured light according to one or more computer vision schemes or algorithms, by processing a portion of a structured light pattern, by time-of-flight (ToF) imaging, simultaneous localization and mapping (SLAM), etc.) to calculate the depth information or the depth sensing subsystem 220 may transmit the data to another device, such as an external implementation of the processing subsystem 210, which may use the data from the depth sensing subsystem 220 to determine the depth information.

[0054] The electronic display 225 can display a two-dimensional or three-dimensional image to the user based on the data received from the processing subsystem 210. In various embodiments, the electronic display 225 includes a single electronic display or multiple electronic displays (e.g., a display for each eye of the user). Examples of the electronic display 225 may include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an inorganic light emitting diode (ILED) display, an active matrix organic light emitting diode (AMOLED) display, a transparent organic light emitting diode (TOLED) display, another suitable display, or some combination thereof. The electronic display 225 may be opaque so that the user cannot see the local environment through the electronic display 225.

[0055] The image capture subsystem 230 may include one or more optical image sensors or cameras that capture and collect image data from the local environment. In some embodiments, the sensors included in the image capture subsystem 230 can provide a stereoscopic view of the local environment, and the processing subsystem 210 can use the stereoscopic view to generate image data that characterizes the local environment and / or the position and orientation of the HMD device 205 in the local environment. For example, the image capture subsystem 230 may include a simultaneous localization and mapping (SLAM) camera or other camera including a wide-angle lens system that captures a wider field of view than the user's eyes can capture. As described herein, when the HMD system 200 is in a boundary definition state, the image capture subsystem 230 can provide a direct view of the real world environment, which is displayed to the user via the electronic display 225.

[0056] In some embodiments, the processing subsystem 210 may process images captured by the image capture subsystem 230 to eliminate distortion caused by the lens system of the image capture subsystem 230 and / or by the separation distance between the two image sensors being significantly greater than or significantly less than the average separation distance between the user's eyes. For example, when the image capture subsystem 230 is or is part of a SLAM camera system, the direct image from the image capture subsystem 230 may appear distorted to the user if shown in an uncorrected format. Image correction or compensation may be performed by the processing subsystem 210 to correct the image and present it to the user with a more natural appearance, so that it appears to the user as if the user is viewing through the electronic display 225 of the HMD device 205. In some embodiments, the image capture subsystem 230 may include one or more image sensors with lenses suitable for providing a direct view of the local environment (in terms of field of view, separation distance, etc.). The image capture subsystem 230 may capture color images or monochrome images.

[0057] In some examples, IMU 240 may represent an electronic subsystem that generates data indicating the position and / or orientation of HMD device 205 based on measurement signals received from one or more position sensors 235 and depth information received from depth sensing subsystem 220 and / or image capture subsystem 230. For example, position sensor 235 may generate one or more measurement signals in response to movement of HMD device 205. Examples of position sensor 235 include one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor used for error correction of IMU 240, or some combination thereof. Position sensor 235 may be located outside of IMU 240, inside of IMU 240, or some combination of these.

[0058] Based on one or more measurement signals from one or more position sensors 235, the IMU 240 can generate data indicating an estimated current position, elevation, and / or orientation of the HMD device 205 relative to an initial position and / or orientation of the HMD device 205. For example, the position sensor 235 may include a plurality of accelerometers that measure translational motion (forward / backward, up / down, left / right) and a plurality of gyroscopes that measure rotational motion (e.g., pitch, yaw, roll). As described herein, the image capture subsystem 230 and / or the depth sensing subsystem 220 can generate data indicating an estimated current position and / or orientation of the HMD device 205 relative to a real-world environment in which the HMD device 205 is used.

[0059] The I / O interface 215 may represent a subsystem or device that allows a user to send action requests and receive responses from the processing subsystem 210 and / or the direction indicator 270. In some embodiments, the direction indicator 270 may include a handheld controller or other device that can be manipulated by the user to provide input to the I / O interface 215, such as orientation and / or position data sensed by a sensor of the direction indicator 270. In other embodiments, the direction indicator 270 may provide passive input to the I / O interface 215. For example, the direction indicator 270 may include a finger or hand of the user, a glove or other wearable object, a handheld object, the user's eyes and / or gaze, and / or another user-manipulable object that can be detected by a sensor of the HMD system 200 to determine orientation and / or position data relative to the direction indicator 270. In some embodiments, the I / O interface 215 may facilitate communication with more than one direction indicator 270. For example, a user may have two direction indicators 270, one for each hand. In some examples, an action request may represent a request to perform a specific action. For example, an action request may be an instruction to start or end the capture of image or video data, or an instruction to perform a specific action within an application or to start or end a boundary-defined state. The I / O interface 215 may include one or more input devices, or may be capable of communicating with one or more input devices. Exemplary input devices may include a keyboard, a mouse, a handheld controller, or any other suitable device for receiving an action request and transmitting the action request to the processing subsystem 210.

[0060] The action request received by the I / O interface 215 can be passed to the processing subsystem 210, and the processing subsystem can perform an action corresponding to the action request. In some embodiments, the direction indicator 270 includes an IMU 240 that captures inertial data indicating an estimated position of the direction indicator 270 relative to an initial position. In some embodiments, the I / O interface 215 and / or the direction indicator 270 can provide tactile feedback to the user according to instructions received from the processing subsystem 210 and / or the HMD device 205. For example, tactile feedback is provided when an action request is received or the processing subsystem 210 transmits an instruction to the I / O interface 215 so that the I / O interface 215 generates or directly generates tactile feedback when the processing subsystem 210 performs an action.

[0061] The processing subsystem 210 may include one or more processing devices or physical processors that provide content to the HMD device 205 based on information received from one or more of the depth sensing subsystem 220, the image capture subsystem 230, the I / O interface 215, and the direction indicator 270. Figure 2 In the example shown, the processing subsystem 210 includes an engine 260, an application storage 250, and a tracking module 255. Some embodiments of the processing subsystem 210 have different Figure 2 Similarly, the functions described further below may be combined with Figure 2 The described approach is distributed among the components of the HMD system 200 .

[0062] The application storage 250 may store one or more applications executed by the processing subsystem 210. In some examples, an application may represent a set of instructions that, when executed by a processor, generates content for presentation to a user. The content generated by the application may be generated in response to input received from the user via movement of the HMD device 205 or the directional indicator 270. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

[0063] The tracking module 255 may calibrate the HMD system 200 using one or more calibration parameters, and may adjust one or more calibration parameters to reduce errors in determining the position of the HMD device 205 or the direction indicator 270. For example, the tracking module 255 may transmit the calibration parameters to the depth sensing subsystem 220 to adjust the focus of the depth sensing subsystem 220 to more accurately determine the position of the structured light elements captured by the depth sensing subsystem 220. The calibration performed by the tracking module 255 may also take into account information received from an IMU 240 in the HMD device 205 and / or another IMU 240 included in the direction indicator 270. In addition, if tracking of the HMD device 205 is lost (e.g., the depth sensing subsystem 220 loses sight of at least a threshold number of structured light elements), the tracking module 255 may recalibrate some or all of the HMD system 200.

[0064] The tracking module 255 may use information from the depth sensing subsystem 220, the image capture subsystem 230, the one or more position sensors 235, the IMU 240, or some combination thereof to track the movement of the HMD device 205 or the direction indicator 270. For example, the tracking module 255 may determine the location of a reference point of the HMD device 205 in a mapping of the real-world environment based on information collected with the HMD device 205. Additionally, in some embodiments, the tracking module 255 may use the portion of data indicating the location and / or orientation of the HMD device 205 and / or the direction indicator 270 from the IMU 240 to predict the future location and / or orientation of the HMD device 205 and / or the direction indicator 270. The tracking module 255 may also provide the estimated or predicted future location of the HMD device 205 or the I / O interface 215 to the engine 260.

[0065] In some embodiments, the tracking module 255 can track other features that can be observed by the depth sensing subsystem 220, the image capture subsystem 230, and / or another system. For example, the tracking module 255 can track one or both hands of the user so that the positioning of the user's hands in the real world environment can be known and utilized. The tracking module 255 can receive and process data to determine, for example, the pointing of a finger of one of the user's hands when the direction indicator 270 includes the user's hand. The tracking module 255 can also receive information from one or more eye tracking cameras included in some embodiments of the HMD device 205 to track the user's gaze.

[0066] Image processing engine 260 can generate a three-dimensional map of an area surrounding some or all of HMD device 205 (i.e., a "local area" or "real world environment") based on information received from HMD device 205. In some embodiments, engine 260 determines depth information for the three-dimensional map of the local area based on information received from depth sensing subsystem 220 related to the techniques used to calculate depth. Engine 260 can calculate depth information using one or more techniques for calculating depth from structured light. In various embodiments, engine 260 uses the depth information to, for example, update a model of the local area and generate content based in part on the updated model.

[0067] The engine 260 may also execute an application within the HMD system 200 and receive position information, acceleration information, velocity information, predicted future position, or some combination thereof of the HMD device 205 from the tracking module 255. Based on the received information, the engine 260 may determine the content to be provided to the HMD device 205 for presentation to the user. For example, if the received information indicates that the user has looked to the left, the engine 260 generates content for the HMD device 205 that corresponds to the user's movement in a virtual environment or in an environment that enhances a local area with additional content. In addition, the engine 260 may perform an action within an application executed on the processing subsystem 210 in response to an action request received from the I / O interface 215 and / or the direction indicator 270, and provide feedback to the user that the action has been performed. The feedback provided may be visual or auditory feedback via the HMD device 205, or tactile feedback via the direction indicator 270.

[0068] Back to Figure 1 , the systems described herein may perform step 110 in a variety of ways. In one example, the reference elevation may have been pre-established and stored. In other examples, the reference elevation may be set by a user, such as by using the HMD device 300 and / or handheld controller 400 to define and / or confirm the reference elevation. For example, the HMD system 200 may receive an indication of the reference elevation from the HMD device 300 and / or handheld controller 400.

[0069] Figure 3is a schematic diagram of an HMD device 300 according to one embodiment of the HMD device 205. The HMD device 300 may include an imaging subsystem and a depth sensing subsystem. The HMD device 300 may be, for example, part of a VR system, an AR system, an MR system, or some combination thereof. In embodiments describing an AR system and / or an MR system, a portion of a front side 302 of the HMD device 300 is at least partially transparent in the visible light band (approximately 380 nanometers (nm) to 750 nm). More specifically, a portion of the HMD device 300 located between the front side 302 of the HMD device 300 and the user's eyes may be at least partially transparent (e.g., a partially transparent electronic display 225). In other embodiments, the front side 302 is opaque, preventing the user from seeing the real world environment. The HMD device 300 may include a front rigid body 305 that houses the electronic display 225 and other components, a user attachment system such as a band 310 that secures the HMD device 300 to the user's head, and a reference point 315 that may characterize the position and / or orientation of the HMD device 300.

[0070] In addition, HMD device 300 may include imaging aperture 320 and illumination aperture 325. An illumination source included in depth sensing subsystem 220 may emit light (e.g., structured light) through illumination aperture 325. An imaging device of depth sensing subsystem 220 may capture light from the illumination source that is reflected or backscattered from a local area through imaging aperture 320. Embodiments of HMD device 300 may also include cameras 340A and 340B, which may be Figure 2 Cameras 340A and 340B may be separated from each other by a distance different than the average separation distance between the user's eyes.

[0071] Front rigid body 305 may include one or more electronic display elements, one or more integrated eye tracking systems, IMU 330, one or more position sensors 335, and reference points 315. IMU 330 may represent an electronic device that generates rapid calibration data based on measurement signals received from one or more position sensors 335. Position sensor 335 may generate one or more measurement signals in response to movement of HMD device 300.

[0072] Figure 4 is a perspective view of an exemplary handheld controller 400 according to some embodiments, which handheld controller 400 may be included in Figure 22. Embodiment of the directional indicator 270 in the HMD system 200. The HMD system 200 may include one or more handheld controllers, such as controller 400. For example, the HMD system 200 may include two handheld controllers 400, one handheld controller 400 for each of the right and left hands of the user. Each handheld controller 400 may be communicatively coupled to the HMD device 205 and / or a computing device (e.g., a personal computer, a processing subsystem 210, etc.). The handheld controller 400 may be communicatively coupled to the HMD device 205 via any suitable wireless and / or wired connection.

[0073] like Figure 4 As shown, the handheld controller 400 may include a grip 402 sized to fit a user's hand. The handheld controller 400 may also include a tracking ring 404 for tracking the position, orientation, and / or movement of the handheld controller 400 relative to the HMD device 205 and / or relative to the real world environment. In some embodiments, the tracking ring 404 may include one or more tracking lights, such as an array of tracking lights 406. The array of tracking lights 406 may include tracking LEDs (e.g., infrared (IR) LEDs) for motion and position tracking purposes to provide 360-degree motion control when using the HMD system 200. The tracking lights 406 may be used to determine the orientation of the controller 400 so that intersections with the floor of the real world environment can be identified in order to "draw" a virtual boundary. The controller 400 may include tracking lights on any suitable portion of the controller 400. In some examples, the tracking lights 406 of the handheld controller 400 may emit light having a wavelength greater than about 700 nm and less than about 900 nm. In one embodiment, tracking light 406 of handheld controller 400 may emit light having a wavelength of approximately 850 nm (e.g., between approximately 840 nm and 860 nm or between approximately 830 nm and 870 nm). In at least one embodiment, cameras 340A and 340B may receive the light emitted by tracking light 406 on handheld controller 400, and tracking module 255 may utilize the received light to determine the position, orientation, and / or movement of handheld controller 400 relative to HMD device 205 and / or another reference frame (e.g., a reference frame of a real-world environment).

[0074] To define the reference elevation, the user may interact with the HMD system 200. For example, the user may be prompted to position the handheld controller 400 in contact with the floor, which may provide a base plane for the real-world environment. The reference elevation may be defined perpendicular to the floor. In some embodiments, components of the HMD device 300 may determine a height above the floor based on the orientation of the HMD device 300 and one or more depth measurements characterizing the distance from the HMD device 300 to the floor. For example, the HMD system 200 may prompt the user to look at the floor of the real-world environment as part of the virtual boundary definition process.

[0075] Some embodiments may further include receiving a confirmation of the reference elevation. An indication of the reference elevation may be presented to the user. For example, the HMD system 200 may show a visual representation of a plane associated with the reference elevation and / or the current height of the HMD device 300 from the reference elevation. The user may then confirm the reference elevation or define / redefine the reference elevation as described above.

[0076] Back to Figure 1 At step 120, one or more systems described herein may establish a virtual boundary of the virtual world environment relative to the reference elevation. For example, the processing subsystem 210 and / or the HMD device 205 and / or the HMD device 300 may determine the virtual boundary based on input from the direction indicator 270 and / or the controller 400 or based on retrieving a previously defined virtual boundary.

[0077] In some embodiments, the term "virtual boundary" may refer to a boundary defined relative to a user's real-world environment and / or a virtual environment. Examples of virtual boundaries include, but are not limited to, the safety boundaries described herein, which may define a safe distance away from obstacles in the real-world environment that a user may encounter during a VR and / or AR experience. Other virtual boundaries may include boundaries indicating an ideal positioning for detection by a sensor, boundaries for limiting a user's movement in a real-world environment, boundaries that may be used to place and / or remove obstacles in a virtual environment, and boundaries for other purposes.

[0078] The systems described herein may perform step 120 in a variety of ways. In one example, the virtual boundary may be loaded from previously stored settings (e.g., default settings) or previous user actions stored in the HMD system 200. In other examples, the user may define the virtual boundary using, for example, the HMD system 200.

[0079] Figure 5A and Figure 5B 5 and 6 respectively present wearing a headset in a real world environment 500 (e.g., a living room) according to some embodiments. Figure 3 HMD device 300 and handheld Figure 44. The real-world environment 500 may include a base plane 502, also referred to as a floor 502, and walls 504A, 504B, 504C, and 504D, collectively referred to as walls 504. The real-world environment 500 may also include a plurality of objects or features within the room that may pose a collision risk when the user's line of sight is blocked by the HMD device 300. For example, the environment 500 may include a fireplace 506A with a protruding mantel 506B, a table 506C, and shelves 506D. Figure 5B As shown, environment 500 may also include sofa 506E. Objects and features 506A, 506B, 506C, 506D, and 506E may be referred to as features 506 of real-world environment 500 along with wall 504.

[0080] In some embodiments, a user may move within the real-world environment 500 in order to move within the virtual environment displayed in the HMD device 300. In other words, as the user moves within the real-world environment 500, the image shown in the electronic display 225 of the HMD device 300 may be updated based on the user's movement. Thus, as the user moves within the real-world environment 500, the user moves relative to the virtual environment. As described in more detail below, embodiments of the systems and methods described herein may enable a user to define a virtual boundary that may be used to prevent the user from colliding with any feature 506 when the user cannot see the real-world environment (e.g., due to an occlusion of the user's real-world view by the HMD device 300).

[0081] Fig. 6A and Figure 6B The user and Figure 5A and Figure 5B The processing subsystem 210 may interact with the representation 600 of the real-world environment 500 to produce a perspective view and an overhead view of the virtual boundary or safety boundary. Because the user's view of the real-world environment 500 may be completely or partially blocked by the HMD device 300, the representation 600 of the real-world environment 500 may be provided to the user in the electronic display 225 of the HMD device 300. The representation 600 may be generated by the image capture subsystem 230 to provide a direct view of the real-world environment 500. In some embodiments, the processing subsystem 210 may perform image correction on the images captured by the image capture subsystem 230 to remove distortion and provide an improved view of the real-world environment 500. For example, the processing subsystem 210 may perform image correction to mitigate distortion caused by the lenses of the cameras 340A and 340B and / or the separation distance between the cameras 340A and 340B.

[0082] like Fig. 6AAs shown, during a boundary definition state where a user can define a virtual boundary, the user can use the controller 400 as a pointer or direction indicator to select an intersection 602. The intersection 602 can be defined as the location of the intersection between the floor 502 and a virtual line 604 defined by the orientation and position of the controller 400, which is held and manipulated by the user. The virtual line 604 can be displayed to the user in the HMD device 300 so that the user can draw a virtual boundary 606 on the floor 502. When the user manipulates the controller 400, a series of intersections (such as the intersection 602) can be combined to form a virtual boundary 606 extending along the floor 502. Figure 6B A top view within a rendering 600 of the real-world environment 500 is shown, depicting an intersection 602 , a virtual line 604 , and a virtual boundary 606 .

[0083] Figure 6C 602 may be defined according to some embodiments. The position and orientation of the handheld controller in the real world environment 500 may be determined based on subsystems (e.g., IMU and / or position tracker) included in the handheld controller 400. In some cases, the user may be instructed by a message displayed in the HMD device 300 to place the handheld controller 400 in contact with the base plane or floor 502 to prepare for the boundary definition process. The user may then activate a button on the controller 400 or hold the controller 400 still for a predetermined amount of time to indicate to the HMD system 200 that the controller 400 is in contact with the floor 502. The user may then be directed to stand so that both the HMD device 300 and the controller 400 are moved away from the floor 502 by a displacement D1, which may be used as the height or elevation of the controller 400 with respect to the floor 502. In some embodiments, the HMD device 300 may determine its elevation above the floor 502 and the distance between the HMD device 300 and the handheld controller 400. For example, the depth sensing subsystem 220 may determine the distance from the floor 502 to the HMD device 300 using structured light or by using a triangulation estimate of a stereo image. The distance between the HMD device 300 and the handheld controller 400 may be subtracted from the height of the HMD device 300 above the floor 502 to determine the displacement D1 or elevation of the handheld controller 400. The orientation of the controller 400 may be used to determine the angle A1 formed by the axis of the controller 400 and the floor 502. Using the positioning, displacement D1, and angle A1 of the controller 400, the point of intersection between the virtual line 604 and the floor 502 may be used as the intersection point 602. By manipulating the controller 400, multiple intersection points 602 may be identified, and these points 602 may be connected to form a virtual boundary 606. Although Figure 6C Floor 502 is shown as the reference elevation, but in other embodiments the reference elevation may be defined relative to another plane in the real world environment.

[0084] Back to Figure 1 At step 130 , one or more systems described herein may receive a request from a user to modify a virtual boundary. For example, the HMD system 200 may receive a request from a user via the controller 400 to modify a virtual boundary.

[0085] In some embodiments, the HMD system 200 may enter a boundary modification state to monitor input for modifying a virtual boundary. The term "boundary modification state" may refer to a state or stage in which a user may modify a virtual boundary. The boundary modification state may be the same or similar stage as a boundary definition state during which a virtual boundary may be defined, or a separate stage. The boundary modification state may be a specific mode of the HMD system 200 dedicated to modifying a virtual boundary. Alternatively, the boundary modification state may be combined with other operating states of the HMD system 200.

[0086] The systems described herein may perform step 130 in a variety of ways. In one example, the user may be notified that the virtual boundary may be inadequate, and the user may initiate a boundary modification state accordingly. The user may be notified of the inadequacy of the virtual boundary in a variety of ways. For example, the user may receive a visual or audible notification, such as through the HMD device 300. The notification may be an icon or may be a more robust visual indicator.

[0087] In some examples, the system may also capture a view of the real world environment using an imaging system of the head mounted display system. The captured view may have lens induced distortion. The system may correct the lens induced distortion in the captured view to produce a compensated view of the real world environment. The system may overlay the virtual boundary on the compensated view of the real world environment and display the compensated view of the real world environment in a display of the head mounted display system during the boundary modification state. Fig. 7A and Figure 7B An exemplary scenario of a compensated view of a real-world environment presented to a user is shown.

[0088] Fig. 7A and Figure 7B A perspective view and a top view are presented, respectively, of a user interacting with a representation 600 of a real-world environment 500 to produce a virtual boundary 606 in accordance with some embodiments. The virtual boundary 606 may define a boundary region 608. The boundary region 608 may be displayed, for example, by the HMD device 300 as a filled shape defined by the virtual boundary 606. The boundary region 608 may be displayed similar to the virtual boundary 606, such as a solid color, or may be a shaded or semi-transparent color corresponding to the virtual boundary 606.

[0089] In some examples, the area defined by the virtual boundary 606, such as the boundary area 608, can be compared to a minimum area threshold. The minimum area threshold can correspond to a minimum area, such as a minimum square feet, in which a user can be expected to safely operate the VR and / or AR system in the real world environment. The minimum area threshold can be a predetermined value or a default value, or can be dynamically determined in the real world environment. The minimum area threshold can include size constraints, such as minimum and / or maximum length and / or width.

[0090] In some examples, the minimum area threshold can be explicitly presented to the user, such as the amount of square feet that the minimum area threshold is to be met or required. In other examples, the minimum area threshold can be represented by displaying the boundary area in a success color when the minimum area threshold is met, or displaying the boundary area in a warning color when the minimum area threshold is not met. For example, when the minimum area threshold is met, the virtual boundary 606 and / or the boundary area 608 can be displayed in green, and when the minimum area threshold is not met, the virtual boundary 606 and / or the boundary area 608 can be displayed in red.

[0091] Upon seeing that the virtual boundary 606 does not meet the minimum area threshold, the user can initiate a boundary modification state to modify the virtual boundary. Alternatively, if the minimum area threshold is not met, the boundary modification state can be automatically initiated. For example, during the boundary definition state, if the virtual boundary 606 does not meet the minimum area threshold, the boundary modification state can be automatically initiated. In addition, the user may initiate a boundary modification state for other reasons. For example, the real world environment may have changed, or the user may wish to fine-tune the virtual boundary, for example if the user prefers a smaller or larger boundary area.

[0092] Back to Figure 1 At step 140, one or more systems described herein may monitor the orientation of the direction indicator to generate orientation data in response to a request from a user. For example, the processing subsystem 210 and / or the HMD device 205 and / or the HMD device 300 may determine the orientation data based on input from the direction indicator 270 and / or the controller 400. In some examples, the orientation data may include and / or be associated with additional input data. For example, the orientation data may include elevation data of the direction indicator relative to a reference elevation. The orientation data may include position data of the direction indicator.

[0093] The systems described herein can perform step 140 in a variety of ways. In one example, the position and orientation of the handheld controller 400 in the real world environment 500 can be determined based on subsystems (e.g., IMU and / or position sensors) included in the handheld controller 400. Figure 6CAs described, D1 may be monitored to determine elevation data, and A1 may be monitored to determine orientation data. Alternatively, the elevation data and orientation data may be determined from sensors external to the controller 400 , such as a camera viewing the controller 400 in the real world environment 500 .

[0094] Back to Figure 1 At step 150, one or more systems described herein may modify the virtual boundary based on the reference elevation and orientation data. For example, the processing subsystem 210 and / or the HMD device 205 and / or the HMD device 300 may modify the virtual boundary based on the elevation data and orientation data from the direction indicator 270 and / or the controller 400 and the reference elevation.

[0095] The systems described herein can perform step 150 in a variety of ways. In one example, an intersection point between a plane and a virtual line extending from the user device at an elevation indicated by the elevation data and an orientation indicated by the orientation data can be determined. Figure 6C As shown, the position of the controller 400, D1, and A1 can define where the virtual line 604 begins, and the point of intersection between the virtual line 604 and the floor 502 can define the intersection 602. Similar to how multiple intersections 602 define a virtual boundary 606, modifications to the virtual boundary 606 can be defined by one or more intersections 602. The intersection 602 can define an extension of the virtual boundary 606.

[0096] The virtual boundary can be modified in various ways. For example, Fig. 7A and Figure 7B 606. The virtual boundary 606 may not meet the minimum area threshold, and the boundary area 608 may be displayed in a warning color. The user can manipulate the controller 400 to define a modification line 610, which may be defined by one or more intersection points 602. Fig. 7A and Figure 7B 6, but modification line 610 may delineate an area surrounding boundary region 608 in order to satisfy a minimum area threshold.

[0097] The virtual border can be modified by adding portions to the virtual border. For example, a portion of virtual border 606 can be extended to add an additional area to border region 608. The additional area can be connected to border region 608, such as to one side of border region 608, or can be a disconnected or otherwise discontinuous area associated with border region 608. The additional area can partially or completely surround border region 608.

[0098] In other examples, modifying the virtual boundary may include subtracting a portion from the virtual boundary. For example, the boundary modification state may include an add state for adding a portion and a subtract state for removing a portion from the virtual boundary. The user may switch between the add state and the subtract state to define and modify the virtual boundary to a desired shape. For example, the user may use the controller 400 to enter the add state and draw an add portion to the virtual boundary 606. The user may use the controller 400 to enter the subtract state and cut out a portion from the virtual boundary 606.

[0099] In yet other examples, modifying the virtual boundary can be achieved by resetting the virtual boundary. For example, the user can send an instruction to reset the virtual boundary. The user can then define the virtual boundary from an empty state. Alternatively, the user can undo / redo one or more modifications.

[0100] The modified virtual boundary can be compared to a minimum area threshold. The comparison can occur dynamically, such as in real time as the user draws the modification. For the purpose of estimating the area, incomplete loops and / or unconnected portions can be automatically connected, such as by connecting the unconnected portions with a straight line of the shortest distance. Thus, the user can see in real time when the minimum area threshold is met. Alternatively, the minimum area comparison can be performed when the user indicates to terminate the boundary modification state. In some examples, the user may not terminate the boundary modification state unless the minimum area threshold is met.

[0101] In addition, when the boundary modification state is terminated, the virtual boundary can be checked for errors. Errors can include, for example, unclosed loops, zero area portions, portions that are too small for the user to reasonably use (which can be compared to a minimum portion area threshold), and other errors. In some examples, errors can be corrected automatically. For example, unclosed loops can be connected, problematic portions can be removed, and so on. In other examples, the user can be warned of the errors, and in some examples, the boundary modification state can be automatically restarted. In some examples, the boundary modification state may not terminate until no errors are detected in the virtual boundary.

[0102] After the user completes the boundary modification state and finalizes the virtual boundary (which may require satisfying a minimum area threshold), the user may interact with the virtual boundary. Fig. 8A and Figure 8B A perspective view and a top view are respectively presented of a user interacting with a virtual boundary defined in a virtual environment according to some embodiments. Fig. 8A and Figure 8BAs shown, a virtual environment 800 can be displayed to a user in an HMD device 300. The virtual environment 800 can represent a captured scene in which a user can move or an artificial environment such as in a video game. The user can move in the virtual environment 800 by moving in the real world environment 500. The virtual environment can include a visual indication of a virtual boundary 606. In other words, the virtual boundary 606 can be visually rendered and presented to the user so that the user can see the virtual boundary 606 whenever the field of view of the HMD device 300 includes the virtual boundary 606. In some embodiments, the user can select a setting so that the virtual boundary 606 always appears, or the virtual boundary 606 only appears when the user is within a threshold distance. Such a threshold distance can depend on the speed or movement of the user and / or the specific part of the user or HMD system 200 that is closest to the boundary.

[0103] In some embodiments, as the user approaches the virtual boundary 606, a boundary wall 802 (shown as elements 802A and 802B) may be rendered in the HMD device 300 to warn the user that he or she is approaching the virtual boundary 606, and therefore warn the user that he or she is approaching features 506 in the real-world environment 500 that present a risk of collision. The boundary wall 802 may be rendered as a series of vertical or horizontal bars, a grid of lines, a grid of dots, etc., which may allow the user to continue to view a portion of the virtual environment 800 through the boundary wall 802. In other embodiments, the boundary wall 802 may be rendered in a manner that completely “blocks” the user's view of a portion of the virtual environment 800. In some embodiments, as the user gets closer to the virtual boundary 606, the rendering of the boundary wall 802 may obstruct the user's view by an increasing amount. In other examples, the boundary wall 802 may be overlaid or otherwise rendered on top of a through-view of the real-world environment (e.g., provided by the image capture subsystem 230).

[0104] like Fig. 8A and Figure 8BAs shown, separate wall portions may be rendered in the HMD device 300. Thus, boundary wall 802A and boundary wall 802B are shown. Boundary wall 802A may be rendered when the distance between the user and the virtual boundary 606 is less than or equal to distance D2. Distance D2 may be configured by the user or automatically configured by the HMD system 200. For example, in some embodiments, a threshold distance between 2 and 5 feet may be used. In some embodiments, the threshold distance may be a function of the user's speed. Boundary wall 802B may be rendered in the HMD device 300 based on distance D3 between the controller 400 and the virtual boundary 606 or based on distance D4 between the HMD device 300 and the virtual boundary 606. In some implementations, the position of the user's hands may be monitored, and when one or both of the user's hands are determined to be too close to the virtual boundary 606, boundary wall 802 may be displayed to the user regardless of whether the user is holding one or more controllers 400.

[0105] In some embodiments, method 100 may also include the operations of generating a physical definition of the real-world environment and storing the virtual boundary (including any modifications) in a memory device in association with the physical definition of the real-world environment in which the virtual boundary is defined, such that the user-defined virtual boundary may be reloaded and used again in the same real-world environment in which it is defined. Fig. 9 As shown, a memory device referred to as memory device 900 may be included in the HMD system 200. The memory device 900 may store a table or virtual boundary library containing a plurality of physical definitions 902A-D. Each physical definition 902A-D may be associated with a virtual boundary 904A-D, respectively. In some embodiments, the virtual boundary library may further indicate whether the entry corresponds to a modification, such as determining a modification history that allows the user to undo / redo the modification. The method 100 may include an operation of performing a real-world environment check, which may include generating a temporary physical definition of the real-world environment and then comparing the definition with the physical definitions 902A-D included in the memory device 900. When a match is found as a result of the comparison, the processing subsystem 210 may allow the HMD system 200 to use the corresponding virtual boundary. When the real-world environment check does not result in a match being found and the user is required to define a new virtual boundary, the processing subsystem 210 may deny the HMD system 200 from using the virtual boundary.

[0106] According to aspects described herein, a user of a VR system can modify a previously defined virtual boundary. Wearing an HMD device, the user can view a virtual boundary overlaid on a pass-through view of a real-world environment. The user can use a handheld controller to draw modifications to the virtual boundary. The HMD device and handheld controller can allow the user to intuitively modify the virtual boundary by "drawing" or "brushing" the modifications. If correction is required, allowing the user to modify the virtual boundary can prevent the user from having to completely redefine the virtual boundary from the beginning. If correction to the virtual boundary is required, the user can also be notified visually and / or audibly, and the virtual boundary can then be corrected in real time. Therefore, a more efficient and intuitive interface for defining and modifying virtual boundaries can be presented to the user.

[0107] As detailed above, the computing devices and systems described and / or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions (such as those contained in the modules described herein). In their most basic configuration, these computing devices can each include at least one memory device and at least one physical processor.

[0108] In some examples, the term "memory device" generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a memory device can store, load and / or maintain one or more modules described herein. Examples of memory devices include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), optical disk drive, cache, a variation or combination of one or more of these components, or any other suitable storage memory.

[0109] In some examples, the term "physical processor" generally refers to any type or form of hardware-implemented processing unit capable of parsing and / or executing computer-readable instructions. In one example, the physical processor can access and / or modify one or more modules stored in the above-mentioned memory device. Examples of physical processors include, without limitation, a microprocessor, a microcontroller, a central processing unit (CPU), a field programmable gate array (FPGA) implementing a soft-core processor, an application-specific integrated circuit (ASIC), a portion of one or more of these components, a variation or combination of one or more of these components, or any other suitable physical processor.

[0110] Although shown as separate elements, the modules described and / or shown herein may represent parts of a single module or application. In addition, in certain embodiments, one or more of these modules may represent one or more software applications or programs that, when executed by a computing device, may enable the computing device to perform one or more tasks. For example, one or more modules described and / or shown herein may represent modules stored and configured to run on one or more computing devices or systems described and / or shown herein. One or more of these modules may also represent all or part of one or more special-purpose computers configured to perform one or more tasks.

[0111] In addition, one or more modules described herein can convert data, physical devices, and / or representations of physical devices from one form to another. For example, one or more modules described herein can receive sensor data to be transformed, transform the sensor data, output the transformation results to display a virtual boundary, use the transformation results to define and / or modify the virtual boundary, and store the transformation results to establish the virtual boundary. Additionally or alternatively, one or more modules described herein can convert a processor, volatile memory, non-volatile memory, and / or any other part of a physical computing device from one form to another by executing on a computing device, storing data on a computing device, and / or otherwise interacting with a computing device.

[0112] In some embodiments, the term "computer-readable medium" generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transmission-type media (such as carrier waves) and non-transitory media, such as magnetic storage media (e.g., hard drives, tape drives, and floppy disks), optical storage media (e.g., compact disks (CDs), digital video disks (DVDs), and BLU-RAY disks), electronic storage media (e.g., solid-state drives and flash memory media), and other distribution systems.

[0113] Embodiments of the present disclosure may include an artificial reality system or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (such as stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or to be used in other ways in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0114] The process parameters and step sequences described and / or illustrated herein are provided as examples only and may be changed as desired. For example, although the steps illustrated and / or described herein may be shown or discussed in a particular order, the steps do not necessarily need to be performed in the order illustrated or discussed. The various exemplary methods described and / or illustrated herein may also omit one or more steps described or illustrated herein, or include additional steps in addition to those disclosed steps.

[0115] The foregoing description has been provided to enable others skilled in the art to best utilize the various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the present disclosure. The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. In determining the scope of the present disclosure, reference should be made to the attached claims and their equivalents.

[0116] Unless otherwise indicated, the terms "connected to" and "coupled to" (and their derivatives) used in the specification and claims should be interpreted as allowing both direct and indirect (i.e., through other elements or components) connections. In addition, the terms "a" or "an" used in the specification and claims should be interpreted to mean "at least one of..." Finally, for ease of use, the terms "including" and "having" (and their derivatives) used in the specification and claims are interchangeable with the word "comprising" and have the same meaning.

Claims

1. A method comprising: capturing, by an imaging system of the head mounted display system, image data associated with a view of a real world environment; displaying, by a display of the head mounted display system, to a user a pass-through view of the real world environment based at least in part on the image data; determining, by the head mounted display system, an elevation of a direction indicator relative to a reference plane of the real world environment; monitoring, by the head mounted display system, a position and orientation of the directional indicator in the real world environment as the directional indicator is moved by a user; determining, by the head mounted display system, a series of intersection points as the user moves the directional indicator, wherein each intersection point in the series of intersection points is determined by detecting the location of an intersection between the reference plane of the real world environment and a virtual line defined by the position and orientation of the directional indicator for each of the series of intersection points; as well as The head mounted display system forms a virtual boundary in the virtual world environment based on a combination of the series of intersections.

2. The method according to claim 1, further comprising: A user is prompted by the head mounted display system to position the direction indicator to contact the reference plane.

3. The method according to claim 1 or 2, wherein: The direction indicator is a handheld controller.

4. The method according to claim 3, wherein: The handheld controller includes an array of tracking lights; as well as The method also includes determining, by the head mounted display system, a position or orientation of the handheld controller using light received from the tracking light via the imaging system.

5. The method according to claim 3 or 4, wherein: The handheld controller includes an inertial measurement unit (IMU) that generates position and orientation data; as well as Monitoring the position and orientation of the handheld controller includes receiving, by the head mounted display system, the position and orientation data from the handheld controller.

6. A method according to any preceding claim, wherein: At least a portion of the image data has lens-induced distortion; as well as The pass-through view displaying the real world environment includes: correcting, by the head mounted display system, distortion caused by the lens in the image data to produce a compensated view of the real world environment; as well as The virtual boundary is displayed by the display of the head mounted display system by superimposing the virtual boundary on the compensated view of the real world environment.

7. The method according to any preceding claim, further comprising: providing, by the display of the head mounted display system, a virtual representation of the reference plane to a user; as well as Confirmation of the reference plane is received from a user at the head mounted display system.

8. The method according to any preceding claim, further comprising: When the user comes within a threshold distance of the virtual boundary, an indication of the virtual boundary is presented to the user by the display of the head mounted display system.

9. The method according to any preceding claim, further comprising: A representation of the virtual line is displayed to a user by the display of the head mounted display system.

10. A head mounted display system, comprising: a display secured to a user attachment system; Imaging system; Handheld controller; as well as A processing system, the processing system being configured to: capturing image data with the imaging system, correlating the image data with a view of a real world environment using the imaging system; displaying, with the display, a pass-through view of the real-world environment to a user based at least in part on the image data; determining an elevation of the handheld controller relative to a reference plane of the real world environment; monitoring a position and orientation of the handheld controller in the real world environment as a directional indicator is moved by a user; determining a series of intersection points as the user moves the directional indicator, wherein each intersection point in the series of intersection points is determined by detecting the location of an intersection between the reference plane in the real world environment and a virtual line defined by the position and orientation of the handheld controller for each of the series of intersection points; and A virtual boundary in the virtual world environment is formed based on a combination of the series of intersection points.

11. The head mounted display system according to claim 10, wherein: The processing system is further configured to prompt a user with the display to position the handheld controller in contact with the reference plane.

12. The head mounted display system according to claim 10 or 11, wherein: The handheld controller includes an array of tracking lights; and The processing system is also configured to determine a position or orientation of the handheld controller using light received from the tracking light via the imaging system.

13. The head mounted display system according to any one of claims 10 to 12, wherein: The handheld controller includes an inertial measurement unit (IMU) that generates position and orientation data; and The position and orientation of the handheld controller are monitored, and the processing system is further configured to receive the position and orientation data from the handheld controller.

14. The head mounted display system according to any one of claims 10 to 13, wherein: At least a portion of the image data has lens-induced distortion; displaying the through-view of the real-world environment, the processing system being further configured to: correct the lens-induced distortion in the image data to produce a compensated view of the real-world environment; and The processing system is also configured to display, with the display, the virtual boundary by superimposing the virtual boundary on the compensated view of the real world environment.

15. The head mounted display system according to any one of claims 10 to 14, wherein: The processing system is also configured to: providing a virtual representation of the reference plane to a user using the display; and Confirmation of the reference plane is received from a user using the handheld controller.

16. The head mounted display system according to any one of claims 10 to 15, wherein: The processing system is further configured to present an indication of the virtual boundary to a user using the display when the user comes within a threshold distance of the virtual boundary.

17. The head mounted display system according to any one of claims 10 to 16, wherein: The processing system is further configured to display a representation of the virtual line to a user using the display.