System and method for modifying safety boundaries for virtual reality systems

Abstract

To provide a system and method which can be used by a user wearing an HMD device to prevent collision with a feature or an obstacle in a real world environment and allows for modifying a virtual boundary in the real world environment.SOLUTION: A disclosed computer-implemented method may include receiving an indication of a reference elevation representing a plane of a real world environment and establishing, with respect to the reference elevation, a virtual boundary for a virtual world environment. The method may include receiving a request from a user to modify the virtual boundary and, in response to the request from the user, monitoring an orientation of a direction indicator to generate orientation data. The method may also include modifying the virtual boundary based on the reference elevation and the orientation data. Various other methods, systems, and computer-readable media are also disclosed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Non-provisional Application No. 16 / 177,424, filed October 31, 2018, the entire disclosure of which is incorporated by reference. [Background technology]

[0002] Virtual reality (VR) and augmented reality (AR) systems can enable users to have a more immersive experience than what regular television or video gaming can offer. While wearing a head-mounted display (HMD), a user can observe different parts of a captured or artificially generated scene simply by orienting their head, just as a user would naturally do to observe a real-world environment. The scene can be presented to the user in the HMD based on the position and orientation of the user's head, such that the scene changes based on changes in the user's head position and orientation. Also, a mobile VR system can take into account the user's movement as the user walks around in the real-world environment, so that the user perceives themselves as moving in the virtual environment.

[0003] While immersive, these features may allow a user to interact with the virtual environment in a way that causes the user to forget important aspects of the user's real-world environment. For example, a user attempting to walk from one location in the virtual environment to another may fail to consider (or be unable to see) real-world obstacles, such as a table, couch, or wall, due to the user's lack of awareness of the real-world environment. This may result in collisions with the real-world environment or features in the real-world environment. Summary of the Invention

[0004] As described in more detail below, the present disclosure describes systems and methods that may enable a user wearing an HMD device to modify virtual boundaries in a real-world environment, which may be used to prevent collisions with features or obstacles in the real-world environment.

[0005] In one example, a computer-implemented method for modifying a virtual boundary may include: (1) receiving an indication of a reference elevation representing a plane of a real-world environment; (2) establishing a virtual boundary for the virtual world environment relative to the reference elevation; (3) receiving a request from a user to modify the virtual boundary; (4) monitoring an orientation of a direction indicator to generate orientation data in response to the request from the user; and (5) modifying the virtual boundary based on the reference elevation and the orientation data.

[0006] In some examples, the orientation data may include altitude data of the altitude of the directional indicator relative to a reference altitude. Revising the virtual boundary may include determining an intersection of a plane with a virtual line extending from the user device at the altitude indicated by the altitude data and at the orientation indicated by the orientation data.

[0007] In some examples, the method may include capturing a view of the real-world environment with an imaging system of a head-mounted display system. The captured view may have lens-induced distortion. The method may also include correcting the lens-induced distortion in the captured view to create a compensated view of the real-world environment, superimposing a virtual boundary on the compensated view of the real-world environment, and displaying the compensated view of the real-world environment on a display of the head-mounted display system during the boundary correction state.

[0008] In some examples, when the virtual boundary meets the minimum area threshold, the virtual boundary may be displayed in a success color, and when the virtual boundary does not meet the minimum area threshold, the virtual boundary may be displayed in a warning color. In some examples, the virtual boundary may be displayed as a filled shape defined by the virtual boundary.

[0009] In some examples, modifying the virtual boundary may include adding or subtracting a portion from the virtual boundary. In some examples, the method may include receiving a confirmation of the reference altitude. In some examples, the method may include receiving an instruction to reset the virtual boundary.

[0010] In some examples, the methods described above may be encoded as computer-readable instructions on a computer-readable medium. For example, the computer-readable medium may include one or more computer-executable instructions that, when executed by at least one processor of a computing device, may cause the computing device to (1) receive an indication of a base altitude representing a plane of the real-world environment, (2) establish a virtual boundary for the virtual world environment relative to the base altitude, (3) monitor an orientation of the direction indicator to generate orientation data, and (4) modify the virtual boundary based on the base altitude and the orientation data.

[0011] In some examples, the orientation data may include altitude data of the altitude of the directional indicator relative to a reference altitude. Modifying the virtual boundary may include determining an intersection of a plane with an imaginary line extending from the directional indicator at the altitude indicated by the altitude data and at the orientation indicated by the orientation data.

[0012] In some examples, the instructions may include instructions for capturing a view of the real-world environment with an imaging system of a head-mounted display system. The captured view may have lens-induced distortion. The instructions may include instructions for correcting the lens-induced distortion in the captured view to create a compensated view of the real-world environment, superimposing a virtual boundary on the compensated view of the real-world environment as a filled shape defined by the virtual boundary, and displaying the compensated view of the real-world environment on a display of the head-mounted display system during the boundary correction state.

[0013] In some examples, when the virtual boundary meets the minimum area threshold, the virtual boundary may be displayed in a success color, and when the virtual boundary does not meet the minimum area threshold, the virtual boundary may be displayed in a warning color.

[0014] In some examples, modifying the virtual boundary may include adding a portion to the virtual boundary or subtracting a portion from the virtual boundary. In some examples, the instructions may include instructions to receive a confirmation of the base altitude. In some examples, the instructions may include instructions to receive instructions to reset the virtual boundary.

[0015] Additionally, a head-mounted display system may include a display fixed to a user attachment system, a directional indicator, and a processing system configured to (1) identify a reference altitude representing a plane of the real-world environment, (2) establish a virtual boundary for the virtual world environment relative to the reference altitude, and (3) modify the virtual boundary based on the reference altitude and orientation data characterizing an orientation of the directional indicator.

[0016] In some examples, the processing system may be further configured to capture a view of the real-world environment with an imaging system of the head-mounted display system. The captured view may have lens-induced distortion. The processing system may be further configured to correct the lens-induced distortion in the captured view to produce a compensated view of the real-world environment, superimpose a virtual boundary as a filled shape defined by the virtual boundary on the compensated view of the real-world environment, and display the compensated view of the real-world environment on a display of the head-mounted display system during the boundary correction state.

[0017] In some examples, when the virtual boundary meets the minimum area threshold, the virtual boundary may be displayed in a success color, and when the virtual boundary does not meet the minimum area threshold, the virtual boundary may be displayed in a warning color.

[0018] In some examples, modifying the virtual boundary may include adding or subtracting portions from the virtual boundary. In some examples, the orientation data may include altitude data for an altitude of the directional indicator. Modifying the virtual boundary may include determining an intersection of a plane with a virtual line extending from the user device at the altitude indicated by the altitude data and at the orientation indicated by the orientation data.

[0019] Features from any of the above-described embodiments may be used in combination with each other in accordance with the general principles described herein. These and other embodiments, features, and advantages will be more fully understood from a reading of the following detailed description in conjunction with the accompanying drawings and claims.

[0020] In one embodiment according to the invention, one or more computer-readable non-transitory storage media may embody software that, when executed, is operable to perform a method according to the invention or a method in any of the above-described embodiments.

[0021] In one embodiment according to the invention, a system may comprise one or more processors and at least one memory coupled to the processors and comprising instructions executable by the processors, the processors being operable, when executing the instructions, to perform a method according to the invention or a method in any of the above-described embodiments.

[0022] In one embodiment according to the invention, a computer program product, preferably comprising a computer-readable non-transitory storage medium, may be operable to perform a method according to the invention or a method in any of the above-described embodiments when executed on a data processing system.

[0023] Embodiments in accordance with the present invention are disclosed in the accompanying claims, which are directed, inter alia, to methods, storage media, and head-mounted displays (HMDs), and any feature recited in one claim category, e.g., methods, may also be claimed in another claim category, e.g., HMDs, systems, and computer program products. Dependencies or references in the accompanying claims are chosen for formality reasons only. However, subject matter resulting from intentional references (e.g., multiple dependencies) to previous claims may also be claimed, and as a result, any combination of a claim and its features may be claimed regardless of the dependencies disclosed and chosen in the accompanying claims. Subject matter that may be claimed includes not only combinations of features set forth in the accompanying claims, but also any other combination of features in the claims, and each feature recited in a claim may be combined with any other feature or other combination of features in the claim. Furthermore, any of the embodiments and features described or shown herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or shown herein or with any of the features in the accompanying claims.

[0024] The accompanying drawings illustrate several exemplary embodiments and are a part of this specification and, together with the following description, illustrate and explain various principles of the present disclosure. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a flow diagram of an exemplary method for modifying a virtual boundary relative to a real-world environment, according to aspects of the present disclosure. [Figure 2] FIG. 1 is a block diagram of an exemplary head-mounted display (HMD) system, according to aspects of the present disclosure. [Figure 3] 3 is a perspective view of an exemplary HMD device that may be included in the HMD system of FIG. 2 according to an embodiment of the present disclosure. [Figure 4] FIG. 3 is a perspective view of an exemplary handheld controller that may be included in the HMD system of FIG. 2 according to an embodiment of the present disclosure. [Figure 5A-5B] 5A and 5B are perspective and top views, respectively, of a user wearing the HMD device of FIG. 3 and holding the handheld controller of FIG. 4 in a real-world environment, according to an embodiment of the present disclosure. [Figures 6A-6B] 1A and 1B are perspective and top views, respectively, of a user interacting with a reproduction of a real-world environment to create a virtual safety boundary, according to aspects of the present disclosure. [Figure 6C] FIG. 1 illustrates a system for interacting with a real-world environment to define a virtual boundary, according to aspects of the present disclosure. [Figures 7A-7B] 1A and 1B are perspective and top views, respectively, of a user continuing to interact with a reproduction of a real-world environment to create a virtual boundary, according to aspects of the present disclosure. [Figure 8A-8B] 1A and 1B are perspective and top views, respectively, of a user interacting with a defined virtual boundary, according to aspects of the present disclosure. [Figure 9] FIG. 1 is a diagram of a memory device including a set of physical definitions corresponding to a set of predefined virtual boundaries, according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0026] Throughout the drawings, like reference numbers and descriptions indicate similar, but not necessarily identical, elements. While the exemplary embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will herein be described in detail. However, the exemplary embodiments described herein are not limited to the particular forms disclosed. Rather, the present disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims.

[0027] The present disclosure is generally directed to systems and methods that enable a user to modify virtual boundaries for the user's real-world environment and / or virtual environment. As described in more detail below, embodiments of the present disclosure may enable a user to modify a virtual boundary on the floor of a real-world environment, for example, by using a handheld controller, a finger, a gaze, or other directional indicator. The user may "draw" (e.g., virtually draw using a directional indicator) modifications to the virtual boundary. While the user is wearing an HMD device, a view of the real-world environment may be provided by one or more cameras disposed on the HMD device. These cameras may enable a pass-through view that shows the real-world environment as if the user were not wearing the HMD device. In this way, the user may see environmental features and obstacles to be avoided and may define and modify a virtual boundary a safe distance away from such features.

[0028] In some embodiments, a user may use a directional indicator to point to a location on the floor of a real-world environment. For example, an HMD system may include a handheld controller that can be used to point to the floor. The handheld controller's position and orientation information may be used to identify a particular point on the floor by the processing subsystem, such as by using the handheld controller's height from the floor and the handheld controller's orientation. During the boundary modification state, a virtual boundary may be displayed on the floor, and a virtual line may appear to the user extending from the handheld controller toward the floor to provide visual feedback to the user while the user modifies the virtual boundary on the floor.

[0029] After modifying the virtual boundary, an indication of the virtual boundary and / or a boundary wall derived from the virtual boundary may be presented to the user in the HMD device whenever the user enters within a threshold distance of the virtual boundary to make the user aware of the real-world environment and / or prevent the user from tripping, falling, or bumping into an object. Relying on the virtual boundary, the user can keep the HMD device on and safely move around the real-world environment to better engage with the virtual environment presented to the user during a motion state. The embodiments described herein may improve VR and AR technologies by providing safety features without requiring additional sensors or dedicated hardware. Furthermore, the embodiments described herein may improve computer performance by providing safety features that can be implemented without requiring additional resources. For example, the safety features may be implemented without requiring significant processing and memory resources from the computer and, therefore, may not adversely affect computing performance.

[0030] The following provides a detailed description of systems and methods that enable a user to modify their virtual boundary relative to a real-world environment and use the virtual boundary to provide visual and / or audio indications to the user to make the user aware of the real-world environment whenever the risk of an accidental collision becomes too high, with reference to FIGS. 1-9. FIG. 1 illustrates an exemplary process for modifying a virtual boundary. FIG. 2 illustrates an exemplary VR system. FIG. 3 illustrates an exemplary HMD device. FIG. 4 illustrates an exemplary directional indicator. FIGS. 5A-5B illustrate a user using an exemplary AR system. FIGS. 6A-6C illustrate how a user may interact with a reproduction of a real-world environment. FIGS. 7A-7B illustrate how a user may modify the virtual boundary. FIGS. 8A-8B illustrate how a user may interact with the virtual boundary. FIG. 9 illustrates an exemplary memory device of a VR system.

[0031] Figure 1 is a flow diagram of an exemplary computer-implemented method 100 for modifying a virtual boundary. The steps illustrated in Figure 1 may be performed by any suitable computer-executable code and / or computing system, including the system(s) illustrated in Figures 2-4. In one example, each of the steps illustrated in Figure 1 may represent an algorithm the structure of which includes and / or is represented by multiple sub-steps, examples of which are provided in more detail below.

[0032] 1, one or more of the systems described herein may receive an indication of a reference altitude representing a plane of a real-world environment at step 110. For example, the processing subsystem 210 of the HMD system 200 shown in FIG.

[0033] In some embodiments, the term "reference altitude" may refer to an altitude corresponding to a reference plane in a real-world environment that may be used to determine the point / location pointed to by the directional indicator. In some embodiments, the reference altitude may be a baseline altitude corresponding to a base plane of the real-world environment. The term "base plane" may refer to the lowest altitude a user may encounter in the real-world environment. Examples of base planes include, but are not limited to, a floor, a table, or the ground. The baseline altitude may represent the altitude of the base plane and may be defined perpendicular to the floor. A virtual boundary for the VR system may be defined and / or modified relative to the reference altitude and / or the corresponding plane. FIG. 2 shows an example VR system that may utilize a virtual boundary.

[0034] FIG. 2 is a block diagram of an embodiment of an HMD system 200 that presents a scene (e.g., a captured scene, an artificially generated scene, or a combination thereof) to a user. The HMD system 200 may operate in a virtual reality (VR) system environment, an augmented reality (AR) system environment, a mixed reality (MR) system environment, or some combination thereof. The HMD system 200 shown in FIG. 2 may include an HMD device 205 that includes or communicates with a processing subsystem 210 and an input / output (I / O) interface 215. The HMD device 205, in some embodiments, may completely obstruct the user's view of the real-world environment. Other embodiments may only partially obstruct the user's view of the real-world environment and / or obstruct the user's view depending on the content displayed on the display of the HMD device 205.

[0035] 2 shows an exemplary HMD system 200 including at least one HMD device 205 and at least one I / O interface 215, in other embodiments, any number of these components may be included in the HMD system 200. For example, there may be multiple HMDs 205, each having an associated I / O interface 215, with each HMD device 205 and I / O interface 215 communicating with the processing subsystem 210. In embodiments in which the processing subsystem 210 is not included within or integrated with the HMD device 205, the HMD device 205 may communicate with the processing subsystem 210 via a wired or wireless connection. In alternative configurations, different and / or additional components may be included in the HMD system 200. Furthermore, the functionality described in connection with one or more of the components shown in FIG. 2 may, in some embodiments, be distributed among the components in a manner different from that described with respect to FIG. 2.

[0036] The HMD device 205 may present a variety of content to the user, including virtual views of artificially rendered virtual world environments and / or augmented views of physical real-world environments augmented with computer-generated elements (e.g., two-dimensional (2D) or three-dimensional (3D) images, 2D or 3D video, sound, etc.). In some embodiments, the presented content includes audio presented via internal or external devices (e.g., speakers and / or headphones) that receive audio information from the HMD device 205, the processing subsystem 210, or both and present audio data based on the audio information. In some embodiments, such speakers and / or headphones may be integrated into or releasably coupled or attached to the HMD device 205. The HMD device 205 may comprise one or more bodies, which may be rigidly or non-rigidly coupled to one another. A rigid coupling between rigid bodies may cause the coupled rigid bodies to act as a single rigid entity. In contrast, a non-rigid coupling between rigid bodies may allow the rigid bodies to move relative to one another. One embodiment of the HMD device 205 is the HMD device 300 shown in FIG. 3 and described in further detail below.

[0037] In some examples, HMD device 205 may include a depth-sensing subsystem 220 (or depth camera system), an electronic display 225, an image capture subsystem 230 including one or more cameras, one or more position sensors 235, and / or an inertial measurement unit (IMU) 240. Other embodiments of HMD device 205 may include an optional eye-tracking or gaze estimation system configured to track the eyes of a user of HMD device 205 to estimate the user's gaze. An optional variable-focus module may be configured to adjust the focus of one or more images displayed on electronic display 225 based on determined eye-tracking information obtained from the eye-tracking system and other components. Some embodiments of HMD device 205 may have different components than those described in connection with FIG. 2 .

[0038] The depth sensing subsystem 220 may capture data describing depth information that characterizes a local real-world area or environment surrounding some or all of the HMD device 205 and / or characterizes the position, velocity, or location of the depth sensing subsystem 220 (and thereby the HMD device 205) within the local area. The depth sensing subsystem 220 can use the collected data to calculate the depth information (e.g., by processing a portion of a structured light pattern based on the captured light according to one or more computer vision methods or algorithms, by time-of-flight (ToF) imaging, simultaneous localization and mapping (SLAM), etc.), or the depth sensing subsystem 220 can transmit this data to another device, such as an external implementation of the processing subsystem 210, which can use the data from the depth sensing subsystem 220 to determine the depth information.

[0039] Electronic display 225 may display two-dimensional or three-dimensional images to the user according to data received from processing subsystem 210. In various embodiments, electronic display 225 includes a single electronic display or multiple electronic displays (e.g., a display for each eye of the user). Examples of electronic display 225 may include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an inorganic light-emitting diode (ILED) display, an active-matrix organic light-emitting diode (AMOLED) display, a transparent organic light-emitting diode (TOLED) display, another suitable display, or some combination thereof. Electronic display 225 may be opaque so that the user cannot see the local environment through electronic display 225.

[0040] The image capture subsystem 230 may include one or more optical image sensors or cameras that capture and collect image data from the local environment. In some embodiments, sensors included in the image capture subsystem 230 may provide a stereoscopic view of the local environment, which may be used by the processing subsystem 210 to generate image data that characterizes the local environment and / or the position and orientation of the HMD device 205 within the local environment. For example, the image capture subsystem 230 may include a simultaneous localization and mapping (SLAM) camera or other camera including a wide-angle lens system that captures a wider field of view than can be captured by the user's eyes. As described herein, the image capture subsystem 230 may provide a pass-through view of the real-world environment that is displayed to the user via the electronic display 225 when the HMD system 200 is in a boundary-defined state.

[0041] In some embodiments, the processing subsystem 210 may process images captured by the image capture subsystem 230 to remove distortions caused by the lens system of the image capture subsystem 230 and / or by a separation distance between two image sensors that is significantly greater or less than the average separation distance between a user's eyes. For example, when the image capture subsystem 230 is or is part of a SLAM camera system, the direct image from the image capture subsystem 230 may appear distorted to the user if presented in an uncorrected format. Image correction or compensation may be performed by the processing subsystem 210 to correct the image and present it to the user with a more natural appearance, so that the image appears to the user as if the user were viewing it through the electronic display 225 of the HMD device 205. In some embodiments, the image capture subsystem 230 may include one or more image sensors with lenses adapted (in terms of field of view, separation distance, etc.) to provide a pass-through view of the local environment. The image capture subsystem 230 may capture color or monochrome images.

[0042] The IMU 240 may, in some examples, represent an electronic subsystem that generates data indicative of the position and / or orientation of the HMD device 205 based on measurement signals received from one or more of the position sensors 235 and from depth information received from the depth sensing subsystem 220 and / or the image capture subsystem 230. For example, the position sensor 235 generates one or more measurement signals in response to movement of the HMD device 205. Examples of the position sensor 235 include one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor used for error correction of the IMU 240, or some combination thereof. The position sensor 235 may be located external to the IMU 240, internal to the IMU 240, or some combination thereof.

[0043] Based on one or more measurement signals from the one or more position sensors 235, the IMU 240 may generate data indicative of an estimated current position, altitude, and / or orientation of the HMD device 205 relative to an initial position and / or orientation of the HMD device 205. For example, the position sensors 235 may include multiple accelerometers for measuring translational motion (forward / backward, up / down, left / right) and multiple gyroscopes for measuring rotational motion (e.g., pitch, yaw, roll). As described herein, the image capture subsystem 230 and / or the depth sensing subsystem 220 may generate data indicative of an estimated current position and / or orientation of the HMD device 205 relative to a real-world environment in which the HMD device 205 is used.

[0044] I / O interface 215 may represent a subsystem or device that allows a user to send action requests and receive responses from processing subsystem 210 and / or directional indicator 270. In some embodiments, directional indicator 270 may comprise a handheld controller or other device that can be manipulated by a user to provide input to I / O interface 215, such as orientation and / or position data sensed by sensors in directional indicator 270. In other embodiments, directional indicator 270 may provide passive input to I / O interface 215. For example, directional indicator 270 may include a user's finger or hand, a glove or other wearable object, a handheld object, a user's eye and / or line of sight, and / or another user-manipulable object that can be detected by sensors in HMD system 200 to determine orientation and / or position data relative to directional indicator 270. In some embodiments, I / O interface 215 may facilitate communication with more than one directional indicator 270. For example, a user may have two directional indicators 270, one on each hand. An action request, in some examples, may represent a request to perform a particular action. For example, the action request may be an instruction to begin or end the capture of image or video data, or an instruction to perform a particular action or enter or exit a boundary-defined state within an application. I / O interface 215 may include or enable communication with one or more input devices. Exemplary input devices may include a keyboard, a mouse, a handheld controller, or any other suitable device for receiving action requests and communicating the action requests to processing subsystem 210.

[0045] Action requests received by I / O interface 215 may be communicated to processing subsystem 210, which may perform an action corresponding to the action request. In some embodiments, directional indicator 270 includes IMU 240, which captures inertial data indicating an estimated position of directional indicator 270 relative to an initial position. In some embodiments, I / O interface 215 and / or directional indicator 270 may provide haptic feedback to the user according to instructions received from processing subsystem 210 and / or HMD device 205. For example, haptic feedback is provided when an action request is received, or when processing subsystem 210 performs an action, processing subsystem 210 communicates instructions to I / O interface 215, causing I / O interface 215 to generate or direct the generation of haptic feedback.

[0046] The processing subsystem 210 may include one or more processing devices or physical processors that provide content to the HMD device 205 according to information received from one or more of the depth-sensing subsystem 220, the image capture subsystem 230, the I / O interface 215, and the directional indicator 270. In the example shown in FIG. 2, the processing subsystem 210 includes an engine 260, an application store 250, and a tracking module 255. Some embodiments of the processing subsystem 210 have different modules or components than those described in connection with FIG. 2. Similarly, the functionality described further below may be distributed among the components of the HMD system 200 in a manner different from that described in connection with FIG. 2.

[0047] The application store 250 may store one or more applications for execution by the processing subsystem 210. An application may, in some examples, represent a group of instructions that, when executed by a processor, generates content for presentation to a user. The content generated by an application may be generated in response to input received from a user via movement of the HMD device 205 or the directional indicator 270. Examples of applications include gaming applications, conferencing applications, video playback applications, or other suitable applications.

[0048] The tracking module 255 may calibrate the HMD system 200 using one or more calibration parameters and may adjust the one or more calibration parameters to reduce errors in determining the position of the HMD device 205 or the directional indicator 270. For example, the tracking module 255 may communicate to the depth-sensing subsystem 220 calibration parameters for adjusting the focus of the depth-sensing subsystem 220 to more accurately determine the positions of the structured light elements captured by the depth-sensing subsystem 220. The calibration performed by the tracking module 255 may also take into account information received from the IMU 240 in the HMD device 205 and / or another IMU 240 included in the directional indicator 270. Furthermore, if tracking of the HMD device 205 is lost (e.g., the depth-sensing subsystem 220 loses line of sight of at least a threshold number of structured light elements), the tracking module 255 may recalibrate some or all of the HMD system 200.

[0049] The tracking module 255 may track the movement of the HMD device 205 or the movement of the directional indicator 270 using information from the depth-sensing subsystem 220, the image capture subsystem 230, the one or more position sensors 235, the IMU 240, or some combination thereof. For example, the tracking module 255 may determine the location of a reference point of the HMD device 205 in a mapping of the real-world environment based on information collected at the HMD device 205. Additionally, in some embodiments, the tracking module 255 may use portions of data indicative of the position and / or orientation of the HMD device 205 and / or the directional indicator 270 from the IMU 240 to predict the future position and / or orientation of the HMD device 205 and / or the directional indicator 270. The tracking module 255 may also provide the engine 260 with estimated or predicted future positions of the HMD device 205 or the I / O interface 215.

[0050] In some embodiments, the tracking module 255 may track other features that may be observed by the depth-sensing subsystem 220, the image capture subsystem 230, and / or another system. For example, the tracking module 255 may track one or both of the user's hands so that the location of the user's hands in the real-world environment can be known and utilized. The tracking module 255 may receive and process data to determine, for example, the pointing direction of one finger of the user's hand when the directional indicator 270 includes the user's hand. The tracking module 255 may also receive information from one or more eye-tracking cameras included in some embodiments of the HMD device 205 to track the user's gaze.

[0051] Image processing engine 260 may generate a three-dimensional mapping of an area surrounding some or all of HMD device 205 (i.e., a "local area" or "real-world environment") based on information received from HMD device 205. In some embodiments, engine 260 determines depth information for the three-dimensional mapping of the local area based on information received from depth sensing subsystem 220 relating to the technique used in calculating depth. Engine 260 may calculate the depth information using one or more techniques when calculating depth from structured light. In various embodiments, engine 260 uses the depth information, for example, to update a model of the local area and generate content based in part on the updated model.

[0052] The engine 260 also executes applications within the HMD system 200 and may receive from the tracking module 255 position information, acceleration information, velocity information, predicted future position of the HMD device 205, or some combination thereof. Based on the received information, the engine 260 may determine content to provide to the HMD device 205 for presentation to the user. For example, if the received information indicates that the user is looking left, the engine 260 generates content for the HMD device 205 that corresponds to the user's movement in a virtual environment or in an environment that augments the local area with additional content. Additionally, the engine 260 may perform actions within applications running on the processing subsystem 210 in response to action requests received from the I / O interface 215 and / or the directional indicator 270 and provide feedback to the user that the action was performed. The provided feedback may be visual or audible feedback via the HMD device 205 or haptic feedback via the directional indicator 270.

[0053] 1 , the systems described herein may perform step 110 in a variety of ways. In one example, the reference altitude may be previously established and stored. In another example, the reference altitude may be set by a user, such as by using the HMD device 300 and / or handheld controller 400 to define and / or confirm the reference altitude. For example, the HMD system 200 may receive an indication of the reference altitude from the HMD device 300 and / or handheld controller 400.

[0054] FIG. 3 is a diagram of an HMD device 300 according to one embodiment of the HMD device 205. The HMD device 300 may include an imaging subsystem and a depth-sensing subsystem. The HMD device 300 may be, for example, part of a VR system, an AR system, an MR system, or some combination thereof. In embodiments illustrating an AR system and / or an MR system, a portion of the front side 302 of the HMD device 300 is at least partially transparent within the visible band (approximately 380 nanometers (nm) to 750 nm). More specifically, a portion of the HMD device 300 between the front side 302 of the HMD device 300 and the user's eyes may be at least partially transparent (e.g., a partially transparent electronic display 225). In other embodiments, the front side 302 is opaque, preventing the user from seeing the real-world environment. The HMD device 300 may include a forward rigid body 305 that houses the electronic display 225 and other components, a user attachment system such as a band 310 that secures the HMD device 300 to the user's head, and reference points 315 that may characterize the position and / or orientation of the HMD device 300.

[0055] Additionally, the HMD device 300 may include an imaging aperture 320 and an illumination aperture 325. An illumination source included in the depth sensing subsystem 220 may emit light (e.g., structured light) through the illumination aperture 325. An imaging device of the depth sensing subsystem 220 may capture light from the illumination source that is reflected or backscattered from the local area through the imaging aperture 320. Embodiments of the HMD device 300 may further include a camera 340A and a camera 340B, which may be components of the image capture subsystem 230 of FIG. 2. The cameras 340A and 340B may be separated from each other by a distance different from the average separation distance between the user's eyes.

[0056] The forward rigid body 305 may include one or more electronic display elements, one or more integrated eye tracking systems, an IMU 330, one or more position sensors 335, and a reference point 315. The IMU 330 may represent an electronic device that generates fast calibration data based on measurement signals received from one or more of the position sensors 335. The position sensors 335 may generate one or more measurement signals in response to movement of the HMD device 300.

[0057] 4 is a perspective view of an exemplary handheld controller 400, which may be an embodiment of the directional indicator 270 included in the HMD system 200 of FIG. 2 , according to some embodiments. The HMD system 200 may include one or more handheld controllers, such as the controller 400. For example, the HMD system 200 may include two handheld controllers 400, one for each of a user's right and left hands. Each handheld controller 400 may be communicatively coupled to the HMD device 205 and / or to a computing device (e.g., a personal computer, the processing subsystem 210, etc.). The handheld controllers 400 may be communicatively coupled to the HMD device 205 via any suitable wireless and / or wired connection.

[0058] As shown in FIG. 4 , the handheld controller 400 may include a grip 402 sized to fit within a user's hand. The handheld controller 400 may also include a tracking loop 404 for tracking the position, orientation, and / or movement of the handheld controller 400 relative to the HMD device 205 and / or relative to the real-world environment. In some embodiments, the tracking loop 404 may include one or more tracking lights, such as an array of tracking lights 406. The array of tracking lights 406 may include tracking LEDs (e.g., infrared (IR) LEDs) used for motion and position tracking purposes to provide 360-degree motion control while using the HMD system 200. The tracking lights 406 may be utilized to determine the orientation of the controller 400 so that intersection points with the floor of the real-world environment can be identified to “draw” a virtual boundary. The controller 400 may include tracking lights on any suitable portion of the controller 400. In some examples, the tracking light 406 of the handheld controller 400 may emit light having a wavelength greater than approximately 700 nm and less than approximately 900 nm. In one embodiment, the tracking light 406 of the handheld controller 400 may emit light having a wavelength of approximately 850 nm (e.g., between approximately 840 nm and 860 nm, or between approximately 830 nm and 870 nm). In at least one embodiment, the cameras 340A and 340B may receive light emitted by the tracking light 406 on the handheld controller 400, and the tracking module 255 may utilize the received light to determine the location, orientation, and / or movement of the handheld controller 400 relative to another frame of reference, such as the frame of reference of the HMD device 205 and / or the real-world environment.

[0059] To define the reference altitude, the user may interact with the HMD system 200. For example, the user may be prompted to place the handheld controller 400 in contact with a floor, which may provide a base plane of the real-world environment. The reference altitude may be defined perpendicular to the floor. In some embodiments, components of the HMD device 300 may determine the height above the floor based on the orientation of the HMD device 300 and one or more depth measurements that characterize the distance from the HMD device 300 to the floor. For example, the HMD system 200 may prompt the user to look at the floor of the real-world environment as part of the virtual boundary definition process.

[0060] Some embodiments may further include receiving confirmation of the reference altitude. The user may be presented with an indication of the reference altitude. For example, the HMD system 200 may show a visual representation of a plane associated with the reference altitude and / or the current height of the HMD device 300 from the reference altitude. The user may then confirm the reference altitude or define / redefine the reference altitude as described above.

[0061] 1 , in step 120, one or more of the systems described herein may establish a virtual boundary for the virtual world environment relative to a reference altitude. For example, processing subsystem 210 and / or HMD device 205 and / or HMD device 300 may determine the virtual boundary based on input from direction indicator 270 and / or controller 400, or based on retrieving a previously defined virtual boundary.

[0062] In some embodiments, the term "virtual boundary" may refer to a boundary defined for a user's real-world environment and / or virtual environment. Examples of virtual boundaries include, but are not limited to, the safety boundaries described herein that may be defined at a safe distance from obstacles in the real-world environment that a user may encounter during a VR and / or AR experience. Other virtual boundaries may include boundaries that indicate ideal locations for detection by sensors, boundaries for restricting a user's movement in the real-world environment, boundaries that may be used to place and / or clear obstacles in the virtual environment, and boundaries for other uses.

[0063] The systems described herein may perform step 120 in a variety of ways. In one example, the virtual boundary may be loaded from a previously stored setting, such as a default setting or a previous user action stored in the HMD system 200. In another example, a user may define the virtual boundary using, for example, the HMD system 200.

[0064] 5A and 5B present perspective and top views, respectively, of a user wearing the HMD device 300 of FIG. 3 and holding the handheld controller 400 of FIG. 4 in a real-world environment 500, such as a living room, according to some embodiments. The real-world environment 500 may include a base plane 502, also referred to as floor 502, and walls 504A, 504B, 504C, and 504D, collectively referred to as walls 504. The real-world environment 500 may further include multiple objects or features within the room that pose a collision risk when the user's view is blocked by the HMD device 300. For example, the environment 500 may include a fireplace 506A with a protruding mantelpiece 506B, a table 506C, and a shelf 506D. The environment 500 may further include a sofa 506E, as shown in FIG. 5B. The objects and features 506A, 506B, 506C, 506D, and 506E, along with the wall 504, may be referred to as features 506 of the real-world environment 500.

[0065] In some embodiments, the user may move within the real-world environment 500 to navigate within the virtual environment displayed in the HMD device 300. In other words, as the user moves within the real-world environment 500, the image shown on the electronic display 225 of the HMD device 300 may be updated based on the user's movement. Thus, the user moves relative to the virtual environment as the user moves through the real-world environment 500. As described in more detail below, embodiments of the systems and methods described herein may allow the user to define a virtual boundary that can be used to prevent the user from colliding with any of the features 506 when the user cannot see the real-world environment (e.g., due to the HMD device 300 blocking the user's real-world view).

[0066] 6A and 6B present perspective and top views, respectively, of a user interacting with a reproduction 600 of the real-world environment 500 of FIGS. 5A and 5B to create a virtual boundary or safety boundary, according to some embodiments. Because the user's view of the real-world environment 500 may be fully or partially blocked by the HMD device 300, the reproduction 600 of the real-world environment 500 may be provided to the user in the electronic display 225 of the HMD device 300. The reproduction 600 may be produced by the image capture subsystem 230 to provide a pass-through view of the real-world environment 500. In some embodiments, the processing subsystem 210 may perform image correction on images captured by the image capture subsystem 230 to remove distortions and provide an improved view of the real-world environment 500. For example, the processing subsystem 210 may perform image correction to mitigate distortions caused by the lenses of the cameras 340A and 340B and / or the separation distance between the cameras 340A and 340B.

[0067] As shown in FIG. 6A , during a boundary definition state in which the user may define a virtual boundary, the user may utilize the controller 400 as a pointer or directional indicator to select an intersection point 602. The intersection point 602 may be defined as the location of the intersection of the floor 502 and a virtual line 604, defined by the orientation and position of the controller 400 held and manipulated by the user. The virtual line 604 may be displayed to the user in the HMD device 300 so that the user can draw a virtual boundary 606 on the floor 502. As the user manipulates the controller 400, a series of intersection points, such as the intersection point 602, may be joined to form a virtual boundary 606 extending along the floor 502. FIG. 6B shows a top view within a reproduction 600 of the real-world environment 500 showing the intersection point 602, the virtual line 604, and the virtual boundary 606.

[0068] 6C is a diagram illustrating how an intersection point 602 may be defined, according to some embodiments. The position and orientation of the handheld controller within the real-world environment 500 may be determined based on subsystems included within the handheld controller 400, such as an IMU and / or a position tracker. In some instances, the user may be instructed by a message displayed in the HMD device 300 to place the handheld controller 400 in contact with a base plane or floor 502 to prepare for the boundary definition process. The user then activates a button on the controller 400 or holds the controller 400 still for a predetermined amount of time to indicate to the HMD system 200 that the controller 400 is in contact with the floor 502. The user is then instructed to stand, which may displace both the HMD device 300 and the controller 400 away from the floor 502 by a displacement D1, which may be used as the height or altitude of the controller 400 above the floor 502. In some embodiments, the HMD device 300 may determine the altitude of the HMD device 300 from the floor 502 and the distance between the HMD device 300 and the handheld controller 400. For example, the depth sensing subsystem 220 may determine the distance of the HMD device 300 from the floor 502 using structured light or by triangulation estimation using stereoscopic images. The distance between the HMD device 300 and the handheld controller 400 may be subtracted from the height of the HMD device 300 from the floor 502 to determine the displacement D1 or the altitude of the handheld controller 400. The orientation of the controller 400 may be used to determine the angle A1 that the axes of the controller 400 make with the floor 502. Using the location of the controller 400, the displacement D1, and the angle A1, the point of intersection of the virtual line 604 and the floor 502 may be used as the intersection point 602. By manipulating the controller 400, multiple intersection points 602 may be identified, and these points 602 may be connected to form a virtual boundary 606. Although Figure 6C shows the floor 502 as the reference elevation, in other embodiments, the reference elevation may be defined relative to another plane in the real-world environment.

[0069] 1 , one or more of the systems described herein may receive a request from a user to modify the virtual boundary at step 130. For example, HMD system 200 may receive a request from a user via controller 400 to modify the virtual boundary.

[0070] In some embodiments, the HMD system 200 may enter a boundary modification state to monitor inputs for modifying the virtual boundary. The term "boundary modification state" may refer to a state or phase during which the virtual boundary may be modified by the user. The boundary modification state may be the same or similar phase as the boundary definition state during which the virtual boundary may be defined, or may be a separate phase. The boundary modification state may be a natural mode of the HMD system 200 solely for modifying the virtual boundary. Alternatively, the boundary modification state may be integrated with other operating states of the HMD system 200.

[0071] The systems described herein may perform step 130 in various ways. In one example, the user may be notified that the virtual boundary may be insufficient, and therefore the user may initiate a boundary correction state. The user may be notified of the insufficiency of the virtual boundary in various ways. For example, the user may receive a visual or audible notification, such as through the HMD device 300. The notification may be an icon or a more robust visual indicator.

[0072] In some examples, the system may also capture a view of the real-world environment with an imaging system of a head-mounted display system. The captured view may have lens-induced distortion. The system may correct the lens-induced distortion in the captured view to create a compensated view of the real-world environment. The system may superimpose a virtual boundary on the compensated view of the real-world environment and display the compensated view of the real-world environment on the display of the head-mounted display system during the boundary correction state. Figures 7A and 7B show exemplary scenarios of compensated views of the real-world environment presented to a user.

[0073] 7A and 7B present perspective and top views, respectively, of a user interacting with a reproduction 600 of a real-world environment 500 to create a virtual boundary 606, according to some embodiments. The virtual boundary 606 may define a boundary region 608. The boundary region 608 may be displayed, for example, by the HMD device 300 as a filled shape defined by the virtual boundary 606. The boundary region 608 may be displayed similarly to the virtual boundary 606, such as a solid color, or may be a shaded or semi-transparent color corresponding to the virtual boundary 606.

[0074] In some examples, the area defined by the virtual boundary 606, e.g., boundary region 608, may be compared to a minimum area threshold. The minimum area threshold may correspond to a minimum area, such as a minimum number of square feet, within which a user may be expected to safely operate the VR and / or AR system in a real-world environment. The minimum area threshold may be a predetermined or default value or may be dynamically determined in the real-world environment. The minimum area threshold may include dimensional limits, such as minimum and / or maximum length and / or width.

[0075] In some examples, the minimum area threshold may be explicitly presented to the user, such as an amount of square footage that must be met or required for the minimum area threshold. In other examples, the minimum area threshold may be represented by displaying a bounded region in a success color when the minimum area threshold is met or in a warning color if the minimum area threshold is not met. For example, virtual boundary 606 and / or bounded region 608 may be displayed in green when the minimum area threshold is met and in red when the minimum area threshold is not met.

[0076] Upon finding that the virtual boundary 606 does not meet the minimum area threshold, the user may initiate the boundary modification state to modify the virtual boundary. Alternatively, the boundary modification state may be initiated automatically if the minimum area threshold is not met. For example, during the boundary definition state, the boundary modification state may be initiated automatically if the virtual boundary 606 does not meet the minimum area threshold. Additionally, the user may initiate the boundary modification state for other reasons. For example, the real-world environment may change, or the user may wish to fine-tune the virtual boundary, such as if the user prefers a smaller or larger boundary area.

[0077] 1 , in step 140, one or more of the systems described herein monitor the orientation of the directional indicator to generate orientation data in response to a request from a user. For example, processing subsystem 210 and / or HMD device 205 and / or HMD device 300 may determine the orientation data based on input from directional indicator 270 and / or controller 400. In some examples, the orientation data may include and / or be associated with additional input data. For example, the orientation data may include altitude data of the directional indicator relative to a reference altitude. The orientation data may include position data of the directional indicator.

[0078] The systems described herein may perform step 140 in various ways. In one example, the position and orientation of handheld controller 400 within real-world environment 500 may be determined based on subsystems included within handheld controller 400, such as an IMU and / or position sensors. As described above with respect to FIG. 6C , D1 may be monitored to determine altitude data and A1 may be monitored to determine orientation data. Alternatively, the altitude data and orientation data may be determined from sensors external to controller 400, such as a camera observing controller 400 in real-world environment 500.

[0079] 1 , in step 150, one or more of the systems described herein may modify the virtual boundary based on the reference altitude and orientation data. For example, processing subsystem 210 and / or HMD device 205 and / or HMD device 300 may modify the virtual boundary based on the reference altitude and altitude and orientation data from direction indicator 270 and / or controller 400.

[0080] The systems described herein may perform step 150 in various ways. In one example, the intersection of a plane with a virtual line extending from the user device at the altitude indicated by the altitude data and at the orientation indicated by the orientation data may be determined. As shown in FIG. 6C , the locations D1 and A1 of the controller 400 may define where the virtual line 604 begins, and the point of intersection of the virtual line 604 with the floor 502 may define the intersection point 602. Similar to how multiple intersection points 602 may define the virtual boundary 606, a modification to the virtual boundary 606 may be defined by one or more intersection points 602. The intersection point 602 may define an extension to the virtual boundary 606.

[0081] The virtual boundary may be modified in various ways. For example, FIGS. 7A and 7B show how a virtual boundary 606 may be modified. The virtual boundary 606 may not meet the minimum area threshold, and the boundary region 608 may be displayed in a warning color. The user may manipulate the controller 400 to define a correction line 610, which may be defined by one or more intersection points 602. Although not fully shown in FIGS. 7A and 7B, the correction line 610 may outline an area surrounding the boundary region 608 to meet the minimum area threshold.

[0082] The virtual boundary may be modified by adding portions to the virtual boundary. For example, a portion of virtual boundary 606 may be extended to add additional area to boundary region 608. The additional area may be connected to boundary region 608, such as connected to a side of boundary region 608, or may be a separate or possibly discontinuous area associated with boundary region 608. The additional area may partially or completely surround boundary region 608.

[0083] In other examples, modifying the virtual boundary may include subtracting portions from the virtual boundary. For example, the boundary modification state may include an addition state for adding portions and a subtraction state for removing portions from the virtual boundary. A user may toggle between the addition state and the subtraction state to define and modify the virtual boundary to a desired shape. For example, a user may use the controller 400 to enter the addition state and draw what to add to the virtual boundary 606. A user may use the controller 400 to enter the subtraction state and cut out a portion from the virtual boundary 606.

[0084] In yet another example, modifying the virtual boundary may be accomplished by resetting the virtual boundary. For example, the user may send an instruction to reset the virtual boundary. The user may then define the virtual boundary from scratch. Alternatively, the user may be able to undo / redo one or more modifications.

[0085] The modified virtual boundary may be compared to a minimum area threshold. The comparison may be performed dynamically, for example, in real time as the user draws the modification. Incomplete loops and / or unconnected portions may be automatically connected, for example, with the shortest distance straight line to connect the unconnected portions, for purposes of estimating the area. Thus, the user may know in real time when the minimum area threshold is met. Alternatively, the minimum area comparison may be performed when the user indicates that they wish to exit the boundary modification state. In some examples, the user may not exit the boundary modification state unless the minimum area threshold is met.

[0086] Additionally, when the boundary correction state ends, the virtual boundary may be checked for errors. Errors may include, for example, unclosed loops, zero area portions, portions that are too small for the user to reasonably use (which may be compared to a minimum portion area threshold), and other errors. In some examples, the errors may be automatically corrected. For example, unclosed loops may be connected, problematic portions may be removed, etc. In other examples, the user may be alerted to the error, and in some examples, the boundary correction state may be automatically restarted. In some examples, the boundary correction state may not end until there are no more errors detected in the virtual boundary.

[0087] After the user completes the boundary modification state, which may require meeting a minimum area threshold, and establishes the virtual boundary, the user may interact with the virtual boundary. FIGS. 8A and 8B are perspective and top views, respectively, of a user interacting with a defined virtual boundary in a virtual environment, according to some embodiments. As shown in FIGS. 8A and 8B , a virtual environment 800 may be displayed to the user at the HMD device 300. The virtual environment 800 may represent a captured scene through which the user can move, or an artificial environment, such as in a video game. The user may navigate within the virtual environment 800 by moving within the real-world environment 500. The virtual environment may include a visual indication of the virtual boundary 606. In other words, the virtual boundary 606 may be visibly rendered and presented to the user so that the user can see the virtual boundary 606 whenever the field of view of the HMD device 300 includes the virtual boundary 606. In some embodiments, the user may select a setting to have the virtual boundary 606 always appear, or to have the virtual boundary 606 appear only when the user is within a threshold distance, which may depend on the speed or movement of the user and / or the particular portion of the user or HMD system 200 that is closest to the boundary.

[0088] In some embodiments, as the user approaches the virtual boundary 606, a boundary wall 802 (shown as elements 802A and 802B) may be rendered in the HMD device 300 to alert the user to their proximity to the virtual boundary 606 and, therefore, to features 506 in the real-world environment 500 that pose a collision risk. The boundary wall 802 may be rendered as a series of vertical or horizontal bars, a grid of lines, a grid of dots, etc., which may allow the user to continue to observe a portion of the virtual environment 800 through the boundary wall 802. In other embodiments, the boundary wall 802 may be rendered in a manner that completely “obstructs” the user's view of a portion of the virtual environment 800. In some embodiments, the rendering of the boundary wall 802 may block an increasing amount of the user's view as the user approaches the virtual boundary 606. In other examples, the boundary wall 802 is overlaid or otherwise rendered on top of a pass-through view of the real-world environment (e.g., provided by the image capture subsystem 230).

[0089] 8A and 8B , separate wall portions may be rendered at the HMD device 300. Thus, boundary wall 802A and boundary wall 802B are shown. Boundary wall 802A may be rendered when the distance between the user and the virtual boundary 606 is less than or equal to distance D2. Distance D2 may be configurable by the user or automatically by the HMD system 200. For example, in some embodiments, a threshold distance between 2 feet and 5 feet may be used. The threshold distance may, in some embodiments, be a function of the user's speed. Boundary wall 802B may be rendered at the HMD device 300 based on distance D3 between the controller 400 and the virtual boundary 606 or based on distance D4 between the HMD device 300 and the virtual boundary 606. In some implementations, the position of the user's hands may be monitored, and boundary wall 802 may be displayed to the user when it is determined that one or more of the user's hands are too close to the virtual boundary 606, regardless of whether the user is holding one or more controllers 400.

[0090] In some embodiments, method 100 may further include the operations of generating a physical definition of the real-world environment and storing the virtual boundary, including any modifications associated with the physical definition of the real-world environment in which the virtual boundary was defined, in a memory device, so that the user-defined virtual boundary can be reloaded and used again in the same real-world environment in which the virtual boundary was defined. As shown in FIG. 9 , a memory device, referred to as memory device 900, may be included in HMD system 200. Memory device 900 may store a table or virtual boundary library containing multiple physical definitions 902A-D. Each of physical definitions 902A-D may be associated with a virtual boundary 904A-D, respectively. In some embodiments, the virtual boundary library may further indicate whether an entry corresponds to a modification, for example, to determine a history of modifications that may allow a user to undo / redo the modification. Method 100 may include the operation of performing a real-world environment test, which may include generating a virtual physical definition of the real-world environment and then comparing the definition to the physical definitions 902A-D included in memory device 900. When the comparison results in a match, the processing subsystem 210 may allow the HMD system 200 to use the corresponding virtual boundary. When the real-world environment inspection results in no match, the processing subsystem 210 may deny the HMD system 200 use of the virtual boundary and require the user to define a new virtual boundary.

[0091] According to aspects described herein, a user of a VR system may modify a previously defined virtual boundary. Wearing an HMD device, the user may observe the virtual boundary overlaid on a pass-through view of the real-world environment. The user may use a handheld controller to draw modifications to the virtual boundary. The HMD device and handheld controller may allow the user to intuitively modify the virtual boundary by “drawing” or “paintbrushing” the modifications. Allowing the user to modify the virtual boundary may prevent the user from having to completely redefine the virtual boundary from scratch if a correction is needed. The user may also be visually and / or audibly notified if a correction is needed to the virtual boundary and may then correct the virtual boundary in real time. Thus, the user may be presented with a more efficient and intuitive interface for defining and modifying the virtual boundary.

[0092] As detailed above, the computing devices and systems described and / or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those contained within the modules described herein. In their most basic configurations, these computing device(s) may each include at least one memory device and at least one physical processor.

[0093] In some examples, the term "memory device" generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a memory device may store, load, and / or maintain one or more of the modules described herein. Examples of memory devices include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), optical disk drive, cache, variations or combinations of one or more of the above, or any other suitable storage memory.

[0094] In some examples, the term "physical processor" generally refers to any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, a physical processor may access and / or modify one or more modules stored in the memory devices described above. Examples of physical processors include, but are not limited to, a microprocessor, a microcontroller, a central processing unit (CPU), a field programmable gate array (FPGA) implementing a soft-core processor, an application-specific integrated circuit (ASIC), portions of one or more of the above, variations or combinations of one or more of the above, or any other suitable physical processor.

[0095] Although shown as separate elements, the modules described and / or illustrated herein may represent portions of a single module or application. Moreover, in some embodiments, one or more of these modules may represent one or more software applications or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks. For example, one or more of the modules described and / or illustrated herein may represent modules stored and configured to run on one or more of the computing devices or systems described and / or illustrated herein. One or more of these modules may also represent all or part of one or more special-purpose computers configured to perform one or more tasks.

[0096] Further, one or more of the modules described herein may transform data, physical devices, and / or representations of physical devices from one form to another. For example, one or more of the modules presented herein may receive sensor data to be transformed, transform the sensor data, output the results of the transformation to display a virtual boundary, use the results of the transformation to define and / or modify the virtual boundary, and store the results of the transformation to establish the virtual boundary. Additionally or alternatively, one or more of the modules presented herein may transform a processor, volatile memory, non-volatile memory, and / or any other portion of a physical computing device from one form to another by executing on a computing device, storing data on a computing device, and / or possibly interacting with a computing device.

[0097] In some embodiments, the term "computer-readable medium" generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transmission-type media such as carrier waves, non-transitory-type media such as magnetic storage media (e.g., hard disk drives, tape drives, and floppy disks), optical storage media (e.g., compact discs (CDs), digital video discs (DVDs), and Blu-ray discs), electronic storage media (e.g., solid-state drives and flash media), and other distribution systems.

[0098] Embodiments of the present disclosure may include or be implemented in conjunction with a virtual reality system. A virtual reality is a form of reality that is conditioned in some manner before presentation to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. A virtual reality content may include entirely generated content or generated content combined with captured (e.g., real-world) content. A virtual reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereo video to create a three-dimensional effect for the viewer). Additionally, in some embodiments, a virtual reality may also be associated with applications, products, accessories, services, or some combination thereof, for example, used to create content in the virtual reality and / or otherwise used in the virtual reality (e.g., performing activities in the virtual reality). A virtual reality system that provides virtual reality content may be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing virtual reality content to one or more observers.

[0099] The process parameters and sequence of steps described and / or illustrated herein are given by way of example only and can be varied as necessary. For example, although the steps illustrated and / or described herein may be illustrated or described in a particular order, these steps do not necessarily have to be performed in the order illustrated or described. The various exemplary methods described and / or illustrated herein can also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.

[0100] The preceding description is provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to the precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the present disclosure. The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. In determining the scope of the present disclosure, reference should be made to the appended claims and their equivalents.

[0101] Unless otherwise noted, the terms "connected to" and "coupled to" (and their derivatives) as used in this specification and in the claims should be interpreted as allowing both direct and indirect connections (i.e., via other elements or components). Furthermore, the terms "a" or "an" as used in this specification and in the claims should be interpreted as meaning "at least one of." Finally, for ease of use, the terms "including" and "having" (and their derivatives) as used in this specification and in the claims are interchangeable with, and have the same meaning as, the word "comprising."

Claims

1. capturing image data associated with a view of a real-world environment using an imaging system of the head-mounted display system; displaying a pass-through view of the real-world environment to a user on a display of the head mounted display system, the pass-through view being based at least in part on the image data; receiving an indication of a reference altitude representing a plane of the real-world environment; monitoring an orientation of a direction indicator to generate first orientation data and establish a virtual boundary of a virtual world environment relative to the reference altitude; displaying the virtual boundary in the pass-through view of the real-world environment; determining whether the dimensions of the virtual boundary meet a minimum threshold; displaying a notification when the dimension of the virtual boundary does not meet the minimum threshold; and receiving a request from the user to modify the virtual boundary; monitoring an orientation of the directional indicator to generate second orientation data in response to the request from the user; modifying the virtual boundary based on the reference altitude and the second orientation data; A method comprising:

2. The method of claim 1 , further comprising, if the dimension does not meet the minimum threshold, presenting an indication of the minimum threshold to the user.

3. The method of claim 1 , wherein modifying the virtual boundary includes resetting the virtual boundary based on the reference altitude and the second orientation data.

4. The method of claim 1 , wherein the dimension comprises a length or width of the imaginary boundary.

5. The method of claim 1 , wherein the dimension comprises an area of ​​the imaginary boundary.

6. 2. The method of claim 1, further comprising displaying the virtual boundary in a first color if the dimension does not meet the minimum threshold, and displaying the virtual boundary in a second color if the dimension meets the minimum threshold.

7. The method of claim 1 , wherein the first orientation data and the second orientation data include altitude data of an altitude of the directional indicator relative to the reference altitude.

8. 8. The method of claim 7, wherein modifying the virtual boundary includes determining an intersection of the plane with a virtual line extending from the direction indicator at the altitude indicated by the altitude data and the orientation indicated by the second orientation data.

9. at least a portion of the captured image data has lens-induced distortion; displaying the pass-through view of the real-world environment includes correcting the lens-induced distortion in the image data to produce a compensated view of the real-world environment; The method of claim 1 , wherein displaying the virtual boundary comprises superimposing the virtual boundary onto the compensating view of the real-world environment.

10. The method of claim 1 , wherein modifying the virtual boundary comprises adding or subtracting portions from the virtual boundary.

11. The method of claim 1 , further comprising receiving a confirmation of the reference altitude.

12. A tangible, non-transitory computer-readable storage medium having stored thereon instructions that, when executed by a processing system, cause the processing system to: capturing image data associated with a view of a real-world environment using an imaging system of the head-mounted display system; displaying a pass-through view of the real-world environment to a user on a display of the head mounted display system, the pass-through view being based at least in part on the image data; receiving an indication of a reference altitude representing a plane of the real-world environment; monitoring an orientation of a direction indicator to generate first orientation data and establish a virtual boundary of a virtual world environment relative to the reference altitude; displaying the virtual boundary in the pass-through view of the real-world environment; determining whether the dimensions of the virtual boundary meet a minimum threshold; displaying a notification when the dimension of the virtual boundary does not meet the minimum threshold; and receiving a request from the user to modify the virtual boundary; monitoring an orientation of the directional indicator to generate second orientation data in response to the request from the user; modifying the virtual boundary based on the reference altitude and the second orientation data; A non-transitory computer-readable storage medium for performing operations including:

13. The non-transitory computer-readable storage medium of claim 12 , further comprising instructions for presenting the minimum threshold to the user if the dimension does not meet the minimum threshold.

14. The non-transitory computer-readable storage medium of claim 12 , wherein modifying the virtual boundary includes resetting the virtual boundary based on the reference altitude and the second orientation data.

15. The non-transitory computer-readable storage medium of claim 12 , wherein the dimension comprises a length or width of the virtual boundary.

16. The non-transitory computer-readable storage medium of claim 12 , wherein the dimensions include an area of ​​the virtual boundary.

17. 13. The non-transitory computer-readable storage medium of claim 12, further comprising instructions for displaying the virtual boundary in a first color if the dimension does not meet the minimum threshold, and displaying the virtual boundary in a second color if the dimension meets the minimum threshold.

18. The non-transitory computer-readable storage medium of claim 12 , wherein the first orientation data and the second orientation data include altitude data of an altitude of the directional indicator relative to the reference altitude.

19. 20. The non-transitory computer-readable storage medium of claim 18, wherein modifying the virtual boundary includes determining an intersection of the plane with a virtual line extending from the direction indicator at an altitude indicated by the elevation data and an orientation indicated by the second orientation data.

20. the captured view of the real-world environment has lens-induced distortion; displaying the pass-through view of the real-world environment includes correcting the lens-induced distortion in the captured view to create a compensated view of the real-world environment; The non-transitory computer-readable storage medium of claim 12 , wherein displaying the virtual boundary comprises superimposing the virtual boundary onto the compensating view of the real-world environment.

21. The non-transitory computer-readable storage medium of claim 12 , wherein modifying the virtual boundary comprises adding or subtracting a portion from the virtual boundary.

22. The non-transitory computer-readable storage medium of claim 12 , further comprising instructions for receiving confirmation of the reference altitude.

23. a display secured to the user attachment system; an imaging system; A direction indicator; Processing system and a head mounted display system comprising: capturing image data associated with a view of a real-world environment using the imaging system; presenting to the user on the display a pass-through view of the real-world environment based at least in part on the image data; and receiving an indication of a reference altitude representing a plane of the real-world environment; monitoring an orientation of the direction indicator to generate first orientation data and establish a virtual boundary of a virtual world environment relative to the reference altitude; displaying the virtual boundary in the pass-through view of the real-world environment; determining whether the dimensions of the virtual boundary meet a minimum threshold; displaying a notification when the dimension of the virtual boundary does not meet the minimum threshold; and receiving a request from the user to modify the virtual boundary; monitoring an orientation of the directional indicator to generate second orientation data in response to the request from the user; modifying the virtual boundary based on the reference altitude and the second orientation data; A head-mounted display system configured to:

24. 24. The head mounted display system of claim 23, wherein the processing system is further configured to present the minimum threshold to the user if the dimension does not meet the minimum threshold.

25. 24. The head mounted display system of claim 23, wherein modifying the virtual boundary includes resetting the virtual boundary based on the reference altitude and the second orientation data.

26. The head mounted display system of claim 23 , wherein the dimensions include a length or width of the virtual boundary.

27. The head mounted display system of claim 23 , wherein the dimensions include an area of ​​the virtual boundary.

28. 24. The head mounted display system of claim 23, wherein the processing system is further configured to display the virtual boundary in a first color if the dimension does not meet the minimum threshold, and to display the virtual boundary in a second color if the dimension meets the minimum threshold.

29. 24. The head mounted display system of claim 23, wherein the first orientation data and the second orientation data include altitude data of an altitude of the directional indicator relative to the reference altitude.

30. 30. The head mounted display system of claim 29, wherein modifying the virtual boundary includes determining an intersection of the plane with a virtual line extending from the direction indicator at an altitude indicated by the altitude data and an orientation indicated by the second orientation data.

31. the captured view of the real-world environment has lens-induced distortion; displaying the pass-through view of the real-world environment includes correcting the lens-induced distortion in the captured view to create a compensated view of the real-world environment; 24. The head mounted display system of claim 23, wherein displaying the virtual boundary comprises superimposing the virtual boundary onto the compensating view of the real world environment.

32. The head mounted display system of claim 23 , wherein modifying the virtual boundary comprises adding or subtracting a portion from the virtual boundary.

33. 24. The head mounted display system of claim 23, wherein the processing system is further configured to receive confirmation of the reference altitude.

Images