Systems and methods for detecting objects within a boundary of a defined space in artificial reality

Abstract

In one embodiment, a method includes generating a plurality of spatial points based on depth measurements of physical objects within a physical space surrounding a user, and determining the possible locations of the physical objects based on the spatial points. The method then renders a virtual space representing the physical space based on the locations of the physical objects. The virtual space may include virtual objects representing the physical objects. The method displays the virtual space to the user, and while displaying the virtual space, receives input from the user indicating the boundaries of subspaces within the virtual space, and detects that at least a portion of the virtual object is within a subspace. Finally, the method updates the virtual space to indicate that the portion of the virtual object is within the subspace.

Description

Technical Field

[0001] This disclosure generally relates to augmented reality, virtual reality, mixed reality, or hybrid reality environments. Background Technology

[0002] Artificial reality is a form of reality that has been modulated in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), surrealism, or combinations and / or derivatives thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., photographs of the real world). Artificial reality content may include video, audio, haptic feedback, or combinations thereof, any of which may be presented in a single channel or multiple channels (such as stereoscopic video that creates a three-dimensional effect for the viewer). Artificial reality may be associated with, for example, applications, products, accessories, services, or combinations thereof used to create content in and / or use in artificial reality (e.g., performing activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including head-mounted displays (HMDs) connected to a host computer system, standalone HMDs, mobile devices or computing systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0003] When immersing oneself in this artificial reality, especially VR, via an HMD, the user's view of the real world may be obstructed by the HMD's physical structure. Because real-world objects and intruders could pose a danger to the user, it's necessary to make the user aware of their presence. Summary of the Invention

[0004] In the context of augmented reality or virtual reality (AR / VR), users can virtually define a space that corresponds to an area in the physical space where they can safely move (e.g., to play games) while wearing a headset (such as an HMD). The boundaries of this space can be drawn along the ground using gestures or controllers (e.g., the orientation of gestures or controllers can resemble a virtual laser pointer), and the VR system then constructs virtual walls along these boundaries by extending 2D boundaries upwards from the ground. However, some users may struggle to grasp the necessary requirements for a clear and safe space before entering VR. Furthermore, when creating the initial boundaries, they may be focused on the actual process of tracking them (e.g., they might be focused on the floor they are drawing boundaries on with a laser pointer) and therefore may not notice unwanted objects left within the boundaries (such as chairs) or objects protruding beyond the boundaries from the outside (such as the edge of a table). These objects may also be invisible to the user while they are wearing an HMD and immersed in VR media, and therefore may pose a safety hazard due to the risk of collision.

[0005] The specific embodiments described herein relate to an intrusion detection system that detects potentially dangerous or vacant spaces and is designed to alert a user when an object is within a user-defined space. The system can generate a point cloud corresponding to observable objects in a room. However, this point cloud may be noisy, so the presence of a point does not necessarily indicate that it corresponds to an actual physical object. Therefore, the system can use them to assess the likelihood that a specific area in the space is occupied by a physical object and alert the user when that likelihood is sufficiently high, rather than relying on these points as absolute indicators of the presence of a physical object.

[0006] In a particular embodiment, the system can generate a virtual space corresponding to the physical area where the user is located, and divide this virtual space into voxels. Using computer vision techniques, the system can detect observable features around the user and generate corresponding point clouds for these features. Each point in the point cloud can have coordinates in the virtual space and fall within a voxel. In a particular embodiment, each voxel can have one of three possible states: idle, occupied, or unknown. A voxel can start from unknown, and rays can be projected from the estimated position of the HMD onto the points in the point cloud to determine which voxels are idle and which are occupied. In a particular embodiment, the presence of a point in a voxel is counted as a vote that the voxel is occupied by a physical object. When a ray is shone toward (or from) a point toward the user, each voxel through which the ray passes receives a vote for the "idle" state, based on the assumption that if the point corresponds to an observable feature visible to the camera of the HMD, then there should be no object in the space between the HMD and the point; otherwise, the feature corresponding to the point will not be observable. After performing this process on each point in the point cloud, the system can determine whether each voxel is likely to be occupied or idle based on the votes it receives.

[0007] However, since these points can be very noisy, especially as their distance from the HMD increases, time and / or density values ​​can be considered when determining whether a voxel in an occupied state is truly occupied. For example, a voxel that has received at least one "occupy" vote may still remain free if: it has received relatively few "occupy" votes (e.g., 1, 2, or 5 points in the point cloud within the voxel), its neighboring voxels are free, and / or it changes state rapidly over time. On the other hand, if a voxel has received relatively many "occupy" votes (e.g., 10, 15, or 20 points in the point cloud within the voxel), occupies neighboring voxels, and / or remains occupied for a threshold amount of time, it can be considered truly occupied by a physical object. These occupied voxels can then be compared to user-drawn boundaries to determine whether the detected object is within or outside the defined space and whether an alert is needed. Once an intruding object is identified, its presence can be alerted to the user in various visual or audio ways.

[0008] In certain embodiments, additional rules can specify whether occupancy voxels in the user-defined space should trigger an alarm. For example, a rule can be configured to filter out occupancy voxels that might correspond to a user's arm or leg. For instance, occupancy voxels that are more than a threshold distance from the user can trigger an alarm for the user, but occupancy voxels within the threshold distance can be ignored (to avoid alerting the user to their own arm or leg).

[0009] There are certain technical challenges in creating the boundaries of a defined space in which users will immerse themselves in artificial reality. When users draw boundaries, they may not understand the requirements for creating a clearly defined and secure space, or they may simply ignore objects within the space. If previously drawn boundaries are automatically recreated in future sessions upon system startup, users may not realize that objects in a larger physical space have been moved and that a once clearly defined space is no longer empty. Furthermore, users can draw the boundaries of a defined space but may not realize that additional free space can be included within the defined space, and that boundaries can be extended.

[0010] Therefore, providing information to users to draw their attention to these issues can help them create an ideal defined space. However, a technical challenge in this regard may include actually detecting any objects that intrude into the user-defined space during the setup process. Therefore, the solutions to this challenge proposed in the embodiments disclosed herein may include using voxels and voting algorithms to evaluate the probability that physical objects exist within the boundaries, classifying voxels with space points that appear to correspond to either occupied or unoccupied space. Another technical challenge may include determining whether there is additional free space within a larger physical space that can be incorporated into the user-defined space. The solutions to this challenge proposed in the embodiments disclosed herein may include informing the user of the existence of such free space and providing suggestions and / or tools to enable the user to edit or redraw the boundaries.

[0011] Certain embodiments disclosed herein may provide one or more technical advantages. As an example, the voting algorithm and time decay process used to detect voxels that may be occupied by physical objects during setup can provide more accurate detection of these objects solely through stereo reconstruction, and can further ensure that any identified occupied voxels are accurately updated when the user moves objects or adjusts boundaries. As another example, reminding the user of the presence of objects within the defined space boundaries while the user is immersed in artificial reality after setup can improve user safety. Similarly, informing the user that they can draw more ideal boundaries can enhance the user's experience with artificial reality media. Certain embodiments disclosed herein may not provide, provide some or all of the above-described technical advantages. Given the accompanying drawings, specification, and claims of this disclosure, one or more other technical advantages will be apparent to those skilled in the art.

[0012] The embodiments disclosed herein are merely examples, and the scope of this disclosure is not limited thereto. Specific embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. Embodiments of the invention are disclosed in particular in the appended claims relating to methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim class (e.g., method) may also be claimed in another claim class (e.g., system). Dependencies or references in the appended claims are chosen solely for formal reasons. However, any subject matter arising from the intentional retrospection to any prior claim (particularly multiple dependencies) may also be claimed so that any combination of claims and their features is disclosed and can be claimed, regardless of the dependency chosen in the appended claims. Claimable subject matter includes not only combinations of features as described in the appended claims but also any other combination of features in the claims, wherein each feature mentioned in a claim may be combined with any other feature or combination of features in the claims. Furthermore, any embodiments and features described or depicted herein may be claimed in individual claims and / or in any combination with any embodiments or features described or depicted herein or with any features of the appended claims.

[0013] In one aspect of the invention, a method is provided, the method comprising: generating a plurality of spatial points based on depth measurements of physical objects within a physical space surrounding a user; determining possible locations of the physical objects based on the spatial points; rendering a virtual space representing the physical space based on the locations of the physical objects, wherein the virtual space includes virtual objects representing the physical objects; displaying the virtual space to the user; and simultaneously displaying the virtual space: receiving from the user input indicating the boundaries of subspaces within the virtual space, and detecting that at least a portion of the virtual objects is within the subspaces; and updating the virtual space to indicate that the portion of the virtual objects is within the subspaces.

[0014] The boundaries can be open.

[0015] The boundary may include a predicted portion; and the portion that detects the virtual object may include the portion that detects the virtual object intersecting with the predicted portion of the boundary within the subspace.

[0016] The method may also include issuing an alert to the user indicating that part of the virtual object.

[0017] The method may also include receiving a second input from the user in response to an alarm, indicating a corrected boundary for excluding that portion of a virtual object from the subspace.

[0018] The method may also include detecting user movement of physical objects to new locations outside the boundary in response to an alarm.

[0019] The method may also include storing the location of the physical object in memory.

[0020] The method may further include: receiving a second user input indicating boundary completion; retrieving the location of a physical object from memory; and issuing an alarm to the user indicating that the portion of the virtual object is within a subspace at that location.

[0021] The method may also include dividing the physical space into multiple voxels, each voxel having a corresponding occupancy status value, wherein the set of voxels corresponding to the position of the physical object may have a corresponding occupied occupancy status value.

[0022] The method may also include determining an occupancy score for each voxel in a subset of spatial points contained within the voxel.

[0023] The method may also include: detecting the user's gaze; and determining whether the detected gaze is directed at the location of a physical object.

[0024] The method may further include: if the detected gaze does not point to the location of a physical object, storing the corresponding occupancy score of each voxel in the voxel set in memory; receiving a second input from the user indicating boundary completion; and issuing an alert to the user indicating the location of the physical object.

[0025] The method may also include displaying the voxel set to the user.

[0026] The method may also include reducing the occupancy score of the voxels in the voxel set if the detected gaze is directed at the location of a physical object.

[0027] The method may also include determining whether a physical object may have been moved based on spatial points.

[0028] When it is determined that a physical object may have been moved, the corresponding occupancy status value of the voxels in that voxel set is updated to unoccupied; and when it is determined that a physical object is unlikely to have been moved, the corresponding occupancy score of the voxels in that voxel set is increased.

[0029] The method may also include: determining the distance between the user and that part of the virtual object; and issuing an alert to the user if the distance is less than a threshold.

[0030] In one aspect of the invention, a system is provided comprising one or more processors and one or more computer-readable nontransitory storage media coupled to the one or more processors and including instructions operable, when executed by the one or more processors, such that the system: generates a plurality of spatial points based on depth measurements of physical objects within a physical space surrounding a user; determines the possible locations of the physical objects based on the spatial points; renders a virtual space representing the physical space based on the locations of the physical objects, wherein the virtual space includes virtual objects representing the physical objects; displays the virtual space to the user; and while displaying the virtual space: receives input from the user indicating the boundaries of subspaces within the virtual space, and detects that at least a portion of the virtual objects is within the subspaces; and updates the virtual space to indicate that the portion of the virtual objects is within the subspaces.

[0031] In one aspect of the invention, one or more computer-readable non-transitory storage media are provided to implement operable software that, when executed, causes: a plurality of spatial points to be generated based on depth measurements of physical objects within a physical space surrounding a user; the possible locations of physical objects to be located based on the spatial points; a virtual space representing the physical space to be rendered based on the locations of the physical objects, wherein the virtual space includes virtual objects representing the physical objects; the virtual space to be displayed to the user; and, while displaying the virtual space: receiving from the user input indicating the boundaries of subspaces within the virtual space, and detecting that at least a portion of the virtual objects is within the subspaces; and updating the virtual space to indicate that the portion of the virtual objects is within the subspaces. Attached Figure Description

[0032] Figure 1 An example method is shown for detecting objects that intrude into the user-defined space during the guardian setup.

[0033] Figure 2 An example network environment associated with artificial reality devices and social network environments is shown.

[0034] Figure 3 An example HMD system is shown that is wirelessly connected to a computer system.

[0035] Figures 4A-4C An example of creating a boundary around a user's guardian is shown.

[0036] Figure 5 An example HMD is shown.

[0037] Figure 6 An example method for determining the occupancy status of a voxel is shown.

[0038] Figure 7 An example procedure for determining the ray projection of an empty voxel is shown.

[0039] Figures 8A-8C An example visualization of the physical objects remaining within the guardian is shown and presented to the user during the guardian's setup.

[0040] Figures 9A-9D This example visualization shows a user drawing a boundary that intersects with various objects in the user's environment.

[0041] Figure 10 An example method for time decay of occupied voxels is shown.

[0042] Figure 11 An example computer system is shown. Detailed Implementation

[0043] Figure 1 An example method 100 for detecting objects intruding into a user-defined space during a guard setup is illustrated. In a particular embodiment, the system can determine whether one or more objects intrude into a defined space occupied by a user wearing an HMD (the defined space around the user may be referred to herein as the "guardian") during a guard setup. Method 100 may begin at step 110, where the intrusion detection system generates spatial points based on depth measurements of physical objects within the physical space surrounding the user. This can be achieved by capturing images of the user's surrounding environment using an outward-facing camera and processing the images to detect observable features of objects. For any detected objects, the system can generate spatial points based on depth measurements of the observable features of these physical objects. For example, these points can be generated using stereo techniques. However, the generated spatial points may be noisy or inaccurate. Therefore, at step 120, the intrusion detection system can determine the possible locations where physical objects may be present based on the spatial points. As will be described in further detail below, the system can use, for example, voxels and voting algorithms to evaluate the likelihood that a location in the space is occupied by a physical object. At step 130, the system can render a virtual space representing the physical space based on the locations of the physical objects. The virtual space may include virtual objects representing the physical objects. In step 140, the system can display the virtual space to the user. In step 150, while displaying the virtual space to the user, the system can receive user input from the user indicating the boundaries of subspaces or guardians within the virtual space. In step 160, while displaying the virtual space to the user, the system can detect that at least a portion of a virtual object representing a physical object is within a guardian. Finally, in step 170, the system can update the displayed virtual space to indicate to the user that the portion of the virtual object is within a guardian.

[0044] Figure 2An example network environment 200 associated with an artificial reality device and a social network environment is shown. Network environment 200 includes users 201 wearing HMDs, a host computer system 230, a social network system 260, and a third-party system 270 connected to each other via a network 210. Although Figure 2 A specific arrangement of user 201, host computer system 230, social networking system 260, third-party system 270, and network 210 is shown, but this disclosure considers any suitable arrangement of user 201, host computer system 230, social networking system 260, third-party system 270, and network 210. By way of example and not limitation, two or more of host computer system 230, social networking system 260, and third-party system 270 may be directly connected to each other, bypassing network 210. As another example, two or more of host computer system 230, social networking system 260, and third-party system 270 may be physically or logically located together, wholly or partially. Furthermore, although... Figure 2 A specific number of users 201, host computer system 230, social networking system 260, third-party system 270, and network 210 are shown, but this disclosure contemplates any suitable number of users 201, host computer system 230, social networking system 260, third-party system 270, and network 210. As an example and not a limitation, network environment 200 may include multiple users 201, host computer system 230, social networking system 260, third-party system 270, and network 210.

[0045] In a particular embodiment, user 201 may be an individual who interacts or communicates with or through social networking system 260. In a particular embodiment, social networking system 260 may be a network-addressable computing system hosting an online social network. Social networking system 260 may generate, store, receive, and transmit social network data, such as user profile data, concept profile data, social graph information, or other suitable data related to the online social network. Social networking system 260 may be accessed directly or via network 210 by other components of network environment 200. In a particular embodiment, social networking system 260 may include an authorization server (or (a plurality of) other suitable components) that allows user 201 to opt in or out of allowing social networking system 260 to record their actions or share their actions with other systems (e.g., third-party system 270), for example by setting appropriate privacy settings. The user's privacy settings may determine what information associated with the user can be recorded, how such information can be recorded, when such information can be recorded, who can record such information, with whom such information can be shared, and for what purpose such information can be recorded or shared. As needed, the authorization server can be used to enforce one or more privacy settings of a user of the social networking system 260 through blocking, data hashing, anonymization, or other suitable techniques. In a particular embodiment, the third-party system 270 may be a network-addressable computing system that can host media, such as games that a user can play via an HMD. The third-party system 270 can generate, store, receive, and transmit media and user data, such as the initial download of the game itself, data used during gameplay, or information about the user playing the game, such as game progress, preferences, or patterns. The third-party system data generated, stored, received, and transmitted can be determined by the preferences or privacy settings of the user stored as social networking data in the social networking system 260. The third-party system 270 can be accessed directly or via network 210 by other components of the network environment 200. In a particular embodiment, one or more users 201 can access the social networking system 260 or the third-party system 270 using one or more host computer systems 230, send data to it, and receive data from it. The host computer system 230 can access the social networking system 260 or the third-party system 270 directly, via network 210, or via a third-party system. As an example and not a limitation, the main computer system 230 may access the third-party system 270 via the social networking system 260. The main computer system 230 may be any suitable computing device, such as a personal computer, laptop, cellular phone, smartphone, tablet, or augmented / virtual reality device.

[0046] This disclosure considers any suitable network 210. By way of example and not limitation, one or more portions of network 210 may include an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless local area network (WLAN), wide area network (WAN), wireless wide area network (WWAN), metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more of these. Network 210 may include one or more networks 210.

[0047] Link 250 can connect host computer system 230, social networking system 260, and third-party system 270 to communication network 210 or to each other. This disclosure contemplates any suitable link 250. In a particular embodiment, one or more links 250 include one or more wired links (e.g., Digital Subscriber Line (DSL) or Cable Data Service Interface Specification (DOCSIS)), wireless links (e.g., Wi-Fi or Global Microwave Access Interoperability (WiMAX)), or optical links (e.g., Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)). In a particular embodiment, each of the one or more links 250 includes an ad hoc network, intranet, extranet, VPN, LAN, WLAN, WAN, WWAN, MAN, a portion of the Internet, a portion of the PSTN, a cellular-based network, a satellite-based network, another link 250, or a combination of two or more such links 250. Links 250 need not be identical throughout the network environment 200. One or more first links 250 may differ from one or more second links 250 in one or more aspects.

[0048] In the context of AR / VR, users wearing HMDs can enter or virtually create game spaces or guardians to interact with specific media within them. For example, a user can play a game inside a guardian. However, only the user wearing the HMD can see the guardian. Therefore, others who cannot see its virtual boundaries may inadvertently enter the guardian. Furthermore, during the setup of a guardian, the user may not notice objects inside or protruding from it. Since these individuals or other intruders could pose a collision hazard to the user, accurate detection of their presence and location is crucial to alert the user to their presence.

[0049] Figure 3 A wireless connection to a computer system 320 (such as...) is shown. Figure 2The example HMD 310 is a host computer system 230. HMD 310 may include one or more displays, a display engine, and computing resources. Conversely, host computer system 320 may include more computing resources. In some embodiments, host computer 320 may transmit each complete rendered frame to HMD 310 for display. In other embodiments, HMD 310 may render frames locally. Although HMD 310 is shown as wirelessly connected to host computer system 320, this disclosure contemplates any suitable type of connection, including wired connections. Similarly, while a single HMD is shown connected to a single host computer system, this disclosure contemplates multiple HMDs connected to one or more host computer systems.

[0050] Figures 4A-4C An example of creating a boundary around a user's guardian is shown. Figure 4A The image illustrates a user wearing an HMD 410 using gestures or a controller 415 that simulates a virtual laser pointer to create a guardian within a larger physical space 400. For example, the user drawing the boundaries of the guardian can treat the ground of the physical space as a bitmap of an image and draw the boundaries on that bitmap. During the actual drawing of the boundaries, the user may only see the outline of the boundaries, while the space inside the guardian is shown in a specific color when the boundaries are closed. The guardian should not contain any physical objects; this allows the user to move freely around the guardian, for example, while wearing the HMD 410 and playing a video game. Therefore, any physical objects 405 within the physical space 400 should be outside the boundaries 420 of the guardian. Thus, a visualization of objects intersecting the boundaries can be shown to the user during boundary creation. As an example, the visualization could be presented to the user in a second color. The boundaries 420 are then squeezed vertically upwards to the appropriate height to create a virtual wall 425, so that the completed guardian virtually surrounds and encircles the user.

[0051] However, users are prone to making mistakes when setting up guardians, and sometimes physical objects that may pose a collision hazard to users may be included in the guardian. Figure 4BAn example is shown where the user ignores object 405 within the guardian during the creation of boundary 420. This oversight can occur for several reasons. For example, the object might be small and easily missed, or the user might be focused on creating the guardian's boundary and might not be looking inside. This oversight can also occur if the user doesn't actively create the boundary but simply instructs the recreation of a previously created boundary. For example, the user might have brought the game system with the HMD to a new physical space and simply instructed the system to start using the same boundary or guardian dimension as the previous session. However, the user might not realize that objects in the new physical space have already been included within that guardian. As another example, the user could restart the system in the original physical space 400 with the same guardian dimension as the previous session without realizing that objects such as object 405 have moved into the guardian since the time of the previous session.

[0052] Figure 4C An example is shown where the boundary remains open, and the user drawing the boundary is unaware that object 435 outside boundary 420 will protrude into the guard as the boundary is drawn. Accordingly, when the system detects that object 435 protrudes beyond boundary 420 (although it appears to be outside the boundary), the user should be alerted to the presence of a portion 440 of object 435 within the guard, whether after the guard has been drawn or while the user is drawing the guard. In some embodiments, the system can detect this protrusion in real time, as the user actually draws the boundary up to and / or beyond the protruding portion 440. In other embodiments, the system can predict the possible location of the undrawn future portion of the boundary based on the user's position, movement, the already drawn boundary portion, and / or the position or orientation of the controller 415. Subsequently, the system can detect that this portion of the virtual object will intersect with the predicted portion of the boundary. Therefore, an alert indicating this portion of the virtual object can be provided to the user. Such an alert may include visual and / or haptic feedback of the protruding portion 440. The user can then move object 435 or be provided with tools to cover the alert or adjust the guard boundary 420.

[0053] Once created, it can verify and inspect any included objects or parts of objects, and then continuously monitor to detect any new intruders.

[0054] Figure 5An example HMD 504 is shown. In a particular embodiment, the HMD 504 may be an AR / VR HMD worn by user 502 within an AR / VR system 500. In a particular embodiment, the AR / VR system 500 may include the HMD 504, a controller 506, and a computing system 508. The HMD 504 may be worn over the user's eyes and provide visual content to user 502 via an internal display (not shown). The HMD 504 may have two separate internal displays, one for each of user 502's eyes. Figure 5 As shown, HMD 504 can completely cover the user's field of vision. By becoming the exclusive provider of visual information for user 502, HMD 504 achieves the goal of providing an immersive virtual reality experience. However, one consequence of doing so is that user 502 cannot see his surrounding physical environment while immersed in the virtual reality media because his vision is blocked by HMD 504. Therefore, the intrusion detection described in this paper is needed to provide the user with real-time visual information about his physical environment.

[0055] The HMD 504 can have an externally facing camera, such as... Figure 5 Four cameras 505A, 505B, 505C, and 505D (not shown) are arranged around the HMD. Although only four cameras 505A-505D are described, the HMD 504 can have any number of cameras facing any direction (e.g., an upward-facing camera for capturing ceiling or room lighting, a downward-facing camera for capturing the user's face and / or part of their body, additional forward or backward-facing cameras, and / or an internal camera for capturing the user's gaze for eye tracking). Externally facing cameras are configured to capture the physical environment around the user and can do so continuously to generate a series of frames (e.g., as video).

[0056] The captured frames can be processed to generate depth measurements of the physical object observed by cameras 505A-505D. Depth can be measured in various ways. In a particular embodiment, depth can be calculated based on stereo images. For example, the camera pair in cameras 505A-505D can share overlapping fields of view and be configured to capture images simultaneously. As a result, the same physical object can be captured simultaneously by both cameras in the pair. For example, a specific feature of the object can now be captured by a single pixel p in the image captured by camera 505A. A The same feature can appear at another pixel p in an image captured by the 505B camera. B As long as the depth measurement system knows that two pixels correspond to the same feature, it can use triangulation techniques to calculate the depth of the observed feature. For example, based on the position of the camera 505A in 3D space and p AA line can be projected from the camera 505A and pass through the pixel position relative to the field of view of the camera 505A. A A similar line can be projected from another camera, the 505B, and passes through pixel p. B Since the two pixels correspond to the same physical feature, the two lines intersect. The two intersecting lines and the imaginary line drawn between the two cameras 505A and 505B form a triangle, which can be used to calculate the distance between the observed feature and either camera 505A or 505B, or the spatial point where the observed feature is located.

[0057] In certain embodiments, these depth measurements can be used to create a depth map of physical space. Such a depth map can be densified because the stereo points generated by the above process can be sparse and concentrated on the edges and textured regions of physical objects. Densification can be enhanced by interpolating the depth map with intrinsic and extrapolation of these sparse stereo points. This enhancement can help obtain accurate reconstructions of textureless surfaces, such as white walls or tabletops, which might otherwise only be sparsely detected.

[0058] In certain embodiments, the pose (e.g., position and orientation) of the HMD 504 within the environment may be required. For example, in order to render an appropriate display for user 502 as user 502 moves around in a virtual environment, system 500 will need to determine his / her position and orientation at any given moment. Based on the HMD's pose, system 500 can further determine the viewpoint of any of cameras 505A-505D or any of the user's eyes. In certain embodiments, the HMD 504 may be equipped with an inertial measurement unit (“IMU”). Data generated by the IMU, along with stereo images captured by the outward-facing cameras, allows system 500 to calculate the pose of the HMD 504 using techniques such as SLAM (Simultaneous Localization and Mapping) or other suitable techniques.

[0059] In a particular embodiment, the artificial reality system 500 may also have one or more controllers 506 that enable the user 502 to provide input. The controllers 506 may communicate with the HMD 504 or a separate computing unit 508 via a wireless or wired connection. The controllers 506 may have any number of buttons or other mechanical input mechanisms. Furthermore, the controllers 506 may have an IMU, allowing the position of the controllers 506 to be tracked. The controllers 506 may be tracked based on predetermined patterns on the controllers. For example, the controllers 506 may have several infrared LEDs, or other known observable features that together form the predetermined pattern. Using a sensor or camera, the system 500 may be able to capture images of the predetermined patterns on the controllers. Based on the observed orientation of these patterns, the system can calculate the position and orientation of the controllers relative to the sensor or camera.

[0060] The AR / VR system 500 may also include a computer unit 508. The computer unit may be a separate unit physically separate from the HMD 504, or it may be integrated with the HMD 504. In embodiments where the computer 508 is a separate unit, the computer 508 may be wirelessly or wiredly coupled to the HMD 504. The computer 508 may be a high-performance device, such as a desktop or laptop computer, or a resource-constrained device, such as a mobile phone. High-performance devices may have a dedicated GPU and a high-capacity or constant power supply. On the other hand, resource-constrained devices may not have a GPU and may have a limited battery capacity. Therefore, algorithms that can be practically used in the AR / VR system 500 may be based on the capabilities of its computer unit 508.

[0061] Figure 6 An example method 600 for determining the occupancy state of voxels is shown. Depth measurements of physical objects generated from images can be inaccurate, especially with increasing distance from the camera. Therefore, 3D points acquired through this process may be noisy and unreliable. To improve the reliability of the collected data, the number of detected 3D points and / or temporal consistency can be considered. Method 600 may begin at step 610, where the physical space 400 is divided into multiple voxels to create a voxel grid. Each voxel may have a given size (e.g., 10cm × 10cm, 20cm × 33cm, etc.), and the size or resolution of the voxels may depend on factors such as memory or processing limitations of the AR / VR system, or the desired level of accuracy in object detection. Each voxel also has, for example, an idle, occupied, or unknown state; all voxels can start in an unknown state upon setup.

[0062] In a particular embodiment, each 3D point in the point cloud falls into a voxel in a voxel grid, and a voxel can contain many 3D points. Each 3D point corresponds to an object or feature that may be detected in physical space, and each point is counted as a vote for the occupancy of its corresponding voxel. If a given voxel contains enough points or votes, the system may be more inclined to believe that these points correspond to one or more actual physical objects within the voxel, rather than just noise. However, each voxel may be relatively large, and thousands of 3D points may be generated in each frame, with many points being very close to each other. Therefore, a re-filtering stage can be performed, in which nearby points within a given voxel are bundled together. For example, if the first voxel contains 15 points, these points can be bundled and treated as a single point with a weight of 15. This bundled point may contribute a weighted vote to the occupancy status of its corresponding voxel, as follows... Figure 7 As stated above.

[0063] In step 620, the number of points within each voxel of the voxel grid is determined. In step 630, the method determines whether the points within each voxel satisfy one or more threshold criteria (e.g., whether the number of points within a threshold is greater than a predefined number, such as 3, 5, 10, etc.). In a particular embodiment, the threshold criteria can be uniformly applied across the entire voxel grid. In other embodiments, different voxels can use different threshold criteria (e.g., the threshold can be a function of the distance between the voxel and the user, such that more distant voxels have higher threshold requirements than closer voxels). If the threshold criteria are satisfied, the method proceeds to step 631, where the voxel's state is updated to "occupied". A higher number of points within a voxel indicates a higher probability that these points accurately correspond to an object, while if only one or a few points are detected, these points are more likely to be noise. Similarly, a large number of points densely clustered together indicates a higher probability that they accurately correspond to a detected object, while if only a few scattered points are detected, these points are more likely to be noise. If the number of points within a voxel is not greater than the threshold, the method proceeds to step 632, where the state of the voxel is further evaluated to determine whether it should be "idle" or "unknown".

[0064] As a supplement to or alternative to step 630, step 640 can be performed. In step 640, the method determines for each detected point of each voxel whether the point has been detected within a threshold time period or a threshold number of frames. If the point has not been detected within the threshold time period, the method proceeds to step 680, in which the point is ignored and no vote for "occupancy" is added to the corresponding voxel. On the other hand, if the point has been detected within the threshold time period, the method proceeds to step 650, in which a vote for "occupancy" is added to the voxel containing the point. The longer a point has been detected, the greater the likelihood that it accurately corresponds to an object. For example, if a point is detected only for a very short time period, or if it appears or disappears in a series of frames, it is likely just noise; on the other hand, if a point is detected and remains detected for a certain period of time, it may correspond to an actual detected object. Therefore, the consistency of the point observed over time can be used to measure the vote for the point. In a particular embodiment, detected points can be filtered by projecting onto past images or frames. For example, the weight of a point that has been observed for the past five frames may be greater than that of a point observed only in the current frame. The method then proceeds to step 660, where the number of “occupied” votes is counted for each voxel.

[0065] In step 670, it is determined whether the number of votes counted is greater than a threshold. If so, the method proceeds to step 671, where the state of the corresponding voxel is set to "occupied"; otherwise, the method proceeds to step 672, where the state of the voxel is further evaluated to determine whether it should be "idle" or "unknown".

[0066] When determining the state of a voxel, one or both of the following two considerations can be used: point density as described with respect to steps 630-632 and time consistency as described with respect to steps 640-680, depending on factors such as the processing resource consumption and capabilities of the AR / VR system.

[0067] In a particular embodiment, once a voxel is assigned its state, the system can detect the number of consecutive voxels with an "occupied" state. If more than a threshold number of consecutive voxels are occupied, the system can determine that there is sufficient evidence to suggest that (multiple) actual objects are located at the locations of these adjacent voxels. Furthermore, time decay can be considered when detecting objects. For example, if a voxel is occupied but adjacent voxels are not occupied, after a certain period of time, the voxel's state can be set to "idle" or "unknown," since it is unlikely that an object would occupy only a single voxel. On the other hand, if a voxel is occupied for less than a threshold amount of time, but adjacent voxels are also occupied, the system can determine that an object exists and is moving through the location of the occupied voxel.

[0068] Figure 7An example procedure for ray projection to determine free voxels is shown. The system determines the state of other voxels in the voxel grid by projecting rays (e.g., R1-R7) from the HMD's viewpoint toward points in the point cloud into physical space 700. As previously described, points within a voxel are counted as votes for that voxel being occupied. For example, points P1, P2, and P3 in voxels v3, v3, and v5 are each counted as "occupied" votes. Therefore, voxel v3 will receive two votes for "occupied," and voxel v5 will receive one vote. Simultaneously, voxels between the HMD and each point intersected by the ray are given "free" votes because the space between the HMD and the intersection points should be empty. For example, ray R3 intersects voxels v14, v15, and v16 before reaching voxel v3, where the target point P2 is located. Therefore, voxels v14, v15, and v16 each receive one "free" vote. Based on these votes, the system can classify voxels as occupied, free, or unknown. For example, a voxel can be considered "idle" if the number of "idle" votes is significantly greater than the number of "occupied" votes. A voxel can be considered "occupied" if the number of "occupied" votes significantly exceeds the number of "idle" votes. A voxel's state can be considered "unknown" if the difference between "idle" and "occupied" votes is negligible. For example, in a particular embodiment, a numerical value can represent the statistical vote count for each voxel. Each "occupied" vote can increase this value by a specific amount (e.g., 1), and each "idle" vote can decrease the value by a specific amount. Therefore, after statistical voting, the numerical value can be negative, positive, or zero. In a particular embodiment, the numerical value can have a predefined range (e.g., -10 to +10). One subrange (e.g., +3 to +10) can correspond to an "occupied" state, a second subrange (e.g., -3 to -10) can correspond to an "idle" state, and a third subrange (e.g., -2 to 2) can correspond to an "unknown" state. A voxel with an "occupied" state indicates that the voxel may contain a physical object.

[0069] However, voxels must be able to dynamically update their occupancy state as the environment changes and object detection progresses, and allowing voxel values ​​to exceed certain thresholds could result in values ​​so large that additional votes cannot change their state within an acceptable timeframe. Therefore, each voxel can have a minimum and a maximum value (in the example above, the minimum could be -10, and the maximum could be 10). Each point or vote within a voxel can further have a set lifetime, and as that lifetime expires, the votes within the voxel can decrease. Thus, over time, the voxel's occupancy state can change from "occupied" to "unknown," as described below. Figure 10 As stated above.

[0070] In a particular embodiment, each voxel may store a floating-point value, where the voxel's state corresponds to a given subrange of values, as described above. Each point within a voxel votes on the state of the occupied voxel (which may be weighted); however, not all votes have the same weight. For example, a noise model may be used to weight the value of a point based on its distance from the HMD camera. With stereo reconstruction, point detection becomes less accurate and unreliable as the distance between a point and the camera increases. Therefore, when calculating the votes for a given voxel, the weight of distant points may be less than that of nearby points.

[0071] When an object poses a danger to a user, a visual and / or audio alarm can be issued to indicate the object's presence and / or location. However, these dangerous objects are typically those within the guard's perimeter, as users expect the space within the guard's boundaries to be safe (e.g., free of obstructions), while objects outside the guard may not be close enough to pose a danger. Therefore, in certain embodiments, a distinction can be made between occupied voxels within the guard and voxels outside the guard. When the physical space is divided into a voxel grid, the voxel positions can be compared to the positions of the guard's boundaries. Voxels considered "occupied" within the boundaries can trigger an alarm, while voxels outside the boundaries can be ignored.

[0072] In certain embodiments, a distance range can be set within the guard to control when an alarm is appropriately issued to the user for a detected intruder. For example, even if an intruder is inside the guard, it may be far enough away from the user that it poses no risk. For instance, an intruder, such as a person or a pet, might quickly cross a section of the guard that is far enough away from the user (e.g., 10, 12, 15 meters) or short enough (e.g., 1 second or less) that it will not, for example, impede their actions while playing a game. In such cases, it may be unnecessary or undesirable to interrupt the game.

[0073] Similarly, a threshold distance can be set around the user, and objects close to the user within that threshold distance may not trigger an alarm. For example, this could prevent alarms from being triggered in response to movement of the user's own arm or leg. The size of the threshold distance can be determined in several ways. As an example, it could simply be a predetermined size. As another example, the threshold could be based on the detection of a controller held by the user, and the distance to which detected objects are ignored could be a function of how close the user is holding the controller to their body (since objects within that distance could be part of the user's body or something the user intentionally touches). Another option to avoid alarms for the user's own movements is to perform user body tracking.

[0074] Figures 8A-8CThis example visualization shows the physical objects remaining within the Guardian presented to the user during Guardian setup. Once the system detects an intruder and determines that an alarm should be issued, there are many options regarding the nature or attributes of that alarm. For example, a particular visualization or audio technique could be based on which voxels are actually occupied, when they were detected, or the attributes of the media viewed by the user through the HMD. As an example, such as Figure 8A As shown, detected intruders (such as chairs) can be displayed to the user as a point cloud. As another example, Figure 8B The detected objects are shown as a visualization of the voxels themselves, displayed as actual occupancy. As yet another example, such as... Figure 8C As shown, intruders protruding into the guard, such as table corners, or those expected to protrude into the guard once the boundary is fully drawn, can be outlined to attract the user's attention. When the user is in the initial setup of the guard or drawing the boundary, the HMD operates in mixed reality or VR mode, where visualizations of the real world (e.g., based on 3D reconstruction techniques) are presented to the user as close to real-time as possible. Therefore, simply representing detected intruders as points, voxels, or lines can help draw their attention without consuming excessive processing power. Furthermore, adding color to the displayed visual alarms can further attract the user's attention when the surrounding physical space is shown in grayscale on an AR-type display.

[0075] Figures 9A-9D This example visualization 900 shows a user drawing boundaries that intersect with various objects in the user's environment. Users typically tend to draw boundaries through objects that intrude into an actually usable guardian (e.g., under a chair or table, on a wall, and through a box). Beyond simply looking down at objects intruding into the guardian, users may consciously draw through these objects for a variety of reasons, such as trying to maximize a smaller playspace, judging objects to be soft / less harmful and therefore posing the least risk if included in the guardian, underestimating the likelihood that objects might interfere with the guardian during play, or being unable to actually move objects. Therefore, users can be encouraged to improve the guardian's security (e.g., by removing intruding objects or redrawing boundaries), or they can be constantly made aware that objects are still intruding into the guardian (e.g., if the object is immovable).

[0076] Figure 9AExample scenario 900 is illustrated, where user 910 unknowingly includes a portion of an object within the guard. User 910 draws a large guard boundary that covers most of the room, extending to the edge of the wall. However, the user draws the boundary while facing forward, and thus draws a chair 920 behind the user. The chair 920 is outside the field of view when the boundary is created, and is therefore unintentionally included within the guard; the chair 920 can remain additionally outside the field of view while the user is immersed in VR media, and user 910 may collide with the chair 920 if the user moves backward. Therefore, an alert can be issued to user 910 to remind them of the presence of an object behind them within their guard. Users 910 can be further provided with tools to resolve this alert, such as the option to edit the boundary to exclude the chair 920, the option to redraw the boundary from scratch, or the option to override the alert. In a particular embodiment, in response to the alert, the system can receive a second input from the user indicating a corrected boundary that excludes a portion of the virtual object representing a physical object from the subspace / guardian.

[0077] Figure 9B Example scenario 900 is shown, where user 910 has intentionally included a portion of an object within the guard. User 910 is aware that a portion of sofa 930 intrudes into their guard. However, sofa 930 may be difficult to move, and the guard may have been drawn in to include as much available space as possible. Therefore, user 910 may prefer an option that allows them to exclude such a space-consuming and volume-consuming entity from their guard boundary. Therefore, the intrusion detection system can provide the user with tools or suggestions for adjusting the guard boundary in a way that excludes sofa 930. An alert can be issued to the user indicating the presence of sofa 930. As an example, the alert could illustrate the area intersecting with sofa 930 with a specific color (e.g., red) to indicate that the boundary should recede at that location, and tools could be provided accordingly to allow user 910 to erase the portion of the boundary intersecting with sofa 930 and redraw that portion so that it is completely in front of sofa 930, so that sofa 930 no longer intersects with the boundary.

[0078] Figure 9C Example scenario 900 is shown, where user 910 has additionally drawn walls across a larger physical space to maximize the available playspace within the guardian. Because such walls may be relatively featureless (e.g., plain white walls), the intrusion detection system can use textured light to detect where such walls intersect with the guardian's boundary. Using an infrared projector, structured light can be projected onto surfaces such as walls to detect their presence and intrusions into the guardian. User 910 can then be provided with tools to adjust the boundary or ignore alarms.

[0079] Figure 9D Example scenario 900 is illustrated, where user 910 has intentionally included parts of the objects within the guard and finds their presence acceptable. As shown, user 910 has drawn the boundaries of the guard using chair 920 and sofa 930. However, if the immersive media the user is viewing is considered static, such as a game that doesn't involve much movement, they might underestimate the extent to which chair 920 will intrude into their guard. Furthermore, due to the nature of the objects (e.g., the cushions of sofa 930), the user might perceive the sofa 930 as posing minimal threat. In this case, user 910 might desire an option to override the alarm and keep the guard as currently drawn. Depending on the nature of the media or game the user is playing, the presence of these objects 920 and 930 within the guard might be acceptable. For example, if user 910 is playing a game involving a lot of arm movement but not much walking, the intrusion detection system can determine whether the objects are tall enough to be considered obstacles to the user's body rather than their hands, and can determine that the presence of shorter objects such as sofa 930 will not obstruct the user's game or safety. User 910 may move or wave their arms on sofa 930; however, if user 910 begins to walk towards sofa 930, the system can issue an alarm to alert the user that there is an obstacle near their feet. As an example, the system can determine the distance between the user and this intrusive portion of sofa 930, and if that distance becomes less than a threshold, issue an alarm to the user. This alarm may include a visualization of sofa 930 or an alert indicating the distance between user 910 and sofa 930.

[0080] Once the intrusion detection system detects an intrusion object, it can provide the user with an alert and / or a suggestion to move any detected object. The system can then detect user movement of the physical object to a new location outside the boundary in response to the alert / suggestion. For example, if the user chooses to move the object instead of redrawing the boundary or overriding the alert, the system can confirm to the user that the object has been moved outside the boundary and can warn the user if the object is still within the boundary before being allowed entry into VR mode. In a particular embodiment, once the intrusion object leaves the boundary, whether by being moved or excluded from the boundary by user editing, the intrusion object will not be tracked.

[0081] The system can remember edits to boundaries and / or specific detected objects for use in future sessions. For example, adjustments to the shape or size of a guard can be saved, and the boundaries of this edit can be automatically recreated in future sessions upon system startup. Furthermore, if the intrusion detection system detects an intrusion object, the intrusion object from a previous session can be remembered, but the user can continue using the VR media without moving the object or adjusting the boundaries (in contrast, if the user did move the object in a previous session, it might not be necessary to remember the object's position). In a future session, if the user is looking in the direction of this remembered intrusion object, the system can assume the object is still there and check if this assumption is correct. If the object is detected again, a new alert can be issued to the user. If the user is not looking in the direction of the remembered object, the system may not detect the object and can therefore ignore it. Alternatively, if the remembered object is within the user's field of vision and therefore detectable, an alert can be issued to the user; if the remembered object is not within the user's field of vision, the system can prompt the user to turn and look in the direction of the remembered object so that the system can determine if the object is still there.

[0082] In a particular embodiment, once the boundaries have been drawn, the system can further recommend optimizations to the guardian. For example, the system can determine which areas of a larger physical space are free and which are occupied, rather than just considering the area within the guardian. If additional space is determined, the system can suggest that the user expand the guardian's boundaries, thereby helping the user create a larger game space. As mentioned above, the system can use structured light projected into the physical space to distinguish unremarkable but occupied areas (e.g., white walls).

[0083] Figure 10An example method 1000 for the time decay of occupied voxels is shown. To ensure that voxels previously identified as occupied are released within an acceptable timeframe, continuous reinforcement of detected obstacles can be used. Each point within a voxel has a lifetime; when that lifetime expires, its vote for its corresponding voxel is deleted unless the feature corresponding to that point is redetected. Thus, if an intruding object is moved, its corresponding 3D point decays over time, and as the number of points within the corresponding voxel decreases, the voxel's state changes from "occupied" to "unknown" or "idle." However, voxels affected by this time decay can be selected as those currently within the user's field of view. Decaying all occupied voxels in the scene can lead to forgetting the entire state of the space around the user, so the system cannot even infer anything detected behind the user a few seconds ago. This forgetting of such a state can negatively impact user safety; for example, if the system detects an intruding object and alerts the user, but the user continues to draw the boundary of the guard instead of immediately fixing or moving the intruding object, the system will not be able to alert the user after the boundary is completed that an obstacle has been detected behind them and needs to be repaired. Therefore, applying a time decay function to the voxel the user is currently viewing allows the system to balance user safety with measurement probability, reconstruction noise, and the user's expectation of how long it may take to update the voxel.

[0084] Method 1000 may begin with step 1010, wherein the intrusion detection system detects physical objects at locations within a subspace of the physical space surrounding the user or within the boundary of a guard. In step 1020, the system may determine occupancy voxels corresponding to the locations of the physical objects, and in step 1030, the system may determine occupancy scores for these occupancy voxels. Subsequently, in step 1040, the system may store these occupancy scores and the location information of the occupancy voxels in memory.

[0085] In step 1050, the intrusion detection system can detect the user's gaze, and in step 1060, the system can determine whether the detected gaze is directed at an occupying voxel (e.g., whether the gaze is directed at the location of a physical object, represented by the occupying voxel). If the user's gaze is not directed at an occupying voxel, and therefore the user is not looking at the object corresponding to the occupying voxel, the system can proceed to step 1061, in which the occupying score and location information of the occupying voxel are stored in memory. Subsequently, the system can receive a second user input from the user indicating that the guard boundary has been completed. The system can then retrieve the location of the physical object from memory and issue an alert to the user, indicating that the portion of the virtual object corresponding to the physical object is located at the storage location of the physical object within the guard, thereby alerting the user to the detection of an obstacle that the user has not yet addressed. If the user subsequently looks at the physical object, the system can display the occupying voxel corresponding to the physical object to the user.

[0086] If the user's gaze is directed at the occupied voxel, the method can proceed to step 1062, where the system can begin time decay of the occupied voxel and thus reduce its occupancy score. As an example, the time decay algorithm can use a constant decay rate, with parameters chosen such that a voxel at a distance from the device (e.g., 2 cm) is released within a timeframe (e.g., 1 second) without being enhanced by stereo detection of 3D points, without unreasonably hindering the user from continuing to play the game.

[0087] In step 1063, the intrusion detection system can determine whether a physical object may have been moved based on spatial points. In a particular embodiment, the system can use, for example, stereo reconstruction to detect whether the physical object is still at the location. If the system detects that the physical object is unlikely to have moved and is therefore likely still at the location, the system can increase the occupancy score of the occupancy voxel and maintain the state of the occupancy voxel as "occupied," thereby offsetting the time decay process. Conversely, if the system detects that the physical object may have moved and is therefore likely no longer at the location (e.g., after being moved by the user in response to an alert indicating its intrusion), the system can update the state of the occupancy voxel to "unoccupied" or "unknown" based on the decreased occupancy score due to the gradual disappearance of stereo points within the voxel. Thus, in the example where the user receives an alert about an object intruding on his guardian at a first location and responds by moving the object away from that first location, the visualization of the intruding object at the first location presented to the user (e.g., as a red voxel) may now gradually disappear over a time frame (e.g., 1 second) as the user views the first location. Because voxels have a maximum value (e.g., a maximum of 10), reducing the number of votes for a voxel and taking up the score does not take too long; in contrast, without this maximum value, decaying voxels by vote could require users to stare at the currently empty first position for an unreasonable amount of time.

[0088] Systems and Methods

[0089] Figure 11An example computer system 1100 is illustrated. In a particular embodiment, one or more computer systems 1100 perform one or more steps of one or more methods described or illustrated herein. In a particular embodiment, one or more computer systems 1100 provide the functionality described or illustrated herein. In a particular embodiment, software running on one or more computer systems 1100 performs one or more steps of one or more methods described or illustrated herein, or provides the functionality described or illustrated herein. The particular embodiments include one or more portions of one or more computer systems 1100. Herein, references to computer systems may cover computing devices and vice versa, where appropriate. Furthermore, references to computer systems may cover one or more computer systems, where appropriate.

[0090] This disclosure contemplates any suitable number of computer systems 1100. This disclosure contemplates computer systems 1100 in any suitable physical form. By way of example and not limitation, computer system 1100 may be an embedded computer system, a system-on-a-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system grid, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of the foregoing. Where appropriate, computer system 1100 may include one or more computer systems 1100; may be single or distributed; may span multiple locations; may span multiple machines; may span multiple data centers; or may reside in a cloud that may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1100 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 1100 may execute one or more steps of the methods described or illustrated herein in real time or in batch mode. Where appropriate, one or more computer systems 1100 may execute one or more steps of the methods described or illustrated herein at different times or in different locations.

[0091] In a particular embodiment, computer system 1100 includes a processor 1102, a memory 1104, a storage device 1106, an input / output (I / O) interface 1108, a communication interface 1110, and a bus 1112. While this disclosure describes and illustrates a particular computer system having a particular number of components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of components in any suitable arrangement.

[0092] In a particular embodiment, processor 1102 includes hardware for executing instructions, such as those constituting a computer program. By way of example, and not limitation, to execute instructions, processor 1102 may retrieve (or fetch) instructions from internal registers, internal caches, memory 1104, or storage device 1106; decode and execute the instructions; and then write one or more results to internal registers, internal caches, memory 1104, or storage device 1106. In a particular embodiment, processor 1102 may include one or more internal caches for data, instructions, or addresses. Where appropriate, this disclosure contemplates that processor 1102 may include any suitable number of suitable internal caches. By way of example, and not limitation, processor 1102 may include one or more instruction caches, one or more data caches, and one or more translation back buffers (TLBs). Instructions in the instruction cache may be copies of instructions in memory 1104 or storage device 1106, and the instruction cache may accelerate the retrieval of those instructions by processor 1102. The data in the data cache may be a copy of the data in memory 1104 or storage device 1106 for instructions executed at processor 1102 to perform operations; the result of a previous instruction executed at processor 1102 accessed by a subsequent instruction executed at processor 1102 or for writing to memory 1104 or storage device 1106; or other suitable data. The data cache can accelerate read or write operations of processor 1102. The TLB can accelerate virtual address translation of processor 1102. In a particular embodiment, processor 1102 may include one or more internal registers for data, instructions, or addresses. Where appropriate, this disclosure contemplates that processor 1102 may include any suitable number of suitable internal registers. Where appropriate, processor 1102 may include one or more arithmetic logic units (ALUs); is a multi-core processor; or includes one or more processors 1102. Although this disclosure describes and illustrates particular processors, this disclosure contemplates any suitable processor.

[0093] In a particular embodiment, memory 1104 includes main memory for storing instructions to be executed by processor 1102 or data to be operated by processor 1102. By way of example and not limitation, computer system 1100 may load instructions from storage device 1106 or another source (e.g., another computer system 1100) into memory 1104. Processor 1102 may then load instructions from memory 1104 into internal registers or internal caches. To execute instructions, processor 1102 may retrieve instructions from internal registers or internal caches and decode them. During or after instruction execution, processor 1102 may write one or more results (which may be intermediate or final results) to internal registers or internal caches. Processor 1102 may then write one or more of these results to memory 1104. In a particular embodiment, processor 1102 executes only the instructions in one or more internal registers or internal caches or memory 1104 (as opposed to storage device 1106 or elsewhere) and operates only on the data in one or more internal registers or internal caches or memory 1104 (as opposed to storage device 1106 or elsewhere). One or more memory buses (each of which may include an address bus and a data bus) may couple processor 1102 to memory 1104. Bus 1112 may include one or more memory buses, as described below. In a particular embodiment, one or more memory management units (MMUs) reside between processor 1102 and memory 1104 and facilitate access to memory 1104 requested by processor 1102. In a particular embodiment, memory 1104 includes random access memory (RAM). Where appropriate, the RAM may be volatile memory. Where appropriate, the RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Furthermore, where appropriate, the RAM may be single-port or multi-port RAM. This disclosure contemplates any suitable RAM. Where appropriate, memory 1104 may include one or more memories. Although this disclosure describes and illustrates specific memories, this disclosure contemplates any suitable memory.

[0094] In a particular embodiment, storage device 1106 includes a mass storage device for data or instructions. By way of example and not limitation, storage device 1106 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk drive, magneto-optical disk drive, magnetic tape drive, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, storage device 1106 may include removable or non-removable (or fixed) media. Where appropriate, storage device 1106 may be internal or external to computer system 1100. In a particular embodiment, storage device 1106 is a non-volatile solid-state memory. In a particular embodiment, storage device 1106 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmable ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically changeable ROM (EAROM), or flash memory, or a combination of two or more of these. This disclosure contemplates mass storage device 1106 in any suitable physical form. Where appropriate, storage device 1106 may include one or more storage control units that facilitate communication between processor 1102 and storage device 1106. Where appropriate, storage device 1106 may include one or more storage devices 1106. Although this disclosure describes and illustrates specific storage devices, this disclosure contemplates any suitable storage device.

[0095] In a particular embodiment, I / O interface 1108 includes hardware, software, or both for providing one or more interfaces for communication between computer system 1100 and one or more I / O devices. Where appropriate, computer system 1100 may include one or more of these I / O devices. One or more of these I / O devices can enable communication between a person and computer system 1100. By way of example and not limitation, I / O devices may include a keyboard, keypad, microphone, display, mouse, printer, scanner, speaker, camera, stylus, tablet computer, touchscreen, trackball, video camera, other suitable I / O devices, or combinations of two or more of these. I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O device and any suitable I / O interface 1108 for the I / O device. Where appropriate, I / O interface 1108 may include one or more device or software drivers to enable processor 1102 to drive one or more of these I / O devices. Where appropriate, I / O interface 1108 may include one or more I / O interfaces 1108. Although this disclosure describes and illustrates specific I / O interfaces, this disclosure contemplates any suitable I / O interface.

[0096] In a particular embodiment, communication interface 1110 includes hardware, software, or both for providing one or more interfaces for communication (injection, packet-based communication) between computer system 1100 and one or more other computer systems 1100 or one or more networks. By way of example and not limitation, communication interface 1110 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired networks, or a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks such as Wi-Fi networks. This disclosure contemplates any suitable network and any suitable communication interface 1110 for use therewith. By way of example and not limitation, computer system 1100 may communicate with one or more portions of an ad hoc network, personal area network (PAN), local area network (LAN), wide area network (WAN), metropolitan area network (MAN), or the Internet, or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system 1100 may communicate with a wireless PAN (WPAN) (e.g., BLUETOOTHWPAN), a Wi-Fi network, a Wi-Fi Max network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these. Where appropriate, computer system 1100 may include any suitable communication interface 1110 for any of these networks. Where appropriate, communication interface 1110 may include one or more communication interfaces 1110. Although specific communication interfaces are described and illustrated in this disclosure, any suitable communication interface is contemplated in this disclosure.

[0097] In a particular embodiment, bus 1112 includes hardware, software, or both for coupling components of computer system 1100 to each other. By way of example and not limitation, bus 1112 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1112 may include one or more buses 1112. Although this disclosure describes and illustrates specific buses, this disclosure contemplates any suitable bus or interconnect.

[0098] Here, where appropriate, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tape, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more thereof. Where appropriate, computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0099] Here, "or" is inclusive rather than exclusive, unless otherwise explicitly stated or the context otherwise indicates. Therefore, "A or B" here means "A, B, or both," unless otherwise explicitly stated or the context otherwise indicates. Furthermore, "and" is both joint and multiple, unless otherwise explicitly stated or the context otherwise indicates. Therefore, "A and B" here means "A and B, jointly or separately," unless otherwise explicitly stated or the context otherwise indicates.

[0100] The scope of this disclosure includes all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments described or illustrated herein that will be understood by those skilled in the art. The scope of this disclosure is not limited to the exemplary embodiments described or illustrated herein. Furthermore, although this disclosure describes and illustrates various embodiments herein as including specific components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or arrangement of any components, elements, features, functions, operations, or steps described or illustrated anywhere herein that will be understood by those skilled in the art. Furthermore, references in the appended claims to a means or system or a component of a means or system adapted to, arranged to, capable of, configured to, enabled, operable to, or operated to perform a particular function cover that means, system, or component, whether or not it or the particular function is activated, turned on, or unlocked, provided that the means, system, or component is so adapted, arranged, capable of, configured to, enabled, operable to, or operated. Moreover, although this disclosure describes or illustrates specific embodiments to provide particular advantages, specific embodiments may not provide, provide some, or all of these advantages.

Claims

1. A method for detecting objects in artificial reality, comprising a computing system: Multiple spatial points are generated based on depth measurements of physical objects within the physical space surrounding the user. Based on the spatial points, determine the possible locations where physical objects may exist; A virtual space representing the physical space is rendered based on the location of the physical object, wherein the virtual space includes a virtual object representing the physical object; The virtual space is displayed to the user; While displaying the virtual space: The user receives input from the user regarding the boundary of the subspace to be created within the virtual space, and In response to receiving the input, at least a portion of the virtual object is detected to be within the subspace; as well as Update the virtual space to indicate that the portion of the virtual object is within the subspace; as well as The method further includes: The physical space is divided into multiple voxels, each voxel having a corresponding occupancy status value, wherein the set of voxels corresponding to the position of the physical object has a corresponding occupied occupancy status value. Based on a subset of the plurality of spatial points contained within the voxels, a corresponding occupancy score is determined for each voxel in the voxel set. Detecting the user's gaze; and Determine whether the detected gaze is directed at the location of the physical object; and The method further includes the following: If the detected gaze is not directed at the location of the physical object, then: The corresponding occupancy score for each voxel in the set is stored in memory; Receive a second input from the user indicating that the boundary has been completed; and Issue an alert to the user indicating the location of the physical object; or If the detected gaze is directed at the location of the physical object, then: Reduce the corresponding occupancy score of the voxels in the set.

2. The method of claim 1, wherein the boundary is open.

3. The method according to claim 1, wherein: The boundary includes a predicted portion; as well as Detecting the portion of the virtual object within the subspace includes: detecting whether the portion of the virtual object intersects with the predicted portion of the boundary.

4. The method according to claim 1, further comprising: Issue an alert to the user indicating the portion of the virtual object; Optionally, In response to the alarm, receive from the user a second input indicating a corrected boundary for excluding the portion of the virtual object from the subspace; and / or In response to the alarm, user movement of the physical object to a new location outside the boundary is detected.

5. The method of claim 1, further comprising storing the location of the physical object in a memory.

6. The method according to claim 5, further comprising: Receive a second user input indicating that the boundary has been completed; Retrieve the location of the physical object from the memory; as well as An alert is issued to the user indicating that the portion of the virtual object is located within the subspace at that location.

7. The method of claim 1, further comprising displaying the voxel set to the user.

8. The method according to claim 1, further comprising: Based on the spatial point, determine whether the physical object may have moved; and optionally, When it is determined that the physical object may have been moved, the corresponding occupancy status value of the voxel in the set is updated to unoccupied; as well as When it is determined that the physical object is unlikely to have been moved, the corresponding occupancy score of the voxels in the set is increased.

9. The method according to claim 1, further comprising: Determine the distance between the user and the portion of the virtual object; as well as If the distance is less than the threshold, an alert is issued to the user.

10. A system for detecting objects in artificial reality, comprising one or more processors and one or more computer-readable nontransitory storage media, said one or more computer-readable nontransitory storage media being coupled to said one or more processors and including instructions operable, said instructions being executable by said one or more processors, such that the system: Multiple spatial points are generated based on depth measurements of physical objects within the physical space surrounding the user. Based on the spatial points, determine the possible locations where physical objects may exist; A virtual space representing the physical space is rendered based on the location of the physical object, wherein the virtual space includes a virtual object representing the physical object; The virtual space is displayed to the user; While displaying the virtual space: The user receives input from the user regarding the boundary of the subspace to be created within the virtual space, and In response to receiving the input, at least a portion of the virtual object is detected to be within the subspace; and Update the virtual space to indicate that the portion of the virtual object is within the subspace; and The instructions also enable the system to: The physical space is divided into multiple voxels, each voxel having a corresponding occupancy status value, wherein the set of voxels corresponding to the position of the physical object has a corresponding occupied occupancy status value. Based on a subset of the plurality of spatial points contained within the voxels, a corresponding occupancy score is determined for each voxel in the voxel set. Detect the user's gaze; as well as Determine whether the detected gaze is directed at the location of the physical object; and The instruction also causes the system to perform one of the following: If the detected gaze is not directed at the location of the physical object, then: The corresponding occupancy score for each voxel in the set is stored in memory; Receive a second input from the user indicating that the boundary has been completed; as well as Issue an alert to the user indicating the location of the physical object; or If the detected gaze is directed at the location of the physical object, then: Reduce the corresponding occupancy score of the voxels in the set.

11. One or more computer-readable non-transitory storage media, comprising software, which, when executed, is operable to: Multiple spatial points are generated based on depth measurements of physical objects within the physical space surrounding the user. Based on the spatial points, determine the possible locations where physical objects may exist; A virtual space representing the physical space is rendered based on the location of the physical object, wherein the virtual space includes a virtual object representing the physical object; The virtual space is displayed to the user; While displaying the virtual space: The user receives input from the user regarding the boundary of the subspace to be created within the virtual space, and In response to receiving the input, at least a portion of the virtual object is detected to be within the subspace; and Update the virtual space to indicate that the portion of the virtual object is within the subspace; When the software is executed, it can also operate as follows: The physical space is divided into multiple voxels, each voxel having a corresponding occupancy status value, wherein the set of voxels corresponding to the position of the physical object has a corresponding occupied occupancy status value. Based on a subset of the plurality of spatial points contained within the voxels, a corresponding occupancy score is determined for each voxel in the voxel set. Detect the user's gaze; as well as Determine whether the detected gaze is directed at the location of the physical object; and When executed, the software can also be operated to perform one of the following: If the detected gaze is not directed at the location of the physical object, then: The corresponding occupancy score for each voxel in the set is stored in memory; Receive a second input from the user indicating that the boundary has been completed; as well as Issue an alert to the user indicating the location of the physical object; or If the detected gaze is directed at the location of the physical object, then: Reduce the corresponding occupancy score of the voxels in the set.

Images