Systems and methods for detecting intrusion while in artificial reality

Through the computing system, virtual objects are generated and voxels are divided, and physical objects inside and outside the virtual boundary are detected, which solves the problem of invasive detection in the virtual reality environment, and accurately alerts to potential collision risks, improving user safety and immersion experience.

CN115136054BActive Publication Date: 2024-11-26CTRL-LABS CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202180009410.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-03-26
Filing Date
2021-02-04
Publication Date
2024-11-26
Estimated Expiration
2041-02-04

AI Technical Summary

Technical Problem

In a virtual reality environment, when users wear head-mounted displays, they cannot detect intrusions in the physical space in real time, resulting in potential collision risks and safety hazards. It is difficult for the prior art to accurately detect and warn users.

Method used

The computing system generates virtual objects, accesses the boundary definition of physical space, generates spatial points based on depth measurements, divides voxels and determines the occupancy state, casts light to evaluate the area occupancy situation, and sends an alert to the user in response to the detected physical object location. .

Benefits of technology

It realizes accurate detection and alerting of physical objects inside and outside the virtual boundary, improves users' security in the virtual reality environment, reduces potential collision risks, and enhances the reliability of the immersive experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115136054B_ABST
    Figure CN115136054B_ABST
Patent Text Reader

Abstract

In one embodiment, a method includes displaying a virtual object through a head mounted display (HMD) to a user wearing the HMD. The method then accesses a boundary definition corresponding to a boundary in a physical space around the user, and generates a plurality of spatial points based on depth measurements of physical objects in the physical space. A location where a physical object may exist is determined based on the spatial points. The method determines whether the location of the physical object is inside the boundary definition, and in response to the determination, issues an alert to the user.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to augmented reality, virtual reality, mixed reality, or hybrid reality environments and systems and methods for detecting intrusions while in an artificial reality. Background Art

[0002] Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (such as stereoscopic video that produces a three-dimensional effect to the viewer). Artificial reality may be associated with, for example, applications, products, accessories, services, or some combination thereof that are used to create content in artificial reality and / or used in artificial reality (e.g., to perform activities therein). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including headphones / head-mounted displays (HMDs) connected to a host computer system, standalone HMDs, mobile devices or computing systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0003] When immersed in such artificial reality (especially VR) via an HMD, the user's view of the real world may be blocked by the physical structure of the HMD. Because objects and intrusions in the real world may pose a danger to the user, the user needs to be made aware of their presence. Summary of the invention

[0004] In the context of augmented reality or virtual reality (AR / VR), a user can virtually create a space corresponding to an area in a physical space, in which the user can safely move (e.g., play games) while wearing a headset such as an HMD. The boundaries of this space can be drawn along the ground using gestures or controllers (e.g., the orientation of the gestures or controllers may be similar to a virtual laser pointer), and the VR system will then build virtual walls along these boundaries by extending the 2D boundaries on the ground upwards. However, these boundaries are virtual and cannot be seen by others who may accidentally enter the space. In addition, when creating the initial boundaries, the user may be focused on the actual process of outlining the initial boundaries (e.g., may be focused on the floor on which he is drawing the boundaries with a laser pointer), and therefore may not notice unwanted objects (such as chairs) that protrude through the boundaries from the outside (such as the edge of a table) or remain within the boundaries. These intrusions or objects may not be visible to the user when the user is wearing the HMD, and therefore may pose a safety hazard to the user due to the risk of collision. And because images captured solely by cameras on an HMD may not be able to detect whether a person or object has actually crossed a virtual boundary, more precise intrusion detection is needed to alert users to danger within a defined space.

[0005] An embodiment of the present invention includes a method, comprising: displaying one or more virtual objects to a user wearing a head-mounted display (HMD) through a computing system; accessing a boundary definition corresponding to a boundary within a physical space surrounding the user; generating a plurality of spatial points based on depth measurements of physical objects within the physical space; determining a location where a physical object may exist based on the spatial points; determining that the location of the physical object is inside the boundary definition; and, in response to determining that the location of the physical object is inside the boundary definition, issuing an alert to the user.

[0006] In some embodiments, the method further comprises dividing the physical space into a plurality of voxels, wherein the voxels are assigned corresponding occupancy state values ​​using spatial points, wherein based on the occupancy state value of at least a first voxel of the plurality of voxels, a location where the physical object may exist is determined.

[0007] In some embodiments, the occupancy state value of the first voxel is determined by: determining a subset of spatial points contained by the first voxel; determining that the subset of spatial points satisfies one or more criteria; and setting the occupancy state value of the first voxel to indicate that the first voxel is occupied.

[0008] In some embodiments, determining that the subset of spatial points satisfies one or more criteria includes determining that a number of spatial points in the subset satisfies a threshold number.

[0009] In some embodiments, determining where the physical object may be located includes determining whether a threshold number of contiguous voxels have corresponding occupancy state values ​​indicating that the contiguous voxels are occupied.

[0010] In some embodiments, alerting the user includes generating a display illustrating a first voxel associated with the physical object.

[0011] In some embodiments, the method further includes: determining that one or more second voxels of the plurality of voxels have corresponding occupancy state values ​​indicating that the one or more second voxels are occupied; and determining that the one or more second voxels are outside the boundary definition, wherein the generated display excludes the one or more second voxels.

[0012] In some embodiments, the method further includes: projecting rays toward locations where physical objects may exist; and determining areas in the physical space that may be unoccupied based on the projected rays.

[0013] In some embodiments, determining a potentially unoccupied area in a physical space includes: dividing the physical space into a plurality of voxels; and identifying a first subset of voxels that intersect with the ray before the projected ray intersects with a second subset of voxels corresponding to locations where physical objects may exist, wherein the potentially unoccupied area in the physical space is determined based on the first subset of voxels.

[0014] In some embodiments, the alert includes an outline or halo of a physical object overlaid on the media viewed by the user.

[0015] In some embodiments, the alert includes opening a portal in the media viewed by the user to allow the user to directly view the physical object.

[0016] In some embodiments, the method further includes: determining an intensity of media viewed by the user; and selecting an alert from a plurality of alert types based on the intensity of the media.

[0017] In some embodiments, the intensity indicates an activity level of the media.

[0018] In some embodiments, the alert is a visual or audio alert, and the corresponding visual representation or audio characteristic of the alert is selected based on the intensity of the media.

[0019] In some embodiments, issuing an alert includes pausing media viewed by the user to issue the alert.

[0020] In some embodiments, the physical object comprises: a physical object outside the boundary definition, and a physical object protruding inside the boundary definition.

[0021] In an embodiment, the method further comprises: receiving input from a user to create the boundary definition; and if a physical object is detected while creating the boundary definition, alerting the user upon receiving the input.

[0022] In an embodiment, the method further comprises: creating a radius of space around the user; and ignoring physical objects detected within the radius.

[0023] An embodiment of the present invention includes a system comprising: one or more processors; one or more computer-readable non-transitory storage media, coupled to one or more of the processors and comprising operable instructions, which when executed by one or more of the processors cause the system to: display one or more virtual objects to a user wearing a head-mounted display (HMD) through the HMD; access a boundary definition corresponding to a boundary within a physical space surrounding the user; generate a plurality of spatial points based on depth measurements of physical objects within the physical space; determine a location where a physical object may exist based on the spatial points; determine that the location of the physical object is inside the boundary definition; and in response to determining that the location of the physical object is inside the boundary definition, issue an alert to the user.

[0024] Embodiments of the present invention include one or more computer-readable non-transitory storage media containing software that, when executed, is operable to: display one or more virtual objects to a user wearing a head-mounted display (HMD) through the HMD; access a boundary definition corresponding to a boundary within a physical space surrounding the user; generate a plurality of spatial points based on depth measurements of physical objects within the physical space; determine locations where physical objects may exist based on the spatial points; determine that the location of the physical object is inside the boundary definition; and, in response to determining that the location of the physical object is inside the boundary definition, issue an alert to the user.

[0025] According to the general principles described herein, features from any embodiment described herein can be used in combination with each other in other aspects and embodiments. In particular, the specific features of the method described above can be used by analogy in a system and a computer-readable non-transitory storage medium according to the general principles described herein.

[0026] Certain embodiments described herein relate to a guarded area intrusion detection system designed to alert a user when an object (e.g., a person, pet, toy, furniture, etc.) enters the user's defined space. The system can generate a point cloud corresponding to observable objects in the room. However, this point cloud can be noisy, so the presence of a point does not necessarily mean that it corresponds to an actual physical object. False positives alerting the user can be distracting and detract from the immersive experience that artificial reality is intended to provide. Therefore, rather than relying on these points as an absolute indicator of the presence of a physical object, the system can use these points to assess the likelihood that a particular area in the space is occupied by a physical object and alert the user when the likelihood is high enough.

[0027] In a particular embodiment, the system may generate a virtual space corresponding to the physical area where the user is located, and divide the virtual space into voxels. Using computer vision techniques, the system may detect observable features of the user's surroundings and generate corresponding point clouds for those features. Each point in the point cloud may have coordinates in the virtual space and fall within a voxel. In a particular embodiment, each voxel may have one of three possible states: free, occupied, or unknown. The voxels may start from unknown, and rays may be cast from the estimated orientation of the HMD toward the points in the point cloud to determine which voxels are free and which are occupied. In a particular embodiment, the presence of a point in a voxel is recorded as a vote for the voxel to be occupied by a physical object. When a ray is cast toward the point (or from the point toward the user), each voxel that the ray passes through will receive a vote for the "free" state based on the following assumption: if the point corresponding to the observable feature is visible to the camera of the HMD, then the space between the HMD and the point should be free of objects; otherwise, the feature corresponding to the point will not be observable. After performing this process for each point in the point cloud, the system can determine whether each voxel is likely to be occupied or free based on the votes it receives.

[0028] However, since these points are noisy, especially as their distance from the HMD increases, temporal and / or density values ​​may be considered when determining whether a voxel with an occupied state is truly occupied. For example, if a voxel receives relatively few "occupied" votes (e.g., 1, 2, or 5 points in the point cloud are within the voxel), its neighboring voxels are free, and / or if it changes state rapidly over time, a voxel that has received at least one "occupied" vote may still be free. On the other hand, if a voxel has received relatively many "occupied" votes (e.g., 10, 15, or 20 points in the point cloud are within the voxel), has occupied neighboring voxels, and / or remains occupied for a threshold amount of time, it may be considered to be truly occupied by a physical object. These occupied voxels can then be compared to the boundaries drawn by the user to determine whether the detected object is within or outside the defined space and whether an alarm needs to be sounded.

[0029] In certain embodiments, additional rules may specify whether occupied voxels in a user's defined space should trigger an alarm. For example, a rule may be configured to filter out occupied voxels that may correspond to a user's arms or legs. For example, occupied voxels that are more than a threshold distance from the user may trigger an alarm for the user, but occupied voxels within the threshold distance may be ignored (so as not to alert the user to his own arms or legs). Similarly, it may be unnecessary or undesirable to trigger an alarm that pulls the user out of the game for objects that are too far away from the user to pose a danger.

[0030] Once an intrusive object is identified, the user may be alerted to its presence in a variety of visual or audio ways that involve varying degrees of disruption to the media the user is viewing. Safety considerations may be used to determine the extent to which the user is removed from the game, and the nature of the game may be used to adjust any visual and / or audio alerts. For example, if the user is playing a very active game that could run afoul of an intrusive object, the game may be paused when an alert is sounded. On the other hand, if the user is playing a calmer, more contemplative game, pausing the game or sounding a flashy alert may be unnecessary.

[0031] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to them. Specific embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. Embodiments according to the present invention are particularly disclosed in the appended claims relating to methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or references in the appended claims are selected only for formal reasons. However, any subject matter arising from intentional tracing back to any previous claim (especially multiple dependencies) may also be claimed, so any combination of claims and their features is disclosed and may be claimed, regardless of the dependencies selected in the appended claims. The subject matter that may be claimed includes not only the combination of features as set forth in the appended claims, but also any other feature combination in the claims, wherein each feature mentioned in the claims may be combined with any other feature or other feature combination in the claims. In addition, any embodiment and feature described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or with any feature of the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 An example method for detecting intrusion into a user's defined space is illustrated.

[0033] Figure 2 An example network environment associated with an artificial reality device is illustrated.

[0034] Figure 3 An example HMD system wirelessly connected to a computer system is illustrated.

[0035] Figure 4A-4C An illustrative example of setting custody zone boundaries is illustrated.

[0036] Figure 5 An example HMD is illustrated.

[0037] Figure 6 An example method for determining the occupancy state of a voxel is illustrated.

[0038] Figure 7 An example process for ray casting to determine free voxels is illustrated.

[0039] Figure 8A-8C Illustrated is an example visualization of physical objects remaining within a care zone presented to a user during setup of the care zone.

[0040] Figure 9A-9CIllustrated is an example visualization of presenting a physical intrusion moving into a controlled area to a user while the user is wearing an HMD.

[0041] Fig.10 An example computer system is illustrated. DETAILED DESCRIPTION

[0042] Figure 1 An example method 100 for determining whether one or more physical objects have invaded a defined space occupied by a user wearing an HMD (the defined space around the user may be referred to herein as a "guarded area") is illustrated. The method may begin at step 110, where a virtual object is displayed to the user through the HMD. At step 120, the system may access the definition of the virtual boundaries of the defined space within the larger physical space. The virtual boundaries of the defined space may be drawn by the user during system setup or automatically generated based on computer vision technology. The system may determine whether any physical objects appear within the physical space. This may be achieved by capturing images of the user's surroundings using an external camera and processing the images to detect observable features of the object. For any detected observable features, the system generates a plurality of spatial points based on depth measurements of these observable features of the physical object at step 130. For example, these points may be generated using stereo technology. However, the generated points may be noisy or inaccurate. Therefore, at step 140, the system determines the location where the detected physical object may exist based on the spatial points. As will be described in more detail below, the system may use, for example, voxels and voting algorithms to evaluate the likelihood that an area in the space is actually occupied by a physical object. Specifically, the system determines whether a possible physical object has intruded into the boundaries of the user's defined space or guard zone at step 150. Since unnoticed physical objects may pose a danger to an unsuspecting user wearing an HMD and therefore unable to see them, at step 160, an alert is issued to the user in response to detecting a physical object within the guard zone.

[0043] Figure 2 An example network environment 200 associated with a social networking system is illustrated. The network environment 200 includes a user 201 wearing an HMD, a host computer system 230, a social networking system 260, and a third-party system 270, which are interconnected via a network 210. Figure 2A particular arrangement of user 201, host computer system 230, social networking system 260, third-party system 270, and network 210 is illustrated, but the present disclosure contemplates any suitable arrangement of user 201, host computer system 230, social networking system 260, third-party system 270, and network 210. By way of example and not limitation, two or more of host computer system 230, social networking system 260, and third-party system 270 may be directly connected to each other, bypassing network 210. As another example, two or more of host computer system 230, social networking system 260, and third-party system 270 may be physically or logically co-located with each other, in whole or in part. Furthermore, although Figure 2 A particular number of users 201, host computer systems 230, social networking systems 260, third-party systems 270, and networks 210 are illustrated, but the present disclosure contemplates any suitable number of users 201, host computer systems 230, social networking systems 260, third-party systems 270, and networks 210. By way of example and not limitation, network environment 200 may include a plurality of users 201, host computer systems 230, social networking systems 260, third-party systems 270, and networks 210.

[0044] In certain embodiments, user 201 may be an individual interacting or communicating with social networking system 260 or through social networking system 260. In certain embodiments, social networking system 260 may be a network-addressable computing system that hosts an online social network. Social networking system 260 may generate, store, receive, and send social networking data, such as, for example, user profile data, concept profile data, social graph information, or other suitable data related to an online social network. Social networking system 260 may be accessed by other components of network environment 200 directly or via network 210. In certain embodiments, social networking system 260 may include an authorization server (or other suitable component(s)) that allows user 201 to opt in or out of having their behavior recorded or shared with other systems (e.g., third-party systems 270), for example, by setting appropriate privacy settings. The user's privacy settings may determine: what information associated with the user may be recorded, how information associated with the user may be recorded, when information associated with the user may be recorded, who may record information associated with the user, with whom information associated with the user may be shared, and for what purpose information associated with the user may be recorded or shared. The authorization server may be used to enforce one or more privacy settings of users of the social networking system 260 by blocking, data hashing, anonymization, or other appropriate techniques. In certain embodiments, the third-party system 270 may be a network addressable computing system that may host media such as games that a user may play through an HMD. The third-party system 270 may generate, store, receive, and send media and user data, such as, for example, the initial download of the game itself, data used during the game, or information about the user's game play, such as game progress, preferences, or modes. The third-party system data generated, stored, received, and sent may be determined by the preferences or privacy settings of the user stored as social networking data in the social networking system 260. The third-party system 270 may be accessed by other components of the network environment 200 directly or via the network 210. In certain embodiments, one or more users 201 may access the social networking system 260 or the third-party system 270, send data to it, and receive data from it using one or more host computer systems 230. The host computer system 230 may access the social networking system 260 or the third-party system 270 directly, via the network 210, or via a third-party system. By way of example and not limitation, host computer system 230 may access third party system 270 via social networking system 260. Host computer system 230 may be any suitable computing device, such as, for example, a personal computer, laptop computer, cellular phone, smartphone, tablet computer, or augmented / virtual reality device.

[0045] The present disclosure contemplates any suitable network 210. By way of example and not limitation, one or more portions of network 210 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more thereof. Network 210 may include one or more networks 210.

[0046] Links 250 may connect host computer system 230, social networking system 260, and third-party system 270 to communication network 210 or to each other. The present disclosure contemplates any suitable links 250. In certain embodiments, one or more links 250 include one or more wireline (such as, for example, digital subscriber line (DSL) or cable data service interface specification (DOCSIS)), wireless (such as, for example, Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)), or optical fiber (such as synchronous optical network (SONET) or synchronous digital hierarchy (SDH)) links. In certain embodiments, one or more links 250 each include a self-organizing network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 250, or a combination of two or more such links 250. Links 250 need not be identical throughout network environment 200. One or more first links 250 may differ from one or more second links 250 in one or more aspects.

[0047] In the context of AR / VR, a user wearing an HMD can enter or virtually create a game space or a guardian area in which to interact with certain specific media. For example, a user can play a game within a guardian area. However, this guardian area is only visible to the user wearing the HMD. Therefore, other people cannot see its virtual boundaries and may inadvertently walk into the guardian area. In addition, during the setting of the guardian area, the user may not notice objects inside the guardian area or protruding objects. Since these people or other intrusions may pose a collision hazard to the user, their presence and position need to be accurately detected to alert the user of their presence.

[0048] Figure 3 A wireless connection to a host computer system 320 (such as Figure 2310) is an example of an HMD 310 connected to a host computer system 230. HMD 310 may include one or more displays, a display engine, and computing resources. In contrast, host computer system 320 may include more computing resources. Host computer 320 may transmit each complete rendered frame to HMD 310 for display. In an alternative embodiment, HMD 310 itself may render the frame locally. Although HMD 310 is illustrated as being wirelessly connected to host computer system 320, the present disclosure contemplates any suitable type of connection, including wired connections. Similarly, although a single HMD is illustrated as being connected to a single host computer system, the present disclosure contemplates multiple HMDs connected to one or more host computer systems.

[0049] Figure 4A-4C An example of creating a defined space around a user, also referred to as a guard zone, is shown. Figure 4A , a user wearing an HMD 410 is shown using gestures or controllers 415 that simulate (e.g., a virtual laser pointer) to create a guard zone within a larger physical space 400. Ideally, this guard zone would be free of any physical objects; this would allow the user to move freely around the guard zone, for example, while wearing the HMD 410 and playing an active video game. Therefore, any physical objects 405 within the physical space 400 should be outside the boundary 420 of the guard zone. The boundary 420 would then be protruded vertically upward to the appropriate height, creating a virtual wall 425, so that the completed guard zone surrounds and encloses the user.

[0050] However, users are prone to error when setting the care zone, and physical objects that may pose a collision hazard to the user are sometimes accidentally included in the care zone. Figure 4B An example is illustrated, wherein the user overlooks the object 430 in the guardianship area during the creation of the boundary 420. This oversight may occur due to several reasons. For example, the object may be very small and easily not noticed, or the user may focus on creating the boundary of the guardianship area and may not view it inside. If the user is not actively creating the boundary, but simply instructs to recreate the previously created boundary, this oversight may also occur. For example, the user may have brought the game system to a new physical space, and simply instructs the system to start using the same boundary or guardianship area dimensions as the previous session. However, the user may not realize that the object in the new physical space has been included in this guardianship area. Alternatively, the user may restart the system in the original physical space 400 with the same guardianship area dimensions as the previous session, without realizing that objects such as object 430 have been moved into the guardianship area since the time of the previous session.

[0051] Figure 4CAn example is illustrated in which once the border is drawn, the user drawing the border does not notice that an object 435 outside the border 420 will protrude into the monitoring area. Therefore, despite appearing to be outside the border, when the system detects that the object 435 protrudes through the border 420, the user should be warned that a portion 440 of the object 435 exists within the monitoring area, whether after the monitoring area is drawn or while the user is drawing the monitoring area. When the user actually draws the border up to the protruding portion 440 and through the protruding portion 440, the system can detect the protruding portion in real time; alternatively, the system can predict the possible location of the border based on the user's orientation, movement, the portion of the border that has been drawn, and / or the position or direction of the controller 415. Once detected, an alert can be provided, such as a visualization of the protruding portion 440, so that the user can correct the monitoring area boundary 420 or move the object 435.

[0052] Once created, the guard zone can be verified and checked for any included objects or object parts, and then continuously monitored to detect any new intrusions.

[0053] Figure 5 An example AR / VR HMD 504 is illustrated as being worn by a user 502 within an AR / VR system 500. In certain embodiments, the AR / VR system 500 may include the HMD 504, a controller 506, and a computing system 508. The HMD 504 may be worn on the user's eyes and provide visual content to the user 502 via an internal display (not shown). The HMD 504 may have two separate internal displays, one for each eye of the user 502. Figure 5 As shown in FIG. 5 , HMD 504 can completely cover the user's field of view. By becoming the exclusive provider of visual information for user 502, HMD 504 achieves the goal of providing an immersive virtual reality experience. However, one consequence of this is that user 502 cannot see the physical environment around him while immersed in the virtual reality media because his vision is blocked by HMD 504. As such, intrusion detection as described herein is needed to provide users with real-time visual information about their physical environment.

[0054] HMD 504 may have an outward-facing camera, such as Figure 5504 is a user-friendly graphical user interface (GUI) of the present invention. The user-friendly graphical user interface (GUI) of the present invention is shown in FIG. 504 , which is a ...

[0055] The captured frames can be processed to generate a depth measurement of the physical object observed by the camera 505A-505D. The depth can be measured in a variety of ways. In a particular embodiment, the depth can be calculated based on a stereo image. For example, the camera pair among the cameras 505A-505D can share an overlapping field of view and be configured to capture images simultaneously. As a result, the same physical object may be captured simultaneously by two cameras in a pair. For example, a specific feature of an object may appear at a pixel pA in the image captured by the camera 505A, and the same feature may appear at another pixel pB in the image captured by the camera 505B. As long as the depth measurement system knows that the two pixels correspond to the same feature, it can use triangulation techniques to calculate the depth of the observed feature. For example, based on the orientation of the camera 505A in the 3D space and the pixel position of pA relative to the field of view of the camera 505A, a line can be projected from the camera 505A and pass through the pixel pA. A similar line can be projected from another camera 505B and pass through the pixel pB. Since the two pixels correspond to the same physical feature, the two lines intersect. The two intersecting lines and an imaginary line drawn between the two cameras 505A and 505B form a triangle that can be used to calculate the distance of the observed feature from the camera 505A or 505B or the point in space where the observed feature is located.

[0056] In certain embodiments, the posture (e.g., position and orientation) of the HMD 504 within the environment may be required. For example, in order to render an appropriate display for the user 502 as the user 502 moves around in the virtual environment, the system 500 will need to determine his position and orientation at any time. Based on the posture of the HMD, the system 500 can further determine the viewpoint of any one of the cameras 505A-505D or the viewpoint of any eye of the user. In certain embodiments, the HMD 504 can be equipped with an inertial measurement unit ("IMU"). The data generated by the IMU and the stereo images captured by the externally facing camera allow the system 500 to calculate the posture of the HMD 504 using, for example, SLAM (simultaneous localization and mapping) or other suitable techniques.

[0057] In certain embodiments, the artificial reality system 500 may also have one or more controllers 506 that enable the user 502 to provide input. The controller 506 may communicate with the HMD 504 or a separate computing unit 508 via a wireless or wired connection. The controller 506 may have any number of buttons or other mechanical input mechanisms. In addition, the controller 506 may have an IMU so that the orientation of the controller 506 can be tracked. The controller 506 may also be tracked based on a predetermined pattern on the controller. For example, the controller 506 may have several infrared LEDs or other known observable features that together form a predetermined pattern. Using a sensor or camera, the system 500 may be able to capture an image of a predetermined pattern on the controller. Based on the orientation of those patterns observed, the system can calculate the orientation and orientation of the controller relative to the sensor or camera.

[0058] The artificial reality system 500 may also include a computer unit 508. The computer unit may be a stand-alone unit physically separated from the HMD 504, or it may be integrated with the HMD 504. In an embodiment where the computer 508 is a separate unit, it may be communicatively coupled to the HMD 504 via a wireless or wired link. The computer 508 may be a high-performance device, such as a desktop or laptop computer, or a resource-limited device, such as a mobile phone. High-performance devices may have a dedicated GPU and a large or constant power supply. On the other hand, resource-limited devices may not have a GPU and may have a limited battery capacity. As such, the algorithm actually used by the AR / VR system 500 may depend on the capabilities of its computer unit 508.

[0059] Depth measurements of physical objects generated based on images can be imprecise, especially as the distance from the camera increases. Therefore, the 3D points obtained through this process can be noisy and unreliable. In order to improve the reliability of the collected data, methods that take into account the number and / or temporal consistency of the detected 3D points can be implemented, such as Figure 6 . Method 600 may begin at step 610, in which the physical space 400 is divided into a plurality of voxels, creating a voxel grid. Each voxel may have a given size (e.g., 10 cm x 10 cm, 20 cm x 33 cm, etc.), and the size or resolution of the voxel may depend on factors such as memory or processing constraints of the AR / VR system or the level of accuracy desired in target detection. Each voxel also has a state such as free, occupied, or unknown; at setup, all voxels may start in an unknown state.

[0060] In a particular embodiment, each 3D point in the point cloud falls into a voxel in the voxel grid, and a voxel can contain many 3D points. Each 3D point corresponds to an object or feature that may be detected in physical space, and each point is recorded as a vote that its corresponding voxel is occupied. If a given voxel contains enough points or votes in it, then the system will have more confidence that these points correspond to one or more actual physical objects within the voxel, and not just noise.

[0061] At step 620, the number of points within each voxel of the voxel grid is determined. At step 630, the method determines whether the points within each voxel meet one or more threshold criteria (e.g., whether the number of points within the threshold is greater than a predefined number, such as 3, 5, 10, etc.). In certain embodiments, the threshold criteria may be uniformly applied across the voxel grid. In other embodiments, different voxels may use different threshold criteria (e.g., the threshold may be a function of the distance between the voxel and the user, so that farther voxels have higher threshold requirements than closer voxels). If the threshold criteria are met, the method proceeds to step 631, in which the state of the voxel is updated to "occupied". A greater number of points within a voxel indicates a higher probability that these points accurately correspond to an object, while if only one or a few points are detected, there is a higher probability that these points are just noise. Similarly, many points that are densely clustered together indicate a higher probability that they accurately correspond to a detected object, while if only a few scattered points are detected, there is a higher probability that these points are just noise. If the number of points within the voxel is not greater than the threshold, the method proceeds to step 632 where the state of the voxel is further evaluated to determine whether it should be "free" or "unknown."

[0062] In addition to or in lieu of step 630, step 640 may be performed. At step 640, for each detected point for each voxel, the method determines whether the point has been detected for a threshold amount of time or a threshold number of frames. If the point has not been detected within the threshold amount of time, the method proceeds to step 680, where the point is ignored and a vote for "occupied" is not added to the corresponding voxel. On the other hand, if the point is detected within the threshold amount of time, the method proceeds to step 650, where an "occupied" vote is added to the voxel containing the point. The longer a point has been detected, the more likely it is that it accurately corresponds to an object. For example, if a point is only detected for a short period of time, or if it flashes in and out of view over a series of frames, it is likely just noise; on the other hand, if a point is detected and remains detected consistently over a certain amount of time, it may correspond to an actual detected object. Thus, the consistency of the point observed over time may be used to weight the vote for the point. For example, a point that has been observed for the past five frames may be weighted more heavily than a point that has only been observed in the current frame. The method then proceeds to step 660 where the number of "occupied" votes is counted for each voxel.

[0063] In step 670, it is determined whether the number of votes counted is greater than a threshold. If so, the method proceeds to step 671, in which the state of the corresponding voxel is set to "occupied"; otherwise, the method proceeds to step 672, in which the state of the voxel is further evaluated to determine whether it should be "free" or "unknown".

[0064] When determining the state of a voxel, one or both of considerations of point density as described with respect to steps 630-632 and temporal consistency as described with respect to steps 640-680 may be used, depending on factors such as processing resource consumption and capabilities of the AR / VR system.

[0065] In particular embodiments, once the voxels have been assigned their states, the system may detect a number of contiguous voxels that have an "occupied" state. If more than a threshold number of contiguous voxels are occupied, the system may determine that there is sufficient evidence that (multiple) actual objects are located at the locations of these adjacent voxels. Additionally, time decay may be taken into account when detecting objects. For example, if a voxel becomes occupied but adjacent voxels do not, after a certain amount of time has passed, the state of that voxel may be set to "free" or "unknown" since it would be unlikely that an object would occupy only a single voxel. On the other hand, if a voxel becomes occupied for less than a threshold amount of time, but adjacent voxels are also occupied, the system may determine that an object exists and is moving through the location of the occupied voxel.

[0066] like Figure 7As shown in , the system can determine the state of other voxels in the voxel grid by projecting rays (e.g., R1-R7) from the viewpoint of the HMD into the physical space 700 toward points in the point cloud. As previously described, points within a voxel are recorded as votes for the voxel to be occupied. For example, points P1, P2, and P3 in voxels v3, v3, and v5 are recorded as "occupied" votes, respectively. Therefore, voxel v3 will receive two votes for "occupied" and voxel v5 will receive one vote. At the same time, voxels between the HMD and each of the points intersected by the ray are given "free" votes because the space between the HMD and the intersection points should be empty. For example, ray R3 intersects voxels v14, v15, and v16 before it reaches voxel v3 where the target point P2 is located. Therefore, voxels v14, v15, and v16 each receive a "free" vote. Based on the votes, the system can consider the voxel to be occupied, free, or unknown. For example, if the number of "free" votes for a voxel is significantly more than the number of "occupied" votes for the voxel, the voxel can be considered "free". If the number of "occupied" votes is significantly more than the "free" votes, the voxel can be considered "occupied". If the difference between the "free" and "occupied" votes is negligible, the state of the voxel can be considered "unknown". For example, in a particular embodiment, the numerical value can represent a statistical vote count for each voxel. Each "occupied" vote can increase the value by 1, and each "free" vote can decrease the value by 1. Therefore, after statistics, the numerical value can be negative, positive, or zero. In a particular embodiment, the numerical value can have a predefined range (e.g., -10 to +10). One sub-range (e.g., +3 to +10) can correspond to the "occupied" state, the second sub-range (e.g., -3 to -10) can correspond to the "free" state, and the third sub-range (e.g., -2 to 2) can correspond to the "unknown" state. A voxel with an "occupied" state implies that the voxel may contain a physical object.

[0067] In certain embodiments, each voxel may store a floating point value, where the state of the voxel corresponds to a given sub-range of values, as discussed above. Each point within a voxel contributes a vote (which may be weighted) to the state of the occupied voxel; however, not all votes are given the same weight. For example, a noise model may be used to weight the value of a point based on the distance of the point from the HMD's camera. Using stereo reconstruction, the detection of points becomes imprecise and unreliable as the distance between the point and the camera increases. Therefore, when counting votes for a given voxel, distant points may be given less weight than close points.

[0068] When an object poses a danger to the user, a visual and / or audio alarm may be issued to the user to indicate the presence and / or location of the object. However, these dangerous objects are typically within the surveillance zone, because the user expects the space within the surveillance zone boundary to be safe (e.g., free of obstacles), while objects outside the surveillance zone may not be close enough to the user to pose a danger. Therefore, in certain embodiments, a distinction may be made between occupied voxels within the surveillance zone and those outside the surveillance zone. When the physical space is divided into a voxel grid, the location of the voxel can be compared to the boundary location of the surveillance zone. Voxels that are considered "occupied" within the boundary may trigger an alarm, while voxels outside the boundary may be ignored.

[0069] In certain embodiments, within a guard zone, a distance range may be set to control when it is appropriate to alert a user to a detected intrusion. For example, even if an intruder is within the guard zone, it may be far enough away from the user that it does not pose a danger. For example, an intruder such as a person or pet may quickly pass through a portion of the guard zone that is far enough (e.g., 10, 12, 15 meters) or short enough (e.g., 1 second or less) from the user so as not to impede his movement, for example, during game play. In this case, it may not be necessary or desirable to interrupt the game.

[0070] Similarly, a threshold distance may be set around the user, and objects within that threshold distance that are close to the user may not cause an alert to be issued. This may prevent an alert from being issued in response to movement of the user's own arm or leg, for example. The size of the threshold distance may be determined in several ways. It may simply be a set predetermined size. Alternatively, the threshold may be based on detection of a controller held by the user, and the distance within that distance at which detected objects are ignored may be a function of how close the user is holding the controller to their body (since objects within that distance are likely to be part of the user's body or something the user intentionally touched). Another option to avoid alerts being issued for the user's own movement is to perform body tracking of the user.

[0071] Once the system has detected an intrusion and determined that an alarm should be sounded, there are many options as to the nature or attributes of that alarm. For example, specific visualization or audio techniques may be based on, for example, which voxels are actually occupied, when they were detected, or based on the attributes of the media the user is viewing through the HMD. As an example, Fig. 8A As shown in FIG, a detected intrusion, such as a chair, can be displayed to the user as a point cloud. As another example, Figure 8B , which is shown as a visualization of the actual occupied voxels themselves. As yet another example, Figure 8CAs illustrated in , intrusions can be outlined to draw the user's attention, such as the corner of a table protruding into the monitoring area, or intrusions that are expected to protrude into the monitoring area once the boundary has been fully drawn. When the user is in the initial setup of the monitoring area or drawing the boundary, the HMD operates in a mixed reality mode or VR mode, in which a visualization of the real world (e.g., based on 3D reconstruction technology) is presented to the user as close to real time as possible. Therefore, simply representing detected intrusions as points, voxels, or lines may help to bring them to the user's attention without consuming too much processing power. In addition, in the case where the AR display displays the surrounding physical space in grayscale, adding color to the displayed visual alert can further attract the user's attention.

[0072] After the surveillance zone is established, the user is fully immersed in VR during media presentation. Therefore, when an intrusion is detected, this element of the real world must be presented on top of the VR content, or the VR content can be interrupted to show the intrusion. An example is in Fig.9A , where a user is immersed in a VR content of a game. An intruder such as a person passes through the guarded area, and the outline 910 of the intruder is overlaid on the VR content and displayed to the user. Instead of the outline, an image of the detected person may be displayed. Fig. 9B As illustrated in , the display of an intrusion passing through a surveillance area may alternatively be shown as a phantom halo 920 rather than a simple outline. The outline of the created outline or halo of the intrusion may be based on voxels of a voxel grid. Occupied voxels determined to correspond to an intrusion may trigger one or more of the cameras on the HMD to capture an image of the area where the intrusion is located. The captured image may then be filtered, such as by a gradient filter, to create an outline displayed to the user. Alternatively, to save resources, the VR system may display only a less explicit point cloud at the location of the detected intrusion. Such a filtering or point display process may be less resource intensive than performing stereo reconstruction to display the captured image directly to the user.

[0073] However, security considerations may dictate more intrusive alerts, particularly in very active games involving a large amount of user movement. In such cases, the game may be paused when an alert is sounded. Additionally or alternatively, as Fig. 9C As shown in , whether paused or in session, a portal 930 effect can be opened in the game, thereby providing a way for the user to view the real world image of the intrusion captured by the camera of the HMD. This opening of the portal can be automatic, or a visual and / or audible alarm can be issued to prompt the user to manually open the portal.

[0074] The characteristics of the visual and / or audio alerts, and the degree to which they pull the user out of the game or disrupt the user's immersive experience, can be further based on characteristics such as the visual / audio intensity or activity level of the media or game being viewed. For example, if the user is playing a very quiet, relaxing game that does not involve much movement, it may not be necessary to interrupt the game for certain types of intrusions, or in some cases it may not even be necessary to sound an alert. If an alert is sounded, such as for an approaching pet, the characteristics of the alert can be selected so as not to startle the user, which would occur if an intrusive, flashing visual or loud audio alert were sounded during a quiet game. On the other hand, if the user is playing a very fast-paced game, or one with bright or psychedelic visual characteristics, a simple outline of an intrusion may not be noticed, and therefore the game may need to be paused, or a visual or audio alert may need to be generated that clearly emerges from such media.

[0075] System and method

[0076] Fig.10 An example computer system 1000 is illustrated. In particular embodiments, one or more computer systems 1000 perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems 1000 provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems 1000 performs one or more steps of one or more methods described or illustrated herein, or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 1000. In this document, references to computer systems may encompass computing devices, and vice versa where appropriate. In addition, references to computer systems may encompass one or more computer systems where appropriate.

[0077] The present disclosure contemplates any suitable number of computer systems 1000. The present disclosure contemplates computer systems 1000 taking any suitable physical form. By way of example and not limitation, computer system 1000 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a computer on a module (COM) or a system on a module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system grid, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more thereof. Where appropriate, computer system 1000 may include one or more computer systems 1000; be single or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. Where appropriate, one or more computer systems 1000 may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein.

[0078] In a particular embodiment, computer system 1000 includes a processor 1002, a memory 1004, a storage device 1006, an input / output (I / O) interface 1008, a communication interface 1010, and a bus 1012. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0079] In a particular embodiment, the processor 1002 includes hardware for executing instructions, such as those that constitute a computer program. By way of example and not limitation, to execute instructions, the processor 1002 may retrieve (or fetch) instructions from an internal register, an internal cache, a memory 1004, or a storage device 1006; decode and execute them; and then write one or more results to an internal register, an internal cache, a memory 1004, or a storage device 1006. In a particular embodiment, the processor 1002 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1002 that includes any suitable number of any suitable internal caches. By way of example and not limitation, the processor 1002 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of instructions in the memory 1004 or the storage device 1006, and the instruction cache may speed up the retrieval of those instructions by the processor 1002. The data in the data cache may be a copy of data in the memory 1004 or storage device 1006 for instructions executed on the processor 1002 to operate on it; the results of previous instructions executed at the processor 1002 for access by subsequent instructions executed at the processor 1002 or for writing to the memory 1004 or storage device 1006; or other suitable data. The data cache may speed up read or write operations of the processor 1002. The TLB may speed up virtual address translation for the processor 1002. In a particular embodiment, the processor 1002 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1002 including any suitable number of any suitable internal registers. Where appropriate, the processor 1002 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 1002. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0080] In a particular embodiment, the memory 1004 includes a main memory for storing instructions for the processor 1002 to execute or data for the processor 1002 to operate on. As an example and not limitation, the computer system 1000 can load instructions from the storage device 1006 or another source (such as, for example, another computer system 1000) to the memory 1004. The processor 1002 can then load the instructions from the memory 1004 to an internal register or an internal cache. In order to execute the instructions, the processor 1002 can retrieve the instructions from the internal register or the internal cache and decode them. During or after the execution of the instructions, the processor 1002 can write one or more results (which can be intermediate or final results) to the internal register or the internal cache. The processor 1002 can then write one or more of those results to the memory 1004. In a particular embodiment, the processor 1002 executes only instructions in one or more internal registers or internal caches or in memory 1004 (as opposed to storage device 1006 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 1004 (as opposed to storage device 1006 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 1002 to the memory 1004. Bus 1012 may include one or more memory buses, as described below. In a particular embodiment, one or more memory management units (MMUs) reside between the processor 1002 and the memory 1004 and facilitate access to the memory 1004 requested by the processor 1002. In a particular embodiment, the memory 1004 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port or multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 1004 may include one or more memories 1004. Although this disclosure describes and illustrates particular memories, this disclosure contemplates any suitable memories.

[0081] In a particular embodiment, the storage device 1006 includes a mass storage device for data or instructions. As an example and not limitation, the storage device 1006 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more thereof. Where appropriate, the storage device 1006 may include a removable or non-removable (or fixed) medium. Where appropriate, the storage device 1006 may be inside or outside the computer system 1000. In a particular embodiment, the storage device 1006 is a non-volatile solid-state memory. In a particular embodiment, the storage device 1006 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more thereof. The present disclosure contemplates a mass storage device 1006 in any suitable physical form. Where appropriate, the storage device 1006 may include one or more storage control units that facilitate communication between the processor 1002 and the storage device 1006. Where appropriate, storage devices 1006 may include one or more storage devices 1006. Although this disclosure describes and illustrates particular storage devices, this disclosure contemplates any suitable storage devices.

[0082] In a particular embodiment, the I / O interface 1008 includes hardware, software, or both, providing one or more interfaces for communication between the computer system 1000 and one or more I / O devices. Where appropriate, the computer system 1000 may include one or more of these I / O devices. One or more of these I / O devices can enable communication between a person and the computer system 1000. As an example and not limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a camera, a stylus, a tablet computer, a touch screen, a trackball, a camera, other suitable I / O devices, or a combination of two or more thereof. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1008 for them. Where appropriate, the I / O interface 1008 may include one or more device or software drivers so that the processor 1002 can drive one or more of these I / O devices. Where appropriate, the I / O interface 1008 may include one or more I / O interfaces 1008. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0083] In a particular embodiment, the communication interface 1010 includes hardware, software, or both, providing one or more interfaces for communication between the computer system 1000 and one or more other computer systems 1000 or one or more networks (such as, for example, packet-based communication). By way of example and not limitation, the communication interface 1010 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other wired network, or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network such as a WI-FI network. The present disclosure contemplates any suitable network and any suitable communication interface 1010. By way of example and not limitation, the computer system 1000 may be connected to a self-organizing network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more parts of the Internet, or a combination of two or more thereof. One or more parts of one or more of these networks may be wired or wireless. As an example, the computer system 1000 may be connected to a wireless PAN (WPAN) (such as, for example, a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more thereof. Where appropriate, the computer system 1000 may include any suitable communication interface 1010 for any of these networks. Where appropriate, the communication interface 1010 may include one or more communication interfaces 1010. Although the present disclosure describes and illustrates particular communication interfaces, the present disclosure contemplates any suitable communication interface.

[0084] In a particular embodiment, bus 1012 includes hardware, software, or both that couples components of computer system 1000 to each other. By way of example and not limitation, bus 1012 may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front side bus (FSB), a hypertransport (HT) interconnect, an industry standard architecture (ISA) bus, an INFINIBAND interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus, or other suitable busses or a combination of two or more thereof. Where appropriate, bus 1012 may include one or more buses 1012. Although the present disclosure describes and illustrates a particular bus, the present disclosure contemplates any suitable bus or interconnect.

[0085] As used herein, where appropriate, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more thereof. Where appropriate, the computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0086] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or the context indicates otherwise. Furthermore, "and" is both conjunctive and plural, unless expressly indicated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or the context indicates otherwise.

[0087] The scope of the present disclosure covers all changes, substitutions, variations, alterations and modifications to the example embodiments described or illustrated herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or illustrated herein. In addition, although the present disclosure describes and illustrates the various embodiments herein as including specific components, elements, features, functions, operations or steps, any of these embodiments may include any combination or arrangement of any components, elements, features, functions, operations or steps described or illustrated anywhere herein that will be understood by those of ordinary skill in the art. In addition, references to devices or systems or components of devices or systems adapted to, arranged to, capable of, configured to, enabled to, operable or operable to perform specific functions in the appended claims cover the device, system, component, regardless of whether it or the specific function is activated, started or unlocked, as long as the device, system or component is so adapted, arranged, capable of, configured, enabled, operable or operable. In addition, although the present disclosure describes or illustrates specific embodiments as providing specific advantages, specific embodiments may not provide these advantages, provide some or all of these advantages.

Claims

1. A method for detecting intrusion in an artificial reality, comprising: Displaying one or more virtual objects to a user wearing the head mounted display (HMD) through the HMD; accessing: (i) a boundary definition corresponding to a boundary within a physical space surrounding the user, and (ii) a threshold distance from the user, wherein a physical object detected inside the boundary definition and beyond the threshold distance triggers an alarm and a physical object detected inside the boundary definition and within the threshold distance is ignored; generating a plurality of spatial points based on depth measurements of physical objects within the physical space; determining a possible location of a physical object based on the spatial point; determining that the position of the physical object is inside the boundary definition; as well as In response to determining that the position of the physical object is inside the boundary definition and exceeds the threshold distance, an alert is issued to the user.

2. The method according to claim 1, further comprising: The physical space is divided into a plurality of voxels, wherein the voxels are assigned corresponding occupancy state values ​​using the spatial points, wherein the location where the physical object may exist is determined based on the occupancy state value of at least a first voxel of the plurality of voxels.

3. The method according to claim 2, wherein the occupancy state value of the first voxel is determined by: determining a subset of the spatial points contained by the first voxel; determining that the subset of spatial points satisfies one or more criteria; and The occupancy state value of the first voxel is set to indicate that the first voxel is occupied.

4. The method of claim 3, wherein determining that the subset of spatial points satisfies the one or more criteria comprises: It is determined that the number of spatial points in the subset satisfies a threshold number.

5. The method of claim 2, wherein determining the location where the physical object may exist comprises: A determination is made as to whether a threshold number of contiguous voxels have corresponding occupancy state values ​​indicating that the contiguous voxels are occupied.

6. The method of claim 2, wherein alerting the user comprises: A display is generated illustrating the first voxel associated with the physical object.

7. The method according to claim 6, further comprising: determining that one or more second voxels of the plurality of voxels have respective occupancy state values ​​indicating that the one or more second voxels are occupied; as well as determining that the one or more second voxels are outside the boundary definition, wherein the display generated excludes the one or more second voxels.

8. The method according to claim 1, further comprising: Casting a ray toward the location where the physical object may exist; as well as Areas of the physical space that may be unoccupied are determined based on the projected light rays.

9. The method of claim 8, wherein determining the area of ​​the physical space that is likely unoccupied comprises: dividing the physical space into a plurality of voxels; as well as identifying a first subset of the voxels that intersect the ray before the ray is cast and intersects a second subset of the voxels corresponding to the locations where the physical objects may be present, Wherein the region of the physical space that is likely to be unoccupied is determined based on the first subset of voxels.

10. The method of claim 1, wherein the alert comprises: An outline or halo of the physical object overlaid on media viewed by the user.

11. The method of claim 1 , wherein the alert comprises: A portal is opened in the media viewed by the user to allow direct viewing of the physical object by the user.

12. The method according to claim 1, further comprising: determining the intensity of media viewed by a user; as well as Based on the intensity of the media, the alert is selected from among a plurality of alert types. The method of claim 12 , wherein the intensity indicates an activity level of the media.

14. The method of claim 12, wherein the alert is a visual or audio alert, and a corresponding visual characteristic or audio characteristic of the alert is selected based on the intensity of the media.

15. The method of claim 1, wherein issuing the alarm comprises: Media viewed by the user is paused to sound an alert.

16. The method of claim 1, wherein the physical object comprises: A physical object outside the boundary definition, the physical object protruding inside the boundary definition.

17. The method according to claim 1, further comprising: receiving input from the user to create the boundary definition; as well as If the physical object is detected when creating the boundary definition, the alert is issued to the user upon receiving the input.

18. The method of claim 1, further comprising: creating a radius of space around said user; as well as Ignore physics objects detected within the said radius.

19. A system for detecting intrusion in artificial reality, comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising operable instructions that, when executed by one or more of the processors, cause the system to: Displaying one or more virtual objects to a user wearing the head mounted display (HMD) through the HMD; accessing: (i) a boundary definition corresponding to a boundary within a physical space surrounding the user, and (ii) a threshold distance from the user, wherein a physical object detected inside the boundary definition and beyond the threshold distance triggers an alarm and a physical object detected inside the boundary definition and within the threshold distance is ignored; generating a plurality of spatial points based on depth measurements of physical objects within the physical space; determining a possible location of a physical object based on the spatial point; determining that the position of the physical object is inside the boundary definition; as well as In response to determining that the position of the physical object is inside the boundary definition and exceeds the threshold distance, an alert is issued to the user.

20. One or more computer-readable non-transitory storage media containing software that, when executed, is operable to: Displaying one or more virtual objects to a user wearing the head mounted display (HMD) through the HMD; accessing: (i) a boundary definition corresponding to a boundary within a physical space surrounding the user, and (ii) a threshold distance from the user, wherein a physical object detected inside the boundary definition and beyond the threshold distance triggers an alarm and a physical object detected inside the boundary definition and within the threshold distance is ignored; generating a plurality of spatial points based on depth measurements of physical objects within the physical space; determining a possible location of a physical object based on the spatial point; determining that the position of the physical object is inside the boundary definition; as well as In response to determining that the position of the physical object is inside the boundary definition and exceeds the threshold distance, an alert is issued to the user.