Dynamic proximity boundary for detecting proximity object

By dynamically adjusting the volume boundaries and using sensor data to detect physical objects, the problem of users ignoring the physical environment in immersive extended real-life experience is solved, and the effect of timely detecting changes in the physical environment in immersive experience is achieved.

CN119065021APending Publication Date: 2024-12-03APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410689872.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-05-21
Filing Date
2024-05-30
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In an immersive extended reality experience, users may ignore the surrounding physical environment due to immersed in virtual content, resulting in unawareness of objects in the physical environment.

Method used

By dynamically generating volume boundaries for detecting close to objects, sensor data is used to determine whether the physical object is within the volume, and if necessary initiate a notification process or modify the presentation of virtual content so that the user is aware of the objects in the physical environment. The volume boundary can be dynamically adjusted according to the speed of the user and the device, ensuring that close physical objects can be identified in time when moving.

Benefits of technology

It effectively improves users' awareness of objects in the physical environment, enhances the security and user experience of immersive experience, and ensures that users can timely detect changes in the surrounding physical environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065021A_ABST
    Figure CN119065021A_ABST
Patent Text Reader

Abstract

The invention discloses a dynamic proximity boundary for detecting a proximity object. Providing visual processing based on proximity to an obstacle includes collecting, by a device, sensor data of an environment, and determining a speed of the device. The boundary of the volume around the device is determined based on the velocity. If a physical object is detected within the boundary based on the sensor data, a notification process is initiated. The boundary of the volume dynamically changes as the speed of the device or user changes.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Many multifunctional electronic devices are capable of generating and presenting extended reality ("XR") content. Typically, these devices utilize an immersive display, such as a heads-up display, through which a user can interact with the XR content. XR content can simulate, in whole or in part, an environment that a person perceives and / or interacts with via an electronic device. However, by the very nature of the immersive experience, the user may be distracted from the surrounding physical environment, which may cause the user to be unaware of objects in the physical environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1A to Figure 1B An exemplary system setup is shown in which the present disclosure may be practiced according to one or more embodiments.

[0003] Figure 2 A technique for implementing a notification process based on boundary intersections according to one or more embodiments is shown in flowchart form.

[0004] Figure 3 A technique for modifying the presentation of virtual content based on boundary intersections in accordance with one or more embodiments is shown in flowchart form.

[0005] Figure 4

[0013] Techniques for dynamically modifying volume boundaries in accordance with one or more embodiments are shown in flow chart form.

[0006] Figure 5 Techniques for performing a notification process according to one or more embodiments are shown in flowchart form.

[0007] Figure 6 An exemplary system diagram of an electronic device according to one or more embodiments is shown.

[0008] Figure 7 A simplified multifunction device according to one or more embodiments is shown in block diagram form. DETAILED DESCRIPTION

[0009] The present disclosure relates to systems, methods, and computer-readable media for dynamically generating a proximity boundary for detecting approaching objects. In general, techniques are disclosed for dynamically determining whether to modify the presentation of virtual content and / or initiate a notification process based on the detection of an approaching object (based on a dynamic proximity boundary).

[0010] According to one or more embodiments, a volume for detecting approaching objects may be defined around a user and / or the user's device. In some embodiments, when a physical object is detected within the volume, the device will execute a response. For example, the device may initiate a notification process to indicate to the user that an approaching object has been detected. Additionally or alternatively, the device may generate or modify a visual processing applied to image data presented by the device to serve as a notification of a detected approaching object.

[0011] According to one or more embodiments, dynamic boundaries can be generated based on the speed of the user and / or the user's device. In some embodiments, the speed of the user's device will be detected, including speed and direction. According to some embodiments, the boundary of the volume around the user will be dynamically modified based on the speed and / or direction of the speed. For example, as the speed of the user increases, the volume around the user and / or the device may also increase. As a result, when the physical object is in the volume, when moving at a faster rhythm, the user will have more time to respond. In some embodiments, the boundary can be changed in a uniform manner. For example, if the volume is spherical, the radius of the ball can increase in all directions as the speed of the user increases. Alternatively, in some embodiments, the volume can be an irregular shape that increases in the direction of the speed. For example, if the user moves forward faster, the boundary in front of the user can increase relative to the user, and the boundary behind the user can change or may not change relative to the user.

[0012] The volume can have a regular shape, such as a sphere, a cylinder, a cube, etc. Alternatively, the volume can have an irregular shape. In some embodiments, the volume can be located around the device and / or the user. For example, the center point of the volume can be based on the device position. Alternatively, the center point of the volume can be based on one or more positions of the user. For example, the position of the user's head or hand can be used. In addition, in some embodiments, the volume can be an area around the user's body, and multiple points can be used, and the volume is generated around the multiple points.

[0013] Sensor data can be collected by the user's device to determine whether a physical object is detected in the environment around the user. In some embodiments, the device can determine the location of the detected physical object based on the sensor data. The location can be compared with the dynamic volume to determine whether the physical object is located within the volume, or has intersected with the volume. If a determination is made that the physical object is detected within the volume boundary, a notification process can be initiated. The notification process can include presenting a notification to the user that the physical object is detected within the boundary. In some embodiments, the notification process can include applying visual processing to the image data presented to the user, so that the visual processing indicates to the user that the physical object is detected within the volume. For example, the visual processing can include modifying the presentation of physical and / or virtual content in the scene, removing virtual content from the scene to expose the physical object, etc.

[0014] In the following description, for the purpose of explanation, a lot of specific details are set forth in order to provide a thorough understanding of the disclosed concept. As part of this specification, some drawings of the present disclosure represent structures and devices in the form of block diagrams to avoid blurring the novel aspects of the disclosed embodiments. In this context, it should be understood that references to numbered drawing elements without associated identifiers (e.g., 100) refer to all instances of drawing elements with identifiers (e.g., 100a and 100b). In addition, as part of this specification, some drawings of the present disclosure can be provided in the form of flow charts. The blocks in any particular flow chart can be presented in a specific order. However, it should be understood that the specific process of any flow chart is only used to illustrate an embodiment. In other embodiments, any various components depicted in the flow chart can be deleted, or components can be executed in different orders, or even simultaneously. In addition, other embodiments may include additional steps that are not shown as a part of the flow chart. The language used in this disclosure is mainly selected for readability and guidance purposes, and it may not be selected to depict or limit the disclosed subject matter. Reference in this disclosure to "one embodiment" or "an embodiment" is meant to include a particular feature, structure, or characteristic described in connection with that embodiment in at least one embodiment, and multiple references to "one embodiment" or "an embodiment" should not be construed as necessarily all referring to the same or different embodiments.

[0015] It will be appreciated that in the development of any actual implementation (as in any development project), many decisions must be made to achieve the developer's specific goals (e.g., to meet system and business-related constraints), and that these goals will vary from one implementation to another. It will also be appreciated that such a development effort may be complex and time-consuming, but will nevertheless be a routine undertaking for those of ordinary skill in the art who benefit from the image capture disclosed herein.

[0016] A physical environment refers to the physical world that people can sense and / or interact with with or without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one law of physics. As an example, an XR system may detect head movement and, in response, adjust the head movement in a manner similar to the way such views and sounds change in the physical environment.

[0017] Graphical content and sound fields presented to a person. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting an XR environment, and in response, adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust characteristics of graphical content in an XR environment in response to a representation of physical movement (e.g., a voice command).

[0018] refer to Figure 1A to Figure 1B , presents an exemplary system arrangement in which the present disclosure may be practiced according to one or more embodiments. In particular, the techniques described herein modify the presentation of virtual content presented to a user of a device when the user is close to a physical object in order to make the user aware of the physical environment. Figure 1A An exemplary electronic device 110 with a display is shown, on which virtual content 115A is presented. For clarity, user 100A refers to a user moving at a first speed at a first location. As will be described below, a user moving at a second speed at a second location will be referred to as 100B. Therefore, reference numeral 100 generally refers to a user. Electronic device 110 may include a device configured to present virtual content to a user. In some embodiments, electronic device 110 may be a wearable device (such as a head mounted device) through which user 100 interacts with a mixed reality environment.

[0019] Virtual content 115A may include augmented reality content, virtual reality content, or any kind of extended reality content. For example, virtual content may be presented while a user is engaged in an immersive experience. Display 125 may be an opaque display, an immersive display (such as an optical see-through display), a pass-through display, or the like. When a user is within a physical environment, the user may use electronic device 110 to view virtual content 115A. The physical environment may have physical objects, such as physical object 105. For example, physical objects in the environment may include static objects, such as furniture, walls, doors, fixed furniture, and the like. Physical objects may also include dynamic objects, such as pets, animals, people, or other objects that may move within the environment.

[0020] According to one or more embodiments, the device 110 can track approaching objects in the physical environment. In some embodiments, the device 110 determines the volume boundary 120A to determine whether the physical object in the environment is a close object. In particular, the electronic device 110 may include one or more sensors that can detect physical objects in the environment. The device may include sensors, such as depth cameras, LIDAR, etc., which can be used to detect physical objects and determine the corresponding positions of the physical objects. If a portion of the physical object falls within the volume boundary 120A, the physical object can be determined to be a close object.

[0021] According to one or more embodiments, the device 110 continuously modifies the volume boundary 120A based on the speed of the user 100A. In some embodiments, the faster the user 100A moves, the larger the bounding volume (thereby identifying farther objects as closer objects). For example, if the user or user device is stationary, a default bounding volume can be used. The volume boundary 120A can be centered on the user 100A or a portion of the user, or on the user device 110. For example, the device location can be used to approximate the user location, and the volume boundary 120A can be generated around the point in 3D space that is determined to be the device location. Figure 1A In the example embodiment, the user 100A does not move, or moves at a sufficiently slow pace so that the volume boundary 120A does not surround or intersect the physical object 105. Therefore, the virtual content 115A is presented on the display 125 without interruption.

[0022] Instead, turn Figure 1B, user 100B is moving at a faster pace and is moving toward physical object 105. In response to the increase in speed, volume boundary 120B also increases. As a result, physical object 105 now intersects with volume boundary 120B. In response, device 110 can implement a notification process, such as notifying the user of an approaching object, or modifying the presentation of virtual content. Here, device 110 provides a notification process by removing at least a portion of the virtual content from the display so that the physical object 105 in the physical environment is visible as displayed object 115B. This can occur, for example, by presenting direct camera data on display 125 so that the user can see physical object 105. As described above, volume boundary 120B can be an irregular shape. Here, volume boundary 120B is an elongated shape. However, it should be understood that in some embodiments, the volume boundary can maintain a regular shape, such as a spherical shape. In addition, in some embodiments, the original volume boundary 120A can be an irregular shape.

[0023] Although not shown, if the user stops moving, the volume boundary 120B can shrink in response to the reduced speed. As a result, the physical object 105 may no longer exist within the volume boundary 120. Therefore, the notification process can be stopped. In the example shown, if the notification process is stopped, the virtual content 115A can be presented on the display 125 so that the physical object 105 is no longer visible, or is less visible than during the notification process. In some embodiments, the shrinking boundary can be large enough so that when the user stops, the physical object 105 remains within the boundary. Therefore, when the physical object 105 is still within the shrinking volume boundary, the notification process can remain effective when the user stops moving.

[0024] According to one or more embodiments, the notification process may include applying visual processing to content displayed on the display 125, such as animation, color change, or other modifications to the virtual object. In addition, in some embodiments, the visual processing may include changing the presentation to the user, such as causing a breakout display, so that the physical object or the characteristics of the physical object are visible to the user. According to some embodiments, various visual processing may be applied, and specific visual processing may be selected based on the proximity of the device or the user of the device to the physical object. For example, multiple volume boundaries may be determined around the user and / or user device, and may be associated with different visual processing. As another example, when the user approaches the physical object, the visual processing may be applied dynamically. For example, a specific animation may be applied to the content presented on the display, such as a wave movement across the displayed content, the movement of the displayed content, etc. As another example, color processing may be applied to the displayed content, such as applying a color overlay to the displayed content or a portion of the displayed content, to indicate to the user that the user is approaching the physical object.

[0025] Figure 2 A technique for implementing a notification process based on a boundary intersection according to one or more embodiments is shown in a flow chart form. For the purpose of explanation, the following steps will be described in the context of FIG. 1. However, it should be understood that various actions may be performed by alternative components. In addition, various actions may be performed in different orders. In addition, some actions may be performed simultaneously, and some actions may not be required, or other actions may be added.

[0026] Flowchart 200 begins at box 205, where the electronic device collects sensor data for the environment. In particular, the device may include one or more sensors that can determine the characteristics of the device surrounding the physical environment. For example, the device may include sensors such as cameras, depth sensors, LIDAR, or other technologies that provide sensor data of objects in the physical environment surrounding the device. Therefore, the sensor data may include image data, depth data, and other data corresponding to the characteristics of the physical environment. In some embodiments, the electronic device may transmit a signal into the environment to detect whether there are any objects in the environment, or may receive signal data and / or an indication of the presence of physical objects in the environment from an additional device.

[0027] Flowchart 200 continues to block 210, where speed is determined. In some embodiments, speed can be determined relative to the device and / or the user. For example, the electronic device may include sensors that can be used to track the speed of the device in a physical environment, such as an accelerometer, a gyroscope, etc. In addition, in some embodiments, the device can be configured to use additional or alternative techniques for determining speed. Examples include vision-based techniques such as simultaneous localization and mapping ("SLAM"), scale-invariant feature transform ("SIFT"), etc. According to one or more embodiments, speed includes rate and / or direction.

[0028] At box 215, the volume boundary is determined based on speed. In some embodiments, volume can be defined as a spatial area with a predetermined relationship with an electronic device and / or a user, so that the volume moves through the physical environment when the user / device moves. For example, the volume can be in a configuration relative to the device, such as a specific distance and orientation relative to the device to depict the volume boundary. Therefore, the position of the device and / or the user in the physical environment is used as an origin or area, from which the volume around the device can be determined. According to one or more embodiments, the volume boundary can be a regular or irregular shape. For example, the volume can be included in a plane in front of the electronic device and / or the user, in an arc around the electronic device and / or the user, in a spherical or cylindrical boundary around the electronic device and / or the user.

[0029] The volume boundary can be dynamically determined, for example, on a per-frame basis corresponding to image frames collected by the device from which sensor data of the environment is collected. The volume boundary is dynamically modified based on the speed of the device. For example, as the speed increases, at least in a direction consistent with the direction of the speed, the bounding volume can increase around the device and / or user. Similarly, as the speed decreases, the bounding volume around the device and / or user can decrease.

[0030] According to one or more embodiments, a volume boundary can be determined between a minimum volume and a maximum volume around a user. For example, if the device and / or user is not moving, the minimum volume can be used. In some embodiments, the minimum volume can include not using the volume around the device and / or user. In some embodiments, for example, when the device and / or user moves quickly, the maximum volume can also be used. The volume boundary can be based on a determined distance from the user, which is determined based on the rate and / or direction of the user's speed.

[0031] Flowchart 200 continues to box 220, where a determination is made as to where a foreign physical object is detected within the boundary. That is, a determination is made as to whether a physical object other than the user and / or user device (or peripheral device) is detected to have intersected the bounding volume. The physical object may be a static object, such as a wall, furniture, appliance, plant, etc. Additionally or alternatively, the physical object may include dynamic objects, such as other people, pets, animals, and other moving objects. Detecting an object may include detecting the presence of any general object, or may include identifying a specific object. In some embodiments, depth information may be used to determine the presence of one or more objects in the environment.

[0032] If no foreign object is detected within the volume boundary at block 220, the flowchart 200 returns to block 205 and continues to collect sensor data from which the velocity can be determined, thereby continuously modifying the volume boundary accordingly. Similarly, the sensor data can be monitored to determine whether a foreign object is detected within the volume boundary.

[0033] Returning to box 220, if a determination is made that a foreign object is detected within the volume boundary, flowchart 200 proceeds to box 230, where a notification process is performed. As will be described in more detail below, the notification process may include displaying or otherwise presenting a notification to the user with an indication that a foreign object is detected within the boundary. For example, this may include a text notification, an audio notification, a visual notification, a tactile notification, etc. Additionally or alternatively, the notification process may include applying visual processing to the image data presented on the device. For example, in some embodiments, virtual content may be removed from the image data presented by the device in order to improve or increase the visibility of the physical object. In other examples, the visual processing may include animation, color changes, opacity changes, brightness, etc.

[0034] Flowchart 200 proceeds to frame 235, where the speed of the environment continues to be monitored. This includes receiving additional sensor data of the environment while presenting the notification process. The sensor data includes data related to the presence of foreign objects in the detection environment and dynamically determining the speed of the user and / or device. For example, the process described above with respect to frame 205, frame 210, and frame 215 can occur, for example, on a per-frame basis while continuing to monitor the speed and environment at frame 325. Then, at frame 240, a determination is made as to whether the foreign object remains within the volume boundary. According to some embodiments, a determination can be made based on the sensor data on a frame-by-frame basis or at some other interval. For example, at each frame, a determination can be made as to whether a foreign object exists for a specific volume determined for the frame. If a determination is made that the foreign object remains within the volume boundary (or the foreign object is within the boundary of a specific frame, regardless of whether it is the same foreign object), the flowchart returns to frame 230, and the notification process is continued.

[0035] Returning to block 240, if a determination is made that no foreign objects are detected within the volume boundary for a particular frame or a particular interval, the flowchart 200 proceeds to block 245 and the notification process is stopped. For example, stopping the notification process may include removing the visual processing, stopping the visual or audio presentation of the notification, restoring and expected opacity, brightness, coloring, animation, etc. The flowchart 200 then proceeds to block 205, and the device continues to monitor the speed and objects of the environment as the device and / or user moves within the physical environment.

[0036] Figure 3 A technique for implementing a notification process based on boundary intersections according to one or more embodiments is shown in flowchart form, the notification process including modification of virtual content. For purposes of explanation, FIG. Figure 2The following steps are described in the context of . However, it should be understood that various actions can be performed by alternative components. In addition, various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions may be added.

[0037] Flowchart 300 begins at box 305, where virtual content is presented on a device. For example, the device may include a wearable device, such as a head-mounted device. The device can be used to interact with virtual content and immersive themes. For example, the head-mounted device may include a display located in front of the user's eyes, and the virtual content is presented on the display. In some embodiments, the virtual content can be presented with a view of the physical environment, or instead of a view of the physical environment. For example, the virtual content may include virtual reality content that replaces the view of the physical environment. As another example, the virtual content may include augmented reality content, in which the virtual content is presented in a view of the physical environment. For example, the device may include pass-through content of the physical environment displayed on the display together with the virtual content.

[0038] Flowchart 300 also includes determining the current speed of the device and / or user, as shown at block 210. Figure 2 As described above, according to one or more embodiments, speed is determined based on sensor data captured by the device and includes rate and / or direction. Figure 2 As described, flowchart 300 also includes determining a volume boundary based on the speed at box 215, and determining whether a foreign object is detected within the volume boundary at box 220. If a determination is made at box 220 that there is no foreign object within the volume boundary, the flowchart proceeds to box 225, where the device continues to monitor the speed and environment in the area to dynamically determine whether a foreign object is detected. Therefore, flowchart 300 returns to box 210, and continuously collects sensor data from which the speed can be determined, thereby continuously modifying the volume boundary accordingly. Similarly, the sensor data can be monitored to determine whether a foreign object is detected within the volume boundary.

[0039] Returning to block 220, if a determination is made that a foreign object is detected within the volume boundary, flowchart 300 proceeds to block 330, where the virtual content is removed from the display. In some embodiments, visual processing can be applied to the virtual content presented at block 305. For example, the virtual content can be removed from the image data presented by the device in order to improve or increase the visibility of the physical environment (including at least a portion of the object that has intersected the volume boundary). In other examples, the visual processing can include animation, color changes, opacity changes, brightness changes, etc. For example, the opacity of the virtual content can be reduced to indicate that a foreign object has been detected.

[0040] Flowchart 300 proceeds to block 332 where a pass-through display is presented. That is, visual processing applied to the virtual content (such as reducing the opacity of the virtual content or otherwise removing image data associated with the virtual content from the frame presented to the user) increases visibility of the pass-through camera data at least in the area where the foreign object is detected.

[0041] At box 235, the velocity of the environment continues to be monitored. This includes receiving additional sensor data for the environment while applying visual processing to enhance the visibility of the physical environment (particularly the detected foreign objects). The sensor data includes data related to detecting the presence of foreign objects in the environment and dynamically determining the velocity of the user and / or device. Then, at box 240, a determination is made as to whether the foreign object remains within the volume boundary. If a determination is made that the foreign object remains within the volume boundary (or the foreign object is within the boundary of a particular frame, regardless of whether it is the same foreign object), the flowchart returns to box 330 and continues to remove virtual content.

[0042] Returning to block 240, if a determination is made that no foreign objects are detected within the volume boundaries for a particular frame or a particular interval, the flowchart 300 proceeds to block 345 and the virtual content is displayed again as expected. For example, a particular application running on the device may direct the virtual content to be presented in a particular manner based on regular use. The removal of the virtual content may be an application-level or system-level override that causes the expected presentation of the virtual content to be modified. When foreign objects are no longer detected within the current bounding volume, the virtual content may be displayed as the source application of the expected content, thereby returning to the expected display position, opacity, etc. The flowchart 300 then proceeds to block 225 and the device continues to monitor the velocity and objects of the environment as the device and / or user moves within the physical environment.

[0043] According to one or more embodiments, the speed described above (e.g., at block 210) can be determined by sensor data on a head mounted device (such as an IMU). In some embodiments, the speed of different parts of the user can be used to determine whether the user is likely to collide with objects in the environment, such as the user swinging an arm or performing a gesture. Thus, in some embodiments, determining the tracking speed of the volume boundary can be based on body tracking data. Figure 4 A flow chart of a technique for dynamically determining a bounding volume based on body tracking data according to one or more embodiments is depicted. In particular, Figure 4 Depicted Figure 2 For the purpose of explanation, FIG. 1 to FIG. Figure 2The following steps are described in the context of . However, it should be understood that various actions can be performed by alternative components. In addition, various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions may be added.

[0044] Flowchart 400 begins at box 210, where the current speed of the user and / or device is determined. For the purposes of this example, the current speed is determined based on the speed of the user of the device. For example, the volume boundary can be bound to a portion of the user, such as the user's hand, arm, or other body part. In this example, flowchart 400 includes receiving body tracking data at box 412. According to one or more embodiments, the electronic device may include one or more sensors configured to detect the movement of the body. For example, this may include cameras and other sensors that are directed downward or away from the user from the head-mounted device so that the user's body movement can be tracked. Body tracking data can be used to determine characteristics about the user's movement. For example, this may include posture, position, speed, rotation, etc. In some embodiments, registration information can be used to determine characteristics about the user, which can achieve body tracking.

[0045] Flowchart 400 continues to box 414, where the speed of at least a portion of the body is determined. In some embodiments, the portion of the body for which the speed is determined is based on the portion of the body bound to the volume boundary. For example, the joints of the hands, arms, legs, etc. can be used as the origin position, around which the volume boundary is generated. Therefore, the speed of the origin can be determined based on the body tracking data received at box 412. For example, the speed includes the rate and direction of the tracked portion of the user's body associated with the volume boundary. In some embodiments, the speed can include the angular velocity when the movement of the tracked body part is moving in the rotational direction.

[0046] At block 215, a volume boundary is determined based on the velocity of the portion of the body as determined at block 414. In some embodiments, a volume may be defined as a region of space having a predetermined relationship to the electronic device and / or user such that the volume moves through the physical environment as the user / device moves.

[0047] At block 416, determining the volume boundary includes determining an origin of the volume. According to one or more embodiments, the origin may be a tracked portion of a body for which velocity is determined. For example, the origin may be a particular joint of the user's body, such as a joint position of an arm, leg, or the like. Note that in some embodiments, the portion of the body used to determine velocity may be different from the origin around which the volume boundary is determined. As an example, the movement of a hand, arm, or leg may be tracked to determine a volume boundary around an origin at the user's head or torso.

[0048] Flowchart 400 ends at box 418, where a volume boundary around a portion of the body is determined based on the velocity of the portion of the body. That is, the boundary of the volume is generated to be positioned at a distance away from an origin (or associated body part) determined based on the determined velocity. As described above, the volume boundary can be dynamically determined, for example, on the basis of each frame corresponding to an image frame collected by a device (sensor data of the environment is collected from the device). The volume boundary is dynamically modified based on the velocity of the portion of the body used as the origin of the volume boundary. For example, as the velocity increases, the boundary volume can increase around the portion of the user's body at least in a direction consistent with the direction of the velocity. Similarly, as the velocity decreases, the boundary volume around the device and / or the user can decrease.

[0049] Figure 5 A flow chart of a technique for performing a notification process based on characteristics of an intersection of a foreign object with a volume according to one or more embodiments is depicted. For purposes of explanation, FIG. Figure 2 The following steps are described in the context of . However, it should be understood that various actions can be performed by alternative components. In addition, various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions may be added.

[0050] Flowchart 400 depicts the process described above with respect to Figure 2 The notification process may include modifying the user experience in the immersive environment based on the detected collision between the foreign object and the bounding volume so that the user is notified of the presence of the foreign object within the bounding object.

[0051] Flowchart 500 begins at block 532, where characteristics of the intersection of a foreign object and a volume boundary are determined. For example, the characteristics of the intersection may include the amount of the volume in which the foreign object was detected, the location within the volume at which the foreign object was detected, etc. For example, the characteristics may be determined based on sensor data received by a user's device that detected the presence of the object. In some embodiments, the particular sensor that captured the data by which the object was detected may indicate the volume boundary at which the foreign object is located.

[0052] Some examples of characteristics of the intersection include the detected proximity of the foreign object to the user, the detected size of the foreign object, the relative position of the foreign object to the user (such as in front of the user or behind the user), etc.

[0053] Flowchart 500 proceeds to block 534, where the notification type is selected based on the characteristics of the intersection. For example, if the characteristics of the intersection indicate that the intersection is not within the user's field of view (e.g., if the intersection is behind the user), the notification type may include an audio notification, while if the intersection is within the user's field of view, a visual indication may be used. As another example, an intersection of objects at a boundary may indicate a less intrusive notification than a foreign object detected closer to the user. For example, when a foreign object is detected at a volume boundary, a moderate visual processing may be performed, such as animation, coloring, opacity, brightness changes of the virtual content, or other visual processing may be applied to the virtual content. Conversely, when a foreign object is detected closer to the user, a more explicit notification may be provided, such as a visual indication, removal of virtual content, etc. Similarly, the type of notification may be associated with a pause period, wherein after a certain amount of time, number of frames, etc., if a foreign object continues to be detected within the volume, the notification process may be stopped. In some embodiments, different notification types may be associated with different pause periods. In some embodiments, other types of notifications may be used, such as tactile feedback, etc.

[0054] Flowchart 500 ends at block 536, where the notification process is initiated based on the selected notification type. The notification process can be performed based on the selected notification type (such as the selected notification characteristics). In some embodiments, the type of notification can also be modified on a per-frame or other periodic basis. Thus, while the same foreign object is detected within the volume, the characteristics of the notification can change. In addition, the notification process can be stopped based on any relevant pause period.

[0055] See also Figure 6 , which depicts a simplified block diagram of an electronic device 600 according to one or more embodiments of the present disclosure. The electronic device 600 may be part of a multi-function device such as a mobile phone, a tablet computer, a personal digital assistant, a portable music / video player, a wearable device, or any other electronic device that includes a camera system. Figure 6 An overall view of a system diagram capable of supporting proximity detection and notification according to one or more embodiments is shown in block diagram form. The electronic device 600 can be connected to other network devices such as mobile devices, tablet devices, desktop devices, and network storage devices (such as servers, etc.) through a network via a network interface 650. In some embodiments, the electronic device 600 can be communicatively connected to other electronic devices via a local network to share sensor data and other information about a shared physical environment.

[0056] The electronic device 600 may include a processor 610, such as a central processing unit (CPU). The processor 610 may include a system on a chip such as those present in a mobile device, and may include one or more dedicated graphics processing units (GPUs). In addition, the processor 610 may include multiple processors of the same or different types. The electronic device 600 may also include one or more memories 620. Each of the memories 620 may include one or more different types of memories, which may be used to perform device functions in conjunction with the processor 610. For example, the memory 620 may include a cache, a ROM, and / or a RAM. The memory 620 may store various programming modules in the form of computer-readable code included on a non-transitory computing readable medium during execution, including an XR module 622, a tracking module 624, and various other applications 628. According to some embodiments, the XR module 622 may provide an immersive experience to the user, for example, through augmented reality, virtual reality, extended reality, augmented reality, etc. The tracking module 624 may utilize data from a camera 640 and / or a sensor 655 (such as a proximity sensor) to determine the location of the electronic device 600, the user of the electronic device 600, and / or other objects in the physical environment.

[0057] The electronic device 600 may also include a storage device 630. The storage device 630 may include one or more different storage devices that can be used to store data used by the programming module shown in the memory 620. For example, the storage device 630 may include one or more non-transitory computer-readable storage media, which include, for example, disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and digital video discs (DVDs)), and semiconductor storage devices (such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs)). The storage device 630 may include VR content 634, which may include augmented reality content, virtual reality content, or any kind of extended reality content. In some embodiments, the storage device 630 may also include a user profile memory 638. The user profile memory can be used to store data related to the user profile for the purpose of detecting approaching objects, such as the user's gait or average speed, which may affect when the user should be prompted about the approaching object. In some embodiments, the user profile memory 638 may include user-specified data, such as a threshold distance at which a notification should be provided.

[0058] The electronic device 600 may also include one or more cameras 640. Each of the cameras 640 may include an image sensor, a lens stack, and other components that can be used to capture images. In one or more embodiments, the camera may be configured to face different directions from the electronic device. For example, the front camera may be positioned in or on the first surface of the electronic device 600, and the rear camera may be positioned in or on the second surface of the electronic device 600. In some embodiments, the camera 640 may include one or more types of cameras, such as an RGB camera, a depth camera, and the like. The electronic device 600 may include one or more sensors 655 that can be used to detect physical objects in the environment. Examples of sensors 655 include LIDAR, and the like.

[0059] In one or more embodiments, the electronic device 600 may also include a display 660. The display 660 may be any kind of display device, such as an LCD (liquid crystal display), an LED (light emitting diode) display, an OLED (organic light emitting diode) display, and the like. In addition, the display 660 may be a translucent display, such as a head-up display, a pass-through display, and the like. The display 660 may present content associated with the XR module 622 or other applications 628. In addition, the display 660 may be configured to present image data of a physical environment, such as image data from a pass-through camera, and the like.

[0060] Although electronic device 600 is depicted as including the above-mentioned multiple components, in one or more embodiments, various components may be distributed over multiple devices. In addition, additional components may be used, and some combination of the functionality of any component may be combined.

[0061] Reference now Figure 7 , which shows a simplified functional block diagram of an exemplary multifunction electronic device 700 according to one embodiment. The multifunction electronic device 700 may include a processor 705, a display 710, a user interface 715, graphics hardware 720, sensors 725 (e.g., proximity sensor / ambient light sensor, accelerometer and / or gyroscope), a microphone 730, one or more audio codecs 735, one or more speakers 740, communication circuitry 745, digital image capture circuitry 750 (e.g., including a camera system), one or more video codecs 755 (e.g., supporting a digital image capture unit), a memory 760, a storage device 765, and a communication bus 770. For example, the multifunction electronic device 700 may be a digital camera or a personal electronic device such as a personal media player, a mobile phone, a head-mounted device, or a tablet computer.

[0062] The processor 705 may execute instructions necessary for implementing or controlling the operation of many functions performed by the device 700 (e.g., generation and / or processing of images disclosed herein). The processor 705 may, for example, drive the display 710 and may receive user input from the user interface 715. The user interface 715 may allow a user to interact with the device 700. For example, the user interface 715 may take on a variety of forms, such as buttons, keypads, dials, click wheels, keyboards, display screens, and / or touch screens. The processor 705 may also, for example, be a system on a chip, such as a system on a chip present in a mobile device, and include a dedicated GPU. The processor 705 may be based on a reduced instruction set computer (RISC) or a complex instruction set computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. The graphics hardware 720 may be dedicated computing hardware for processing graphics and / or assisting the processor 705 in processing graphics information. In one embodiment, the graphics hardware 720 may include a programmable GPU.

[0063] Image capture circuit 750 may include two (or more) lens assemblies 780A and 780B, each of which may have a separate focal length. For example, lens assembly 780A may have a short focal length relative to the focal length of lens assembly 780B. Each lens assembly has a separate associated sensor element, such as sensor element 790A and sensor element 790B shown. Alternatively, two or more lens assemblies may share a common sensor element. Image capture circuit 750 may capture still images and / or video images. The output from image capture circuit 750 may be processed at least in part by the following devices: one or more video codecs 755, and / or processor 705 and / or graphics hardware 720, and / or a dedicated image processing unit or pipeline incorporated within circuit 750. The images thus captured may be stored in memory 760 and / or storage device 765.

[0064] The sensor and camera circuit 750 may capture still images and video images that may be processed in accordance with the present disclosure at least in part by one or more video codecs 755, and / or the processor 705 and / or the graphics hardware 720, and / or a dedicated image processing unit incorporated within the circuit 750. The images thus captured may be stored in the memory 760 and / or the storage device 765. The memory 760 may include one or more different types of media used by the processor 705 and the graphics hardware 720 to perform device functions. For example, the memory 760 may include a memory cache, a read-only memory (ROM), and / or a random access memory (RAM). The storage device 765 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage device 765 may include one or more non-transitory computer-readable storage media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and digital video disks (DVDs)), and semiconductor storage devices (such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs). The memory 760 and the storage device 765 may be used to tangibly hold computer program instructions or codes organized into one or more modules and written in any desired computer programming language. When executed by, for example, the processor 705, such computer program code may implement one or more of the methods described herein.

[0065] There are many different types of electronic systems that enable a person to sense and / or interact with various XR environments. Examples include: head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers. Head-mounted systems can

[0066] With an integrated opaque display and one or more speakers. Alternatively, the head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). The head-mounted system can incorporate one or more imaging sensors for capturing images or videos of the physical environment, and / or one or more microphones for capturing audio of the physical environment. The head-mounted system can have a transparent or translucent display instead of an opaque display. The transparent or translucent display can have a medium through which light representing the image is guided to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, silicon-based liquid crystal, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some specific implementations, the transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project virtual objects into a physical environment, for example as a hologram or on a physical surface.

[0067] The scope of the subject matter disclosed in this disclosure should be determined with reference to the appended claims, along with the full scope of equivalents to which the claims are entitled.In the appended claims, the terms "including" and "in which" are used as the plain-English equivalents of the respective terms "comprising" and "wherein."

Claims

1. A method, comprising: A first set of sensor data of the environment is collected by the device; determining a speed of the device; determining a volume boundary of a volume around the device based on the velocity; detecting physical objects in the environment based on the first set of sensor data; as well as Based on determining that the physical object intersects the volume boundary, a notification process is initiated.

2. The method according to claim 1, wherein: The device is configured to present virtual content on a display, and wherein the notification process includes removing the virtual content from the display.

3. The method according to claim 1, wherein: Determining the volume boundary includes: An origin of the volume is determined based on the device.

4. The method according to claim 3, wherein: The origin is determined based on body tracking data.

5. The method according to claim 4, wherein: The volume boundary is determined based on the body tracking data.

6. The method according to any one of claims 1 to 5, wherein: The notification process includes performing at least one selected from the group consisting of an audio notification, a visual notification, and a tactile notification.

7. The method according to claim 6, wherein: Initiating the notification process includes: determining a direction of the velocity; and Select the notification type based on the direction.

8. A computer program product comprising computer readable code executable to: A first set of sensor data of the environment is collected by the device; determining a speed of the device; determining a volume boundary of a volume around the device based on the velocity; detecting physical objects in the environment based on the first set of sensor data; as well as Based on determining that the physical object intersects the volume boundary, a notification process is initiated.

9. The computer program product according to claim 8, wherein: The device is configured to present virtual content on a display, and wherein the notification process includes removing the virtual content from the display.

10. The computer program product of claim 8, wherein: Determining the volume boundary includes: An origin of the volume is determined based on the device.

11. The computer program product according to claim 10, wherein: The origin is determined based on body tracking data.

12. The computer program product according to any one of claims 8 to 11, wherein: The notification process includes performing at least one selected from the group consisting of an audio notification, a visual notification, and a tactile notification.

13. The computer program product according to any one of claims 8 to 11, wherein: Detecting a physical object in the environment based on the first set of sensor data includes computer readable code executable to: Depth information indicative of objects in the environment is obtained.

14. The computer program product according to any one of claims 8 to 11, further comprising computer readable code for performing the following functions: detecting a change in the speed to a second speed less than the speed; and The volume boundary is modified based on the second speed.

15. The computer program product of claim 14, further comprising computer readable code for performing the following functions: The notification process is stopped based on determining that the physical object is not within the modified volume boundary.

16. A system, comprising: one or more processors; as well as One or more computer readable media, the one or more computer readable media comprising computer readable code, the computer readable code being executable by the one or more processors to perform the following operations: Collect the first set of sensor data for the environment; determining a speed of the device; determining a volume boundary of a volume surrounding a system device based on the velocity; detecting physical objects in the environment based on the first set of sensor data; as well as Based on determining that the physical object intersects the volume boundary, a notification process is initiated.

17. The system according to claim 16, further comprising: monitor; as well as Computer readable code for: Virtual content is presented on a display, wherein the notification process includes removing the virtual content from the display.

18. A system according to any one of claims 16 to 17, wherein: The computer readable code for detecting a physical object in the environment based on the first set of sensor data includes computer readable code for: Depth information indicative of objects in the environment is obtained.

19. The system of any one of claims 16 to 17, further comprising computer readable code for: detecting a change in the speed to a second speed less than the speed; and The volume boundary is modified based on the second speed.

20. The system of claim 19, further comprising computer readable code for: The notification process is stopped based on determining that the physical object is not within the modified volume boundary.