Presenting an environment based on user movement

Abstract

The present disclosure relates to presenting an environment based on user movement. In an example process, a computer-generated reality environment including a virtual object is presented, and a user movement occurring in a physical environment is detected. In response to determining that the detected user movement is toward the virtual object and the virtual object occludes a real object in the physical environment, it is determined whether the detected user movement is directed to the virtual object or the real object. In accordance with a determination that the detected user movement is directed to the real object, a visual appearance of the virtual object is modified, wherein modifying the visual appearance of the virtual object includes displaying at least a portion of a presentation of the real object. In accordance with a determination that the detected user movement is directed to the virtual object, the presentation of the virtual object is maintained to occlude the real object.

Description

BACKGROUND 1. TECHNICAL FIELD

[0002] The present disclosure relates generally to computer-generated reality systems, and more particularly to techniques for providing computer-generated reality environments.

[0003] 2. Description of Related Art

[0004] As electronic devices increase in functionality, and improve in their ability to output high quality visual displays, applications become more immersive. One such example is the increasing mainstream demand for computer-generated reality applications. SUMMARY

[0005] The techniques described herein can be used to provide computer-generated reality environments and facilitate user interaction with computer-generated reality environments. Such techniques can optionally supplement or replace other methods for providing computer-generated reality environments. Such techniques can improve the user experience, and enable computer-generated reality interfaces (e.g., 3D interfaces) with advanced functionality.

[0006] In some embodiments, a computer-generated reality environment including a virtual object is presented (e.g., via a display device), and user movement occurring in a physical environment is detected (e.g., via one or more sensors). In response to determining that the detected user movement is toward the virtual object and the virtual object occludes a real object in the physical environment, it is determined whether the detected user movement is directed to the virtual object or the real object. In accordance with a determination that the detected user movement is directed to the real object, a visual appearance of the virtual object is modified, wherein modifying the visual appearance of the virtual object includes displaying a presentation of at least a portion of the real object. In accordance with a determination that the detected user movement is directed to the virtual object, the presentation of the virtual object is maintained to occlude the real object.

[0007] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0008] In some embodiments, an electronic device includes a display device, one or more sensors, one or more processors, and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: presenting, via the display device, a computer-generated reality environment that includes a virtual object; detecting, via the one or more sensors, a user movement occurring in a physical environment; and in response to determining that the detected user movement is toward the virtual object and the virtual object occludes a real object in the physical environment: determining whether the detected user movement is directed to the virtual object or the real object; in accordance with a determination that the detected user movement is directed to the real object, modifying a visual appearance of the virtual object, wherein modifying the visual appearance of the virtual object includes displaying a presentation of at least a portion of the real object; and in accordance with a determination that the detected user movement is directed to the virtual object, maintaining the presentation of the virtual object to occlude the real object.

[0009] In some embodiments, an electronic device includes: means for presenting a computer-generated reality environment that includes a virtual object; means for detecting a user movement occurring in a physical environment; and means for, in response to determining that the detected user movement is toward the virtual object and the virtual object occludes a real object in the physical environment: determining whether the detected user movement is directed to the virtual object or the real object; in accordance with a determination that the detected user movement is directed to the real object, modifying a visual appearance of the virtual object, wherein modifying the visual appearance of the virtual object includes displaying a presentation of at least a portion of the real object; and in accordance with a determination that the detected user movement is directed to the virtual object, maintaining the presentation of the virtual object to occlude the real object. BRIEF DESCRIPTION OF DRAWINGS

[0010] For a better understanding of the various described embodiments, reference should be made to the Detailed Description below taken in connection with the following drawings in which like reference numerals refer to corresponding portions of the disclosure.

[0011] Figures 1A-1B An example system used in various computer-generated reality technologies is depicted.

[0012] Figure 2 An example physical environment is depicted.

[0013] Figure 3 An example computer-generated reality environment, in accordance with some embodiments, is depicted.

[0014] Figure 4 An example computer-generated reality environment, in accordance with some embodiments, is depicted.

[0015] Figure 5An example computer-generated reality environment is depicted in accordance with some embodiments.

[0016] Figure 6 An example computer-generated reality environment is depicted in accordance with some embodiments.

[0017] Figure 7 An example computer-generated reality environment is depicted in accordance with some embodiments.

[0018] Figure 8 An example computer-generated reality environment is depicted in accordance with some embodiments.

[0019] Figure 9 An example computer-generated reality environment is depicted in accordance with some embodiments.

[0020] Figure 10 An example computer-generated reality environment is depicted in accordance with some embodiments.

[0021] Figure 11 A flow diagram of an example process for providing a computer-generated reality environment is depicted in accordance with some embodiments. DETAILED DESCRIPTION

[0022] The following description sets forth exemplary methods, parameters, and the like. Such description is not intended to limit the scope of this disclosure but is provided as a description of exemplary embodiments.

[0023] Various embodiments of electronic systems and techniques are described for using such systems in connection with various computer-generated reality technologies.

[0024] A physical environment (or real environment) refers to a physical world that people are able to sense and / or interact with without aid of electronic systems. A physical environment such as a physical park includes physical articles (or physical objects or real objects) such as physical trees, physical buildings, and physical people. People are able to directly sense and / or interact with a physical environment such as through sight, touch, hearing, taste, and smell.

[0025] In contrast, a computer-generated reality (CGR) environment refers to a fully or partially simulated environment that people experience and / or interact with via an electronic system. In CGR, a subset of a person’s physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner that comports with at least one physical law. For example, a CGR system can detect a person’s head turning, and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to characteristics of virtual objects in a CGR environment can be made in response to representations of physical motions (e.g., voice commands).

[0026] People can sense and / or interact with CGR objects using any of their senses, including sight, hearing, touch, taste, and smell. For example, a person can sense and / or interact with audio objects that create a 3D or spatial audio environment that provides a perception of point audio sources in 3D space. As another example, audio objects can enable audio transparency that selectively incorporates ambient sounds from a physical environment with or without computer-generated audio. In certain CGR environments, a person can sense and / or only interact with audio objects.

[0027] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment (virtual environment) refers to a simulated environment designed to be completely based on computer-generated sensory inputs for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of a person’s presence within the computer-generated environment, and / or through a simulation of a subset of a person’s physical motions within the computer-generated environment.

[0028] In contrast to VR environments, which are designed to be completely based on computer-generated sensory inputs, mixed reality (MR) environments refer to simulated environments that are designed to incorporate sensory inputs or representations thereof from a physical environment in addition to computer-generated sensory inputs (e.g., virtual objects). On a virtual continuum, MR environments are anywhere between fully physical environments on one end and VR environments on the other end, excluding the two extremes.

[0029] In some MR environments, computer-generated sensory inputs can respond to changes in sensory inputs from the physical environment. Additionally, some electronic systems for presenting MR environments can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical articles from the physical environment or representations thereof). For example, a system can cause motion such that a virtual tree appears stationary relative to a physical ground.

[0030] Examples of MR include augmented reality and augmented virtuality. An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed over a physical environment, or a representation thereof. For example, an electronic system for presenting an AR environment can have a transparent or translucent display through which a person can directly view a physical environment. The system can be configured to present virtual objects on the transparent or translucent display so that a person, using the system, perceives the virtual objects superimposed over the physical environment. Alternatively, a system can have an opaque display and one or more imaging sensors that capture images or video of the physical environment, which are representations of the physical environment. The system combines the images or video with virtual objects and presents the combination on the opaque display. A person, using the system, views the physical environment indirectly via the images or video of the physical environment and perceives the virtual objects superimposed over the physical environment. As used herein, video of a physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system captures images of the physical environment using one or more image sensors and uses those images in presenting the AR environment on the opaque display. Further alternatively, a system can have a projection system that projects virtual objects into the physical environment, e.g., as a hologram or on a physical surface, so that a person, using the system, perceives the virtual objects superimposed over the physical environment.

[0031] An AR environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, a system can transform one or more sensor images to impose a selected perspective (e.g., viewpoint) that is different from the perspective captured by the imaging sensors. As another example, a representation of a physical environment can be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions can be representative but not true versions of the originally captured images. As another example, a representation of a physical environment can be transformed by graphically eliminating or blurring portions thereof.

[0032] An augmented virtual (AV) environment refers to a simulated environment in which virtual or computer-generated environments combine with sensory inputs from a physical environment. The sensory inputs can be representations of one or more characteristics of the physical environment. For example, an AV park can have virtual trees and virtual buildings, but a person’s face is realistically reproduced from images taken of a physical person. As another example, a virtual object can take on the shape or color of a physical item imaged by one or more imaging sensors. As another example, a virtual object can take on a shadow that conforms to the position of the sun in the physical environment.

[0033] There are many different types of electronic systems that enable a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields having integrated display capability, windows having integrated display capability, displays formed as lenses that are designed to be placed on a person’s eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of a physical environment, and / or one or more microphones for capturing audio of the physical environment. Rather than an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representative of an image is directed to a person’s eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, hologram medium, optical combiner, optical reflector, or any combination thereof. In one example, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects graphical images onto a person’s retinas. Projection systems can also be configured to project virtual objects into a physical environment, for example, as a hologram or on a physical surface.

[0034] Figure 1A and Figure 1B An example system 100 used in various computer-generated reality technologies is shown.

[0035] In some embodiments, as Figure 1AAs shown, system 100 includes device 100a. Device 100a includes various components, such as processor 102, RF circuitry 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components optionally communicate over communication bus 150 of device 100a.

[0036] In some embodiments, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, a mobile device, or a laptop computer), and other elements of system 100 are implemented in a head-mounted display (HMD) device designed to be worn by a user, with device 200 in communication with the base station device. In some embodiments, device 100a is implemented in the base station device or the HMD device.

[0037] As Figure 1B shown, in some embodiments, system 100 includes two (or more) devices in communication, such as over a wired or wireless connection. First device 100b (e.g., a base station device) includes processor 102, RF circuitry 104, and memory 106. These components optionally communicate over communication bus 150 of device 100b. Second device 100c (e.g., a head-mounted device) includes various components, such as processor 102, RF circuitry 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch- sensitive surface 122. These components optionally communicate over communication bus 150 of device 100c.

[0038] In some embodiments, system 100 is a mobile device. In some embodiments, system 100 is a head-mounted display (HMD) device. In some embodiments, system 100 is a wearable HUD device.

[0039] System 100 includes processor 102 and memory 106. Processor 102 includes one or more general processors, one or more graphics processors, and / or one or more digital signal processors. In some embodiments, one or more memories 106 are one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions that are configured to be executed by one or more processors 102 to perform the techniques described below.

[0040] The system 100 includes RF circuitry 104. The RF circuitry 104 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, an intranet, and / or a wireless network), and / or a wireless network (such as a cellular telephone network and a wireless local area network (LAN)). The RF circuitry 104 optionally includes circuitry for communicating using near-field communication and / or short-range communication (such as Bluetooth®). ) for communicating using near-field communication and / or short-range communication (such as Bluetooth®).

[0041] The system 100 includes a display 120. In some embodiments, the display 120 includes a first display (e.g., a left-eye display panel) and a second display (e.g., a right-eye display panel), each display for displaying images to a respective eye of the user. Corresponding images are displayed on the first display and the second display at the same time. Optionally, the corresponding images include representations of the same virtual objects and / or the same physical objects from different viewpoints, resulting in a parallax effect that provides the user with a stereoscopic effect of objects on the display. In some embodiments, the display 120 includes a single display. For each eye of the user, corresponding images are displayed on a first region and a second region of the single display at the same time. Optionally, the corresponding images include representations of the same virtual objects and / or the same physical objects from different viewpoints, resulting in a parallax effect that provides the user with a stereoscopic effect of objects on the single display.

[0042] In some embodiments, the system 100 includes one or more touch-sensitive surfaces 122 for receiving user input, such as tap inputs and swipe inputs. In some embodiments, the display 120 and the touch-sensitive surfaces 122 form a touch-sensitive display.

[0043] The system 100 includes an image sensor 108. The image sensor 108 optionally includes one or more visible light image sensors, such as a charge-coupled device (CCD) sensor, and / or a complementary metal-oxide-semiconductor (CMOS) sensor operable to obtain images of physical objects from the real environment. The image sensor also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, to detect infrared light from the real environment. For example, an active IR sensor includes an IR emitter, such as an IR dot emitter, to emit infrared light into the real environment. The image sensor 108 also optionally includes one or more event cameras configured to capture movement of physical objects in the real environment. The image sensor 108 also optionally includes one or more depth sensors configured to detect a distance of physical objects from the system 100. In some embodiments, the system 100 uses a CCD sensor, an event camera, and a depth sensor in combination to detect a physical environment around the system 100. In some embodiments, the image sensor 108 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical objects in the real environment from two different perspectives. In some embodiments, the system 100 uses the image sensor 108 to receive user input, such as a gesture. In some embodiments, the system 100 uses the image sensor 108 to detect a position and an orientation of the system 100 and / or the display 120 in the real environment. For example, the system 100 uses the image sensor 108 to track a position and an orientation of the display 120 relative to one or more fixed objects in the real environment.

[0044] In some embodiments, the system 100 includes one or more microphones 112. The system 100 uses the microphone 112 to detect sound from a user and / or the real environment of the user. In some embodiments, the microphone 112 includes a microphone array (including multiple microphones) that is optionally operated in tandem in order to identify ambient noise or to localize a sound source in a space of the real environment.

[0045] The system 100 includes an orientation sensor 110 to detect an orientation and / or movement of the system 100 and / or the display 120. For example, the system 100 uses the orientation sensor 110 to track changes in a position and / or an orientation of the system 100 and / or the display 120, such as with respect to physical objects in the real environment. The orientation sensor 110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0046] Referring now to Figures 2-10 exemplary techniques for providing a CGR environment are described.

[0047] Figure 2A physical environment in which a user is using (e.g., holding or wearing) a device 200 is depicted. In some embodiments, the device is an embodiment of the system 100, or can be an embodiment that is part of the system 100, such as the device 100a. In some embodiments, the device is a handheld device (e.g., a tablet) that includes a display through which the user can directly view the physical environment (e.g., with pass-through video). The device 200 is configured to present virtual objects on the display such that the user perceives the virtual objects superimposed over the physical environment. In some embodiments, a second device (e.g., an external display) can be connected to the device 200 to provide processing and / or presentation capabilities. Figure 2 In the illustrated embodiment, the device 200 is a handheld device (e.g., a tablet) that includes a display through which the user can directly view the physical environment (e.g., with pass-through video). The device 200 is configured to present virtual objects on the display such that the user perceives the virtual objects superimposed over the physical environment. In some embodiments, a second device (e.g., an external display) can be connected to the device 200 to provide processing and / or presentation capabilities.

[0048] Figure 2 A cup 202A and a table 202B are depicted, both of which are physical objects in the physical environment. As discussed below with respect to Figures 3-10 the user interacts with a CGR environment that includes both real objects (or representations thereof) and virtual objects.

[0049] Figure 3 The CGR environment is depicted from the perspective of the user using the device 200. As Figure 3 shown, the device 200 presents (e.g., displays) a virtual castle 204 superimposed on the table 202B such that the virtual castle 204 appears to be disposed in front of the cup 202A on the table 202B. In some embodiments, the virtual castle 204 is a computer-generated object that does not have a counterpart in the physical environment. In embodiments that incorporate pass-through video, the CGR environment includes a representation of the table 202B that is generated using captured images of the physical environment.

[0050] The virtual castle 204 is opaque and appears to be in front of the cup 202A from the perspective of the user. In embodiments that incorporate pass-through video, the device 200 displays the virtual castle 204 without displaying a representation of the cup 202A that would be generated using captured images of the physical environment if the virtual castle 204 did not obscure the cup. Thus, the user cannot see the cup 202A (or, in the case of pass-through video, a representation of the cup 202A).

[0051] In Figures 3-4 the illustrated embodiment, although the cup 202A is not visible in the CGR environment, the user knows the approximate location of the cup 202A (e.g., because the user placed it in Figure 2 the location shown) and begins to reach for the cup 202A in the physical environment with a hand 206.

[0052] As the user reaches for the cup 202A, the device 200 uses image sensors (e.g., 108) to detect the user’s movement. For example, as the hand 206 moves in the physical environment toward the cup 202A, the device 200 uses the image sensors to obtain information about the hand 206 by capturing images of the physical environment. In some embodiments, the image sensors are located at the device 200, at a device external to the device 200, or a combination thereof.

[0053] In response to detecting the user’s movement, the device 200 determines that the detected user movement is toward the virtual castle 204 (e.g., because the virtual castle 204 is between the user and the cup 202A). In some embodiments, the device 200 determines that the virtual castle 204 occludes the physical cup 202A (e.g., in addition to determining that the detected user movement is toward the virtual castle 204).

[0054] With reference to Figure 4 When the user initially reaches forward, it can not be clear what the user is reaching for. For example, the user can be reaching for the virtual castle 204, the cup 202A, some other object, or for no particular object. Using information about the CGR environment, information obtained about the user’s movement (e.g., the pose, position, velocity, acceleration, etc. of the hand 206), and / or information obtained about the user (e.g., gaze, pupil measurement, previous user behavior), the device 200 determines how to present (e.g., modify) the CGR environment in a way that is consistent with the user’s likely intent (e.g., based on whether the detected user movement is directed toward the virtual castle 204 or the cup 202A).

[0055] As described below, various conditions can be used to determine how to present the CRG environment. These conditions can be based on one or more factors, such as distance, pose, gaze, velocity, or pupil measurement. In some embodiments, a level of certainty is determined about whether the detected user movement is directed toward the virtual castle 204 or the cup 202A. For example, if it is determined that one condition has been met that is consistent with a user movement being directed toward a particular object, then a level of certainty is assigned to the movement being directed toward that object. Conversely, if it is determined that multiple conditions have been met that are consistent with a user movement being directed toward a particular object, then a higher level of certainty is assigned to the movement being directed toward that object. In some embodiments, a device external to the device 200, such as a base station device in communication with the device 200, determines the level of certainty assigned to the movement being directed toward an object.

[0056] In some embodiments, device 200 presents the CGR environment based on a distance between a user (e.g., 206) and a virtual object (e.g., 204). For example, device 200 determines whether a distance between the user and a reference point (e.g., a location of a virtual object or a real object) exceeds a threshold (e.g., non-zero) distance. If the distance exceeds the threshold distance (e.g., the user’s hand is far from the virtual object or the real object), device 200 determines that the user’s movement is not directed to a physical object located behind the virtual object (e.g., device 200 assigns a relatively lower level of certainty to the movement being directed to the physical object). Conversely, if the distance does not exceed the threshold distance (e.g., the user’s hand is close to the virtual object or the real object), device 200 determines that the user’s movement is directed to a physical object located behind the virtual object (e.g., device 200 assigns a higher level of certainty to the user’s movement being directed to the physical object).

[0057] Returning to Figure 4 , in response to detecting the user’s movement, device 200 determines that the distance between hand 206 and virtual castle 204 or cup 202A exceeds the threshold distance (e.g., the user’s movement is less certain to be directed to cup 202A). In some embodiments, in accordance with this determination, device 200 maintains the visual appearance of virtual castle 204. For example, device 200 does not change the level of opacity of virtual castle 204. Thus, virtual castle 204 remains opaque, and the user is still unable to see cup 202A (e.g., in embodiments that incorporate pass-through video, device 200 continues to forego displaying a representation of cup 202A).

[0058] At Figure 5 , the user continues to move hand 206 toward virtual castle 204. As the user moves hand 206 closer to virtual castle 204, device 200 continues to capture information about hand 206 using image sensors. In response to detecting the user’s movement, device 200 updates its determination as to whether the detected user movement is directed to virtual castle 204 or cup 202A. For example, device 200 determines whether the distance between hand 206 and virtual castle 204 or cup 202A exceeds the threshold distance. Upon determining that the distance does not exceed the threshold distance, device 200 changes the visual appearance of virtual castle 204, as indicated by the dashed outline of virtual castle 204. Device 200 modifies the visual appearance by reducing the opacity of all or a portion of virtual castle 204 from an initial value (e.g., 100%) to a final value (e.g., 25%, 50%, 75%). Note that Figure 5 the dashed outline in virtual castle 204 of

[0059] In some embodiments, device 200 changes the visual appearance of virtual castle 204 based at least in part on determining that the pose corresponds to cup 202A. For example, in Figure 5 In the illustrated embodiment, hand 206 is curled into the shape of cup 202A and positioned in a similar orientation, which indicates that the movement is directed toward cup 202A (e.g., increases the level of certainty that the movement is directed toward cup 202A).

[0060] In some embodiments, techniques other than changing the level of opacity can be used to modify the visual appearance of a virtual object. For example, modifying the visual appearance can include one or more of the following operations: obscuring the virtual object or a portion thereof, applying a dissolve pattern to the virtual object or a portion thereof, or applying a dither pattern to the virtual object or a portion thereof. For example, modifying the visual appearance of virtual castle 204 can include stopping the display of the columnar portion of virtual castle 204 such that an aperture appears, allowing the user to see cup 202A behind virtual castle 204.

[0061] Returning to Figure 5 , in the case where hand 206 is near virtual castle 204 or cup 202A, device 200 reduces the level of opacity to guard against the user's movement actually being directed toward cup 202A instead of virtual castle 204. Thus, because of the reduced level of opacity of virtual castle 204, the user can see cup 202A, and thus the user can easily reach for cup 202A.

[0062] In Figure 6 , the user continues to reach forward past the front face of virtual castle 204. As the user moves hand 206 past virtual castle 204 to grasp cup 202A, device 200 determines with a higher level of certainty that the user's movement is directed toward cup 202A. In some embodiments, device 200 determines that the user's movement is very likely directed toward cup 202A when it detects that hand 206 has traveled past a portion of virtual castle 204. Thus, device 200 further reduces the level of opacity of virtual castle 204, as shown by the dashed outline of virtual castle 204 in Figure 6 It is noted that the dashed outline of virtual castle 204 in Figure 6 represents a lower level of opacity than the level of opacity of virtual castle 204 in Figure 5 .

[0063] As shown in Figures 5-6 , device 200 modifies the visual appearance of virtual castle 204 based on the level of certainty of the object that the user's movement is directed toward. In Figure 5At this point, device 200 determines that the user's movement is pointing towards cup 202A at a low level of certainty (e.g., 15%, 30%, 45%). Therefore, device 200 reduces the opacity level of the virtual castle 204 to a first opacity level (e.g., 95%, 80%, 65%). Figure 6 At this point, device 200 determines with a high level of certainty (e.g., 65%, 80%, 95%) that the user is moving towards cup 202A. Therefore, device 200 further reduces the opacity level of the virtual castle 204 to a second opacity level (e.g., 45%, 30%, 15%). In some embodiments, the determined level of certainty does not affect the visual appearance of the virtual object (e.g., 204). For example, in some embodiments, when the visual appearance of the virtual object changes, the opacity level changes to a predetermined level, independent of the level of certainty.

[0064] Figures 7-10 An exemplary representation of a CGR environment based on user movement is shown. Similar to... Figure 3 , Figure 7 A perspective view depicting a user wearing an HMD device in a CGR environment is shown, where the virtual castle 204 is opaque and positioned in front of the cup 200B, thus preventing the user from seeing the cup 202A. Figure 3 In contrast, users exhibit different postures (e.g., the position, orientation, or configuration of their hands, face, body, etc.). For example, in Figure 7 In the illustrated embodiment, hand 206 is oriented with the palm facing down and the index finger extended, while Figure 7 In the middle, hand 206 is oriented with the palm facing sideways and the fingers in a curled position.

[0065] exist Figure 8 At this point, the user extends towards the virtual castle 204. Device 200 uses an image sensor to detect the user's movement. In response to detecting the user's movement, device 200 determines that the user is moving towards the virtual castle 204. In some embodiments, device 200 determines that the virtual castle 204 is obstructing the cup 202A.

[0066] Using information about hand 206 obtained through an image sensor, device 200 determines the pose of hand 206. Utilizing the pose of hand 206, device 200 determines whether the pose corresponds to a feature of a nearby object (e.g., a physical or virtual object within a threshold distance (e.g., a non-zero threshold distance) of hand 206).

[0067] refer to Figure 8Device 200 determines that the gesture of hand 206 corresponds to virtual castle 204 (e.g., not cup 202A). For example, device 200 determines the gesture corresponds to virtual castle 204 because virtual castle 204 has virtual button 208, which is an activatable button provided for user interaction with virtual objects, and hand 206 has an extended index finger. Device 200 obtains data indicating that virtual button 208 is associated with one or more gestures. For example, virtual button 208 is associated with gestures that may be used to activate the button (e.g., a hand with an extended index finger, such as...). Figure 8 (As shown). Based on the acquired data, device 200 determines that the gesture of hand 206 matches one of these gestures (e.g., within a threshold). Therefore, device 200 determines that the user intends to interact with virtual castle 204. In some embodiments, device 200 sorts the correspondence level of the gesture of hand 206 with one or more nearby objects (e.g., objects within a threshold distance of hand 206). Device 200 determines that the object with which the user intends to interact is the object with the highest correspondence to the hand gesture.

[0068] like Figure 8 As shown, when it is determined that the user is moving towards the virtual castle 204, the device 200 maintains the visual appearance of the virtual castle 204 (for example, the device 200 does not reduce the opacity level of the virtual castle 204).

[0069] exist Figure 9 At this point, the user moves toward the virtual castle 204 by placing the index finger of hand 206 on the virtual button 208 and activates the virtual button 208. In response, the device 200 modifies the representation of the virtual castle 204 to include flags and banners, such as... Figure 10 As shown. In Figures 7-10 In this implementation, as device 200 continues to determine that the user intends to interact with virtual castle 204, device 200 maintains the opacity level of virtual castle 204 as the user moves toward virtual castle 204. In some implementations, if the user intention is determined to be at a certainty level exceeding a predetermined threshold (e.g., 70%, 80%, 90%), device 200 maintains the opacity level of virtual castle 204. In some implementations, even if it is determined that the user is moving toward virtual castle 204, device 200 reduces the opacity level (e.g., reduces it by 10%) (e.g., the HMD determines the user intention to interact with virtual castle 204 with a high level of certainty). In some implementations, if the certainty level does not exceed a predetermined threshold, device 200 modifies the visual appearance of virtual castle 204 as the user (e.g., hand 206) approaches virtual castle 204, as described above. Figures 4-5 The subject of discussion.

[0070] In some embodiments, in response to detecting the user movement, device 200 determines whether the detected user movement is directed to virtual castle 204 or cup 202A, where the determination is based on a speed and / or acceleration associated with the user movement (e.g., based on a change in speed, device 200 determines that hand 206 will stop moving at virtual button 208; based on a change in speed, device 200 determines that hand 206 will move past virtual button 208 and stop at or near cup 202A). For example, referring to Figures 7-8 , the user moves hand 206 toward virtual castle 204. In some embodiments, in response to detecting the movement, device 200 determines whether a speed of the movement exceeds a threshold (e.g., non-zero) speed. If the speed exceeds the threshold speed, device 200 determines that the user movement is directed to cup 202A. Upon determining that the user movement is directed to cup 202A, device 200 lowers the opacity level of virtual castle 204. Conversely, if the speed does not exceed the threshold speed, device 200 determines that the user movement is directed to virtual castle 204. Upon determining that the user movement is directed to virtual castle 204, device 200 maintains the opacity level of virtual castle 204, or returns the opacity level to its fully opaque level if the opacity level had been previously lowered. For example, the user can initially stretch forward quickly, which causes device 200 to lower the opacity level of virtual castle 204. However, as the user approaches virtual castle 204, the user slows down. Accordingly, device 200 raises the opacity level of virtual castle 204 to the fully opaque level.

[0071] In some embodiments, in response to detecting the user movement, device 200 determines whether the detected user movement is directed to virtual castle 204 or cup 202A, where the determination is based on a gaze of the user. In some embodiments, device 200 uses image sensors for gaze tracking while the user is moving. For example, referring to Figures 7-8 , the user moves hand 206 toward virtual castle 204. In some embodiments, in response to detecting the movement, device 200 determines which object the gaze is directed to. If the gaze is directed to cup 202A, device 200 determines that the user movement is directed to cup 202A. Conversely, if the gaze is directed to virtual castle 204, device 200 determines that the user movement is directed to virtual castle 204.

[0072] In some embodiments, in response to detecting the user movement, device 200 determines whether the detected user movement is directed to virtual castle 204 or cup 202A, where the determination is based on a pupil measurement (e.g., a size of the pupil). For example, referring to Figures 7-9As the user moves hand 206 toward virtual castle 204, in some embodiments, device 200 determines a change in the user’s pupil size in response to detecting the movement. The size of the pupil can provide an indication that the user is approaching an object to interact with. For example, as the user approaches virtual button 208, the size of the user’s pupil can increase due to the expectation of activating virtual button 208. In some embodiments, if the change in size exceeds a predetermined (e.g., non-zero) threshold, device 200 determines that the user’s movement is directed toward the nearby object (e.g., virtual castle 204). In some embodiments, if the change in size does not exceed the predetermined threshold, device 200 determines that the user is not approaching an object that the user intends to interact with.

[0073] In some embodiments, device 200 can check one or more conditions (e.g., distance, pose, velocity, gaze, pupil measurement) to determine whether the detected user movement is directed toward virtual castle 204 or cup 202A. As discussed above with respect to Figures 3-6 In some embodiments, device 200 can check one or more conditions (e.g., distance, pose, velocity, gaze, pupil measurement) to determine whether the detected user movement is directed toward virtual castle 204 or cup 202A. As discussed above with respect to Figure 5 In some embodiments, device 200 determines that the pose of hand 206 matches (e.g., within a threshold) a pose associated with cup 202A. In some embodiments, by checking the pose in addition to checking the distance, device 200 increases the level of certainty in determining whether the detected user movement is directed toward virtual castle 204 or cup 202A. In some embodiments, particular conditions can be given more weight in determining whether the detected user movement is directed toward virtual castle 204 or cup 202A. For example, the pose condition can be given more weight than the distance condition, or vice versa. Thus, if the two conditions suggest different objects, the condition with more weight will determine which object the user’s movement is directed toward.

[0074] Turning now to Figure 11 a flowchart depicting an example process 1100 for providing a CGR environment. Process 1100 can be performed using a device (e.g., 100a, 100c, or 200) having a display device and one or more sensors. Although process 1100 is described herein with reference to device 200, process 1100 can be performed using any suitable device having a display device and one or more sensors. Figure 11The blocks of process 1100 are depicted in a particular order, but these blocks can be performed in other orders. Some operations in method 1100 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted. Additionally, additional operations are optionally performed.

[0075] At block 1102, the device presents (e.g., via a display device) a CGR environment that includes a virtual object (e.g., 204). In some embodiments, the display device includes an opaque display, and presenting the CGR environment includes presenting the virtual object and pass-through video of the physical environment via the opaque display. In some embodiments, the display device includes a transparent or translucent display (e.g., an additional display) through which the physical environment is directly viewable, and presenting the CGR environment includes presenting the virtual object via the transparent or translucent display.

[0076] At block 1104, the device detects (e.g., via one or more sensors) a user movement (e.g., movement of hand 206) that occurs in the physical environment. In some embodiments, the device obtains data representing the user movement that occurs in the physical environment. In some embodiments, the device detects (e.g., via one or more sensors) a user gesture that occurs in the physical environment, or obtains data representing the user gesture that occurs in the physical environment. In some embodiments, the device detects a user gaze (e.g., via one or more sensors), or obtains data representing the user gaze.

[0077] At block 1106, the device determines whether the user movement is directed at a real object (e.g., 202A) in the physical environment (e.g., a real object occluded by a virtual object in the CGR environment). In some embodiments, determining whether the user movement is directed at a real object includes determining whether the user movement is directed at a virtual object or a real object. In some embodiments, determining whether the user movement is directed at a real object (or, e.g., a virtual object) includes predicting where the detected user movement will stop. In some embodiments, the determination of whether the user movement is directed at a real object (or, e.g., a real object) is performed using a machine learning algorithm. For example, the device determines whether the user movement is directed at a real object based at least in part on previous user movements (e.g., previous hand gestures or movement velocity).

[0078] In some embodiments, the device determines whether the user movement is directed at the real object (or, e.g., the virtual object) in response to determining that the user movement is toward the virtual object and that the virtual object occludes the real object in the physical environment. In some embodiments, determining that the user movement is toward the virtual object includes determining that a distance between the virtual object and a location of the user movement does not exceed a threshold distance. In some embodiments, determining that the virtual object occludes the real object in the physical environment includes determining that the virtual object at least partially overlaps the real object in the CGR environment. In some embodiments, determining that the virtual object occludes the real object in the physical environment includes determining that the virtual object at least partially occludes a view of the real object from a user perspective of the CGR environment.

[0079] At block 1108, in accordance with a determination that the user movement is directed at the real object, the device modifies a visual appearance of the virtual object (e.g., the device changes a transparency of at least a portion of the virtual object).

[0080] In some embodiments, determining that the user movement is directed at the real object includes determining that a user pose corresponds to a feature of the real object (e.g., a hand of the user is in a pose that matches a shape of the real object (or a portion thereof), which indicates that the user intends to grasp the real object). In some embodiments, determining that the user movement is directed at the real object includes determining that a user pose does not correspond to a feature of the virtual object. For example, if the virtual object includes a virtual button, the pose does not correspond to a gesture that the user would use to activate (e.g., push) the virtual button.

[0081] In some embodiments, determining that the detected user movement is directed at the real object includes determining that a velocity associated with the detected user movement exceeds a threshold velocity. In some embodiments, the device determines a velocity and / or acceleration of the user movement over time to predict where the user movement will stop (e.g., whether the user movement will stop at a boundary of the virtual object or at a boundary of the real object). In some embodiments, determining that the user movement is directed at the real object includes determining that the detected user movement will stop at a location that is closer to the real object than to the virtual object (e.g., the detected user movement will stop within a boundary of the real object). In some embodiments, determining that the user movement is directed at the real object includes determining that a user gaze is directed at the real object.

[0082] In some embodiments, modifying the visual appearance of the virtual object includes ceasing presentation of at least a portion of the virtual object. In some embodiments, modifying the visual appearance of the virtual object includes presenting at least a portion of the real object. In embodiments with a transparent or semi-transparent display, presenting the real object includes allowing the user to view the real object by not presenting content (e.g., a virtual object) over the real object. In some embodiments, modifying the visual appearance of the virtual object includes: in accordance with a determination that the user movement is directed to the real object with a first level of confidence, modifying the visual appearance (e.g., transparency) of the virtual object by a first amount; and in accordance with a determination that the user movement is directed to the real object with a second level of confidence that is different from the first level of confidence, modifying the visual appearance of the virtual object by a second amount that is different from the first amount.

[0083] At block 1110, in accordance with a determination that the user movement is not directed to the real object (e.g., a determination that the user movement is directed to the virtual object), the device maintains presentation of the virtual object (e.g., the device maintains display of the virtual object to occlude the real object). In some embodiments, determining that the user movement is directed to the virtual object includes determining that a user gaze is directed to the virtual object. In some embodiments, determining that the user movement is directed to the virtual object includes determining that a user pose corresponds to a feature of the virtual object. In some embodiments, determining that the user movement is directed to the virtual object includes determining that a user pose does not correspond to a feature of the real object. In some embodiments, determining that the user movement is directed to the virtual object includes determining that the detected user movement will stop at a location that is closer to the virtual object than to the real object (e.g., the detected user movement will stop within a boundary of the virtual object).

[0084] Executable instructions for performing features of the process 1100 described above can optionally be included in a transitory or non-transitory computer-readable storage medium (e.g., memory 106) or other computer program product configured to be executed by one or more processors (e.g., processor(s) 102).

[0085] Aspects of the techniques described above envisage the possibility of collecting and using personal information to provide CGR experiences. Such information should be collected with the informed consent of the user.

[0086] An entity that handles such personal information will comply with established privacy practices and / or privacy policies (e.g., policies certified by a third party) that are (1) generally recognized as complying or exceeding industry or government requirements, (2) accessible to users, (3) updated as necessary, and (4) in compliance with applicable laws. An entity that handles such personal information will use those information for reasonable and lawful purposes, without sharing or selling outside of those lawful purposes.

[0087] However, users can optionally limit access to / usage of personal information. For example, users can opt-in or opt-out of collection of their personal information. Moreover, although aspects of the techniques described above can be susceptible to improvement in performance, the present application has been developed for purposes of education and illustration based upon the understanding of the principles involved. The application, thus, can be practiced as embodied in the specification and claims attached hereto or as modified when viewed in light of the following claims in which:

[0088] The foregoing description, for purposes of explanation, describes specific embodiments of the present application. However, the illustrative discussions above are not intended to be exhaustive or to limit the application to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments are chosen and described in order to best explain the principles of the application and its practical applications, and to thereby enable others skilled in the art to best utilize the application and various embodiments with various modifications as are suited to the particular use contemplated.

[0089] While the present disclosure and embodiments have been described with reference to the figures, it will be understood that various changes and modifications can be suggested to one skilled in the art and it is intended that the present application encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method for presenting a computer-generated reality environment, the method comprising: presenting, via a display device, the computer-generated reality environment, the computer-generated reality environment including a virtual object; detecting, via one or more sensors, a user movement occurring in a physical environment; and in response to determining that the detected user movement is towards the virtual object and the virtual object occludes a real object in the physical environment: determining whether the detected user movement is directed at the virtual object or the real object; in accordance with a determination that the detected user movement is directed at the real object, modifying a visual appearance of the virtual object, wherein modifying the visual appearance of the virtual object includes presenting at least a portion of the real object; and in accordance with a determination that the detected user movement is directed at the virtual object, maintaining the presentation of the virtual object to occlude the real object.

2. The method of claim 1, wherein determining that the detected user movement is towards the virtual object includes determining that a distance between the virtual object and a location of the user movement does not exceed a threshold distance.

3. The method of claim 1, further comprising: detecting, via the one or more sensors, a user gesture occurring in the physical environment, wherein the determining that the detected user movement is directed at the real object includes determining that the detected user gesture corresponds to a feature of the real object.

4. The method of claim 1, further comprising: detecting, via the one or more sensors, a user gesture occurring in the physical environment, wherein the determining that the detected user movement is directed at the real object includes determining that the detected user gesture does not correspond to a feature of the virtual object.

5. The method of claim 1, wherein the determining that the detected user movement is directed at the real object includes determining that a velocity associated with the detected user movement exceeds a threshold velocity.

6. The method of claim 1, further comprising: detecting, via the one or more sensors, a user gaze, wherein the determining that the detected user movement is directed at the real object includes determining that the detected user gaze is directed at the real object.

7. The method of claim 1, wherein modifying the visual appearance of the virtual object includes: in accordance with a determination that the detected user movement is directed at the real object with a first level of confidence, modifying the visual appearance of the virtual object with a first magnitude; and in accordance with a determination that the detected user movement is directed at the real object with a second level of confidence that is different from the first level of confidence, modifying the visual appearance of the virtual object with a second magnitude that is different from the first magnitude.

8. The method of claim 1, wherein determining that the virtual object occludes the real object from the physical environment includes determining that the virtual object at least partially overlaps the real object in the computer-generated reality environment.

9. The method of claim 1, wherein determining that the virtual object occludes the real object from the physical environment comprises determining that the virtual object at least partially blocks a view of the real object from a user perspective of the computer-generated reality environment.

10. The method of claim 1, wherein determining whether the detected user movement is directed to the virtual object or the real object comprises predicting where the detected user movement will stop.

11. The method of claim 1, wherein modifying the visual appearance of the virtual object comprises ceasing to present at least a portion of the virtual object.

12. The method of claim 1, wherein determining whether the detected user movement is directed to the virtual object or the real object is performed using a machine learning algorithm.

13. An electronic device, the electronic device comprising: a display device; and means for performing the method of any of claims 1-12.

14. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for: presenting, via a display device, a computer-generated reality environment that includes a virtual object; detecting, via one or more sensors, a user movement occurring in a physical environment; and in response to determining that the detected user movement is directed toward the virtual object and that the virtual object occludes a real object in the physical environment: determining whether the detected user movement is directed to the virtual object or the real object; in accordance with a determination that the detected user movement is directed to the real object, modifying a visual appearance of the virtual object, wherein modifying the visual appearance of the virtual object includes presenting at least a portion of the real object; and in accordance with a determination that the detected user movement is directed to the virtual object, maintaining the presentation of the virtual object to occlude the real object.

15. The computer-readable storage medium of claim 14, wherein determining that the detected user movement is directed toward the virtual object includes determining that a distance between the virtual object and a location of the user movement does not exceed a threshold distance.

16. The computer-readable storage medium of claim 14, wherein the one or more programs further include instructions for: detecting, via the one or more sensors, a user gesture occurring in the physical environment, wherein the determination that the detected user movement is directed to the real object includes determining that the detected user gesture corresponds to a feature of the real object.

17. The computer-readable storage medium of claim 14, wherein the one or more programs further include instructions for: detecting, via the one or more sensors, a user gesture occurring in the physical environment, wherein the determination that the detected user movement is directed to the real object includes determining that the detected user gesture does not correspond to a feature of the virtual object.

18. The computer-readable storage medium of claim 14, wherein the determining that the detected user movement is directed at the real object includes determining that a speed associated with the detected user movement exceeds a threshold speed.

19. The computer-readable storage medium of claim 14, wherein the one or more programs further include instructions for: detecting, via the one or more sensors, a user gaze, wherein the determining that the detected user movement is directed at the real object includes determining that the detected user gaze is directed at the real object.

20. The computer-readable storage medium of claim 14, wherein modifying the visual appearance of the virtual object includes: in accordance with a determination that the detected user movement is directed at the real object with a first level of confidence, modifying the visual appearance of the virtual object with a first magnitude; and in accordance with a determination that the detected user movement is directed at the real object with a second level of confidence that is different from the first level of confidence, modifying the visual appearance of the virtual object with a second magnitude that is different from the first magnitude.

21. The computer-readable storage medium of claim 14, wherein determining that the virtual object occludes the real object from the physical environment includes determining that the virtual object at least partially overlaps the real object in the computer-generated reality environment.

22. The computer-readable storage medium of claim 14, wherein determining that the virtual object occludes the real object from the physical environment includes determining that the virtual object at least partially blocks a view of the real object from a user perspective of the computer-generated reality environment.

23. The computer-readable storage medium of claim 14, wherein the determining that the detected user movement is directed at the virtual object or the real object includes predicting where the detected user movement will stop.

24. The computer-readable storage medium of claim 14, wherein modifying the visual appearance of the virtual object includes ceasing to present at least a portion of the virtual object.

25. The computer-readable storage medium of claim 14, wherein the determining that the detected user movement is directed at the virtual object or the real object is performed using a machine learning algorithm.

26. An electronic device, the electronic device comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: presenting, via a display device, a computer-generated reality environment that includes a virtual object; detecting, via one or more sensors, user movement occurring in a physical environment; and in response to determining that the detected user movement is directed toward the virtual object and that the virtual object occludes a real object in the physical environment: determining whether the detected user movement is directed at the virtual object or the real object; in accordance with a determination that the detected user movement is directed to the real object, modifying a visual appearance of the virtual object, wherein modifying the visual appearance of the virtual object includes presenting at least a portion of the real object; and in accordance with a determination that the detected user movement is directed to the virtual object, maintaining the presentation of the virtual object to occlude the real object.

27. The electronic device of claim 26, wherein determining that the detected user movement is directed toward the virtual object includes determining that a distance between the virtual object and a location of the user movement does not exceed a threshold distance.

28. The electronic device of claim 26, wherein the one or more programs further include instructions for: detecting, via the one or more sensors, a user gesture occurring in the physical environment, wherein the determination that the detected user movement is directed to the real object includes determining that the detected user gesture corresponds to a feature of the real object.

29. The electronic device of claim 26, wherein the one or more programs further include instructions for: detecting, via the one or more sensors, a user gesture occurring in the physical environment, wherein the determination that the detected user movement is directed to the real object includes determining that the detected user gesture does not correspond to a feature of the virtual object.

30. The electronic device of claim 26, wherein the determination that the detected user movement is directed to the real object includes determining that a velocity associated with the detected user movement exceeds a threshold velocity.

31. The electronic device of claim 26, wherein the one or more programs further include instructions for: detecting, via the one or more sensors, a user gaze, wherein the determination that the detected user movement is directed to the real object includes determining that the detected user gaze is directed to the real object.

32. The electronic device of claim 26, wherein modifying the visual appearance of the virtual object includes: in accordance with a determination that the detected user movement is directed to the real object with a first level of confidence, modifying the visual appearance of the virtual object by a first amount; and in accordance with a determination that the detected user movement is directed to the real object with a second level of confidence that is different from the first level of confidence, modifying the visual appearance of the virtual object by a second amount that is different from the first amount.

33. The electronic device of claim 26, wherein determining that the virtual object occludes the real object from the physical environment includes determining that the virtual object at least partially overlaps the real object in the computer-generated reality environment.

34. The electronic device of claim 26, wherein determining that the virtual object occludes the real object from the physical environment includes determining that the virtual object at least partially blocks a view of the real object from a user perspective of the computer-generated reality environment.

35. The electronic device of claim 26, wherein determining whether the detected user movement is directed to the virtual object or the real object includes predicting where the detected user movement will stop.

36. The electronic device of claim 26, wherein modifying the visual appearance of the virtual object includes ceasing to present at least a portion of the virtual object.

37. The electronic device of claim 26, wherein determining whether the detected user movement is directed to the virtual object or the real object is performed using a machine learning algorithm.

Images