Presenting content associated with real-world user interface

Abstract

Some examples of the present disclosure relate to systems and methods for presenting content associated with a real-world user interface in an environment. A user interface of a first object in a physical environment may be detected by an electronic device. In some examples, in response to detecting the user interface of the first object, the electronic device presents content associated with the user interface of the first object in an environment independent of a location of the first object in the physical environment according to one or more criteria being met. In some examples, the content associated with the user interface of the first object includes a timer. In some examples, the content associated with the user interface of the first object includes information corresponding to video content. The environment is optionally a computer-generated environment, and the electronic device optionally includes a head mounted display.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 586,169, filed on September 28, 2023, and U.S. Patent Application No. 18 / ---,---, filed on September ----, 2024, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure generally relates to systems and methods for presenting content in an environment. Background Art

[0004] Some computer graphics environments provide two-dimensional and / or three-dimensional environments in which at least some of the objects displayed for user viewing are virtual and generated by a computer. For example, a virtual object can be viewed simultaneously with one or more objects of the physical environment in the environment. Summary of the invention

[0005] Some examples of the present disclosure relate to systems and methods for presenting content in an environment associated with a user interface of a physical object in a physical environment. In some examples, at an electronic device in communication with one or more displays and one or more input devices, the electronic device detects a user interface of a first object in a physical environment of a user of the electronic device via the one or more input devices. In some examples, in response to detecting the user interface of the first object and based on determining that one or more first criteria are met, the electronic device presents first content associated with the user interface of the first object via one or more displays in an environment independent of the position of the first object in the physical environment. In some examples, the environment is a three-dimensional environment, and the first content corresponds to a virtual and / or computer-generated object associated with the user interface of the first object.

[0006] In some examples, the first content includes one or more virtual representations presented in an area of ​​the environment that is independent of the location of the first object relative to the user's current viewpoint. For example, the first content is presented in the environment at a location that does not correspond to the location of the first object in the environment. For example, the first content is presented in the environment when the user interface of the first object is not included in the user's current field of view. For example, the first content is presented in the environment when the user interface of the first object is outside a threshold distance from a location corresponding to the user's current viewpoint.

[0007] In some examples, one or more first criteria include criteria that are satisfied when the intention of the user of the electronic device is to interact with the user interface of the first object. For example, when the electronic device detects the user interface of the first object, the electronic device detects the user's interaction with the user interface of the first object. For example, the electronic device detects the user's attention to the user interface of the first object. For example, the electronic device detects the user's physical interaction with the user interface of the first object in the physical environment. For example, the electronic device detects the user interaction with the virtual element presented in the environment. In some examples, one or more first criteria include criteria that are satisfied when one or more settings associated with the first content stored in the user profile have a first state. For example, the user profile is associated with the user of the electronic device. In some examples, one or more first criteria include criteria that are satisfied when an operating mode of the electronic device is currently active. For example, the operating mode is used to present content associated with the user interface of an object in the physical environment in the environment. For example, the operating mode is associated with an application that can be accessed by the user of the electronic device.

[0008] In some examples, the user interface of the first object displays a timer set to a first time interval, and the first content includes a timer set to the first time interval. For example, the timer included in the first content corresponds to the timer displayed by the user interface of the first object. In some examples, the user interface of the first object includes video content displayed by a physical display in a physical environment, and the first content includes information corresponding to the video content. For example, the first content includes metadata associated with the video content. In some examples, the user interface of the first object includes video content, and the first content includes supplementary content associated with the video content. For example, the supplementary content includes subtitles and / or supplementary information associated with the video content. For example, the subtitles and / or supplementary information are not displayed by the physical display of the first object in the physical environment. In some examples, presenting the first content includes providing an audio output and / or presenting a picture-in-picture view of the video content in the environment.

[0009] A comprehensive description of these examples is provided in the drawings and detailed description, and it should be understood that this summary does not limit the scope of the present disclosure in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a better understanding of the various examples described herein, reference should be made to the following detailed description and the following drawings. Throughout the drawings, like reference numerals generally refer to corresponding parts.

[0011] Figure 1 An electronic device presenting an extended reality environment according to some examples of the present disclosure is illustrated.

[0012] Figure 2A block diagram illustrating an example architecture of a device according to some examples of the present disclosure.

[0013] FIG. 3A to FIG. 3F An electronic device that presents content in an environment corresponding to a user interface of an object in a physical environment according to some examples of the present disclosure is illustrated.

[0014] FIG. 4A to FIG. 4J An electronic device that presents content in an environment corresponding to a user interface of an object in a physical environment according to some examples of the present disclosure is illustrated.

[0015] Figure 5 A flow diagram illustrating an example process for presenting content in an environment associated with a user interface of an object in a physical environment according to some examples of the present disclosure. DETAILED DESCRIPTION

[0016] Some examples of the present disclosure relate to systems and methods for presenting content in an environment associated with a user interface of an object in a physical environment. In some examples, at an electronic device in communication with one or more displays and one or more input devices, the electronic device detects a user interface of a first object in a physical environment of a user of the electronic device via the one or more input devices. In some examples, in response to detecting the user interface of the first object and based on determining that one or more first criteria are met, the electronic device presents first content associated with the user interface of the first object via one or more displays in an environment independent of the position of the first object in the physical environment. In some examples, the environment is a three-dimensional environment, and the first content corresponds to a virtual and / or computer-generated object associated with the user interface of the first object.

[0017] In some examples, the first content includes one or more virtual representations presented in an area of ​​the environment that is independent of the location of the first object relative to the user's current viewpoint. For example, the first content is presented in the environment at a location that does not correspond to the location of the first object in the environment. For example, the first content is presented in the environment when the user interface of the first object is not included in the user's current field of view. For example, the first content is presented in the environment when the user interface of the first object is outside a threshold distance from a location corresponding to the user's current viewpoint.

[0018] In some examples, one or more first criteria include criteria that are satisfied when the intention of the user of the electronic device is to interact with the user interface of the first object. For example, when the electronic device detects the user interface of the first object, the electronic device detects the user's interaction with the user interface of the first object. For example, the electronic device detects the user's attention to the user interface of the first object. For example, the electronic device detects the user's physical interaction with the user interface of the first object in the physical environment. For example, the electronic device detects the user interaction with the virtual element presented in the environment. In some examples, one or more first criteria include criteria that are satisfied when one or more settings associated with the first content stored in the user profile have a first state. For example, the user profile is associated with the user of the electronic device. In some examples, one or more first criteria include criteria that are satisfied when an operating mode of the electronic device is currently active. For example, the operating mode is used to present content associated with the user interface of an object in the physical environment in the environment. For example, the operating mode is associated with an application that can be accessed by the user of the electronic device.

[0019] In some examples, the user interface of the first object displays a timer set to a first time interval, and the first content includes a timer set to the first time interval. For example, the timer included in the first content corresponds to the timer displayed by the user interface of the first object. In some examples, the user interface of the first object includes video content displayed by a physical display in a physical environment, and the first content includes information corresponding to the video content. For example, the first content includes metadata associated with the video content. In some examples, the user interface of the first object includes video content, and the first content includes supplementary content associated with the video content. For example, the supplementary content includes subtitles and / or supplementary information associated with the video content. For example, the subtitles and / or supplementary information are not displayed by the physical display of the first object in the physical environment. In some examples, presenting the first content includes providing an audio output and / or presenting a picture-in-picture view of the video content in the environment.

[0020] In some examples, a three-dimensional object is displayed in a computer-generated three-dimensional environment at a particular orientation that controls one or more behaviors of the three-dimensional object (e.g., when the three-dimensional object moves within the three-dimensional environment). In some examples, the orientation in which the three-dimensional object is displayed in the three-dimensional environment is selected by a user of the electronic device or automatically selected by the electronic device. For example, when initiating presentation of the three-dimensional object in the three-dimensional environment, the user may select a particular orientation of the three-dimensional object, or the electronic device may automatically select the orientation of the three-dimensional object (e.g., based on the type of the three-dimensional object).

[0021] In some examples, a three-dimensional object may be displayed in a three-dimensional environment in a world-locked orientation, a body-locked orientation, a tilt-locked orientation, or a head-locked orientation, as described below. As used herein, an object displayed in a three-dimensional environment in a body-locked orientation has a distance and orientation offset relative to a portion of the user's body (e.g., the user's torso). Alternatively, in some examples, the body-locked object has a fixed distance from the user, and the orientation of the content does not reference any part of the user's body (e.g., may be displayed in the same basic direction relative to the user regardless of head and / or body movement). Additionally or alternatively, in some examples, the body-locked object may be configured to always maintain gravity or horizon (e.g., perpendicular to gravity) alignment, so that changes in the head and / or body in the rolling direction will not cause the body-locked object to move within the three-dimensional environment. Instead, translational movement in either configuration will cause the body-locked object to reposition within the three-dimensional environment to maintain the distance offset.

[0022] As used herein, an object displayed in a head-locked orientation in a three-dimensional environment has a distance and orientation offset relative to the user's head. In some examples, the head-locked object moves within the three-dimensional environment when the user's head moves (when the user's viewpoint changes).

[0023] As used herein, an object displayed in a world-locked orientation in a three-dimensional environment has no distance or orientation offset relative to the user.

[0024] As used herein, an object displayed in a three-dimensional environment with a tilt-locked orientation (referred to herein as a tilt-locked object) has a distance offset relative to a user, such as a portion of a user's body (e.g., a user's torso) or a user's head. In some examples, the tilt-locked object is displayed with a fixed orientation relative to the three-dimensional environment. In some examples, the tilt-locked object moves according to a polar (e.g., spherical) coordinate system centered on a pole passing through the user (e.g., the user's head). For example, the tilt-locked object is moved in the three-dimensional environment based on the movement of the user's head in a spherical space around the user's head (e.g., centered on the user's head). Thus, if the user tilts his head relative to gravity (e.g., up or down in a pitch direction), the tilt-locked object will follow the head tilt and move radially along the sphere, so that the tilt-locked object is repositioned within the three-dimensional environment to the same distance offset relative to the user as before the head tilt, while optionally maintaining the same orientation relative to the three-dimensional environment. In some examples, if the user moves his head in a rolling direction (e.g., clockwise or counterclockwise) relative to gravity, the tilt-locked object is not repositioned within the three-dimensional environment.

[0025] Figure 1An electronic device 101 is illustrated that presents an extended reality (XR) environment (e.g., a computer-generated environment that optionally includes representations of physical and / or virtual objects) according to some examples of the present disclosure. In some examples, such as Figure 1 As shown, the electronic device 101 is a head mounted display or other head mounted device configured to be worn on the head of the user of the electronic device 101. Figure 2 An example of an electronic device 101 is described by using an architectural block diagram of FIG. Figure 1 As shown, electronic device 101 and table 106 are located in a physical environment. The physical environment may include physical features, such as physical surfaces (e.g., floors, walls) or physical objects (e.g., tables, lights, etc.). In some examples, electronic device 101 may be configured to detect and / or capture an image of the physical environment including table 106 (shown in the field of view of electronic device 101).

[0026] In some examples, such as Figure 1 As shown, the electronic device 101 includes one or more internal image sensors 114a (eg, as described below with reference to FIG. 1 ) oriented toward the user's face. Figure 2 101 ). In some examples, internal image sensor 114a is used for eye tracking (e.g., detecting the gaze of a user). Internal image sensor 114a is optionally arranged on the left and right portions of display 120 to enable eye tracking of the left and right eyes of the user. In some examples, electronic device 101 also includes external image sensors 114b and 114c facing outward from the user to detect and / or capture the physical environment of electronic device 101 and / or movement of the user's hands or other body parts.

[0027] In some examples, the display 120 has a field of view visible to the user (e.g., it may or may not correspond to the field of view of the external image sensors 114b and 114c). Because the display 120 is optionally part of the head-mounted device, the field of view of the display 120 is optionally the same as or similar to the field of view of the user's eyes. In other examples, the field of view of the display 120 may be smaller than the field of view of the user's eyes. In some examples, the electronic device 101 may be an optical see-through device, in which the display 120 is a transparent or translucent display through which parts of the physical environment can be directly viewed. In some examples, the display 120 may be included in a transparent lens and may overlap all or only a portion of the transparent lens. In other examples, the electronic device may be a video pass-through device, in which the display 120 is an opaque display configured to display an image of the physical environment captured by the external image sensors 114b and 114c. Although a single display 120 is shown, it should be understood that the display 120 may include a stereo display pair.

[0028] In some examples, in response to a trigger, the electronic device 101 may be configured to display in the XR environment Figure 1 The cube shown represents a virtual object 104 that does not exist in the physical environment but is displayed in the XR environment positioned on top of a real-world table 106 (or a representation thereof). Optionally, in response to detecting the plane of the table 106 in the physical environment 100, the virtual object 104 can be displayed on the surface of the table 106 in the XR environment displayed via the display 120 of the electronic device 101.

[0029] It should be understood that virtual object 104 is a representative virtual object and may include and render one or more different virtual objects (e.g., virtual objects with various dimensions, such as two-dimensional or other three-dimensional virtual objects) in a three-dimensional XR environment. For example, a virtual object may represent an application or user interface displayed in an XR environment. In some examples, a virtual object may represent content corresponding to an application and / or displayed via a user interface in an XR environment. In some examples, virtual object 104 is optionally configured to be interactive and responsive to user input (e.g., air gestures, such as air pinch gestures, air tap gestures, and / or air touch gestures), so that a user virtually touches, taps, moves, rotates, or otherwise interacts with virtual object 104.

[0030] In some examples, displaying an object in a three-dimensional environment may include interaction with one or more user interface objects in the three-dimensional environment. For example, initiation of display of an object in a three-dimensional environment may include interaction with one or more virtual options / indicators displayed in the three-dimensional environment. In some examples, when display of an object is initiated in a three-dimensional environment, an electronic device may track the user's gaze as an input for identifying one or more virtual options / indicators as targets for selection. For example, gaze may be used to identify one or more virtual options / indicators as targets for selection using another selection input. In some examples, a virtual option / indicator may be selected using hand tracking input detected via an input device that communicates with the electronic device. In some examples, an object displayed in a three-dimensional environment may be moved and / or reoriented in the three-dimensional environment based on movement input detected via an input device.

[0031] In the following discussion, an electronic device that communicates with a display generation component and one or more input devices is described. It should be understood that the electronic device optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand tracking device, an eye tracking device, a stylus, etc. In addition, as described above, it should be understood that the described electronic device, display, and touch-sensitive surface are optionally distributed between two or more devices. Therefore, as used in the present disclosure, information displayed on or by an electronic device is optionally used to describe information output by the electronic device to be displayed on an independent display device (touch-sensitive or non-touch-sensitive). Similarly, as used in the present disclosure, input received on an electronic device (e.g., touch input received on a touch-sensitive surface of an electronic device, or touch input received on the surface of a stylus) is optionally used to describe input received on an independent input device, and the electronic device receives input information from the independent input device.

[0032] The device typically supports a variety of applications, such as one or more of the following applications: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, a television (TV) channel browsing application, and / or a digital video player application.

[0033] Figure 2 A block diagram of an example architecture of a device 201 according to some examples of the present disclosure is illustrated. In some examples, the device 201 includes one or more electronic devices. For example, the electronic device 201 can be a portable device, an auxiliary device that communicates with another device, a head-mounted display, etc. In some examples, the electronic device 201 corresponds to the above reference Figure 1 An electronic device 101 is described.

[0034] like Figure 2 As illustrated, the electronic device 201 optionally includes various sensors, such as one or more hand tracking sensors 202, one or more position sensors 204, one or more image sensors 206 (optionally corresponding to Figure 1114a and / or external image sensors 114b and 114c in the embodiment of the present invention), one or more touch-sensitive surfaces 209, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other audio sensors, one or more body tracking sensors (e.g., torso and / or head tracking sensors), one or more display generation components 214 (optionally corresponding to Figure 1 The display 120 in the electronic device 201, one or more speakers 216, one or more processors 218, one or more memories 220 and / or communication circuits 222. One or more communication buses 208 are optionally used for communication between the above components of the electronic device 201.

[0035] The communication circuit 222 optionally includes circuits for communicating with electronic devices, networks such as the Internet, intranets, wired networks and / or wireless networks, cellular networks, and wireless local area networks (LANs). The communication circuit 222 optionally includes circuits for using near field communication (NFC) and / or short-range communication such as A circuit for communicating.

[0036] Processor 218 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 220 is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) storing computer-readable instructions, which are configured to be executed by processor 218 to perform the techniques, processes, and / or methods described below. In some examples, memory 220 may include more than one non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may be any medium (e.g., excluding signals) that can tangibly contain or store computer-executable instructions for use by or in combination with an instruction execution system, device, and apparatus. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include, but is not limited to, a magnetic storage device, an optical storage device, and / or a semiconductor storage device. Examples of such storage devices include magnetic disks, optical disks based on compact disc (CD), digital versatile disc (DVD) or Blu-ray technology, and persistent solid-state memory (such as flash memory, solid-state drives, etc.).

[0037] In some examples, display generation component 214 includes a single display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), or other type of display). In some examples, display generation component 214 includes multiple displays. In some examples, display generation component 214 may include a display with touch capabilities (e.g., a touch screen), a projector, a holographic projector, a retinal projector, a transparent or translucent display, etc. In some examples, electronic device 201 includes touch-sensitive surfaces 209 for receiving user inputs (such as tap inputs and swipe inputs or other gestures). In some examples, display generation component 214 and touch-sensitive surface 209 form a touch-sensitive display (e.g., a touch screen integrated with electronic device 201 or a touch screen that communicates with electronic device 201 outside electronic device 201).

[0038] The electronic device 201 optionally includes an image sensor 206. The image sensor 206 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real-world environment. The image sensor 206 also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. The image sensor 206 also optionally includes one or more cameras configured to capture the movement of physical objects in the real-world environment. The image sensor 206 also optionally includes one or more depth sensors configured to detect the distance of the physical object from the electronic device 201. In some examples, information from the one or more depth sensors may allow the device to identify an object in the real-world environment and distinguish it from other objects in the real-world environment. In some examples, the one or more depth sensors may allow the device to determine the texture and / or topography of an object in the real-world environment.

[0039] In some examples, the electronic device 201 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment around the electronic device 201. In some examples, the image sensor 206 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor work in tandem and are optionally configured to capture different information of physical objects in the real-world environment. In some examples, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some examples, the electronic device 201 uses the image sensor 206 to detect the positioning and orientation of the electronic device 201 and / or the display generation component 214 in the real-world environment. For example, the electronic device 201 uses the image sensor 206 to track the positioning and orientation of the display generation component 214 relative to one or more fixed objects in the real-world environment.

[0040] In some examples, the electronic device 201 includes a microphone 213 or other audio sensor. The electronic device 201 optionally uses the microphone 213 to detect sounds from the user and / or the user's real-world environment. In some examples, the microphone 213 includes a microphone array (multiple microphones) that optionally operate in conjunction to identify ambient noise or locate a sound source in the space of the real-world environment.

[0041] The electronic device 201 includes a location sensor 204 for detecting the location of the electronic device 201 and / or the display generating component 214. For example, the location sensor 204 may include a global positioning system (GPS) receiver that receives data from one or more satellites and allows the electronic device 201 to determine the absolute location of the device in the physical world.

[0042] The electronic device 201 includes an orientation sensor 210 for detecting the orientation and / or movement of the electronic device 201 and / or the display generation component 214. For example, the electronic device 201 uses the orientation sensor 210 to track changes in the position and / or orientation of the electronic device 201 and / or the display generation component 214, such as relative to physical objects in the real world environment. The orientation sensor 210 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0043] In some examples, the electronic device 201 includes a hand tracking sensor 202 and / or an eye tracking sensor 212 and / or other body tracking sensors, such as a leg tracking sensor, a torso tracking sensor, and / or a head tracking sensor. The hand tracking sensor 202 is configured to track the position / location of one or more parts of the user's hand, and / or the movement of one or more parts of the user's hand relative to the extended reality environment, relative to the display generation component 214, and / or relative to another defined coordinate system. The eye tracking sensor 212 is configured to track the position and movement of the user's gaze (more generally, eyes, face, or head) relative to the real world or extended reality environment and / or relative to the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented together with the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented separately from the display generation component 214.

[0044] In some examples, the hand tracking sensor 202 (and / or other body tracking sensors, such as leg, torso, and / or head tracking sensors) may use an image sensor 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that captures three-dimensional information from the real world including one or more body parts (e.g., the hands, legs, torso of a human user). In some examples, the hands may be resolved with sufficient resolution to distinguish fingers and their corresponding positioning. In some examples, one or more image sensors 206 are positioned relative to the user to define a field of view and an interaction space of the image sensor 206, in which the finger / hand position, orientation, and / or movement captured by the image sensor is used as input (e.g., to distinguish from the user's resting hand or other hands of other people in the real-world environment). Tracking fingers / hands for input (e.g., gestures, touches, taps, etc.) may be advantageous because it does not require the user to touch, hold, or wear any type of beacon, sensor, or other marker.

[0045] In some examples, eye tracking sensor 212 includes at least one eye tracking camera (e.g., an infrared (IR) camera) and / or an illumination source (e.g., an IR light source such as an LED) that emits light toward the user's eyes. The eye tracking camera may be pointed toward the user's eyes to receive reflected IR light from the light source directly or indirectly from the eyes. In some examples, both eyes are tracked separately by corresponding eye tracking cameras and illumination sources, and focus / gaze may be determined by tracking both eyes. In some examples, one eye (e.g., the dominant eye) is tracked by one or more corresponding eye tracking cameras / illumination sources.

[0046] The electronic device 201 is not limited to Figure 2Rather than components and configurations of the electronic device 201, it may include fewer components, other components, or additional components in a variety of configurations. In some examples, the electronic device 201 may be implemented between two electronic devices (e.g., as a system). In some such examples, each (or more) electronic device may each include one or more of the same components discussed above, such as various sensors, one or more display generating components, one or more speakers, one or more processors, one or more memories, and / or communication circuits. One or more people using the electronic device 201 are optionally referred to herein as one or more users of the device.

[0047] Attention is now directed to the interaction of one or more virtual objects displayed in an environment presented at an electronic device, e.g., corresponding to electronic device 201. In some examples, the electronic device presents content in an environment associated with a user interface included in the physical environment of a user of the electronic device.

[0048] FIG. 3A to FIG. 3F An electronic device that presents content in the context of a user interface corresponding to a real-world object (e.g., an object included in a physical environment) according to some examples of the present disclosure is illustrated. In some examples, the content includes a virtual representation of a timer shown by the user interface corresponding to the real-world object (e.g., including content displayed on a display). In some examples, FIG. 3A to FIG. 3F The electronic device 302 shown has one or more characteristics of the electronic device 101 and / or the electronic device 201 described above. In some examples, such as FIG. 3A to FIG. 3F As shown, the electronic device 302 is a head-mounted display that includes a display generation component 330 (e.g., or optionally one or more display generation components having one or more characteristics of the display generation component 214 as described above) that presents the environment 304 to the user 306 (e.g., using a transparent or semi-transparent display). In some examples, the electronic device 302 includes one or more image sensors 314a-314c (e.g., including one or more characteristics of the image sensors 114a-114c and / or the image sensor 206 as described above) configured to detect the physical environment 308 (e.g., as shown in the top view 310) and / or the movement of one or more parts of the user 306 (e.g., hands, head and / or eyes) and / or the attention of the user 306 (e.g., head orientation, gaze). The physical environment 308 has one or more characteristics of the real-world environment and / or the physical environment described above.

[0049] In some examples, environment 304 is a three-dimensional environment presented to user 306 via display generation component 330 of electronic device 302. In some examples, environment 304 is an extended reality (XR) environment having one or more characteristics of the above-described XR environment. For example, from the current viewpoint of user 306, one or more virtual objects (e.g., reference objects) in physical environment 308 may be displayed in a plurality of different ways. Figure 3C The virtual representation 312a shown and described) and / or one or more physical objects (e.g., reference Figure 1 The real world table 106 shown and described and / or referenced Figure 3A 306) is visible (e.g., via a video pass-through or optical see-through of physical environment 308 visible to user 306 via display generation component 330). In some examples, environment 304 is a virtual reality environment (e.g., environment 304 is fully or partially immersive (e.g., user 306 controls the level of virtual immersion via one or more input devices of electronic device 302)).

[0050] Figure 3A An electronic device 302 is illustrated detecting a user interface of an object in a physical environment 308. In some examples, the user interface of the object is visible to a user 306 via a display generation component 330. In some examples, the electronic device 302 detects a real-world user interface 316 of a real-world microwave oven 336. Figure 3A As shown, additional objects included in the physical environment 308 are visible to the user 306 (e.g., through display generation component 330) in the environment 304 (e.g., through video pass-through or optical see-through). For example, from the current viewpoint of the user 306, a real-world stove 332 included in the physical environment 304 is visible in the environment 308. In some examples, the real-world user interface 316 includes a touchpad and / or display of the real-world microwave 336, which the user 306 uses to enter a desired cooking time or view a timer indicating the progress of the cooking time.

[0051] FIG. 3A to FIG. 3FAlso illustrated is an overhead view 310 of a physical environment 308 (e.g., including user 306). In some examples, the physical environment 308 is an indoor space that includes multiple rooms. For example, as shown in the overhead view 310, the physical environment 308 includes a first room 324a (e.g., a kitchen) and a second room 324b (e.g., a living room). As shown in the overhead view 310, the first room 324a includes a real-world microwave 336 and a real-world stove 332. In some examples, the physical environment 308 is an indoor and / or outdoor space that includes multiple physical objects (e.g., indoor furniture, appliances, doors, walls, floors, outdoor furniture, outdoor structures, and / or natural objects) that are visible to the user 306 in the environment 304 via video pass-through and / or optical see-through (e.g., via a display generation component 330). In some examples, one or more virtual objects (e.g., such as reference images) are visible to the user 306 via video pass-through and / or optical see-through. Figure 3C , Figure 3E and Figure 3F The virtual representations 312a-312b-c shown and described are presented in the environment 304 and are visible to the user 306 (e.g., by the display generation component 330) simultaneously with a plurality of physical objects included in the physical environment 308. In some examples, the movement of the user 306 in the physical environment 308 corresponds to the movement of the current viewpoint of the user 306 relative to the environment 304. For example, the movement of the user 306 in the physical environment 308 (e.g., caused by the physical movement of the user 306) from the first room 324a to the second room 324b corresponds to the movement of the current viewpoint of the user 306 relative to the environment 304 from a first area of ​​the environment 304 associated with the first room 324a to a second area of ​​the environment 304 associated with the second room 324b. In some examples, the movement of the current viewpoint of the user 306 relative to the environment 304 causes one or more virtual objects presented in the world-locked orientation in the first area of ​​the environment 304 to no longer be visible to the user 306. In some examples, movement of the current viewpoint of user 306 relative to environment 304 causes a view of one or more physical objects included in physical environment 308 to change while maintaining a view of one or more virtual objects in environment 304 presented in a body-locked orientation or a head-locked orientation from the current viewpoint of user 306.

[0052] In some examples, electronic device 302 detects user 306's interaction with real-world microwave oven 336 and / or real-world user interface 316 while detecting real-world microwave oven 336 and / or real-world user interface 316. For example, electronic device 302 optionally detects real-world user interface 316 and / or user interaction with real-world user interface 316 using image sensors 314a-314c. In some examples, detecting user 306's interaction with real-world microwave oven 336 and / or real-world user interface 316 includes detecting hand 338 of user 306 providing input to real-world user interface 316 and / or a change in the appearance of real-world user interface 316 (e.g., a change in a time interval displayed by real-world user interface 316 based on input provided by user 306). In some examples, information associated with user 306's interaction with real-world user interface 316 is stored by electronic device 302 in a memory (e.g., with reference to FIG. 1 ). Figure 2 For example, information associated with the user 306's interaction with the real-world user interface 316 is used to present content in the environment 304 associated with the real-world user interface 316 (e.g., as described below).

[0053] Figure 3B Illustrated with a Figure 3A In some examples, the electronic device 302 detects further interaction of the user 306 with the real-world user interface 316 (e.g., the user 306 presses a start button on the real-world user interface 316 to start the real-world microwave 336). In some examples, the electronic device 302 optionally detects a change in the appearance of the real-world user interface 316 (e.g., using image sensors 314a-314c). For example, when an interaction of the user 306 with the real-world user interface 316 is detected (e.g., as shown in FIG. 1 ), the electronic device 302 may detect an update of the appearance of the real-world user interface 316 based on the input provided by the user 306. Specifically, a two-minute timer is set on the real-world microwave 336 based on the user input. In some examples, the electronic device 302 detects further interaction of the user 306 with the real-world user interface 316 (e.g., the user 306 presses a start button on the real-world user interface 316 to start the real-world microwave 336). Figure 3A), the electronic device 302 detects that the display of the real-world user interface 316 changes based on the interaction (e.g., the electronic device 302 detects that one or more numbers are added to the real-world user interface 316 (e.g., associated with the selected time interval)). For example, after detecting the interaction of the user 306 with the real-world user interface 316, the electronic device 302 optionally detects the time interval displayed by the real-world user interface 316. For example, after detecting the interaction of the user 306 with the real-world user interface 316, the electronic device 302 detects that the time interval displayed by the real-world user interface 316 changes (e.g., corresponding to a sequential countdown of a two-minute timer). In some examples, the electronic device 302 stores information associated with the user's interaction with the real-world user interface 316, the updated appearance of the real-world user interface, and / or the changed appearance of the real-world user interface (e.g., in a memory). For example, the electronic device 302 stores information corresponding to the time interval displayed by the real-world user interface 316. In some examples, the time interval (e.g., and / or the change in the time interval displayed by the real-world user interface 316) is used to present a virtual representation in the environment 304 (e.g., as shown in FIG. 2 ). Figure 3C shown and described).

[0054] In some examples, the electronic device 302 presents virtual content associated with the real-world user interface 316 in the environment 304 based on one or more criteria being satisfied. For example, the electronic device 302 determines whether the virtual content associated with the real-world user interface 316 should be presented in the environment 304 using the one or more criteria. Presenting the content associated with the real-world user interface 316 in the environment 304 when the one or more criteria are met ensures that the content is presented in the environment 304 when the user 306 intends and / or expects the content to be presented in the environment 304. For example, the one or more criteria include criteria that are satisfied when the electronic device 302 detects a user interaction with the real-world user interface 316 (e.g., as referenced in FIG. 1 ). Figure 3AFor example, one or more criteria include criteria that are satisfied when the electronic device 302 detects a visual appearance of the real-world user interface 316 (e.g., a time interval and / or countdown of a timer displayed by the real-world user interface 316 (e.g., optionally after detecting a user interaction with the real-world user interface 316)). For example, one or more criteria include criteria that are satisfied when the electronic device 302 detects an intent of the user 306 to interact with the real-world user interface 316 (e.g., further interact). For example, the electronic device 302 detects the intent of the user 306 to interact with the real-world user interface 316 based on the user 306 pointing at (e.g., paying attention to) the real-world user interface 316 for a threshold time period (e.g., 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds) (e.g., the electronic device 302 detects the orientation of the electronic device 302 and / or the gaze of the user 306 (e.g., using a reference having a position). Figure 2 One or more characteristics of eye tracking sensor 212 are shown and described herein as one or more eye tracking sensors).

[0055] In some examples, the one or more criteria include criteria that are satisfied when the electronic device 302 is operating in the operating mode before and / or during detection of the real-world user interface 316 (e.g., before and / or during interaction of the user 306 with the real-world user interface 316). Establishing criteria for presenting content associated with the real-world user interface in the environment 304 (including operating the electronic device 302 in the operating mode) provides the user 306 with discretion to decide when the electronic device 302 should present content associated with one or more real-world user interfaces in the environment 304. In some examples, the operating mode is associated with an application that can be accessed by the user 306 (e.g., via the electronic device 302). For example, when the application is launched (e.g., when the user 306 interacts with the real-world user interface 316), the user 306 may select the electronic device 302 to present content associated with the real-world user interface in the environment 304. Figure 3A In some examples, the application is associated with presenting content associated with one or more real-world objects (e.g., and / or one or more real-world user interfaces of one or more real-world objects) in environment 304. For example, operating electronic device 302 in operating mode prompts electronic device 302 to detect one or more user interfaces (e.g., real-world user interface 316) and present content associated with the one or more user interfaces in environment 304. In some examples, if electronic device 302 is not operating in operating mode (e.g., as shown in FIG. 1 ) before and / or while user 306 interacts with real-world user interface 316, the application is associated with presenting content associated with one or more real-world objects (e.g., and / or one or more real-world user interfaces of one or more real-world objects) in environment 304. Figure 3A), the electronic device 302 abandons presenting the content in the environment 304 associated with the real-world user interface 316.

[0056] In some examples, the one or more criteria include criteria that are satisfied when one or more settings stored in a user profile of user 306 have a first state. For example, the user profile includes one or more settings for presenting (or not presenting) content associated with one or more objects (e.g., or one or more user interfaces of one or more objects) in environment 304. For example, the one or more settings include a setting for presenting content associated with cooking in environment 304 (e.g., if the setting is a first state, the electronic device 302 presents content associated with real-world user interface 316, and if the setting is a second state different from the first state, the electronic device 302 forgoes presenting content associated with real-world user interface 316).

[0057] In some examples, based on one or more criteria being met, the electronic device 302 presents content associated with the real-world user interface 316 in the environment 304. In some examples, the content associated with the real-world user interface 316 is presented at a location that is different (e.g., independent) from the location of the real-world microwave oven 336 and / or the user interface 316. For example, the content is not presented at a location in the environment 304 that corresponds to the location of the real-world microwave oven 336 and / or the user interface 316. In some examples, the content associated with the real-world user interface 316 is initially presented at the location of the real-world user interface 316 and then moved to a different location (e.g., optionally using an animation). In some examples, the content is displayed at a predetermined location, such as in a corner of the display or along an edge of the display (e.g., the center of the top edge, the lower right corner, etc.). In some examples, the content is presented in the environment 304 in a body-locked orientation and / or a head-locked orientation from the current viewpoint of the user 306 (e.g., the content remains in the field of view during changes in the viewpoint of the user 306 relative to the environment 304).

[0058] In some examples, presenting content associated with real-world user interface 316 in environment 304 includes adding the content associated with real-world user interface 316 to a user interface (e.g., such as a reference page) accessible to user 306 through environment 304. Figure 3E For example, in response to one or more criteria being met, the electronic device 302 adds the content to the system user interface accessible via the user input and abandons the content that is visible in the environment 304 from the current viewpoint of the user 306 through one or more virtual representations (e.g., such as reference images). Figure 3CIn some examples, the electronic device 302 may present content via one or more virtual representations in the environment 304 (e.g., in addition to adding the content to the system user interface). In some examples, the electronic device 302 may present content via one or more virtual representations in the environment 304 (e.g., in addition to adding the content to the system user interface) based on one or more determinations made by the electronic device 302 (e.g., such as based on the location of the current viewpoint of the user 306 as described below).

[0059] In some examples, presenting content associated with the real-world user interface 316 in the environment 304 includes presenting one or more virtual representations (e.g., reference to the real-world user interface 316) in the environment 304. Figure 3C In some examples, based on one or more criteria for presenting content associated with the real-world user interface 316 in the environment 304 being satisfied, the electronic device determines whether to present the one or more virtual representations in the environment 304 from the user's current viewpoint. In some examples, the electronic device 302 determines whether to present the one or more virtual representations in the environment 304 based on the location (e.g., position and / or orientation) of the user 306's current viewpoint (e.g., and the electronic device 302) in the environment 304. In some examples, the electronic device 302 presents the one or more virtual representations in the environment 304 based on determining that the location in the environment 304 corresponding to the user's 306 current viewpoint is (e.g., has moved) outside a threshold distance from the real-world microwave oven 336 (e.g., after a timer is started). Thus, in Figure 3B 310, a reference line 344 is illustrated. In some examples, the movement of the user 306 in the physical environment 304 beyond the reference line 344 corresponds to the movement of the current viewpoint of the user 306 relative to the environment 304 beyond a threshold distance from the real-world microwave oven 336 (e.g., in response to the movement of the user 306 relative to the environment 304 beyond the threshold distance from the real-world microwave oven 336, the electronic device 302 presents one or more virtual representations in the environment 308). In some examples, the threshold distance is 0.1m, 0.2m, 0.5m, 1m, 2m, 5m, 10m, 20m, or 30m from the real-world microwave oven 336 relative to the environment 304. In some examples, the electronic device 302 presents one or more virtual representations based on determining that the user 306 has moved to a different room within the indoor space (e.g., the reference line 344 corresponds to an area of ​​the physical environment 308 that separates the first room 324a from the second room 324b). In some examples, electronic device 302 presents one or more virtual representations in environment 304 based on determining that real-world user interface 316 is not within the field of view of user 306 or electronic device 302. For example, Figure 3BAs shown, when the real-world user interface 316 is within the field of view of the user 306, the electronic device 302 optionally forgoes presenting content associated with the real-world user interface 316 in the environment 304. In some examples, the electronic device 302 presents one or more virtual representations in the environment 304 based on determining that the movement of the current viewpoint of the user 306 exceeds an orientation threshold (e.g., 0.1 degrees, 0.5 degrees, 1 degree, 2 degrees, 5 degrees, 10 degrees, 20 degrees, 30 degrees, 40 degrees, 45 degrees, 50 degrees, 60 degrees, 70 degrees, 80 degrees, 90 degrees, 100 degrees, 120 degrees, or 180 degrees) relative to the environment 304. For example, when one or more criteria (e.g., as described above) are met, the viewpoint of the user 306 has a first orientation relative to the environment 304, and has a second orientation different from the first orientation relative to the environment 304 after the one or more criteria are met. Based on the difference between the first orientation and the second orientation exceeding the orientation threshold, the electronic device 302 presents one or more virtual representations in the environment 304. Presenting one or more virtual representations in environment 304 based on a position (e.g., location and / or orientation) corresponding to a current viewpoint of user 306 relative to environment 304 ensures that content is visible to user 306 when useful. For example, if a virtual representation includes a timer corresponding to a time interval displayed by real-world user interface 316, and real-world user interface 316 is currently visible to user 306, then it is not necessary to present the virtual representation in environment 304 (e.g., because the time interval displayed by real-world user interface 316 is already visible to user 306).

[0060] Figure 3C An electronic device 302 is illustrated presenting content associated with a real-world user interface 316 in an environment 304 in response to a user 306 moving to a second room 324b. Specifically, a virtual representation 312a is presented in the environment 304. In some examples, the electronic device 302 presents the virtual representation 312a in the environment 304 because one or more criteria (e.g., as described above) for presenting content associated with the real-world user interface 316 in the environment 304 are met. In some examples, the electronic device 302 determines when to present the virtual representation 312a in the environment 304 based on the location (e.g., position and / or orientation) of the current viewpoint of the user 306 (e.g., the electronic device 302 presents the virtual representation 312a when the user 306 moves to the room 326b). Figure 3C As shown, the virtual representation 312a includes a Figure 3B A timer (e.g., Figure 3C The view of the environment 304 is shown Figure 3BFor example, the electronic device 302 updates the timer included in the virtual representation 312a to correspond to the current value of the timer presented by the real-world user interface 316, regardless of whether the real-world user interface 316 is currently visible to the user 306 or the electronic device 302. For example, when the real-world user interface 316 is visible to the user 306 or the electronic device 302 and the virtual representation 312a is not presented in the environment 304, the electronic device 302 tracks the amount of time remaining on the timer displayed by the real-world user interface 316. For example, when the real-world user interface 316 is not visible to the user 306 or the electronic device 302 and the virtual representation 312a is presented in the environment 304, the electronic device 302 presents the correct amount of time remaining to the timer included in the virtual representation 312 (e.g., the presented amount of time remaining corresponds to the amount of time remaining on the timer displayed by the real-world user interface 316 because the electronic device 302 continues to update the timer regardless of whether the real-world user interface 316 is visible).

[0061] In some examples, the virtual representation 312a includes one or more selectable options. Figure 3C As shown, the virtual representation 312a includes a first selectable option 334a and a second selectable option 334b. In some examples, the first selectable option 334a can be selected to stop presenting the virtual representation 312a in the environment 304, and the second selectable option 334b ​​can be selected to pause or continue the countdown of the timer presented by the virtual representation 312a. In some examples, the first selectable option 334a and the second selectable option 334b ​​can be selected by a selection input corresponding to attention (e.g., gaze) directed to the first selectable option 334a or the second selectable option 334b, while the user 306 optionally performs an air gesture (e.g., air pinch, air long pinch, air tap, air drag or air pinch). In some examples, the electronic device 302 presents a virtual representation 312a with a subset of the above selectable options or without one or more selectable options (e.g., presenting a virtual representation 312a without the first selectable option 334a and / or the second selectable option 334b). In some examples, the virtual representation 312a is presented in a head-locked orientation and / or a body-locked orientation relative to the current viewpoint of the user 306 (e.g., the virtual representation 312a is not displayed in a world-locked orientation (e.g., at a location corresponding to the real-world user interface 316 and / or the real-world microwave oven 336)).

[0062] exist Figure 3C, virtual representation 312a is presented at the lower right corner of environment 304 from the current viewpoint of user 306. In some examples, virtual representation 312a is alternatively presented in an area of ​​environment 304 that is different from the lower right corner of environment 304 (e.g., the upper right corner, left corner, bottom, top, and / or side of the display area of ​​environment 304 from the current viewpoint of user 306). In some examples, virtual representation 312a is presented at a location relative to the current viewpoint of user 306 that avoids visual (e.g., spatial) conflict with one or more objects visible in environment 304 (e.g., such as Figure 3C As shown, from the current viewpoint of user 306, virtual representation 312a does not overlap and / or intersect with real-world TV 340 and real-world table 342. In some examples, virtual representation 312a is presented at a certain position that overlaps in front of one or more objects in environment 304 from the current viewpoint of user 304 (e.g., electronic device 302 presents virtual representation 312a at a closer distance in environment 304 relative to the current viewpoint of user 306 than one or more objects visible in environment 306). In some examples, virtual representation 312a is presented at a default display position relative to the current viewpoint of user 306 (e.g., the display position is established by one or more default settings of electronic device 302). In some examples, virtual representation 312a is presented at a display position set by user 306 (e.g., user 306 establishes the display position of virtual representation 312a through one or more settings accessed via a system and / or settings user interface of electronic device 302, and / or the display position is included in a user profile associated with user 306).

[0063] In some examples, the electronic device 302 maintains presentation of the virtual representation 312a in the environment 304 until the user 306 provides input corresponding to a request to stop presenting the virtual representation 312a (e.g., by selecting the first selectable option 334a). In some examples, the electronic device 302 maintains presentation of the virtual representation 312a in the environment 304 until a countdown of a timer is completed (e.g., the electronic device 302 automatically stops presentation of the virtual representation 312a after notifying the user 306 that the timer is completed). In some examples, the electronic device 302 presents the virtual representation 312a having one or more dynamic visual characteristics in the environment 304. For example, the electronic device 302 presents the virtual representation 312a having different amounts of brightness (e.g., the virtual representation 312a flashes / blinks in the environment 304) and / or colors (e.g., the color of the virtual representation 312a changes based on the amount of time remaining). In some examples, the electronic device 302 periodically presents the virtual representation 312a (e.g., as shown in reference to Figure 3FIn some examples, the virtual representation 312a is presented without text (e.g., without text such as Figure 3C The word "timer" shown in the figure) and / or virtual container (e.g., without Figure 3C 312a).

[0064] FIG. 3D to FIG. 3E An example of presenting content associated with a real-world user interface 316 that is different from the virtual representation 312a in the environment 304 is illustrated. Specifically, in some examples, the electronic device 302 adds content associated with the real-world user interface 316 (e.g., a timer) to the system user interface (e.g., reference Figure 3E In some examples, in addition to presenting the virtual representation 312a in the environment 304, the electronic device 302 also adds content associated with the real-world user interface 316 to the system user interface. In some examples, the system user interface can be accessed by a user 306 in the environment 304 through one or more user inputs. For example, in Figure 3D In the example, user 306 provides input including attention (e.g., gaze 318) directed to an area (e.g., an upper area) of environment 304. In some examples, the area of ​​environment 304 to which gaze 318 is directed is different from Figure 3D An area of ​​the environment 304 in which the virtual representation 312a is displayed (e.g., a lower area, a corner, a side area, and / or a blank space of the environment 304 (e.g., an area of ​​the environment 304 that does not include virtual and / or real-world objects)). In some examples, the area in the environment 304 to which the gaze 318 is directed is a default area in the environment 304 associated with presenting a system user interface (e.g., corresponding to a default setting of the electronic device 302). In some examples, the area in the environment 304 to which the gaze 318 is directed is set by the user 306 (e.g., through one or more settings associated with a user profile). In some examples, the input provided by the user 306 includes the gaze 318 directed to the area of ​​the environment 304 for a threshold amount of time (e.g., 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds). For example, based on the gaze 318 being directed to the area of ​​the environment 304 for less than the threshold amount of time, the electronic device 302 abandons presenting the system user interface in the environment 304. In some examples, input provided by user 306 includes an air gesture (e.g., in addition to gaze 318 directed toward an area of ​​environment 304). For example, the air gesture includes an air pinch, a long air pinch (e.g., an air pinch held for 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds), an air tap, and / or an air pointing performed by a hand of user 306.

[0065] Figure 3E This example shows a response to Figure 3D In some examples, the system user interface 320 is presented in the environment 304 based on input provided by the user 306 in the environment. In some examples, the system user interface 320 includes one or more virtual elements, including notifications, icons, and / or widgets associated with one or more applications. Figure 3E As shown, system user interface 320 includes virtual representation 346a and virtual representation 346b. In some examples, virtual representation 346a corresponds to a notification received from a messaging application (e.g., the notification is associated with a text message received from electronic device 302 (e.g., and / or optionally from a device in communication with electronic device 302)). In some examples, virtual representation 346b corresponds to a notification received from a calendar application (e.g., the notification is associated with an event). In some examples, system user interface 320 includes Figure 3E In some examples, the system user interface 320 includes one or more menus for viewing and / or updating settings (e.g., system, device, and / or user settings). In some examples, the system user interface 320 includes one or more visual indications not included in the system user interface 320 (e.g., the system user interface 320 includes an indication of the current time, date, and / or weather (e.g., the outdoor temperature at a location associated with the current location of the electronic device 302)). Figure 3E As shown, electronic device 302 changes the visual appearance of environment 304 when presenting system user interface 320. For example, electronic device 302 reduces the opacity, color, saturation, brightness, and / or sharpness of one or more objects (e.g., including representations of virtual objects and / or real-world objects) in environment 304 when presenting system user interface 320. In some examples, electronic device 302 does not change the visual appearance of environment 304 when presenting system user interface 320.

[0066] like Figure 3E As shown, the system user interface 320 includes a virtual representation 312b. In some examples, the virtual representation 312b includes a corresponding Figure 3B A timer (e.g., Figure 3E The view of the environment 304 is shown Figure 3B In some examples, electronic device 302 responds to user 306 and FIG. 3A to FIG. 3B The virtual representation 312b is added to the system user interface 320 (e.g., the electronic device 302 detects that the real world user interface 316 is interactively displayed). Figure 3A User input shown and / or Figure 3BIn some examples, electronic device 302 presents virtual representation 312a in environment 304 (e.g., based on determining that a location corresponding to a current viewpoint of user 306 is greater than a threshold distance from real-world microwave oven 336 (e.g., as shown in FIG. 1 ). FIG. 3B to FIG. 3C 312a in the environment 304, then when the system user interface 320 including the virtual representation 312b is presented with the virtual representation 312b, the electronic device 302 stops presenting the virtual representation 312a in the environment 304. In some examples, the virtual representation 312b includes one or more selectable options for controlling a timer included in the virtual representation 312b (e.g., selectable options for releasing, stopping, and / or resuming a timer).

[0067] In some examples, the electronic device 302 adds multiple virtual representations (e.g., corresponding to content associated with one or more real-world objects and / or user interfaces) to the system user interface 320. For example, the electronic device 302 detects user interactions with one or more real-world user interfaces that are different from the real-world user interface 316 (e.g., real-world user interfaces associated with a real-world washing machine, dryer, TV, mobile device, laptop, and / or wearable device (e.g., smart watch)), and based on one or more criteria being met (e.g., as described above), the electronic device 302 includes one or more virtual representations of content associated with the one or more real-world user interfaces (e.g., corresponding to the reference user interfaces). FIG. 4F to FIG. 4I The virtual representation 312b and / or virtual representations 412a-412e shown and described are added to the system user interface 320. In some examples, one or more of the virtual representations include a timer that is not associated with the real-world user interface (e.g., the timer is set by user input other than interaction with the real-world user interface (e.g., verbal input and / or user input provided through an application)).

[0068] Figure 3F An example of presenting content associated with real-world user interface 316 that is different from virtual representation 312a and virtual representation 312b in environment 304 is illustrated. Figure 3F In some examples, in addition to adding virtual representation 312b to system user interface 320, electronic device 302 also periodically presents virtual representation 312c (e.g., as shown in FIG. Figure 3EFor example, during a first time period, the electronic device 302 detects the real-world user interface 316 and / or user interactions with the real-world user interface 316, and stores information associated with the real-world user interface 316 (e.g., stores a time interval displayed by the real-world user interface 316). For example, at a second time period after the first time period (e.g., when less than a threshold amount of time remains on a timer displayed by the real-world user interface 316), the electronic device 302 presents the virtual representation 312c in the environment 304. In some examples, the electronic device 302 presents one or more notifications in the environment 304 corresponding to the amount of time remaining on the timer associated with the real-world user interface 316. Figure 3F As shown, the virtual representation 312c includes an indication that 10 seconds remain in the timer (e.g., Figure 3F The view of the environment 304 is shown Figure 3B The view of the environment 304 shown is presented 110 seconds later). In some examples, the virtual representation 312c is presented when different amounts of time remain in the timer (e.g., 0 seconds, 5 seconds, 15 seconds, 20 seconds, 25 seconds, 30 seconds, or 60 seconds, and / or 2 minutes, 5 minutes, 10 minutes, 15 minutes, 20 minutes, 25 minutes, 30 minutes, 60 minutes, or 120 minutes). In some examples, the periodicity of the presentation of the virtual representation 312c is set by one or more default settings of the electronic device 302. In some examples, the periodicity of the presentation of the virtual representation 312c is set by the user 306 (e.g., by one or more settings associated with a user profile, or one or more settings associated with an application). In some examples, presenting the virtual representation 312c in the environment 304 includes providing an audio output (e.g., by an output device (e.g., a speaker) of the electronic device 302). In some examples, the electronic device 302 stops presenting the virtual representation 312c in the environment 304 after a threshold amount of time (e.g., 1 second, 2 seconds, 5 seconds, 10 seconds, 15 seconds, 20 seconds, 25 seconds, 30 seconds, 60 seconds, or 120 seconds). In some examples, the electronic device 302 stops presenting the virtual representation 312c in the environment 304 in response to a user input corresponding to a request to stop presenting the representation 312c (e.g., the user input includes a selection of a selectable option for stopping presenting the virtual representation 312c, a verbal input (e.g., a voice command), and / or an air gesture).

[0069] In some examples, based on the electronic device 302 detecting user interactions with one or more real-world user interfaces different from the real-world user interface 316 (e.g., as described above) and tracking multiple timers (e.g., multiple timers are added to the system user interface 320), the electronic device 302 presents a virtual representation 312c corresponding to a prioritized timer among the multiple timers. For example, the prioritized timer corresponds to a recent timer (e.g., the user 306 has recently interacted with the real-world user interface associated with the prioritized timer). For example, the prioritized timer corresponds to a timer with a remaining amount of time less than a threshold amount of time (e.g., less than 1 second, 5 seconds, 10 seconds, 15 seconds, 20 seconds, 25 seconds, 30 seconds, or 60 seconds, and / or less than 2 minutes, 5 minutes, 10 minutes, 15 minutes, 20 minutes, 25 minutes, 30 minutes, 60 minutes, or 120 minutes). For example, the prioritized timer corresponds to a timer that the electronic device 302 determines to prioritize based on detecting user input (e.g., the user 306 provides a speech input corresponding to a request for a prioritized timer and / or the timer is set to be prioritized by the user 306 through one or more settings of a user profile).

[0070] FIG. 4A to FIG. 4J An electronic device that presents content in an environment of a user interface corresponding to a real-world object (e.g., an object included in a physical environment) according to some examples of the present disclosure is illustrated. In some examples, the content includes one or more virtual representations associated with video content displayed by a physical display of a real-world TV. In some examples, the one or more virtual representations include information and / or supplemental content associated with the video content displayed by the real-world TV in the physical environment (e.g., the information and / or supplemental content includes subtitles, metadata, annotations, and / or information associated with the video content (e.g., optionally, the information and / or supplemental content is not included in the video content displayed by the real-world TV)). In some examples, the electronic device 402 has a reference FIG. 3A to FIG. 3E One or more features of the electronic device 302 are shown and described. FIG. 4A to FIG. 4J As shown, the electronic device 402 includes a display generation component 430 (for example, the display generation component 430 has a reference FIG. 3A to FIG. 3F In some examples, the image sensors 414a-414c have one or more characteristics of the image sensors 314a-314c. In some examples, the environment 404 has a reference FIG. 3A to FIG. 3F One or more characteristics of environment 304 are shown and described. In some examples, physical environment 408 (eg, shown in top view 410) has reference FIG. 3A to FIG. 3F One or more characteristics of physical environment 308 are shown and described.

[0071] Figure 4A A user 406 is illustrated viewing video content 444 displayed by a real-world user interface (e.g., a physical display) of a real-world TV 440 in a physical environment 408. In some examples, the electronic device 402 detects the real-world user interface of the real-world TV 440 (e.g., using image sensors 414a-414c). In some examples, the user 406 views the video content 444 displayed by the real-world TV 440 through the display generation component 430 (e.g., the physical display of the real-world TV 440 is visible through video pass-through or optical see-through). In some examples, the video content 444 includes movies, television shows, live television shows (e.g., sporting events broadcast on cable television or streamed through a streaming service), and / or online videos from a video sharing service or social media application. In some examples, video content 444 includes content projected from an electronic device (e.g., electronic device 402 or a different electronic device (e.g., such as a mobile device, a laptop, and / or a wearable device (e.g., a smart watch or a head-mounted display)) to real-world TV 440. As shown in top view 410, real-world TV 440 is included in room 424b of physical environment 408. In some examples, physical environment 408 includes one or more objects other than real-world TV 440 that are visible to user 406 via video pass-through and / or optical see-through (e.g., real-world table 442).

[0072] In some examples, electronic device 402 detects video content 444 displayed by real-world TV 440 (e.g., using image sensors 414a-414c and / or one or more audio sensors (e.g., with reference to Figure 2In some examples, the electronic device 402 detects the attention (e.g., gaze) of the user 406 directed toward a location in the environment 404 corresponding to the real-world TV 440 (e.g., in addition to detecting the video content 444 displayed by the real-world TV 440). In some examples, the electronic device 402 identifies the video content 444 displayed by the real-world TV 440 through a database (e.g., the database includes information for identifying the video content 444 (e.g., images, videos, and / or audio prints associated with the video content 444)). For example, the database is stored in a memory by the electronic device 402, or is stored by a device (e.g., owned and / or operated by a third party) that communicates with the electronic device 402 (e.g., via a network). In some examples, the electronic device 402 receives information corresponding to the video content 444 from an application associated with the video content 444 (e.g., the application can be accessed on the real-world TV 440 and / or the electronic device 402). For example, electronic device 402 uses information corresponding to video content 444 to present content associated with video content 444 in environment 404 (e.g., reference FIG. 4F to FIG. 4I Virtual representations 412a-412e) are shown and described.

[0073] In some examples, in response to detecting a real-world user interface of real-world TV 440, video content 444, and / or user 406's attention directed toward video content 444 (e.g., for a threshold period of time (e.g., 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds)), electronic device 402 presents virtual element 420a in environment 404, such as Figure 4A In some examples, virtual element 420a can interact to present content (e.g., virtual content) associated with video content 444 in environment 404. For example, virtual element 420a can interact via user input corresponding to attention directed to virtual element 420a (e.g., user 406 directs gaze toward virtual element 420a (e.g., for a threshold time period, such as 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds)). Figure 4A As shown, from the current viewpoint of user 406, virtual element 420a is presented in the lower right corner of environment 404. In some examples, virtual element 420a is alternatively presented in a different area of ​​environment 404 (e.g., the lower left corner, upper corner, upper area, lower area, left area, right area, and / or empty space of environment 404) from the current viewpoint of user 406. In some examples, virtual element 420a is presented with different Figure 4AOne or more visual characteristics of those shown (e.g., brightness, color, saturation, sharpness, size, opacity of different sizes). In some examples, virtual element 420a is presented with one or more dynamic visual characteristics (e.g., changing brightness and / or color (e.g., flickering)). In some examples, virtual element 420a is periodically presented in environment 404 (e.g., every 1 second, 2 seconds, 5 seconds, 10 seconds, 30 seconds, 60 seconds, 90 seconds, or 120 seconds) for a threshold time period (e.g., 1 second, 2 seconds, 5 seconds, 10 seconds, 30 seconds, 60 seconds, 90 seconds, or 120 seconds). In some examples, presenting virtual element 420a in environment 404 includes providing audio output (e.g., through one or more output devices of electronic device 402). In some examples, when the electronic device 402 detects that the user 406 has not interacted with the virtual element 420a within a threshold time period (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, 100 seconds, or 120 seconds), the electronic device 402 stops presenting the virtual element 420a in the environment 404. In some examples, the electronic device 402 stops presenting the virtual element 420a in the environment 404 in response to input provided by the user 406 corresponding to a request to stop presenting the virtual element 420a in the environment 404 (e.g., the input includes an air gesture, an interaction with a selectable option included in the virtual element 420a, a speech input, and / or an actuation of a hardware button of the electronic device 402).

[0074] Figure 4B The user 406 is shown as being focused on the virtual element 420a. Figure 4B As shown, gaze 418 of user 406 is directed toward virtual element 420a. In some examples, gaze 418 is directed toward virtual element 420a for a threshold amount of time (e.g., 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds). In some examples, detecting user 406's attention directed toward virtual element 420a includes detecting an air gesture (e.g., air pinch, long air pinch (e.g., air pinch held for 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds), air tap, and / or air pointing) performed by a hand of user 406 (e.g., optionally simultaneously detecting gaze 418 directed toward virtual element 420a). In some examples, in response to detecting attention (e.g., gaze 418) directed toward virtual element 420a (e.g., for greater than a threshold amount of time), electronic device 402 presents content associated with video content displayed by real-world TV 440 in environment 404 (e.g., such as reference Figures 4F to 4G In some examples, in response to detecting that attention is directed toward virtual element 420a, electronic device 402 presents a second virtual element in environment 404 (e.g., as shown in reference Figure 4Cshown and described).

[0075] Figure 4C This example shows a response to Figure 4B An electronic device 402 is provided to present a virtual element 420b in an environment 404 based on input provided by a user 406 in the real world TV 440. In some examples, the virtual element 420b is presented by the electronic device 402 to confirm that the intention of the user 406 is to present content associated with the video content 444 displayed by the real world TV 440 in the environment 404. In some examples, the virtual element 420b includes one or more selectable options 460a and 460b. In some examples, the selectable option 460a can be selected to present content associated with the video content 444 in the environment 404. In some examples, the selectable option 460b can be selected to stop the presentation of the virtual element 420b in the environment 404 (for example, the selectable option 460b can be selected to abandon the presentation of the content associated with the video content 444 in the environment 404). In some examples, the selectable options 460a and 460b can be selected by user input including directing attention to the selectable option 460a or 460b. Figure 4C As shown, the electronic device 402 detects user input including gaze 418 directed toward selectable option 460b (e.g., for a threshold time period (e.g., 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds)). In some examples, the user input includes an air gesture (e.g., when the attention is directed toward selectable option 460a or 460b). Figure 4C As shown, an air gesture (e.g., an air pinch) is optionally performed by the user using hand 438. In some examples, in response to user input other than selection of selectable option 460b (e.g., user 406 provides verbal input, an air gesture, and / or actuates a hardware button of electronic device 402), electronic device 402 stops presenting virtual element 420b in environment 404. It should be appreciated that in some examples, electronic device 402 does not respond to a selection by a user. Figure 4B 406 in the present virtual element 420b. Figure 4B The electronic device 402 presents content associated with the video content 444 in the environment 404 (e.g., as shown in FIG. Figure 4F-4I It should be appreciated that in some examples, the electronic device 402 presents the virtual element 420b in the environment 404 in response to one or more criteria being met, the one or more criteria being different from when a user detects a virtual element 420b being present in the environment 404. Figure 4BFor example, the electronic device 402 does not present the virtual element 420a in the environment 404, and in response to the electronic device 402 detecting the real-world user interface of the real-world TV 440, the video content 444, and / or the user 406's attention directed toward the real-world TV 440, the electronic device 402 presents the virtual element 420b in the environment 404.

[0076] Figure 4D This example shows a response to Figure 4C The electronic device 402 presents content corresponding to the video content 416 in the environment 404 in response to the input provided by the user 406 in the environment 404. Figure 4C The selection of the selectable option 460b in the embodiment of the present invention abandons the presentation of the content in the environment 404. Figure 4D As shown, in response to Figure 4C In response to input provided by user 406 in the environment 404, electronic device 402 stops presenting virtual element 420b in environment 404. In some examples, such as Figure 4D As shown, the electronic device 402 abandons presenting the virtual element 420a in the environment 404 when stopping presenting the virtual element 420b in the environment 404 (for example, because the electronic device 402 Figure 4C The input provided by user 406 in environment 404 identifies that the intention of user 406 is not to present content associated with video content 444 in environment 404).

[0077] Figure 4E Illustrated Figure 4C , in which user 406 provides input corresponding to a selection of selectable option 460a (e.g., instead of selectable option 460b). Figure 4E As shown, the electronic device 402 detects the gaze 418 directed to the selectable option 460a (e.g., for a threshold time period (e.g., 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds)). In addition, as shown in FIG. Figure 4E As shown, electronic device 402 detects user 406 performing an air gesture (e.g., air pinch) with hand 438 (e.g., while user 406's attention is directed to selectable option 460a). Figure 4E The input provided by user 406 in environment 404 corresponds to a request to present content associated with video content 444 in environment 404. In some examples, in response to a user input different from an air gesture selecting selectable option 460a (e.g., user 406 providing a verbal input, providing an air gesture, and / or actuating a hardware button of electronic device 402), electronic device 402 presents content associated with video content 444 in environment 404.

[0078] In some examples, the electronic device 402 presents one or more types of content associated with the video content 444 in the environment 404. For example, the content associated with the video content 444 presented by the electronic device 402 includes subtitles, audio output, picture-in-picture, metadata, and / or visual graphics including supplementary information associated with the video content 444. In some examples, the type of content presented by the electronic device 402 in the environment 404 is determined based on user input. For example, the electronic device 402 presents a type of content based on one or more settings stored in a user profile (e.g., the one or more settings are set by the user 406 and correspond to a preferred type of content). For example, the electronic device 402 presents a type of content corresponding to an audio input (e.g., a verbal command provided by the user 406). For example, the electronic device 402 presents a type of content based on the position of the user 406 relative to the physical environment 408 (e.g., if the real-world TV 440 is in the field of view of the user 406, the electronic device 402 presents a visual graphic including supplementary information and abandons the picture-in-picture presentation of the video content 444).

[0079] Figure 4F This example shows a response to Figure 4E The virtual representation 412a is presented in the environment 404 based on the input provided by the user 406 in the environment. Figure 4F As shown, virtual representation 412a includes subtitle content presented in environment 404. In some examples, the subtitle content corresponds to subtitles of video content 444 (e.g., the subtitles are presented in environment 404 in synchronization with video content 444 displayed by real-world TV 440). For example, after electronic device 402 identifies video content 444 displayed by real-world TV 440, electronic device 402 retrieves subtitle information for video content 444 (e.g., from a database stored in memory, another electronic device in communication with electronic device 402, and / or an application (e.g., a streaming service application) associated with video content 444).

[0080] exist Figure 4F4, virtual representation 412a is presented in an area below real-world TV 440 in environment 404 from the current viewpoint of user 406. In some examples, subtitles are presented at a default location in environment 404 and / or at a location preferred by user 406 in environment 404 (e.g., user 406 sets a preferred location for virtual element 420a in one or more settings of electronic device 402 and / or in one or more settings of a user profile). In some examples, virtual representation 412a is presented in an empty space of environment 404 (e.g., an area of ​​environment 404 that does not include virtual and / or real-world objects). In some examples, virtual representation 412a is presented in a different type of content associated with video content 444 (e.g., a different type of content associated with reference to video content 444). Figures 4G to 4H In some examples, the electronic device 402 presents the virtual representation 412a when the video content 444 is included in the field of view of the user 406 (e.g., based on the video content 444 not being within the field of view of the user 406, the electronic device 402 abandons presenting the virtual representation 412a in the environment 404). In some examples, when subtitles are included in the video content 444 (e.g., subtitles corresponding to the video content 444 are displayed by the real-world TV 440 in the physical environment 408), the electronic device 402 abandons presenting the virtual representation 412a. In some examples, the electronic device 402 presents the virtual representation 412a based on the audio level (e.g., in decibels) detected from the real-world TV 440. For example, based on the detected audio level being less than a threshold amount, the electronic device 402 presents the virtual representation 412a in the environment 404.

[0081] Figure 4G This example shows a response to Figure 4E Virtual representations 412b-412d are presented in environment 404 based on input provided by user 406 in environment 404. Figure 4GAs shown, virtual representations 412b and 412c include metadata and / or supplemental information associated with video content 444. For example, video content 444 is a broadcast sporting event (e.g., a baseball game), and virtual representations 412b and 412c include metadata and / or information associated with the sporting event (e.g., virtual representation 412b includes a play sheet for the baseball game, and virtual representations 412b-412c include statistics for one or more players of the baseball game). In some examples, video content 444 is a movie or television show, and virtual representations 412b-412d include information associated with the movie or television show (e.g., cast, episode name / number, run time, and / or description). In some examples, video content 444 is live television, and virtual representations 412b-412d include a broadcast schedule and / or a channel guide. In some examples, the video content 444 is an online video from a video sharing service or social media application, and the virtual representations 412b-412d include the video name, description, playback controls, a playback progress bar, user profile information (e.g., associated with the creator of the video), and / or user comments (e.g., from one or more viewers of the video).

[0082] In some examples, content corresponding to metadata and / or supplemental information associated with video content 444 is presented via one or more virtual representations. For example, presenting the content includes presenting metadata and / or information in a virtual object and / or virtual container (e.g., metadata and / or information included in a virtual object presented in environment 404). Figure 4G, from the current viewpoint of the user 406, the virtual representations 412b-412d are presented in a lower area of ​​the environment 404. In some examples, the virtual representations 412b-412d are alternatively presented in different areas of the environment 404 (e.g., the lower left corner, the upper corner, the upper area, the lower area, the left area, the right area, and / or the empty space of the environment 404) from the current viewpoint of the user 406. In some examples, from the current viewpoint of the user 406, the content corresponding to the metadata and / or supplemental information associated with the video content 444 is presented in a default location (e.g., as defined by one or more system and / or device settings) or a user preferred location (e.g., as defined by one or more settings included in a user profile) in the environment 404. In some examples, the virtual representations 412b-412d are capable of being moved in the environment 404 from the current viewpoint of the user 406 by user input (e.g., the user input includes attention and / or an air gesture directed to one or more of the virtual representations 412b-412d). In some examples, the electronic device 402 stops presenting the virtual representations 412b-412d in response to the user input (e.g., corresponding to a request to stop presenting the virtual representations 412b-412d). For example, the virtual representations 412b-412d include one or more selectable options for stopping presenting the virtual representations 412b-412d, and the user input corresponds to a selection of the one or more selectable options. For example, the user input includes a verbal input (e.g., a voice command) and / or an air gesture.

[0083] Figure 4H 4 illustrates virtual representations 412b-412d presented in environment 404 when real-world TV 440 is not visible in environment 404 from the current viewpoint of user 406. As shown in top view 410, user 406 has moved from room 424b to room 424a in physical environment 408 (e.g., the movement of user 406 in physical environment 408 corresponds to a change in the current viewpoint of user 406 relative to environment 404). As a result of the movement of user 406, video content 444 displayed by real-world TV 440 is no longer visible to user 406. In some examples, such as Figure 4H As shown, electronic device 402 maintains presentation of virtual representations 412b-412d in environment 404. For example, virtual representations 412b-412d are presented in environment 404 independently of the position of real-world TV 440 in head-locked orientation and / or body-locked orientation. Figure 4G In contrast, virtual representations 412b-412d are Figure 4HThe virtual representations 412b-412d are alternatively presented in different locations in the environment 404 (eg, the electronic device 402 presents the virtual representations 412b-412d in an empty space in the environment 404 from the current viewpoint of the user 406).

[0084] Fig. 4I Illustrated Figure 4H An alternative example of , which includes presenting a virtual representation 412e in environment 404 in response to user 406 moving toward room 424a. In some examples, virtual representation 412e corresponds to a picture-in-picture presentation of video content 444. In some examples, based on user 406 and room 424a, the user can select a virtual representation 412e and select a virtual representation 412e. FIG. 4A to FIG. 4B and Figure 4E 4, the electronic device 402 determines that the user 406's intention is to present the virtual representation 412e in the environment 404 when the real-world TV 440 (e.g., and the video content 444 displayed by the real-world TV) is not visible. Fig. 4I As shown, in response to a change in the current viewpoint of the user 406 that causes the real-world TV 440 to no longer be visible in the environment 404, the electronic device 402 presents the virtual representation 412e in the environment 404. Optionally, the electronic device 402 provides an audio output 450 (e.g., by a Fig. 4I 412e) (e.g., the audio output is synchronized with the video content 444 presented by the virtual representation 412e). In some examples, when the real-world TV 440 is not included in the field of view of the user 406 and audio is detected from the real-world TV 440 (e.g., audio of a threshold amount (e.g., decibels) is detected), the electronic device 402 abandons and / or stops providing the audio output 450. In some examples, the virtual representation 412e may be presented in one or more locations in the environment 404, as described with reference to the virtual representations 414b-414d above (e.g., the virtual representation 412e is presented in a location based on a default system setting or a user setting, and / or the virtual representation 412e is capable of being moved in the environment 404 by user input). In some examples, the electronic device 402 stops presenting the virtual representation 412e in response to detecting a user input corresponding to a request to stop presenting the virtual representation 412e (e.g., including one or more characteristics of the request to stop presenting the virtual representations 412b-412d as described above).

[0085] Figure 4JThe electronic device 402 is illustrated as stopping presenting the virtual representation 412e in the environment 404 in response to the movement of the user 406 that makes the real-world TV 440 visible in the user's field of view. As shown in the top view 410, the user 406 has moved from the room 424a to the room 424b in the physical environment 408. Based on the movement of the user 406 from the room 424a to the room 424b, the video content 444 displayed by the real-world TV 440 is included in the field of view of the user 406. Based on the video content 444 displayed by the real-world TV 440 being visible to the user 406 and detectable in the environment 404, the electronic device 402 stops presenting the virtual representation 412e in the environment 404. In some examples, based on one or more user preferences, when the video content 444 is visible to the user 406, the electronic device 402 does not present content associated with the video content 444 (e.g., Figure 4J In some examples, when the electronic device 402 stops presenting the virtual representation 412e in the environment 404, one or more different types of content associated with the video content 444 may be displayed in the environment 404 (e.g., as shown in FIG. FIG. 4F to FIG. 4H For example, content associated with video content 444 that was presented in environment 404 before electronic device 402 presented virtual representation 412e is presented again in environment 404 by electronic device 402 (e.g., at the same location in environment 404 from the current viewpoint of user 406). In some examples, when virtual representation 412e stops being presented in environment 404, electronic device 402 stops providing audio output 450.

[0086] It should be understood that Figures 3A to 4JThe content (e.g., virtual representations 312a-312c and 412a-412e) presented in (e.g., in environments 304 and 404) is exemplary, and in some examples, the content is presented in association with a different real-world user interface from the depicted real-world user interface (e.g., a user interface of an object other than a microwave oven and / or a TV). For example, a user of an electronic device (e.g., having one or more characteristics of electronic device 302 and / or 402) interacts with a real-world washing machine, and the electronic device presents a timer in an environment corresponding to the timer displayed by the user interface of the real-world washing machine. For example, when the user is in an area of ​​the physical environment in which the washing machine is not located, the timer is presented by the electronic device. For another example, a user of the electronic device interacts with website content on a user interface of a laptop and / or mobile device, and in response to the detected interaction, the electronic device presents information associated with the website content in the environment. For example, the presented information includes supplemental information to the website content. For example, information is presented based on a user interface that does not include a laptop and / or mobile phone in the user's field of view (e.g., when the user moves to an area of ​​the physical environment where the laptop and / or mobile device is not located, the electronic device presents website content in the environment).

[0087] Figure 5 A flowchart illustrating an example process for presenting content associated with a user interface of an object included in a physical environment in an environment according to some examples of the present disclosure. In some examples, process 500 begins at an electronic device in communication with one or more displays and one or more input devices. In some examples, the electronic device is optionally similar to or corresponding to Figure 2 The head mounted display of the device 201. Figure 5 As shown, in some examples, at 502, the electronic device detects a user interface of a first object in the physical environment of a user of the electronic device via one or more input devices. Figure 3A As shown, electronic device 302 detects a real-world user interface of a real-world object in a physical environment (e.g., real-world user interface 316 of a real-world microwave oven 336 in physical environment 308). Figure 4A As shown, electronic device 402 detects a real-world user interface (eg, display) of a real-world TV (eg, real-world TV 440 ) in a physical environment (eg, physical environment 408 ).

[0088] In some examples, at 504, in response to detecting the user interface of the first object, based on determining that one or more first criteria are satisfied, the electronic device presents, via one or more displays, first content associated with the user interface of the first object in an environment independent of the location of the first object in the physical environment. Figure 3C As shown, the electronic device 302 presents a virtual representation 312a in the environment 304, the virtual representation including a corresponding Figure 3B A timer for a time interval displayed in the real-world user interface 316. For example, Figure 4F As shown, electronic device 402 presents virtual representation 412a corresponding to subtitles associated with video content 444 displayed by real-world TV 440 in environment 404. Figures 4G to 4H As shown, electronic device 402 presents virtual representations 412b-412d in environment 404 corresponding to metadata and / or information associated with video content 444 displayed by real-world TV 440. For example, Fig. 4I As shown, the electronic device 402 presents a virtual representation 412 e corresponding to a picture-in-picture presentation of video content 444 displayed by the real-world TV 440 .

[0089] In some examples, the one or more criteria include a criterion that is satisfied when the electronic device detects (e.g., and / or determines) that the user of the electronic device intends to interact with the user interface of the first object. FIG. 3A to FIG. 3B , the electronic device 302 determines that the user 306 intends to interact with the real-world user interface 316 by detecting the user 306's attention (e.g., gaze 318) directed toward the real-world user interface 316 and / or the user 306's physical interaction with the real-world user interface 316. For example, in Figure 4A , the electronic device 402 detects the user 406's intention to interact with the real-world TV 440 by detecting the video content 444 displayed by the real-world TV 440 and / or the user 406's attention (e.g., gaze 418) directed toward the real-world TV 440. For example, FIG. 4B to FIG. 4C As shown, electronic device 402 detects user 406's intention to interact with real-world TV 440 by detecting interaction with virtual element 420a and / or virtual element 420b. In some examples, presenting first content associated with a user interface of a first object in an environment independent of the location of the first object in the physical environment includes presenting the first content at a location in the environment that does not correspond to the location of the first object in the environment from the current viewpoint of the user of the electronic device. For example, Figure 3B As shown, when the real-world user interface 316 of the real-world microwave oven 336 is not visible in the environment 304 from the current viewpoint of the user 306, the electronic device 302 presents the virtual representation 312a in the environment 304 (e.g., when the user 336 is at a threshold distance from the real-world user interface 316 of the real-world microwave oven 306, the electronic device 302 presents the virtual representation 312a in the environment 304). Figures 4G to 4HAs shown, electronic device 402 presents virtual representations 412b-412d in environment 404 when real-world TV 440 is visible and when real-world TV 440 is not visible from the current viewpoint of user 406. For example, Fig. 4I As shown, the electronic device 402 presents the virtual representation 412 e in the environment 404 when the real-world TV 440 (eg, and the video content 444 ) is not visible from the current viewpoint of the user 406 .

[0090] It should be understood that process 500 is an example, and more, fewer, or different operations may be performed in the same or different order. In addition, the operations in the above process 500 are optionally performed by running an information processing device such as a general-purpose processor (e.g., as described in relation to Figure 2 The one or more functional modules in the dedicated chip and / or Figure 2 other components to achieve this.

[0091] Therefore, in accordance with the above, some examples of the present disclosure relate to a method, the method comprising: at an electronic device in communication with one or more displays and one or more input devices, detecting, via the one or more input devices, a user interface of a first object in a physical environment of a user of the electronic device. In some examples, the method further comprises: in response to detecting the user interface of the first object, presenting, via the one or more displays, first content associated with the user interface of the first object in an environment independent of the position of the first object in the physical environment based on determining that one or more first criteria are satisfied.

[0092] Additionally or alternatively, in some examples, presenting the first content in the environment includes presenting one or more virtual representations in a first area of ​​the environment relative to a current viewpoint of the user, where the first area relative to the current viewpoint of the user is independent of a position of the first object relative to the current viewpoint of the user.

[0093] Additionally or alternatively, in some examples, based on the user interface determining that a field of view of the environment from a current viewpoint of the user does not include the first object, one or more virtual representations are presented in the environment.

[0094] Additionally or alternatively, in some examples, one or more virtual representations are presented in the environment based on determining that the user interface of the first object is outside a threshold distance from a location corresponding to a current viewpoint of the user in the environment.

[0095] Additionally or alternatively, in some examples, the one or more first criteria include criteria that are satisfied when an intent of a user to interact with the user interface of the first object is detected when the user interface of the first object is detected.

[0096] Additionally or alternatively, in some examples, detecting the user's intent to interact with the user interface of the first object includes detecting input directed to the user interface of the first object.

[0097] Additionally or alternatively, in some examples, detecting a user's intention to interact with a user interface of a first object includes: after detecting input of the user interface directed to the first object, presenting a virtual element in the environment, the virtual element being interactive to present first content in the environment; and detecting input corresponding to the user interaction with the virtual element via one or more input devices.

[0098] Additionally or alternatively, in some examples, the one or more first criteria include a criterion that is satisfied when one or more settings associated with the first content stored in the user profile have a first state, and the criterion is not satisfied when one or more settings associated with the first content stored in the user profile have a second state different from the first state.

[0099] Additionally or alternatively, in some examples, the one or more first criteria include criteria that are satisfied when an operating mode is currently active, the operating mode including presenting content associated with one or more objects included in the physical environment in the environment.

[0100] Additionally or alternatively, in some examples, presenting the first content in the environment includes: during a first time period, storing information associated with the user interface of the first object without presenting a first virtual representation associated with the user interface of the first object in the environment; and during a second time period after the first time period, presenting one or more virtual representations associated with the user interface of the first object in the environment via one or more displays based on the stored information.

[0101] Additionally or alternatively, in some examples, presenting the first content in the environment includes presenting the first content in a system user interface of the electronic device.

[0102] Additionally or alternatively, in some examples, the user interface of the first object includes a timer set to a first time interval in the physical environment, and the first content presented in the environment includes a timer set to the first time interval.

[0103] Additionally or alternatively, in some examples, the user interface of the first object includes video content displayed via a physical display in the physical environment, and the first content includes information corresponding to the video content.

[0104] Additionally or alternatively, in some examples, presenting the first content in the environment includes presenting supplemental content associated with the video content.

[0105] Additionally or alternatively, in some examples, presenting the first content in the environment includes outputting audio associated with the video content to the user.

[0106] Additionally or alternatively, in some examples, presenting the first content in the environment includes presenting a picture-in-picture view of the video content.

[0107] Some examples of the present disclosure relate to an electronic device comprising: one or more processors; a memory; and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for executing any of the above methods.

[0108] Some examples of the present disclosure relate to a non-transitory computer-readable storage medium that stores one or more programs that include instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any of the above methods.

[0109] Some examples of the present disclosure relate to an electronic device comprising one or more processors, a memory, and a component for performing any of the above methods.

[0110] Some examples of the present disclosure relate to an information processing apparatus used in an electronic device, the information processing apparatus including a component for executing any one of the above methods.

[0111] For the purpose of explanation, the foregoing description is described by reference to specific examples. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Based on the above teachings, many modifications and variations are possible. The examples are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so as to enable other technical personnel in the field to best use the present disclosure and the various described examples with various modifications suitable for the specific purpose contemplated.

Claims

1. A method, comprising: At an electronic device in communication with one or more displays and one or more input devices: detecting, via the one or more input devices, a user interface of a first object in a physical environment of a user of the electronic device; as well as In response to detecting the user interface of the first object, based on determining that one or more first criteria are satisfied, presenting first content associated with the user interface of the first object via the one or more displays in an environment independent of the position of the first object in the physical environment.

2. A method according to claim 1, wherein presenting the first content in the environment includes presenting one or more virtual representations in a first area of ​​the environment relative to a current viewpoint of the user, wherein the first area relative to the current viewpoint of the user is independent of a position of the first object relative to the current viewpoint of the user.

3. The method of claim 2, wherein the one or more virtual representations are presented in the environment in accordance with the user interface determining that a field of view of the environment from the current viewpoint of the user does not include the first object.

4. The method of claim 2, wherein the one or more virtual representations are presented in the environment based on determining that the user interface of the first object is outside a threshold distance from a location corresponding to the current viewpoint of the user in the environment. 5 . The method of claim 1 , wherein the one or more first criteria include a criterion that is satisfied when an intent of the user to interact with the user interface of the first object is detected when the user interface of the first object is detected. The method of claim 5 , wherein detecting the user's intention to interact with the user interface of the first object comprises detecting input directed to the user interface of the first object.

7. The method of claim 6, wherein detecting the user's intention to interact with the user interface of the first object comprises: After detecting the input of the user interface directed to the first object, presenting a virtual element in the environment, the virtual element being interactive to present the first content in the environment; as well as Input corresponding to a user interaction with the virtual element is detected via the one or more input devices.

8. The method according to claim 1, wherein: The one or more first criteria include criteria that are satisfied when one or more settings associated with the first content stored in the user profile have a first state; and The criterion is not met when the one or more settings associated with the first content stored in the user profile have a second state different from the first state.

9. The method of claim 1, wherein the one or more first criteria comprise criteria that are satisfied when an operating mode is currently active, the operating mode comprising presenting, in the environment, content associated with one or more objects included in the physical environment.

10. The method of claim 1, wherein presenting the first content in the environment comprises: storing information associated with the user interface of the first object without presenting a first virtual representation associated with the user interface of the first object in the environment during a first time period; as well as During a second time period after the first time period, one or more virtual representations associated with the user interface of the first object in the environment are presented via the one or more displays based on the stored information. The method of claim 1 , wherein presenting the first content in the environment comprises presenting the first content in a system user interface of the electronic device.

12. The method of claim 1, wherein the user interface of the first object includes a timer set to a first time interval in the physical environment, and the first content presented in the environment includes a timer set to the first time interval.

13. The method of claim 1, wherein the user interface of the first object comprises video content displayed via a physical display in the physical environment, and the first content comprises information corresponding to the video content. The method of claim 13 , wherein presenting the first content in the environment comprises presenting supplemental content associated with the video content.

15. The method of claim 13, wherein presenting the first content in the environment comprises outputting audio associated with the video content to the user.

16. The method of claim 13, wherein presenting the first content in the environment comprises presenting a picture-in-picture view of the video content.

17. An electronic device, comprising: one or more processors; Memory; and One or more programs, the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 16.

18. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 16.

Images