System and method for image capture for augmented reality applications using peripheral objects

Abstract

The disclosure relates to systems and methods for image capture for augmented reality applications using peripheral objects. The first device optionally presents a representation of the input device at a first location in the three-dimensional environment having a location and orientation in the three-dimensional environment based on a location and orientation of the input device in the physical environment. While presenting the representation of the input device at the first position in the three-dimensional environment, the first device optionally detects an input to capture one or more images from a) a viewing angle corresponding to the input device and / or b) a viewing angle corresponding to the representation of the input device, and in response to detecting the input, displays the one or more images at the first position in the three-dimensional environment. The first device optionally captures the one or more images from a) the viewing angle corresponding to the input device and / or b) the viewing angle corresponding to the representation of the input device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 508,486, filed on June 15, 2023, and U.S. Patent Application No. 18 / 675,923, filed on May 28, 2024, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure generally relates to computer systems that provide computer-generated experiences, including but not limited to electronic devices that provide extended reality experiences via displays. Background Art

[0004] Cameras capture images or videos in a physical environment, which can be viewed. Summary of the invention

[0005] Some aspects of the present disclosure provide systems and methods for capturing one or more images and / or audio of an environment such as an extended reality environment (e.g., a virtual environment, an augmented reality environment, a mixed reality environment) or a physical environment. Some aspects of the present disclosure provide systems and methods for presenting one or more images and / or audio of a captured environment in the same environment in which the one or more images and / or audio were captured and / or in an environment different from the environment corresponding to the one or more images and / or audio captured. For example, some aspects of the present disclosure provide systems and methods for reproducing (e.g., displaying) a two-dimensional image of a three-dimensional environment (such as a three-dimensional virtual environment, a three-dimensional augmented reality environment, and / or a three-dimensional physical environment). Some aspects of the present disclosure provide systems and methods for presenting a two-dimensional image in the same environment in which the one or more images were captured and / or in an environment different from the environment corresponding to the one or more images and / or audio captured.

[0006] In some aspects, the first device communicates with the input device, and the first device displays the virtual environment from the perspective of a user immersed in the virtual environment. In some aspects, the view of the virtual environment displayed by the first device is based on the position and / or orientation of the first device in the physical environment. In some aspects, when displaying the virtual environment, the first device displays a representation of the input device (e.g., a three-dimensional virtual object), which has a position and orientation in the virtual environment based on the position and orientation of the input device in the physical environment. In some aspects, the first device captures one or more images of the virtual environment from the perspective of the representation of the input device in the virtual environment. In some aspects, the first device captures the audio of the virtual environment from the perspective of the representation of the input device in the virtual environment (e.g., a virtual space perspective). In some aspects, the first device displays one or more images of the virtual environment (e.g., one or more two-dimensional images) on the virtual object from the perspective of the virtual object in the virtual environment (optionally opposite to the perspective of the user immersed in the virtual environment).

[0007] A full description of these examples is provided in the drawings and detailed description, and it should be understood that the summary presented herein does not limit the scope of the present disclosure in any way. Additionally, the drawings and detailed description provide other examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] For a better understanding of the various examples described herein, reference should be made to the following detailed description and the following drawings. Throughout the drawings, like reference numerals generally refer to corresponding parts.

[0009] Figure 1 A block diagram illustrating an example architecture of a system according to some examples of the present disclosure.

[0010] Figure 2 A physical environment including a first device and an input device having a first position and orientation is shown according to some examples of the present disclosure, and a virtual environment including a representation of the input device having a first position and orientation based on the first position and orientation of the input device in the physical environment is shown.

[0011] Figure 3 Some examples according to the present disclosure are shown Figure 2 a physical environment in which the input device has a second position and orientation, and a representation including the input device is shown Figure 2 The virtual environment has a second position and orientation based on a second position and orientation of the input device in the physical environment.

[0012] Figure 4 Some examples according to the present disclosure are shown Figure 2 physical environment, and shows Figure 2A virtual environment of the invention, wherein the representation of the input device includes a representation of a display screen in a first position and orientation, the display screen displaying a view of the virtual environment from the perspective of the representation of the input device.

[0013] Figure 5 FIG. 1 shows an example of a first device from the perspective of a user according to some examples of the present disclosure. Figure 2 A view of the virtual environment.

[0014] Figure 6 Some examples according to the present disclosure are shown Figure 2 physical environment, and shows Figure 2 A virtual environment wherein the representation of the input device includes a representation of a display screen in a first position and orientation, the display screen displaying a view of the physical environment from the perspective of the input device.

[0015] Figure 7 Some examples according to the present disclosure are shown Figure 2 physical environment (but including a second person in the physical environment and a third person with a position and orientation that displays Figure 2 of the virtual environment), and shows Figure 2 A virtual environment in which a representation of the input device has a third position and orientation based on a third position and orientation of the input device in the physical environment.

[0016] Figure 8 Some examples according to the present disclosure are shown Figure 3 physical environment, and shows Figure 3 A virtual environment of the invention, wherein the representation of the input device includes a representation of a display screen in a second position and orientation, the display screen displaying an image of the augmented reality environment.

[0017] Fig. 9 is a flow chart illustrating operations and communications performed by a first device and / or an input device according to some examples of the present disclosure.

[0018] Fig.10 A diagram illustrating a method for capturing one or more images of an environment according to some examples of the present disclosure is shown. DETAILED DESCRIPTION

[0019] Some aspects of the present disclosure provide systems and methods for capturing one or more images and / or audio of an environment such as an extended reality environment (e.g., a virtual environment, an augmented reality environment, a mixed reality environment) or a physical environment. Some aspects of the present disclosure provide systems and methods for presenting one or more images and / or audio of a captured environment in the same environment in which the one or more images and / or audio were captured and / or in an environment different from the environment corresponding to the one or more images and / or audio captured. For example, some aspects of the present disclosure provide systems and methods for reproducing (e.g., displaying) a two-dimensional image of a three-dimensional environment (such as a three-dimensional virtual environment, a three-dimensional augmented reality environment, and / or a three-dimensional physical environment). Some aspects of the present disclosure provide systems and methods for presenting a two-dimensional image in the same environment in which the one or more images were captured and / or in an environment different from the environment corresponding to the one or more images and / or audio captured.

[0020] In some aspects, the first device communicates with the input device, and the first device displays the virtual environment from the perspective of a user immersed in the virtual environment. In some aspects, the view of the virtual environment displayed by the first device is based on the position and / or orientation of the first device in the physical environment. In some aspects, when displaying the virtual environment, the first device displays a representation of the input device (e.g., a three-dimensional virtual object), which has a position and orientation in the virtual environment based on the position and orientation of the input device in the physical environment. In some aspects, the first device captures one or more images of the virtual environment from the perspective of the representation of the input device in the virtual environment. In some aspects, the first device captures the audio of the virtual environment from the perspective of the representation of the input device in the virtual environment (e.g., a virtual space perspective). In some aspects, the first device displays one or more images of the virtual environment (e.g., one or more two-dimensional images) on the virtual object from the perspective of the virtual object in the virtual environment (optionally opposite to the perspective of the user immersed in the virtual environment).

[0021] Figure 1 A block diagram illustrating an example architecture of a system 201 according to some examples of the present disclosure is shown.

[0022] System 201 includes a first electronic device 220 and a second electronic device 230. The first electronic device 220 and the second electronic device 230 are communicatively coupled. The first electronic device 220 is optionally a head mounted device, a mobile phone, a smart phone, a tablet computer, a laptop computer, an auxiliary device that communicates with another device, and / or another type of portable, non-portable, wearable and / or non-wearable device. The second electronic device 230 is optionally similar or different in kind from the first electronic device. For example, when the first electronic device 220 is a head mounted device, the second electronic device 230 is optionally a mobile phone.

[0023] like Figure 1 As shown, the first electronic device 220 optionally includes the following (e.g., communicates with the following): various sensors (e.g., one or more hand tracking sensors 202, one or more position sensors 204, one or more image sensors 206A, one or more touch-sensitive surfaces 209A, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other audio sensors, etc.), one or more display generation components 214A, one or more speakers 216, one or more processors 218A, one or more memories 220A, and / or communication circuit system 222A. The second electronic device 230 optionally includes various sensors (e.g., one or more image sensors such as cameras 224B, one or more touch-sensitive surfaces 209B, and / or one or more microphones 228), one or more display generation components 214B, one or more processors 218B, one or more memories 220B, and / or communication circuit system 222B. One or more communication buses 208A and 208B are optionally used for communication between the above components of devices 220 and 230, respectively. First electronic device 220 and second electronic device 230 optionally communicate via a wired or wireless connection (eg, via communication circuitry 222A-222B).

[0024] The communication circuit systems 222A, 222B optionally include circuit systems for communicating with electronic devices, networks such as the Internet, intranets, wired networks and / or wireless networks, WiFi, cellular networks (e.g., 3G, 5G), and wireless local area networks (LANs). The communication circuit systems 222A, 222B optionally include circuit systems for using near field communication (NFC) and / or short-range communication such as A system of circuits for communicating.

[0025] Processors 218A, 218B optionally include one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 220A, 220B is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) storing computer-readable instructions, which are configured to be executed by processors 218A, 218B to perform the techniques, processes, and / or methods described below. In some examples, memory 220A, 220B may include more than one non-transitory computer-readable storage medium (optionally, its storage is configured to be executed by processors 218A, 218B to perform the techniques, processes, and / or methods 1000 described below). A non-transitory computer-readable storage medium may be any medium (e.g., excluding signals) that can tangibly contain or store computer-executable instructions for use by or in conjunction with an instruction execution system, device, and apparatus. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memory such as flash memory, solid-state drives, and the like.

[0026] In some examples, display generation components 214A, 214B include a single display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), or other type of display). In some examples, display generation components 214A, 214B include multiple displays. In some examples, display generation components 214A, 214B may include a display with touch capabilities (e.g., a touch screen), a projector, a holographic projector, a retinal projector, etc. In some examples, electronic devices 220 and / or 230 include touch-sensitive surfaces 209A and 209B for receiving user inputs such as tap inputs and swipe inputs or other gestures, respectively. For example, electronic device 230 optionally includes a touch-sensitive surface, while electronic device 220 does not include a touch-sensitive surface. In some examples, display generation components 214A, 214B and touch-sensitive surfaces 209A, 209B form a touch-sensitive display (e.g., a touch screen integrated with devices 220 and 230, respectively, or a touch screen that communicates with devices 220 and 230, respectively, outside devices 220 and 230).

[0027] Electronic devices 220 and / or 230 optionally include image sensors. Image sensors 206A and / or 206B optionally include one or more visible light image sensors (such as charge coupled device (CCD) sensors) and / or complementary metal oxide semiconductor (CMOS) sensors operable to obtain images of physical objects from a real-world environment. Image sensors 206A and / or 206B also optionally include one or more infrared (IR) sensors, such as passive IR sensors or active IR sensors, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. Image sensors 206A and 206B also optionally include one or more cameras 224A and 224B, respectively, configured to capture images of objects in the physical environment. Image sensors 206A and / or 206B also optionally include one or more depth sensors, which are configured to detect the distance of physical objects from device 220 / 230. In some examples, system 201 utilizes data from one or more depth sensors to identify and distinguish objects in the real-world environment from other objects in the real-world environment and / or determine the texture and / or topography of objects in the real-world environment.

[0028] In some examples, electronic devices 220 and / or 230 use CCD sensors, event cameras, and depth sensors in combination to detect the physical environment around devices 220 and / or 230. In some examples, image sensor 206A includes a first image sensor and a second image sensor. The first image sensor and the second image sensor work in tandem and are optionally configured to capture different information of physical objects in a real-world environment. In some examples, the first image sensor is a visible light image sensor, and the second image sensor is a depth sensor. In some examples, image sensor 206B includes two image sensors and is configured to perform functions similar to the above functions of image sensor 206A. In some examples, device 220 / 230 uses image sensor 206A to detect the position and orientation of device 220 / 230 and / or display generation component 214A / 214B in a real-world environment. For example, device 220 / 230 uses image sensor 206A / 206B to track the position and orientation of display generation component 214A / 214B relative to one or more fixed objects in a real-world environment. In some examples, system 201 uses image sensor 206A to detect the position and orientation of electronic device 230 relative to electronic device 220 .

[0029] In some examples, device 220 includes microphone 213 or other audio sensor. Device 220 uses microphone 213 to detect sounds from the user and / or the user's real-world environment. In some examples, microphone 213 includes a microphone array (multiple microphones) that optionally operate in concert to identify ambient noise or locate a sound source in the space of the real-world environment.

[0030] Device 220 includes a location sensor 204 for detecting the location of device 220 and / or display generating component 214A. For example, location sensor 204 may include a global positioning system (GPS) receiver that receives data from one or more satellites and allows device 220 to determine the absolute location of the device in the physical world.

[0031] Device 220 includes orientation sensor 210 for detecting orientation and / or movement of device 220 and / or display generation component 214A. For example, device 220 uses orientation sensor 210 to track changes in the position and / or orientation of device 220 and / or display generation component 214A, such as changes relative to physical objects in a real-world environment. Orientation sensor 210 optionally includes one or more gyroscopes, one or more accelerometers, and / or one or more inertial measurement units (IMUs).

[0032] In some examples, device 220 includes hand tracking sensor 202 and / or eye tracking sensor 212. Hand tracking sensor 202 is configured to track the position / location of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to an extended reality environment, relative to display generation component 214A, and / or relative to another defined coordinate system. Eye tracking sensor 212 is configured to track the position and movement of a user's gaze (more generally, eyes, face, or head) relative to the real world or extended reality environment and / or relative to display generation component 214A. In some examples, hand tracking sensor 202 and / or eye tracking sensor 212 are implemented together with display generation component 214A. In some examples, hand tracking sensor 202 and / or eye tracking sensor 212 are implemented separately from display generation component 214A.

[0033] In some examples, the hand tracking sensor 202 may use an image sensor 206A (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that captures three-dimensional information from the real world including one or more hands (e.g., one or more hands of a human user). In some examples, the hands may be resolved with sufficient resolution to distinguish fingers and their corresponding positioning. In some examples, one or more image sensors 206A are positioned relative to the user to define a field of view and interaction space of the image sensor 206A, in which the finger / hand position, orientation, and / or movement captured by the image sensor is used as input (e.g., to distinguish from the user's resting hand or other hands of other people in the real-world environment). Tracking fingers / hands for input (e.g., gestures, touches, taps, etc.) may be advantageous because it does not require the user to touch, hold, or wear any type of beacon, sensor, or other marker.

[0034] In some examples, eye tracking sensor 212 includes at least one eye tracking camera (e.g., an infrared (IR) camera) and / or an illumination source (e.g., an IR light source such as an LED) that emits light toward the user's eyes. The eye tracking camera can be pointed at the user's eyes to receive reflected IR light from the light source directly or indirectly from the eyes. In some examples, both eyes are tracked separately by corresponding eye tracking cameras and illumination sources, and focus / gaze can be determined by tracking both eyes. In some examples, one eye (e.g., the dominant eye) is tracked by corresponding eye tracking camera / illumination source.

[0035] Devices 220 / 230 and system 201 are not limited to Figure 1 201, may include fewer components, other components, or additional components in multiple configurations. In some examples, system 201 can be implemented in a single device. One or more people using device 220 / 230 or system 201 are optionally referred to herein as one or more users of the device. Additionally or alternatively, in some examples, electronic device 220 tracks a device that is not an electronic device (e.g., a cup, a box, a pen, or another device). For example, electronic device 220 optionally tracks an object that is not any of the illustrated components of device 230.

[0036] Figure 2 A physical environment 300 and a virtual environment 302 are shown in which a user 304 is immersed (e.g., fully immersed). The physical environment 300 (e.g., a room, an office) includes the user 304, a first device 306 (e.g., a head mounted display system, an extended reality (XR) display system), and an input device 308. The first device 306 optionally includes Figure 1 One or more features of device 220. Input device 308 optionally includes Figure 1One or more features of device 230. Figure 2 , the physical environment 300 also includes a chair 310 , windows 312 a and 312 b , a plant 316 on a floor 318 , and a wall 319 .

[0037] In some examples, input device 308 is an electronic device, such as a mobile phone, a laptop, a watch, a remote control. In some examples, input device 308 is a non-electronic object, such as a non-electronic block, a non-electronic cup, a non-electronic wallet, or another non-electronic object. Further discussion of input device 308 is provided below.

[0038] exist Figure 2, in the physical environment 300, the input device 308 has a position and orientation within a representative coordinate system 314a. Angle 314b optionally represents an angle corresponding to the orientation of the input device 308 in the physical environment 300 (e.g., relative to the horizontal plane of the physical environment 300 or another reference object (e.g., relative to the first device 306), an axis, or a plane). Vector 314c is optionally a vector representing a vector (e.g., a position vector) between the first device 306 and the input device 308, wherein the length (e.g., magnitude) of the vector represents the distance between the positions of the first device 306 and the input device 308 in the physical environment 300. In some examples, the input device 308 includes a sensor (e.g., a position sensor, an orientation sensor, an accelerometer, a gyroscope, an IMU, or other sensor) for detecting the position and orientation of the input device 308 in the physical environment 300; the first device 306 optionally receives a transmission of data corresponding to the sensor data detected via the input device 308, which is used to detect the position and orientation of the input device 308 in the physical environment 300 (optionally relative to the position and orientation of the first device 306 in the physical environment 300). In some examples, the first device 306 detects the position and orientation of the input device via a sensor (e.g., an image sensor) of the first device 306. For example, the position of the input device 308 relative to the first device 306 and / or the distance between the input device 308 and the first device 306 (as shown by vector 314c) is optionally detected by the first device 306. In some examples, the first device 306 detects the position and orientation of the input device 308 via a data stream (e.g., data packets) from the input device 308 (optionally via Bluetooth) or another suitable wired or wireless medium. In some examples, the first device 306 uses the spatial relationship of the first device 306 and the input device 308 (e.g., the distance between the first device 306 and the input device 308 in the physical environment 300 and the orientation of the input device 308 relative to the first device 306 (e.g., relative to an exterior forward direction of the first device 306)) to understand the position and orientation of the input device 308 relative to the first device 306. Additionally, the orientation of the input device 308 is optionally detected by the first device 306. Thus, the position and orientation of the input device 308 are optionally relative to a reference in the physical environment 300 (e.g., a spatial reference, an orientation reference, a gravity direction, or another type of reference), such as relative to the floor 318, gravity, and / or the first device 306.

[0039] exist Figure 2In , a user 304 is immersed in a virtual environment 302 (e.g., a virtual scene) via a first device 306. The virtual environment optionally includes a visual scene in which the user is fully or partially immersed, such as a campsite, a scene of the sky, outer space, and / or other suitable virtual scenes. In some examples, the virtual environment is a simulated three-dimensional environment that is optionally displayed in a three-dimensional environment instead of a representation of a physical environment (e.g., full immersion) or optionally simultaneously with a representation of the physical environment (e.g., partial immersion). Some examples of virtual environments include lake environments, mountain environments, sunset scenes, sunrise scenes, night environments, meadow environments, and / or concert scenes. In some examples, the virtual environment is based on a real physical location (such as a museum and / or aquarium), or based on a location designed by an artist. Therefore, displaying the virtual environment in a three-dimensional environment optionally provides the user with a virtual experience as if the user were physically located in the virtual environment. In Figure 2 , the user is optionally fully immersed in the virtual environment 302.

[0040] exist Figure 2 , the first device 306 displays a virtual environment 302 that includes a representation of an input device 320 and virtual objects, including a representation of an apple tree 322, a representation of apples 324 on a virtual ground, and a representation of a table 326. The representation of the input device 320 (e.g., a virtual object) is optionally a virtual camera (e.g., a representation of an image capture device) simulated by the first device 306, which can capture images or videos in the virtual environment 302 from the perspective of the representation of the input device 320. In some examples, the representation of the input device 320 is a representation of a camera, such as an analog camera, a digital camera, a film camera, a video camera, or another type of camera or image capture device. In some examples, the representation of the input device 320 includes a tripod and / or a selfie stick (optionally, regardless of whether the input device 308 includes a tripod and / or a selfie stick). In some examples, the appearance of the input device 308 in the physical environment 300 is similar or identical to the appearance of the representation of the input device 320 simulated by the first device 306 in the virtual environment 302. In some examples, the appearance of the input device 308 in the physical environment is different from the appearance of the representation of the input device 320 simulated by the first device 306 in the virtual environment 302. In an example, the input device 308 is optionally a cup, and the representation of the input device 320 in the virtual environment is optionally a camera. In other examples, the input device 308 is optionally a phone, and the representation of the input device 320 is a virtual representation of a phone or a virtual representation of a fixed camera.

[0041] exist Figure 2 , the representation of input device 320 is oriented to the image capture representation of apple 324, as shown by viewing boundaries 327a and 327b (e.g., representing the field of view of the representation of the aperture of the input device). Figure 2 302, in the virtual environment 302, the representation of the input device 320 has a position and orientation (e.g., in a representative virtual coordinate system 315a) that is based on the (real) position and orientation of the input device 308 in the physical environment 300 (e.g., based on the position and orientation of the input device 308 within the representative coordinate system 314a). For example, angle 315b optionally corresponds to angle 314b, but in the virtual environment 302, and vector 315c optionally corresponds to vector 314c, but in the virtual environment 302. In the virtual environment 302, angle 315b and / or vector 315c are optionally the same as angle 314b and / or vector 314c in the physical environment 300 (or are otherwise based on (e.g., are a function of) angle 314b and / or vector 314c in the physical environment 300). When the first device 306 detects a change in the position and / or orientation of the input device 308, the first device 306 optionally updates the position and / or orientation of the representation of the input device 320 in the virtual environment 302. Figure 2 In the example, the first device 306 is displayed in the virtual environment 302 with a first orientation and a first position (the first orientation and the first position are based on the input device 308 in Figure 2 Continuing with this example, in Figure 2 , the representation of the input device 320 is oriented to the image captured representation of the apple 324, and in response to the first device 306 detecting a change in the position and / or orientation of the input device 308, the change in the position and / or orientation of the input device 308 is determined to result in Figure 3 The first device 306 optionally updates the position and orientation of the representation of the input device 320 so that the representation of the input device 320 is oriented (e.g., reoriented) to the image captured representation of the apple tree 322, such as Figure 3 Shown is viewing borders 329a and 329b overlying the representation of apple tree 322. Thus, first device 306 optionally orients the position and / or orientation of the representation of input device 320 based on the position and / or orientation of input device 308.

[0042] Return to Figure 2, in virtual environment 302, the view of virtual environment 302 from the perspective of the representation of input device 320 includes the representation of apple 324, but does not include the representation of apple tree 322 and the representation of table 326 because the former (e.g., the representation of apple 324) is within viewing boundaries 327a and 327b corresponding to the representation of input device 320, while the latter (e.g., the representation of apple tree 322 and the representation of table 326) are outside viewing boundaries 327a and 327b corresponding to the representation of input device 320. Just as a physical camera optionally includes zoom functionality, the viewing angle boundaries of the representation of input device 320 may be zoomed in or out (e.g., via user input) and / or the magnification may be modified so that, for example, the image sensing view of the representation of input device 320 optionally includes the representation of apple tree 322 and table 326 in addition to the representation of apple 324 (e.g., zoomed out), or the image sensing view of the representation of input device 320 includes only a portion of the representation of apple 324 (e.g., zoomed in). It should be noted that the representation of the input device 320 optionally includes a focus mode, a zoom mode, a filter mode, a selfie mode, a landscape orientation, a portrait orientation, a simulated flash, a representation of a lens, a representation of film, and / or other digital, analog, and / or physical features of any type of physical camera. In some examples, the first device 306 displays a view of the virtual environment 302 from the perspective (e.g., position and orientation) of the representation of the input device 320 on the representation of the input device 320, such as Figure 4 As shown, the representation of input device 320 includes image data 329 that includes a representation of apple 330.

[0043] In some examples, the representation of input device 320 is as follows Figure 2When oriented and positioned as shown, the first device 306 optionally detects input to capture an image corresponding to a viewpoint of a representation of the input device 320. In some examples, the first device 306 detects capture input including a gaze and / or gesture directed to the input device 308 and / or the representation of the input device 320 via a sensor. For example, a gaze and / or gesture directed to a user interface of the representation of the input device 320, a virtual button displayed on the representation of the input device 320, and / or a physical touch-sensitive portion of the input device 308 (e.g., detection of a touch) or a button of the input device 308 (e.g., detection of pressing a button of the input device 308) is detected. The gaze and / or gesture optionally corresponds to the capture input. In some examples, a physical button on the input device 308 is mapped to a virtual button displayed by the first device 306 on the representation of the input device 320, so that a selection of a first physical button on the input device 308 corresponds to a selection of a first virtual button or user interface element on the representation of the input device 320. In some examples, one or more or all virtual buttons displayed by the first device 306 on the representation of the input device 320 do not correspond to physical buttons on the input device 308, but the virtual buttons can be triggered by input (e.g., touch, capacitive, press, or hold input) at the location of a physical button on the input device 308 or at a location without a physical button on the input device 308. In response to detecting the capture input, the first device 306 optionally captures a viewpoint of the virtual environment 302 from the perspective of the representation of the input device 320, such as Figure 5 shown.

[0044] Figure 5 Shows from Figure 2 From the perspective of the user 304 of the first device 306 Figure 2 A view of a virtual environment 302. Figure 5 In, from Figure 2 From the perspective of the user 304 of the first device 306 Figure 2 The view of the virtual environment 302 includes a representation of the input device 320 and virtual objects, including a representation of an apple tree 322, a representation of apples 324, a representation of a table 326, and a second representation of an apple 330 displayed on the representation of the input device 320. The representation of apples 324 and the representation of apples 330 correspond to the same representation of apples in the virtual environment 302, even though the first device 306 displays two different representations of the same apple at two different locations. Figure 5, first device 306 displays two representations of the same apple in virtual environment 302 because, in the view of virtual environment 302 from the perspective of user 304, the representation of input device 320 is positioned and oriented so that it does not completely obstruct visibility of the representation of apple 324. In some examples, based on the position and orientation of the representation of input device 320, first device 306 displays a second representation of apple 330 without displaying the representation of apple 324, optionally because, in the view of virtual environment 302 from the perspective of user 304, the representation of input device 320 completely obscures display of the representation of apple 324 (optionally such that first device displays the representation of apple 330 without displaying the representation of apple 324). Additionally, it should be noted that the representation of apple 330 is optionally a two-dimensional image of the representation of apple 324, which is optionally three-dimensional.

[0045] exist Figure 5 , the representation of the input device 320 (e.g., a virtual camera) includes a representation of an apple 330 on a virtual ground, but does not include representations of an apple tree 322 and a table 326 outside the coverage of the representation of the input device 320. Thus, the image is optionally captured (e.g., via the first device 306) from the perspective (e.g., position and orientation) of the representation of the input device 320 (e.g., a virtual camera) displayed by the first device 306 rather than from the perspective (e.g., position and orientation) of the first device 306 in the virtual environment; thus, the first device 306 optionally simulates a representation of the input device 320 that captures an image or video in the virtual environment and displays the image or video on the representation of the input device 320, which is visually similar to how a physical camera (including a user interface) can capture an image or video of a physical environment and display the image or video on the user interface of the physical camera. In some examples, the captured image includes an image orientation, such as a portrait, square, or landscape orientation. In some examples, the image orientation of the captured image is based on the image orientation functionality of the representation of input device 320, which is optionally based on or not based on the image orientation functionality of input device 308. In some examples, the representation of input device 320 includes more, the same number and the same kinds, or fewer image orientation types than input device 308. In some examples, the representation of input device 320 includes a single image orientation functionality, while input device 308 includes more than a single image orientation functionality (e.g., portrait and landscape). In some examples, the representation of input device 320 includes image orientation functionality, while input device 308 does not include any image orientation functionality for capturing images in physical environment 300.

[0046] Figure 6 Some examples according to the present disclosure are shown Figure 2 The physical environment and Figure 2A virtual environment wherein a representation of an input device 320 has a first position and orientation and includes a representation of a display screen at the first position and orientation, the display screen displaying a view of the physical environment from the perspective of the input device. Figure 6 In the physical environment 300 of FIG. 3 , the input device 308 optionally includes an image sensor (e.g., an image capture device or component) that captures image data of a portion of the physical environment 300 including the plant 316, as shown by viewing boundaries 336a and 336b. The input device 308 optionally transmits the image data to the first device 306, and in response, the first device 306 displays the image data captured by the input device 308 in the physical environment 300. Figure 6 , an input device 308 in the physical environment 300 drives the position and orientation of a representation of an input device 320 in the virtual environment 302 (and, optionally, an image displayed on the representation of the input device 320 in the virtual environment 302). For example, when the input device 308 is facing a first direction in the physical environment 300 (e.g., has a first orientation in the physical environment 300), the representation of the input device 320 is optionally facing a first direction in the virtual environment 302 (e.g., has a first orientation in the virtual environment 302) and includes image data of the physical environment 300 from the input device 308 facing the first direction in the physical environment 300; when the input device 308 is facing a second direction in the physical environment 300 that is different from the first direction (e.g., has a second orientation in the physical environment 300), the representation of the input device 320 is optionally facing a second direction in the virtual environment 302 that is different from the first direction in the physical environment 300 (e.g., has a second orientation in the virtual environment 302) and includes image data of the physical environment 300 from the input device 308 facing the second direction in the physical environment 300.

[0047] In addition, Figure 6 In , input device 308 optionally detects (e.g., captures) image data in physical environment 300 and then transmits the image data to first device 306, which image data includes a view of plant 316 in the example shown. Figure 6 In response to receiving a transmission of image data from input device 308, first device 306 displays image data 332 (e.g., images of plant 316 and floor 318) on a representation of input device 320 in virtual environment 302. Figure 6, although the representation of the input device 320 in the virtual environment 302 is optionally oriented to capture the representation of the apple 324 in the virtual environment 302, the representation of the input device 320 includes image data 332 from the input device 308 in the physical environment 300, but does not include image data from the virtual environment 302 (e.g., does not include the representation of the apple 324). Thus, while the user is immersed in the virtual environment, the first device 306 optionally uses image data from the physical camera as a window to the physical environment. Thus, while the user remains immersed in the virtual environment 302 (e.g., without having to exit the immersive experience), the first device 306 optionally utilizes one or more image sensors of the input device 308 to provide a controllable view of the physical environment 300. In some examples, the first device 306 includes an image sensor that captures images of the physical environment 300, and when the first device 306 displays the virtual environment 302, the first device 306 displays a view of the physical environment 300 on the representation of the input device 320 that is based on image data of the physical environment 300 captured via an image sensor of the first device 306 (e.g., an external image sensor) and driven by the position and orientation of the input device 308, optionally independent of whether the input device 308 includes an image sensor for capturing images of the physical environment 300. In some examples, the image data of the physical environment 300 that the first device 306 displays on the representation of the input device 320 is a combination of image data of the physical environment 300 from image data detected via the image sensor of the first device 306 and image data from image data detected via the image sensor of the input device 308.

[0048] In some examples, input device 308 includes a display component configured to display a user interface, and displays the user interface including an image of the portion of the physical environment. Figure 6 The input device 308 of the first device 306 optionally displays a user interface including the plant 316, while the first device 306 displays the image data of the plant 316 on the representation of the input device 320 in the virtual environment 302. In some examples, the input device 308 includes a display component that is configured to display the user interface and forgo displaying a user interface that includes an image of the portion of the physical environment that includes the plant 316, while the first device 306 displays the image data of the plant 316 on the representation of the input device 320. For example, Figure 6The input device 308 of the first device 306 optionally forgoes displaying a user interface including the plant 316, while the first device 306 displays the image data of the plant 316 on the representation of the input device 320 in the virtual environment 302 (even though the input device 308 optionally includes a display component). Thus, when the input device 308 includes a display component, the display component is optionally active or inactive (e.g., when inactive (e.g., turned off), the input device 308 optionally saves power), while the first device 306 displays a view of the physical environment 300 from the perspective of the input device 308 on the representation of the input device 320.

[0049] Figure 7 Some examples according to the present disclosure are shown Figure 2 physical environment (but including a second person in the physical environment and a third person with a position and orientation and showing Figure 2 of the virtual environment), and shows Figure 2 A virtual environment in which a representation of an input device has a third position and orientation. Figure 7 In the physical environment 300, the input device 308 optionally includes a sensor (e.g., a position, image, orientation sensor, or other physical space perception sensor as described above) and a display screen. Figure 7 In , the input device 308 communicates with the first device 306 and provides a view of the virtual environment 302 (e.g., two-dimensional image data) to a second person who does not see the display of the first device 306 (e.g., does not see the virtual environment 302 (e.g., a three-dimensional virtual environment)), which is displayed on the display of the input device 308. For example, the first device 306 optionally detects the position and orientation of the input device 308, and then transmits the image data of the virtual environment 302 displayed via the first device 306 based on the position and orientation of the input device 308 in the physical environment 300 (e.g., converted into the spatial relationship of the virtual environment 302 displayed via the first device 306). Figure 7 , since the view of the virtual environment 302 is directed toward the representation of the table 326, the first device transmits a corresponding view of the virtual environment 302, which includes an image 352 (e.g., a two-dimensional image) of the (three-dimensional) representation of the table 326 (and optionally, a virtual floor on which the representation of the table 326 stands), as shown by viewing boundaries 340a and 340b. Thus, in some examples, the first device 306 optionally drives a view of the virtual environment 302 (which is based on the position and orientation of the input device 308 relative to the first device 306) to the input device 308, and the input device 308 optionally presents the view on the display. These features optionally provide a user who is not immersed in the virtual environment 302 with a view of the virtual environment 302 displayed by the first device 306, and the view can be controlled using the position and orientation of the input device 308.

[0050] Figure 8 According to some examples Figure 3 physical environment, and shows Figure 3 A virtual environment of the invention, wherein the representation of the input device includes a representation of a display screen in a second position and orientation, the display screen displaying an image of the augmented reality environment.

[0051] exist Figure 8 In the physical environment 300 of FIG. 3 , the input device 308 (optionally including an active image sensor) is positioned and oriented to capture image data including a portion of a wall 319 of the physical environment, as shown by viewing boundaries 342 a and 342 b, while in the virtual environment 302, the first device 306 displays a representation of the input device 320 having a position and orientation in the virtual environment 302 based on the position and orientation of the input device 308 in the physical environment 300. Figure 8 The position and orientation (and zoom level) of the representation of input device 320 is configured to capture a representation of apple tree 322 in virtual environment 302, as shown by viewing boundaries 344a and 344b. Input device 308 optionally transmits image data to first device 306, and first device 306 optionally synthesizes the image data of first device 306 with image data in virtual environment 302 from the perspective of the representation of input device 320 to generate a representation of apple tree 322 in virtual environment 302. Figure 8 The augmented reality image is displayed on the representation of the input device 320. Figure 8 , the first device 306 displays a representation of a wall 319 of the physical environment 300 and a representation of an apple tree 322 of the virtual environment 302 on a representation of the input device 320. In some examples, the augmented reality image is synthesized at the first device. In some examples, the augmented reality image is synthesized at the input device. In some examples, the augmented reality image is synthesized at a device different from the first device and the input device. In some examples, the synthesized image includes one or more images in the physical environment (e.g., an image of a plant 316 or a chair 310) and a virtual background corresponding to the virtual environment (e.g., the background of the virtual environment 302). Thus, in some examples, the first device generates an augmented reality image based on the position and orientation of the input device in the physical environment.

[0052] Fig. 9 900 is a flowchart illustrating operations and communications optionally performed by the first device 306 and / or the input device 308 according to some examples.

[0053] exist Fig. 9In some examples, the first device optionally presents (902) an environment, such as a virtual reality environment or an augmented reality environment, via a display. While presenting the environment, the first device or the input device optionally initiates (904) a virtual camera mode. In some examples, various operating modes are presented via the first device (and / or the input device), such as a selfie mode, a window mode of the physical environment (such as a reference Figure 6 The input device is a window to the virtual environment and the user of the first device is immersed in the virtual environment (such as reference Figure 7 described), or another mode.

[0054] exist Fig. 9 In, the input device optionally transmits (906) position and orientation data to the first device. Fig. 9 , the first device optionally detects or otherwise obtains (908) position and orientation data of the input device from the input device, and displays (910) (optionally in response to detecting or obtaining the position and orientation data or optionally in response to initiating one of the virtual camera modes) a virtual camera (e.g., a representation of the input device 320) having a position and / or orientation based on the position and / or orientation data of the input device. Additionally or alternatively, the first device detects the position and orientation data of the input device via a sensor of the first device without transmitting the position and orientation data of the input device from the input device.

[0055] exist Fig. 9 In, optionally, after the input device transmits the position and orientation data to the first device, the input device transmits (912) updated position and orientation data to the first device. Fig. 9, the first device optionally detects (914) updated position and orientation data of the input device from the input device, and in response, displays (916) a virtual camera (e.g., a representation of the input device 320) having a position and / or orientation based on the updated position and / or orientation data of the input device. For example, the first device optionally updates the position of the virtual camera to be based on the updated position and / or orientation data of the input device (e.g., the first device optionally visually moves the virtual camera from a first position and orientation in the virtual environment based on the position and orientation data of the input device from block 908 to a second position and orientation in the virtual environment based on the updated position and orientation data of the input device). While displaying the virtual camera having a position and / or orientation based on the updated position and / or orientation data of the input device, the input device optionally detects (918) a capture input for capturing one or more images from the perspective of the virtual camera and transmits (920) the detection to the first device, or optionally the first device detects the capture input. In response to detecting the capture input detection, the first device optionally captures (922) a view of the environment from the perspective of the virtual camera, and optionally displays (924) the captured view of the environment. The first device optionally transmits (926) the captured view of the environment to an input device, which may receive (928), store, and / or display the captured view of the environment.

[0056] Fig.10 A diagram of a method 1000 for capturing one or more images (e.g., of an environment and / or including real objects or virtual objects) is shown. In some examples, the method 1000 is performed on a first device (such as a display) that is optionally in communication with a display and one or more input devices (including an input device). Figure 2 The first device 306 (including Figure 2 The display and input device 308)) is executed.

[0057] In some examples, the first device presents (1002) a three-dimensional environment via a display, such as Figure 2 Virtual environment 302.

[0058] In some examples, the first device presents (1004) via the display and in the three-dimensional environment a representation of the input device at a first location in the three-dimensional environment, such as Figure 2 In some examples, the representation of the input device 320 has a position and orientation in the three-dimensional environment based on the position and orientation of the input device in the physical environment of the first device, such as Figure 2The position and orientation of input device 320 within representative coordinate system 315a (and with reference to angle 315b and vector 315c in physical environment 302) is based on the position and orientation of input device 308 within representative coordinate system 314a (and with reference to angle 314b and vector 314c in physical environment 300).

[0059] In some examples, when presenting the three-dimensional environment from a first viewpoint in the three-dimensional environment and presenting a representation of an input device at a first position in the three-dimensional environment, the first device detects (1006) a perspective (e.g., position, orientation, and / or view) corresponding to the input device in the physical environment (e.g., from a first viewpoint in the three-dimensional environment). Figure 2 a) a perspective (e.g., position, orientation, and / or view) corresponding to a representation of the input device 308 in the three-dimensional environment 300 (e.g., from a perspective of the input device 308 in the physical environment 300) and / or b) a perspective (e.g., position, orientation, and / or view) corresponding to a representation of the input device in the three-dimensional environment (e.g., from a perspective of the input device 308 in the physical environment 30 Figure 2 The input of one or more images is captured by the first device (from the perspective of a representation of the input device 320 in the virtual environment 302 in the first device). The input is optionally received and / or transmitted from the input device (and to the first device) at the first device.

[0060] In some examples, in response to detecting input to capture one or more images from a) a perspective corresponding to an input device in a physical environment and / or b) a perspective corresponding to a representation of an input device in a three-dimensional environment, the first device captures (1008) one or more images from a) a perspective corresponding to an input device in a physical environment and / or b) a perspective corresponding to a representation of an input device in a three-dimensional environment. For example, in response to detecting input to capture one or more images from a) a perspective corresponding to an input device in a physical environment and / or b) a perspective corresponding to a representation of an input device, Figure 2 The first device 306 corresponds to the physical environment 300 from a) Figure 2 The viewing angle of the input device 308 and / or b) corresponds to Figure 2 One or more images are captured from the perspective of a representation of the virtual environment 302 by the input device 320 .

[0061] In some examples, the three-dimensional environment is a virtual reality environment in which a user of the first device is immersed, such as Figure 2 In some examples, one or more images are from b) corresponding to a representation of an input device in a three-dimensional environment (e.g., Figure 2 A virtual reality environment (e.g., a representation of an input device 320 in a virtual environment 302) from the perspective of Figure 2 One or more images of a virtual environment 302).

[0062] In some examples, the three-dimensional environment is a virtual reality environment in which a user of the first device is immersed, such as Figure 2 The virtual environment 302. In some examples, the one or more images are one or more images of the physical environment from a) a perspective corresponding to an input device in the physical environment, such as Figure 6 The image data 332 is shown, which is optionally shown when the user of the first device 306 is immersed in Figure 6 is captured in the virtual environment 302.

[0063] In some examples, the three-dimensional environment is an augmented reality environment. For example, the first device 306 optionally presents one or more virtual objects to the physical environment 300. In some examples, the one or more images are viewed from a) a perspective corresponding to an input device in the physical environment and b) a perspective corresponding to a representation of the input device in the three-dimensional environment (e.g., Figure 8 One or more images of an augmented reality environment (combination of a virtual environment 302 and a physical environment 300) such as Figure 8 The image data 360 is shown.

[0064] In some examples, based on determining that the position and orientation of the input device in the physical environment are the first position and the first orientation, the representation of the input device has the first position and the first orientation in the three-dimensional environment, such as with reference to Figure 2 As shown and described, and based on determining that the position and orientation of the input device in the physical environment are a second position and a second orientation that are different from the first position and the first orientation in the physical environment, the representation of the input device has a second position and a second orientation in the three-dimensional environment that are different from the first position and the first orientation in the three-dimensional environment, such as reference Figure 3 As shown and described.

[0065] In some examples, the input device includes a position sensor and / or an orientation sensor, such as described above with reference to Figure 2 The input device 308 is the sensor discussed above.

[0066] In some examples, the orientation of the representation of the input device is based on orientation data from an orientation sensor of the input device, such as due to Figure 3 The input device 308 and Figure 2 A change in orientation between the input device 308, Figure 3 The orientation of the representation of the input device 320 is different from Figure 2 The orientation of the representation of the input device 320.

[0067] In some examples, the position and orientation of the representation of the input device in the three-dimensional environment is based on image data of the input device detected by an image sensor of the first device, such as described above with reference to Figure 2 The first device 306 is discussed.

[0068] In some examples, the position and orientation of the representation of the input device in the three-dimensional environment is relative to the position and orientation of the first device, such as described above with reference to Figure 2 The first device 306 is discussed. For example, Figure 2 The origin of the coordinate system 314a is optionally at the first device 306 (e.g., relative to the first device) and / or the angle 314b and / or the vector 314c are optionally relative to Figure 2 The first device 306 such that the position and / or orientation of the input device 308 is relative to Figure 2 The position and / or orientation of the first device 306 is determined and / or optionally by Figure 2 The first device 306 determines.

[0069] In some examples, presenting a representation of the input device at the first location in the three-dimensional environment includes presenting a user interface that includes selectable modes for operating the representation of the input device, the selectable modes including a selectable selfie mode. Figure 2 The first device 306 optionally presents a selfie mode on the representation of the input device 320, which, when selected, initiates a process for the first device to display a view of the virtual environment corresponding to the virtual camera sensor on the front of the representation of the input device 320 on the representation of the input device 320. In some examples, the first device (e.g., Figure 2 The first device 306) in a virtual environment (e.g., Figure 2 In some examples, when operating in selfie mode, and in response to a capture input, the first device (e.g., Figure 2 The first device 306 captures the virtual environment (e.g., Figure 2 The first device may display an image of an avatar of a user (e.g., a virtual space representation of a user, optionally including characteristics of the user, such as a head, arms, glasses, legs, and / or other body parts) in a virtual environment 302 of the user's virtual environment, and the first device may display an image of an avatar of a user (e.g., a virtual space representation of a user, optionally including characteristics of the user, such as a representation of ... Figure 4 The image is displayed on a display device 320 of the input device 320). In some examples, the selectable mode includes an option to modify the representation type of the input device 320 (e.g., switching from a representation of a digital camera to a representation of an analog camera, optionally including functionality corresponding to a digital camera and / or an analog camera).

[0070] In some examples, via a first device (e.g., Figure 2 The first device 306) detects input to capture one or more images from a) a perspective corresponding to an input device and / or b) a perspective corresponding to a representation of the input device.

[0071] In some examples, the input is detected via the input device before the input is detected at the first device to capture one or more images from a) a perspective corresponding to the input device and / or b) a perspective corresponding to a representation of the input device. Figure 2 The sensor of the input device 308 optionally detects the input and then optionally sends Figure 2 The first device 306 transmits a notification to indicate that an input has been detected to capture one or more images from a) a perspective corresponding to the input device and / or b) a perspective corresponding to a representation of the input device.

[0072] In some examples, the first device captures audio associated with one or more images from the perspective of the first device (or a user of the first device) in a three-dimensional environment (e.g., a virtual space perspective). Figure 2 When the virtual environment 302, Figure 2 The first device 306 optionally presents audio associated with the virtual environment 302, the audio corresponding to audio detected at the position and / or orientation of the first device 306 in the virtual environment 302, which is optionally based on the position and / or orientation of the first device 306 in the physical environment 300. Continuing with this example, when the first device 306 detects an input to capture one or more images of the virtual environment 302, in addition to capturing one or more images of the virtual environment 302 from the perspective of the representation of the input device 320, the first device 306 optionally initiates a process to capture audio from the perspective of the corresponding position of the first device 306 in the virtual environment 302.

[0073] In some examples, the first device captures audio associated with one or more images from the perspective of a representation of an input device in a three-dimensional environment (e.g., a virtual space perspective). Figure 2 When the virtual environment 302, Figure 2 The first device 306 optionally presents audio associated with the virtual environment 302 that corresponds to spatial audio from the position and / or orientation of the representation of the input device 320 in the virtual environment 302. Continuing with this example, when the first device 306 detects an input to capture one or more images of the virtual environment 302, in addition to capturing one or more images of the virtual environment from the perspective of the representation of the input device 320, the first device optionally initiates a process to capture audio from the perspective of the representation of the input device 320 in the virtual environment 302.

[0074] In some examples, the first device saves the captured one or more images to the first device, an input device, or a remote server, such as the first device 306 saving the captured one or more images to the first device, an input device, or a remote server. Figure 5 The Apple 330 is saved to Figure 3 Input device 308.

[0075] In some examples, in response to detecting a connection with an input device or a first device (e.g., Figure 2 An input device 308 in the physical environment 300 or Figure 2 The method further comprises: performing a predefined interaction (e.g., a gaze trigger and / or an air gesture such as a pinch gesture) with a first device 306 of the present invention, performing a display and presenting in the three-dimensional environment a representation of the input device at a first location in the three-dimensional environment (1004), the representation having a first position in the three-dimensional environment based on the position and orientation of the input device in the physical environment (e.g., Figure 2 The position and orientation of the input device 308 in the physical environment 300 (e.g., Figure 2 representation of input device 320 in virtual environment 302).

[0076] In some examples, an input device (e.g., Figure 2 The input device 308) is a mobile phone.

[0077] In some examples, an input device (e.g., Figure 2 The input device 308) includes one or more image capture components, such as an image sensor or camera.

[0078] In some examples, presenting, via the display and in the three-dimensional environment, a representation of the input device at the first location in the three-dimensional environment includes presenting one or more images on the representation of the input device, such as on Figure 5 The representation of the input device 320 is presented on Figure 5 The Apple 330 is shown.

[0079] In some examples, one or more images (e.g., Figure 8 The representation of the input device 320 is presented on Figure 8 The image data 360) is a combination of: first image data detected by an image sensor of an input device; and Figure 8 In some examples, a first portion of image data (e.g., the first image data) of one or more images is processed at the input device, such as corresponding to a first image data of the input device 306 and a second image data that is presented on a display of the first device 306 and is different from the image data detected by the image sensor of the input device. Figure 8 The image sensor component of the input device 308 or the image data of the physical environment 300 captured by the image sensor component, and processing a second portion of the image data of the one or more images (e.g., second image data) at the first device, such as corresponding to Figure 8 The first device 306 or image data of the virtual environment 302 captured by the first device.

[0080] In some examples, the first device (including a display and / or one or more processors) can perform method 1000 and / or any of the additional and / or alternative operations disclosed. In some examples, a non-transitory computer-readable storage medium stores one or more programs including instructions that, when executed by a processor, cause the first device to perform method 1000 and / or any of the additional and / or alternative operations disclosed.

[0081] Various aspects of the disclosed examples, such as aspects of the examples shown in the figures and details in the present disclosure may be combined. In addition, although the disclosed examples are fully described with reference to the drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of the disclosed examples as defined by the appended claims.

Claims

1. A method comprising: At a first device in communication with a display and an input device: presenting a three-dimensional environment via the display; presenting, via the display and in the three-dimensional environment, a representation of the input device at a first location in the three-dimensional environment, wherein the representation of the input device has a location and orientation in the three-dimensional environment that is based on the location and orientation of the input device in the physical environment; while presenting the three-dimensional environment and presenting the representation of the input device at the first position in the three-dimensional environment, detecting input to capture one or more images from a) a perspective corresponding to the input device and / or b) a perspective corresponding to the representation of the input device; as well as In response to detecting the input, the one or more images are captured from a) the perspective corresponding to the input device and / or b) the perspective corresponding to the representation of the input device.

2. The method according to claim 1, wherein: The three-dimensional environment is a virtual reality environment, and The one or more images include one or more images of the virtual reality environment from b) the perspective corresponding to the representation of the input device in the three-dimensional environment.

3. The method according to claim 1, wherein: The three-dimensional environment is a virtual reality environment, and The one or more images include one or more images of the physical environment from a) the perspective corresponding to the input device in the physical environment.

4. The method according to claim 1, wherein: The three-dimensional environment is an augmented reality environment, and The one or more images include one or more images of the augmented reality environment from a) the perspective corresponding to the input device in the physical environment and b) the perspective corresponding to the representation of the input device in the three-dimensional environment.

5. The method according to any one of claims 1 to 4, wherein: Based on determining that the position and the orientation of the input device are a first position and a first orientation in the physical environment, the representation of the input device has a first position and a first orientation in the three-dimensional environment, and Based on determining that the position and the orientation of the input device are a second position and a second orientation in the physical environment that are different from the first position and the first orientation in the physical environment, the representation of the input device has a second position and a second orientation in the three-dimensional environment that are different from the first position and the first orientation in the three-dimensional environment.

6. The method according to any one of claims 1 to 5, wherein the input device comprises a position sensor and / or an orientation sensor.

7. A method according to any one of claims 1 to 6, wherein the orientation of the representation of the input device is based on orientation data from an orientation sensor of the input device.

8. The method of any one of claims 1 to 7, wherein the position and the orientation of the representation of the input device in the three-dimensional environment are based on image data of the input device detected by an image sensor of the first device.

9. A method according to any one of claims 1 to 8, wherein presenting the representation of the input device at the first position in the three-dimensional environment includes presenting a user interface, the user interface including optional modes for operating the representation of the input device, the optional modes including a selfie mode.

10. The method according to any one of claims 1 to 9, wherein: The input is detected via the first device.

11. The method according to any one of claims 1 to 10, wherein: The input is detected via the input device prior to the input being detected at the first device.

12. The method according to any one of claims 1 to 11, comprising: Audio associated with the three-dimensional environment is captured from the perspective of the first device in the three-dimensional environment, the audio being associated with the one or more images.

13. The method according to any one of claims 1 to 12, comprising: Audio associated with the three-dimensional environment is captured from the perspective of the representation of the input device in the three-dimensional environment, the audio being associated with the one or more images.

14. The method according to any one of claims 1 to 13, wherein: In response to detecting a predefined interaction with the input device or the first device, performing presentation, via the display and in the three-dimensional environment, of the representation of the input device at the first position in the three-dimensional environment, the representation having a position and orientation in the three-dimensional environment based on the position and the orientation of the input device in the physical environment.

15. The method according to any one of claims 1 to 14, wherein the input device is a mobile phone.

16. A method according to any one of claims 1 to 15, wherein the input device comprises an image capture device, and wherein capturing the one or more images of the environment from a) corresponding to the perspective of the input device in the physical environment comprises capturing the one or more images via the image capture device of the input device.

17. The method according to any one of claims 1 to 16, wherein: Presenting the representation of the input device at the first position in the three-dimensional environment via the display and in the three-dimensional environment includes presenting the one or more images on the representation of the input device.

18. The method according to any one of claims 1 to 17, wherein: The one or more images are a composite of: first image data detected by an image sensor of the input device, and second image data presented via the display and different from image data detected by the image sensor of the input device, processing a first portion of the image data of the one or more images at the input device, and A second portion of the image data for the one or more images is processed at the first device.

19. A first device, comprising: monitor; and one or more processors, The first device is configured to perform the method according to any one of claims 1 to 18.

20. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by a processor, cause a first device to perform the method according to any one of claims 1 to 18.

Images