Displaying augmented reality media feeds using media links

Abstract

The disclosure relates to displaying augmented reality media feeds using media links. Some examples of the present disclosure relate to systems and methods for communicating image sensor data to a computer system using one or more media links that allow the image sensor data to be displayed within a three-dimensional environment. In some examples, while presenting an electronic device that includes an image sensor (e.g., a camera or some other imaging device) communicatively coupled within the three-dimensional environment, the electronic device establishes a communication link with the image sensor using a visual code displayed on the electronic device, an indication is received to display image sensor data from the image sensor communicatively coupled to the electronic device. In some examples, when presenting a three-dimensional environment that includes a first predefined location within the three-dimensional environment, the electronic device facilitates a user to select an image sensor for display at the predefined location.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 586,578, filed on September 29, 2023, and U.S. Patent Application No. 18 / 882,622, filed on September 11, 2024, the contents of both applications are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present invention generally relates to a media link for displaying image sensor data within a three-dimensional environment. Background Art

[0004] Some electronic devices include cameras configured to capture images and / or video. Some electronic devices including one or more cameras are wired to a display to enable viewing of the camera feed. Summary of the invention

[0005] Some examples of the present disclosure relate to systems and methods for transmitting image sensor data (e.g., a media feed from a media sensor) to a computer system using one or more media links that allow the image sensor data to be displayed within a three-dimensional environment (e.g., a user viewing the three-dimensional environment and using the computer system uses one or more displays of the computer system to view the image sensor data within the three-dimensional environment, rather than viewing the image sensor data displayed on another electronic device visible within the three-dimensional environment). In one or more examples of the present disclosure, the system and method include receiving an instruction to display image sensor data from an image sensor communicatively coupled to the electronic device while presenting a three-dimensional environment that includes an image sensor (e.g., a camera or some other imaging device) communicatively coupled to the three-dimensional environment. In one or more examples, the instruction can be received in the form of a visual code displayed on the electronic device that is visible in the three-dimensional environment. In some examples, a user of the computer system selects the visual code (e.g., by looking at the visual code and / or providing input selecting the visual code). In response to the selection of the visual code, the computer system establishes a communication link with the image sensor, transmits the image sensor data via the communication link, and then displays the image sensor data in the three-dimensional environment. In this way, a user can view the image sensor feed using one or more displays of the computer system, rather than viewing the image sensor data displayed on a display of the electronic device, which results in better image fidelity. In one or more examples, the visual code may include Internet Protocol (IP) information and screen resolution information for establishing a communication link with the image sensor, for transmitting the image sensor data, and / or for displaying the image sensor data within a three-dimensional environment.

[0006] In one or more examples, the image sensor data is initially anchored (e.g., displayed) at a location within the three-dimensional environment corresponding to the location of the visual code or an electronic device displaying the visual code. In some examples, a user of the computer system may move the location of the display of the anchored image sensor data. In one or more examples, the updated location may be stored as the initial anchor location of the corresponding image sensor and image sensor data feed. Additionally or alternatively, the user of the computer system may move the location of the displayed image sensor data within the three-dimensional environment (e.g., change the position and / or orientation). Additionally or alternatively, the user may adjust the size of the display of the image sensor data within the three-dimensional environment (e.g., make the display of the image sensor data within the three-dimensional environment smaller or larger).

[0007] In one or more examples, the system and method include, while presenting a three-dimensional environment including a first predefined position within the three-dimensional environment, receiving an indication to display image sensor data within the three-dimensional environment. In some examples, the indication is provided in the form of a user of the computer system providing input to the computer system at an image sensor connection user interface to initiate display of the image sensor data. In one or more examples, the input at the image sensor connection user interface includes selecting an image sensor data feed from a list of image sensor data feeds displayed on the image sensor connection user interface. In one or more examples, and in response to receiving a selection of the image sensor data feed, the computer system establishes a communication link with the image sensor and begins to directly or indirectly receive image sensor data from the image sensor. In some examples, the computer system displays the image sensor data at a predefined position within the three-dimensional environment. Similar to the examples above, in some examples, the user is able to move the displayed image sensor data within the three-dimensional environment (e.g., change the position of displaying the image sensor data and / or within the three-dimensional environment where the image sensor data is displayed) and / or is able to adjust the size of the displayed image sensor data. In some examples of the present disclosure, the predefined position within the three-dimensional environment is associated with an electronic display visible in the three-dimensional environment. In some examples, the image sensor data may be initially displayed at a location in a three-dimensional environment, where the electronic display is located within the three-dimensional environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] For a better understanding of the various examples described herein, reference should be made to the following detailed description and the following drawings. Throughout the drawings, like reference numerals generally refer to corresponding parts.

[0009] Figure 1 An electronic device presenting an extended reality environment according to some examples of the present disclosure is shown.

[0010] Figure 2 A block diagram illustrating an example architecture of a device according to some examples of the present disclosure.

[0011] FIG. 3A to FIG. 3E An example system for displaying image sensor data in a three-dimensional environment according to some examples of the present disclosure is shown.

[0012] Figure 4 An example flow chart illustrating a method of displaying image sensor data in a three-dimensional environment according to examples of the present disclosure is shown.

[0013] FIG. 5A to FIG. 5E An example system for displaying image sensor data in a three-dimensional environment according to some examples of the present disclosure is shown.

[0014] Figure 6 An example flow chart illustrating a method of displaying image sensor data in a three-dimensional environment according to examples of the present disclosure is shown.

[0015] Figure 7 An example communication link between an image sensor and a computer system according to some examples of the present disclosure is shown. DETAILED DESCRIPTION

[0016] Some examples of the present disclosure relate to systems and methods for transmitting image sensor data (e.g., a media feed from a media sensor) to a computer system using one or more media links that allow the image sensor data to be displayed within a three-dimensional environment (e.g., a user viewing the three-dimensional environment on the computer system views the image sensor data from one or more displays of the computer system, rather than viewing the image sensor data displayed on another electronic device visible within the three-dimensional environment). In one or more examples of the present disclosure, the system and method include receiving an instruction to display image sensor data from an image sensor communicatively coupled to the electronic device while presenting a three-dimensional environment that includes an image sensor (e.g., a camera or some other imaging device) communicatively coupled to the electronic device within the three-dimensional environment. In one or more examples, the instruction can be received in the form of a visual code displayed on the electronic device that is visible in the three-dimensional environment. In some examples, a user of the computer system selects the visual code (e.g., by looking at the visual code and / or providing input selecting the visual code). In response to the selection of the visual code, the computer system establishes a communication link with the image sensor, transmits the image sensor data via the communication link, and then displays the image sensor data in the three-dimensional environment. In this way, a user can view the image sensor feed using one or more displays of the computer system, rather than viewing the image sensor data displayed on a display of the electronic device, which results in better image fidelity. In one or more examples, the visual code may include Internet Protocol (IP) information and screen resolution information for establishing a communication link with the image sensor, for transmitting the image sensor data, and / or for displaying the image sensor data within a three-dimensional environment.

[0017] In one or more examples, the image sensor data is initially anchored (e.g., displayed) at a location within the three-dimensional environment corresponding to the location of the visual code or an electronic device displaying the visual code. In some examples, a user of the computer system may move the location of the display of the anchored image sensor data. In one or more examples, the updated location may be stored as the initial anchor location of the corresponding image sensor and image sensor data feed. Additionally or alternatively, the user of the computer system may move the location of the displayed image sensor data within the three-dimensional environment (e.g., change the position and / or orientation). Additionally or alternatively, the user may adjust the size of the display of the image sensor data within the three-dimensional environment (e.g., make the display of the image sensor data within the three-dimensional environment smaller or larger).

[0018] In one or more examples, the system and method include, while presenting a three-dimensional environment including a first predefined position within the three-dimensional environment, receiving an indication to display image sensor data within the three-dimensional environment. In some examples, the indication is provided in the form of a user of the computer system providing input to the computer system at an image sensor connection user interface to initiate display of the image sensor data. In one or more examples, the input at the image sensor connection user interface includes selecting an image sensor data feed from a list of image sensor data feeds displayed on the image sensor connection user interface. In one or more examples, and in response to receiving a selection of the image sensor data feed, the computer system establishes a communication link with the image sensor and begins to directly or indirectly receive image sensor data from the image sensor. In some examples, the computer system displays the image sensor data at a predefined position within the three-dimensional environment. Similar to the examples above, in some examples, the user is able to move the displayed image sensor data within the three-dimensional environment (e.g., change the position of displaying the image sensor data and / or within the three-dimensional environment where the image sensor data is displayed) and / or is able to adjust the size of the displayed image sensor data. In some examples of the present disclosure, the predefined position within the three-dimensional environment is associated with an electronic display visible in the three-dimensional environment. In some examples, the image sensor data may be initially displayed at a location in a three-dimensional environment, where the electronic display is located within the three-dimensional environment.

[0019] Figure 1 A computer system 101 is shown that presents an extended reality (XR) environment (e.g., a computer-generated environment that optionally includes representations of physical and / or virtual objects) according to some examples of the present disclosure. In some examples, Figure 1 As shown, the computer system 101 is a head mounted display or other head mounted device configured to be worn on the head of a user of the computer system 101. Figure 2 An example of a computer system 101 is described by using an architectural block diagram of FIG. Figure 1 As shown, computer system 101 and desk 106 are located in a physical environment. The physical environment may include physical features, such as physical surfaces (e.g., floors, walls) or physical objects (e.g., desks, lights, etc.). In some examples, computer system 101 may be configured to detect and / or capture an image of the physical environment including desk 106 (shown in the field of view of computer system 101).

[0020] In some examples, such as Figure 1 As shown, the computer system 101 includes one or more internal image sensors 114a (eg, as described below with reference to FIG. 1 ) oriented toward the user's face. Figure 2101 ). In some examples, internal image sensor 114a is used for eye tracking (e.g., detecting the gaze of a user). Internal image sensor 114a is optionally arranged on the left and right portions of display 120 to enable eye tracking of the user's left and right eyes. In some examples, computer system 101 also includes external image sensors 114b and 114c facing outward from the user to detect and / or capture the physical environment of computer system 101 and / or the movement of the user's hands or other body parts.

[0021] In some examples, the display 120 has a field of view visible to the user (e.g., it may or may not correspond to the field of view of the external image sensors 114b and 114c). Because the display 120 is optionally part of the head-mounted device, the field of view of the display 120 is optionally the same as or similar to the field of view of the user's eyes. In other examples, the field of view of the display 120 may be smaller than the field of view of the user's eyes. In some examples, the computer system 101 may be an optical see-through device, in which the display 120 is a transparent or translucent display through which portions of the physical environment can be directly viewed. In some examples, the display 120 may be included in a transparent lens and may overlap all or only a portion of the transparent lens. In other examples, the computer system may be a video pass-through device, in which the display 120 is an opaque display configured to display an image of the physical environment captured by the external image sensors 114b and 114c.

[0022] In some examples, in response to a trigger, computer system 101 may be configured to display in the XR environment Figure 1 1 , the virtual object 104 is represented by a cube shown in the figure, which does not exist in the physical environment, but is displayed in the XR environment positioned on top of the real-world table 106 (or its representation). Optionally, in response to detecting the plane of the table 106 in the physical environment 100, the virtual object 104 can be displayed on the surface of the table 106 in the XR environment displayed via the display 120 of the computer system 101.

[0023] It should be understood that virtual object 104 is a representative virtual object and may include and render one or more different virtual objects (e.g., virtual objects with various dimensions, such as two-dimensional or other three-dimensional virtual objects) in a three-dimensional XR environment. For example, a virtual object may represent an application or user interface displayed in an XR environment. In some examples, a virtual object may represent content corresponding to an application and / or displayed via a user interface in an XR environment. In some examples, virtual object 104 is optionally configured to be interactive and responsive to user input (e.g., air gestures, such as air pinch gestures, air tap gestures, and / or air touch gestures), so that a user virtually touches, taps, moves, rotates, or otherwise interacts with virtual object 104.

[0024] In some examples, displaying an object in a three-dimensional environment may include interaction with one or more user interface objects in the three-dimensional environment. For example, initiation of display of an object in a three-dimensional environment may include interaction with one or more virtual options / indicators displayed in the three-dimensional environment. In some examples, when display of an object in a three-dimensional environment is initiated, a computer system may track the user's gaze as an input for identifying one or more virtual options / indicators as targets for selection. For example, gaze may be used to identify one or more virtual options / indicators as targets for selection using another selection input. In some examples, a virtual option / indicator may be selected using hand tracking input detected via an input device that communicates with the computer system. In some examples, an object displayed in a three-dimensional environment may be moved and / or reoriented in the three-dimensional environment based on movement input detected via an input device.

[0025] In the following discussion, a computer system that communicates with a display generation component and one or more input devices is described. It should be understood that the computer system optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand tracking device, an eye tracking device, a stylus, etc. In addition, as described above, it should be understood that the described computer system, display, and touch-sensitive surface are optionally distributed between two or more devices. Therefore, as used in the present disclosure, information displayed on or by a computer system is optionally used to describe information output by the computer system to be displayed on an independent display device (touch-sensitive or non-touch-sensitive). Similarly, as used in the present disclosure, input received on a computer system (e.g., touch input received on a touch-sensitive surface of a computer system, or touch input received on the surface of a stylus) is optionally used to describe input received on an independent input device, and the computer system receives input information from the independent input device.

[0026] The device typically supports a variety of applications, such as one or more of the following applications: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, a television channel browsing application, and / or a digital video player application.

[0027] Figure 2 A block diagram of an exemplary architecture of a computer system device 201 according to some examples of the present disclosure is shown. In some examples, the computer system 201 includes one or more computer systems. For example, the computer system 201 can be a portable device, an auxiliary device that communicates with another device, a head-mounted display, etc. In some examples, the computer system 201 corresponds to the above reference Figure 1 Computer system 101 is described.

[0028] like Figure 2 As illustrated in FIG. 1 , the computer system 201 optionally includes various sensors, such as one or more hand tracking sensors 202, one or more position sensors 204, one or more image sensors 206 (optionally corresponding to Figure 1 114a and / or external image sensors 114b and 114c in the embodiment of the present invention), one or more touch-sensitive surfaces 209, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other audio sensors, one or more body tracking sensors (e.g., torso and / or head tracking sensors), one or more display generation components 214 (optionally corresponding to Figure 1 1 ), one or more speakers 216, one or more processors 218, one or more memories 220, and / or communication circuits 222. One or more communication buses 208 are optionally used for communication between the above components of computer system 201.

[0029] The communication circuit 222 optionally includes circuits for communicating with a computer system, a network such as the Internet, an intranet, a wired network and / or a wireless network, a cellular network, and a wireless local area network (LAN). The communication circuit 222 optionally includes circuits for communicating using near field communication (NFC) and / or short range communication such as A circuit for communicating.

[0030] Processor 218 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 220 is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) storing computer-readable instructions, which are configured to be executed by processor 218 to perform the techniques, processes, and / or methods described below. In some examples, memory 220 may include more than one non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may be any medium (e.g., excluding signals) that can tangibly contain or store computer-executable instructions for use by or in combination with an instruction execution system, device, and apparatus. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include, but is not limited to, a magnetic storage device, an optical storage device, and / or a semiconductor storage device. Examples of such storage devices include magnetic disks, optical disks based on compact disc (CD), digital versatile disc (DVD) or Blu-ray technology, and persistent solid-state memory (such as flash memory, solid-state drives, etc.).

[0031] In some examples, display generation component 214 includes a single display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), or other type of display). In some examples, display generation component 214 includes multiple displays. In some examples, display generation component 214 may include a display with touch capabilities (e.g., a touch screen), a projector, a holographic projector, a retinal projector, a transparent or semi-transparent display, etc. In some examples, computer system 201 includes touch-sensitive surfaces 209 for receiving user inputs (such as tap inputs and swipe inputs or other gestures). In some examples, display generation component 214 and touch-sensitive surface 209 form a touch-sensitive display (e.g., a touch screen integrated with computer system 201 or a touch screen that communicates with computer system 201 outside computer system 201).

[0032] Computer system 201 optionally includes image sensor 206. Image sensor 206 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real-world environment. Image sensor 206 also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. Image sensor 206 also optionally includes one or more cameras configured to capture the movement of physical objects in the real-world environment. Image sensor 206 also optionally includes one or more depth sensors configured to detect the distance of the physical object from computer system 201. In some examples, information from one or more depth sensors may allow the device to identify an object in the real-world environment and distinguish it from other objects in the real-world environment. In some examples, one or more depth sensors may allow the device to determine the texture and / or topography of an object in the real-world environment.

[0033] In some examples, the computer system 201 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment around the computer system 201. In some examples, the image sensor 206 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor work in tandem and are optionally configured to capture different information of physical objects in the real-world environment. In some examples, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some examples, the computer system 201 uses the image sensor 206 to detect the position and orientation of the computer system 201 and / or the display generation component 214 in the real-world environment. For example, the computer system 201 uses the image sensor 206 to track the position and orientation of the display generation component 214 relative to one or more fixed objects in the real-world environment.

[0034] In some examples, computer system 201 includes microphone 213 or other audio sensor. Computer system 201 optionally uses microphone 213 to detect sounds from a user and / or the user's real-world environment. In some examples, microphone 213 includes an array of microphones (multiple microphones) that optionally operate in concert to identify ambient noise or locate a sound source in the space of a real-world environment.

[0035] Computer system 201 includes a location sensor 204 for detecting the location of computer system 201 and / or display generating component 214. For example, location sensor 204 may include a Global Positioning System (GPS) receiver that receives data from one or more satellites and allows computer system 201 to determine the absolute location of the device in the physical world.

[0036] Computer system 201 includes orientation sensor 210 for detecting orientation and / or movement of computer system 201 and / or display generation component 214. For example, computer system 201 uses orientation sensor 210 to track changes in the position and / or orientation of computer system 201 and / or display generation component 214, such as relative to physical objects in a real-world environment. Orientation sensor 210 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0037] In some examples, the computer system 201 includes a hand tracking sensor 202 and / or an eye tracking sensor 212 and / or other body tracking sensors, such as a leg tracking sensor, a torso tracking sensor, and / or a head tracking sensor. The hand tracking sensor 202 is configured to track the position / location of one or more parts of the user's hand, and / or the movement of one or more parts of the user's hand relative to the extended reality environment, relative to the display generation component 214, and / or relative to another defined coordinate system. The eye tracking sensor 212 is configured to track the position and movement of the user's gaze (more generally, eyes, face, or head) relative to the real world or extended reality environment and / or relative to the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented together with the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented separately from the display generation component 214.

[0038] In some examples, the hand tracking sensor 202 (and / or other body tracking sensors, such as leg, torso, and / or head tracking sensors) may use an image sensor 206 (e.g., one or more IR cameras, three-dimensional (3D) cameras, depth cameras, etc.) that captures three-dimensional information from the real world including one or more body parts (e.g., the hand, leg, torso, or head of a human user). In some examples, the hand may be resolved with sufficient resolution to distinguish fingers and their corresponding positioning. In some examples, one or more image sensors 206 are positioned relative to the user to define a field of view and interaction space of the image sensor 206, in which the finger / hand position, orientation, and / or movement captured by the image sensor is used as input (e.g., to distinguish from the user's resting hand or other hands of other people in the real-world environment). Tracking fingers / hands for input (e.g., gestures, touches, taps, etc.) may be advantageous because it does not require the user to touch, hold, or wear any type of beacon, sensor, or other marker.

[0039] In some examples, eye tracking sensor 212 includes at least one eye tracking camera (e.g., an infrared (IR) camera) and / or an illumination source (e.g., an IR light source such as an LED) that emits light toward the user's eyes. The eye tracking camera may be pointed toward the user's eyes to receive reflected IR light from the light source directly or indirectly from the eyes. In some examples, both eyes are tracked separately by corresponding eye tracking cameras and illumination sources, and focus / gaze may be determined by tracking both eyes. In some examples, one eye (e.g., the dominant eye) is tracked by one or more corresponding eye tracking cameras / illumination sources.

[0040] Computer system 201 is not limited to Figure 2 Rather than having components and configurations, the computer system 201 may include fewer components, other components, or additional components in a variety of configurations. In some examples, the computer system 201 may be implemented between two computer systems (e.g., as a system). In some such examples, each (or more) computer system may each include one or more of the same components discussed above, such as various sensors, one or more display generation components, one or more speakers, one or more processors, one or more memories, and / or communication circuits. One or more people using the computer system 201 are optionally referred to herein as one or more users of the device.

[0041] Attention is now directed to a three-dimensional environment including displayed image sensor data presented at a computer system (eg, corresponding to computer system 101 ), and in particular, systems and methods for receiving image sensor data at a computer system and directly displaying the image sensor data within a three-dimensional environment.

[0042] FIG. 3A to FIG. 3E An example system for displaying image sensor data in a three-dimensional environment according to some examples of the present disclosure is shown. In one or more examples, Figure 3A An example three-dimensional environment 302 rendered by computer system 101 is shown. In some examples, the three-dimensional environment 302 is an XR environment that includes some of the rendering of the physical environment. For example, the three-dimensional environment 302 includes a table 308, which is a physical table rendered in the three-dimensional environment. In one or more examples, objects on the table 308 may also include real-world elements. For example, in one or more examples, an electronic device 304 displayed within the three-dimensional environment 302 is a real-world electronic device that includes a display as shown. The electronic device is optionally a tablet computer, a desktop computer, a laptop computer, a television, a display monitor, etc. Similarly, the three-dimensional environment 302 includes an image sensor 306 that is a real-world image sensor. In one or more examples, an image sensor refers to any device that captures visual data, including but not limited to: a camera (e.g., red-green-blue, infrared, etc.), an acoustic imager, an optical imaging and ranging (lidar) sensor, or any electronic device capable of rendering an electronic image by collecting image data from a real-world object. Figure 3A In the example of FIG. 3 , the image sensor 306 is shown as a microscope, but the example of a microscope is not meant to be limiting, and the image sensor 306 may be implemented by any device incorporating the features described above.

[0043] In one or more examples, and as Figure 3A As shown in , the image sensor 306 is communicatively coupled to the electronic device 304. In some examples, the image sensor 306 can be communicatively coupled to the electronic device 304 using a wired connection (as shown), such as a universal serial bus (USB, such as USB-C), a high-definition multimedia interface (HDMI), a DisplayPort, or other interface. Additionally or alternatively, the image sensor 306 can be communicatively coupled to the electronic device 304 using one or more wireless connections (such as Bluetooth, Wi-Fi, a peer-to-peer network, etc.). In one or more examples, by being communicatively coupled to the image sensor 306, the electronic device 304 can display the image sensor data captured by the image sensor 306. As shown in Figure 3A, the electronic device 304 includes a display that can render image data provided by the image sensor 306. In one or more examples, a user of the computer system 101 can therefore view the image sensor data by viewing the display of the electronic device 304 via the display of the computer system 101. However, viewing the image sensor data in this manner (e.g., viewing the display of the electronic device 304 through the display of the computer system) can result in suboptimal conditions for viewing the image sensor data of the image sensor 306. For example, if the display of the electronic device 304 is refreshed asynchronously with the display of the computer system 101, the user may experience visual degradation of the image sensor data in the form of visual artifacts and other distortions. In addition, the display of the electronic device 304 can impose physical constraints on the display of the image sensor data due to its position in the three-dimensional environment 302, its resolution, orientation, etc. Therefore, in one or more examples, instead of displaying the image sensor data through the display of the electronic device 304, or in addition to displaying the image sensor data through the display of the electronic device 304, the computer system 101 can directly receive the image sensor data and display the image sensor data in the three-dimensional environment, as described in further detail below. By directly acquiring image sensor data and displaying it in a three-dimensional environment, computer system 101 may eliminate and / or mitigate visual degradation associated with displaying image sensor data on a display of electronic device 304 as described above.

[0044] In one or more examples, the electronic device 304 may include one or more applications stored in a memory. When the electronic device 304 detects that an image sensor (such as image sensor 306) is communicatively coupled to the device, the one or more applications may generate a visual code 310. Additionally or alternatively, the visual code 310 may be displayed on the electronic device 304 in response to one or more user inputs at the electronic device 304. Additionally or alternatively, the visual code 310 may be permanently fixed to the device 304 (e.g., using a label on the device itself) and therefore no display is required to display the visual code. In one or more examples, the visual code 310 may include information about the image sensor 306 embedded in the code, so that when the computer system 101 accesses the visual code 310 (captures an image of the visual code 310), the computer system 101 can establish a communication link with the image sensor 306. Establishing a communication link enables computer system 101 to receive image sensor data from image sensor 306, which computer system 101 can use to display the feed of image sensor 306 directly (e.g., independent of the display of electronic device 304) within three-dimensional environment 302. In one or more examples, visual code 310 can be implemented as a Quick Response ("QR") code, DotCode, App Clip code, barcode, data matrix, etc. Additionally or alternatively, visual code 310 can be implemented as a custom visual code that can be specifically customized to provide connection information to computer system 101 so that computer system 101 can directly receive image sensor data. In some examples, visual code 310 can be used to pair computer system 101 with electronic device 304 so that computer system 101 indirectly receives image sensor data from image sensor 306 (e.g., in the case where image sensor 306 does not have wireless communication capabilities).

[0045] In one or more examples, the visual code 310 may include information embedded within the code that provides information to the computer system 101 to establish a communication link with the image sensor 306 so that the computer system 101 can render / display the image sensor data within the three-dimensional environment 302. Additionally or alternatively, in one or more examples, the visual code 310 may include information embedded within the code that provides information to the computer system regarding specifications of the image sensor data so that the computer system 101 can render / display the image sensor data within the three-dimensional environment 302. For example, regarding information regarding establishing a communication link with the image sensor 306, the visual code 310 may include communication link information, such as Internet Protocol (“IP”) address information or other types of information that may allow the computer system 101 to (directly or indirectly) establish a communication link with the image sensor 306. In some examples, the visual code 310 may include information related to the image sensor data (such as aspect ratio or screen resolution), or information related to the display of the electronic device 304 (such as size, screen resolution), or location information that the computer system 101 may use to display the image sensor data within the three-dimensional environment 302.

[0046] In one or more examples, once the visual code 310 is displayed on the display of the electronic device 304, the user of the computer system 101 can initiate the process of displaying the image sensor data from the image sensor 306 by providing an instruction to the computer system 101 to display the image sensor data (e.g., using one or more parts of their body). For example, in one or more examples, the computer system 101 can be responsive to detecting that the user's gaze 312 (and / or the orientation of the computer system) is directed toward the image sensor data. Figure 3B The location within the three-dimensional environment 302 at which the visual code 310 is located is shown, optionally for a threshold duration and optionally in addition to another input (e.g., a gesture, a voice command, etc.), to initiate a process of displaying image sensor data from the image sensor 306. In some examples, a user can set a default location within the three-dimensional environment 302 to display image sensor data (before displaying the image sensor data), and as discussed below, the location at which the image sensor data is displayed can be changed after the image sensor data has been displayed in the three-dimensional environment. In some examples, the computer system 101 can automatically initiate a process of displaying the image sensor data without requiring input from the user.

[0047] exist Figure 3BIn the example of , in response to detecting a user's gaze 312 in the direction of a location within the three-dimensional environment 302 where a visual code 310 is located (e.g., on a display of an electronic device 304), the computer system 101 can display an image sensor activation user interface 314 for receiving user input confirming that the user of the computer system 101 wants to display image sensor data from the image sensor 306. In some examples, the image sensor activation user interface 314 includes selectable options that the user can select. In response to detecting a selection of the selectable option of the image sensor activation user interface 314 (e.g., by detecting a user's hand 316 performing an air gesture such as an air pinch while also detecting a user's gaze 312 directed at the user interface 314), the computer system 101 establishes a communication link to the image sensor 306 (e.g., using information embedded in the visual code 310) and displays the image sensor data from the image sensor 306, such as the image sensor data from the image sensor 306. Figure 3C shown.

[0048] exist Figure 3C In the example of , and after establishing a communication link with the image sensor 306 as described above (and in further detail below), the computer system 101 receives image sensor data from the image sensor 306 and displays the image sensor data as an image sensor data feed 318 within the three-dimensional environment 302. In one or more examples, the image sensor data feed 318 provides a real-time or near real-time (e.g., less than a threshold latency, such as 1 ms, 10 ms, 50 ms, 100 ms, etc.) display of image data being collected by the image sensor 306. Thus, in one or more examples, when the image sensor 306 is capturing video, the feed 318 is represented by the computer system 101 in the three-dimensional environment 302 as a video feed. In one or more examples, when the image sensor 306 is capturing still images, the feed 318 is represented by the computer system 101 in the three-dimensional environment 302 as a still image feed.

[0049] In one or more examples, the computer system 101 renders and / or displays the feed 318 according to one or more display specifications (such as size, screen resolution, or position) associated with the display of the electronic device or one or more image specifications associated with the image sensor data of the image sensor 306. For example, and as described above, the visual code 310 may have one or more specifications embedded with the code, such as the aspect ratio of the image data and the resolution of the image sensor data. Additionally or alternatively, the image specifications may be embedded in the actual image sensor data received from the image sensor 306 at the computer system 101 or sent separately from the electronic device to the computer system. In the case where the image specifications are embedded in the image sensor data, the computer system 101 can extract the image specifications from the image sensor data and use them to render and display the image sensor data within the three-dimensional environment 302. In some examples, rendering / displaying the image sensor data based on the image specifications or display specifications may include displaying the feed 318 according to the aspect ratio and / or resolution specified in the image specifications or according to the size, screen resolution, or position specified in the display specifications.

[0050] In one or more examples, and as Figure 3CAs shown, feed 318 is initially displayed at the location of the display of electronic device 304, so that the feed is superimposed on the display, thereby giving the appearance that the image sensor data is being displayed by electronic device 304. In some examples, computer system 101 stops displaying visual code 310 in the three-dimensional environment while displaying the image sensor data superimposed on the display of electronic device 304. Thus, in one or more examples, the screen resolution and aspect ratio provided in the image specification can be used to display feed 318 in the same manner as if electronic device 304 is displaying feed 318. In some examples, display specifications (described above) can also include positioning and size information, which can be used to position and resize display feed 318 within three-dimensional environment 302. Additionally or alternatively, computer system 101 can use visual code 310 as an "anchor" for positioning display feed 310 within three-dimensional environment 302 (e.g., a fixed position within three-dimensional environment 302). However, in examples where computer system 101 receives image sensor data and displays the feed directly, a user of the computer system can have an improved view of the image sensor data, as compared to examples where the image sensor data is actually displayed by electronic device 304 and viewed through a display of computer system 101. Additionally or alternatively, a user of the computer system can have greater control and flexibility over how the image sensor data is displayed. For example, in one or more examples, in response to detecting a user gaze 312 directed toward feed 318, and upon detecting a user's hand 316 performing an air gesture, computer system 101 can move feed 318 to a different location within three-dimensional environment 302 commensurate with the movement of the user's hand, such as Figure 3D shown.

[0051] In one or more examples, and as Figure 3DAs shown, image sensor data feed 318 has been moved from being displayed at or above the display of electronic device 304 to being displayed in a second position within the three-dimensional environment. Optionally, the display of electronic device 304 can be turned off. In some examples, electronic device 304 can display feed 318 at electronic device 304 and at a second position at the same time (e.g., if other users are viewing electronic device 304 without a computer system (such as computer system 101)). In some examples, when image sensor data feed 318 has been moved to the second position, electronic device 304 can continue to display visual code 310, thereby providing a visual anchor for computer system 101. The second position where feed 318 is displayed by computer system 101 is commensurate with the movement of the user's hand. Therefore, the distance and position from the original display position to the new display position of feed 318 are based on the direction and amount of movement of the user's hand 316. Because computer system 101 now receives image sensor data from image sensor 306 (directly or indirectly), the user can customize the position where feed 318 is displayed within three-dimensional environment 302. In examples where the feed is displayed on the electronic device 304 without the computer system having access to the underlying image sensor data, such customization would not be possible because the image sensor data would be "locked" to the position of the electronic device 304. In one or more examples, customization of the feed 318 is not limited to simply moving the position of the feed, but can also include adjusting the size of the feed 318. For example, upon detecting the user's gaze 312 and when the user performs an air gesture with their hand 316 (such as expanding the distance between their thumb and index finger), the computer system 101 can enlarge the size of the feed 318, such as Figure 3E As shown. The examples of using gaze and performing air gestures to enlarge the size of feed 318 are provided as examples and should not be considered limitations of the present disclosure. In one or more examples, input for modifying the size of feed 318 can come from other types of input (e.g., gaze without gesture, gesture without gaze, input using one or more user interfaces or affordances, button presses, touch input to a touch screen or touch sensor panel, voice input, etc.).

[0052] In one or more examples, and as Figure 3EAs shown, feed 318 has been resized (in this case, made larger) based on the movement of user's hand 316 (and specifically, the amount of movement of the user's thumb and index finger). In some examples, a user of computer system 101 can similarly cause feed 318 to also be reduced in size (e.g., by bringing their thumb and index finger together), or by using other inputs including receiving input through one or more user interfaces or affordances. In one or more examples, computer system 101 resizes feed 318 based on the movement of user's hand 316 and also based on the image specifications provided to computer system 101 (as described above). For example, when zooming in on feed 318, computer system 101 can display feed 318 at a larger size, but still maintain the aspect ratio and screen resolution associated with feed 318.

[0053] In some examples, and using the communication link established above, a user may send commands to image sensor 306. For example, a user may use one or more gestures to change the zoom of image sensor 306 or change the positioning of an optical lens of image sensor 306. In some examples, a user may initiate a command to image sensor 306 by viewing image sensor 306 or an image sensor data feed while performing a gesture.

[0054] Figure 4 An example flow chart illustrating a method or process for displaying image sensor data in a three-dimensional environment according to an example of the present disclosure is shown. In one or more examples, Figure 4 The process 400 shows the FIG. 3A to FIG. 3E . In one or more examples, one or more displays (e.g., of computer system 101) present a three-dimensional environment including an electronic device having a display (e.g., electronic device 304 displayed in three-dimensional environment 302). The electronic device is communicatively coupled to an image sensor (e.g., electronic device 304 is communicatively coupled to image sensor 306). While presenting the three-dimensional environment including the electronic device, computer system 101 receives (402) an instruction to display image sensor data from the image sensor communicatively coupled to the electronic device. In one or more examples, the three-dimensional environment including the electronic device is presented to the computer system 101 via a visual code (e.g., Figure 3AThe computer system 101 may include a display device 304 (e.g., a visual code 310 shown) that provides an indication of displaying image sensor data from the image sensor to the computer system 101, the visual code being selectable by a user (e.g., by the user gazing at the visual code and / or performing an air gesture). Additionally or alternatively, the computer system 101 receives an indication from a visual code that appears (e.g., on a display of the device 304) in the three-dimensional environment without requiring the user to select the visual code. In some examples, in response to receiving a selection from the user or otherwise detecting and / or selecting a visual code, the computer system 101 optionally displays a user interface that the user can select to initiate display of the image sensor data within the three-dimensional environment.

[0055] In some examples, in response to receiving an indication to display image sensor data from an image sensor communicatively coupled to the electronic device, computer system 101 establishes (404) a communication link to the image sensor communicatively coupled to the electronic device. In one or more examples, computer system 101 uses connection information embedded in the visual code to establish a communication link with the image sensor. For example, the visual code includes IP information embedded therein that provides an IP address at which the image sensor data can be accessed. Therefore, in one or more examples, computer system 101 uses the IP information to establish a communication link with the image sensor to obtain access to the image sensor data.

[0056] In one or more examples, after computer system 101 establishes a communication link with the image sensor, computer system 101 receives (406) image sensor data via the communication link and displays (408) image sensor data (such as a Figure 3C 318 in the image sensor data feed. In some examples, the image sensor data feed may be initially displayed on the electronic device (so that it appears as if the image sensor data is being displayed on a display of the electronic device). In some examples, the computer system 101 uses display information embedded in the visual code or sent as part of the image sensor data to display the image sensor data in a three-dimensional environment (e.g., via the same communication link or a different communication link). For example, the display information may include an aspect ratio and screen resolution associated with the image sensor data and a location in three-dimensional space where the image sensor data is to be displayed. In one or more examples, and using the display information, the computer system 101 displays the image sensor data according to the display information (e.g., image specifications) within the three-dimensional environment.

[0057] It should be understood that process 400 is an example and more, fewer, or different operations may be performed in the same or different order. For example, establishment of a communication link and / or display of image sensor data in a three-dimensional environment may be initiated without displaying or receiving user interface input (e.g., automatically or using other inputs without displaying a user interface). In addition, the operations in process 400 described above are optionally performed by running an information processing device such as a general purpose processor (e.g., as described with respect to Figure 2 or one or more functional modules in a dedicated chip and / or through Figure 2 to be implemented by other components.

[0058] FIG. 5A to FIG. 5E An example system for displaying image sensor data in a three-dimensional environment according to some examples of the present disclosure is shown. In one or more examples, Figure 5A An example three-dimensional environment 514 is shown rendered by computer system 101. In some examples, three-dimensional environment 514 is an extended reality environment that includes a visualization of the user's physical environment, and computer system 101 enhances the visualization by placing other visualizations in the three-dimensional environment to create an extended reality environment. Thus, in one or more examples, three-dimensional environment 514 includes sofa 506 and table 508, which are physical sofas and tables, respectively, incorporated into the three-dimensional environment.

[0059] In one or more examples, the three-dimensional environment 514 includes a predefined area 502 for displaying one or more image sensor data feeds (similar to the image sensor data feeds 318 described above). In some examples, the predefined area 502 may correspond to a physical electronic device (e.g., a monitor or television) visible within the three-dimensional environment 514. Additionally or alternatively, the predefined area 502 may correspond to some other physical object in the three-dimensional environment, such as a picture frame, a window, a door, etc. In some examples, the predefined area 502 does not correspond to any physical object visible within the three-dimensional environment 514, but is a virtual area that may float in space and is defined by the computer system 101 and placed by the computer system 101 (or a user) in the three-dimensional environment 514 for the purpose of specifying an area within the three-dimensional environment 514 where image sensor data may be displayed.

[0060] In one or more examples, and as Figure 5AAs shown, computer system 101 displays an image sensor connection user interface 504 for displaying an image sensor data feed. In some examples, image sensor connection user interface 504 can be displayed adjacent to predefined area 502, thereby indicating that user interface 504 corresponds to displaying image sensor data at predefined area 502. In one or more examples, and in response to detecting a selection of image sensor connection user interface 504 (e.g., by detecting a gaze 512 of a user of the computer system directed toward user interface 504 while a hand 510 of the user performs an air gesture (such as an air pinch), computer system 101 displays an image sensor selection user interface 516, as shown in FIG. Figure 5B shown.

[0061] like Figure 5B As shown, the image sensor selection user interface 516 includes one or more selectable options 518a-518c for facilitating a user of the computer system 101 to select an image sensor data feed from a list of one or more image sensor data feeds available for display on the computer system 101. In one or more examples, each of the selectable options 518a-518c corresponds to an image sensor data feed in the one or more image sensor data feeds available for display within the three-dimensional environment 514 and specifically at the predefined area 502. In some examples, each of the image sensor data feeds displayed on the image sensor selection user interface 516 belongs to an image sensor data feed that has been sent to the computer system 101 with an indication that the image sensor data feed is available for display. For example, for each selectable option 518a-518c, a device that is part of an image sensor for collecting the image sensor data feed or a device that is communicatively coupled to the image sensor (e.g., corresponding to the electronic device 304) sends (or broadcasts) information about the image sensor data feed to the computer system 101. In some examples, this information may include the IP address where the feed can be accessed, the aspect ratio of the feed, and resolution information about the feed (similar to the information described above regarding FIG. 3A to FIG. 3E In one or more examples, upon receiving the information about the image sensor data feed, the computer system 101 stores the information in a memory of the computer system and generates a selectable option on the image sensor selection user interface 516 that the user can select to initiate display of the image sensor data feed within the three-dimensional environment 514. The image sensors corresponding to the selectable items 518a-518c are similar to those described above with respect to FIG. 3A to FIG. 3EIn one or more examples, in response to receiving a selection of a selection option 518a-518c (e.g., in response to detecting a user's gaze 512 on selectable option 518b while also detecting a hand 510 performing an air gesture), the computer system 101 can establish a communication link with the image sensor and display an image sensor data feed corresponding to the selected selectable option, such as Figure 5C In one or more examples, computer system 101 uses information associated with the selected image sensor data feed to establish a communication link (eg, by establishing a communication link at an IP address associated with the image sensor data feed).

[0062] exist Figure 5C In one or more examples, and after establishing a communication link with an image sensor associated with a feed of a selected item as described above (and described in further detail below), computer system 101 receives image sensor data from the selected image sensor and displays the image sensor data as image sensor data feed 520 within three-dimensional environment 514. In one or more examples, image sensor data feed 520 provides a real-time or near real-time display of image data being collected by the image sensor associated with feed 520. Thus, in one or more examples, if the image sensor associated with feed 520 is displaying video, feed 520 will be represented as a video feed in three-dimensional environment 514. In one or more examples, if the image sensor associated with feed 520 is providing still images, feed 520 will be displayed by computer system 101 as a still image feed.

[0063] In one or more examples, the computer system 101 renders and displays the feed 520 according to one or more image specifications and / or display specifications associated with the image sensor data of the image sensor data feed 520. For example, and as described above, the computer system 101 may store information associated with the feed 520 (before selecting the feed 520), including the aspect ratio of the image data and the resolution of the image sensor data, or according to the size, screen resolution, or position specified in the display specifications. Additionally or alternatively, the image and / or display specifications may be embedded in the actual image sensor data received from the image sensor at the computer system 101. In the case where the image and / or display specifications are embedded within the image sensor data, the computer system 101 may extract the specifications from the image sensor data and use them to render and display the image sensor data within the three-dimensional environment 514. In some examples, rendering / displaying the image sensor data based on the image and / or display specifications may include displaying the feed 520 according to the aspect ratio and / or resolution specified in the image specifications or according to the size, screen resolution, or position specified in the display specifications.

[0064] In one or more examples, and as Figure 5C As shown, feed 520 is initially displayed at predefined area 502, such that the feed is superimposed on the predefined area. In one or more examples, a user of computer system 10 can have control and flexibility over how image sensor data is displayed. For example, in one or more examples, in response to detecting a user gaze 512 directed toward feed 520, and upon detecting a user's hand 510 performing an air gesture, computer system 101 can move feed 520 to a different location within three-dimensional environment 514 commensurate with the movement of the user's hand, such as Figure 5D shown.

[0065] In one or more examples, and as Figure 5D As shown, image sensor data feed 520 has moved from being displayed above predefined area 502 to being displayed in a second position within three-dimensional environment 514. The second position at which feed 520 is displayed by computer system 101 is commensurate with the movement of the user's hand 510. Thus, the distance and position from the original display position to the new display position of feed 520 is based on the direction and amount of movement of the user's hand 510. Because computer system 101 is now receiving image sensor data from the image sensor (directly or indirectly), it can now allow the user to customize the position at which feed 520 is displayed within three-dimensional environment 514. In one or more examples, customization of feed 520 is not limited to simply moving the position of the feed, but can also include adjusting the size of feed 520. For example, upon detecting a user's gaze 512 and upon the user performing an air gesture with hand 510 (such as widening the distance between their thumb and index finger), computer system 101 can enlarge the size of feed 520, such as by moving the user's hand 510. Figure 5E shown.

[0066] In one or more examples, and as Figure 5E As shown, feed 518 has been resized (in this case, made larger) based on the movement of the user's hand 510 (and specifically, the amount of movement of the user's thumb and index finger). In some examples, the user of computer system 101 can similarly reduce the size of feed 520 as well (e.g., by bringing their thumb and index finger together). In one or more examples, computer system 101 can resize feed 520 not only based on the movement of the user's hand 510, but also based on the image specifications provided to computer system 101 (as described above). For example, when zooming in on feed 520, computer system 101 can display feed 520 at a larger size while still maintaining the aspect ratio and screen resolution associated with feed 520. FIG. 5A to FIG. 5EIn the examples of , although a single image sensor data feed is shown, the example should not be considered limiting, and the examples described above may be applied to multiple image sensor data feeds displayed simultaneously within a three-dimensional environment.

[0067] Figure 6 An example flow chart illustrating a method of displaying image sensor data in a three-dimensional environment according to an example of the present disclosure is shown. In one or more examples, Figure 6 The process 600 shows the FIG. 5A to FIG. 5E In some examples, when a three-dimensional environment (such as three-dimensional environment 514) is displayed via one or more displays (wherein the three-dimensional environment includes a first predefined location within the three-dimensional environment (such as Figure 5A The computer system 101 receives (602) an instruction to display image sensor data from a first image sensor at a first predefined location within the three-dimensional environment. For example, in one or more examples, the computer system 101 selects a display on a user interface (e.g., Figure 5A A user of one or more image sensor data feeds on an interface 504 in FIG. 1 receives an instruction to display the image sensor data.

[0068] In some examples, in response to receiving an indication to display image sensor data from the first image sensor at the first predefined location, computer system 101 establishes (604) a communication link to the first image sensor. In one or more examples, computer system 101 stores connection information (e.g., IP address information) associated with the first image sensor and uses the information to establish a communication link with the image sensor, thereby generating a communication link with the first image sensor for receiving image sensor data from the first image sensor, which image sensor data can be used for the purpose of displaying the image sensor data within the three-dimensional environment.

[0069] In some examples, after establishing a communication link with the first image sensor, the computer system 101 receives (606) image sensor data from the first image sensor via the established communication link and displays (608) image sensor data from the first image sensor at a first predefined location in the three-dimensional environment. In some examples, the image sensor data feed can be initially displayed at a predefined location, but can be moved and resized in the three-dimensional environment. In some examples, the computer system 101 uses display information stored by the computer system and associated with the image sensor to display the image sensor data in the three-dimensional environment. For example, the display information may include an aspect ratio and a screen resolution associated with the image sensor data. In one or more examples, and using the display information, the computer system 101 displays the image sensor data according to the display information (e.g., image specifications) within the three-dimensional environment.

[0070] It should be understood that process 600 is an example, and more, fewer, or different operations may be performed in the same or different order. In addition, the operations in process 400 described above are optionally performed by running an information processing device such as a general-purpose processor (e.g., as described in relation to Figure 2 or one or more functional modules in a dedicated chip and / or through Figure 2 to be implemented by other components.

[0071] Figure 7 An example communication link between an image sensor and a computer system according to some examples of the present disclosure is shown. In one or more examples, Figure 7 Example system 700 shows a communication link between an external image sensor and computer system 101 as described above with respect to processes 400 and 600. In one or more examples, image sensor 706 (described above with respect to FIG. 3A to FIG. 3E and FIG. 5A to FIG. 5E Detailed description) is connected to transmitter 702, and the transmitter is configured to send the image sensor data to computer system 101 (e.g., similar to or identical to computer system 101 or 201 described above) once a communication link has been established between image sensor 706 and computer system 710. Figure 7 In the example of , transmitter 702 is shown as a separate component from image sensor 706, however, with respect to transmitter 702 and video card 704 (described below), any or all of these components may be integrated as part of the same device with image sensor 706, and the functionality may be implemented by one or more processors.

[0072] In one or more examples, and in examples where the image sensor 706 and the transmitter 702 are implemented on separate devices or are not integrated, the image sensor 706 may be connected to the transmitter 702 via a video card 704. In one or more examples, the video card 704 is configured to receive data from the image sensor 706 using a variety of interface formats (such as HDMI, DisplayPort, Mini DisplayPort, Video Graphics Array (VGA), and Digital Visual Interface (DVI)). In one or more examples, the video card may accept a variety of input types (e.g., interface formats) as described above, but may convert and output any received signal to a common format such as USB-C. In one or more examples, the video card 704 is optional. For example, in the case where the transmitter 702 is integrated with the image sensor 706, the video card 704 may be unnecessary because the image sensor 706 and the transmitter 702 may access a common memory.

[0073] In one or more examples, the transmitter 702 may send any received image sensor data to a receiver 708 that is communicatively coupled to or is part of the computer system 101. In one or more examples, the transmitter 702 and the receiver 708 may be communicatively coupled to each other across a variety of media including both wired and wireless transmission media. In one or more examples, the transmitter 702 may send the image sensor data to the receiver 708 using an Internet protocol, wherein each of the transmitter 702 and the receiver 708 has its own IP address and is connected to each other via a public or private computer network (such as the Internet or a peer-to-peer network). In some examples, the receiver 708 receives data from the transmitter 702, which may be sent to the computer system 101 or otherwise accessed by the computer system 101 (via a wired or wireless connection). The computer system 101 then uses the data to render and / or display the received image sensor data according to the examples described above.

[0074] Therefore, according to the above, some examples of the present disclosure relate to a method, which includes: at a computer system that communicates with one or more displays and one or more input devices: when presenting a three-dimensional environment including an electronic device with a display via the one or more displays (wherein the electronic device is communicatively coupled to an image sensor), receiving an indication to display image sensor data from an image sensor communicatively coupled to the electronic device; in response to receiving an indication to display image sensor data from the image sensor communicatively coupled to the electronic device, establishing a communication link to the image sensor communicatively coupled to the electronic device; receiving image sensor data from the image sensor via the communication link; and displaying the image sensor data from the image sensor in the three-dimensional environment.

[0075] Optionally, receiving the indication to display image sensor data from an image sensor communicatively coupled to the electronic device includes detecting that a user of the computer system is looking at a display of the electronic device while the display of the electronic device displays the visual code.

[0076] Optionally, in response to receiving an indication at the electronic device that the electronic device is communicatively coupled to the image sensor, the visual code is displayed on the electronic device.

[0077] Optionally, the visual code is a Quick Response (QR) code.

[0078] Optionally, the visual code includes internet protocol information for accessing image sensor data from the image sensor.

[0079] Optionally, the visual code comprises one or more of resolution information and aspect ratio information associated with the image sensor data from the image sensor.

[0080] Optionally, the visual code includes one or more of screen resolution information of the electronic device, size information, or a location of displayed information.

[0081] Optionally, receiving an indication to display image sensor data from an image sensor communicatively coupled to the electronic device also includes: in response to detecting that a user of the computer system is looking at a display of the electronic device when the display of the electronic device displays the visual code, displaying an image sensor activation user interface in the three-dimensional environment, and receiving a first input at the image sensor activation user interface.

[0082] Optionally, displaying the image sensor data from the image sensor comprises displaying the image sensor data at a location within the three-dimensional environment corresponding to a display of the electronic device.

[0083] Optionally, displaying image sensor data received from the image sensor comprises displaying the image sensor data at a predetermined location within the three-dimensional environment.

[0084] Optionally, displaying image sensor data from an image sensor in a three-dimensional environment includes displaying the image sensor data from the image sensor at a first location within the three-dimensional environment, and wherein the method further includes: while displaying the image sensor data from the image sensor at the first location within the three-dimensional environment, receiving a first input from a first part of a user, the first input comprising a first air gesture directed toward the displayed image sensor data followed by movement of the first part of the user, and in response to receiving the first input, updating the location of the displayed image sensor data in the three-dimensional environment based on the detected movement of the first part of the user, wherein the updated location of the displayed image sensor data is a second location different from the first location within the three-dimensional environment.

[0085] Optionally, displaying image sensor data from an image sensor in a three-dimensional environment includes displaying the image sensor data from the image sensor in the three-dimensional environment according to a first size, and wherein the method further includes: while displaying the image sensor data from the image sensor in the three-dimensional environment according to the first size, receiving a first input from a first part of a user, the first input including a first air gesture directed toward the displayed image sensor data followed by movement of the first part of the user, and in response to receiving the first input, updating a size of the displayed image sensor data according to the detected movement of the first part of the user, wherein the updated size of the displayed image sensor data is a second size different from the first size.

[0086] Optionally, the communication link is a direct communication link to the image sensor.

[0087] Optionally, the communication link is an indirect communication link via an electronic device.

[0088] In accordance with the above, some examples of the present disclosure relate to a method, comprising: at a computer system that communicates with one or more displays and one or more input devices: when presenting a three-dimensional environment via the one or more displays (wherein the three-dimensional environment includes a first predefined position within the three-dimensional environment), receiving an indication to display image sensor data from a first image sensor at a first predefined position within the three-dimensional environment; in response to receiving an indication to display image sensor data from the first image sensor at the first predefined position, generating a communication link to the first image sensor; receiving image sensor data from the first image sensor via the communication link; and displaying the image sensor data from the first image sensor at the first predefined position within the three-dimensional environment via the one or more displays.

[0089] Optionally, receiving an indication to display image sensor data from a first image sensor of the one or more image sensors includes: when presenting a three-dimensional environment including a first predefined location, displaying an image sensor connection user interface at the predefined location within the three-dimensional environment, receiving a first input at the image sensor connection user interface to display image sensor data at the predefined location, in response to receiving the first input at the image sensor connection user interface, displaying an image sensor selection user interface for selecting one of the one or more pre-image sensors, and while displaying the image sensor selection user interface, receiving a second input corresponding to selection of the first image sensor of the one or more image sensors.

[0090] Optionally, the method further comprises: while displaying the received image sensor data from the image sensor at a first predefined position within the three-dimensional environment, receiving a first input from a first part of the user, the first input comprising a first air gesture directed toward the displayed image sensor data followed by movement of the first part of the user, and in response to receiving the first input, updating the position of the displayed image sensor data in the three-dimensional environment based on the detected movement of the first part of the user, wherein the updated position of the displayed image sensor data is a second position different from the first predefined position within the three-dimensional environment.

[0091] Optionally, the received indication to display image sensor data from the first image sensor at the first predefined location comprises receiving internet protocol information for accessing image sensor data from the first image sensor.

[0092] Optionally, the received indication to display the image sensor data from the first image sensor at the first predefined position comprises receiving one or more of resolution information and aspect ratio information for displaying the image sensor data from the first image sensor.

[0093] Optionally, displaying image sensor data from an image sensor in a three-dimensional environment includes displaying the image sensor data from the image sensor in the three-dimensional environment according to a first size, and wherein the method further includes: when displaying the received image sensor data from the image sensor in the three-dimensional environment according to the first size, receiving a first input from a first part of a user, the first input including a first air gesture directed to the displayed image sensor data followed by movement of the first part of the user; and in response to receiving the first input, updating the size of the displayed image sensor data according to the detected movement of the first part of the user, wherein the updated size of the displayed image sensor data is a second size different from the first size.

[0094] Optionally, the predefined area in the three-dimensional environment corresponds to a representation of an electronic device having a display in the three-dimensional environment.

[0095] Optionally, displaying the image sensor data from the first image sensor comprises displaying the image sensor data at a location within the three-dimensional environment corresponding to a display of the electronic device.

[0096] Optionally, the communication link is a direct communication link to the image sensor.

[0097] Optionally, the communication link is an indirect communication link via an electronic device.

[0098] Some examples of the present disclosure relate to an electronic device comprising: one or more processors; a memory; and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for executing any of the above methods.

[0099] Some examples of the present disclosure relate to a non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any of the above methods.

[0100] Some examples of the present disclosure relate to an electronic device comprising one or more processors, a memory, and a component for performing any of the above methods.

[0101] Some examples of the present disclosure relate to an information processing apparatus used in an electronic device, the information processing apparatus including a component for executing any one of the above methods.

[0102] For the purpose of explanation, the foregoing description is described by reference to specific examples. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Based on the above teachings, many modifications and variations are possible. The examples are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so as to enable other technical personnel in the field to best use the present disclosure and the various described examples with various modifications suitable for the specific purpose contemplated.

Claims

1. A method, comprising: At a computer system in communication with one or more displays and one or more input devices: receiving an instruction to display image sensor data from an image sensor communicatively coupled to the electronic device while presenting, via the one or more displays, a three-dimensional environment including an electronic device having a display, wherein the electronic device is communicatively coupled to the image sensor; in response to receiving an indication to display image sensor data from the image sensor communicatively coupled to the electronic device, establishing a communication link to the image sensor communicatively coupled to the electronic device; receiving image sensor data from the image sensor via the communication link; as well as The image sensor data from the image sensor is displayed in the three-dimensional environment.

2. The method of claim 1 , wherein receiving the indication of displaying image sensor data from the image sensor communicatively coupled to the electronic device comprises detecting that a user of the computer system is looking at the display of the electronic device while the display of the electronic device displays a visual code. 3 . The method of claim 2 , wherein the visual code is displayed on the electronic device in response to receiving an indication at the electronic device that the electronic device is communicatively coupled to the image sensor.

4. The method of claim 2, wherein the visual code is a Quick Response (QR) code.

5. The method of claim 2, wherein the visual code comprises Internet protocol information for accessing the image sensor data from the image sensor.

6. The method of claim 2, wherein the visual code comprises one or more of resolution information and aspect ratio information associated with the image sensor data from the image sensor.

7. The method of claim 2, wherein the visual code comprises information related to the display of the electronic device, the information comprising one or more of screen resolution information, size information, and display position information of the electronic device.

8. The method of claim 2, wherein receiving the indication to display image sensor data from the image sensor communicatively coupled to the electronic device further comprises: displaying an image sensor activated user interface in the three-dimensional environment in response to detecting that the user of the computer system is looking at the display of the electronic device while the display of the electronic device displays the visual code; as well as A first input is received at the image sensor activated user interface. 9 . The method of claim 1 , wherein displaying the image sensor data from the image sensor comprises displaying the image sensor data at a location within the three-dimensional environment corresponding to a display of the electronic device. 10 . The method of claim 1 , wherein displaying the image sensor data received from the image sensor comprises displaying the image sensor data at a predetermined location within the three-dimensional environment.

11. The method of claim 1 , wherein displaying the image sensor data from the image sensor in the three-dimensional environment comprises displaying the image sensor data from the image sensor at a first location within the three-dimensional environment, and wherein the method further comprises: while displaying the image sensor data from the image sensor at the first location within the three-dimensional environment, receiving a first input from a first portion of the user, the first input comprising a first mid-air gesture directed toward the displayed image sensor data followed by movement of the first portion of the user; as well as In response to receiving the first input, updating a position of the displayed image sensor data in the three-dimensional environment based on the detected movement of the first part of the user, wherein the updated position of the displayed image sensor data is a second position different from the first position within the three-dimensional environment.

12. The method of claim 1 , wherein displaying the image sensor data from the image sensor in the three-dimensional environment comprises displaying the image sensor data from the image sensor in the three-dimensional environment according to a first size, and wherein the method further comprises: while displaying the image sensor data from the image sensor in the three-dimensional environment according to the first dimension, receiving a first input from a first portion of the user, the first input comprising a first air gesture directed toward the displayed image sensor data followed by movement of the first portion of the user; as well as In response to receiving the first input, a size of the displayed image sensor data is updated according to the detected movement of the first portion of the user, wherein the updated size of the displayed image sensor data is a second size different from the first size.

13. The method of claim 1, wherein the communication link is a direct communication link to the image sensor. The method of claim 1 , wherein the communication link is an indirect communication link via the electronic device.

15. An electronic device in communication with a display generation component and one or more input devices, the electronic device comprising: one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 14.

16. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by one or more processors of an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 14.

17. An electronic device in communication with a display generation component and one or more input devices, the electronic device comprising: one or more processors; Memory; means for receiving an indication to display image sensor data from an image sensor communicatively coupled to an electronic device having a display while presenting a three-dimensional environment including an electronic device having a display via the one or more displays, wherein the electronic device is communicatively coupled to the image sensor; means for establishing a communication link to the image sensor communicatively coupled to the electronic device in response to receiving an indication from the image sensor communicatively coupled to the electronic device to display image sensor data; means for receiving image sensor data from said image sensor via said communication link; as well as Means for displaying the image sensor data from the image sensor in the three-dimensional environment.

Images