Display environment representation

Abstract

A method includes displaying a home ER environment represented by home ER world coordinates, including one or more stereo model view representations of one or more corresponding ER environments. Each stereo model view representation includes ER objects arranged in spatial relationships according to corresponding ER world coordinates. In some embodiments, in response to detecting an input directed to a first stereo model view representation, the method includes transforming the home ER environment. Transforming the home ER environment includes transforming spatial relationships between a subset of the ER objects according to the home ER world coordinates and the corresponding ER world coordinates. In some embodiments, in response to detecting an input associated with a first stereo model view representation of a plurality of stereo model view representations, the method includes changing the display of the first stereo model view representation of the plurality of stereo model view representations from a first viewing vector to a second viewing vector while maintaining the arrangement of the ER objects.

Description

Technical Field

[0001] The present disclosure relates to display content, and in particular, to corresponding representations of display environments. Background Art

[0002] The previously available device may display content within an operating environment. In augmented reality (AR) applications, the operating environment corresponds to a physical (e.g., real-world) environment that includes physical objects, and the previously available device displays virtual objects superimposed on a graphical representation of the physical environment. In virtual reality (VR) applications, the operating environment corresponds to a virtual environment that includes purely virtual objects.

[0003] However, the device lacks a mechanism for enabling three-dimensional (3D) manipulation of displayed content within the operating environment. The device also lacks a mechanism for displaying virtual objects such that virtual objects and physical objects obscure each other within the operating environment. Summary of the invention

[0004] According to some specific implementations, a method is performed at an electronic device having one or more processors, non-volatile memory, one or more input devices, and a display device. The method includes displaying, via the display device, a home augmented reality (ER) environment represented by home ER world coordinates, including a first stereoscopic model view representation of a first ER environment. The first stereoscopic model view representation includes one or more ER objects in the ER objects arranged in a spatial relationship according to the first ER world coordinates. The method includes detecting, via the one or more input devices, a first input directed to the first stereoscopic model view representation. The method includes, in response to detecting the first input, transforming the home ER environment in the following manner: ceasing to display the first stereoscopic model view representation within the home ER environment; transforming the spatial relationship between a subset of the one or more ER objects according to the home ER world coordinates and the first ER world coordinates; and based on the transformation, displaying, via the display device, the subset of the one or more ER objects within the home ER environment.

[0005] According to some specific implementations, a method is performed at an electronic device having one or more processors, non-transitory memory, one or more input devices, and a display device. The method includes displaying a plurality of stereo model view representations from corresponding plurality of viewing vectors via the display device. The plurality of stereo model view representations correspond to a plurality of augmented reality (ER) environments. Each of the plurality of stereo model view representations is associated with a corresponding ER world coordinate set characterizing the corresponding ER environment. The plurality of stereo model view representations includes a first stereo model view representation of the plurality of stereo model view representations displayed from a first viewing vector. The first stereo model view representation of the plurality of stereo model view representations includes one or more ER objects of the ER objects arranged according to the first ER world coordinate set. The method includes detecting an input associated with the first stereo model view representation of the plurality of stereo model view representations via the one or more input devices. The method includes, in response to detecting the input, changing the display of the first stereo model view representation of the plurality of stereo model view representations from the first viewing vector to a second viewing vector while maintaining the one or more ER objects arranged according to the first ER world coordinate set.

[0006] According to some specific implementations, an electronic device includes one or more processors, a non-volatile memory, one or more input devices, and a display device. One or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of operations of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of an electronic device, cause the device to perform or cause the execution of operations of any of the methods described herein. According to some specific implementations, an electronic device includes a device for performing or causing the execution of operations of any of the methods described herein. According to some specific implementations, an information processing device for use in an electronic device includes a device for performing or causing the execution of operations of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] For a better understanding of the various described implementations, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the drawings.

[0008] Figure 1 is a block diagram of an example of a portable multifunction device according to some implementations.

[0009] Figures 2A to 2V are examples of operating environments according to some specific implementations.

[0010] Figure 3 is a flow chart of a method for changing a perspective view of a stereoscopic model view representation of an ER environment based on user input according to some implementations.

[0011] Figure 4 is a flow chart of a method of transforming spatial relationships between a subset of ER objects according to respective coordinates associated with corresponding stereo model view representations and home ER world coordinates according to some implementations. Summary of the invention

[0013] In some cases, the device may display one or more objects within the operating environment. In some applications, the device modifies the display of the current operating environment. Modifying the current operating environment may include displaying different (e.g., new) virtual objects at corresponding predetermined locations within the current operating environment, regardless of user input. Displaying virtual objects at predetermined locations within the current operating environment is problematic because virtual objects and physical (e.g., real-world) objects that are also within the current operating environment may obstruct (e.g., occlude) each other, thereby reducing the functionality provided by the current operating environment.

[0014] In contrast, various implementations include systems, methods, and electronic devices that, in response to detecting a first input directed to a first stereo model view representation of a first ER environment, transform a subset of ER objects included in a first stereo model view representation to a home ER environment based on home ER world coordinates and first ER world coordinates. In some implementations, the method includes moving the subset of ER objects in response to detecting a second input. For example, in some implementations, the method includes moving the subset of ER objects relative to physical objects within the home ER environment. As a result, obstructions (e.g., occlusions) between the physical objects and the ER objects are negated, thereby enabling a richer set of functionality relative to the home ER environment.

[0015] In some cases, the device can display content in an operating environment. For example, in a conventional video conferencing application, the operating environment corresponds to a shared communication environment. The device displays a remotely located individual associated with the shared communication environment based on the corresponding recorded video stream of the individual. Therefore, individuals can interact with each other in a graphical manner. The device displays content in an operating environment in a two-dimensional (2D) manner, such as a planar 2D video stream associated with a video conference. However, the device lacks a mechanism for enabling three-dimensional (3D) manipulation of displayed content in an operating environment.

[0016] In contrast, various specific implementations include methods, systems, and electronic devices that enable changing the perspective of a first stereo model view representation of a plurality of displayed stereo model view representations based on user input. The plurality of stereo model view representations correspond to a plurality of augmented reality (ER) environments. Each of the plurality of stereo model view representations is associated with a respective ER world coordinate set characterizing the respective ER environment. Based on the user input, the electronic device changes the perspective of a first stereo model view representation of the plurality of stereo model view representations while maintaining a previous arrangement of ER objects therein according to the respective ER world coordinate set. Thus, the first stereo model view representation of the plurality of stereo model view representations can be manipulated from a 3D perspective, thereby enabling a richer set of functionality with respect to the ER environment. DETAILED DESCRIPTION

[0017] Reference will now be made in detail to implementations, embodiments of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described implementations. However, it will be apparent to one of ordinary skill in the art that the various described implementations can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail so as not to unnecessarily obscure various aspects of the implementation.

[0018] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact without departing from the scope of the various described specific implementations. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0019] The terms used in the description of various embodiments described herein are only for the purpose of describing specific embodiments, and are not intended to be limiting. As used in the description of various embodiments described and in the appended claims, the singular forms "one", "an" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their groupings.

[0020] As used herein, the term "if" is optionally interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0021] Various examples of electronic systems and techniques for using such systems in connection with various augmented reality techniques are described.

[0022] A physical set refers to a world that individuals can sense and / or interact with without the use of electronic systems. A physical set, such as a physical park, includes physical elements, such as physical wildlife, physical trees, and physical plants. People can directly sense and / or otherwise interact with a physical set, for example, using one or more senses, including vision, smell, touch, taste, and hearing.

[0023] In contrast to a physical set, an augmented reality (ER) set or computer generated reality (CGR) set refers to a fully (or partially) computer generated set that a person can sense and / or otherwise interact with using an electronic system. In ER / CGR, the movement of a person is locally monitored, and in response thereto, at least one property corresponding to at least one virtual object in the ER / CGR set is changed in a manner consistent with one or more laws of physics. For example, in response to the ER / CGR system detecting that the person is looking up, the ER / CGR system can adjust various audio and graphics presented to the person in a manner consistent with the way such sounds and appearance would change in a physical set. Adjustments to properties of virtual objects in the ER / CGR set can also be made, for example, in response to representations of movement (e.g., voice commands).

[0024] A person may sense and / or interact with ER / CGR objects using one or more senses, such as vision, smell, taste, touch, and hearing. For example, a person may sense and / or interact with objects that create a multi-dimensional or spatial acoustic scenery. A multi-dimensional or spatial acoustic scenery provides an individual with the perception of discrete sound sources in a multi-dimensional space. Such objects may also implement acoustic transparency, which may selectively incorporate audio from the physical scenery with or without computer-generated audio. In some ER / CGR scenery, a person may sense and / or interact with only audio objects.

[0025] Virtual reality (VR) is an example of ER / CGR. A VR set refers to an augmented set that is configured to include only computer-generated sensory input for one or more senses. The VR set includes a plurality of virtual objects that a person can sense and / or interact with. A person can sense and / or interact with virtual objects in a VR set by simulating at least some of the person's actions within the computer-generated set and / or by simulating a person or his / her presence within the computer-generated set.

[0026] Mixed reality (MR) is another example of ER / CGR. An MR set refers to an augmented set that is configured to integrate computer-generated sensory input (e.g., virtual objects) with sensory input from a physical set or a representation of sensory input from a physical set. On the reality spectrum, MR sets are between and do not include fully physical sets at one end and VR sets at the other end.

[0027] In some MR sets, computer-generated sensory inputs may be adjusted based on changes in sensory inputs from the physical set. Additionally, some electronic systems used to render the MR set may detect position and / or orientation relative to the physical set to enable interaction between real objects (i.e., physical elements from the physical set or representations thereof) and virtual objects. For example, the system may detect movement and adjust computer-generated sensory inputs accordingly so that, for example, a virtual tree appears stationary relative to a physical structure.

[0028] Augmented reality (AR) is an example of MR. AR scenery refers to an enhanced scenery in which one or more virtual objects are superimposed on a physical scenery (or a representation thereof). For example, an electronic system may include an opaque display and one or more imaging sensors for capturing a video and / or image of a physical scenery. For example, such a video and / or image may be a representation of a physical scenery. The video and / or image is combined with a virtual object, where the combination is then displayed on an opaque display. The physical scenery can be viewed indirectly by a person via an image and / or video of the physical scenery. Therefore, a person can observe a virtual object superimposed on the physical scenery. When the system captures an image of the physical scenery and uses the captured image to display the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, a transparent or translucent display may be included in an electronic system for displaying an AR scenery, so that an individual can directly view the physical scenery through a transparent or translucent display. Virtual objects may be displayed on a translucent or transparent display, so that an individual observes virtual objects superimposed on the physical scenery. In another example, a projection system may be utilized to project virtual objects onto the physical scenery. For example, virtual objects may be projected onto a physical surface, or as a hologram, such that an individual observes the virtual object superimposed on a physical setting.

[0029] AR scenery may also refer to an augmented scenery in which a representation of a physical scenery is modified by computer-generated sensory data. For example, at least a portion of the representation of the physical scenery can be graphically modified (e.g., enlarged) so that the modified portion can still represent the originally captured image (but not an exact copy). Alternatively, when providing video pass-through, one or more sensor images may be modified to apply a specific viewpoint that is different from the viewpoint captured by the image sensor. For another example, a portion of the representation of the physical scenery may be changed by graphically blurring or eliminating the portion.

[0030] Augmented Virtual (AV) is another example of MR. An AV set refers to an augmented set that is a virtual or computer-generated set combined with one or more sensory inputs from a physical set. Such sensory inputs may include representations of one or more features of the physical set. Virtual objects may, for example, incorporate colors associated with physical elements captured by an imaging sensor. Alternatively, virtual objects may adopt features consistent with, for example, current weather conditions corresponding to the physical set, such as weather conditions identified via imaging, online weather information, and / or weather-related sensors. For another example, an AR park may include virtual structures, plants, and trees, although animals within the AR park set may include features accurately replicated from images of physical animals.

[0031] Various systems allow people to sense and / or interact with the ER / CGR scenery. For example, a head-mounted system may include one or more speakers and an opaque display. As another example, an external display (e.g., a smart phone) may be incorporated into the head-mounted system. The head-mounted system may include a microphone for capturing audio of the physical scenery and / or an image sensor for capturing images / video of the physical scenery. A transparent or semi-transparent display may also be included in the head-mounted system. The semi-transparent or transparent display may, for example, include a substrate through which light (representing an image) is directed to a person's eyes. The display may also include an LED, an OLED, a liquid crystal on silicon, a laser scanning light source, a digital light projector, or any combination thereof. The substrate through which the light is transmitted may be an optical reflector, a holographic substrate, an optical waveguide, an optical combiner, or any combination thereof. The transparent or semi-transparent display may, for example, selectively transition between a transparent / semi-transparent state and an opaque state. As another example, the electronic system may be a projection-based system. In a projection-based system, retinal projection may be used to project an image onto a person's retina. Alternatively, the projection-based system may also project the virtual object into a physical setting, such as, for example, projecting the virtual object as a hologram or onto a physical surface. Other examples of ER / CGR systems include a window configured to display graphics, headphones, earphones, a speaker arrangement, a lens configured to display graphics, a head-up display, a car windshield configured to display graphics, an input mechanism (e.g., a controller with or without haptic capabilities), a desktop or laptop computer, a tablet computer, or a smart phone.

[0032] Figure 11 is a block diagram of an example of a portable multifunction device 100 (also sometimes referred to herein as an "electronic device 100" for brevity) according to some specific implementations. The electronic device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an input / output (I / O) subsystem 106, a speaker 111, a touch-sensitive display system 112, an inertial measurement unit (IMU) 130, an image sensor 143 (e.g., a camera), a contact intensity sensor 165, an audio sensor 113 (e.g., a microphone), an eye tracking sensor 164 (e.g., included in a head mounted device (HMD)), a hand tracking sensor 150, a body posture sensor 190, and other input or control devices 116. In some implementations, the electronic device 100 corresponds to one of a mobile phone, a tablet computer, a laptop computer, a wearable computing device, a head mounted device (HMD), a head mounted housing (e.g., the electronic device 100 slides into or is otherwise attached to the head mounted housing), etc. In some implementations, the head mounted housing is shaped to form a receiver for receiving the electronic device 100 having a display.

[0033] In some implementations, peripherals interface 118, one or more processing units 120, and memory controller 122 are, optionally, implemented on a single chip, such as chip 103. In some other implementations, they are, optionally, implemented on separate chips.

[0034] The I / O subsystem 106 couples input / output peripherals on the electronic device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an image sensor controller 158, an intensity sensor controller 159, an audio controller 157, an eye tracking controller 162, one or more input controllers 160 for other input or control devices, an IMU controller 132, a hand tracking controller 180, a body gesture controller 195, and a privacy subsystem 170. The one or more input controllers 160 receive / send electrical signals to / from other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative implementations, one or more input controllers 160 are optionally coupled to (or not coupled to) any of the following: a keyboard, an infrared port, a universal serial bus (USB) port, a stylus, and / or a pointer device such as a mouse. One or more buttons optionally include an increase / decrease button for volume control of the speaker 111 and / or the audio sensor 113. One or more buttons optionally include a push button. In some implementations, other input or control devices 116 include a positioning system (e.g., GPS) that obtains information about the position and / or orientation of the electronic device 100 relative to the physical environment.

[0035] The touch-sensitive display system 112 provides an input interface and an output interface between the electronic device 100 and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some specific implementations, some visual outputs or all visual outputs correspond to user interface objects. As used herein, the term "indicator" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed to a graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0036] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic and / or tactile contact. The touch-sensitive display system 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In an exemplary implementation, the point of contact between the touch-sensitive display system 112 and the user corresponds to a finger or stylus of the user.

[0037] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other specific implementations. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive technology, resistive technology, infrared technology, and surface acoustic wave technology, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof.

[0038] The user optionally uses any suitable object or appendage, such as a stylus, finger, etc., to contact the touch-sensitive display system 112. In some implementations, the user interface is designed to work with finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some implementations, the electronic device 100 converts the rough finger-based input into a precise pointer / cursor position or command for performing the action desired by the user.

[0039] The speaker 111 and the audio sensor 113 provide an audio interface between the user and the electronic device 100. The audio circuit receives audio data from the peripheral device interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into a sound wave audible to humans. The audio circuit also receives an electrical signal converted from a sound wave by an audio sensor 113 (e.g., a microphone). The audio circuit converts the electrical signal into audio data, and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit by the peripheral device interface 118. In some specific implementations, the audio circuit also includes an earphone jack. The earphone jack provides an interface between the audio circuit and a removable audio input / output peripheral device, and the removable audio input / output peripheral device is a headset such as an output-only headset or a headset with both output (e.g., a single-ear headset or a dual-ear headset) and input (e.g., a microphone).

[0040] The inertial measurement unit (IMU) 130 includes an accelerometer, a gyroscope, and / or a magnetometer to measure various forces, angular rates, and / or magnetic field information relative to the electronic device 100. Therefore, according to various specific implementations, the IMU detects one or more position change inputs of the electronic device 100, such as the electronic device 100 being shaken, rotated, moved in a specific direction, etc.

[0041] Image sensor 143 captures still images and / or video. In some implementations, optical sensor 143 is located on the back of electronic device 100, opposite the touch screen on the front of electronic device 100, so that the touch screen can be used as a viewfinder for still image and / or video image acquisition. In some implementations, another image sensor 143 is located on the front of electronic device 100, so that an image of the user is acquired (e.g., for selfies, for video conferencing when the user views other video conference participants on the touch screen, etc.). In some implementations, the image sensor is integrated into the HMD.

[0042] The contact force sensor 165 detects the intensity of contact on the electronic device 100 (e.g., touch input on the touch-sensitive surface of the electronic device 100). The contact force sensor 165 is coupled to the force sensor controller 159 in the I / O subsystem 106. The contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact force sensor 165 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the physical environment. In some implementations, at least one contact force sensor 165 is juxtaposed or adjacent to the touch-sensitive surface of the electronic device 100. In some implementations, at least one contact force sensor 165 is located on the back of the electronic device 100.

[0043] The eye tracking sensor 164 detects eye gaze of a user of the electronic device 100 and generates eye tracking data indicative of the eye gaze of the user. In various implementations, the eye tracking data includes data indicative of a user's fixation point (e.g., a focus point) on a display panel, such as a display panel within a head mounted device (HMD), a head mounted housing, or a heads up display.

[0044] The hand tracking sensor 150 acquires hand tracking data indicating the position of the user's hand. In various implementations, the electronic device 100 utilizes the hand tracking data to manipulate the display of the stereoscopic model view representation of the ER environment. For example, in some implementations, the electronic device 100 moves the stereoscopic model view representation to track the current position of the user's hand. For another example, the electronic device 100

[0045] The body posture sensor 190 acquires body posture data indicating the position of the user's head or body. In various implementations, the electronic device 100 utilizes the body posture data in order to manipulate the display of the stereoscopic model view representation of the ER environment. For example, the body posture data indicates that the user is turning his / her head sideways, and the electronic device 100 manipulates the perspective view of the stereoscopic model view representation of the ER environment accordingly.

[0046] In various implementations, the electronic device 100 includes a privacy subsystem 170 that includes one or more privacy setting filters associated with user information, such as user information included in eye gaze data and / or body position data associated with the user. In some implementations, the privacy subsystem 170 selectively prevents and / or limits the electronic device 100 or a portion thereof from acquiring and / or transmitting user information. To this end, the privacy subsystem 170 receives user preferences and / or selections from the user in response to prompting the user to make user preferences and / or selections. In some implementations, the privacy subsystem 170 prevents the electronic device 100 from acquiring and / or transmitting user information unless and until the privacy subsystem 170 obtains informed consent from the user. In some implementations, the privacy subsystem 170 anonymizes (e.g., scrambles or obscures) certain types of user information. For example, the privacy subsystem 170 receives user input that specifies which types of user information the privacy subsystem 170 anonymizes. As another example, the privacy subsystem 170 specifies (e.g., automatically) anonymizing certain types of user information that may include sensitive and / or identifying information independently of the user. As another example, privacy system 170 denies access to a first individual having a first access level that does not satisfy access level criteria associated with the ER session, and allows access to a second individual having a second access level that satisfies the access level criteria.

[0047] Figures 2A to 2V is an example of an operating environment 200 according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein.

[0048] For this reason, Figure 2A As shown, operating environment 200 includes an electronic device 203 held by user 210 and a physical table 220. In some implementations, electronic device 203 corresponds to a mobile device such as a smart phone, a laptop, a tablet computer, etc. In some implementations, electronic device 203 and Figure 1 The electronic device 203 is similar to and modified from the electronic device 100 in FIG.

[0049] According to various implementations, the electronic device 203 is configured to present a home ER environment 206 to the user 210. The home ER environment 206 is characterized by home ER world coordinates 208. In some implementations, the home ER world coordinates 208 define a set of points that exist in a three-dimensional (3D) space. In some implementations, the home ER environment 206 corresponds to a purely virtual environment, and thus the electronic device 203 does not display a physical table 220 within the home ER environment 206, regardless of the position of the electronic device 203 relative to the operating environment 200.

[0050] In some implementations, the home ER environment 206 corresponds to an AR environment. For example, in some implementations, the electronic device 203 is configured to present the home ER environment 206 and enable the video signal to pass through at least a portion of the physical objects within the operating environment 200 therein. As an example, referring to Figure 2F , after the user 210 wearing the electronic device 203 rotates the orientation of the electronic device 203 toward the physical table 220, the electronic device 203 displays the physical table 220 within the home ER environment 206. As another example, in some implementations, the electronic device 203 includes a transparent or additional display that enables optical see-through of the operating environment 200, including physical objects within the operating environment 200.

[0051] The home ER environment 206 includes a first stereo model view representation 230 of the first ER environment and a second stereo model view representation 240 of the second ER environment. In some implementations, the stereo model view representation represents a purely virtual environment. On the other hand, in some implementations, the stereo model view representation at least partially represents a real environment. For example, in some implementations, the stereo model view representation includes structural features of the real environment, such as the walls, floor, and ceiling of the user's living room. For another example, in some implementations, the stereo model view representation includes physical objects within virtual structural features, such as a real table within a virtual room.

[0052] The first stereoscopic model view representation 230 includes one or more of the ER objects arranged in a spatial relationship according to a first ER world coordinate 232. For example, the first ER world coordinate 232 defines a corresponding set of points existing in a 3D space. That is, the first stereoscopic model view representation 230 includes a television 234, an avatar 236, and a sofa 238. The avatar 236 corresponds to a virtual representation of an individual associated with (e.g., connected to) the first ER environment. For example, in some implementations, the avatar 236 is associated with an ER session that enables a graphical representation of the individual to be displayed within the first ER environment. In some implementations, the electronic device 203 dynamically updates the first stereoscopic model view representation 230, such as playback of video content via the television 234, or movement of the avatar 236, to reflect changes within the first ER environment.

[0053] The second stereo model view representation 240 includes one or more of the ER objects arranged in a spatial relationship according to the second ER world coordinate 242. That is, the second stereo model view representation 240 includes a sideboard 244 and a chair 246. For example, the second ER world coordinate 242 defines a corresponding set of points existing in 3D space. One of ordinary skill in the art will appreciate that the stereo model view representation may include any number and type of ER objects arranged in a variety of ways. One of ordinary skill in the art will appreciate that the stereo model view representation may include any number of avatars.

[0054] In some implementations, the electronic device 203 corresponds to a head mounted device (HMD) worn by the user 210. The HMD presents the above reference Figure 2A In some implementations, the HMD includes an image sensor associated with the field of view 204, and the HMD displays AR content including physical objects within the home ER environment 206. In some implementations, the HMD is similar to and adapted from Figure 1 In some implementations, the HMD is configured to present the home ER environment 206 and enable the video signal to pass through at least a portion of the physical objects within the operating environment 200. As another example, in some implementations, the HMD includes a transparent or additional display that enables optical see-through of the operating environment 200, including the physical objects within the operating environment 200.

[0055] In some implementations, the HMD includes an integrated display (e.g., a built-in display) that displays the home ER environment 206. In some implementations, the HMD includes a head-mounted housing. In various implementations, the head-mounted housing includes an attachment area to which another device having a display can be attached. In various implementations, the head-mounted housing is shaped to receive another device including a display (e.g., Figure 2A For example, in some implementations, the electronic device 203 slides / snaps onto or is otherwise attached to the head-mounted housing. In some implementations, a display of the device attached to the head-mounted housing presents (e.g., displays) the home ER environment 206.

[0056] FIG. 2B to FIG. 2H The respective perspectives of the first stereo model view representation 230 and the second stereo model view representation 240 are shown changing based on various user inputs. Figure 2B As shown, the electronic device 203 detects an input 250 associated with the second stereo model view representation 240. The input 250 requests to move the second stereo model view representation 240 upward along the z-axis of the home ER world coordinate 208 to a position above the first stereo model view representation 230. For example, in some implementations, an eye tracking sensor of the electronic device 203 (e.g., integrated therein) detects the input 250, wherein the input 250 corresponds to an eye gaze input indicating that the eye gaze of the user 210 is directed to the second stereo model view representation 240. For another example, in some implementations, a hand tracking sensor of the electronic device 203 (e.g., integrated therein) detects the input 250, wherein the input 250 corresponds to a hand tracking movement performed by the user 210 to point to the second stereo model view representation 240.

[0057] In response to detecting Figure 2B The electronic device 203 changes the display of the second stereo model view representation 240 from the first viewing vector to the second viewing vector by moving the second stereo model view representation 240 to a position above the first stereo model view representation 230, such as Figure 2C . The viewing vector provides one or more of a field of view (FOV), a pose / rotation coordinate, a translation coordinate, a viewing angle, etc. relative to the user 210 and the home ER coordinate system 208. When changing the display of the second stereo model view representation 240 from the first viewing vector to the second viewing vector, the electronic device 203 maintains the sideboard 244 and the chair 246 according to the second ER world coordinate 242. In other words, the electronic device 203 maintains the spatial relationship between the sideboard 244 and the chair 246 within the second stereo model view representation 240. For example, Figure 2B and Figure 2C As shown, the longer edge of the sideboard 244 extends substantially along (eg, parallel to) the y-axis of the second ER world coordinate 242. Figure 2B and Figure 2C As shown, the bottom of the legs of the sideboard 244 and the bottom of the legs of the chair 246 are located at the z=0 position of the z-axis of the second ER world coordinate 242. For another example, the electronic device 203 keeps the sideboard 244 and the chair 246 along Figure 2B and Figure 2C The corresponding positions of the xy plane of the second ER world coordinate 242 relative to each other.

[0058] like Figure 2D As shown, the electronic device 203 detects movement input 251, such as via an IMU of the electronic device 203. The movement input 251 corresponds to the user 210 moving clockwise (as viewed from above) around the first stereo model view representation 230 and the second stereo model view representation 240 within the operating environment 200. In response to detecting Figure 2D The mobile input 251 in the display, the field of view 204 associated with the electronic device 203 and thus associated with the displayed home ER environment 206 includes a portion of the physical table 220, such as Figure 2E shown.

[0059] Further in response to detecting Figure 2D In response to the mobile input 251 in the home ER environment 206, the electronic device 203 changes the display of the first stereoscopic model view representation 230 and the second stereoscopic model view representation 240 according to the rotation of the home ER environment 206, such as Figure 2E That is, the electronic device 203 rotates according to the corresponding rotation of the home ER world coordinate 208 (such as by Figure 2D and Figure 2E The first stereoscopic model view representation 230 and the ER objects therein (e.g., the television 234, the avatar 236, and the sofa 238) are rotated 90 degrees. For example, the view of the avatar 236 is rotated from Figure 2D The front view of the avatar 236 is changed to Figure 2E When rotating the first stereo model view representation 230, the electronic device 203 maintains the television 234, the avatar 236, and the sofa 238 according to the first ER world coordinate 232. In other words, the electronic device 203 maintains the spatial relationship between the television 234, the avatar 236, and the sofa 238 within the first stereo model view representation 230.

[0060] In addition, if Figure 2E As shown, the electronic device 203 rotates according to the corresponding home ER world coordinates 208 (such as by Figure 2D and Figure 2E The second stereoscopic model view representation 240 and the ER objects therein (e.g., the sideboard 244 and the chair 246) are rotated 90 degrees as indicated by the rotation of the second ER world coordinate 242 between Figure 2D The front view of the Chinese food cabinet 244 is changed to Figure 2EA side view of the dining cabinet 244. When the second stereoscopic model view representation 240 is rotated, the electronic device 203 maintains the dining cabinet 244 and the chair 246 according to the second ER world coordinate 242. In other words, the electronic device 203 maintains the spatial relationship between the dining cabinet 244 and the chair 246 within the second stereoscopic model view representation 240.

[0061] like Figure 2F As shown, the electronic device 203 detects a selection input 252 pointing to the first stereo model view representation 230. For example, the selection input 252 is detected by an eye tracking sensor of the electronic device 203 or by a hand tracking sensor of the electronic device 203. In response to detecting Figure 2F The electronic device 203 stops displaying the second stereoscopic model view representation 240 and keeps displaying the first stereoscopic model view representation 230. Figure 2G shown.

[0062] like Figure 2H As shown, the electronic device 203 detects a manipulation input 253 directed to the first stereoscopic model view representation 230. The manipulation input 253 rotates the first stereoscopic model view representation 230 counterclockwise (as viewed from above) by 90 degrees relative to the home ER world coordinate 208. Figure 2H and Fig.2I As shown, the first ER world coordinate 232 associated with the first stereoscopic model view representation 230 is rotated according to the manipulation input 253, while the home ER world coordinate 208 does not change. Therefore, the corresponding views of the television 234, the avatar 236, and the sofa 238 are in Figure 2H and Fig.2I Change from side view to front view.

[0063] Figures 2J to 2N 23. The ER session is associated with a first ER environment. The ER session enables corresponding graphical representations of individuals to be in the first ER environment at the same time. For example, the ER session enables a particular individual represented by avatar 236 to be in the first ER environment. In some implementations, electronic device 203 receives a request to join the ER session and displays a corresponding indication. For example, in some implementations, the indication corresponds to ER join interface 254, such as Figure 2J In addition, Figure 2JAs shown, the electronic device 203 plays (e.g., via a speaker) a first set of voice data 256 associated with a particular individual who is associated with (e.g., connected to) the ER session. For example, the first set of voice data 256 is a verbal request to join the ER session, such as "Hey, Bob, join my ER environment and we can watch TV together virtually." For another example, the first set of voice data 256 includes ambient noise associated with the first ER environment, such as noise from the television 234.

[0064] like Figure 2K and Figure 2L As shown, the electronic device 203 displays the animation of the avatar 236 based on the corresponding movement of the specific individual. Figure 2K As shown, avatar 236 moves from standing next to sofa 238 to sitting on sofa 238 facing television 234, as shown by movement arrow 257, which is shown for illustrative purposes only. Figure 2L As shown, the electronic device 203 displays a first stereoscopic model view representation 230 , in which the avatar 236 has moved from beside a sofa 238 to sitting on the sofa 238 and facing the television 234 .

[0065] like Figure 2M As shown, the electronic device 203 obtains a second set of voice data 258 from the user 210 (e.g., via an audio sensor) and provides the second set of voice data 258 to a specific individual. For example, the second set of voice data 258 is a verbal request to join the ER session, such as "Hey, Jill, OK, I'll join your ER environment, I want to watch the baseball game on TV."

[0066] Figure 2N to Figure 2V The spatial relationship between the subset of ER objects within the first stereoscopic model view representation 230 is shown as being transformed according to the first ER world coordinates 232 and the home ER world coordinates 208. Figure 2N As shown, the first stereo model view representation 230 has a first height value indicated by a first reference measurement 260a, which is shown for illustrative purposes only.

[0067] like Figure 2N As shown, the electronic device 203 detects an accept input 259 that accepts the request to join the ER session. Figure 2N In accepting input 259, the electronic device 203: stops displaying the first stereoscopic model view representation 230, transforms the subset of ER objects within the first stereoscopic model view representation 230 according to the first ER world coordinate 232 and the home ER world coordinate 208, and displays the subset of ER objects based on the transformation, such as Fig.2O In some specific implementations, Fig.2OAs shown, electronic device 203 displays a subset of ER objects within bounded ER environment 261. Fig.2O As shown, bounded ER environment 261 has a second height value indicated by second reference measurement result 260b, which is shown for illustrative purposes only. Fig.2O As shown, the second height value indicated by the second reference measurement result 260b is greater than the first height value indicated by the first reference measurement result 260a. One of ordinary skill in the art will appreciate that other dimensional components (e.g., depth and width) of the bounded ER environment 261 are greater than the corresponding dimensional components of the first stereo model view representation 230.

[0068] Displaying bounded ER environment 261 can help user 210 reposition a subset of ER objects to avoid physical objects and thus avoid visual occlusion, as described below. In some implementations, electronic device 203 displays a subset of ER objects without bounded ER environment 261 .

[0069] like Fig.2O As shown, the subset of ER objects includes a transformed television 264 based on the television 234, a transformed avatar 266 based on the avatar 236, and a transformed sofa 268 based on the sofa 238. The transformed television 264, the transformed avatar 266, and the transformed sofa 268 are arranged according to the transformed ER world coordinates 262 based on the first ER world coordinates 232. Compared with the first stereo model view representation 230 and the ER objects therein, the bounded ER environment 261 and the subset of ER objects therein more accurately represent the ER objects that can be later incorporated into or replace the home ER environment 206. For example, in some implementations, transforming the subset of ER objects corresponds to enlarging each subset of ER objects according to a zoom factor. The zoom factor can be predetermined or can be specified by user input. For example, in some implementations, the zoom factor corresponds to a real size. In this example, the subset of ER objects looks substantially the same as they would after being incorporated into or replacing the home ER environment 206.

[0070] In some implementations, the electronic device 203 arranges the transformed television 264, the transformed avatar 266, and the transformed sofa 268 to match (within an error threshold) the corresponding arrangement of the television 234, the avatar 236, and the sofa 238 within the first stereoscopic model view representation 230 defined by the first ER world coordinates 232 and the home ER world coordinates 208. For example, Figure 2N Compared to the corresponding ER object in the first stereoscopic model view representation 230, Fig.2O The subset of ER objects (264, 266, 268) shown are similarly positioned and oriented relative to each other.

[0071] In some implementations, the transformation of the subset of ER objects causes the relative distances between the transformed television 264 , the transformed avatar 266 , and the transformed couch 268 to match the relative distances of the corresponding ER objects within the first stereoscopic model view representation 230 within an error threshold.

[0072] In some implementations, the transformation of the ER object subsets corresponds to changing the resolution of some or all of the ER object subsets. For example, in some implementations, the electronic device 203 performs object recognition (e.g., semantic segmentation or object tracking) to identify dynamic ER objects (e.g., avatar 236) and reduce the resolution of static ER objects (e.g., sofa 238).

[0073] In some implementations, the bounded ER environment 261 and a subset of the ER objects are occluded by one or more physical objects within the home ER environment 206. For example, Fig.2O As shown, bounded ER environment 261 and the subset of ER objects are initially displayed along the back wall of home ER environment 206. Therefore, if user 210 requests to merge the subset of ER objects into home ER environment 206, transformed television 264 will be behind and therefore blocked by the back wall, thereby reducing the experience of user 210. Therefore, according to various implementations, electronic device 203 can move the subset of ER objects to avoid physical occlusion.

[0074] For example, see Figure 2P , electronic device 203 detects input 270 corresponding to moving bounded ER environment 261 and the subset of ER objects therein away from the back wall of home ER environment 206. In response to detecting Figure 2P Based on the input 270 in the embodiment, the electronic device 203 moves the bounded ER environment 261 and the subset of ER objects therein away from the back wall and above the physical table 220, such as Figure 2Q However, Figure 2Q As shown, the transformed avatar 266 and the transformed sofa 268 obscure a portion of the physical table 220.

[0075] like Figure 2R As shown, electronic device 203 detects input 272 corresponding to further moving bounded ER environment 261 and the subset of ER objects therein downwardly along the z-axis of home ER world coordinate 208 away from physical table 220. In response to detecting Figure 2R , the electronic device 203 moves the bounded ER environment 261 and the ER objects therein downward along the z-axis of the home ER world coordinate 208. Figure 2S As shown, the transformed avatar 266 and the transformed sofa 268 no longer obstruct the physical table 220 , thereby improving the experience of the user 210 .

[0076] like Figure 2T As shown, electronic device 203 detects input 274 directed to bounded ER environment 261. Input 274 corresponds to a request to enter the first ER environment. In some specific implementations, in response to detecting Figure 2T , the electronic device 203 adds a real-size (eg, actual-size) television 282, a real-size avatar 284, and a real-size sofa 286 to the home ER environment 206, such as Figure 2U It is worth noting that Figure 2U As shown, real-size television 282 is within field of view 204, but real-size avatar 284 and real-size sofa 286 are not. However, if user 210 rotates electronic device 203 90 degrees clockwise (as viewed from above), such as by turning his or her head toward real-size avatar 284, real-size avatar 284 and real-size sofa 286 will be within field of view 204.

[0077] In some implementations, adding a subset of ER objects to the home ER environment 206 corresponds to resizing the subset of ER objects to true size. Figure 2U As shown, the transformed avatar 266 and the transformed sofa 268 do not obscure the physical table 220. Therefore, the user 210 can interact with the real-size ER object without providing additional input, resulting in a more enjoyable experience and reducing the processor, memory, and battery utilization of the electronic device 203.

[0078] In some implementations, in response to detecting Figure 2T In response to input 274 in the home ER environment 206, the electronic device 203 replaces the home ER environment 206 with the first ER environment 290, such as when the first ER environment 290 is a pure VR environment, such as Figure 2V The first ER environment 290 includes a real-size TV 282 , a real-size avatar 284 , and a real-size sofa 286 , but does not include a physical table 220 .

[0079] Figure 3 is a flow chart of a method 300 for changing the perspective of a stereoscopic model view representation of an ER environment based on user input according to some implementations. In various implementations, the method 300 or a portion thereof is performed by an electronic device (e.g., Figure 1 The electronic device 100, Figures 2A to 2VIn various implementations, method 300 or a portion thereof is performed by an HMD. In some implementations, method 300 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 300 is performed by a processor that executes a code stored in a non-transitory computer-readable medium (e.g., a memory). In various implementations, some operations in method 300 are optionally combined, and / or the order of some operations is optionally changed.

[0080] As shown in block 302, in some implementations, method 300 includes generating a plurality of stereoscopic model view representations corresponding to a plurality of ER environments, respectively. In some implementations, method 300 includes obtaining a plurality of representation vectors, which respectively provide a plurality of spatial representations of the corresponding plurality of ER environments. Each of the plurality of representation vectors includes a plurality of object label values ​​that respectively identify one or more ER objects. Each of the plurality of representation vectors also includes a plurality of relative position values ​​that provide respective positions of the one or more ER objects relative to each other. In addition, method 300 includes generating a plurality of stereoscopic model view representations of the corresponding plurality of ER environments from the corresponding plurality of ER environments and the plurality of representation vectors according to the relative position values. For example, the object label value may include an instance object label value and / or a semantic object label value. For another example, the object label value may include one sub-value or multiple sub-values. For another example, a particular volumetric region of an ER environment may include three objects and therefore have three corresponding object label values, such as "first object", "second object" and "third object". The object label value may be associated with an object and / or a feature of an object (e.g., a table, a chair, a corner, an edge, etc.). Additionally, the object tag value can identify a moving object, such as an avatar of a particular individual within the ER environment. In some implementations, the spatial representation corresponds to a volumetric (eg, three-dimensional (3D)) representation.

[0081] In some implementations, generating multiple stereo model view representations is based on a zoom factor, such as scaling down the corresponding multiple ER environments according to the zoom factor. For example, in some implementations, the zoom factor is predetermined. For another example, the zoom factor is obtained via user input, such as input of an affordance indicated by pointing to a touch-sensitive surface of an electronic device or input corresponding to a user's eye gaze position. For another example, in some implementations, method 300 includes determining the zoom factor based on a physical object within the physical environment. To this end, in some implementations, method 300 includes: obtaining, via an image sensor, transparent image data defined by a field of view of a physical environment associated with the image sensor; identifying one or more physical objects within the physical environment within the transparent image data; and determining the zoom factor based on the one or more physical objects. For example, the zoom factor may be based on the corresponding position and / or size (e.g., volume) of the one or more physical objects, such as scaling down more when there are relatively large physical objects within the physical environment. In some implementations, the zoom factor is based on the size (e.g., volume) of the corresponding ER environment.

[0082] As shown in block 304, method 300 includes displaying, via a display device, a plurality of stereo model view representations from corresponding plurality of viewing vectors. The plurality of stereo model view representations correspond to a plurality of augmented reality (ER) environments. Each of the plurality of stereo model view representations is associated with a corresponding ER world coordinate set characterizing the corresponding ER environment. The plurality of stereo model view representations includes a first stereo model view representation of the plurality of stereo model view representations displayed from a first viewing vector. The first stereo model view representation of the plurality of stereo model view representations includes a first or more ER objects of the ER objects arranged according to a first ER world coordinate set. As another example, each of the plurality of stereo model view representations corresponds to a bounded area associated with a corresponding ER world coordinate set. An orientation, position, etc. of a particular plurality of ER objects within the corresponding stereo model view representation is based on the corresponding ER world coordinate set. The ER objects may include a combination of movable inanimate objects (e.g., a chair or table), animate objects (e.g., one or more avatars representing an individual), and inanimate structural objects (e.g., a ceiling, a wall, a boundary of a stereo model view representation). In some implementations, the electronic device displays the plurality of stereo model view representations based on determining that a spatial proximity threshold is satisfied. For example, the stereo model view representations are spaced far enough apart from each other so as to avoid mutual occlusion. A specific viewing vector provides one or more of the following: field of view (FOV), pose / rotation coordinates, translation coordinates, viewing angle, and the like.

[0083] In some implementations, method 300 includes performing a home ER setting (such as Figures 2A to 2VIn the home ER environment 206 in FIG. 1 , multiple stereoscopic model views are displayed. As an example, referring to FIG. Figure 2A , the electronic device 203 displays a first stereo model view representation 230 of a first ER environment and a second stereo model view representation 240 of a second ER environment within the home ER environment 206. Continuing with the example, the first stereo model view representation 230 includes a television 234, an avatar 236, and a sofa 238 arranged according to the first ER world coordinates 232. Continuing with the example, the second stereo model view representation 240 includes a sideboard 244 and a chair 246 arranged according to the second ER world coordinates 242.

[0084] In some implementations, the method 300 includes displaying, within a particular one of the plurality of stereo model view representations, a record of activities within a corresponding ER environment associated with the particular one of the plurality of stereo model view representations. In some implementations, the record includes activities within the particular one of the plurality of stereo model view representations over a previous period of time. For example, the record includes activities over the past day indicating that five persons entered and left the corresponding ER environment, and three of the persons drew on a whiteboard within the corresponding ER environment.

[0085] In some implementations, the electronic device displays multiple stereo model view representations based on control values. For example, these control values ​​can be set via user input. The control values ​​can include, for example, a combination of whether to mute, whether the preview user can communicate with individuals who have joined the first ER session, and / or whether the preview user sees a history (e.g., a log) of who has joined and left and when.

[0086] In some specific implementations, the electronic device displays multiple stereo model view representations based on eye gaze data. To this end, method 300 includes obtaining eye gaze data indicating an eye gaze position, wherein displaying multiple stereo model view representations is based on the eye gaze data. For example, when the eye gaze data indicates that the eye gaze position is closest to a specific one of the multiple stereo model view representations, the electronic device renders and displays the specific one of the multiple stereo model view representations with high resolution (e.g., emphasized). For another example, when the eye gaze data indicates that the eye gaze position is not within a threshold distance from another stereo model view representation of the multiple stereo model view representations for a threshold amount of time, the electronic device does not display the other stereo model view representation of the multiple stereo model view representations. For another example, the electronic device stops performing real-time updates (e.g., keeping static) on the content of a portion of the multiple stereo model view representations that is not within a threshold distance from the eye gaze position.

[0087] As shown in block 306, in some implementations, the plurality of stereoscopic model view representations correspond to reduced-size representations of the corresponding plurality of ER environments. As an example, Figure 2N The television 234 in the first stereoscopic model view representation 230 corresponds to Figure 2U 282 is a reduced size version of the real-size television 282 shown. In some implementations, the number of ER objects within a particular stereo model view representation is the same as or less than the number of ER objects with the corresponding ER environment. For example, the method 300 includes selectively rendering and / or displaying certain objects, such as displaying life objects (e.g., avatars 236) but not rendering structural features of the home ER environment (e.g., walls). In some implementations, the method 300 includes rendering different features of a particular stereo model view representation at different fidelity levels (e.g., resolutions). By selectively rendering and / or displaying, the electronic device reduces utilization of processing and battery resources.

[0088] As shown in block 308, in some implementations, displaying the plurality of stereoscopic model view representations includes displaying a portion of the plurality of stereoscopic model view representations in an animated manner. Figure 2K and Figure 2L , the electronic device 203 displays the movement of the avatar 236 as the individual associated with the avatar 236 is moving within his or her corresponding physical environment, resulting in the resulting movement of his or her avatar 236 within the ER environment.

[0089] As shown in block 310, in some implementations, at least a subset of the corresponding plurality of ER environments are respectively associated with a plurality of ER environments corresponding to the plurality of ER sessions. Each of the plurality of ER environments enables graphical representations of individuals to appear simultaneously within the ER environments. According to various implementations, each of the one or more corresponding individuals is associated with a corresponding access level that satisfies access level criteria associated with the corresponding ER session, such as by Figure 1 Thus, for example, a particular individual who does not have a pre-existing relationship with any of the individuals currently associated with the ER environment cannot view the corresponding stereo model view representation.

[0090] As shown in block 312, in some implementations, at least a subset of the plurality of stereoscopic model view representations includes one or more ER representations (e.g., avatars) of one or more corresponding individuals. Each of the one or more corresponding individuals is associated with a corresponding ER session. As an example, referring to Figure 2A, the electronic device 203 displays an avatar 236 associated with an ER session, which is itself associated with a first ER environment. In some specific implementations, the method 300 includes providing two-way audio between the electronic device and the avatar within the stereo model view representation. To this end, the method 300 includes playing a first set of voice data via a speaker of the electronic device while displaying multiple stereo model view representations. The first set of voice data is associated with one or more corresponding individuals, and the one or more corresponding individuals are associated with a specific ER session, and the specific ER session is associated with the corresponding ER environment. For example, the first set of voice data is emitted by the one or more corresponding individuals. For another example, the playback of the first set of voice data is real-time or delayed via buffering. For another example, the first set of voice data can be spatialized so that it sounds like it comes from within the stereo model view representation. In addition, the method 300 includes obtaining a second set of voice data from a user associated with the electronic device via an audio sensor of the electronic device. In addition, the method 300 includes providing the second set of voice data to the corresponding ER environment so that the second set of voice data can be heard by one or more corresponding individuals associated with the specific ER session. The second set of voice data can be presented to participants of the corresponding ER environment in a variety of ways, such as spatialized audio from above the corresponding ER environment.

[0091] As shown in block 314, in some implementations, a first stereo model view representation in the plurality of stereo model view representations corresponds to an augmented reality (AR) representation of the first ER environment. For example, the first stereo model view representation in the plurality of stereo model view representations includes AR content superimposed on environmental data associated with physical features of the operating environment. For example, referring to Figure 2A , the sofa 238 within the first stereoscopic model view representation 230 can correspond to a physical sofa associated with the first ER environment. In some implementations, the method 300 includes acquiring (e.g., via an image sensor) environmental data (e.g., pass-through image data), generating ER objects, and overlaying the ER objects on the environmental data.

[0092] As shown in block 316, in some implementations, a first stereo model view representation of the plurality of stereo model view representations corresponds to a virtual reality (VR) representation of the first ER environment. For example, the second stereo model view representation 240, including structural objects (e.g., floors and walls), the sideboard 244, and the chair 246 correspond to ER objects.

[0093] As shown in block 318, method 300 includes detecting, via one or more input devices, an input associated with a first stereo model view representation in the plurality of stereo model view representations. As shown in block 320, in some implementations, the input is directed to the first stereo model view representation in the plurality of stereo model view representations. As an example, referring to Figure 2H, the manipulation input 253 points to the first stereo model view representation 230. For example, the input changes the orientation (e.g., rotation) of the first stereo model view representation in the plurality of stereo model view representations. For another example, the input moves the first stereo model view representation in the plurality of stereo model view representations along an axis, such as translating the first stereo model view representation in the plurality of stereo model view representations along an xy axis.

[0094] As shown in block 322, in some implementations, the input corresponds to a change in position of the electronic device relative to a first stereo model view representation in the plurality of stereo model view representations from a first pose to a second pose. As shown in block 324, in some implementations, the input points to the plurality of stereo model view representations. For example, the electronic device includes an IMU that detects a change in position of the electronic device. As an example, referring to Figure 2D , the electronic device 203 detects a movement input 251 corresponding to the user 210 moving around the first stereo model view representation 230 and the second stereo model view representation 240 within the operating environment 200. In response to detecting the movement input 251, the electronic device 203 changes the first stereo model view representation 230 and the second stereo model view representation 240 from a first posture to a second posture.

[0095] As indicated at block 326, method 300 includes, in response to detecting the input, changing display of a first stereo model view representation of the plurality of stereo model view representations from the first viewing vector to a second viewing vector while maintaining the first one or more ER objects arranged according to the first ER world coordinate set. As an example, referring to Figure 2D and Figure 2E In response to detecting the movement input 251, the electronic device 203 changes the respective viewing vectors associated with the first stereo model view representation 230 and the second stereo model view representation 240 accordingly, as indicated by the respective changes to the first ER world coordinate 232 and the second ER world coordinate 242. For another example, referring to Figure 2H and Fig.2I In response to detecting the manipulation input 253 directed toward the first stereo model view representation 230 , the electronic device 203 correspondingly rotates the first stereo model view representation 230 , as indicated by a change in the first ER world coordinates 232 relative to the home ER world coordinates 208 .

[0096] Figure 4 is a flow chart of a method 400 for converting spatial relationships between a subset of ER objects according to respective coordinates associated with corresponding stereo model view representations and home ER world coordinates according to some implementations. In various implementations, the method 400 or a portion thereof is performed by an electronic device (e.g., Figure 1 The electronic device 100, Figures 2A to 2VIn various implementations, method 400 or a portion thereof is performed by an HMD. In some implementations, method 400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some embodiments, method 400 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). In various implementations, some operations in method 400 are optionally combined, and / or the order of some operations is optionally changed.

[0097] As shown in block 402, in some implementations, method 400 includes displaying a plurality of stereoscopic model view representations corresponding to a plurality of ER environments within a home ER environment. The plurality of stereoscopic model view representations includes a first stereoscopic model view representation of a first ER environment. As an example, referring to Figure 2A , the electronic device 203 displays a first stereo model view representation 230 of a first ER environment and a second stereo model view representation 240 of a second ER environment within the home ER environment 206. Continuing with the example, the first stereo model view representation 230 includes a television 234, an avatar 236, and a sofa 238 arranged according to the first ER world coordinates 232. Continuing with the example, the second stereo model view representation 240 includes a sideboard 244 and a chair 246 arranged according to the second ER world coordinates 242.

[0098] In some implementations, as shown in block 404, method 400 includes detecting a selection input for selecting a first stereoscopic model view representation from the plurality of stereoscopic model view representations. Figure 2F , the electronic device 203 detects a selection input 252 pointing to the first stereo model view representation 230. In some specific implementations, as shown in block 406, the selection input is based on eye gaze data acquired by, for example, an eye tracking sensor integrated in the electronic device or HMD. For example, the eye gaze data indicates that the user's eye gaze position points to the first stereo model view representation in the multiple stereo model view representations.

[0099] As shown in block 408, method 400 includes displaying a home ER environment represented by the home ER world coordinates, including a first stereoscopic model view representation of a first ER environment. The first stereoscopic model view representation includes one or more of the ER objects arranged in a spatial relationship according to the first ER world coordinates. In some implementations, displaying the first stereoscopic model view representation includes stopping displaying the remaining of the plurality of stereoscopic model view representations. In some implementations, displaying the first stereoscopic model view representation includes displaying the remaining of the plurality of stereoscopic model view representations at a lower resolution than the first stereoscopic model view representation. In some implementations, displaying the first stereoscopic model view representation includes stopping performing real-time updates on the content of the first stereoscopic model view representation. In some implementations, the first stereoscopic model view representation corresponds to a reduced-size representation of the first ER environment. For example, the one or more ER objects include a combination of movable objects (e.g., a chair or table), life objects (e.g., one or more avatars representing corresponding individuals), structural objects (e.g., a ceiling, a wall, a border of the stereoscopic model view representation), an anchor object (e.g., a bookshelf), etc. The orientation, position, etc. of the one or more ER objects are based on the first ER world coordinates. In some implementations, as shown in block 410, the home ER environment includes one or more physical objects associated with home ER world coordinates. Figure 2A , the home ER environment 206 includes a physical table 220 .

[0100] As shown in block 412, method 400 includes detecting a first input directed to a first stereoscopic model view representation. Figure 2N , accepting input 259 accepts the invitation to join the ER session associated with the first stereo model view representation 230. For example, the first input is associated with an interest in initiating (e.g., joining or displaying) the first ER environment, such as voice input, picking up or dropping the first stereo model view representation (e.g., as detected via a hand sensor), etc.

[0101] As shown in block 414, method 400 includes, in response to detecting the first input, transforming the home ER environment. According to some implementations, transforming the home ER environment includes ceasing to display the first stereoscopic model view representation within the home ER environment, and transforming a spatial relationship between a subset of one or more ER objects according to the home ER world coordinates and the first ER world coordinates. As shown in block 416, method 400 includes displaying the subset of the one or more ER objects within the home ER environment based on the transformation. As an example, in response to detecting the selection Figure 2N In response to receiving input 259 of the first stereoscopic model view representation 230, the electronic device 203 displays a bounded ER environment 261 including a transformed television 264, a transformed avatar 266, and a transformed sofa 268, as shown in FIG. Fig.2O As shown. For example, the subset of the one or more ER objects corresponds to the corresponding representation of the corresponding ER object. For another example, the subset of the one or more ER objects may include some or all of the one or more ER objects. In some implementations, transforming the spatial relationship includes increasing the corresponding size of the subset of the one or more ER objects compared to the same object in the first stereo model view representation. In some implementations, transforming the spatial relationship includes increasing the distance between the subset of the one or more ER objects compared to the same object in the first stereo model view representation while maintaining the relative distance between the subset of the one or more ER objects. For example, in the first stereo model view representation, the ER stapler is 1 inch away from the ER table and 3 inches away from the ER wall, and the transformation changes the distance between the ER stapler and the ER table to 2 inches and the distance between the ER stapler and the ER wall to 6 inches. In some implementations, transforming the spatial relationship includes moving the subset of the one or more ER objects relative to the home ER environment, such as dragging the subset of the one or more ER objects along the xy axis. In some implementations, transforming the spatial relationship includes continuously increasing the corresponding size of the subset of the one or more ER objects until a true size representation of the subset of the one or more ER objects is achieved.

[0102] In some implementations, method 400 includes, in response to detecting the first input, displaying the subset of the one or more ER objects from a first view vector, as shown in block 418. For example, a particular view vector provides one or more of: field of view (FOV), pose / rotation coordinates, translation coordinates, viewing angle, etc.

[0103] In some implementations, as shown in block 420, method 400 includes moving the subset of the one or more ER objects based on the identified physical object. To this end, method 400 includes acquiring, via an image sensor, environmental data (e.g., pass-through image data) defined by a field of view associated with the image sensor, wherein the environmental data is associated with a physical environment including the one or more physical objects. In addition, method 400 includes identifying within the environmental data a particular one of the one or more physical objects that is within a spatial proximity threshold of the subset of the one or more ER objects, and moving the subset of the one or more ER objects relative to the one or more physical objects based on the particular one of the one or more physical objects. For example, method 400 includes identifying the particular one of the one or more physical objects via, for example, instance segmentation and / or semantic segmentation. For another example, the particular one of the one or more physical objects is larger than a volume threshold and therefore may occlude the subset of the one or more ER objects. In some implementations, the particular one of the one or more physical objects satisfies the spatial proximity threshold relative to the subset of the one or more ER objects. For example, when the distance of the particular one of the one or more physical objects from the subset of the one or more ER objects is less than a threshold distance, the spatial proximity threshold is satisfied. In some implementations, method 400 includes displaying an indicator indicating the particular one of the one or more physical objects, such as a color overlay or animation on a surface of a portion of the particular one of the one or more physical objects (e.g., “Don't move the ER environment down because there is a big wall there”).

[0104] In some embodiments, as shown in box 422, method 400 includes detecting a second input. As an example, the second input is directed to a subset of the one or more ER objects. For example, the second input is directed to a separate enable representation, such as pressing the left arrow to move the stereo model view representation to the left. In some embodiments, the second input has three degrees of freedom. For example, the stereo model view representation can rotate, rotate and / or translate along the xz plane, but not move up or down in the y direction. In some embodiments, the second input is detected via an inertial measurement unit (IMU) such as a gyroscope or accelerometer. For example, the second input corresponds to a change in position of the electronic device from a first posture to a second posture relative to the subset of the one or more ER objects.

[0105] In some implementations, as shown in block 424, the second input is detected via a hand tracking sensor or an eye tracking sensor. To this end, in some implementations, method 400 includes: detecting the second input via a hand tracking sensor; acquiring hand tracking data from the hand tracking sensor based on the second input; and determining that the second input is directed to the subset of the one or more ER objects based on the hand tracking data. For example, method 400 includes comparing a hand position indicated by the hand tracking data with a position of the subset of the one or more ER objects. In some implementations, method 400 includes: detecting the second input via an eye tracking sensor; acquiring eye tracking data from the eye tracking sensor based on the second input; and determining that the second input is directed to the subset of the one or more ER objects based on the eye tracking data. For example, method 400 includes determining the second input directed to the subset of the one or more ER objects by comparing an eye gaze position indicated by the eye gaze data with a position of the subset of the one or more ER objects.

[0106] In some implementations, as shown in block 426, method 400 includes, in response to detecting the second input, changing the display of the subset of the one or more ER objects from the first viewing vector to the second viewing vector while maintaining the subset of the one or more ER objects arranged according to the first ER world coordinate. For example, changing the display of the subset of the one or more ER objects from the first viewing vector to the second viewing vector may include a combination of rotating, flipping, or translating along a fixed axis by a nominal rotation amount. As an example, in response to detecting the second input, Figure 2R , the electronic device 203 moves the transformed television 264, the transformed avatar 266, and the transformed sofa 268 downward from above the physical table 220 along the z-axis of the home ER world coordinate 208 to Figure 2S Continuing with the example, the electronic device 203 maintains the arrangement of the transformed television 264, the transformed avatar 266, and the transformed sofa 268 according to the transformed ER world coordinates 262. In some implementations, as shown in block 428, based on the change in position of the electronic device from the first pose to the second pose, the display of the subset of the one or more ER objects is changed from the first view vector to the second view vector. In some implementations, as shown in block 430, the changing the display of the subset of the one or more ER objects from the first view vector to the second view vector includes moving the subset of the one or more ER objects relative to the one or more physical objects.

[0107] In some implementations, as shown in block 432, method 400 includes detecting an input requesting entry into a first ER environment. Figure 2T, the input 274 points to the bounded ER environment 261 and corresponds to a request to enter the bounded ER environment 261. For example, the input 274 corresponds to one of an eye gaze input, a gesture, a voice input, a touch input, etc. For another example, the input 274 corresponds to placing or dropping the stereo model view representation on the ground so that the stereo model view representation can be expanded to a real size, or so that the stereo model view representation can replace the current environment.

[0108] In some implementations, as shown in block 434, method 400 includes, in response to detecting an input requesting entry into the first ER environment, adding the subset of the one or more ER objects to the home ER environment. As an example, in response to detecting an input requesting entry into the first ER environment, Figure 2T 274, the electronic device 203 adds the transformed television 264, the transformed avatar 266, and the transformed sofa 268 to the home ER environment 206 including the physical table 220, as shown in FIG. Figure 2U In some implementations, adding the subset of one or more ER objects to the home ER environment includes overlaying the subset of one or more ER objects on physical objects within the home ER environment. In some implementations, adding the subset of one or more ER objects to the home ER environment includes skinning physical features (e.g., structural features such as walls, floors) of the first ER environment and adding physical features in addition to adding the subset of one or more ER objects to the home ER environment.

[0109] In some implementations, as shown in block 436, method 400 includes, in response to detecting an input requesting entry into the first ER environment, replacing the home ER environment with the first ER environment that includes a subset of the one or more ER objects. For example, replacing the home ER environment with the first ER environment corresponds to replacing the home ER environment with a pure VR environment. As an example, in response to detecting an input requesting entry into the first ER environment, replacing the home ER environment with the first ER environment that includes a subset of the one or more ER objects. Figure 2T , the electronic device 203 replaces the home ER environment 206 with the first ER environment 290, such as Figure 2V As shown. Figure 2U Unlike the illustrated home ER environment 206 , the first ER environment 290 corresponds to a pure VR environment and therefore does not include the physical table 220 .

[0110] The present disclosure describes various features, none of which can achieve the benefits described herein alone. It should be understood that the various features described herein can be combined, modified or omitted, which is obvious to those of ordinary skill in the art. Other combinations and sub-combinations other than those specifically described herein will be obvious to those of ordinary skill and are intended to form a part of the present disclosure. Various methods are described herein in conjunction with various flow chart steps and / or stages. It should be understood that in many cases, certain steps and / or stages can be combined together so that the multiple steps and / or stages shown in the flow chart can be performed as a single step and / or stage. In addition, certain steps and / or stages can be divided into additional sub-components to be performed independently. In some cases, the order of steps and / or stages can be rearranged, and certain steps and / or stages can be completely omitted. In addition, the methods described herein should be understood to be broadly interpretable so that additional steps and / or stages other than those shown and described herein can also be performed.

[0111] Some or all of the methods and tasks described herein may be performed and fully automated by a computer system. In some cases, a computer system may include multiple different computers or computing devices (e.g., physical servers, workstations, storage arrays, etc.) that communicate and interoperate over a network to perform the functions. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in a memory or other non-transient computer-readable storage medium or device. The various functions disclosed herein may be implemented in such program instructions, but alternatively may be implemented in a dedicated circuit (e.g., ASIC or FPGA or GP-GPU) of a computer system Some or all of the disclosed functions. In the case where a computer system includes multiple computing devices, these devices may be located in the same location or not in the same location. The results of the disclosed methods and tasks may be persistently stored by converting physical storage devices such as solid-state memory chips and / or disks into different states.

[0112] The various processes defined herein contemplate the option of obtaining and utilizing a user's personal information. For example, such personal information may be utilized to provide an improved privacy screen on an electronic device. However, to the extent such personal information is collected, such information should be obtained with the user's informed consent. As described herein, users should understand and control the use of their personal information.

[0113] Personal information will be used by appropriate parties only for lawful and reasonable purposes. Parties utilizing such information will adhere to privacy policies and practices that at a minimum comply with applicable laws and regulations. In addition, such policies should be comprehensive, user accessible, and recognized to meet or exceed government / industry standards. In addition, parties may not distribute, sell, or otherwise share such information except for any reasonable and lawful purpose.

[0114] However, users may limit the extent to which parties may access or otherwise obtain personal information. For example, settings or other preferences may be adjusted so that users may determine whether their personal information may be accessed by various entities. Furthermore, while some features defined herein are described in the context of using personal information, aspects of these features may be implemented without the use of such information. For example, if user preferences, account names, and / or location history are collected, that information may be obfuscated or otherwise generalized so that the information does not identify the respective user.

[0115] The present disclosure is not intended to be limited to the specific implementations shown herein. Various modifications to the specific implementations described in the present disclosure may be apparent to those skilled in the art, and the general principles defined herein may be applied to other specific implementations without departing from the essence or scope of the present disclosure. The teachings of the present invention provided herein may be applied to other methods and systems, and are not limited to the above methods and systems, and the elements and actions of the various specific implementations described above may be combined to provide more specific implementations. Therefore, the novel methods and systems described herein may be implemented in a variety of other forms; in addition, various omissions, substitutions, and changes may be made to the forms of the methods and systems described herein without departing from the essence of the present disclosure. The attached claims and their equivalents are intended to cover such forms or modified forms that fall within the scope and essence of the present disclosure.

Claims

1. A method for displaying a representation of an environment, comprising: At an electronic device including one or more processors, a non-transitory memory, one or more input devices, and a display device: displaying, via the display device, a plurality of stereo model view representations from a corresponding plurality of viewing vectors, wherein the plurality of stereo model view representations correspond to a plurality of augmented reality ER environments, wherein each of the plurality of stereo model view representations is associated with a respective ER world coordinate set characterizing the respective ER environment, wherein the plurality of stereo model view representations comprises a first stereo model view representation of the plurality of stereo model view representations displayed from a first viewing vector and a second stereo model view representation of the plurality of stereo model view representations displayed from a third viewing vector, and wherein the first stereo model view representation of the plurality of stereo model view representations comprises first one or more of the ER objects arranged according to a first ER world coordinate set; detecting, via the one or more input devices, an input associated with the first of the plurality of stereo model view representations; as well as In response to detecting the input, changing the display of the first stereo model view representation in the multiple stereo model view representations from the first viewing vector to a second viewing vector while maintaining the first one or more ER objects arranged according to the first ER world coordinate set, and without changing the display of the second stereo model view representation in the multiple stereo model view representations from a third viewing vector. 2 . The method of claim 1 , wherein the input is for the first stereo model view representation among the plurality of stereo model view representations.

3. The method of claim 2, wherein the one or more input devices include a hand tracking sensor, the method further comprising: detecting the input via the hand tracking sensor; acquiring hand tracking data from the hand tracking sensor based on the input; as well as It is determined from the hand tracking data that the input is for the first stereo model view representation of the plurality of stereo model view representations.

4. The method of claim 2, wherein the one or more input devices include an eye tracking sensor, the method further comprising: detecting the input via the eye tracking sensor; acquiring eye tracking data from the eye tracking sensor based on the input; as well as A determination is made based on the eye tracking data that the input is for the first stereo model view representation of the plurality of stereo model view representations. 5 . The method according to claim 1 , wherein the input corresponds to a change in position of the electronic device relative to the first stereo model view representation of the plurality of stereo model view representations from a first posture to a second posture.

6. The method of claim 5, wherein the one or more input devices include a position change sensor, the method further comprising: detecting the input via the position change sensor; acquiring position change data from the position change sensor based on the input; as well as The change in position of the electronic device is determined based on the position change data.

7. The method according to any one of claims 1 to 4, wherein the input is also associated with the second stereo model view representation in the multiple stereo model view representations, and wherein the second stereo model view representation in the multiple stereo model view representations includes a second one or more ER objects in the ER objects arranged according to a second ER world coordinate set, the method further comprising, in response to detecting the input, changing the display of the second stereo model view representation in the multiple stereo model view representations from the third viewing vector to a fourth viewing vector while maintaining the second one or more ER objects arranged according to the second ER world coordinate set.

8. The method of any one of claims 1 to 4, wherein the plurality of stereo model view representations correspond to reduced-scale representations of a corresponding plurality of ER environments.

9. The method of any one of claims 1 to 4, wherein the first stereo model view representation of the plurality of stereo model view representations corresponds to a virtual reality (VR) representation of a first ER environment.

10. The method of any one of claims 1 to 4, wherein the first of the plurality of stereo model view representations corresponds to an augmented reality (AR) representation of a first ER environment, and wherein the first of the plurality of stereo model view representations includes AR content overlaid on environment data associated with physical features of the first ER environment.

11. The method of any one of claims 1 to 4, wherein displaying the plurality of stereo model view representations comprises animating a portion of the plurality of stereo model view representations.

12. The method according to any one of claims 1 to 4, further comprising: obtaining a plurality of representation vectors that respectively provide a plurality of spatial representations of a corresponding plurality of ER environments, wherein each of the plurality of representation vectors includes a plurality of object label values ​​that respectively identify one or more ER objects, and wherein each of the plurality of representation vectors also includes a plurality of relative position values ​​that provide respective positions of the one or more ER objects relative to each other; as well as The plurality of stereo model view representations of the corresponding plurality of ER environments are generated from the corresponding plurality of ER environments and the plurality of characterization vectors according to the plurality of relative position values. 13 . The method of claim 12 , wherein generating the plurality of stereo model view representations comprises scaling down the corresponding plurality of ER environments according to a scaling factor.

14. The method of claim 13, further comprising obtaining a zoom request input specifying the zoom factor via the one or more input devices.

15. The method of claim 13, wherein the electronic device comprises an image sensor, the method further comprising: acquiring, via the image sensor, transmission image data defined by a field of view of a physical environment associated with the image sensor; identifying one or more physical objects in the physical environment within the transparent image data; as well as The scaling factor is determined based on the one or more physical objects.

16. The method of any one of claims 1 to 4, wherein at least a subset of the corresponding plurality of ER environments are respectively associated with a plurality of ER environments corresponding to a plurality of ER sessions that are different from one another, and wherein each of the plurality of ER environments enables an individual to perform graphical representations simultaneously within the ER environment.

17. The method of claim 16, wherein each of at least a subset of the plurality of stereo model view representations corresponding to at least the subset of the corresponding plurality of ER environments comprises one or more ER representations of one or more corresponding individuals associated with a corresponding ER session.

18. The method of claim 17, wherein each of the one or more corresponding individuals is associated with a corresponding access level that satisfies access level criteria associated with the corresponding ER session.

19. The method according to claim 16, further comprising: Playing a first set of voice data via a speaker of the electronic device while displaying the plurality of stereo model view representations, wherein the first set of voice data is associated with one or more corresponding individuals, the one or more corresponding individuals are associated with a specific ER session, and the specific ER session is associated with a corresponding ER environment; acquiring a second set of voice data from a user associated with the electronic device via an audio sensor of the electronic device; and The second set of voice data is provided to the corresponding ER environment so that the second set of voice data can be heard by the one or more corresponding individuals associated with the particular ER session.

20. The method of claim 16, wherein each of the at least one subset of the plurality of stereo model view representations corresponding to at least the subset of the corresponding plurality of ER environments is associated with a corresponding ER session, and wherein displaying the plurality of stereo model view representations is based on historical data about the electronic devices participating in the plurality of ER sessions.

21. The method according to any one of claims 1 to 4, further comprising acquiring eye gaze data indicating eye gaze positions via an eye tracking sensor, wherein displaying the plurality of stereo model view representations is based on the eye gaze data.

22. The method according to any one of claims 1 to 4, further comprising: receiving, via the one or more input devices, a stereo model selection input selecting the first stereo model view representation of the plurality of stereo model view representations; as well as In response to receiving the stereo model selection input: displaying, via the display device, an ER environment associated with the first stereo model view representation of the plurality of stereo model view representations; as well as The display of the plurality of stereo model view representations is stopped.

23. The method according to any one of claims 1 to 4, further comprising displaying, within a specific stereo model view representation among the multiple stereo model view representations, an activity record within a corresponding ER environment associated with the specific stereo model view representation among the multiple stereo model view representations.

24. The method according to any one of claims 1 to 4, wherein displaying the plurality of stereo model view representations is based on a control value.

25. An electronic device comprising: one or more processors; non-transitory memory; one or more input devices; Display devices; and One or more programs, wherein the one or more programs are stored in the non-volatile memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing or causing the execution of any of the methods according to claims 1 to 24.

26. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by an electronic device, cause the electronic device to perform or cause the performance of any one of the methods according to claims 1 to 24.

27. An electronic device comprising: one or more processors; non-transitory memory; one or more input devices; Display devices; and Means for performing or causing the performance of any of the methods according to claims 1 to 24.

28. An information processing apparatus for use in an electronic device, comprising means for performing or causing the performance of any of the methods according to claims 1 to 24.

29. A method for displaying a representation of an environment, comprising: At an electronic device including one or more processors, a non-transitory memory, one or more input devices, and a display device: displaying, via the display device, a home ER environment represented by the home ER world coordinates, including a first stereo model view representation of a first ER environment different from the home ER environment, wherein the first stereo model view representation includes one or more of the ER objects arranged in a spatial relationship according to the first ER world coordinates, and wherein the one or more ER objects are associated with a first appearance when displayed within the first stereo model view representation of the first ER environment; detecting, via the one or more input devices, a first input to the first stereo model view representation; and In response to detecting the first input, transforming the home ER environment by: ceasing to display the first stereoscopic model view representation within the home ER environment, transforming the spatial relationship between the subset of the one or more ER objects according to the home ER world coordinates and the first ER world coordinates, and Based on the transformation, the subset of the one or more ER objects within the home ER environment is displayed via the display device, and wherein the subset of the one or more ER objects is associated with a second appearance different from the first appearance when displayed within the home ER environment.

30. The method of claim 29, further comprising: detecting a second input via the one or more input devices while displaying the subset of the one or more ER objects within the home ER environment based on the transforming; as well as In response to detecting the second input, the subset of the one or more ER objects is added to the home ER environment.

31. The method of claim 29, further comprising: detecting a second input via the one or more input devices while displaying the subset of the one or more ER objects within the home ER environment based on the transforming; as well as In response to detecting the second input, the home ER environment is replaced with the first ER environment including the subset of the one or more ER objects.

32. The method according to any one of claims 29 to 31, wherein: In response to detecting the first input, displaying the subset of the one or more ER objects from a first view vector, the method further comprising: detecting a second input via the one or more input devices while displaying the subset of the one or more ER objects within the home ER environment based on the transforming; and In response to detecting the second input, changing display of the subset of the one or more ER objects from the first view vector to a second view vector while maintaining the subset of the one or more ER objects arranged according to the first ER world coordinate.

33. The method of claim 32, wherein the second input is for the subset of the one or more ER objects.

34. The method of claim 33, wherein the one or more input devices include a hand tracking sensor, the method further comprising: detecting the second input via the hand tracking sensor; acquiring hand tracking data from the hand tracking sensor based on the second input; as well as It is determined based on the hand tracking data that the second input is for the subset of the one or more ER objects.

35. The method of claim 33, wherein the one or more input devices include an eye tracking sensor, the method further comprising: detecting the second input via the eye tracking sensor; acquiring eye tracking data from the eye tracking sensor based on the second input; as well as A determination is made based on the eye tracking data that the second input is for the subset of the one or more ER objects.

36. A method according to claim 32, wherein the home ER environment includes one or more physical objects associated with the home ER world coordinates, and wherein changing the display of the subset of the one or more ER objects from the first viewing vector to the second viewing vector includes moving the subset of the one or more ER objects relative to the one or more physical objects.

37. A method according to claim 32, wherein the second input corresponds to a change in position of the electronic device relative to the subset of the one or more ER objects from a first posture to a second posture, and wherein changing the display of the subset of the one or more ER objects from the first viewing vector to the second viewing vector is based on the change in position of the electronic device from the first posture to the second posture.

38. The method according to any one of claims 29 to 31, further comprising: acquiring, via an image sensor, environmental data defined by a field of view associated with the image sensor, wherein the environmental data is associated with a physical environment including one or more physical objects; identifying within the environmental data a particular physical object of the one or more physical objects that is within a spatial proximity threshold of the subset of the one or more ER objects; and The subset of the one or more ER objects is moved relative to the one or more physical objects based on the particular physical object of the one or more physical objects.

39. The method according to any one of claims 29 to 31, further comprising: displaying, via the display device, a plurality of stereo model view representations corresponding to a plurality of ER environments within the home ER environment, wherein the plurality of stereo model view representations include the first stereo model view representation; as well as In response to detecting the first input, the first stereo model view representation is selected from the plurality of stereo model view representations.

40. The method according to any one of claims 29 to 31, further comprising: displaying, via the display device, a plurality of stereo model view representations corresponding to a plurality of ER environments within the home ER environment, wherein the plurality of stereo model view representations include the first stereo model view representation; as well as detecting, via the one or more input devices, a selection input selecting the first stereo model view representation from among the plurality of stereo model view representations; as well as In response to detecting the selection input, display of the first stereo model view representation within the home ER environment is maintained and display of the remaining ones of the plurality of stereo model view representations is ceased.

41. The method of claim 40, further comprising acquiring eye gaze data indicative of an eye gaze location via an eye tracking sensor, wherein the selection input is based on the eye gaze location.

42. The method of any one of claims 29 to 31, wherein the first ER environment is associated with an ER session, the ER session enabling individuals to simultaneously work with corresponding graphical representations within the first ER environment.

43. The method of claim 42, wherein the first ER environment includes one or more ER representations respectively associated with one or more individuals connected to the ER session.

44. The method of claim 43, wherein each of the one or more individuals has a respective access level that satisfies access level criteria associated with the ER session.

45. The method of claim 43, further comprising: When displaying the first stereo model view representation: playing, via a speaker of the electronic device, a first set of voice data associated with the one or more individuals connected to the ER session; acquiring a second set of voice data from a user associated with the electronic device via an audio sensor of the electronic device; and The second set of voice data is provided to the one or more individuals connected to the ER session.

46. ​​The method according to any one of claims 29 to 31, further comprising: obtaining a representation vector providing a spatial representation of the first ER environment, wherein the representation vector includes a plurality of object label values ​​that respectively identify the one or more ER objects, and wherein the representation vector further includes a plurality of relative position values ​​that provide respective positions of the one or more ER objects relative to each other; as well as The first stereo model view representation is generated according to the first ER environment and the characterization vector.

47. The method of any one of claims 29 to 31, wherein displaying the first stereo model view representation comprises animating a portion of the one or more ER objects.

48. An electronic device comprising: one or more processors; non-transitory memory; one or more input devices; Display devices; and One or more programs, wherein the one or more programs are stored in the non-volatile memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing or causing the execution of any of the methods described in claims 29 to 47.

49. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by an electronic device, cause the electronic device to perform or cause the performance of any of the methods of claims 29 to 47.

50. An electronic device comprising: one or more processors; non-transitory memory; one or more input devices; Display devices; and Means for performing or causing the performance of any of the methods according to claims 29 to 47.

51. An information processing apparatus for use in an electronic device, comprising means for performing or causing the performance of any of the methods of claims 29 to 47.

Images