Portal content for communication sessions

The problem of HMD's difficulty in capturing backgrounds is solved by using environmental sensor data to fill and update the background during a video chat session, and the accuracy and privacy of the background data are achieved.

CN119094502APending Publication Date: 2024-12-06APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410709044.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-05-31
Filing Date
2024-06-03
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

During a video chat session, headsets (HMDs) have difficulty capturing and updating participants’ background environments, resulting in problems with insufficient or inaccurate background representations.

Method used

By capturing environmental sensor data on the sender device, filling data gaps, and using parameters such as blurring, not depicting others, providing limited views, etc., to generate and update background data, ensure that some or all of the background environment is blurred, blocking the background. other people or objects, retain user privacy.

Benefits of technology

This enables the provision of actual background or background representation during a communication session, addressing HMD's limitations in updating users and background views, ensuring the accuracy and privacy of background data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094502A_ABST
    Figure CN119094502A_ABST
Patent Text Reader

Abstract

The invention relates to portal content for communication sessions. Devices, systems, and methods are disclosed that provide a view of a three-dimensional (3D) environment (e.g., a viewer room) having a portal that provides a view of a background environment of another user (e.g., a sender) during a communication session. For example, a process at a first electronic device may include presenting a view of a first 3D environment. Data representing the second 3D environment based at least in part on sensor data captured in the physical environment of the second electronic device may be obtained. Portal content based on the data representing the second 3D environment and viewpoints within the first 3D environment may be determined. A portal having the portal content may be displayed in the view of the first 3D environment, wherein the portal content depicts a portion of the second 3D environment viewed from the viewpoint through the portal.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to electronic devices that use sensors to provide views during a communication session, the views including views of representations of one or more of the environments of the electronic devices participating in the session. Background Art

[0002] Various techniques are used to present communication sessions, such as video conferencing, interactive gaming sessions, and other interactive social experiences. For example, participants may see realistic or non-realistic representations (e.g., avatars) of users participating in the session. However, a representation of at least a portion of the sender's environment (e.g., background) needs to be provided to give some context of where the sender is calling from. Summary of the invention

[0003] Various embodiments disclosed herein include devices, systems, and methods for providing a view of a three-dimensional (3D) environment (e.g., a viewer's room) having a portal that provides a representation of another user's environment (e.g., a sender's room) and a view of the user's representation (e.g., an avatar). The representation does not provide a fully immersive view, but may be displayed at a relatively small viewing portal at a fixed position within a larger 3D environment (e.g., a portal). Providing at least a portion of a representation of the sender's background within the portal may be intended to give some context of where the sender is calling from. However, a head mounted device (HMD) may be limited in updating the view of the user and the user's background because, from the perspective of a typical video chat session (e.g., positioning the camera 1 to 2 meters in front of the user to capture both the user and live background data), an outward-facing camera may not be able to capture the background environment during a communication session. Therefore, when two users wear an HMD during a communication session, if it is desired to provide an actual ("live") background or at least a representation of the background during the session, the system may utilize a previously captured image of at least a portion of the environment or the environment prior to the communication session, or the system may fabricate any gaps. Then, based on the sender's position relative to the background and / or the viewer's viewpoint position, the background may be displayed and updated accordingly.

[0004] In some implementations, contextual data may be provided by a sender device that captures sensor data of its environment, potentially filling in data gaps (e.g., fictitious content), and may be provided to viewer devices using parameters (e.g., blurring, not depicting other people, providing limited views (e.g., 180° OV), using update criteria based on changes / new content, etc.). Processing to determine contextual data for portal content (e.g., blurring, not depicting other people, providing limited views (e.g., 180° OV), using update criteria based on changes / new content, etc.) may be performed at the sender device, the viewer device, or a combination thereof. For example, the sender device may limit the amount of portal content sent to the viewer device such that the content may be blurred or a limited view of the background may be provided.

[0005] In some implementations, the portal may provide multi-directional views of other environments that change as the viewer moves relative to the portal (e.g., depending on the viewpoint). The portal may present a portal view of received 180° stereoscopic image / video background content on a plane / surface displayed in a 3D space (e.g., VR or MR). In some implementations, during the capture of the sender background, the sender device may update low-frequency screenshots (RGB images), provide a current depth map, and optionally include additional metadata such as head orientation / pose. The sender device and / or viewer device are able to fill in gaps / holes in the background (e.g., occlusions during room scans, unscanned portions of the room, etc.) and periodically provide updates to the viewer. For example, background data may be updated for portal content as the sender moves around their environment and provides additional views for the sender electronic device to capture additional sensor data.

[0006] In some implementations, certain features of the portal content can be limited in the amount of data provided, such as providing viewpoint-dependent views, privacy features that blur portions of the background, masking out people or other moving objects in the background, etc. Thus, user privacy can be preserved by providing only some of the user's background information (e.g., blurring portions or all of the background environment, not depicting other people or other objects in the background, providing limited views (e.g., 180° OV), using update criteria based on changes in the background environment and / or new content, etc.).

[0007] Generally speaking, one innovative aspect of the subject matter described in this specification can be embodied in a method of using a first electronic device including one or more processors, the method including the following acts: presenting a view of a first 3D environment; obtaining data representing a second 3D environment, the data representing the second 3D environment based at least in part on sensor data captured in a physical environment of the second electronic device; determining portal content based on the data representing the second 3D environment and a viewpoint within the first 3D environment; and displaying a portal with portal content in the view of the first 3D environment, wherein the portal content depicts a portion of the second 3D environment viewed through the portal from the viewpoint.

[0008] These and other embodiments can optionally include one or more of the following features.

[0009] In some aspects, the portal content is based on data that synthesizes a representation of a portion of the second 3D environment that is not represented in sensor data captured in the physical environment of the second electronic device.

[0010] In some aspects, the portal content is updated based on detecting a change in the second 3D environment or in the physical environment of the second electronic device.

[0011] In some aspects, obtaining data representative of the second 3D environment includes obtaining parameters associated with the data representative of the second 3D environment.

[0012] In some aspects, the parameter identifies a field of view or an orientation of the second 3D environment, and wherein determining the portal content is further based on the parameter.

[0013] In some aspects, determining the portal content includes blurring portions of the second 3D environment based on the identified field of view or orientation of the second 3D environment.

[0014] In some aspects, the method further includes an act of obtaining data representing a user of the second electronic device, wherein determining the portal content is further based on the data representing the user of the second electronic device, and wherein the portal content depicts a representation of the user of the second electronic device in front of a portion of the second 3D environment. In some aspects, determining the portal content includes blurring a portion of the second 3D environment that is behind the representation of the user of the second electronic device. In some aspects, the data representing the second 3D environment depicts a less than 360-degree view of the second 3D environment. In some aspects, the data representing the second 3D environment depicts a 360-degree view of the second 3D environment.

[0015] In some aspects, the method further includes an act of determining a location to display the portal within the view of the first 3D environment based on the viewpoint. In some aspects, the method further includes an act of changing content of the portal based on a change in the viewpoint within the first 3D environment.

[0016] In some aspects, displaying the portal with portal content in the view of the first 3D environment is based on determining a positional relationship of the viewpoint relative to the portal. In some aspects, the position of the portal within the first 3D environment is constant as the viewpoint changes within the first 3D environment. In some aspects, the position of the portal within the first 3D environment changes based on changes in the viewpoint within the first 3D environment.

[0017] In some aspects, the data representing the second 3D environment includes a stereoscopic image pair including left-eye content corresponding to a left-eye viewpoint and right-eye content corresponding to a right-eye viewpoint. In some aspects, the data representing the second 3D environment includes a 180-degree stereoscopic image. In some aspects, the data representing the second 3D environment includes two-dimensional (2D) image data and depth data.

[0018] In some aspects, determining portal content includes rendering at least a portion of data representing a second 3D environment on at least a portion of the sphere. In some aspects, the data representing the second 3D environment includes a three-dimensional (3D) model. In some aspects, the data representing the second 3D environment is obtained during a communication session between the first electronic device and the second electronic device.

[0019] In some aspects, sensor data captured in the physical environment of the second electronic device is obtained by one or more sensors of the second electronic device. In some aspects, the first 3D environment is an extended reality (XR) environment. In some aspects, the first electronic device or the second electronic device includes a head mounted device (HMD).

[0020] Generally speaking, one innovative aspect of the subject matter described in this specification may be embodied in a method of using a first electronic device comprising one or more processors and one or more sensors, the method comprising the following actions: obtaining sensor data captured via one or more sensors in a physical environment associated with the first electronic device; determining data representing a first three-dimensional (3D) environment, wherein the data representing the first 3D environment is generated at least in part based on the sensor data and parameters identifying an orientation or field of view of the first electronic device; and providing the data representing the first 3D environment to a second electronic device.

[0021] These and other embodiments can optionally include one or more of the following features.

[0022] In some aspects, determining data representative of the first 3D environment includes synthesizing data representative of portions of the first 3D environment that are not represented in sensor data captured in a physical environment of the first electronic device.

[0023] In some aspects, data synthesis is performed based on detecting a change in position of the first electronic device within the first 3D environment. In some aspects, data synthesis is performed based on identifying another portion of the first 3D environment that is not represented in the sensor data.

[0024] In some aspects, the data representing the first 3D environment is updated based on detecting a change in the first 3D environment. In some aspects, the data representing the first 3D environment is updated based on detecting a change in the physical environment of the second electronic device. In some aspects, the data representing the first 3D environment is updated based on detecting a change in the position of the first electronic device exceeding a threshold.

[0025] In some aspects, the method further includes the acts of determining a first lighting condition associated with an area of ​​the first 3D environment based on the data representing the first 3D environment; and updating the data representing the first 3D environment for the area associated with the first lighting condition in the first 3D environment.

[0026] In some aspects, determining data representing the first 3D environment includes determining coverage of a background associated with a physical environment of the first electronic device based on the sensor data, and in response to determining that the coverage of the captured background of the physical environment is below a threshold amount, providing synthetic data as the data representing the first 3D environment.

[0027] In some aspects, the blur effect is applied by the first electronic device to at least a portion of the data representing the first 3D environment provided to the second electronic device. In some aspects, the parameters identifying the orientation or field of view of the first electronic device are based on determining a posture of the first electronic device. In some aspects, the method also includes an action of providing data representing a user of the first electronic device to the second electronic device. In some aspects, the data representing the user of the first electronic device is provided at a higher frequency than the data representing the first 3D environment.

[0028] In some aspects, the second electronic device is configured to display a view of a portion of the data representing the first 3D environment within a portal within the view of the second 3D environment. In some aspects, the first electronic device and the second electronic device operatively communicate during the communication session. In some aspects, the first electronic device or the second electronic device includes an HMD.

[0029] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0031] Figure 1A to Figure 1B Example electronic devices are shown operating in a physical environment according to some implementations.

[0032] Figure 2 According to some specific implementations Figure 1A to Figure 1B A view of a three-dimensional (3D) physical environment provided by a device.

[0033] Figure 3 According to some specific implementations, Figure 1A to Figure 1B Example electronic devices operating in different physical environments.

[0034] Figure 4 According to some specific implementations, Figure 3 An exemplary 3D environment generated by a physical environment and a portal displaying portal content within the 3D environment.

[0035] Figure 5 According to some specific implementations, Figure 4 Example interactions of portal content displayed within a portal.

[0036] Figure 6 Example electronic devices are shown operating in different physical environments during a communication session according to some implementations.

[0037] 7-10 illustrate example environments for displaying a portal with portal content in a view of a first 3D environment that depicts a portion of a second 3D environment viewed through the portal from a different viewpoint, according to some implementations.

[0038] Fig.11is a process flow diagram illustrating an exemplary process of providing portal content based on data from a first 3D environment for display in a portal within a view of a second 3D environment, according to some implementations.

[0039] Fig.12 is a flow chart illustrating a method for displaying a portal having portal content in a view of a first 3D environment, the portal content depicting a portion of a second 3D environment viewed through the portal from a different viewpoint, according to some implementations.

[0040] Fig.13 is a flow chart illustrating another method for displaying a portal having portal content in a view of a first 3D environment that depicts a portion of a second 3D environment viewed through the portal from a different viewpoint, according to some implementations.

[0041] Fig.14 is a block diagram of an electronic device according to some specific implementations.

[0042] Fig.15 is a block diagram of a head mounted device (HMD) according to some specific implementations.

[0043] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, for clarity, the sizes of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features.

[0044] describe

[0045] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0046] Figure 1A to Figure 1B Exemplary electronic devices 105 and 110 are shown operating in physical environment 100. Figure 1A to Figure 1BIn the example of , the physical environment 100 is a room including a table 120. The electronic devices 105 and 110 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 100 and objects therein, as well as information about the user 102 of the electronic devices 105 and 110 and evaluate the physical environment and objects therein. The information about the physical environment 100 and / or the user 102 can be used to provide visual and audio content and / or to identify the current location of the physical environment 100 and / or the location of the user within the physical environment 100.

[0047] In some implementations, a view of an extended reality (XR) environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via electronic devices 105 (e.g., wearable devices such as HMDs, etc.) and / or 110 (e.g., handheld devices such as mobile devices, tablet computing devices, laptop computers, etc.). Such an XR environment may include a view of a 3D environment generated based on camera images and / or depth camera images of the physical environment 100 and a representation of the user 102 based on the camera images and / or depth camera images of the user 102. Such an XR environment may include virtual content positioned at a 3D position relative to a 3D coordinate system (e.g., 3D space) associated with the XR environment, which 3D coordinate system may correspond to the 3D coordinate system of the physical environment 100.

[0048] In some embodiments, video (e.g., pass-through video depicting a physical environment) is received from an image sensor of a device (e.g., device 105 or device 110) and used to render an XR environment. In other embodiments, optical perspective may be used to render an XR environment by overlaying virtual content on a view of the physical environment seen through a translucent or transparent display. In some embodiments, a 3D representation of the virtual environment is aligned with a 3D coordinate system of the physical environment. The size of the 3D representation of the virtual environment may be generated based on, in particular, the scale of the physical environment or the positioning of open spaces, floors, walls, etc., so that the 3D representation is configured to align with corresponding features of the physical environment. In some embodiments, a viewpoint within a 3D coordinate system may be determined based on the position of the electronic device within the physical environment. The viewpoint may be determined based on, in particular, image data, depth sensor data, motion sensor data, etc., which may be retrieved via a virtual inertial odometer system (VIO), a simultaneous localization and mapping (SLAM) system, etc.

[0049] Figure 2 2 is an example of an operating environment 200 of a device (e.g., device 105 and / or device 110) used within physical environment 100 and an example view 205 from the device according to some implementations. Specifically, operating environment 200 shows user 102 viewing an example of a device 105 and / or device 110 in physical environment 100. Figure 1A to Figure 1B120 in the physical environment 100. As shown, in operating environment 200, user 102 has placed a device (e.g., device 105 and / or device 110) at a far edge of table 120 to initiate a scan of environment 100. Operating environment 200 illustrates, for example, a process of creating an environment representation 210 of the current physical environment 100 to identify key features of the physical environment 100. A handheld device (e.g., device 110) is shown, however, an HMD (e.g., device 105) may also be used to capture the background of the physical environment 100 (e.g., portions of the environment behind the user) that may be utilized during a communication session. For example, Figure 2 The background scene of the captured environment is shown so that when the user wears the HMD (e.g., device 105) during a communication session, the system can use previously captured background scene sensor data (e.g., when the HMD was previously facing the background scene) to generate a representation of the background, because the HMD will not be able to capture background data during a live communication session unless the user faces the HMD toward the area. In other words, because the sensors (e.g., cameras) on the HMD are typically positioned close to the user's face to capture facial and body images and capture images of the environment from the user's perspective, the HMD is typically unable to capture live image data of the background from another user's perspective (e.g., the example viewpoint 214 of the device sitting at a table).

[0050] The environment representation 210 shows an example representation of the physical environment 100 from a viewpoint 214 corresponding to the perspective of the electronic device 105 / 110 as depicted by the position indicator 212. The environment representation 210 includes appearance and / or location / positioning information as indicated by objects 222 (e.g., wall hangings 150), objects 224 (e.g., plants 125), and objects 226 (e.g., table 120). In addition, the environment representation 210 identifies the appearance and / or location of the user 102, as shown in the representation 220. In some embodiments, the environment representation 210 may include a representation of the environment 100 generated using previously captured scene sensor data (e.g., for the portion of the physical environment 100 behind the user 102) and a representation of the user 102 using current sensor data. In these embodiments, a representation for the portion of the physical environment 100 in front of the user 102 may not be included in the environment representation 210 (e.g., the object 226 representing the table 120 may not be included).

[0051] like Figure 2As shown, device 105 or device 110 can use environment representation 210 to provide a view 205 of 3D environment 250 from the perspective of device 105 or device 110 (e.g., from the perspective of an XR environment such as a position indicator 212 of a forward-facing camera or representing a forward-facing camera view of device 105 or device 110). For example, view 205 shows 3D environment 250, including representation 260 of plant 125, representation 265 of wall hanging 150, representation 270 of table 120, and representation 280 of user 102. As described above, in some implementations, representations of portions of environment 100 that are located in front of user 102 may not be included in environment representation 210 and may not be presented in view 205 (e.g., representation 270 of table 120). Representations 260, 265, 270 may be images of actual objects (e.g., videos), may be views of each physical object as seen through a transparent or semi-transparent display, may be virtual content representing each physical object, or representations 260, 265, 270 may be a combination of virtual content and images and / or pass-through video (e.g., an XR experience). Similarly, representation 280 of user 102 may be an actual video of user 102, may be generated virtual content (e.g., an avatar) representing user 102, or may be a view of user 102 as seen through a transparent or semi-transparent display, as further described herein. In some embodiments, as further described below, in addition to or alternatively to presenting view 205, device 105 or device 110 may provide environment representation 210 or another representation derived therefrom to a remote device during a multi-user communication session.

[0052] Figure 3 An exemplary electronic device 305 is shown operating in a physical environment 300. Specifically, Figure 3 Shown in Figure 1A to Figure 1B An exemplary electronic device 305 is shown operating in a physical environment (e.g., physical environment 300) different from the physical environment (e.g., physical environment 100). Figure 3 In the example of FIG. 3 , the physical environment 300 is a room including a sofa 320, a wall hanging 350, and a television screen 370. The electronic device 305 includes one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 300 and objects therein and information about the user 302 of the electronic device 105, as well as to evaluate the physical environment and objects therein. The information about the physical environment 300 and / or the user 302 can be used to provide visual and audio content and / or to identify the current location of the physical environment 300 and / or the location of the user within the physical environment 300.

[0053] In some implementations, a view of an XR environment may be provided to one or more participants (e.g., user 302 and / or other participants not shown, such as user 102) via an electronic device 305 (e.g., a wearable device such as an HMD and / or a handheld device such as a mobile device, a tablet computing device, a laptop computer, etc. (e.g., device 110)). Such an XR environment may include a view of a 3D environment generated based on a camera image and / or a depth camera image of the physical environment 300 and a representation of the user 302 based on the camera image and / or the depth camera image of the user 302. Such an XR environment may include virtual content positioned at a 3D position relative to a 3D coordinate system (i.e., 3D space) associated with the XR environment, which 3D coordinate system may correspond to the 3D coordinate system of the physical environment 300.

[0054] In some embodiments, video (e.g., pass-through video depicting a physical environment) is received from an image sensor of a device (e.g., device 305) and used to render an XR environment. In other embodiments, optical perspective may be used to render an XR environment by overlaying virtual content on a view of the physical environment seen through a translucent or transparent display. In some embodiments, a 3D representation of the virtual environment is aligned with a 3D coordinate system of the physical environment. The size of the 3D representation of the virtual environment may be generated based on, in particular, the scale of the physical environment or the positioning of open spaces, floors, walls, etc., so that the 3D representation is configured to align with corresponding features of the physical environment. In some embodiments, a viewpoint within a 3D coordinate system may be determined based on the position of the electronic device within the physical environment. The viewpoint may be determined based on, in particular, image data, depth sensor data, motion sensor data, etc., which may be retrieved via a virtual inertial odometer system (VIO), a simultaneous localization and mapping (SLAM) system, etc.

[0055] Figure 4 According to some specific implementations, Figure 3 4 and 5. An exemplary 3D environment 400 generated from a physical environment 300 and a portal 480 displaying portal content 485 within the 3D environment. Portal 480 may also be referred to herein as a projection of a 3D image. 3D environment 400 includes representations 425, 450, and 470 of a sofa 320, a wall hanging 350, and a television screen 370 of the physical environment 300, respectively. 3D environment 400 also includes portal content 485 displayed to form the shape of portal 480. In some specific implementations, the shape of portal 480 can be any geometric shape capable of projecting content (e.g., a 3D virtual shape such as a hemisphere, also referred to as a "snowball" view). The portal content 485 being displayed by portal 480 constitutes a portal (e.g., a projection of an image), as discussed herein.

[0056] The electronic device 305 provides a view of the physical environment 300, which includes a depiction of the 3D environment 400 from a viewpoint 420 (e.g., also referred to herein as a viewer position), which in this example is determined based on the position of the electronic device 305 in the physical environment 300 (e.g., the viewpoint of the user 302, also referred to herein as the "viewer position" or "viewer viewpoint"). Therefore, as the user 302 moves relative to the physical environment 300 with the electronic device 305 (e.g., HMD), the viewpoint 420 corresponding to the position of the electronic device 305 moves relative to the 3D environment 400. The view of the 3D environment provided by the electronic device changes based on the change of the viewpoint 420 relative to the 3D environment 400. In some specific implementations, the 3D environment 400 does not include a representation of the physical environment 300, for example, only includes virtual content corresponding to a virtual reality environment.

[0057] Figure 5 According to some specific implementations, Figure 4 Example interactions of portal content 485 displayed within portal 480. For example, Figure 5 An exemplary "direct" interaction involving a user's hand 502 virtually touching a UI element of a user interface (e.g., portal content 485) within portal 480 is shown. In this example, user 302 is using device 305 to view and interact with an XR environment, which may include user interface 530 (e.g., portal content 485) within a view of XR environment 510. A direct interaction identification process may use sensor data and / or UI information to determine, for example, which UI element the user's hand is virtually touching and / or where on the UI element the interaction is occurring. Direct interaction may additionally (or alternatively) involve evaluating user activity to determine the user's intent, e.g., whether the user intends to perform a direct tap gesture through a UI element or a swipe / scroll motion along the UI element. Such identification may utilize information about UI elements, e.g., about location, size, element type, types of interactions that can be performed on the element, types of interactions enabled on the element, which of a set of potential target elements for user activity accepts which types of interactions, and the like.

[0058] Figure 5 It further shows Figure 3A view of an XR environment 510 provided via device 305 of virtual elements within a 3D physical environment, wherein a user 302 can perform interactions. In this example, the user 302 makes gestures relative to content presented in the view of the XR environment 510 provided by a device (e.g., device 305). The view of the XR environment 510 includes an exemplary user interface 530 of an application and a drawing 570 of a television screen 370 (e.g., a representation of a physical object, which may be viewed as a pass-through video or may be a direct view of the physical object through a transparent or semi-transparent display). In addition, the view of the XR environment 510 includes a representation 522 of the hand / arm 502 of the user 302. Providing such a view may involve determining 3D properties of the physical environment 300 and positioning virtual content, such as the user interface 530, in a 3D coordinate system corresponding to the physical environment 300.

[0059] exist Figure 5 In the example of , user interface 530 includes various content items, including background portion 535, application portion 540, control element 532 and scroll bar 550. Application portion 540 is displayed in a view provided by device 305 together with 3D effects. For illustrative purposes, user interface 530 (e.g., portal content 485 displayed within portal 480) is simplified, and the user interface may actually include any level of complexity, any number of content items and / or a combination of 2D and / or 3D content. User interface 530 may be provided by various types of operating systems and / or applications, including but not limited to messaging applications, web browser applications, content viewing applications, content creation and editing applications, or any other application that can display, present, or otherwise use visual and / or audio content.

[0060] In this example, the background portion 535 of the user interface 530 is flat. In this example, the background portion 535 includes all aspects of the user interface 530 being displayed except for the icons 532 and the scroll bar 550. Displaying the background portion of the user interface of the operating system or application as a flat surface can provide various advantages. Doing so can provide an easy-to-understand or otherwise easy-to-use user interface of the application using a portion of the XR environment. In some specific implementations, the shape of the user interface (e.g., portal 485) can be curved, such as a hemisphere, to provide different depth views of the content within the user interface as the content is presented within the view of the 3D environment. In some specific implementations, multiple user interfaces (e.g., corresponding to multiple different applications) are presented sequentially and / or simultaneously within the XR environment (e.g., within one or more colliders or other such components).

[0061] In some implementations, the position and / or orientation of such one or more user interfaces may be determined to facilitate visibility and / or use. Within a 3D environment, one or more user interfaces may be in a fixed position and orientation. In such a case, user movement will not affect the position or orientation of the user interface within the 3D environment.

[0062] The position of the user interface within the 3D environment may be based on determining the distance of the user interface from the user (e.g., from an initial or current user position). The position and / or distance from the user may be determined based on a variety of criteria, including but not limited to criteria that consider application type, application functionality, content type, content / text size, environment type, environment size, environment complexity, environment lighting, the presence of other people in the environment, use of the application or content by multiple users, user preferences, user input, and numerous other factors.

[0063] Figure 6 An exemplary operating environment 600 is shown of electronic devices 105 and 305 operating in different physical environments 100 and 300, respectively, during a communication session (e.g., when the electronic devices 105, 305 share information with each other or with an intermediary device such as a communication session server) according to some implementations. Figure 6 In this example, the physical environment 100 is a room including a wall hanging 150, a plant 125, and a table 120. The electronic device 105 includes one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 100 and objects therein and information about the user 102 of the electronic device 105, as well as to evaluate the physical environment and objects therein. For example, the electronic device 105 can use the captured information to generate an environmental representation of the physical environment 100 that is similar or identical to the environmental representation 210 described above. Information about the physical environment 100 and / or the user 102 (e.g., the environmental representation) can be used to provide visual and audio content during a communication session. For example, the communication session can provide one or more participants (e.g., users 102, 302) of the 3D environment with a view generated based on a camera image and / or a depth camera image of the physical environment 100 and a representation of the user 102 based on the camera image and / or the depth camera image of the user 102.

[0064] In this example, it is shown Figure 3 300, which is a portion of a room including a wall hanging 350 and a sofa 320, and a view of a user 302 wearing an electronic device 305 (e.g., an HMD). The electronic device 305 includes one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 300 and objects therein and information about the user 302 of the electronic device 302, as well as to evaluate the physical environment and objects therein, as described herein with reference to Figure 3 For example, the electronic device 305 may use the captured information to generate an environment representation of the physical environment 300 that is similar to the environment representation 210 generated for the physical environment 100 described above. Information about the physical environment 300 and / or the user 302 may be used to provide visual and audio content during the communication session. For example, the communication session may provide a view of a 3D environment generated based on a camera image and / or a depth camera image (from the electronic device 105) of the physical environment 100 and a representation of the user 102. For example, a representation of the 3D environment (e.g., environment representation 210) may be sent by a device 105 using the communication session instruction set 612 that communicates with the device 305 using the communication session instruction set 622 (e.g., via the network connection 602). Similarly, a representation of the 3D environment (e.g., similar to environment representation 210) may be sent by a device 305 using the communication session instruction set 622 that communicates with the device 105 using the communication session instruction set 612 (e.g., via the network connection 602). However, it should be noted that representations of the users 102, 302 may be provided in other 3D environments. For example, a communication session may involve representations of either or both of the users 102, 302 positioned within any fully virtual environment or an XR environment that includes some representations of the physical environment and some representations of the virtual environment and is viewable by other users within the portal. In the examples of FIGS. 7-10 described next, such views of the portal are shown, as well as different aspects of the views of the user representations and the context of each user environment relative to the viewpoint.

[0065] Figures 7 to 10 illustrate exemplary environments for displaying a portal with portal content in a view of a first 3D environment, the portal content depicting a portion of a second 3D environment viewed through the portal from different viewpoints, according to some specific implementations. Specifically, Figures 7 to 10 illustrate changes in the content view within a portal of a user (e.g., user 102) wearing or using a device such as an HMD (e.g., device 305) during a communication session with another user (e.g., user 302) wearing and / or using another device such as an HMD (e.g., device 105). For example, Figures 7 to 10 illustrate different portal content views as user 302 moves around in his room (e.g., physical environment 300) during the communication session and thus there are different viewpoints (e.g., viewpoints 720, 820, 920, 1020) and / or user 102 moves in his room (e.g., physical environment 100). As mentioned above, the HMD may be limited in its ability to update the view of the user and the user's background because, from the perspective of a typical video chat session (e.g., with the camera positioned 1-2 meters in front of the user to capture both user and background data), the outward-facing camera may not be able to capture the background environment (e.g., the portion of the physical environment behind the user) during the communication session. Therefore, when two users wear the HMD during a communication session, if it is desired to provide an actual ("live") background or at least a representation of the background during the session, the system may represent the background using a previously captured image of the environment (e.g., an image of at least a portion of the environment captured prior to the communication session), or the system may fabricate some of the background (e.g., any gaps or invisible portions in the background) or all of it. Then, based on the sender's position relative to the background and / or the viewer's viewpoint position, the background may be displayed and updated accordingly.

[0066] Fig. 7A , Fig. 8A , Fig. 9A and Fig. 10A The image is shown from different viewpoints (e.g., viewpoints 720, 820, 920, 1020). Figure 4 The exemplary environments 700A, 800A, 900A and 1000A of the exemplary 3D environment 400 are shown in FIG. Fig. 7A , Fig. 8A , Fig. 9A and Fig. 10A Depicted is a 3D environment 400 including a portal 480 displaying portal content 485 , and respective representations 425 , 450 , and 470 of a sofa 320 , a wall hanging 350 , and a television screen 370 of the physical environment 300 .

[0067] Figure 7B , Figure 8B , Fig. 9B and Fig. 10BThe image is shown from different viewpoints (e.g., viewpoints 720, 820, 920, 1020). Figure 3 The exemplary environments 700B, 800B, 900B and 1000B of the physical environment 300 are shown in FIG. Figure 7B , Figure 8B , Fig. 9B and Fig. 10B A view of the physical environment 300 from a particular viewpoint and one or more objects that may be within the view, such as a television screen 370 and a sofa 320, are depicted. Figure 7B , Figure 8B , Fig. 9B and Fig. 10B The view may be an actual view of a user (e.g., user 302) from a particular viewpoint prior to wearing device 305 (e.g., an HMD), or may be a pass-through video or optical see-through view of the physical environment 300 via device 305 prior to displaying virtual content (e.g., portal content 485).

[0068] FIG. 7C to FIG. 7D , FIG. 8C to FIG. 8D , FIG. 9C to FIG. 9D and FIG. 10C to FIG. 10D Exemplary views 700C-D, 800C-D, 900C-D, and 1000C-D of 3D environment 400 are shown from different viewpoints (e.g., viewpoints 720, 820, 920, 1020, respectively). Specifically, FIG. 7C to FIG. 7D , FIG. 8C to FIG. 8D , FIG. 9C to FIG. 9D and FIG. 10C to FIG. 10DDepicted are views of an XR environment representing the physical environment 300 and views of portal content 485 within a portal 480 from a particular viewpoint (e.g., a viewer position of the user 302 within the physical environment 300). In these examples of views 700C-D, 800C-D, 900C-D, and 1000C-D, during a communication session, the electronic device 305 provides different views (e.g., views 705, 805, 905, and 1005, respectively) of a portal (e.g., portals 780, 880, 980, and 1080, respectively) that enable the user 302 to view a representation of at least a portion of the user 102 (e.g., user representations 740, 840, 940, and 1040, respectively), as well as a representation of at least a portion of the environment / background behind the user 102 in the physical environment 100 (e.g., backgrounds 730, 830, 930, and 1030, respectively). For example, user 302 may view a representation of user 102 and a representation of at least a portion of user 102's physical environment 100 (e.g., user 102's office / room). Additionally, depending on the viewpoint, the view may include a representation of television screen 370 (e.g., representations 770, 870, 970, and 1070, respectively). In some implementations, the representation of user 102 (e.g., avatar) and / or background views may provide a live, real-time view of user 102, for example, based on sensor data including images of user 102 and other sensor data obtained during the communication session. As user 102 moves around, makes gestures, and makes facial expressions, corresponding movements, gestures, and expressions may be displayed for the representation in each view. For example, when user 102 moves two feet to the left in physical environment 100, each view 705, 805, 905, and 1005 may show the representation (e.g., avatar) moving two feet to the left in the view corresponding to the movement of user 102.

[0069] for FIG. 7A to FIG. 7D At viewpoint 720, user 302 and / or device 305 is facing directly towards portal content 485 of portal 480. For example, during a normal communication session (e.g., a video chat session), each user is facing directly towards each other and is positioned at a specific distance from a specific viewpoint (e.g., 1-2 meters from the camera source).

[0070] exist Figure 7C and Fig.7D , view 705 of portal 780 includes representation 740 of user 302 appearing to be looking directly back at viewpoint 720 (eg, looking directly at user 102). However, Figure 7C and Fig.7D The context data 730 between the two may be different, which may be based on one or more characteristics / factors further described herein (e.g., missing data, privacy characteristics / ambiguity, updated content, etc.). Figure 7C1 (eg, a live image or an unaltered representation of the wall hanging 150), while Fig.7D The representation 734 of the wall hanging 150 in has been filtered, such as by a privacy masking feature (e.g., masking any personally identifiable information, such as pictures or photos of the individual or family members). For example, Fig.7D The frame representing 734 of the wall hanging 150 is retained, but the content within the frame has been blocked.

[0071] exist Figure 8C and Fig.8D 7 , the view 805 of the portal 880 includes a representation 840 of the user 102 that appears to be looking slightly away from the viewpoint 820. For example, the avatar of the user 102 appears to continue looking at the initial viewpoint 720 of FIG. 7 . In other words, the representation of the user 102 during the communication session may be anchored at the 3D space within the portal 880, but the user 302 may look around the background 830 of the 3D space (e.g., changing the viewpoint). However, Figure 8C and Fig.8D The context data 830 between the two may be different, which may be based on one or more characteristics / factors further described herein (e.g., missing data, privacy characteristics / ambiguity, updated content, etc.). Figure 8C The representation 832 of the wall hanging 150 in FIG. 1 is presented as having the same or similar appearance as that shown in the physical environment 100 of FIG. 1 (e.g., a live image or an unaltered representation of the wall hanging 150), while Fig.8D The representation 834 of the wall hanging 150 in has been filtered, such as by a blurring technique (e.g., providing at least a portion of a representation of the sender's background within the portal to give some context of where the sender is calling from, but not providing a completely clear picture of that environment / background).

[0072] exist Fig. 9C and Fig.9D 7 , the view 905 of the portal 980 includes a representation 940 of the user 102 that appears to be looking slightly away from the viewpoint 920. For example, the avatar of the user 102 appears to continue looking at the initial viewpoint 720 of FIG. 7 . In other words, the representation of the user 102 during the communication session may be anchored at the 3D space within the portal 980, but the user 302 may look around the background 930 of the 3D space (e.g., changing the viewpoint). Fig. 9A As shown in , user 302 at viewpoint 920 appears to be looking from a larger angle, enabling user 302 to see different angles of the background / environment of user 102 (e.g., physical environment 100). Fig. 9C and Fig.9DThe context data 930 between the two may be different, which may be based on one or more characteristics / factors further described herein (e.g., missing data, privacy characteristics / ambiguity, updated content, etc.). Fig. 9C 100 (e.g., poor lighting conditions) are shown, which may be detected in the environment 100 and thus have a less than ideal appearance in the background data 930. In some implementations, the systems and techniques described herein can detect the light source anomaly 932 and provide corrections. Thus, for example, Fig.9D An example of fictitious background content is shown that is used to correct such anomalies and / or update the background 930 based on one or more factors (e.g., updating a portion of a virtual background that overlays a real background representation, or replacing the entire background with fictitious content (e.g., a virtual background during a video chat session)). Specifically, Fig.9D Included are representations 942 of doors, walls, and floors added to background 930. In some implementations, additional content (e.g., representation 942) can be added based on updated sensor data from device 105 as the user moves around in their physical environment 100, or generated by device 305 or device 105 based on acquired semantic data (e.g., semantic 3D point cloud) identifying a door that may be located at that particular location. In other words, background data (e.g., door) is fabricated based on the identified incomplete data.

[0073] exist Fig. 10C and Fig. 10D 7 , the view 1005 of the portal 1080 includes a small portion of the representation 1040 of the user 102 because the viewpoint 1020 is at a very wide angle relative to the initial viewpoint 720 of FIG. 7 . In other words, the representation of the user 102 during the communication session may be anchored at the 3D space within the portal 1080, but the user 302 may look around the background 1030 of the 3D space (e.g., change the viewpoint). Fig. 10A As shown, user 302 at viewpoint 1020 appears to be viewing from a very wide angle and almost Fig. 10A The planes of portal 480 shown in FIG. 1 are aligned so that user 302 can see different portions of user 102's background / environment (e.g., physical environment 100) if the system allows the user to view other areas. However, Fig. 10C and Fig. 10D The context data 1030 between the two may be different based on one or more characteristics / factors further described herein (e.g., missing data, privacy characteristics / ambiguity, updated content, etc.). Fig. 10CA representation 1032 of plants 125 of physical environment 100 is shown in background 1030. For example, sensor data obtained from device 105 of physical environment 100 allows the system to generate representation 1032 (eg, an area in the background captured by the device). Fig. 10D An example of fictitious background content is shown that is used to update the background 1030 based on one or more factors (e.g., updating a portion of a virtual background that covers a real background representation, or replacing the entire background with fictitious content (e.g., a virtual background during a video chat session)). Specifically, Fig. 10D A representation 1034 of a fake plant is included. Representation 1034 may be generated to represent a known location of a plant based on semantic data, or may be randomly generated due to the absence of known data for the area. Alternatively, a blurred and / or colored background (e.g., a faded colored wall, such as a curtain background for a portrait or photo) may be applied instead of fictitious additions for unknown areas in background data 1030. In some implementations, additional content (e.g., representation 1034) may be added based on updated sensor data from device 105 as the user moves around in their physical environment 100, or generated by device 305 or device 105 based on acquired semantic data (e.g., a semantic 3D point cloud) that identifies a plant that may be located at that particular location. In other words, background data (e.g., a plant) is fictitious based on the identified data to fill in the incomplete data.

[0074] Fig.11 1 is a process flow diagram illustrating an exemplary process 1100 of providing portal content based on data from a first 3D environment for display in a portal within a view of a second 3D environment according to some implementations. In some implementations, the process 1100 is performed on a device (e.g., device 105, 110, 305, etc.) such as a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. In some implementations, the process flow 1100 is performed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the process flow 1100 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0075] Process flow 1100 obtains sensor data 1102 from a first physical environment (e.g., device 105 obtains sensor data of physical environment 100). Sensor data 1102 may include image data, depth data, location information, etc. For example, sensors on a device (e.g., a camera, IMU, etc. on device 105, 110, 305, etc.) may capture information about the position, location, movement, posture, etc. of a user's head and / or body and the environment.

[0076] In an example implementation, a portal content generation system for process 1100 may include a portal content instruction set 1120 and a combined representation instruction set 1140. The portal content instruction set 1120 may include one or more modules that may then be used to analyze the sensor data 1102. The portal content instruction set 1120 may include a motion module 1122 for determining motion trajectory data from a motion sensor of one or more objects. The portal content instruction set 1120 may include a positioning module 1124 configured with instructions executable by a processor to obtain sensor data (e.g., RGB data, depth data, etc.) using one or more techniques and track the position of a mobile device (e.g., device 105, 305, etc.) in a 3D coordinate system (e.g., tracking as a user moves around in a 3D environment to determine a particular viewpoint, as discussed herein). The portal content instruction set 1120 may include an object detection module 1126 that may analyze RGB images from a light intensity camera and / or a sparse depth map from a depth camera (e.g., a time-of-flight sensor) and other sources of physical environment information (e.g., camera positioning information such as a position sensor from a camera's SLAM system, VIO, etc.) to identify objects (e.g., people, pets, etc.) in the light intensity image sequence. In some implementations, the object detection module 1126 uses machine learning to perform object recognition. In some implementations, the machine learning model is a neural network (e.g., an artificial neural network), a decision tree, a support vector machine, a Bayesian network, etc. For example, the object detection module 1126 uses an object detection neural network unit to recognize objects and / or an object classification neural network to classify each type of object.

[0077] The portal content instruction set 1120 may also include an occlusion module 1128 for detecting occlusions in the object model. For example, if the viewer viewpoint changes and occlusions are detected, the system may then make a determination to fabricate any gaps in data that may be missing based on the occlusions detected between one or more objects. For example, an initial room scan may not capture image data for the area behind the table 120 of FIG. 1 , but if the user 102 moves to a location where the viewpoint may reveal areas that were occluded by the initial captured viewpoint, the occlusion module 1128 may indicate which area may need to be fabricated based on the surrounding (known) data from the initial room scan.

[0078] The portal content instruction set 1120 may also include a privacy module 1130 that may control the amount of blurring or masking of specific areas of background data to be displayed to another user during a communication session based on one or more user settings and / or default system settings. For example, based on a threshold distance setting, only a certain radial distance (e.g., a 5-foot radius) around the user may be displayed within the portal content, and then the remainder of the background data may be blurred. Additionally, for privacy purposes, all background data may be blurred. Additionally (or alternatively), the identified objects for which personally identifiable information is displayed may be modified. For example, if Fig.7D As shown, the representation 734 of the wall hanging 150 is modified (eg, assuming that the mountain depicted on the wall hanging 150 may also have family members included in the picture).

[0079] The portal content instruction set 1120 may also include an environment representation module 1132 and / or a user representation module 1134 for generating data to be used for representations of contextual data and user representations as described herein.

[0080] The portal content instruction set 1120 generates portal content data 1136 using one or more modules and sends it to a combined representation instruction set 1140 configured to generate a combined representation 1150 (e.g., a virtual portal within a view of a 3D environment such as an XR environment). In some implementations, the portal content data 1136 may include an environment representation similar to or the same as the environment representation 210 described above.

[0081] The combined representation instruction set 1140 may obtain sensor data 1104 from the viewer environment (e.g., the device 305 obtains sensor data of the physical environment 300). The sensor data 1104 may include image data, depth data, location information, etc. For example, sensors on the device (e.g., a camera, an IMU, etc. on the device 305) may capture information about the position, location, movement, posture, etc. of the user's head and / or body and the environment. The combined representation instruction set 1140 may include a 3D environment representation module 1142 for generating a view of a representation of the viewer environment (e.g., an optical see-through, a pass-through video, or a 3D model of the viewer environment) based on the obtained sensor data 1104. The combined representation instruction set 1140 may also include a portal content representation module 1144 for generating a view of the portal including a representation of the sender environment based on the obtained portal content data 1136. Thus, the combined representation instruction set 1140 generates a combined representation 1150 based on combining the 3D environment representation of the physical environment 300 with the generated portal, the generated portal including the portal content generated from the sender physical environment 100, as shown in Figures 7-10.

[0082] One or more modules included in the portal content instruction set 1120 may be executed at a sender device, a viewer device, or a combination thereof. For example, the device 105 (e.g., the sender device) obtains sensor data 1102 (e.g., a room scan) of the physical environment 100 and sends the sensor data to the device 305 (e.g., the viewer device) for analysis to generate portal content, and thus the portal content (e.g., avatar and background data) will be updated based on the updated sensor data from the device 305. In addition (or alternatively), the device 105 (e.g., the sender device) obtains sensor data 1102 (e.g., a room scan) of the physical environment 100 and also analyzes the sensor data to generate portal content, and then sends the portal content to the device 305 for display and viewing. In addition (or alternatively), the analysis and different decision points of when to fabricate new content, blur one or more features, etc. can be performed by both the sender device and the viewer device.

[0083] Fig.12 1 is a flowchart illustrating a method 1200 for displaying a portal with portal content in a view of a first 3D environment, the portal content depicting a portion of a second 3D environment viewed through the portal from a viewpoint, according to some implementations. In some implementations, a device such as an electronic device 105, 110, 305 performs the method 1200. In some implementations, the method 1200 is performed on a mobile device (e.g., device 110), a desktop computer, a laptop computer, an HMD (e.g., device 105, 305), or a server device. The method 1200 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1200 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory). In some implementations, the device performing the method 1200 includes a processor and one or more sensors.

[0084] In an exemplary implementation, method 1200 is performed at a first electronic device having a processor. Specifically, the following blocks are performed at a viewer device (e.g., HMD) such as device 305, and provide a view of a 3D environment (e.g., viewer's room) with a portal that provides a view of the background environment of another user (e.g., sender). The portal can provide a multi-directional view of the other environment that changes as the viewer moves relative to the portal (e.g., depending on the viewpoint). The background data is provided by the other user's device capturing sensor data of the other user's environment, thereby potentially filling in data gaps / fictional content, and can be provided to the viewer device using parameters (e.g., blurring, not depicting other people, providing limited views (e.g., 180℉ OV), using update criteria based on changes / new content, etc.).

[0085] At block 1210 , method 1200 presents a view of a first three-dimensional (3D) environment. For example, as shown in FIG. 7 , a viewing device (device 305 ) presents a view 705 that includes a representation of physical environment 300 .

[0086] At block 1220, method 1200 obtains data representing a second 3D environment based at least in part on sensor data captured in the physical environment of the second electronic device. Fig.11 As shown, a viewing device (e.g., device 305) located in a physical environment 300 (e.g., a first environment) may obtain portal content data 1136, which may include an environment representation of a second 3D environment generated based on sensor data (e.g., sensor data 1102) of a second 3D environment (e.g., physical environment 100 obtained from a room scan via device 110 or device 105). In some specific implementations, data representing a second user (e.g., an avatar) of a second electronic device may also be received. For example, portal content data 1136 may also include a representation of user 102 of device 105 or device 110 generated based on sensor data (e.g., sensor data 1102). In some specific implementations, data representing the second 3D environment may include various types of 3D representations, which may include, but are not limited to, 180° or 360° stereo images (e.g., spherical images, equirectangular projections, etc.), 2D images, and depth data / height maps or 3D models / meshes. In some specific implementations, data representing the second 3D environment may be received from a sender during or before the communication session. In some implementations, data representing the second 3D environment can be based on outward-facing sensors on the second device / HMD and / or fictional content.

[0087] In some implementations, the data representing the second 3D environment depicts a 360 degree view of the second 3D environment. In some implementations, the data representing the second 3D environment depicts less than a 360 degree view (e.g., a 180 degree FOV) of the second 3D environment.

[0088] In some implementations, the data representing the second 3D environment includes a stereoscopic image pair including left-eye content corresponding to a left-eye viewpoint and right-eye content corresponding to a right-eye viewpoint. For example, the data representing the second 3D environment may include a 180-degree stereoscopic image and / or a spherical image or an equirectangular projection. Additionally or alternatively, in some implementations, the data representing the second 3D environment includes two-dimensional (2D) image data and depth data (e.g., a 2D image and depth data / height map). In some implementations, the data representing the second 3D environment includes a 3D model (e.g., a 3D mesh representing a background environment).

[0089] In some implementations, data representing the second 3D environment is obtained during a communication session between the first electronic device and the second electronic device. Figure 6 As shown, electronic devices 105 and 305 operate in different physical environments 100 and 300, respectively, during a communication session (eg, when electronic devices 105, 305 share information with each other or with an intermediary device such as a communication session server).

[0090] At block 1230, method 1200 determines portal content based on data representing the second 3D environment and a viewpoint within the first 3D environment (e.g., a viewer viewpoint of the portal). For example, as shown in FIGS. 7-10 , device 305 (e.g., a viewer device) of user 302 determines portal content 485 to be displayed with portal 480.

[0091] In some implementations, the portal content is based on data that synthesizes a representation of a portion of the second 3D environment that is not represented in the sensor data captured in the physical environment of the second electronic device. For example, during capture of the sender background, the sender device may update a low-frequency screenshot (e.g., RGB image) and a depth map (and possibly some metadata, such as head orientation / pose) during the communication session. In addition, the viewer device is able to fill in holes (e.g., update an incomplete map via an auto-fill neural network) and update the background data. In some implementations, the viewer device (e.g., device 305) may update the portal content using update criteria based on a change in the user's viewpoint, new content, the sender changing their position, a new / different background view, detection of objects in motion, etc.

[0092] In some implementations, the portal content is updated based on detecting a change in the second 3D environment or in the physical environment of the second electronic device. For example, the context of the portal content 485 can be updated based on the sender device or the viewing device detecting new content in the context of the data representing the second 3D environment, the sender or the viewer changing their position, a new / different context view, etc. In some implementations, determining the portal content includes rendering at least a portion of the data representing the second 3D environment on at least a portion of the sphere.

[0093] At block 1240, method 1200 displays a portal with portal content in the view of the first 3D environment, wherein the portal content depicts a portion of the second 3D environment viewed through the portal from the viewpoint. For example, as shown in FIGS. 7-10 , device 305 (e.g., viewer device) of user 302 displays portal content 485 within the view of portal 480.

[0094] In some implementations, obtaining data representing the second 3D environment includes obtaining a parameter associated with the data representing the second 3D environment. In some implementations, the parameter identifies a field of view or orientation of the second 3D environment, and wherein determining the portal content is further based on the parameter. For example, the system can update the background data based on a viewer viewpoint change, as shown in the different viewpoint scenes of Figures 7 to 10.

[0095] In some implementations, determining the portal content includes blurring portions of the second 3D environment based on the identified field of view or orientation of the second 3D environment. For example, a privacy blur can be applied to a portion of the background data (e.g., masking any personally identifiable information, such as pictures or photos of an individual or family member). Fig.7D As shown, the representation 734 of the wall hanging 150 has been filtered (e.g., Fig.7D The frame representing 734 of the wall hanging 150 is retained, but the content within the frame has been shielded from view of the portal content 485).

[0096] In some implementations, method 1200 further includes obtaining data representing a user of the second electronic device, wherein determining the portal content is further based on the data representing the user of the second electronic device, and wherein the portal content depicts a representation of the user of the second electronic device in front of a portion of the second 3D environment (e.g., displaying an avatar with a background). In some implementations, determining the portal content includes blurring a portion of the second 3D environment behind the representation of the user of the second electronic device (e.g., applying a slight blur to the entire background behind the avatar).

[0097] In some implementations, method 1200 further includes determining a location within the view of the first 3D environment to display the portal based on the viewpoint. In some implementations, method 1200 further includes changing the portal content based on a change in the viewpoint within the first 3D environment. For example, as shown in FIGS. 7 to 10 , as a user (e.g., a viewer) moves their viewpoint, the view of the sender's environmental background and avatar changes, allowing the viewer to look around the 180° stereoscopic image portal. The sender avatar may continue to look forward, but the viewer may explore the limited portal content view of the 180° stereoscopic image to some extent.

[0098] In some implementations, displaying the portal with the portal content in the view of the first 3D environment is based on determining a positional relationship (e.g., distance, orientation, etc.) of a viewpoint (viewer's head or device) relative to the portal. For example, the positional relationship can be within or outside a threshold distance from the visual content, within a sphere determined based on the visual content, etc.

[0099] In some implementations, the position of the portal within the first 3D environment is constant as the viewpoint changes within the first 3D environment. For example, when a user (e.g., a viewer, such as user 302) moves around in their environment (e.g., physical environment 300), portal 480 stays in a fixed position (e.g., at the same 3D position). Alternatively, in some implementations, the position of the portal within the first 3D environment changes based on changes in the viewpoint within the first 3D environment. For example, when a user (e.g., a viewer, such as user 302) moves around in their environment (e.g., physical environment 300), portal 480 may move with the user. For example, the portal may appear to remain at the same viewing distance in front of user 302, or portal 480 may remain in the same 3D position, but pivot so that it always faces the user based on the user's viewpoint. Alternatively, in some implementations, the portal may move based on other changes in the environment, such as interruption events (e.g., another person or other object blocking the view of portal 480).

[0100] Fig.13 1 is a flowchart illustrating a method 1300 for generating and providing a representation of a first 3D environment according to some implementations. In some implementations, a device such as an electronic device 105, 110, 305, etc. performs the method 1300. In some implementations, the method 1300 is performed on a mobile device (e.g., device 110), a desktop computer, a laptop computer, an HMD (e.g., device 105, 305), or a server device. The method 1300 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1300 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory). In some implementations, the device performing the method 1300 includes a processor and one or more sensors.

[0101] In an exemplary implementation, method 1300 is performed at a first electronic device having a processor and one or more sensors (e.g., an outward-facing sensor on a device, such as an HMD). Specifically, the following blocks are performed at a sender device (e.g., an HMD) such as device 105, and provide a sender-side perspective of the process of method 1200. For example, method 1200 provides a view of a user's (e.g., sender's) environment to be viewed within a portal within a view of a 3D environment (e.g., a viewer's room). The portal may provide a multi-directional view of the sender's environment that changes as the sender changes position within its environment or as the viewer moves relative to the portal (e.g., depending on the viewpoint). Data representing the user's 3D environment is provided by a user device that captures sensor data of the user's environment, potentially filling in data gaps / fictional content, and may be provided to a viewer device using parameters (e.g., blurring, not depicting other people, providing a limited view (e.g., 180℉ OV), using update criteria based on changes / new content, etc.). During capture of the sender's environment, the sender device may update the low-frequency screenshot (RGB image) + depth map (and metadata such as head orientation / pose), fill holes, and periodically provide updates to the viewer.

[0102] At block 1310, method 1300 obtains sensor data captured via one or more sensors in a physical environment associated with a first electronic device. Sensor data 1102 may include image data, depth data, location information, etc. For example, a sensor on a device (e.g., a camera, IMU, etc. on device 105 or device 110) may capture information about the position, location, motion, posture, etc. of a user's head and / or body and the environment.

[0103] At block 1320, method 1300 determines data representing a first 3D environment, the data generated based at least in part on the sensor data and a parameter identifying an orientation or field of view of the first electronic device. Fig.11 As shown, a viewing device (e.g., device 105 or device 110) located in a physical environment 100 (e.g., a first environment) can obtain sensor data (e.g., sensor data 1102) of the first environment (e.g., physical environment 100 obtained from a room scan via device 110 or device 105). In some implementations, data representing the first 3D environment can be generated and / or updated based on identifying parameters of an orientation or field of view based on a gesture of the first electronic device, so that a second electronic device (e.g., a viewer device) can render an appropriate portion of the environment in an inner portal.

[0104] In some implementations, data representing a first user of the first electronic device (e.g., an avatar) may also be obtained. In some implementations, the data representing the first 3D environment may include various types of 3D representations, which may include, but are not limited to, 180° or 360° stereo images (e.g., spherical images, equirectangular projections, etc.), 2D images, and depth data / height maps or 3D models / meshes. In some implementations, the data representing the first 3D environment may be based on outward-facing sensors and / or fictional content on the first device / HMD.

[0105] At block 1330 , method 1300 provides data representing the first 3D environment to a second electronic device.

[0106] In some implementations, the data representing the first 3D environment is based on synthesizing data representing a portion of the first 3D environment that is not represented in the sensor data captured in the physical environment of the first electronic device. For example, during the capture of the sender environment, the sender device (e.g., device 105) may update a low-frequency screenshot (e.g., an RGB image) and provide a current depth map (and possibly some metadata, such as head orientation / pose). In addition, the sender device may fill in holes (e.g., update an incomplete map via an auto-fill neural network) and update the data representing the first 3D environment. In some implementations, the sender device may update the data representing the first 3D environment using update criteria based on a change in the user's viewpoint, new content, the sender changing its position, a new / different background view, object detection in motion, etc. In some implementations, the synthesized information is determined based on detecting a change in the viewpoint within the second 3D environment (e.g., fictitious information based on a change in the viewpoint of the viewer). In some implementations, the synthesized information is determined based on identifying another portion of the first 3D environment that was not captured by the sensor (e.g., fictitious information based on detecting a new background that was not obtained by the sensor).

[0107] In some implementations, the portal content is updated based on detecting a change in the first 3D environment or in the physical environment of the second electronic device. For example, the data representing the first 3D environment may be updated based on the sender device detecting new content in the environment, the sender or viewer changing their position, a new / different background view, etc. In some implementations, the data representing the first 3D environment is updated based on detecting a change in the physical environment of the first electronic device. For example, the sender device may update the data representing the first 3D environment based on new content in the environment, the sender changing their position, a new / different background view, etc.

[0108] In some implementations, the data representing the first 3D environment is updated based on detecting that the change in the position of the first electronic device exceeds a threshold. For example, the sender device may update the data representing the first 3D environment (and the avatar) in response to the sender electronic device moving a threshold distance (e.g., moving more than three meters to five meters, or another configurable threshold distance set by the system and / or user).

[0109] In some implementations, the method 1300 further includes: determining a first lighting condition associated with an area of ​​the first 3D environment based on the data representing the first 3D environment; and updating the data representing the first 3D environment for the area associated with the first lighting condition in the first 3D environment. For example, as shown in FIG. 9 , the system can detect a poorly lit portion of the representation (e.g., an extinguished light source and / or a poor lighting condition, such as Fig. 9C 932) and replace the data with synthetic data (e.g., false ceiling, wall color, etc.), or update areas of the first 3D environment based on a tone map, etc.

[0110] In some implementations, determining data representing the first 3D environment includes determining coverage of a background associated with a physical environment of the first electronic device based on the sensor data, and in response to determining that coverage of the captured background of the physical environment is below a threshold amount, including synthetic background data for a portion of the first 3D environment in the data representing the first 3D environment. For example, the system may be configured to send a default background if coverage of the representation is below a certain threshold amount (e.g., if the room scan covers less than 50% of the sender's current position and current viewpoint in the physical environment, the system may display a default background relative to the fictional content to fill the gap).

[0111] In some implementations, a blur effect is applied by a first electronic device (e.g., a sender device, device 105) to at least a portion of data representing a first 3D environment that is provided to a second electronic device (e.g., a viewer device, device 305). For example, the sender can apply a slight blur to the entire representation of the first 3D environment (e.g., where the blur is performed by the sender rather than the recipient / viewer).

[0112] In some implementations, the second electronic device is configured to display a view of the data representing the first 3D environment within a portal within a view of the second 3D environment (e.g., VR or XR). In some implementations, the first electronic device and the second electronic device operatively communicate during the communication session. For example, Figure 6As shown, electronic devices 105 and 305 operate in different physical environments 100 and 300, respectively, during a communication session (e.g., when electronic device 105 (e.g., sender device) and electronic device 305 (e.g., viewer device) share information with each other or with an intermediate device such as a communication session server).

[0113] Fig.14 1 is a block diagram of an exemplary device 1400. Device 1400 illustrates an exemplary device configuration of the devices described herein (e.g., device 105, device 110, device 305, etc.). Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 1400 includes one or more processing units 1402 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 1406, one or more communication interfaces 1408 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1410, one or more displays 1412, one or more internal-facing and / or external-facing image sensor systems 1414, memory 1420, and one or more communication buses 1404 for interconnecting these and various other components.

[0114] In some implementations, the one or more communication buses 1404 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices and sensors 1406 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0115] In some implementations, one or more displays 1412 are configured to present a view of a physical or graphical environment to a user. In some implementations, one or more displays 1412 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more displays 1412 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, device 10 includes a single display. As another example, device 10 includes a display for each eye of the user.

[0116] In some implementations, the one or more image sensor systems 1414 are configured to obtain image data corresponding to at least a portion of the physical environment 105. For example, the one or more image sensor systems 1414 include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 1414 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 1414 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.

[0117] Memory 1420 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, memory 1420 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. Memory 1420 optionally includes one or more storage devices remotely located from one or more processing units 1402. Memory 1420 includes non-transitory computer-readable storage media.

[0118] In some implementations, the memory 1420 or a non-transitory computer-readable storage medium of the memory 1420 stores an optional operating system 1430 and one or more instruction sets 1440. The operating system 1430 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 1440 includes executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set 1440 is software that can be executed by one or more processing units 1302 to implement one or more of the techniques described herein.

[0119] The instruction set 1440 includes a portal content instruction set 1442 that generates portal content data, and a representation instruction set 1444 that generates and displays a representation of a context and / or a user. The instruction set 1440 may be embodied as a single software executable file or multiple software executable files. In some implementations, the portal content instruction set 1442 and the representation instruction set 1444 may be executed by the processing unit 1402 using one or more of the techniques described herein or as may be appropriate in other ways. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0120] Although the instruction set 1440 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. Fig.14 The instructions set is intended to be more of a functional description of the various features present in a particular implementation rather than a structural representation of the implementations described herein. As will be appreciated by one of ordinary skill in the art, items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how features are distributed among them will vary depending on the implementation and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0121] Fig.15 A block diagram of an exemplary head mounted device 1500 is shown according to some implementations. The head mounted device 1500 includes a housing 1501 (or shell) that houses various components of the head mounted device 1500. The housing 1501 includes (or is coupled to) eye pads (not shown) disposed at the proximal (of the user 25) end of the housing 1501. In various implementations, the eye pads are plastic or rubber pieces that comfortably and snugly hold the head mounted device 1500 in place on the face of the user 25 (e.g., around the eyes 35 of the user 25).

[0122] Housing 1501 houses display 1510, which displays images, emits light toward or onto the eye 35 of user 25. In various implementations, display 1510 emits light through an eyepiece having one or more optical elements 1505, which refract the light emitted by display 1510, so that the display appears to user 25 as a virtual distance farther than the actual distance from the eye to display 1510. For example, optical element 1505 may include one or more lenses, waveguides, other diffractive optical elements (DOEs), etc. In order to enable user 25 to focus on display 1510, in various implementations, the virtual distance is at least greater than the minimum focal length of the eye (e.g., 7 cm). In addition, in order to provide a better user experience, in various implementations, the virtual distance is greater than 1 meter.

[0123] Housing 1501 also houses a tracking system that includes one or more light sources 1522, camera 1524, camera 1532, camera 1534, and controller 1580. One or more light sources 1522 emit light onto the eyes of user 25, which is reflected as a light pattern (e.g., a flashing circle) that can be detected by camera 1524. Based on the light pattern, controller 1580 can determine the eye tracking characteristics of user 25. For example, controller 1580 can determine the gaze direction and / or blink state (eyes open or closed) of user 25. As another example, controller 1580 can determine the pupil center, pupil size, or focus point. Therefore, in various specific implementations, light is emitted by one or more light sources 1522, reflected from the eyes of user 25, and detected by camera 1524. In various specific implementations, light from the eyes of user 25 reflects from a hot mirror or passes through an eyepiece before reaching camera 1524.

[0124] The display 1510 emits light within a first wavelength range, and the one or more light sources 1522 emit light within a second wavelength range. Similarly, the camera 1524 detects light within the second wavelength range. In various implementations, the first wavelength range is a visible wavelength range (e.g., a wavelength range of approximately 400 nm-700 nm within the visible spectrum), and the second wavelength range is a near infrared wavelength range (e.g., a wavelength range of approximately 700 nm-1400 nm within the near infrared spectrum).

[0125] In various implementations, eye tracking (or specifically, determined gaze direction) is used to enable user interaction (e.g., user 25 selects an option on display 1510 by looking at it), provide perforated rendering (e.g., presenting a higher resolution in the area of ​​display 1510 that user 25 is looking at and presenting a lower resolution elsewhere on display 1510), or correct distortion (e.g., for images to be provided on display 1510). In various implementations, one or more light sources 1522 emit light toward the eyes of user 25, which is reflected in the form of multiple flashes.

[0126] In various implementations, camera 1524 is a frame / shutter based camera that generates images of the user's 25 eyes at a particular point in time or multiple points in time at a frame rate. Each image includes a matrix of pixel values ​​corresponding to pixels of the image, the pixels corresponding to locations of the camera's light sensor matrix. In a specific implementation, each image is used to measure or track pupil dilation by measuring changes in pixel intensity associated with one or both of the user's pupils.

[0127] In various implementations, camera 1524 is an event camera comprising multiple light sensors (e.g., a light sensor matrix) at multiple corresponding positions, which generates an event message indicating a specific position of a particular light sensor in response to the particular light sensor detecting a change in light intensity.

[0128] In various implementations, camera 1532 and camera 1534 are frame / shutter based cameras that can generate images of the face of user 25 at a specific point in time or multiple points in time at a frame rate. For example, camera 1532 captures an image of the user's face below the eyes, and camera 1534 captures an image of the user's face above the eyes. The images captured by camera 1532 and camera 1534 may include light intensity images (e.g., RGB) and / or depth image data (e.g., time of flight, infrared, etc.).

[0129] It should be understood that the specific implementations described above are cited by way of example, and the present disclosure is not limited to what has been particularly shown and described above. On the contrary, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to a person skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0130] As described above, one aspect of the present technology is to collect and use sensor data that may include user data to improve the user experience of electronic devices. The present disclosure contemplates that, in some cases, the collected data may include personal information data that uniquely identifies a particular person or can be used to identify the interests, characteristics, or tendencies of a particular person. Such personal information data may include motion data, physiological data, demographic data, location-based data, phone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0131] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, personal information data can be used to improve the content viewing experience. Therefore, the use of such personal information data may enable planned control of electronic devices. In addition, the present disclosure also anticipates other uses of personal information data that benefit users.

[0132] The disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for the privacy and security of maintaining personal information data. For example, personal information from users should be collected for the legal and reasonable purposes of entities, and not shared or sold outside these legal purposes. In addition, such collection should only be carried out after the user's informed consent. In addition, such entities should take any required steps to safeguard and protect access to such personal information data, and ensure that other people who can access personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices.

[0133] Regardless of the foregoing, the present disclosure also contemplates specific implementations in which users selectively prevent the use or access of personal information data. That is, the present disclosure contemplates that hardware elements or software elements may be provided to prevent or block access to such personal information data. For example, with respect to a content delivery service customized for a user, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" of participating in the collection of personal information data during registration for the service. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information but allow anonymous information to be transmitted for use in improving the functionality of the device.

[0134] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences or settings based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information.

[0135] In some embodiments, data is stored using a public / private key system that only allows the owner of the data to decrypt the stored data. In some other implementations, data can be stored anonymously (e.g., without identification and / or personal information about the user, such as legal name, username, time and location data, etc.). In this way, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some implementations, a user can access their stored data from a user device different from the user device used to upload the stored data. In these cases, the user may need to provide login credentials to access their stored data.

[0136] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0137] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0138] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0139] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined and / or divided into sub-blocks. Some boxes or processes can be executed in parallel.

[0140] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0141] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0142] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0143] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0144] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method, comprising: At a first electronic device having a processor: presenting a view of a first three-dimensional 3D environment; obtaining data representing a second 3D environment, the data representing the second 3D environment based at least in part on sensor data captured in a physical environment of a second electronic device; determining portal content based on the data representing the second 3D environment and a viewpoint within the first 3D environment; as well as A portal having the portal content is displayed in the view of the first 3D environment, wherein the portal content depicts a portion of the second 3D environment viewed through the portal from the viewpoint. 2 . The method of claim 1 , wherein the portal content is based on data that synthetically represents a portion of the second 3D environment that is not represented in sensor data captured in the physical environment of the second electronic device. 3 . The method of claim 1 , wherein the portal content is updated based on detecting a change in the second 3D environment or in the physical environment of the second electronic device. The method of claim 1 , wherein obtaining the data representing the second 3D environment comprises obtaining parameters associated with the data representing the second 3D environment. 5 . The method of claim 4 , wherein the parameter identifies a field of view or an orientation of the second 3D environment, and wherein determining portal content is further based on the parameter. 6 . The method of claim 5 , wherein determining portal content comprises blurring some of the portions of the second 3D environment based on the identified field of view or orientation of the second 3D environment.

7. The method of claim 1 , further comprising obtaining data representing a user of the second electronic device, wherein determining the portal content is further based on the data representing the user of the second electronic device, and wherein the portal content depicts the representation of the user of the second electronic device in front of the portion of the second 3D environment. 8 . The method of claim 7 , wherein determining portal content comprises blurring the portion of the second 3D environment behind the representation of the user of the second electronic device. 9 . The method of claim 1 , wherein the data representing the second 3D environment depicts less than a 360 degree view of the second 3D environment.

10. The method of claim 1, wherein the data representing the second 3D environment depicts a 360 degree view of the second 3D environment.

11. The method according to claim 1, further comprising: A location within the view of the first 3D environment at which to display the portal is determined based on the viewpoint.

12. The method according to claim 1, further comprising: The portal content is changed based on a change in the viewpoint within the first 3D environment.

13. The method of claim 1, wherein displaying the portal with the portal content in the view of the first 3D environment is based on determining a positional relationship of the viewpoint relative to the portal.

14. The method of claim 1, wherein a position of the portal within the first 3D environment is constant as the viewpoint changes within the first 3D environment.

15. The method of claim 1, wherein a position of the portal within the first 3D environment changes based on a change in the viewpoint within the first 3D environment.

16. The method of claim 1, wherein the data representing the second 3D environment comprises a stereoscopic image pair including left-eye content corresponding to a left-eye viewpoint and right-eye content corresponding to a right-eye viewpoint.

17. The method of claim 1, wherein the data representing the second 3D environment comprises a 180 degree stereoscopic image.

18. The method of claim 1, wherein the data representing the second 3D environment comprises two-dimensional (2D) image data and depth data.

19. The method of claim 1, wherein determining portal content comprises rendering at least a portion of the data representing the second 3D environment on at least a portion of a sphere.

20. A device, comprising: a non-transitory computer-readable storage medium; as well as One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations including: presenting a view of a first three-dimensional 3D environment; obtaining data representing a second 3D environment, the data representing the second 3D environment based at least in part on sensor data captured in a physical environment of a second electronic device; determining portal content based on the data representing the second 3D environment and a viewpoint within the first 3D environment; as well as A portal having the portal content is displayed in the view of the first 3D environment, wherein the portal content depicts a portion of the second 3D environment viewed through the portal from the viewpoint.