Presenting communication data based on the environment

Abstract

Various embodiments disclosed herein include devices, systems, and methods for presenting communication data. In various embodiments, a first device associated with a first person includes a display, a non-volatile memory, and one or more processors coupled to the display and the non-volatile memory. In some embodiments, a method includes obtaining communication data associated with a second device corresponding to a second person. In some embodiments, the method includes determining whether the first device and the second device are in a shared environment. In some embodiments, the method includes displaying a representation of the second person based on the communication data associated with the second device in response to determining that the first device and the second device are not in a shared environment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 855,155, filed May 31, 2019, which is incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to presenting communication data based on context. Background Art

[0004] Some devices are capable of generating and presenting augmented reality (ER) sets. Some ER sets include virtual sets that are simulated replacements for physical sets. Some ER sets include augmented sets that are modified versions of physical sets. Some devices that present ER sets include mobile communication devices such as smart phones, head mounted displays (HMDs), glasses, heads up displays (HUDs), and optical projection systems. Most previously available devices that present ER sets are ineffective at presenting communication data. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0006] Figures 1A to 1G is an illustration of an example operating environment according to some implementations.

[0007] FIG. 2A to FIG. 2D is a flowchart representation of a method of presenting communication data according to some specific implementations.

[0008] FIG. 3A to FIG. 3C is a flowchart representation of a method of masking communication data according to some specific implementations.

[0009] Figure 4 is a block diagram of an apparatus for presenting communication data according to some implementations.

[0010] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0011] Various embodiments disclosed herein include devices, systems, and methods for presenting communication data. In various embodiments, a first device associated with a first person includes a display, a non-transitory memory, and one or more processors coupled to the display and the non-transitory memory. In some embodiments, a method includes obtaining communication data associated with a second device corresponding to a second person. In some embodiments, the method includes determining whether the first device and the second device are in a shared environment. In some embodiments, the method includes displaying an augmented reality (ER) representation of the second person based on the communication data associated with the second device in response to determining that the first device and the second device are not in a shared environment.

[0012] Various embodiments disclosed herein include devices, systems, and methods for masking communication data. In various embodiments, a first device includes an output device, a non-transitory memory, and one or more processors coupled to the output device and the non-transitory memory. In some embodiments, a method includes, when the first device is in a communication session with a second device, obtaining communication data associated with the second device. In some embodiments, the method includes determining that the first device and the second device are in a shared physical setting. In some embodiments, the method includes masking a portion of the communication data so as to prevent the output device from outputting the portion of the communication data.

[0013] According to some specific implementations, the device includes one or more processors, non-transitory memory, and one or more programs. In some specific implementations, one or more programs are stored in non-transitory memory and executed by one or more processors. In some specific implementations, one or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of a device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes one or more processors, non-transitory memory, and a device for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0014] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0015] Various examples of electronic systems and techniques for using such systems in connection with various augmented reality techniques are described.

[0016] A physical set refers to a world that individuals can sense and / or interact with without the use of electronic systems. A physical set, such as a physical park, includes physical elements, such as physical wildlife, physical trees, and physical plants. People can directly sense and / or otherwise interact with a physical set, for example, using one or more senses, including vision, smell, touch, taste, and hearing.

[0017] In contrast to a physical set, an augmented reality (ER) set refers to a fully (or partially) computer-generated set that various people can sense and / or otherwise interact with using electronic systems. In ER, the movement of a person is monitored in part, and in response thereto, at least one property corresponding to at least one virtual object in the ER set is changed in a manner consistent with one or more laws of physics. For example, in response to the ER system detecting that the person is looking up, the ER system may adjust various audio and graphics presented to the person in a manner consistent with the manner in which such sounds and appearances would change in a physical set. Adjustments to properties of virtual objects in the ER set may also be made, for example, in response to representations of movement (e.g., voice commands).

[0018] A person may sense and / or interact with ER objects using one or more senses, such as vision, smell, taste, touch, and hearing. For example, a person may sense and / or interact with objects that create a multi-dimensional or spatial acoustic scenery. A multi-dimensional or spatial acoustic scenery provides an individual with the perception of discrete sound sources in a multi-dimensional space. Such objects may also implement acoustic transparency, which may selectively incorporate audio from the physical scenery with or without computer-generated audio. In some ER scenery, a person may sense and / or interact with only audio objects.

[0019] Virtual reality (VR) is an example of ER. A VR set refers to an augmented set that is configured to include only computer-generated sensory input for one or more senses. The VR set includes a plurality of virtual objects that a person can sense and / or interact with. A person can sense and / or interact with virtual objects in a VR set by simulating at least some of the person's actions within the computer-generated set and / or by simulating a person or his / her presence within the computer-generated set.

[0020] Mixed reality (MR) is another example of ER. An MR set refers to an augmented set that is configured to integrate computer-generated sensory input (e.g., virtual objects) with sensory input from a physical set or a representation of sensory input from a physical set. On the reality spectrum, MR sets are between and do not include fully physical sets at one end and VR sets at the other end.

[0021] In some MR sets, computer-generated sensory inputs may be adjusted based on changes in sensory inputs from the physical set. Additionally, some electronic systems used to render the MR set may detect position and / or orientation relative to the physical set to enable interaction between real objects (i.e., physical elements from the physical set or representations thereof) and virtual objects. For example, the system may detect movement and adjust computer-generated sensory inputs accordingly so that, for example, a virtual tree appears stationary relative to a physical structure.

[0022] Augmented reality (AR) is an example of MR. AR scenery refers to an enhanced scenery in which one or more virtual objects are superimposed on a physical scenery (or a representation thereof). For example, an electronic system may include an opaque display and one or more imaging sensors for capturing a video and / or image of a physical scenery. For example, such a video and / or image may be a representation of a physical scenery. The video and / or image is combined with a virtual object, where the combination is then displayed on an opaque display. The physical scenery can be viewed indirectly by a person via an image and / or video of the physical scenery. Therefore, a person can observe a virtual object superimposed on the physical scenery. When the system captures an image of the physical scenery and uses the captured image to display the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, a transparent or translucent display may be included in an electronic system for displaying an AR scenery, so that an individual can directly view the physical scenery through a transparent or translucent display. Virtual objects may be displayed on a translucent or transparent display, so that an individual observes virtual objects superimposed on the physical scenery. In another example, a projection system may be utilized to project virtual objects onto the physical scenery. For example, virtual objects may be projected onto a physical surface, or as a hologram, such that an individual observes the virtual object superimposed on a physical setting.

[0023] AR scenery may also refer to an augmented scenery in which a representation of a physical scenery is modified by computer-generated sensory data. For example, at least a portion of the representation of the physical scenery can be graphically modified (e.g., enlarged) so that the modified portion can still represent the originally captured image (but not an exact copy). Alternatively, when providing video pass-through, one or more sensor images may be modified to apply a specific viewpoint that is different from the viewpoint captured by the image sensor. For another example, a portion of the representation of the physical scenery may be changed by graphically blurring or eliminating the portion.

[0024] Augmented Virtual (AV) is another example of MR. An AV set refers to an augmented set that is a virtual or computer-generated set combined with one or more sensory inputs from a physical set. Such sensory inputs may include representations of one or more features of the physical set. Virtual objects may, for example, incorporate colors associated with physical elements captured by an imaging sensor. Alternatively, virtual objects may adopt features consistent with, for example, current weather conditions corresponding to the physical set, such as weather conditions identified via imaging, online weather information, and / or weather-related sensors. For another example, an AR park may include virtual structures, plants, and trees, although animals within the AR park set may include features accurately replicated from images of physical animals.

[0025] Various systems allow people to sense and / or interact with the ER scenery. For example, a head-mounted system may include one or more speakers and an opaque display. As another example, an external display (e.g., a smart phone) may be incorporated into the head-mounted system. The head-mounted system may include a microphone for capturing audio of the physical scenery and / or an image sensor for capturing images / video of the physical scenery. A transparent or translucent display may also be included in the head-mounted system. A translucent or transparent display may, for example, include a substrate through which light (representing an image) is directed to a person's eyes. The display may also include an LED, an OLED, a silicon-based liquid crystal, a laser scanning light source, a digital light projector, or any combination thereof. The substrate through which the light is transmitted may be an optical reflector, a holographic substrate, an optical waveguide, an optical combiner, or any combination thereof. A transparent or translucent display may, for example, selectively transition between a transparent / translucent state and an opaque state. As another example, the electronic system may be a projection-based system. In a projection-based system, retinal projection may be used to project an image onto a person's retina. Alternatively, the projection-based system may also project the virtual object into a physical setting, such as, for example, projecting the virtual object as a hologram or onto a physical surface. Other examples of ER systems include a window configured to display graphics, headphones, earphones, a speaker arrangement, a lens configured to display graphics, a head-up display, a car windshield configured to display graphics, an input mechanism (e.g., a controller with or without haptic capabilities), a desktop or laptop computer, a tablet computer, or a smart phone.

[0026] When a first device associated with a first person communicates with a second device associated with a second person, it may sometimes be unwise to present a video or ER representation of the second person on the first device. For example, if the first device and the second device are in the same environment, it may be unwise to present a video or ER representation of the second person on the first device because the first person can see the second person. In some scenarios, it may be helpful to indicate the type of the second device so that the first person knows how to interact with the second person. For example, if the second device provides the second person with a limited view of the surroundings of the first device, the first person does not need to point to areas of the surroundings that are not visible to the second person.

[0027] The present disclosure provides methods, devices and / or systems that allow a first device to present communication data associated with a second device based on the presence state of the second device. When the first device acquires communication data associated with the second device, if the second device is not in the same environment as the first device, the first device presents a video or ER representation of the second person. If the second device is in the same environment as the first device, the first device presents a transparent transmission of the environment and abandons presenting the video or ER representation encoded in the communication data. In some scenarios, when the second device includes an HMD, the first device presents the ER representation of the second person, and when the second device includes a non-HMD device (e.g., a handheld device such as a tablet or smart phone, a laptop and / or a desktop computer), the first device presents the video.

[0028] When a first device associated with a first person communicates with a second device associated with a second person, presenting network audio and video may result in a degraded experience due to inaudible speech and insensible video. For example, if the first device and the second device are in the same physical setting, the first person will likely hear both network audio through the first device and direct audio from the second person. Interference between the network audio and the direct audio may result in inaudible speech. Similarly, if the first device displays an ER representation of the second person while the second device is in the same physical setting as the first device, the first person may view the ER representation of the second person instead of the second person, resulting in a degraded communication experience.

[0029] The present disclosure provides methods, devices and / or systems for masking communication data when a first device and a second device are in the same physical setting. If the second device is in the same physical setting as the first device, the first device masks the network audio to reduce interference between the network audio and the direct audio. Masking the network audio when the second device is in the same physical setting as the first device allows the first person to listen to the direct audio. If the second device is in the same physical setting as the first device, the first device masks the video or ER representation of the second person indicated by the communication data. Abandoning the display of the video or ER representation of the second person improves the user experience of the first person by allowing the first person to view the second person. In some scenarios, the first device presents a transparent transmission of the physical setting, and the first person sees the second person via the transparent transmission.

[0030] Figure 1A is a block diagram of an exemplary operating environment 1 according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 1 includes a first environment 10 (e.g., a first set) and a second environment 40 (e.g., a second set). In some implementations, the first environment 10 includes a first physical set (e.g., a first physical environment), and the second environment 40 includes a second physical set (e.g., a second physical environment). In some implementations, the first environment 10 includes a first ER set, and the second environment 40 includes a second ER set.

[0031] like Figure 1A As shown, the first environment 10 includes a first person 12 associated with (eg, operating) a first electronic device 14, and the second environment 40 includes a second person 42 associated with (eg, operating) a second electronic device 44. Figure 1A In the example of FIG. 1 , the first person 12 is holding the first electronic device 14, and the second person 42 is holding the second electronic device 44. In various implementations, the first electronic device 14 and the second electronic device 44 include handheld devices (e.g., tablet computers, smart phones, or laptop computers). In some implementations, the first electronic device 14 and the second electronic device 44 are not head-mounted devices (e.g., non-HMD devices, such as handheld devices, desktop computers, or watches).

[0032] In various specific implementations, the first electronic device 14 and the second electronic device 44 communicate with each other via a network 70 (e.g., a portion of the Internet, a wide area network (WAN), a local area network (LAN), etc.). When the first electronic device 14 and the second electronic device 44 communicate with each other, the first electronic device 14 transmits first communication data 16, and the second electronic device 44 transmits second communication data 46. The first communication data 16 includes first audio data 18 captured by a microphone of the first electronic device 14, and first video data 20 captured by an image sensor (e.g., a camera, such as a front-facing camera) of the first electronic device 14. Similarly, the second communication data 46 includes second audio data 48 captured by a microphone of the second electronic device 44 and second video data 50 captured by an image sensor of the second electronic device 44.

[0033] The first electronic device 14 receives the second communication data 46, and the second electronic device 44 receives the first communication data 16. Figure 1A As shown, the first electronic device 14 presents the second communication data 46, and the second electronic device 44 presents the first communication data 16. For example, the first electronic device 14 outputs (e.g., plays) the second audio data 48 via the speaker of the first electronic device 14, and displays the second video data 50 on the display of the first electronic device 14. Similarly, the second electronic device 44 outputs the first audio data 18 via the speaker of the second electronic device 44, and displays the first video data 20 on the display of the second electronic device 44. In various specific implementations, the first audio data 18 encodes the voice input provided by the first person 12, and the first video data 20 encodes the video stream of the first person 12. Similarly, the second audio data 48 encodes the voice input provided by the second person 42, and the second video data 50 encodes the video stream of the second person 42.

[0034] In various implementations, the first electronic device 14 determines whether the second electronic device 42 is in the first environment 10. When the first electronic device 14 determines that the second electronic device 44 is in the first environment 10, the first electronic device 14 considers the second electronic device 44 to be local. In various implementations, when the first electronic device 14 determines that the second electronic device 44 is local, the first electronic device 14 changes the presentation of the second communication data 46. In various implementations, in response to determining that the second electronic device 44 is local, the first electronic device 14 masks a portion of the second communication data 46. For example, in some implementations, in response to determining that the second electronic device 44 is local, the first electronic device 14 abandons outputting the second audio data 48 and / or abandons displaying the second video data 50.

[0035] exist Figure 1AIn the example of , the first electronic device 14 determines that the second electronic device 44 is not in the first environment 10. When the first electronic device 14 determines that the second electronic device 44 is not in the first environment 10, the first electronic device 14 considers that the second electronic device 44 is remote. Figure 1A As shown, in response to determining that the second electronic device 44 is remote, the first electronic device 14 presents the second communication data 46. For example, in response to the second electronic device 44 being remote, the first electronic device 14 outputs the second audio data 48 and / or displays the second video data 50. In some implementations, in response to determining that the second electronic device 44 is remote, the first electronic device 14 abandons masking the second communication data 46.

[0036] See also Figure 1B , a first head mounted device (HMD) 24 worn by the first person 12 presents (e.g., displays) a first ER scenery 26 according to various implementations. Figure 1B The first person 12 is shown holding the first HMD 24, but in various implementations, the first person 12 wears the first HMD 24 on the head of the first person 12. In some implementations, the first HMD 24 includes an integrated display (e.g., a built-in display) that displays the first ER scenery 26. In some implementations, the first HMD 24 includes a head-mounted shell. In various implementations, the head-mounted shell includes an attachment area to which another device with a display can be attached. For example, in some implementations, the first electronic device 14 can be attached to the head-mounted shell. In various implementations, the head-mounted shell is shaped to form a container for receiving another device (e.g., the first electronic device 14) including a display. For example, in some implementations, the first electronic device 14 slides / snaps into the head-mounted shell or is otherwise attached to the head-mounted shell. In some implementations, the display of the device attached to the head-mounted shell presents (e.g., displays) the first ER scenery 26. In various implementations, examples of the first electronic device 14 include a smart phone, a tablet computer, a media player, a laptop computer, and the like.

[0037] exist Figure 1BIn the example of , the first ER scenery 26 presents (e.g., displays) the second video data 50 within a card graphical user interface (GUI) element 28. In some implementations, the card GUI element 28 is within a similarity threshold of a card (e.g., the visual appearance of the card GUI element 28 is similar to the visual appearance of the card). In some implementations, the card GUI element 28 is referred to as a video card. In some implementations, the first HMD 24 presents the second video data 50 in the card GUI element 28 based on the type of the second electronic device 44. For example, in response to determining that the second electronic device 44 is a non-HMD (e.g., a handheld device such as a tablet, a smart phone, or a laptop, a desktop computer, or a watch), the first HMD 24 presents the second video data 50 in the card GUI element 28. In some implementations, the first HMD 24 outputs (e.g., plays) the second audio data 48 via a speaker of the first HMD 24. In some implementations, the first HMD 24 spatializes the second audio data 48 to provide an appearance that the second audio data 48 originates from the card GUI element 28. In some implementations, the first HMD 24 changes the position of the card GUI element 28 within the first ER scenery 26 based on the movement of the second person 42 and / or the second electronic device 44 within the second environment 40. For example, the first HMD 24 moves the card GUI element 28 in the same direction as the second person 42 and / or the second electronic device 44.

[0038] See also Figure 1C , the second HMD 54 worn by the second person 42 presents (e.g., displays) a second ER scenery 56 according to various implementations. Figure 1C The second person 42 is shown holding the second HMD 54, but in various implementations, the second person 42 wears the second HMD 54 on the head of the second person 42. In some implementations, the second HMD 54 includes an integrated display (e.g., a built-in display) that displays the second ER scenery 56. In some implementations, the second HMD 54 includes a head-mounted shell. In various implementations, the head-mounted shell includes an attachment area to which another device with a display can be attached. For example, in some implementations, the second electronic device 44 can be attached to the head-mounted shell. In various implementations, the head-mounted shell is shaped to form a container for receiving another device (e.g., the second electronic device 44) including a display. For example, in some implementations, the second electronic device 44 slides / snaps into the head-mounted shell or is otherwise attached to the head-mounted shell. In some implementations, the display of the device attached to the head-mounted shell presents (e.g., displays) the second ER scenery 56. In various implementations, examples of the second electronic device 44 include a smart phone, a tablet computer, a media player, a laptop computer, and the like.

[0039] like Figure 1C As shown, in some implementations, the first communication data 16 includes the first environment data 22, and the second communication data 46 includes the second environment data 52. In some implementations, the first environment data 22 indicates various characteristics and / or features of the first environment 10, and the second environment data 52 indicates various characteristics and / or features of the second environment 40. In some implementations, the first environment data 22 indicates physical elements located in the first environment 10, and the second environment data 52 indicates physical elements allocated in the second environment 40. In some implementations, the first environment data 22 includes a grid map of the first environment 10, and the second environment data 52 includes a grid map of the second environment 40. In some implementations, the first environment data 22 indicates a body posture of the first person 12, and the second environment data 52 indicates a body posture of the second person 42. In some implementations, the first environment data 22 indicates a facial expression of the first person 12, and the second environment data 52 indicates a facial expression of the second person 42. In some implementations, the first environmental data 22 includes a mesh map of the face of the first person 12 , and the second environmental data 52 includes a mesh map of the face of the second person 42 .

[0040] In various implementations, the first HMD 24 presents the ER object 30 representing the second person 42. In some implementations, the first HMD 24 generates the ER object 30 based on the second environmental data 52. For example, in some implementations, the second environmental data 52 encodes the ER object 30 representing the second person 42. In various implementations, the ER object 30 includes an ER representation of the second person 42. For example, in some implementations, the ER object 30 includes an avatar of the second person 42. In some implementations, the second environmental data 52 indicates a body posture of the second person 42, and the ER object 30 has a posture that is within a similarity to the body posture of the second person 42. In some implementations, the second environmental data 52 indicates a physical facial expression of the second person 42, and the ER face of the ER object 30 has an ER expression that is within a similarity to the physical facial expression of the second person 42. In various implementations, the second environmental data 52 indicates a movement of the second person 42, and the ER object 30 mimics the movement of the second person 42.

[0041] In various implementations, the second HMD 54 presents an ER object 60 representing the first person 12. In some implementations, the second HMD 54 generates the ER object 60 based on the first environment data 22. For example, in some implementations, the first environment data 22 encodes the ER object 60 representing the first person 12. In various implementations, the ER object 60 includes an ER representation of the first person 12. For example, in some implementations, the ER object 60 includes an avatar of the first person 12. In some implementations, the first environment data 22 indicates a body posture of the first person 12, and the ER object 60 has a posture that is within a similarity to the body posture of the first person 12 (e.g., within a similarity threshold of the body posture of the first person 12). In some implementations, the first environment data 22 indicates a physical facial expression of the first person 12, and the ER face of the ER object 60 has an ER expression that is within a similarity to the physical facial expression of the first person 12 (e.g., within a similarity threshold of the physical facial expression of the first person 12). In various implementations, the first environment data 22 indicates the movement of the first person 12 , and the ER object 60 mimics the movement of the first person 12 .

[0042] In various implementations, the first HMD 24 presents the ER object 30 based on the type of device associated with the second person 42. In some implementations, the first HMD 24 generates and presents the ER object 30 in response to determining that the second person 42 is using an HMD rather than a non-HMD. In some implementations, in response to acquiring the second communication data 46 including the second environment data 52, the first HMD 24 determines that the second person 42 is using a second HMD 54. In some implementations, the first HMD 24 spatializes the second audio data 48 to provide the appearance that the second audio data 48 originates from the ER object 30. Presenting the ER object 30 enhances the user experience of the first HMD 24. For example, presenting the ER object 30 provides the appearance that the second person 42 is in the first environment 10, even though the second person 42 is actually remote.

[0043] exist Figure 1D In the example of FIG. 1 , the first person 12 and the second person 42 are in the first environment 10. Therefore, the first electronic device 14 and the second electronic device 44 are in the first environment 10. Figure 1D In the example of , the first electronic device 14 determines that the second electronic device 44 is in the first environment 10. When the first electronic device 14 and the second electronic device 44 are in the same environment, the environment can be referred to as a shared environment (e.g., a shared physical setting). When the first electronic device 14 detects that the second electronic device 44 is in the same environment as the first electronic device 14, the first electronic device 14 considers the second electronic device 44 to be local (e.g., not remote).

[0044] In various implementations, in response to determining that the second electronic device 44 is local, the first electronic device 14 masks a portion of the second communication data 46. For example, in some implementations, the first electronic device 14 forgoes displaying the second video data 50 on the display of the first electronic device 14. Because the first electronic device 14 does not display the second video data 50, the first electronic device 14 allows the first person 12 to view the second person 42 without being distracted by the presentation of the second video data 50. In some implementations, the first electronic device 14 forgoes playing the second audio data 48 through the speaker of the first electronic device 14. In some implementations, not playing the second audio data 48 allows the first person 12 to hear the voice 80 of the second person 42. Because the first electronic device 14 does not play the second audio data 48, the second audio data 48 does not interfere with the voice 80 of the second person 42, thereby allowing the first person 12 to hear the voice 80. Similarly, in some implementations, in response to determining that the first electronic device 14 is local, the second electronic device 44 masks a portion of the first communication data 16 (e.g., the second electronic device 44 abandons displaying the first video data 20 and / or abandons playing the first audio data 18). As described herein, in some implementations, the first communication data 16 includes the first environment data 22 (e.g., Figure 1C ), and the second communication data 46 includes the second environment data 52 (as shown Figure 1C In such implementations, when the first electronic device 14 and the second electronic device 44 are local, the first electronic device 14 masks a portion of the second environment data 52 and the second electronic device 44 masks a portion of the first environment data 22.

[0045] In some implementations, the first electronic device 14 displays a first video pass-through 74 of the first environment 10. In some implementations, the first electronic device 14 includes an image sensor (e.g., a rear-facing camera) with a first field of view 72. The first video pass-through 74 represents a video feed captured by the image sensor of the first electronic device 14. Because the second person 42 is in the first field of view 72, the first video pass-through 74 includes a representation of the second person 42 (e.g., a video feed of the second person 42). Similarly, in some implementations, the second electronic device 44 displays a second video pass-through 78 of the first environment 10. In some implementations, the second electronic device 44 includes an image sensor with a second field of view 76. The second video pass-through 78 represents a video feed captured by the image sensor of the second electronic device 44. Because the first person 12 is in the first field of view 72, the second video pass-through 78 includes a representation of the first person 12 (e.g., a video feed of the first person 12).

[0046] See also Figure 1EIn response to determining that the second electronic device 44 is local, the first HMD 24 masks a portion of the second communication data 46. For example, in some implementations, the first HMD 24 abandons displaying the second video data 50 and / or the ER object 30 representing the second person 42 (e.g., Figure 1C ). In some implementations, the first HMD 24 forgoes playing the second audio data 48 through the speaker of the first HMD 24. In some implementations, not playing the second audio data 48 allows the first person 12 to hear the voice 80 of the second person 42. Since the first HMD 24 does not play the second audio data 48, the second audio data 48 does not interfere with the voice 80 of the second person 42, thereby allowing the first person 12 to hear the voice 80.

[0047] In some implementations, the first HMD 24 presents a first pass-through 84 of the first environment 10. In some implementations, the first HMD 24 includes an environmental sensor (e.g., a depth sensor such as a depth camera, and / or an image sensor such as a rear-facing camera) having a first detection field 82. In some implementations, the first pass-through 84 includes a sensor similar to Figure 1D 1. The video pass-through of the first video pass-through 74 is shown. In some implementations, the first pass-through 84 includes an optical pass-through in which light (e.g., natural light or artificial light) from the first environment 10 is allowed to enter the eyes of the first person 12. Since the second person 42 is in the first detection field 82, the first pass-through 84 includes a representation of the second person 42 (e.g., a video feed of the second person 42, or light reflected from the second person 42). Presenting the first pass-through 84 enhances the user experience provided by the first HMD 24 because the first pass-through 84 tends to have lower latency than displaying the second video data 50 or generating the ER object 30 based on the second environment data 52.

[0048] refer to Figure 1F In response to determining that the first HMD 24 is local, the second HMD 54 masks a portion of the first communication data 16. For example, in some implementations, the second HMD 54 forgoes displaying the first video data 20 or the ER object 60 representing the first person 12 (e.g., Figure 1C ). In some implementations, the second HMD 54 forgoes playing the first audio data 18 through the speakers of the second HMD 54. In some implementations, not playing the first audio data 18 allows the second person 42 to hear the voice of the first person 12 (e.g., direct audio from the first person 12). Since the second HMD 54 does not play the first audio data 18, the first audio data 18 does not interfere with the voice of the first person 12, thereby allowing the second person 42 to hear the voice.

[0049] In some implementations, the second HMD 54 presents a second pass-through 88 of the first environment 10. In some implementations, the second HMD 54 includes an environmental sensor (e.g., a depth sensor such as a depth camera, and / or an image sensor such as a rear-facing camera) having a second detection field 86. In some implementations, the second pass-through 88 includes a sensor similar to Figure 1D 88. In some implementations, the second pass-through 88 includes an optical pass-through in which light (e.g., natural light or artificial light) from the first environment 10 is allowed to enter the eyes of the second person 42. Since the first person 12 is in the second detection field 86, the second pass-through 88 includes a representation of the first person 12 (e.g., a video feed of the first person 12, or light reflected from the first person 12). Presenting the second pass-through 88 enhances the user experience provided by the second HMD 54 because the second pass-through 88 tends to have lower latency than displaying the first video data 20 or generating the ER object 60 based on the first environment data 22.

[0050] like Figure 1G As shown, in some implementations, the first HMD 24 communicates with multiple devices. Figure 1G A third environment 90 including a third person 92 associated with a third electronic device 94 is shown. Figure 1G Also shown is a fourth environment 100 including a fourth person 102 associated with a third HMD 104 (eg, the fourth person 102 is wearing the third HMD 104 on the head of the fourth person 102). Figure 1G In the example of FIG. 1 , the first HMD 24 communicates with the second HMD 54, the third electronic device 94, and the third HMD 104. In some implementations, the first HMD 24, the second HMD 54, the third electronic device 94, and the third HMD 104 are in a conference call (e.g., in a conference call session).

[0051] In some implementations, the first HMD 24 determines that the second HMD 54 is local because the second HMD 54 is in the same environment as the first HMD 24. Therefore, as described herein, in some implementations, the first HMD 24 masks a portion of the communication data associated with the second HMD 24. Additionally, as described herein, in some implementations, the first HMD 24 presents a first pass-through 84 of the first environment 10. Figure 1G As shown, the first HMD 24 presents a first pass-through 84 within the first ER scenery 26 .

[0052] In some implementations, the first HMD 24 determines that the third electronic device 94 is remote because the third electronic device 94 is not in the same environment as the first HMD 24. In some implementations, the first HMD 24 determines that the third electronic device 94 is a non-HMD (e.g., a tablet, smartphone, media player, laptop, or desktop computer). Figure 1G In the example of , the first HMD 24 displays a card GUI element 96 that includes video data 98 ​​associated with (eg, originating from) a third electronic device 94 .

[0053] In some implementations, the first HMD 24 determines that the third HMD 104 is remote because the third HMD 104 is not in the same environment as the first HMD 24. In some implementations, the first HMD 24 determines that the fourth person 102 is using an HMD-type device. Figure 1G As shown, the first HMD 24 displays an ER object 106 representing the fourth person 102. In some implementations, the ER object 106 includes an ER representation of the fourth person 102. In some implementations, the ER object 106 includes an avatar of the fourth person 102.

[0054] Figure 2A 200 is a flowchart representation of a method 200 for presenting communication data. In various specific implementations, the method 200 is performed by a first device associated with a first person. In some specific implementations, the first device includes a display, a non-transient memory, and one or more processors coupled to the display and the non-transient memory. In some specific implementations, the method 200 is performed by a first electronic device 14, a second electronic device 44, a first HMD 24, and / or a second HMD 54. In some specific implementations, the method 200 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 200 is performed by a processor that executes a code stored in a non-transient computer-readable medium (e.g., a memory).

[0055] As indicated at block 210, in various implementations, method 200 includes obtaining communication data associated with (eg, originating from or generated by) a second device corresponding to a second person. Figure 1A As shown, the first electronic device 14 obtains second communication data 46 associated with the second electronic device 44. In some specific implementations, the method 200 includes receiving the communication data via a network. In some specific implementations, the communication data includes audio data (e.g., network audio, such as Figures 1A to 1F In some implementations, the communication data includes video data (e.g., Figures 1A to 1FIn some implementations, the communication data includes environmental data (e.g., Figure 1C In some implementations, the communication data (eg, video data and / or environmental data) encodes an ER object (eg, an avatar) representing the second person.

[0056] As shown in block 220, in various implementations, method 200 includes determining whether the first device and the second device are in a shared environment. In some implementations, method 200 includes determining, by the first device, whether the second device is in the same environment as the first device. For example, first electronic device 14 determines whether second electronic device 44 is in first environment 10. In some implementations, method 200 includes determining, by the first device, whether the second device is local or remote.

[0057] As indicated at block 230, in various implementations, method 200 includes, in response to determining that the first device and the second device are not in a shared environment, displaying an ER representation of the second person based on communication data associated with the second device. In some implementations, displaying the ER representation includes displaying video data included in the communication data. For example, Figure 1A As shown, the first electronic device 14 displays the second video data 50 in response to determining that the second electronic device 44 is not in the first environment 10. In some specific implementations, displaying the ER representation includes displaying the ER scene, displaying a card GUI element within the ER scene, and displaying the video data associated with the second device within the card GUI element. Figure 1B As shown, the first HMD 24 displays the first ER scenery 26, the card GUI element 28 within the first ER scenery 26, and the second video data 50 within the card GUI element 28. In some specific implementations, displaying the ER representation includes generating an ER object based on the video data and / or environmental data associated with the second device, and displaying the ER object in the ER scenery. Figure 1C As shown, the first HMD 24 generates and displays an ER object 30 representing a second person 42 .

[0058] refer to Figure 2B As shown in block 240, in some implementations, method 200 includes determining a device type of the second device, and generating an ER representation of the second person based on the device type. Figure 1B to Figure 1C As shown, the first HMD 24 determines the device type of the device associated with the second person 42 and generates an ER representation of the second person based on the device type. Figure 1B In the example of , the first HMD 24 determines that the second person 42 is using a non-HMD, and the first HMD 24 decodes the second video data 50 and displays it within the card GUI element 28. Figure 1C In the example of , the first HMD 24 determines that the second person 42 is using the HMD, and the first HMD 24 generates and displays the ER object 30 representing the second person 42.

[0059] As indicated at block 242, in some implementations, method 200 includes, in response to the device type being the first device type, generating a first type of ER representation of the second person based on the communication data associated with the second device. As indicated at block 242a, in some implementations, generating the first type of ER representation of the second person includes generating a three-dimensional (3D) ER object (e.g., an avatar) representing the second person. As indicated at block 242b, in some implementations, the first device type includes an HMD. For example, Figure 1C As shown, the first HMD 24 generates an ER object 30 representing the second person 42 in response to determining that the second person 42 is using the HMD.

[0060] As shown in box 244, in some embodiments, method 200 includes, in response to the device type being the second device type, generating a second type of ER representation of the second person based on the communication data associated with the second device. As shown in box 244a, in some embodiments, generating the second type of ER representation of the second person includes generating a two-dimensional (2D) ER object representing the second person. As shown in box 244b, in some embodiments, the second type of ER representation includes a video of the second person. In some embodiments, the video is encoded in the communication data associated with the second device. As shown in box 244c, in some embodiments, the second device type includes a handheld device (e.g., a smart phone, a tablet, a laptop, a media player, and / or a watch). As shown in box 244d, in some embodiments, the second device type includes a device that is not an HMD (e.g., a non-HMD such as a handheld device, a desktop computer, a television, and / or a projector). As shown in box 244e, in some embodiments, method 200 includes generating a second type of ER representation of the second person in conjunction with a card (e.g., a card GUI element, such as Figure 1B The GUI element 28) of the card shown in the figure displays the second type of ER representation of the second person in the GUI element within the similarity. Figure 1B As shown, the first HMD 24 displays the second video data 50 in response to determining that the second person 42 is associated with the non-HMD. In some specific implementations, the method 200 includes changing the position of the GUI element (e.g., the card GUI element 28) within the first ER scenery based on the movement of the second person and / or the second electronic device within the second environment. For example, the method 200 includes moving the GUI element in the same direction as the second person and / or the second electronic device.

[0061] refer to Figure 2CIn some implementations, as shown in block 250, the first device includes one or more speakers, and method 200 includes outputting audio corresponding to the second person via the one or more speakers. In some implementations, the audio is spatialized to provide an appearance that the audio originates from the ER representation of the second person. For example, Figure 1B As described above, the first HMD 24 spatializes the second audio data 48 to provide the appearance that the second audio data 48 originates from the card GUI element 28. Similarly, as with respect to Figure 1C As described, the first HMD 24 spatializes the second audio data 48 to provide the appearance that the second audio data 48 originates from the ER object 30 representing the second person 42 .

[0062] As indicated at block 252, in some implementations, method 200 includes generating audio based on communication data associated with the second device. Figure 1A As shown, the first electronic device 14 extracts the second audio data 48 from the second communication data 46. In some implementations, the method 200 includes decoding the communication data to identify the audio (eg, the first electronic device 14 decodes the second communication data 46 to identify the second audio data 48).

[0063] As shown in block 254, in some implementations, the method 200 includes generating early reflections to provide the appearance that audio is being reflected from a surface. For example, the first HMD 24 generates early reflections for the second audio data 48 to provide the appearance that the sound corresponding to the second audio data 48 is being reflected from a surface of the first environment 10. In some implementations, the method 200 includes outputting the early reflections before outputting the audio (e.g., the first HMD 24 outputs the early reflections before playing the second audio data 48). In some implementations, the method 200 includes outputting the early reflections and the audio simultaneously (e.g., the first HMD 24 plays the early reflections of the second audio data 48 and the second audio data 48 simultaneously). In some implementations, the method 200 includes generating early reflections based on the type of the first environment. In some implementations, the first environment is a physical set, and the early reflections provide the appearance that the audio is being reflected from a physical surface of the physical set. In some implementations, the first environment is an ER set (e.g., a virtual environment), and the early reflections provide the appearance that the audio is being reflected from an ER surface (e.g., a virtual surface) of the ER set.

[0064] As shown in block 256, in some implementations, the method 200 includes generating late reverberations to provide the appearance that the audio is echoing. For example, the first HMD 24 generates late reverberations for the second audio data 48 to provide the appearance that the sound corresponding to the second audio data 48 is echoing in the first environment 10. In some implementations, the method 200 includes outputting the late reverberations after outputting the audio (e.g., the first HMD 24 outputs the late reverberations after playing the second audio data 48). In some implementations, the method 200 includes generating the late reverberations based on the type of the first environment. In some implementations, the first environment is a physical set, and the late reverberations provide the appearance that the audio is echoing in the physical set. In some implementations, the first environment is an ER set (e.g., a virtual environment), and the early reflections provide the appearance that the audio is reflecting off an ER surface (e.g., a virtual surface) of the ER set.

[0065] As indicated at block 260, in some implementations, method 200 includes, in response to determining that the first device and the second device are in a shared environment, forgoing displaying the ER representation of the second person. In some implementations, method 200 includes forgoing displaying video data included in the communication data associated with the second device. For example, Figure 1D As shown, in response to determining that the second electronic device 44 is in the same environment as the first electronic device 14, the first electronic device 14 abandons displaying the second video data 50. Similarly, Figure 1E As shown, in response to determining that the second electronic device 44 is in the same environment as the first electronic device 14, the first HMD 24 abandons displaying the second video data 50. In some specific implementations, the method 200 includes abandoning displaying the ER representation of the second person generated based on the communication data associated with the second person (for example, when the second person is local, abandoning displaying the avatar of the second person). For example, Figure 1F As shown, in response to determining that the second HMD 54 is in the same environment as the first HMD 24, the first HMD 24 relinquishes displaying the ER object 30 representing the second person.

[0066] As shown in block 262, in some implementations, method 200 includes transparently presenting a shared environment. Figure 1E to Figure 1F As shown, the first HMD 24 presents a first transmittance 84 of the first environment 10. As shown in block 262a, in some implementations, the method 200 includes presenting an optical transmittance of the shared environment. Figure 1E to Figure 1F As described above, in some embodiments, the first transparent transmission 84 includes optical transparent transmission. As shown in block 262b, in some embodiments, the method 200 includes video transparent transmission of the display shared environment. For example, display Figure 1D The first video pass-through 74 shown in FIG. Figure 1E to Figure 1FAs described above, in some implementations, the first passthrough 84 includes video passthrough. As shown in block 262c, in some implementations, the method 200 includes displaying a window and presenting the passthrough in the window. For example, Figure 1E to Figure 1F As shown, in Figure 1B A first passthrough 84 is shown within a card GUI element similar to the card GUI element 28 shown.

[0067] refer to Figure 2D As shown in block 220a, in some implementations, the shared environment includes a shared physical setting. Figure 1A As described above, in some implementations, the first environment 10 includes a first physical scene. As shown in block 220b, in some implementations, the shared environment includes a shared ER scene. For example, as shown relative to Figure 1A As mentioned above, in some implementations, the first environment 10 includes a first ER scene.

[0068] As shown in block 220c, in some implementations, determining whether the first device and the second device are in a shared environment includes determining whether an identifier (ID) associated with the second device can be detected via short-range communication. Exemplary short-range communication includes Bluetooth, Wi-Fi, near field communication (NFC), ZigBee, etc. For example, with respect to Figure 1A , the first electronic device 14 determines whether the first electronic device 14 can detect the ID associated with the second electronic device 44. Figure 1A In the example of , the first electronic device 14 does not detect the ID associated with the second electronic device 44 through short-range communication. Therefore, the first electronic device 14 determines that the second electronic device 44 is remote. Figure 1C In some implementations, the first HMD 24 determines that the second HMD 54 is remote because the first HMD 24 cannot detect the ID of the second HMD 54 via short-range communication. Figure 1F , the first HMD 24 determines that the second HMD 54 is local because the first HMD 24 detects the ID of the second HMD 54 via short-range communication.

[0069] As shown in block 220d, in some implementations, determining whether the first device and the second device are in a shared environment includes determining whether audio received via a microphone of the first device is within a similarity to audio encoded in communication data associated with the second device. For example, in some implementations, method 200 includes determining whether direct audio from the second person is within a similarity to network audio associated with the second device. For example, relative to Figure 1DIn some implementations, the first electronic device 14 determines that the second electronic device 44 is local because the voice 80 of the second person 42 is within a similarity to the second audio data 48 encoded in the second communication data 46 .

[0070] As shown in block 220e, in some implementations, determining whether the first device and the second device are in a shared environment includes determining whether the second person is in the shared environment based on an image captured via an image sensor of the first device. Figure 1D In some implementations, the first electronic device 14 captures an image corresponding to the first field of view 72, and performs facial detection / recognition on the image to determine whether the second person 42 is in the first environment 10. Since the second person 42 is in the first field of view 72, after performing facial detection / recognition on the image, the first electronic device 14 determines that the second person 42 is local.

[0071] Figure 3A 300 is a flowchart representation of a method 300 for masking communication data. In various specific implementations, the method 300 is performed by a first device, which includes an output device, a non-transitory memory, and one or more processors coupled to the output device and the non-transitory memory. In some specific implementations, the method 300 is performed by a first electronic device 14, a second electronic device 44, a first HMD 24, and / or a second HMD 54. In some specific implementations, the method 300 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 300 is performed by a processor that executes a code stored in a non-transitory computer-readable medium (e.g., a memory).

[0072] As shown in block 310, in various implementations, method 300 includes obtaining communication data associated with (e.g., originating from or generated by) the second device while the first device is in a communication session with the second device. Figure 1D As shown, the first electronic device 14 obtains second communication data 46 associated with the second electronic device 44. In some specific implementations, the method 300 includes receiving the communication data via a network. In some specific implementations, the communication data includes audio data (e.g., network audio, such as Figures 1A to 1F In some implementations, the communication data includes video data (e.g., Figures 1A to 1F In some implementations, the communication data includes environmental data (e.g., Figure 1C In some implementations, the communication data (eg, video data and / or environmental data) encodes an ER object (eg, an avatar) representing the second person.

[0073] As shown in block 320, in various implementations, method 300 includes determining that the first device and the second device are in a shared environment. In some implementations, method 300 includes determining, by the first device, that the second device is in the same environment as the first device. For example, with respect to Figure 1D , the first electronic device 14 determines that the second electronic device 44 is in the first environment 10. In some implementations, the method 300 includes determining, by the first device, that the second device is local.

[0074] As shown in block 330, in various implementations, method 300 includes masking a portion of the communication data to prevent the output device from outputting the portion of the communication data. In some implementations, masking a portion of the communication data includes forgoing presenting the portion of the communication data. For example, with respect to Figure 1D , the first electronic device 14 forgoes presenting the second communication data 46 in response to determining that the second electronic device 44 is local. In various implementations, when the second device is local, masking a portion of the communication data enhances the user experience of the first device by allowing the first person to directly hear and / or see the second person without the delay associated with network communications.

[0075] refer to Figure 3B As shown in block 320a, in some implementations, determining whether the first device and the second device are in a shared physical setting includes detecting an identifier associated with the second device via short-range communication. Exemplary short-range communications include Bluetooth, Wi-Fi, near field communication (NFC), ZigBee, etc. For example, with respect to Figure 1D In some implementations, the first electronic device 14 determines that the second electronic device 44 is local in response to detecting an ID associated with the second electronic device 44 via short-range communication. Figure 1F In some implementations, the first HMD 24 determines that the second HMD 54 is local in response to detecting the ID of the second HMD 54 via short-range communication.

[0076] As shown in block 320b, in some implementations, determining that the first device and the second device are in a shared physical setting includes detecting, via a microphone of the first device, first audio that is within a similarity to second audio encoded in the communication data (e.g., within a similarity threshold of the second audio). In some implementations, method 300 includes determining that the second device is local in response to detecting direct audio that is within a similarity to network audio encoded in the communication data. For example, relative to Figure 1EIn some implementations, the first HMD 24 determines that the second electronic device 44 is local in response to detecting the voice 80 via the microphone of the first HMD 24 and determining that the voice 80 is within a similarity to the second audio data 48 encoded in the second communication data 46.

[0077] As shown in block 320c, in some implementations, determining that the first device and the second device are in a shared physical setting includes detecting, via an image sensor of the first device, a person associated with the second device. Figure 1D In some implementations, the first electronic device 14 captures an image corresponding to the first field of view 72, and performs facial detection / recognition on the image to determine whether the second person 42 is in the first environment 10. Since the second person 42 is in the first field of view 72, after performing facial detection / recognition on the image, the first electronic device 14 determines that the second device 44 is local.

[0078] As shown in block 330a, in some implementations, the output device includes a speaker, and masking a portion of the communication data includes masking an audio portion of the communication data to prevent the speaker from playing the audio portion of the communication data. Figure 1D As described, in some specific implementations, the first electronic device 14 masks the second audio data 48 to prevent the speaker of the first electronic device 14 from playing the second audio data 48. As described herein, forgoing playing the second audio data 48 at the first electronic device 14 enhances the user experience of the first electronic device 14 by allowing the first person 12 to listen to the speech 80 spoken by the second person 42 without interference from the second audio data 48.

[0079] As indicated at block 330b, in some implementations, the output device includes a display, and masking a portion of the communications data includes masking a video portion of the communications data to prevent the display from displaying the video portion of the communications data. Figure 1E As described, in some implementations, the first HMD 24 masks the second video data 50 so as to prevent the display of the first HMD 24 from displaying the second video data 50. As described herein, forgoing displaying the second video data 50 at the first HMD 24 enhances the user experience of the first HMD 24 by not distracting the first person 12 from the second video data 50 and allowing the first person 12 to see the first pass-through 84 of the first environment 10.

[0080] As shown in block 330c, in some implementations, the communication data encodes an ER representation of a person associated with the second device, and masking a portion of the communication data includes forgoing displaying the ER representation of the person. Figure 1FAs shown, in some implementations, the first HMD 24 forgoes displaying the ER object 30 representing the second person 42 in response to determining that the second person 42 is in the same environment as the first HMD 24. As described herein, forgoing displaying the ER object 30 at the first HMD 24 enhances the user experience of the first HMD 24 by allowing the first person 12 to see the first pass-through 84 associated with a lower latency than generating and displaying the ER object 30.

[0081] As shown in block 340, in some implementations, method 300 includes presenting a pass-through of the shared physical scene. Figure 1E to Figure 1F As shown, the first HMD 24 presents a first pass-through 84 of the first environment 10. As shown in block 340a, in some implementations, presenting the pass-through includes presenting an optical pass-through of the shared physical scenery. Figure 1E to Figure 1F As described above, in some implementations, the first pass-through 84 includes an optical pass-through, wherein the first HMD 24 allows light from the first environment 10 to reach the eyes of the first person 12. As shown in block 340b, in some implementations, the presentation pass-through includes a video pass-through showing the shared physical scenery. Figure 1E to Figure 1F As mentioned above, in some specific implementations, the first pass-through 84 includes video pass-through, wherein the display of the first HMD 24 displays a video stream corresponding to the first detection field 82 .

[0082] See also Figure 3C In some implementations, the method 300 includes detecting movement of the second device away from the shared physical setting, and abandoning masking a portion of the communication data to allow the output device to output the portion of the communication data, as shown in block 350. For example, in some implementations, the first electronic device 14 detects that the second electronic device 44 has left the first environment 10, and the first electronic device 14 abandons masking the second communication data 46 in response to detecting that the second electronic device 44 has left the first environment 10.

[0083] As shown in block 350a, in some implementations, detecting movement of the second device away from the shared physical setting includes determining that an identifier associated with the second device cannot be detected via short-range communication. For example, in some implementations, the first electronic device 14 and / or the first HMD 24 determines that an ID associated with the second electronic device 44 and / or the second HMD 54 cannot be detected via short-range communication.

[0084] As shown in block 350b, in some implementations, detecting movement of the second device away from the shared physical setting includes determining that the first audio detected via the microphone of the first device is not within a similarity to the second audio encoded in the communication data. For example, in some implementations, the first electronic device 14 and / or the first HMD 24 determines that the audio detected via the microphone of the first electronic device 14 and / or the first HMD 24 does not match the second audio data 48.

[0085] As shown in block 350c, in some implementations, detecting movement of the second device away from the shared physical setting includes determining that environmental data captured by an environmental sensor of the first device indicates that a person associated with the second device has moved away from the shared physical setting. For example, in some implementations, the first electronic device 14 and / or the first HMD 24 determines that environmental data captured by an environmental sensor of the first electronic device 14 and / or the first HMD 24 (e.g., images captured by a camera and / or depth data captured by a depth sensor) indicates that the second person 42 is not in the first environment 10.

[0086] As shown in block 350d, in some implementations, the output device includes a speaker, and abandoning masking a portion of the communication data includes outputting an audio portion of the communication data via the speaker. For example, in response to determining that the second electronic device 44 and / or the second HMD 54 has left the first environment 10, the first electronic device 14 and / or the first HMD 24 outputs the second audio data 48.

[0087] As shown in block 350e, in some implementations, outputting the audio portion includes spatializing the audio portion to provide the appearance that the audio portion is originating from the ER representation of the person associated with the second device. Figure 1B to Figure 1C As described, the first electronic device 14 and / or the first HMD 24 spatializes the second audio data 48 to provide the appearance that the second audio data 48 is originating from the card GUI element 28 or the ER object 30 representing the second person 42 .

[0088] As shown in block 350f, in some implementations, the output device includes a display, and wherein abandoning masking a portion of the communication data includes displaying a video portion of the communication data on the display. For example, in some implementations, in response to detecting that the second electronic device 44 has left the first environment 10, the first electronic device 14 and / or the first HMD 24 displays the second video data 50.

[0089] As shown in block 350g, in some implementations, the communication data encodes an ER representation of a person associated with the second device, and abstaining from masking a portion of the communication data includes displaying the ER representation of the person. For example, in some implementations, in response to detecting that the second HMD 54 has left the first environment 10, the first HMD 24 displays the ER object 30 representing the second person 42.

[0090] Figure 4 4 is a block diagram of a device 400 for presenting / masking communication data according to some implementations. Although some specific features are shown, those of ordinary skill in the art will recognize from the present disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some implementations, the device 400 includes one or more processing units (CPUs) 401, a network interface 402, a programming interface 403, a memory 404, an environmental sensor 407, one or more input / output (I / O) devices 409, and one or more communication buses 405 for interconnecting these and various other components.

[0091] In some implementations, a network interface 402 is provided to, among other purposes, establish and maintain a metadata tunnel between a cloud-hosted network management system and at least one private network including one or more compatible devices. In some implementations, one or more communication buses 405 include circuits for interconnecting and controlling communications between system components. Memory 404 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 404 optionally includes one or more storage devices located remotely from one or more CPUs 401. Memory 404 includes non-transitory computer-readable storage media.

[0092] In various implementations, the environment sensor 407 includes an image sensor. For example, in some implementations, the environment sensor 407 includes a camera (e.g., a scene-facing camera, an outward-facing camera, or a rear-facing camera). In some implementations, the environment sensor 407 includes a depth sensor. For example, in some implementations, the environment sensor 407 includes a depth camera.

[0093] In some implementations, the memory 404 or the non-transitory computer-readable storage medium of the memory 404 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 406, a data acquirer 410, an environment analyzer 420, and an ER experience generator 430. In various implementations, the device 400 executes FIG. 2A to FIG. 2D In various implementations, the device 400 performs FIG. 3A to FIG. 3C The method 300 is shown. In various implementations, the device 400 implements the first electronic device 14, the second electronic device 44, the first HMD 24, the second HMD 54, the third electronic device 94, and / or the third HMD 104.

[0094] In some implementations, the data acquirer 410 acquires data. In some implementations, the data acquirer 410 acquires communication data associated with another device (e.g., Figures 1A to 1F In some implementations, the data acquirer 410 performs at least a portion of the method 200. For example, in some implementations, the data acquirer 410 performs the following steps: Figure 2A In some implementations, the data acquirer 410 performs at least a portion of the method 300. For example, in some implementations, the data acquirer 410 performs the operations represented by the block 210 shown in FIG. Figure 3A The operations are represented by block 310. To this end, data fetcher 410 includes instructions 410a and heuristics and metadata 410b.

[0095] As described herein, in some implementations, the environment analyzer 420 determines whether the device 400 and the other device are in a shared environment. For example, the environment analyzer 420 determines whether the first electronic device 14 and the second electronic device 44 are in the first environment 10. In some implementations, the environment analyzer 420 performs at least a portion of the method 200. For example, in some implementations, the environment analyzer 420 performs the following steps: Figure 2A and Figure 2D In some implementations, the environment analyzer 420 performs at least a portion of the method 300. For example, in some implementations, the environment analyzer 420 performs the operations represented by the block 220 in FIG. Figure 3A and Figure 3B The operations are represented by block 320 in FIG. To this end, environment analyzer 420 includes instructions 420a and heuristics and metadata 420b.

[0096] In some implementations, in response to environment analyzer 420 determining that the other device is remote (e.g., not in the same environment as device 400), ER experience generator 430 displays an ER representation of a person associated with the other device. In some implementations, ER experience generator 430 performs at least a portion of method 200. For example, in some implementations, ER experience generator 430 performs the steps of: FIG. 2A to FIG. 2CIn some implementations, in response to environment analyzer 420 determining that the other device is local (e.g., in the same environment as device 400), ER experience generator 430 masks a portion of the communication data. In some implementations, ER experience generator 430 performs at least a portion of method 300. For example, in some implementations, ER experience generator 430 performs operations represented by blocks 230, 240, 250, and 260. FIG. 3A to FIG. 3C The operations are represented by blocks 330, 340, and 350. To this end, ER experience generator 430 includes instructions 430a and heuristics and metadata 430b.

[0097] In some implementations, one or more I / O devices 409 include a device for capturing and communicating with the environment (e.g., Figures 1A to 1G One or more sensors for environmental data associated with the first environment 10 shown in the figure. For example, in some specific implementations, one or more I / O devices 409 include an image sensor (e.g., a camera), an ambient light sensor (ALS), a microphone, and / or a position sensor. In some specific implementations, one or more I / O devices 409 include a display (e.g., an opaque display or an optical see-through display) and / or a speaker for presenting communication data.

[0098] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0099] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0100] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and in the appended claims, the singular forms of "a", "an" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "include" and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0101] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method, comprising: At a first device associated with a first person, the first device comprising a display, a non-transitory memory, and one or more processors coupled to the display and the non-transitory memory: obtaining communication data associated with a second device corresponding to a second person; determining whether the first device and the second device are in a shared environment; and In response to determining that the first device and the second device are not in a shared environment, a virtual representation of the second person is displayed based on the communication data associated with the second device, wherein the virtual representation of the second person is generated based on a device type of the second device corresponding to the second person.

2. The method according to claim 1, further comprising: The device type of the second device is determined.

3. The method according to claim 2, further comprising: In response to the device type being a first device type, the virtual representation of the first type of the second person is generated based on the communication data associated with the second device. 4 . The method of claim 3 , wherein generating the virtual representation of the first type of the second person comprises generating a three-dimensional (3D) virtual object representing the second person.

5. The method of any one of claims 3 and 4, wherein the first device type comprises a head mounted device (HMD).

6. The method according to claim 2, further comprising: In response to the device type being a second device type, the virtual representation of the second person of a second type is generated based on the communication data associated with the second device. 7 . The method of claim 6 , wherein generating the virtual representation of the second type of the second person comprises generating a two-dimensional (2D) virtual object representing the second person.

8. The method of claim 6, wherein the virtual representation of the second type comprises a video of the second person, wherein the video is encoded in the communication data associated with the second device.

9. The method of any one of claims 6 to 8, wherein the second device type comprises a handheld device.

10. The method of any one of claims 6 to 8, wherein the second device type comprises a device that is not a head mounted device.

11. The method according to any one of claims 6 to 8, further comprising: The virtual representation of the second type of the second person is displayed within a graphical user interface (GUI) element within a card similarity threshold.

12. The method according to any one of claims 1 to 4 and 6 to 8, wherein the first device comprises one or more speakers, and the method further comprises: Audio corresponding to the second person is output via the one or more speakers, wherein the audio is spatialized to provide an appearance that the audio is originating from the virtual representation of the second person.

13. The method of claim 12, further comprising generating the audio based on the communication data associated with the second device.

14. The method of claim 12, further comprising generating early reflections to provide the appearance that the audio is reflecting off a surface.

15. The method of claim 12, further comprising generating late reverberation to provide the audio with the appearance of having an echo.

16. The method according to any one of claims 1 to 4, 6 to 8 and 13 to 15, further comprising: In response to determining that the first device and the second device are in a shared environment, displaying the virtual representation of the second person is abandoned.

17. The method according to claim 16, further comprising: Presenting a transparent transmission of the shared environment. The method of claim 17 , wherein presenting the see-through comprises presenting an optical see-through of the shared environment. The method of claim 17 , wherein presenting the pass-through comprises displaying a video pass-through of the shared environment.

20. The method according to any one of claims 17 to 19, wherein presenting the pass-through comprises displaying a window and presenting the pass-through within the window.

21. The method of any one of claims 1 to 4, 6 to 8, 13 to 15 and 17 to 19, wherein the shared environment comprises a shared physical set.

22. The method of any one of claims 1 to 4, 6 to 8, 13 to 15, and 17 to 19, wherein the shared environment comprises a shared augmented reality (ER) set.

23. The method of any one of claims 1 to 4, 6 to 8, 13 to 15, and 17 to 19, wherein determining whether the first device and the second device are in the shared environment comprises determining whether an identifier associated with the second device is detectable via short-range communication.

24. A method according to any one of claims 1 to 4, 6 to 8, 13 to 15 and 17 to 19, wherein determining whether the first device and the second device are in the shared environment includes determining whether audio received via a microphone of the first device is within a similarity threshold of audio encoded in the communication data associated with the second device.

25. The method of any one of claims 1 to 4, 6 to 8, 13 to 15, and 17 to 19, wherein determining whether the first device and the second device are in the shared environment comprises determining whether the second person is in the shared environment based on an image captured via an image sensor of the first device.

26. An apparatus comprising: one or more processors; monitor; non-transitory memory; as well as One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to perform any of the methods of claims 1 to 25.

27. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device having a display, cause the device to perform any of the methods of claims 1 to 25.

28. An apparatus comprising: one or more processors; monitor; non-transitory memory; as well as Means for causing the device to perform any of the methods according to claims 1 to 25.

Images