Methods, electronic devices, storage devices, and program products for use in a communication environment

Abstract

This disclosure relates to techniques for collaboration in multi-user communication environments. One such technique includes receiving, at a first communication device, data associated with a multi-user communication session between a first user of the first communication device and a second user of a second communication device; presenting, at the first communication device, a non-extended reality graphical user interface (GUI) including a non-extended reality representation of a virtual object included in the multi-user communication session and a representation of the second user based on the data associated with the multi-user communication session; and in response to data indicating an interaction, updating, at the first communication device, the non-extended reality GUI to show the interaction between the representation of the second user and the virtual object.

Description

Background Technology

[0001] This disclosure relates generally to multi-user environments. More specifically, but not in a limiting sense, this disclosure relates to techniques and systems for discovering extended reality (XR) services.

[0002] Some devices are capable of generating and presenting XR environments (XREs). An XRE environment can include a fully or partially simulated environment that people perceive and / or interact with via electronic systems. In an XRE, a subset of a person's physical motion or a representation thereof is tracked, and in response, one or more characteristics of one or more virtual elements simulated in the XRE are adjusted in a manner consistent with at least one physical law. Some XREs allow multiple users to interact with each other within the XRE. However, there is a need for improved techniques for discovering XR services and users. Attached Figure Description

[0003] Figure 1 A simplified system diagram according to one or more implementation schemes is shown in block diagram form.

[0004] Figure 2 A diagram illustrating an example operating environment based on one or more implementation schemes is shown.

[0005] Figures 3 to 5 This is a block diagram illustrating an exemplary communication environment according to various aspects of this disclosure.

[0006] Figure 6 This is a block diagram illustrating an exemplary communication environment using gestures according to various aspects of this disclosure.

[0007] Figure 7 This is a block diagram illustrating an exemplary communication environment using device sound according to various aspects of this disclosure.

[0008] Figure 8 This is a block diagram illustrating an exemplary communication environment using user voice according to various aspects of this disclosure.

[0009] Figure 9 Techniques for identifying XR services according to various aspects of this disclosure are shown.

[0010] Figure 10A and Figure 10B An exemplary system for various XR technologies is shown according to one or more embodiments. Detailed Implementation

[0011] This disclosure relates to techniques for users seeking to collaborate in multi-user communication sessions such as XR communication sessions (XRC). An XRC session can be advertised as available for other devices locally to join, and these other devices may be known or unknown to the first user. Additionally, multiple advertised XRC sessions may be locally available for joining. To facilitate joining a desired XRC session, techniques for identifying available XRC sessions may be desired. Therefore, the techniques described herein provide methods for identifying available XRC sessions to join.

[0012] Humans can interact with and / or perceive the physical environment or physical world without the aid of electronic devices. The physical environment can include physical features, such as physical objects or surfaces. An example of a physical environment is a physical forest that includes physical plants and animals. Humans can directly perceive and / or interact with the physical environment through various means, such as hearing, vision, taste, touch, and smell. In contrast, humans can interact with and / or perceive a fully or partially simulated XR environment (XRE) using electronic devices. This XRE can include mixed reality (MR) content, augmented reality (AR) content, virtual reality (VR) content, and so on. Using an XR system, a person's physical movements, or some of their representations, can be tracked, and in response, the characteristics of virtual objects simulated in the XRE can be adjusted in a manner consistent with at least one physical law. For example, the XR system can detect movement of the user's head and adjust the graphical and auditory content presented to the user (similar to how such views and sounds change in a physical environment). For example, the XR system can detect movement of the electronic device (e.g., mobile phone, tablet, laptop, etc.) that presents the XRE, and adjust the graphical and auditory content presented to the user (similar to how such views and sounds change in a physical environment). In some cases, the XR system can adjust the characteristics of the graphical content in response to other inputs such as representations of physical motion (e.g., voice commands).

[0013] Many different types of electronic systems enable users to interact with and / or sense XREs. A non-exclusive list of examples includes head-up displays (HUDs), head-mounted systems, projection-based systems, windows or vehicle windshields with integrated display capabilities, displays formed as lenses placed over the user's eyes (e.g., contact lenses), head-mounted receivers / headsets, input systems with or without haptic feedback (e.g., wearable or handheld controllers), speaker arrays, smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have opaque displays and one or more speakers. Other head-mounted systems may be configured to accept opaque external displays (e.g., smartphones). Head-mounted systems may include one or more image sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light is directed to the user's eyes. Displays can utilize various display technologies, such as uLED, OLED, LED, liquid crystal on silicon, laser scanning light sources, digital light projection, or combinations thereof. Optical waveguides, optical reflectors, holographic media, optical combiners, or combinations thereof, or other similar technologies can be used as the medium. In some implementations, transparent or translucent displays can be selectively controlled to become opaque. Projection-based systems can utilize retinal projection technology, which projects graphic images onto a user's retina. Projection systems can also project virtual objects into the physical environment (e.g., as holograms or onto physical surfaces).

[0014] For the purposes of this disclosure, an XR communication (XRC) session refers to an XR environment (XRE) in which two or more devices are participating.

[0015] For the purposes of this disclosure, a local XRC device refers to a current device that is described in an XRC session or is being controlled by the described user.

[0016] For the purposes of this disclosure, XRC devices located in the same location refer to two or more devices that share a physical environment and XRC session, such that users of devices located in the same location can experience the same physical objects and XR objects.

[0017] For the purposes of this disclosure, a remote XRC device refers to an auxiliary device located in a physical environment isolated from the current local XRC device. In one or more embodiments, the remote XRC device may be a participant in an XRC session.

[0018] For the purposes of this disclosure, a shared virtual element refers to an XR object that is visible to participants in an XR session or otherwise can be experienced by those participants in an XRE.

[0019] For the purposes of this disclosure, an XRC environment (XRCE) refers to a computing environment or container capable of hosting an XRC session of an application. XRCE enables applications to run within an XRC session. In some cases, XRCE allows users of an XRC session to interact with the hosted application within that session.

[0020] For the purposes of this disclosure, an XRCE instance refers to the XRCE of the current device described or controlled by the described user. An XRCE instance can allow a user to participate in an XRC session and run applications within that session.

[0021] For the purposes of this disclosure, a second XRCE instance refers to the XRCE of an auxiliary device in an XRC session other than the local XRCE instance, or one that is being controlled by a second user. The second XRCE instance can be remote or located in the same location.

[0022] For the purposes of this disclosure, an XRCE application refers to an application that can be executed in the context of XRCE.

[0023] For the purposes of this disclosure, a second XRCE application refers to the XRCE application of an auxiliary device in an XRC session, or an XRCE application that is being controlled by a second user rather than a local XRCE application. The second XRCE application can be remote or located at the same location.

[0024] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the accompanying drawings of this disclosure are block diagrams representing structures and devices to avoid obscuring the novel aspects of the disclosed concepts. For clarity, not all features of actual specific embodiments may be described. Additionally, as part of this specification, some of the drawings of this disclosure are provided in the form of flowcharts. The blocks in any particular flowchart may be presented in a specific order. However, it should be understood that the specific order of any given flowchart is only for illustrative purposes of one embodiment. In other embodiments, any of the various elements depicted in the flowcharts may be omitted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. Furthermore, other embodiments may include additional steps not shown as part of the flowcharts. Moreover, the language used in this disclosure has been primarily chosen for readability and instructional purposes and may not have been chosen to define or limit the subject matter of the invention, thereby resorting to the necessary claims to determine such inventive subject matter. In this disclosure, reference to “an implementation” or “implementation” means that a particular feature, structure or characteristic described in connection with that implementation is included in at least one implementation of the disclosed subject matter, and the repeated references to “an implementation” or “implementation” should not be construed as necessarily referring to all of the same implementation.

[0025] It should be understood that in any actual implementation of development (as in any software and / or hardware development project), numerous decisions must be made to achieve the developer's specific goals (e.g., compliance with system and business-related constraints), and these goals may differ between different implementations. It should also be understood that such development work can be complex and time-consuming, but nevertheless, it remains routine work for those of ordinary skill in the art who design and implement graphical modeling systems in benefit from this disclosure.

[0026] See Figure 1This diagram illustrates a simplified block diagram of an XR electronic device 100 communicatively connected via a network 105 to an additional XR electronic device 110 and a network storage device 115, according to one or more embodiments of the present disclosure. The XR electronic device differs from other electronic devices by displaying the XRE in a manner that allows the user of the XR electronic device to perceive the XRE as a three-dimensional (3D) interactive experience. In contrast, other electronic devices may be able to display a two-dimensional “window” that can be displayed as a 3D perspective projection onto the XRE. The XR electronic device 100 can be part of a multi-functional device such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable device, head-mounted system, projection-based system, base station, laptop computer, desktop computer, network device, or any other electronic system as described herein. The XR electronic device 100, the additional XR electronic device 110, and / or the network storage device 115 may additionally or alternatively include one or more additional devices (such as server devices, base stations, accessory devices, etc.) in which various functions can be accommodated, or various functions can be distributed across these additional devices. Examples of networks such as network 105 include, but are not limited to, local area networks (such as Universal Serial Bus (USB) networks), organizational LANs, and wide area networks (such as the Internet). According to one or more embodiments, XR electronic device 100 is used to participate in XR XRC sessions. It should be understood that various components and functions within XR electronic device 100, additional XR electronic device 110, and network storage device 115 may be distributed differently across devices or across additional devices. XR electronic device 100 may include a network interface 150 for communicating with other devices by connecting networking components such as radio, infrared, and / or visible light transceivers. Network interface 150 may connect to wired or wireless network components or both.

[0027] XR electronic device 100 may include one or more processors 125, such as a central processing unit (CPU). Processor 125 may include a system-on-a-chip (such as a system-on-a-chip present in a mobile device) and may include one or more dedicated graphics processing units (GPUs). Additionally, processor 125 may include multiple processors of the same or different types. XR electronic device 100 may also include memory 135. Memory 135 may include one or more different types of memory that can be used in conjunction with processor 125 to perform device functions. For example, memory 135 may include cache, ROM, RAM, or any kind of transient or non-transitory computer-readable storage medium capable of storing computer-readable code. Memory 135 may store various programming modules for execution by processor 125, including XR module 165, communication device identification module 170, and various other application programs 175. XR electronic device 100 may also include storage device 130. Storage device 130 may include one or more non-transitory computer-readable media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and magnetic tapes, optical media (such as CD-ROMs and digital video optical discs (DVDs)), and semiconductor storage devices (such as electrically programmable read-only memory (EPROMs) and electrically erasable programmable read-only memory (EEPROMs)). According to one or more embodiments, storage device 130 may be configured to store content item 160.

[0028] XR electronics 100 may also include one or more cameras 140 or other sensors 145, such as depth sensors that can determine the depth of a scene. In one or more embodiments, each of the one or more cameras 140 may be a conventional RGB camera or a depth camera. Furthermore, the cameras 140 may include stereo cameras or other multi-camera systems, time-of-flight camera systems, etc. Information generated by the sensors 145 and the one or more cameras 140 may be used by the communication device identification module 170 to detect other XR devices located in the same location and / or to connect with other XR devices located in the same location. The communication device identification module 170 may access the network interface 150 and / or certain types of sensors 145 to obtain information for facilitating connections with other XR devices located in the same location or to transmit such information to facilitate connections with other XR devices located in the same location.

[0029] XR electronic device 100 may also include a display 155. The display device 155 may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display may be configured to selectively become opaque. Projection-based systems may employ retinal projection technology, which projects graphic images onto the human retina. Projection systems may also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0030] According to one or more embodiments, memory 135 may include one or more modules comprising computer-readable code executable by processor 125 to perform functions. The memory may include, for example, an XR module 165 that can be used to provide XRE to a local XRC device in an XRC session. The XRC session may be a computing environment supporting a shared experience for XR electronics 100 and additional XR electronics 110.

[0031] The memory 135 may also include an OS module 180 for supporting the basic functions of the XR electronic device 100 and managing the hardware. The OS module 180 provides an environment in which applications 175 can be executed. Applications 175 may include, for example, computer applications that can be experienced in an XR session by multiple devices, such as electronic device 100 and attached XR electronic device 110.

[0032] Although the electronic device 100 is depicted as including the aforementioned components, in one or more embodiments, the various components may be distributed across multiple devices. Therefore, while certain calls and transmissions are described herein with respect to the specific system depicted, in one or more embodiments, various calls and transmissions may be directed differently based on the functions of different distributions. Additionally, additional components may be used, and certain combinations of the functions of any components may be combined.

[0033] Figure 2 A diagram illustrating an example operating environment according to one or more embodiments is shown. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the examples in the specific embodiments disclosed herein. Therefore, as a non-limiting example, the first operating environment 240 includes a first physical environment, while the second operating environment 250 includes a second physical environment.

[0034] like Figure 2As shown, the first operating environment 240 includes a first user 220 utilizing the first electronic device 200, and the second operating environment 250 includes a second user 232 utilizing the second electronic device 210. In one or more embodiments, the first electronic device 200 and the second electronic device 210 include mobile devices, such as handheld devices, wearable devices, etc.

[0035] In one or more embodiments, the first electronic device 200 and the second electronic device 210 communicate with each other via a network 205. Examples of the network 205 may include, for example, the Internet, a wide area network (WAN), a local area network (LAN), etc. In one or more embodiments, the first electronic device 200 and the second electronic device 210 may participate in an XRC session.

[0036] Although electronic device 200 and second electronic device 210 can participate in a common XRC session, the virtual environment can be presented differently on each device. As shown, electronic device 200 can depict the physical objects of the first operating environment 240. As shown, physical table 222 can be depicted as virtual table 224 on display 242. In one or more embodiments, display 242 can be a perspective display, and virtual table 224 can be simply a view of physical table 222 through display 242.

[0037] The display 242 of the electronic device 200 may also include an avatar 226 corresponding to a user 232 in the second operating environment 250. For the purposes of this disclosure, the avatar may include a virtual representation of the user. The avatar may depict real-time actions of the corresponding user 232, including movement, updated location, and / or interaction with various physical and / or virtual components within the XRC session.

[0038] According to one or more implementations, an XRCE can be an XRE that supports one or more XRCE applications or other modules that provide a description of XR objects across all participating devices, such as electronic device 200 and second electronic device 210. As shown in display 242, presentation panel 230A is an example of a virtual object that can be visible to all participating devices.

[0039] As an example, returning to environment 250, the second electronic device 210 includes a display 252 on which a presentation panel virtual object 230B is depicted. It should be understood that in one or more embodiments, although the same virtual object may be visible on all participating devices, the virtual object may be rendered differently depending on the location of the electronic device, the orientation of the electronic device, or other physical or virtual characteristics associated with electronic devices 200 and 210 and / or the XRCE depicted within displays 242 and 252.

[0040] Returning to environment 250, another characteristic of the XRC session is that while virtual objects can be shared across participating devices, the physical world can appear different. Therefore, the physical chair 234 is depicted as a virtual chair 236. As described above, and as in one or more embodiments, the display 252 can be a perspective display, and the virtual chair 236 can be a view of the physical chair 234 through the perspective display 252. Furthermore, the second electronic device 210 depicts an avatar 238 corresponding to the user 220 within the physical environment 240.

[0041] Generally, communication devices can be located in the same location or remotely. When two XRC devices share a physical environment, they can be located in the same location. Otherwise, the XRC devices are remote. It may be desirable to establish an XRC session between XRC devices located in the same location. In some cases, users may want to announce or create XRC sessions so that other users located in the same location can join other XRC devices. For example, users at a conference may want to use an XRC session to give presentations to conference participants located in the same location. Users may want to announce the availability of the XRC session to the XRC devices of conference participants located in the same location. Figure 3 A diagram illustrating XRC session discovery 300 according to various aspects of this disclosure is shown. In this example, a second XRC device 302 associated with a second user 304 may be configured to transmit an XRC session announcement 306. The XRC session announcement 306 may be transmitted using a relatively short-range signal. In some cases, the short-range signal may be a relatively low-power signal that typically does not penetrate solid surfaces such as walls. For example, the XRC session announcement 306 may be transmitted as a Quick Response (QR) code displayed on a screen. In some cases, it may be desirable to transmit the XRC session announcement 306 in a non-visual manner. As an example, the XRC session announcement 306 may be encoded and broadcast via an infrared (IR) signal, such as via an IR light-emitting diode (LED) or an IR laser, such as an IR dot projector or a time-of-flight camera. In other cases, the XRC session may be announced using conventional low-power protocols, such as Bluetooth. ® Low power (BLE) (Bluetooth is a registered trademark owned by Bluetooth SIG, Inc.), or the use of radio frequencies, such as millimeter wave frequencies, to communicate, frequencies that are easily absorbed by objects in their propagation path.

[0042] The transmitted XRC session notification 306 can be received by the first XRC device 308 associated with the first user 310. For example, the first XRC device 308 can receive the XRC session notification 306 via a camera, an IR receiver, or a suitable radio frequency receiver. After receiving the XRC session notification 306, the first XRC device 308 can display an indication that an XRC session is available. In some cases, this indication may include identification information 312 associated with the first user. In some cases, the first XRC device 308 can display an image of the scene, including the second XRC device 302 and / or the second user 304. For example, the first user 310 can point the camera of the first XRC device 308 at the second XRC device 302 and / or the second user 304, and the first XRC device 308 can display an image of the corresponding scene. Based on the identification information 312, the first XRC device 308 can determine that the second XRC device 302 is a candidate communication device for establishing an XRC session with it, and update the graphical user interface of the first XRC device 308 to indicate that the candidate communication device is available for establishing an XRC session with it. For example, the scene 314 displayed on the first XRC device 308 can be updated to indicate the second XRC device 302 in the displayed scene 314 and indicate that an XRC session can be established with the second XRC device 302. The first XRC device 308 can then send a request to the second XRC device 302 to join the XRC session. For example, a user interface element allowing a first user 310 of the first XRC device 302 to establish an XRC session with the second XRC device 302 based on the XRC session announcement 306 can be presented to them. Based on an instruction received, for example, via this user interface element, the first XRC device 302 can transmit a request to the second XRC device 302 to join the XRC session.

[0043] As described above, the XRC session announcement 306 may also include identification information 312. In some cases, identification information 312 may include information identifying the second user 304. For example, identification information 312 may include a hash based on the user identity of the second user 304. Transmitting the user hash helps allow for a level of user privacy while still allowing other devices to determine whether the user is a known user. In some cases, the first XRC device 308 may receive the user identity hash of the second user 304 and compare it with the contact hashes of known users. For example, contact hashes may be generated and stored for each of the user's contacts. The received user identity hash may be compared with these stored contact hashes to determine whether the second user 304 is known to the first user 310. If the first user 310 knows the second user 304, an indication of the second user 304 may be displayed along with an indication that the second user 304 is attempting to establish an XRC session. If the first XRC device 308 receives an instruction to join an XRC session from the first user 310, the first XRC device 308 sends a request 316 to join the XRC session to establish an XRC session between the first XRC device 308 and the second XRC device 302.

[0044] In some cases, XRC session announcements can be received from unknown users. How the first XRC device 308 responds to XRC session announcements 306 from unknown users can be configurable. For example, the first XRC device 308 can be configured to ignore all XRC session announcements. In this case, the first XRC device 308 can discard the received XRC session announcement 306, or the first XRC device can completely block receiving XRC session announcements. As another example, the first XRC device 308 can be configured to disallow XRC session announcements from unknown users. In this case, the first XRC device 308 can receive the user identity hash of the second user 304 and determine that the second user 304 is unknown to the first XRC device 308. The first XRC device 308 can then discard the received XRC session announcement 306. In other cases, the first XRC device 308 can be configured to allow XRC session announcements from unknown users. In this case, the first XRC device 308 can display an instruction to the first user 310 regarding the received XRC session announcement 306. If the first user 310 receives another instruction to accept joining the XRC session, the first XRC device 308 transmits a join request 316 to the second XRC device 302 to establish an XRC session.

[0045] Figure 4This is a block diagram illustrating an exemplary communication environment 400 according to various aspects of the present disclosure. According to various aspects of the present disclosure, instead of a user identity hash or other than a user identity hash, the identification information 312 transmitted by the second XRC device 302 in an XRC session announcement 306 may include location information 418. The announced location information 418 may be information related to the location of the second XRC device 302 in a reference coordinate system. For example, the second XRC device 302 may include one or more location sensors, such as a Global Positioning System (GPS) sensor, a compass, etc., which may be used to provide the location information 418 in a reference coordinate system (such as GPS coordinates and orientation coordinates). In some cases, the location information 418 may be based on information determined from sensors or components not specifically designed to provide location information. For example, the location information 418 may be based on cellular signal triangulation, nearby Wi-Fi networks, beacons such as Bluetooth. ® Signal strength of beacons and access points, relative positions of other nearby radio frequency sources, comparisons of images captured by cameras with known locations, and computer vision-based localization techniques (e.g., Simultaneous Localization and Mapping (SLAM), Visual Inertial Odometry (VIO), etc.) can all be used to generate this information. As a more specific example, it can be based on the relative position information of identified shapes and / or planes. Image data (such as point clouds or traditional visible light digital images) can be processed to determine ground planes, ground plane size information, and / or shape information to describe objects in the image data via shape information. In some cases, objects can be identified and assigned corresponding shapes. In other cases, one or more shapes can be used to describe objects, such as by one or more shapes most similar to the corresponding object. Relative position information can then be generated based on how shapes and / or ground planes / ground plane size information in the shape set are oriented relative to each other.

[0046] In other cases, location information 418 can be extracted based on characteristics of XRC session announcement 306. For example, the first XRC device 308 can determine the distance to the second XRC device 302 based on the signal strength of XRC session announcement 306. This signal strength can be compared with a predetermined signal strength or with the transmitted signal strength included in XRC session announcement 306. As another example, the first XRC device 308 can measure the Doppler effect, directivity, signal strength across multiple antennas, or other characteristics of the received XRC session announcement 306 to determine location information about the second XRC device 302. In some cases, certain techniques for determining location information may be preferred, and other techniques may be used when the preferred techniques are unavailable. For example, GPS coordinates may be preferred, but when GPS signals are unavailable, nearby Wi-Fi networks can be used to determine location information. If nearby Wi-Fi networks are unavailable (e.g., unidentified, Wi-Fi disabled, etc.), shape recognition can be used.

[0047] The location information 418 can be used to help indicate which XRC session can be established with the second XRC device 302. For example, multiple XRC devices may advertise XRC sessions in an area, and a user may want to ensure they are attempting to connect to the intended XRC session. The location information 418 can be used to determine which of the multiple XRC devices a particular XRC session advertisement is associated with. In some cases, the first XRC device 308 can determine the relative position of the second XRC device 302 based on the received location information 418 along with the position of the first XRC device 308. When displaying an image of the scene, the first XRC device 308 can attempt to identify candidate communication devices in the image of the scene based on the location information 418. For example, the first XRC device 308 can use object recognition technology to identify XRC devices and / or users within the image of the scene. The first XRC device 308 can also use, for example, depth information, the position of the first XRC device 308, and an indication of the direction the first XRC device 308 is facing to determine the relative position of the identified XRC device and / or user. The depth information can be provided, for example, by a depth sensor, a time-of-flight sensor, or depth sensing based on multiple images. The position and orientation indication of the first XRC device 308 can be obtained, for example, using techniques similar to those used to determine the position information 418. The relative positions of the identified XRC device and / or user can then be compared with the determined relative position of the second XRC device 302 to identify candidate communication devices and update the displayed scene 314 to indicate the second XRC device 302 within the displayed scene 314.

[0048] In some cases, the first XRC device 308 can be configured to ignore received XRC session announcements where the distance to the second XRC device 302 is greater than a threshold distance. In this case, the first XRC device 308 can compare the announced location information 418 with its own location to determine the distance to the second XRC device 302. If the determined distance is greater than the threshold distance, the received XRC session announcement 306 can be discarded.

[0049] In some cases, location information 418 may not be accurate enough to determine which XRC device an XRC session announcement is associated with. For example, GPS accuracy can be reduced in conditions with limited or no sky view. Figure 5 As shown, the identification information 312 transmitted by the second XRC device 302 may include orientation information 518. Orientation information 518 may be included along with or in lieu of any other information, such as location information 418. Orientation information 518 may describe the orientation of the user and / or the XRC device associated with the user. For example, the second XRC device 302 may include one or more sensors that can be used to provide orientation information describing how the second XRC device 302 is oriented. For example, gyroscopes, gravity and / or altimeter sensors may be used to provide tilt, angle, and height information. In some cases, other sensors, such as optical cameras (e.g., using computer vision-based techniques such as SLAM, VIO, etc.), depth sensors, tilt sensors, etc., may be used to determine the orientation of the XRC device in space. In another example, orientation information may describe the user's orientation; for example, the second XRC device 302 may be associated with one or more body position sensors and / or imaging devices that can provide orientation information describing how the user is oriented in space and / or how the user is oriented relative to the second XRC device 302. For example, orientation information can indicate that a second user is facing a specific direction, how parts of their body are positioned relative to gravity, and / or how the user is oriented relative to other people or objects around them. This orientation information 518 can be encoded and included together with identification information 312 and transmitted in XRC session announcement 306.

[0050] This orientation information can also be used to help indicate which XRC session can be established with the second XRC device 302. In some cases, the orientation information 518 can be used to determine which of a plurality of XRC devices a particular XRC session notification is associated with. For example, the orientation information 518 may indicate that the second XRC device 302 is facing a specific direction and tilted upwards by a first number of degrees and tilted to the left by a second number of degrees, and / or a third number of inches above the ground. In some cases, the first XRC device 308 may use object recognition technology to identify XRC devices and / or users within an image of the scene.

[0051] The first XRC device 308 can also determine the orientation information of the identified XRC device and / or user, for example, by using object recognition technology to scale and / or rotate the identified XRC device and human pose estimation to determine the user's orientation information. As part of the object recognition process, the first XRC device 308 can, for example, determine the rotation information of the identified XRC device. Based on this rotation information and the orientation of the first XRC device 308, the orientation of the identified XRC device can be determined. In some cases, height information can be determined based on, for example, the height and tilt information of the identified XRC device in an image of the scene relative to the second XRC device 308. The determined orientation information can then be compared with received orientation information 518 to identify candidate communication devices. The first XRC device 308 can then compare the determined orientation information of the identified XRC device with similarly oriented XRC devices to identify candidate communication devices. The displayed scene 314 can then be updated to indicate the second XRC device 302 and / or the second user 304 in the displayed scene 314 as candidate devices.

[0052] In some cases, orientation information 518 from multiple XRC session announcements 306 can be matched over time, for example, to increase the confidence in correctly identifying candidate communication devices. For instance, if multiple identified XRC devices have orientations within a threshold distance of a determined orientation, orientation information 518 from multiple XRC session announcements 306 over a period of time can be sampled and compared with multiple determined orientations within the same time period. Obtaining orientation information 518 from multiple XRC session announcements 306 over time can be helpful because multiple users associated with an XRC device are less likely to similarly orient their respective devices over time.

[0053] Orientation information 518 can also be used in conjunction with other information that may be included in the identification information to identify candidate communication devices. For example, location information can be used to narrow down a group of XRC devices, and then the orientation of the XRC devices in that group can be determined and matched with the received orientation information 518.

[0054] Figure 6 This is a block diagram illustrating an exemplary communication environment using gesture 600 according to various aspects of this disclosure. In some cases, an XRC device may request one or more users to perform one or more gestures. These gestures may be used to help indicate which device and / or user an XRC session can be established with. In this example, a second XRC device 302 may instruct a second user 304 to perform a specific gesture, such as a thumbs-up gesture 620 or other gestures such as raising a specific arm, pointing, nodding, etc. In some cases, the second user 304 may select the specific gesture to perform, for example, from a menu of possible gestures displayed on the user interface of the second XRC device 302. The second XRC device 302 may include gesture information 618 as part of identification information 312 in an XRC session announcement 306. Gesture information 618 may be included instead of user identity hashes, orientation information, location information, etc., or otherwise. Gesture information 618 may describe the gesture requested by or by the second user 304. For example, gesture information 618 may be based on features or gesture vectors or descriptors that describe the expected gesture in an orientation-invariant manner.

[0055] Upon receiving XRC session notification 306 and gesture information 618, the first XRC device 308 can use object recognition technology to identify a user making the gesture described by gesture information 618 within an image of the scene. For example, the first XRC device 308 can apply feature or gesture recognition technology to search for the gesture described by gesture information. If a matching gesture is found in the image of the scene, the displayed scene 314 can be updated to identify the second XRC device 302 and / or the second user 304 in the displayed scene 314 as candidate devices.

[0056] In some situations, multiple gestures can be used, for example, to increase the confidence in correctly identifying candidate communication devices. For instance, multiple people can raise their hands simultaneously. Matching multiple gestures reduces the likelihood that multiple people will make the same gesture.

[0057] In some cases, gesture information 618 can also be used in conjunction with other information that may be included in the identification information to identify candidate communication devices. For example, location or orientation information can be used to narrow down a group of XRC devices and / or associated users, and then the gestures of the associated users can be matched with the received gesture information 618.

[0058] Figure 7This is a block diagram illustrating an exemplary communication environment using device sound 700 according to various aspects of this disclosure. In some cases, the XRC device may emit one or more sounds to help indicate which device an XRC session can be established with. In this example, the second XRC device 302 may include audio signal information 718 as part of identification information 312 in the XRC session announcement 306. Audio signal information 718 may be included instead of user identity hash, orientation information, location information, etc., or other information. Audio signal information 718 may describe an audio signal 720 that can be transmitted by the second XRC device 302. The second XRC device 302 may transmit the audio signal 720 as described by audio signal information 718 after or simultaneously with the transmission of XRC session announcement 306. For example, audio signal information 718 may describe the number of beeps at a certain frequency, followed by a pause of a certain duration, followed by a longer beep, etc. The second XRC device 302 may transmit audio signal 720 while transmitting XRC session announcement 306. In some cases, audio signals 720 can be transmitted at frequencies inaudible to the human ear, such as ultrasonic frequencies.

[0059] Upon receiving XRC session notification 306 and audio signal information 718, the first XRC device 308 may attempt to detect the transmitted audio signal 720. For example, the first XRC device 308 may include one or more microphones capable of receiving audio signals. These audio signals may be analyzed to determine whether the received audio signal corresponding to the audio signal information 718 has been detected. In some cases, the first XRC device 308 may include multiple microphones and use stereo localization technology to determine the direction of the received audio signal. In some cases, the first XRC device 308 may rotate in various directions to help determine the direction of the received audio signal. For example, the first XRC device 308 may request the first user 310 to rotate the first XRC device 308 left and right multiple times to help better locate the received audio or provide a wider stereo baseline. The displayed scene 314 can then be updated to identify the second XRC device 302 and / or the second user 304 in the displayed scene 314 as candidate devices based on the determined direction of the received audio signal.

[0060] In some cases, the audio signal information 718 can also be used in conjunction with other information that may be included in the identification information to identify candidate communication devices. For example, positioning or orientation information can be used to narrow the direction in which the audio signal 720 is being attempted to be detected. Furthermore, detected gestures performed by other users can increase the confidence in correctly identifying candidate communication devices.

[0061] Figure 8This is a block diagram illustrating an exemplary communication environment using user voice 800 according to various aspects of this disclosure. In some cases, the XRC device may utilize user-generated voice to help indicate which device an XRC session can be established with. In this example, the second XRC device 302 may include user audio information 818 as part of identification information 312 in the XRC session announcement 306. User audio information 818 may be included instead of user identity hash, orientation information, location information, etc., or other information. User audio information 818 may describe user audio 820 transmitted by the second user 304. In some cases, user audio information 818 may be a voice fingerprint of the second user 304 or encoded speech of the second user 304.

[0062] Upon receiving XRC session notification 306 and user audio information 818, the first XRC device 308 may attempt to detect user audio 820. For example, the first XRC device 308 may include one or more microphones capable of receiving audio signals. These audio signals may be analyzed to determine whether the received audio signal corresponds to user audio information 818. For example, the first XRC device 308 may include detecting the presence of speech in the received audio signal and comparing the detected speech with a second user's voice fingerprint or coded speech. In some cases, the first XRC device 308 may include multiple microphones and use stereo localization technology to determine the direction of the received audio signal. In some cases, the first XRC device 308 may rotate in various directions to help determine the direction of the received audio signal. For example, the first XRC device 308 may request the first user 310 to rotate the first XRC device 308 left and right multiple times to help better locate the received audio or provide a wider stereo baseline. The displayed scene 314 can then be updated to identify the second XRC device 302 and / or the second user 304 in the displayed scene 314 as candidate devices based on the determined direction of the received audio signal.

[0063] In some cases, user audio information 818 can also be used in conjunction with other information that may be included in the identification information to identify candidate communication devices. For example, location or orientation information can be used to narrow the direction in which user audio 820 is being attempted to be detected. Furthermore, detected gestures performed by other users or audio output by another device can increase the confidence in correctly identifying candidate communication devices.

[0064] Figure 9A technique 900 for identifying an XR service according to various aspects of this disclosure is illustrated. At block 902, a first communication device receives a message indicating the availability of a second communication device for a communication session, the message including identification information indicating the current posture of the second communication device or its user. For example, the second communication device may transmit an XRC session notification. This XRC session notification may include identification information for the second communication device and / or a second user associated with the second communication device. In some cases, the identification information may include location information, orientation information, gesture information, audio signal information, and / or user audio information. The XRC session notification may be received by the first communication device. At block 904, the first communication device receives image data corresponding to an image of a scene including a candidate communication device. For example, the first communication device may include a camera that can be pointed at the second communication device, and the first communication device may display the corresponding image on its display. At block 906, the first communication device determines that the second communication device is a candidate communication device based on the identification information and the image data. In some cases, the image data may include other image-based inputs, such as point or point data (e.g., point clouds) from, for example, an IR point projector, a laser projector, and / or a time-of-flight camera. For example, the first communication device may use identification information along with location information, orientation information, gesture information, audio signal information, and / or user audio information (if available) to identify the second communication device in the image. At block 908, the first communication device updates the graphical user interface depicting the candidate communication device to indicate that the candidate communication device is available for the communication session indicated in the message. For example, the first communication device may update the displayed image with an indication that an XRC session can be established with the second communication device. At block 910, the first communication device sends a request to join the communication session in response to receiving a selection of a candidate communication device. For example, the first communication device may send a response to an XRC session announcement requesting to join or begin an XRC session.

[0065] Figure 10A and Figure 10B An exemplary system 1000 for use in various XR technologies is shown.

[0066] In some examples, such as Figure 10A As shown, system 1000 includes device 1000A. Device 1000A includes various components such as processor 1002, RF circuitry 1004, memory 1006, image sensor 1008, orientation sensor 1010, microphone 1012, position sensor 1016, speaker 1018, display 1020, and touch-sensitive surface 1022. These components optionally communicate via communication bus 1050 of device 1000A.

[0067] In some examples, components of system 1000 are implemented in a base station device (e.g., a computing device, such as a remote server, mobile device, or laptop computer), and other components of system 1000 are implemented in a second device (e.g., a head-mounted device). In some examples, device 1000A is implemented in either the base station device or the second device.

[0068] like Figure 10B As shown, in some examples, system 1000 includes two (or more) devices that communicate via wired or wireless connections. A first device 1000B (e.g., a base station device) includes a processor 1002, RF circuitry 1004, and memory 1006. These components optionally communicate via a communication bus 1050 of device 1000B. A second device 1000c (e.g., a head-mounted device) includes various components such as processor 1002, RF circuitry 1004, memory 1006, image sensor 1008, orientation sensor 1010, microphone 1012, position sensor 1016, speaker 1018, display 1020, and touch-sensitive surface 1022. These components optionally communicate via a communication bus 1050 of device 1000c.

[0069] System 1000 includes processor 1002 and memory 1006. Processor 1002 includes one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 1006 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 1002 to perform the techniques described below.

[0070] System 1000 includes RF circuitry 1004. RF circuitry 1004 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, intranets), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 1004 optionally includes circuitry for using near-field communication and / or short-range communication (such as Bluetooth). ® The circuit that enables communication.

[0071] System 1000 includes a display 1020. Display 1020 may have an opaque display. Display 1020 may also have a transparent or translucent display, which may be combined with a substrate through which light representing an image is directed to an individual's eye. Display 1020 may incorporate LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The light-transmitting substrate may be an optical waveguide, an optical combiner, a light reflector, a holographic substrate, or any combination of these substrates. In one example, the transparent or translucent display may selectively switch between an opaque state and a transparent or translucent state. Other examples of display 1020 include head-up displays, automotive windshields capable of displaying graphics, windows capable of displaying graphics, lenses capable of displaying graphics, tablet computers, smartphones, and desktop or laptop computers. Alternatively, system 1000 may be designed to receive an external display (e.g., a smartphone). In some examples, System 1000 is a projection-based system that uses retinal projection to project images onto an individual's retina or to project virtual objects into a physical setting (e.g., onto a physical surface or as a hologram).

[0072] In some examples, system 1000 includes a touch-sensitive surface 1022 for receiving user input, such as tap input and swipe input. In some examples, display 1020 and touch-sensitive surface 1022 form a touch-sensitive display.

[0073] System 1000 includes an image sensor 1008. Image sensor 1008 optionally includes one or more visible light image sensors, such as charge-coupled device (CCD) sensors and / or complementary metal-oxide-semiconductor (CMOS) sensors, operable to obtain images of physical elements from a physical scene. One or more image sensors also optionally include one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light from the physical scene. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the physical scene. Image sensor 1008 also optionally includes one or more event cameras configured to capture movement of physical elements in the physical scene. Image sensor 1008 also optionally includes one or more depth sensors configured to detect the distance between physical elements and system 1000. In some examples, system 1000 uses a combination of CCD sensors, event cameras, and depth sensors to detect the physical scene around system 1000. In some examples, image sensor 1008 includes a first image sensor and a second image sensor. The first and second image sensors are optionally configured to capture images of physical elements in a physical setting from two different perspectives. In some examples, system 1000 uses image sensor 1008 to receive user input, such as gestures. In some examples, system 1000 uses image sensor 1008 to detect the position and orientation of system 1000 and / or display 1020 in the physical setting. For example, system 1000 uses image sensor 1008 to track the position and orientation of display 1020 relative to one or more fixed elements in the physical setting.

[0074] In some examples, system 1000 includes microphone 1012. System 1000 uses one or more microphones 1012 to detect sound from a user and / or the user's physical set. In some examples, the one or more microphones 1012 include a microphone array (including multiple microphones) that optionally cooperate to identify ambient noise or locate sound sources in the space of the physical set.

[0075] System 1000 includes an orientation sensor 1010 for detecting the orientation and / or movement of system 1000 and / or display 1020. For example, system 1000 uses orientation sensor 1010 to track changes in the position and / or orientation of system 1000 and / or display 1020, such as relative to physical elements in a physical setting. Orientation sensor 1010 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0076] The techniques defined herein take into account options for obtaining and utilizing users' personal information. For example, one aspect of the techniques of this invention is to automatically determine whether a particular device can display an XR view of an XRC session. However, with regard to the collection of such personal information, it should be obtained with the user's informed consent so that the user understands and controls the use of their personal information.

[0077] Parties authorized to access personal information will use the information only for lawful and reasonable purposes and will comply with privacy policies and practices that at least meet appropriate laws and regulations. Furthermore, such policies should be comprehensive, user-accessible, and considered to meet or exceed government / industry standards. In addition, personal information may not be distributed, sold, or otherwise shared except for any reasonable and lawful purpose.

[0078] However, users can limit the extent to which parties can access personal information. The processes and devices described herein can allow for changes to settings or other preferences that enable users to control access to their personal information. Furthermore, while some of the characteristics defined herein are described in the context of the use of personal information, aspects of these characteristics can be implemented without the need for such information. As an example, a user's personal information can be masked or otherwise generalized so that it does not identify the specific user from whom it is obtained.

[0079] It should be understood that the above description is intended to be exemplary and not restrictive. Material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims and to provide that material in the context of a particular embodiment, variations of which will be readily apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in combination with each other). Therefore, Figure 9 The specific arrangement of the steps or actions shown or Figures 1 to 8 and Figures 10A to 10B The arrangement of the elements shown should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents. In the appended claims, the terms “comprising” and “therein” are used as common English equivalents of the corresponding terms “including” and “wherein”.

Claims

1. A method used in a communication environment, the method comprising: A message indicating the availability of a second communication device for a communication session is received at a first communication device, the message including identification information indicating the current posture of the second communication device or the user of the second communication device; Image data corresponding to an image of a scene including a candidate communication device is received at the first communication device, wherein the image is captured by the camera of the first communication device; At the first communication device, the second communication device is determined to be the candidate communication device based on the identification information and the image data; as well as In response to determining that the second communication device is the candidate communication device: A graphical user interface for the candidate communication device is presented in a view identifying the scene at the first communication device, wherein the graphical user interface indicates the availability of the second communication device for the communication session; as well as In response to receiving a selection of the candidate communication device, a request to join the communication session is sent from the first communication device.

2. The method of claim 1, wherein the communication session corresponds to a shared extended reality experience.

3. The method according to any one of claims 1 to 2, wherein the current pose includes the position of the second communication device in a reference coordinate system, and wherein determining that the second communication device is the candidate communication device based on the identification information and the image data includes: Determine the local position of the first communication device in the reference coordinate system; The relative position of the candidate communication device with respect to the first communication device is determined based on the image data; The candidate position of the candidate communication device in the reference coordinate system is determined based on the local position and the relative position. as well as The candidate locations are matched with the locations of the announcements.

4. The method of claim 3, wherein the reference coordinate system corresponds to a global coordinate system, and wherein the local location is determined based on a Global Positioning System device included in the first communication device.

5. The method of claim 1, wherein the current pose includes the orientation of the second communication device in a reference coordinate system, and wherein determining that the second communication device is the candidate communication device based on the identification information and the image data includes: Determine the local orientation of the first communication device in the reference coordinate system; The relative orientation of the candidate communication device with respect to the first communication device is determined based on the image data; The candidate orientation of the candidate communication device in the reference coordinate system is determined based on the local orientation and the relative orientation. as well as The candidate orientations are matched with the orientations of the announcements.

6. The method of claim 1, wherein the current gesture includes a gesture of a user of the second communication device, and wherein determining that the second communication device is the candidate communication device based on the identification information and the image data includes: Based on the image data, determine the candidate gestures of the user of the candidate communication device; as well as The candidate gestures are matched with the gestures in the announcement.

7. The method of claim 1, wherein the current posture includes the height of the user of the second communication device, and wherein determining that the second communication device is the candidate communication device based on the identification information and the image data includes: Determine the local altitude of the user of the first communication device; The height difference between the user of the first communication device and the user of the candidate communication device is determined based on the image data. The candidate height of the user of the candidate communication device is determined based on the local height and the difference; and The candidate height is matched with the height of the announcement.

8. The method of claim 1, wherein the identification information includes an audio signature of the announcement, and wherein determining that the second communication device is the candidate communication device based on the identification information and the image data comprises: The sound signature mentioned in the detection announcement; Determine the source direction of the detected audio signature of the announcement; The relative position of the candidate communication device with respect to the first communication device is determined based on the image data; as well as Match the source direction with the relative position.

9. The method of claim 8, wherein the announced voice signature corresponds to the voice fingerprint of the user of the second communication device.

10. A computer-readable storage device storing instructions executable by one or more processors to perform the following operations: A message indicating the availability of a second communication device for a communication session is received at a first communication device, the message including identification information indicating the current posture of the second communication device or the user of the second communication device; Image data corresponding to an image of a scene including a candidate communication device is received at the first communication device, wherein the image is captured by the camera of the first communication device; At the first communication device, the second communication device is determined to be the candidate communication device based on the identification information and the image data; as well as In response to determining that the second communication device is the candidate communication device: A graphical user interface for the candidate communication device is presented in a view identifying the scene at the first communication device, wherein the graphical user interface indicates the availability of the second communication device for the communication session; as well as In response to receiving a selection of the candidate communication device, a request to join the communication session is sent from the first communication device.

11. The computer-readable storage device of claim 10, wherein the communication session corresponds to a shared augmented reality experience.

12. The computer-readable storage device according to any one of claims 10 to 11, wherein the current attitude information includes the position of the second communication device in a reference coordinate system, and wherein determining that the second communication device is the candidate communication device based on the identification information and the image data includes: Determine the local position of the first communication device in the reference coordinate system; The relative position of the candidate communication device with respect to the first communication device is determined based on the image data; The candidate position of the candidate communication device in the reference coordinate system is determined based on the local position and the relative position. as well as The candidate locations are matched with the locations of the announcements.

13. The computer-readable storage device of claim 12, wherein the reference coordinate system corresponds to a global coordinate system, and wherein the local location is determined based on a Global Positioning System device included in the first communication device.

14. The computer-readable storage device of claim 10, wherein the current pose includes the orientation of the second communication device in a reference coordinate system, and wherein determining that the second communication device is the candidate communication device based on the identification information and the image data includes: Determine the local orientation of the first communication device in the reference coordinate system; The relative orientation of the candidate communication device with respect to the first communication device is determined based on the image data; The candidate orientation of the candidate communication device in the reference coordinate system is determined based on the local orientation and the relative orientation. as well as The candidate orientations are matched with the orientations of the announcements.

15. The computer-readable storage device according to any one of claims 10 to 11, wherein the current gesture includes a gesture of a user of the second communication device, and wherein determining that the second communication device is the candidate communication device based on the identification information and the image data comprises: Based on the image data, determine the candidate gestures of the user of the candidate communication device; as well as The candidate gestures are matched with the gestures in the announcement.

16. The computer-readable storage device of any one of claims 10 to 11, wherein the current posture includes the height of the user of the second communication device, and wherein determining that the second communication device is the candidate communication device based on the identification information and the image data comprises: Determine the local altitude of the user of the first communication device; The height difference between the user of the first communication device and the user of the candidate communication device is determined based on the image data. The candidate height of the user of the candidate communication device is determined based on the local height and the difference; and The candidate height is matched with the height of the announcement.

17. The computer-readable storage device according to any one of claims 10 to 11, wherein the identification information includes an audio signature of the announcement, and wherein determining that the second communication device is the candidate communication device based on the identification information and the image data comprises: The sound signature mentioned in the detection announcement; Determine the source direction of the detected audio signature of the announcement; The relative position of the candidate communication device with respect to the first communication device is determined based on the image data; as well as Match the source direction with the relative position.

18. The computer-readable storage device of claim 17, wherein the announced voice signature corresponds to the voice fingerprint of a user of the second communication device.

19. An electronic device, the electronic device comprising: One or more processors; as well as A memory that stores instructions that can be executed by one or more processors to perform the method according to any one of claims 1 to 9.

20. A computer program product comprising a computer program, the computer program including computer code, the computer code being executable by one or more processors to perform the method according to any one of claims 1 to 9.

Images