Session privacy for third-party applications

Abstract

The present disclosure relates to techniques for collaboration in multi-user communication sessions, including receiving, at a first device, a request to launch a first application in a multi-user communication session with a second device; presenting an environment of the first application in the multi-user communication session; receiving, by one or more processes of the multi-user communication session, data from the first user, the data including input data and first communication data; providing, by the one or more processes, the input data to the first application; providing, by the one or more processes, the first communication data to the second device; receiving, by the one or more processes, second communication data from the second device; and presenting, by the one or more processes, the second communication data in the environment of the first application, wherein the first application is restricted from accessing the first communication data and the second communication data.

Description

Related application citation

[0001] This application is a divisional application of the invention patent application with international application number PCT / US2021 / 050468, international application date of September 15, 2021, entry into the Chinese national phase date of May 23, 2023, Chinese national application number 202180078741.X, and invention title "Session Privacy of Third-Party Applications". Background Technology

[0002] This disclosure relates in its entirety to multi-user environments. More specifically, but not as a limitation, this disclosure relates to techniques and systems for session privacy of third-party applications executing in the context of multi-user communication environments such as extended reality (XR) environments (XRE).

[0003] Some devices are capable of generating and presenting XREs. XREs can include fully or partially simulated environments that people perceive and / or interact with via electronic systems. In an XRE, a subset of a person's physical motion or a representation thereof is tracked, and in response, one or more properties of one or more virtual objects simulated in the XRE are adjusted in a manner consistent with at least one physical law. Some XREs allow multiple users to interact with each other within the XRE. However, what is needed is technology that hosts applications within the XRE while simultaneously managing users' private information. Attached Figure Description

[0004] Figure 1 A simplified system diagram according to one or more implementation schemes is shown in block diagram form.

[0005] Figure 2 A diagram illustrating an exemplary operating environment according to one or more implementation schemes is shown.

[0006] Figure 3A and Figure 3B This is a line drawing illustrating an interface for bringing the application experience to a multi-user communication extended reality environment, according to various aspects of this disclosure.

[0007] Figure 4 This is a diagram illustrating the application experience according to various aspects of this disclosure.

[0008] Figure 5 This is a flowchart illustrating techniques for collaboration in a multi-user communication session according to various aspects of this disclosure.

[0009] Figures 6A to 6B An exemplary system for use in various extended reality technologies is shown. Detailed Implementation

[0010] This disclosure relates to techniques for enabling users to collaborate within an extended reality (XRE) environment to share an application experience. Some applications can be experienced by multiple users within an XRE. In some cases, it is desirable for certain applications to present an application environment in place of the XRE. For example, a game application or a multi-user drawing application may present its own environment in place of the XRE while retaining certain elements of the XRE, such as avatars and voice communication between participants. In some cases, privacy concerns may arise regarding allowing applications potentially access to voice communication, visual representations of participants, or identifiable information about participants in the application experience. Therefore, it is desirable to allow applications to present an application environment in place of the XRE while preventing applications from accessing potentially sensitive information.

[0011] Humans can interact with and / or perceive the physical environment or physical world without the aid of electronic devices. The physical environment can include physical features, such as physical objects or surfaces. An example of a physical environment is a physical forest that includes physical plants and animals. Humans can directly perceive and / or interact with the physical environment through various means, such as hearing, vision, taste, touch, and smell. In contrast, humans can use electronic devices to interact with and / or perceive a fully or partially simulated extended reality (XR) environment. This XR environment can include mixed reality (MR) content, augmented reality (AR) content, virtual reality (VR) content, and so on. Using an XR system, a person's physical movements, or some of their representations, can be tracked, and in response, the characteristics of virtual objects simulated in the XR environment can be adjusted in a manner consistent with at least one physical law. For example, the XR system can detect movement of the user's head and adjust the graphical and auditory content presented to the user (similar to how such views and sounds change in a physical environment). For example, the XR system can detect movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) presenting the XR environment and adjust the graphical and auditory content presented to the user (similar to how such views and sounds change in a physical environment). In some cases, the XR system can adjust the characteristics of the graphical content in response to other inputs such as representations of physical motion (e.g., voice commands).

[0012] Many different types of electronic systems enable users to interact with and / or perceive XR environments. A non-exclusive list of examples includes head-up displays (HUDs), head-mounted systems, projection-based systems, windows or vehicle windshields with integrated display capabilities, displays formed as lenses placed over the user's eyes (e.g., contact lenses), head-mounted receivers / headsets, input systems with or without haptic feedback (e.g., wearable or handheld controllers), speaker arrays, smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have opaque displays and one or more speakers. Other head-mounted systems may be configured to accept opaque external displays (e.g., smartphones). Head-mounted systems may include one or more image sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays, rather than opaque displays. Transparent or semi-transparent displays may have a medium through which light is directed to the user's eyes. Displays can utilize various display technologies, such as uLED, OLED, LED, liquid crystal on silicon, laser scanning light sources, digital light projection, or combinations thereof. Optical waveguides, optical reflectors, holographic media, optical combiners, or combinations thereof, or other similar technologies can be used as the medium. In some implementations, transparent or translucent displays can be selectively controlled to become opaque. Projection-based systems can utilize retinal projection technology, which projects graphic images onto a user's retina. Projection systems can also project virtual objects into the physical environment (e.g., as holograms or onto physical surfaces).

[0013] For the purposes of this disclosure, an XR communication (XRC) session refers to an XR multi-user communication session, such as an XRE in which two or more devices are participating.

[0014] For the purposes of this disclosure, a local XRC device refers to the current device described in the XRC session or controlled by the described user.

[0015] For the purposes of this disclosure, a co-located XRC device refers to two devices that share a physical environment and an XRC session, so that users of the co-located devices can experience the same physical objects and XR objects.

[0016] For the purposes of this disclosure, a remote XRC device refers to an auxiliary device located in a separate physical environment from the current local XRC device. In one or more embodiments, the remote XRC device may be a participant in an XRC session.

[0017] For the purposes of this disclosure, a shared virtual element refers to an XR object that is visible to participants in an XR session or that participants in an XR session can otherwise experience.

[0018] For the purposes of this disclosure, an XRC compute environment (XRCE) refers to a compute environment or container capable of hosting an XRC session of an application. XRCE enables applications to run within an XRC session. In some cases, XRCE allows users of an XRC session to interact with the hosted application within the XRC session.

[0019] For the purposes of this disclosure, an XRCE instance refers to the XRCE of the current device described or controlled by the described user. An XRCE instance allows a user to participate in an XRC session and run applications within that session.

[0020] For the purposes of this disclosure, a second XRCE instance refers to an auxiliary device's XRCE in an XRC session other than the local XRCE instance, or an XRCE controlled by a second user. The second XRCE instance can be remote or co-located.

[0021] For the purposes of this disclosure, an XRCE application refers to an application that can be executed in the context of XRCE.

[0022] For the purposes of this disclosure, a second XRCE application refers to an auxiliary device's XRCE application in an XRC session, other than the local XRCE application, or an XRCE application controlled by a second user. The second XRCE application may be remote or co-located.

[0023] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the accompanying drawings of this disclosure are block diagrams representing structures and devices to avoid obscuring the novel aspects of the disclosed concepts. For clarity, not all features of actual specific embodiments may be described. Additionally, as part of this specification, some of the drawings of this disclosure are provided in the form of flowcharts. The blocks in any particular flowchart may be presented in a specific order. However, it should be understood that the specific order of any given flowchart is only for illustrative purposes of one embodiment. In other embodiments, any of the various elements depicted in the flowcharts may be omitted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. Furthermore, other embodiments may include additional steps not shown as part of the flowcharts. Moreover, the language used in this disclosure has been primarily chosen for readability and instructional purposes and may not have been chosen to define or limit the subject matter of the invention, thereby resorting to the necessary claims to determine such inventive subject matter. In this disclosure, reference to “an implementation” or “implementation” means that a particular feature, structure or characteristic described in connection with that implementation is included in at least one implementation of the disclosed subject matter, and the repeated references to “an implementation” or “implementation” should not be construed as necessarily referring to all of the same implementation.

[0024] It should be understood that in any actual implementation of development (as in any software and / or hardware development project), numerous decisions must be made to achieve the developer's specific goals (e.g., compliance with system and business-related constraints), and these goals may differ between different implementations. It should also be understood that such development work can be complex and time-consuming, but nevertheless, it remains routine work for those of ordinary skill in the art who design and implement graphical modeling systems in benefit from this disclosure.

[0025] See Figure 1This diagram illustrates a simplified block diagram of an electronic device 100, communicatively connected via a network 105 to an additional electronic device 110 and a network storage device 115, according to one or more embodiments of this disclosure. The electronic device 100 may be a multi-functional device such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable device, head-mounted system, projection-based system, base station, laptop computer, desktop computer, network device, or part of any other electronic system as described herein. The electronic device 100, the additional electronic device 110, and / or the network storage device 115 may additionally or alternatively include one or more additional devices (such as server devices, base stations, accessory devices, etc.) that can accommodate various functions or may distribute various functions across these additional devices. Exemplary networks such as network 105 include, but are not limited to, local area networks (such as Universal Serial Bus (USB) networks), organizational LANs, and wide area networks (such as the Internet). According to one or more embodiments, the electronic device 100 is used to participate in an XRC session. It should be understood that the various components and functions within electronic device 100, additional electronic device 110, and network storage device 115 may be distributed differently on the device or on the additional device.

[0026] Electronic device 100 may include one or more processors 125, such as a central processing unit (CPU). Processor 125 may include a system-on-a-chip (such as a system-on-a-chip present in a mobile device) and may include one or more dedicated graphics processing units (GPUs). Additionally, processor 125 may include multiple processors of the same or different types. Electronic device 100 may also include memory 135. Memory 135 may include one or more different types of memory that can be used in conjunction with processor 125 to perform device functions. For example, memory 135 may include cache, ROM, RAM, or any kind of transient or non-transitory computer-readable storage medium capable of storing computer-readable code. Memory 135 may store various programming modules for execution by processor 125, including XR module 165, XRCE module 170, and various other application programs 175. Electronic device 100 may also include storage device 130. Storage device 130 may include one or more non-transitory computer-readable storage media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and magnetic tapes, optical media (e.g., CD-ROMs and digital video optical discs (DVDs)), and semiconductor storage devices (e.g., electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM)). According to one or more embodiments, storage device 130 may be configured to store content item 160.

[0027] Electronic device 100 may also include one or more cameras 140 or other sensors 145, such as depth sensors that can determine the depth of a scene. In one or more embodiments, each of the one or more cameras 140 may be a conventional RGB camera or a depth camera. Furthermore, cameras 140 may include stereo cameras or other multi-camera systems, time-of-flight camera systems, etc. Electronic device 100 may also include a display 155. The display device 155 may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display may be configured to selectively become opaque. Projection-based systems may employ retinal projection technology, which projects graphic images onto the human retina. Projection systems may also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0028] According to one or more embodiments, memory 135 may include one or more modules comprising computer-readable code executable by processor 125 to perform functions. The memory may include, for example, an XR module 165 that can be used to provide XRE for a local XRC device. XRCE module 170 may generate an XRE-enabled environment. For example, XRCE module 170 may provide service and / or application programming interfaces that can be used by XRE and / or hosted applications to interoperate with the operating system (OS) and / or hardware of electronic device 100. An XRC session may be a computing environment supporting a shared experience for electronic device 100 and attached electronic devices 110.

[0029] Memory 135 may also include an OS module 180 for supporting the basic functions of electronic device 100 and managing the hardware of electronic device 100. OS module 180 provides an environment in which application 175 can be executed. In some cases, XRCE module 170 and XR module 165 can run in the context of OS module 180. In other cases, XRCE module 170 and XR module 165 can process the basic functions of electronic device 100 and manage the hardware of electronic device 100 in parallel with or in place of OS module 180. XRCE module 170 also provides an environment in which application 175 can be executed. Application 175 may include, for example, computer applications that can be experienced by multiple devices, such as electronic device 100 and additional electronic device 110, in an XRC session.

[0030] Although electronic device 100 is described as including the numerous components described above, in one or more embodiments, the various components may be distributed across multiple devices. Therefore, although certain calls and transmissions are described herein with respect to the specific system depicted, in one or more embodiments, various calls and transmissions may be directed differently based on the functions of different distributions. Additionally, additional components may be used, and certain combinations of the functions of any components may be combined.

[0031] Figure 2 A diagram illustrating an exemplary operating environment according to one or more embodiments is shown. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the examples in the specific embodiments disclosed herein. Therefore, as a non-limiting example, operating environment 240 includes a first physical environment, while operating environment 250 includes a second physical environment.

[0032] Figure 2 A diagram illustrating an exemplary operating environment according to one or more embodiments is shown. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the examples in the specific embodiments disclosed herein. Therefore, as a non-limiting example, operating environment 240 includes a first physical environment, while operating environment 250 includes a second physical environment.

[0033] like Figure 2 As shown, the first environment 240 includes a first user 220 using the first electronic device 200, and the second environment 250 includes a second user 232 using the second electronic device 210. In one or more embodiments, the first electronic device 200 and the second electronic device 210 include mobile devices, such as handheld devices, wearable devices, etc.

[0034] In one or more embodiments, the first electronic device 200 and the second electronic device 210 communicate with each other via a network 205. Examples of the network 205 may include, for example, the Internet, a wide area network (WAN), a local area network (LAN), etc. In one or more embodiments, the first electronic device 200 and the second electronic device 210 may participate in an XRC session.

[0035] Although electronic device 200 and second electronic device 210 can participate in a common XRC session, the XRE can be rendered differently on each device. As shown, electronic device 200 can depict physical objects of environment 240. As shown, physical table 222 can be depicted as virtual table 224 on display 242. In one or more embodiments, display 242 can be a perspective display, and virtual table 224 can be simply a view of physical table 222 through display 242.

[0036] The display 242 of the electronic device 200 may also include an avatar 226 corresponding to a user 232 in the physical environment 250. For the purposes of this disclosure, the avatar may include a virtual representation of the user. The avatar may depict real-time actions of the corresponding user 232, including movement, updated location, and / or interaction with various physical and / or virtual components within an XRC session. The avatar may or may not simulate the user's physical characteristics, and may or may not simulate the user's facial expressions.

[0037] According to one or more implementations, XRCE can be an XRC session supporting one or more XRCE applications or other modules that provide depictions of objects across all participating devices, such as electronic device 200 and second electronic device 210. As shown in display 242, presentation panel 230A is an example of virtual objects visible to all participating devices.

[0038] As an example, returning to environment 250, the second electronic device 210 includes a display 252 on which a presentation panel virtual object 230B is depicted. It should be understood that in one or more embodiments, although the same virtual object may be visible across all participating devices, it may be rendered differently depending on the location of the electronic device, the orientation of the electronic device, or other physical or virtual characteristics associated with electronic devices 200 and 210 and / or the XRCE depicted within displays 242 and 252.

[0039] Returning to environment 250, another characteristic of the XRC session is that while virtual objects can be shared across participating devices, the physical world can appear different. Therefore, the physical chair 234 is depicted as a virtual chair 236. As described above, and as in one or more embodiments, the display 252 can be a perspective display, and the virtual chair 236 can be a view of the physical chair 234 through the perspective display 252. Furthermore, the second electronic device 210 depicts an avatar 238 corresponding to the user 220 within the physical environment 240.

[0040] According to one or more embodiments, virtual objects such as presentation panel 230 can be rendered in conjunction with XRCE applications. In one or more embodiments, multiple XRCE applications can be executed within an XRCE. For example, the XRCE can host XRCE applications as virtual objects within the XRE of the XRCE. Although hosted within the XRCE, the XRCE applications can be executed as processes separate from one or more processes of the XRCE. As discussed further below, applications can be configured to run within or outside the XRCE. For example, based on the operating environment or context in which the application is executing, the application can include multiple operating modes, such as XRCE mode and non-XRCE mode. It can be determined that an application is running within the context of the XRCE and operating in XRCE mode, such as through an application programming interface (API). When running outside the XRCE, the application can run in non-XRCE mode. For example, when running in the context of the electronic device's OS rather than within the XRCE, the application runs in non-XRCE mode. As used herein, an XRCE application or a managed application can also refer to an application running in XRCE mode, as well as an application configured only to run within the context of XRCE. XRCE can provide a container or computing environment in which XRCE applications can run, much like an operating system. These containers can be presented to the user as virtual objects, making it appear as if the XRCE application is executing in a window, on a surface, or on another virtual object.

[0041] In some cases, an XRCE application executes within the context of a specific XRCE instance. For example, a first user associated with a first device can participate in an XRC session with a second user associated with a second device. In this example, the XRCE application executes on both the first and second devices. Based on input from the first user, the first device can execute a first XRCE application hosted in its local XRCE instance. The first XRCE application can share data with a second XRCE application hosted in a second XRCE instance and executed on the second device. Executing an application hosted within the context of a specific XRCE instance helps provide privacy measures for local users within the XRCE. In some cases, the XRCE instance can execute on the local device. In other cases, the XRCE instance can execute at least partially on the network or on the hosted device.

[0042] In some cases, users of an XRC session may want to share the application experience of an XRCE application with other users in the XRC session. For example, a group of users might want to experience an XRCE application together, such as a game, group drawing application, productivity application, etc., as a shared XRCE application experience within the XRC session. In some cases, applications may be configured to run inside or outside of XRCE. For example, an application may determine that it is running within the context of XRCE, such as through an API, and operating in XRCE mode. Allowing XRCE applications to access data from XRCE can raise privacy concerns. For example, it may not be desirable to allow XRCE applications to access voice data, avatars, or other communication data between users sharing an XRCE application experience. To help maintain privacy, XRCE applications may be restricted in the types of data they can receive.

[0043] Figure 3A This is a line drawing illustrating an XRCE interface 300 for bringing an XRCE application experience to an XRCE according to various aspects of this disclosure. In some cases, the XRCE 302 may display a view within a shared XRC session. The XRCE 302 may include interfaces such as menus, gestures, voice commands, or other interfaces that a user can utilize to select an application to bring to the XRCE 302. In this example, the XRCE 302 includes a menu 304 for bringing an application experience to the XRCE 302. The menu 304 may be subdivided, for example, to categorize available XRCE applications. In this example, selecting the Games category 306 displays a list 308 of available applications that can be brought to the XRCE 302.

[0044] In some cases, XRCE can run, for example, as a background application. For instance, a user can switch out of the active XRCE and switch to another application to browse and / or select the application brought into the XRC session. Figure 3BThis is a line drawing illustrating another interface for bringing an application experience into an XRC session 350, according to various aspects of this disclosure. In this example, the interface for sharing content executes outside the context of the XRCE. As an example, a user can browse an application or content on their device and access an interface for bringing the application experience into the XRCE, such as using a button, long press, selecting content items, etc. In this example, the user can select application 352 and access a context menu 354 associated with application 352. The context menu 354 includes menu items 356 for bringing the application experience into the XRCE. In some cases, a user can access an XRCE application from within the XRCE, while the XRCE application is not visible to other users of the XRCE. In some cases where the XRCE is not running, an XRCE instance can be started, and then the application can be brought into the XRCE.

[0045] In some cases, XRCE may include functionality that an application can utilize. For example, a first user may be in an XRC session with a second and a third user, and the first user may begin an application experience with the second user's XRCE application. Instructions such as flags or switches may be used to invoke or launch the XRCE application while the XRCE application is executing within the context of the XRC session. In some cases, initiating an XRCE application experience among multiple users in an XRC session may launch an instance of the XRCE application on each user's local device. For example, the XRC session may invoke the XRCE application on each user's local device participating in the application experience. The application may have access to one or more APIs that allow the application to utilize features provided by the XRCE session. For example, the XRCE application may be able to call APIs provided by XRCE to access functionality provided by XRCE. In some cases, XRCE may provide data transfer, networking, audio, video, and avatar processing capabilities that the XRCE application can use when executing within the context of XRCE. Continuing this example, if a participant in the application experience, such as a second user, does not have an XRCE application, an XRCE application, such as one running on the first user's local device, can utilize the networking capabilities of XRCE and transmit or otherwise acquire a version of the XRCE application for use during the application experience. In some cases, the version of the XRCE application may offer reduced or different functionality compared to the full XRCE application. The instance of the XRCE application associated with the first user can then use the networking capabilities provided by XRCE to transmit data to other instances of the XRCE application.

[0046] According to various aspects of this disclosure, graphical data from an XRCE application can be incorporated into the video stream of an XRC session along with visual representation data. Visual representation data is graphical data representing a user, such as an avatar and / or video data from the user. Visual representation data is collected, for example, by the XRCE instance and distributed among other devices participating in the XRC session. In some cases, visual representation data may be processed by another XRCE application. This other XRCE application may be integrated into the XRCE, for example, as a separate first-party logical component of the XRCE, to allow for the sending of individual updates, or as a first or third-party trusted application.

[0047] Users can be associated with avatars within the XRE. Within the XRE, avatars are generated, packaged into video streams, and distributed to other participants in the XRC session. Animation of the avatars and / or effects applied to them, along with video (if any) between participants in the XRC session, can also be handled by the XRCE instance and can be based on information from the video stream.

[0048] When an XRCE application runs within the context of an XRCE instance, such as during an application experience, graphics provided by the XRCE application may be passed to the XRCE instance via an API and sent as part of the video stream of an XRC session. In some cases, the XRCE application's graphics environment may be displayed as part of the XRC session's graphics environment. In other cases, during an application experience, the XRCE instance's graphics environment may be replaced by the XRCE application's graphics environment. Figure 4This is a diagram illustrating an application experience 400 according to various aspects of this disclosure. In this exemplary application experience 400, instead of the graphical environment of an XRCE instance, a graphical environment of an XRCE application 402 is shown. In some cases, graphical data generated by the XRCE application may be sent to the XRCE instance, for example, via an API and integrated into the video stream of an XRC session along with visual representation data such as avatars. According to various aspects of this disclosure, the XRCE instance continues to draw avatars representing a first user 404 and a second user 406 into the graphical environment of the XRCE application 402. For example, in the case where two users are participating in an XRCE experience, the XRCE application may provide location information on where to display the two avatars representing the two users. Animations, effects, etc., of the avatars may be rendered by the XRCE instance in the graphical environment of the XRCE application 402 in a manner similar to those animations, effects, etc., of the avatars rendered in the graphical environment of the XRC. For example, the XRCE may have access to the avatar information of a local user, and the XRCE may render the avatar representing the local user at a location corresponding to the location information provided by the XRCE application. Similarly, the local XRCE can receive avatar information from the second XRCE application. Avatar information may include information for rendering and / or animate display of a second avatar representing the second user. The XRCE application may render the second avatar at a location corresponding to the location information provided by the XRCE application. In some cases, the second XRCE application may also provide location information for one or more avatars. In some cases, animations or graphical representations of the user may be adjusted or changed based on the user's experience with the application.

[0049] When a video stream is received, the input video stream is processed by the local XRCE instance, and the XRCE application's graphics data can be rendered directly by the XRCE instance, or the XRCE application's graphics data can be extracted from the received video stream and passed to the XRCE application for processing and rendering. Because the XRCE application only has access to the graphics data generated or used by the XRCE application, the XRCE application cannot access the user's visual representation.

[0050] In some cases, an XRCE application may be able to control aspects of visual representation data. For example, an XRCE application may request the XRCE instance not to display a user's graphical representation. In such cases, the user's avatar and / or other graphical representations may be hidden. As another example, an XRCE application may provide the XRCE instance with location and / or size information for the graphical representation. This helps allow the XRCE application to instruct where, for example, the user's avatar should be positioned within the XRCE application environment to help optimize the application experience. In this example, a first user 404 is positioned relative to a second user 406 across a game board generated by the XRCE application. In some cases, an XRCE application may change or update the location and / or size of a user's graphical representation. In some cases, an XRCE application may provide location information on a per-participant basis. For example, an XRCE application may be able to access non-personally identifiable information about users, such as user identifiers, via the XRCE instance. The location and / or size information of the user's graphical representation may then be provided separately for each user identifier. This non-personally identifiable information may be changed or randomized between instances of the application experience. In some cases, an XRCE application may include login or account functionality within the XRCE application. In such cases, participants can log in or create accounts, and the XRCE application may be able to associate non-personally identifiable information provided by the XRCE instance with the account or login.

[0051] In some respects, audio content associated with or representing a user can also be processed by the XRCE instance of the XRCE application. For example, audio data generated by the XRCE application, such as music, sound effects, and interactive sounds, can be passed from the local XRCE application to the XRCE instance of a specific device. In some cases, the XRCE application may also pass metadata about the audio data, such as timing information and direction information, to the XRCE instance to help control the playback of the audio data. The XRCE instance can then transmit the representation or indication of the audio to other devices participating in the XRC session of the application experience so that it can be reproduced by other devices. Voice audio data between participants in the application experience can be processed by the XRCE instance. For example, during an XRC session, voice audio data is received from the participants of the XRC session to generate an audio stream, which is appropriately passed between the devices participating in the XRC session and played back by the devices participating in the XRC session. The XRCE instance continues to process voice audio data during the application experience. Audio data generated by the XRCE application can be sent to the XRCE instance, for example, via an API and integrated into the audio stream. For example, an XRCE application can provide audio data that can be incorporated into an audio stream of an XRCE instance and sent to other participants for playback. The received audio stream can be processed by the XRCE instance, and the audio data of the XRCE application can either be directly presented by the XRCE instance, or the audio data can be extracted from the received audio stream and passed to the XRCE application for playback, without requiring the XRCE application to access other audio data, such as speech audio data, in the remainder of the audio stream. Because the XRCE application only has access to the audio data generated or used by the XRCE application, it cannot access speech audio data between participants in the application experience. In some cases, the XRCE application may be able to control aspects of the speech audio data, for example, by providing the XRCE with directionality, silence, or other information. This information can be provided on a participant-by-participant basis or for all participants in the application experience. In some cases, the speech audio data can be processed and / or transmitted by another XRCE application. This other XRCE application can be integrated into the XRCE, for example, as a first logical component of the XRCE, or as a trusted application from a first or third party.

[0052] Figure 5This is a flowchart illustrating techniques for collaboration in a multi-user communication session 500 according to various aspects of this disclosure. At block 502, a request to launch a first application in a multi-user communication session with a second device is received. For example, multiple users may collaborate in an XRC session, and the first user may begin an application experience using an XRCE application. The first user may launch the XRCE application, for example, via an XRCE interface such as a menu. In other cases, the first user may activate an XRCE instance on the first device and launch the XRCE application and begin the application experience, for example, via an OS interface or an application interface. At block 504, the environment of the first application is presented in the multi-user communication session. For example, the XRCE application may be presented within an XRC session. In some cases, the environment of the XRCE application may be presented instead of the XRE of the XRC session. At block 506, one or more processes of the multi-user communication session receive data, including input data and first communication data. For example, the first user may provide control input and voice information to the XRCE application. This input may be received by the XRCE instance. In some cases, the multi-user communication session may be executed using one or more processes. For example, a multi-user communication session may include a process that renders a graphical interface, another process that handles communication, etc. In some cases, a multi-user communication session may include a process for interfacing with one or more XRCE applications. This process for interfacing with one or more XRCE applications may be separate from other processes in the multi-user communication session. For example, a first process may provide and / or receive communication data from other devices in the multi-user communication session, and another separate process may receive information from and / or present the environment of the XRCE application in the multi-user communication session. In some cases, communication data may be processed by one or more processes integrated into the XRCE, for example, as a first part of the XRCE's logic components. At block 508, one or more processes provide input data to a first application. For example, control input may be provided to the XRCE application by the XRCE instance. At block 510, one or more processes provide first communication data to a second device. For example, the XRCE instance may transmit received voice input data to another device participating in the application experience in the XRC session. The voice input data may be received by the first device, for example, by the microphone of the first device. Voice input can be processed by the XRCE instance and provided to the XRC session but not to the XRCE application. At box 512, one or more processes receive second communication data from a second device. For example, the XRCE instance can receive voice data from another device participating in the application experience within the XRC session. At box 514, one or more processes present the second communication data within the context of the first application, where the first application is restricted to accessing both the first and second communication data.For example, the XRCE instance presents the received voice data to the first user instead of providing the received voice data to the XRCE application. The received voice data may be presented, for example, by one or more speakers of the first device.

[0053] Figure 6A and Figure 6B An exemplary system 600 for various extended reality technologies is depicted.

[0054] In some examples, such as Figure 6A As shown, system 600 includes device 600a. Device 600a includes various components such as processor 602, RF circuitry 604, memory 606, image sensor 608, orientation sensor 610, microphone 612, position sensor 616, speaker 618, display 620, and touch-sensitive surface 622. These components optionally communicate via communication bus 650 of device 600a.

[0055] In some examples, components of system 600 are implemented in a base station device (e.g., a computing device, such as a remote server, mobile device, or laptop computer), and other components of system 600 are implemented in a second device (e.g., a head-mounted device). In some examples, device 600A is implemented in either the base station device or the second device.

[0056] like Figure 6B As shown, in some examples, system 600 includes two (or more) devices that communicate via wired or wireless connections. A first device 600B (e.g., a base station device) includes a processor 602, RF circuitry 604, and memory 606. These components optionally communicate via a communication bus 650 of device 600B. A second device 600C (e.g., a head-mounted device) includes various components such as processor 602, RF circuitry 604, memory 606, image sensor 608, orientation sensor 610, microphone 612, position sensor 616, speaker 618, display 620, and touch-sensitive surface 622. These components optionally communicate via a communication bus 650 of device 600C.

[0057] System 600 includes processor 602 and memory 606. Processor 602 includes one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 606 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 602 to perform the techniques described below.

[0058] System 600 includes RF circuitry 604. RF circuitry 604 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, intranets), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 604 optionally includes circuitry for using near-field communication and / or short-range communication (such as Bluetooth). ® The circuit for communication.

[0059] System 600 includes a display 620. Display 620 may have an opaque display. Display 620 may also have a transparent or translucent display, which may be combined with a substrate through which light representing an image is directed to an individual's eye. Display 620 may incorporate LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The light-transmitting substrate may be an optical waveguide, an optical combiner, a light reflector, a holographic substrate, or any combination of these substrates. In one example, the transparent or translucent display may selectively switch between an opaque state and a transparent or translucent state. Other examples of display 620 include head-up displays, automotive windshields capable of displaying graphics, windows capable of displaying graphics, lenses capable of displaying graphics, tablet computers, smartphones, and desktop or laptop computers. Alternatively, system 600 may be designed to receive an external display (e.g., a smartphone). In some examples, system 600 is a projection-based system that uses retinal projection to project images onto an individual's retina or to project virtual objects into a physical scene (e.g., onto a physical surface or as a hologram).

[0060] In some examples, system 600 includes a touch-sensitive surface 622 for receiving user input such as tap and swipe input. In some examples, display 620 and touch-sensitive surface 622 form a touch-sensitive display.

[0061] System 600 includes an image sensor 608. Image sensor 608 optionally includes one or more visible light image sensors, such as charge-coupled device (CCD) sensors and / or complementary metal-oxide-semiconductor (CMOS) sensors, operable to acquire images of physical elements from a physical scene. One or more image sensors also optionally include one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light from the physical scene. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the physical scene. Image sensor 608 also optionally includes one or more event cameras configured to capture movement of physical elements in the physical scene. Image sensor 608 also optionally includes one or more depth sensors configured to detect the distance between physical elements and system 600. In some examples, system 600 uses a combination of CCD sensors, event cameras, and depth sensors to detect the physical scene around system 600. In some examples, image sensor 608 includes a first image sensor and a second image sensor. The first and second image sensors are optionally configured to capture images of physical elements in a physical setting from two different perspectives. In some examples, system 600 uses image sensor 608 to receive user input, such as gestures. In some examples, system 600 uses image sensor 608 to detect the position and orientation of system 600 and / or display 620 in the physical setting. For example, system 600 uses image sensor 608 to track the position and orientation of display 620 relative to one or more fixed elements in the physical setting.

[0062] In some examples, system 600 includes microphone 612. System 600 uses microphone 612 to detect sound from a user and / or the user's physical set. In some examples, microphone 612 includes a microphone array (including multiple microphones) that optionally cooperate to identify ambient noise or locate sound sources in the space of the physical set.

[0063] System 600 includes an orientation sensor 610 for detecting the orientation and / or movement of system 600 and / or display 620. For example, system 600 uses orientation sensor 610 to track changes in the position and / or orientation of system 600 and / or display 620 relative to physical elements in a physical setting. Orientation sensor 610 may optionally include one or more gyroscopes and / or one or more accelerometers.

[0064] The technologies defined in this document take into account options for obtaining and utilizing users' personal information. For example, such personal information may be used to provide an improved XRE experience and application experience on electronic devices. However, with regard to the collection of such personal information, it should be obtained with the user's informed consent, enabling the user to know and control the use of their personal information.

[0065] Parties with access to personal information will use it solely for legitimate and reasonable purposes and will comply with privacy policies and practices that at least meet appropriate laws and regulations. Furthermore, such policies should be comprehensive, user-accessible, and considered to meet or exceed government / industry standards. In addition, personal information will not be distributed, sold, or otherwise shared outside of any legitimate and reasonable purpose.

[0066] However, users can limit the extent to which parties can access personal information. The processes and devices described herein may allow for changes to settings or other preferences that enable users to control access to their personal information. Furthermore, while some of the characteristics defined herein are described in the context of the use of personal information, aspects of these characteristics can be implemented without the need for such information. As an example, a user's personal information may be obscured or otherwise generalized so that it does not identify the specific user from whom it is obtained.

[0067] It should be understood that the above description is intended to be exemplary and not restrictive. Material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims and to provide that material in the context of a particular embodiment, variations of which will be readily apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in combination with each other). Therefore, Figure 5 The specific arrangement of the steps or actions shown or Figures 1 to 4 The arrangement of the elements shown in Figure 6 should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents. In the appended claims, the terms “comprising” and “therein” are used as common English equivalents of the corresponding terms “including” and “wherein”.

Claims

1. A method for collaboration in a multi-user communication session, comprising: A request to launch a first application is received in a multi-user communication session, wherein the first device and the second device are active in the multi-user communication session; The first device receives data, which includes input data and first communication data. The input data is provided by the first device to the first application; as well as The first communication data is transmitted to the second device, wherein the first application is restricted from accessing the first communication data.

2. The method according to claim 1, further comprising: The first device receives the second communication data from the second device; as well as The second communication data is presented by the first device together with the first application. The first application is restricted from accessing the second communication data.

3. The method of claim 1, wherein, based on determining that the first application is not a trusted application, the first application is restricted from accessing the first communication data.

4. The method according to claim 1, further comprising: Upon receiving the request to launch the first application, the first device launches the first application in an extended reality computing environment, wherein the first communication data is transmitted by the extended reality computing environment without providing the first communication data to the first application.

5. The method of claim 1, wherein the input data is configured to control the functions of the first application.

6. The method of claim 1, wherein the first communication data includes identifiable information related to a user of the first device.

7. The method of claim 1, wherein the first communication data includes voice audio data received from a user of the first device.

8. A system comprising: One or more processors; as well as One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media comprising computer-readable code executable by the one or more processors to perform the following operations: A request to launch a first application is received in a multi-user communication session, wherein the first device and the second device are active in the multi-user communication session; The first device receives data, which includes input data and first communication data. The input data is provided by the first device to the first application; as well as The first communication data is transmitted to the second device, wherein the first application is restricted from accessing the first communication data.

9. The system of claim 8, further comprising computer-readable code that performs the following operations: The first device receives second communication data from the second device; and The second communication data is presented by the first device together with the first application. The first application is restricted from accessing the second communication data.

10. The system of claim 8, wherein, based on determining that the first application is not a trusted application, the first application is restricted from accessing the first communication data.

11. The system of claim 8, further comprising computer-readable code that performs the following operations: launching the first application in an extended reality computing environment upon receiving the request to launch the first application, wherein the first communication data is transmitted by the extended reality computing environment without providing the first communication data to the first application.

12. The system of claim 8, wherein the first communication data includes identifiable information related to a user of the first device.

13. The system of claim 8, wherein the first communication data includes voice audio data received from a user of the first device.

14. A non-transitory computer-readable medium comprising computer-readable code executable by one or more processors to perform the method according to any one of claims 1-7.

15. A method for collaboration in a multi-user communication session, comprising: User input data is received at a first device that participates in a shared extended reality environment with a second device, wherein the shared extended reality environment hosts a shared application that can be accessed by both the first and second devices within the shared extended reality environment; as well as Based on the determination that a first portion of the user input data corresponds to a restricted category, the first portion of the user input data is transferred from a first instance of the shared extended reality environment on the first device to a second instance of the shared extended reality environment on the second device, while bypassing the shared application.

16. The method of claim 15, further comprising: Based on the determination that the second part of the user input data corresponds to an unrestricted category, the second part of the user input data is provided to a first instance of the shared application on the first device for transmission to a second instance of the shared application on the second device.

17. The method of claim 16, wherein the shared application is updated by the second portion of the user input data.

18. The method of claim 15, wherein in response to determining that the shared application is an untrusted application, the determination that the first portion of the user input data corresponds to a restricted category is performed.

19. The method of claim 18, wherein the determination that the first portion of the user input data corresponds to a restricted category is performed by a first application separate from the shared application.

20. The method of claim 15, wherein the first portion of the user input data includes user identifiable information, and the first portion of the user input data is determined to correspond to the first category.

21. The method of claim 15, wherein the first portion based on the user input data includes user communication data, and the first portion of the user input data is determined to correspond to the first category.

22. A system comprising: One or more processors; as well as One or more computer-readable media, the one or more non-transitory computer-readable media comprising computer-readable code executable by the one or more processors to perform the following operations: User input data is received at a first device that participates in a shared extended reality environment with a second device, wherein the shared extended reality environment hosts a shared application that can be accessed by both the first and second devices within the shared extended reality environment; as well as Based on the determination that a first portion of the user input data corresponds to a restricted category, the first portion of the user input data is transferred from a first instance of the shared extended reality environment on the first device to a second instance of the shared extended reality environment on the second device, while bypassing the shared application.

23. The system of claim 22, further comprising computer-readable code that performs the following operations: Based on the determination that the second part of the user input data corresponds to an unrestricted category, the second part of the user input data is provided to a first instance of the shared application on the first device for transmission to a second instance of the shared application on the second device.

24. The system of claim 23, wherein the shared application is updated by the second portion of the user input data.

25. The system of claim 22, wherein in response to determining that the shared application is an untrusted application, the determination that the first portion of the user input data corresponds to a restricted category is performed.

26. The system of claim 25, wherein the determination that the first portion of the user input data corresponds to a restricted category is performed by a first application separate from the shared application.

27. The system of claim 22, wherein the first portion based on the user input data includes user communication data, the first portion of the user input data being determined to correspond to the first category.

28. A non-transitory computer-readable medium comprising computer-readable code executable by one or more processors to perform the method according to any one of claims 15-21.

Images