Group communication platform

Abstract

The present disclosure generally relates to a group communication platform. A group communication platform facilitates sharing an application environment with other users. The platform may receive a request to initiate a group session for a local user and a remote user. An out-of-process network connection may be established for the group session with a system communication channel between a local computing device associated with the local user and a remote computing device associated with the remote user. A system call may be received from a local instance of a first application on the local computing device to communicate local data to a remote instance of the first application on the remote computing device via the out-of-process network connection. The local data may be communicated to the remote instance of the first application on the remote computing device via the off-process network connection and the system communication channel. The local data may include state data for the local instance of the first application for updating a state of the remote instance of the first application.

Description

[0001] This application is a divisional application of the application with international application number PCT / US2021 / 026393, international application date April 8, 2021, date of entry into the Chinese national phase October 13, 2022, national application number 202180028285.8, and invention name “Group Communication Platform”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Patent Application No. 17 / 224,072, filed on April 6, 2021, which claims priority to U.S. Provisional Patent Application No. 63 / 011,966, filed on April 17, 2020, the disclosures of each of which are hereby incorporated herein in their entirety. Technical Field

[0004] This specification generally relates to multi-user environments in computing platforms. Background Art

[0005] Extended reality applications create simulated environments that users can interact with using electronic interfaces. When these interactions are shared with other users, they become richer and more interesting experiences. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Certain features of the subject technology are set forth in the appended claims. However, for illustrative purposes, several specific implementations of the subject technology are set forth in the following drawings.

[0007] Figure 1 An exemplary network environment is shown in which a group communication platform can operate in accordance with aspects of the subject technology.

[0008] Figure 2 An exemplary computing device is shown upon which aspects of the subject technology may be implemented.

[0009] Figure 3 is a block diagram illustrating components of data flow managed by a group communication platform in accordance with aspects of the subject technology.

[0010] Figure 4 A flow diagram illustrating an exemplary process for managing a group session of a shared XR environment in accordance with aspects of the subject technology is shown.

[0011] Figure 5 An exemplary computing device is shown that can be used to implement aspects of the subject technology. DETAILED DESCRIPTION

[0012] The specific embodiments shown below are intended to be descriptions of various configurations of the subject technology and are not intended to represent the only configuration in which the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the specific embodiments. The specific embodiments include specific details intended to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details described herein, and one or more other specific implementations can be used to practice. In one or more specific implementations, structures and components are shown in block diagram form to avoid blurring the concepts of the subject technology.

[0013] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, the extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR system are adjusted in a manner that complies with at least one physical law. For example, the XR system can detect head movement, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting an XR environment, and in response, adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust features of graphical content in an XR environment in response to representations of physical movement (e.g., voice commands).

[0014] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0015] Extending a simulation environment such as an XR environment to allow multiple users to interact with each other and share the experience of the simulation environment brings a richer and more interesting experience to the user. However, a multi-user simulation environment requires establishing and maintaining a network connection through which data for updating and synchronizing the states of corresponding applications executed on different computing devices is shared. When a user wishes to share more than one XR application at the same time, the multi-user simulation environment becomes more complicated.

[0016] The subject technology provides a group communication platform as a system-level process integrated into the operating system of a computing device. The group communication platform facilitates multiple users to join a group session that will be presented (e.g., simultaneously) in a common XR environment via their respective computing devices. For example, the subject technology establishes and maintains a system-level network connection between the operating systems on the respective computing devices for the group session. This system-level network connection is exposed to higher-level applications and system services to facilitate communication with corresponding applications and system services on another computing device without requiring each application and / or system service to establish and maintain its own network connection with the other computing device. In addition, the system-level network connection can be an exposed out-of-process network connection to allow multiple applications and / or system services to utilize the connection, rather than using multiple network connections associated with corresponding applications and / or system services within the same group session.

[0017] According to aspects of the subject technology, a local user may wish to initiate a group session with a remote user to share and present (e.g., concurrently present) the group session in a common XR environment. The local user and the remote user may be close to each other, such as in the same room, or may be geographically distant from each other, such as on opposite sides of the globe. A system may receive a request to initiate a group session of a local user and a remote user. In response to the request, an out-of-process network connection to a system communication channel between a local computing device associated with the local user and a remote computing device associated with the remote user may be established for the group session. The system communication channel may include a connection to a relay server that manages connections and communications with all participants in the group session. Alternatively, the system communication channel may include a peer-to-peer connection between the local computing device and the remote computing device.

[0018] One or more applications may be shared between participants in a group session. Data may be exchanged between respective instances of the applications to update respective states of the applications in a common XR environment. For example, a local instance of a first application on a local computing device may make a system call to pass local data to a remote instance of the first application on a remote computing device via an out-of-process network connection. In response to the system call, the local data may be passed to the remote instance of the first application on the remote computing device via the out-of-process network connection and a system communication channel. Similarly, a local instance of a second application participating in the group session may make a system call to pass local data to a remote instance of the second application on the remote computing device via an out-of-process network connection. The local data of the second application may then be passed to the remote instance of the second application using the same out-of-process network connection used by the first application.

[0019] Figure 1An exemplary network environment in which a group communication platform can operate according to aspects of the subject technology is shown. However, not all of the components depicted can be used in all specific implementations, and one or more specific implementations may include additional or different components than those shown in the figures. Changes in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0020] The network environment 100 includes computing devices 102, 104, 106, and 108 (hereinafter 102-108), a server 110, and a network 112. The network 112 may communicatively couple (directly or indirectly), for example, any two or more of the computing devices 102-108 and the server 110. In one or more specific implementations, the network 112 may be an interconnected network that may include the Internet and / or devices communicatively coupled to the Internet. Any two of the computing devices 102-108 (e.g., Figure 1 The computing devices 104 and 106 shown in FIG. 1 may be communicatively coupled using a peer-to-peer connection such as using Bluetooth, near field communication, Wi-Fi, etc. For purposes of illustration, the network environment 100 is illustrated in FIG. Figure 1 1 as including computing devices 102-108 and server 110; however, network environment 100 may include any number of computing devices and any number of servers.

[0021] One or more of the computing devices 102-108 may be, for example, a portable computing device such as a laptop computer, a smart phone, a smart watch, a tablet device, a wearable device, etc., or any other type of device that includes, for example, one or more wireless interfaces, such as a WLAN radio, a cellular radio, a Bluetooth radio, a Zigbee radio, a near field communication (NFC) radio, and / or other radios. Figure 1 , by way of example, computing device 102 is depicted as a smartphone, computing device 104 is depicted as a laptop, computing device 106 is depicted as a tablet device, and computing device 108 is depicted as another smartphone. Wearable devices may include headphones, goggles, glasses, and other types of head-mountable devices. Head-mountable devices may also include an arrangement in which a smartphone may be arranged to form a headset worn by a user. Each of computing devices 102-108 may be and / or may include the following references to Figure 2 The device and / or the following reference Figure 5 All or part of the computing device described.

[0022] The server 110 may be and / or may include the following references Figure 5All or part of the device described. Server 110 may include one or more servers, such as a server cloud. For illustrative purposes, a single server 110 is shown and discussed with reference to various operations. Server 110 may be a relay server that is configured to manage connections with two or more of the computing devices 102-108 that have joined the group session to facilitate data communications between applications executed on the computing devices 102-108. Server 110 may manage unicast communications between corresponding pairs of computing devices 102-108 in the group session. Server 110 may also manage fan-out communications from one of the computing devices 102-108 to other participants in the group session. However, these operations and other operations discussed herein may be performed by one or more servers, and each different operation may be performed by the same or different servers.

[0023] Figure 1 Also included are user A, user B, user C, and user D, which are associated with computing devices 102-108, respectively. A user may be considered associated with a particular computing device based on the user logging into the computing device using a set of credentials for a user account that grants access to the resources and content of the computing device. The computing device itself may authenticate the credentials, or may pass the credentials to an authentication server for authentication. Although Figure 1 One user is shown associated with each depicted computing device, but a particular user may be associated with more than one computing device.

[0024] As described in detail below, the subject technology facilitates the creation of a group session for two or more of users AD to share a first application. For example, user A may wish to initiate a group session including user A and user B. According to aspects of the subject technology, an out-of-process network connection to a system communication channel between computing device 102 and computing device 104 managed by server 110 may be established on each of computing device 102 and computing device 104. Data may be exchanged between respective instances of the first application on computing device 102 and computing device 104 via the out-of-process network connection and the system communication channel to update the respective states of the instances of the first application. In this way, user A and user B may experience and be presented in (e.g., simultaneously presented in) an environment generated by the first application.

[0025] Figure 2 An exemplary computing device 200 for implementing various aspects of the subject technology is shown. The computing device 200 may correspond to Figure 1108. However, not all of the components depicted may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and types of these components may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, or fewer components may be provided.

[0026] As shown, the computing device 200 includes a processor 202 and a memory 204. The processor 202 may include appropriate logic components, circuits, and / or code that can process data and / or control the operation of the computing device 200. For example, the processor 202 may be implemented in software (e.g., subroutines and code), hardware (e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic components, discrete hardware components, or any other appropriate devices), and / or a combination of the two. In this regard, the processor 202 may enable control signals to be provided to various other components of the computing device 200. The processor 202 may also control the transfer of data between various parts of the computing device 200. In addition, the processor 202 may enable the implementation of an operating system or otherwise execute code to manage the operation of the computing device 102.

[0027] The memory 204 may include suitable logic components, circuits, and / or code to enable storage of various types of information, such as received data, generated data, code, and / or configuration information. The memory 204 may include, for example, random access memory (RAM), read-only memory (ROM), flash memory, and / or magnetic storage devices. Figure 2 As shown, memory 204 may contain code or instruction sequences that can be executed by processor 202 to implement various software components. For example, memory 204 may contain code for operating system 206, exclusive application 208, shared applications 210 and 212, and system application 214. Figure 2 Only one example of an exclusive application and a system application and two examples of a shared application are included, but the subject technology is not limited to these numbers. More than one exclusive application and a system application and / or more than two shared applications may exist in memory 204 and they may be available for execution by processor 202.

[0028] Operating system 206 manages the resources of computing device 200 and facilitates the communication of data and control signals between components of computing device 200. According to aspects of the subject technology, a group communication platform is integrated into operating system 206 such that operating system 206 facilitates, at a system level, a user of computing device 200 to share experiences or functionality provided by one or more of the applications executed on computing device 200 with other users via other respective computing devices executing other instances of one or more applications.

[0029] The exclusive application 208 represents an application that operates in a full screen mode when executed by the processor 202. In this regard, the exclusive application 208 presents control over the user interface of the computing device 200 while the exclusive application 208 is active. For example, the exclusive application 208 may generate an XR environment that is presented for display on the computing device 200 and for interaction by a user via the user interface of the computing device 200. Only one exclusive application may be active on the computing device 200 at a time.

[0030] The shared applications 210 and 212 represent applications operating in a multitasking environment that allows more than one application to be active at one time in a graphical user interface of the computing device 200. Each of the shared applications 210 and 212 can generate XR objects that coexist within the overall XR environment. The system application 214 manages the overall XR environment by controlling the launching, moving, and closing of the shared applications 210 and 212. A user of the computing device 200 can interact with different XR objects within the XR environment via the user interface of the computing device 200. In this regard, the system application 214 behaves like an exclusive application, controlling the user interface of the computing device 200.

[0031] The XR environment may include one or more XR objects that a user can interact with or experience. XR objects may represent different sensory experiences (e.g., visuals, sounds). For example, an XR object may be a visually rendered object that a user can observe using the XR environment. The visually rendered object may be for the user to observe only, or may include a user interface element that allows the user to interact with the XR object and / or access functionality provided by the underlying code of the XR object. Another XR object may be an audio object that provides spatial audio for the XR environment. The spatial audio object may be played using the audio components of the computing device 200 and may correspond to the visual XR object and its corresponding position in the overall XR environment.

[0032] Figure 3is a block diagram illustrating components of data flows managed by a group communication platform according to aspects of the subject technology. However, not all of the depicted components may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figure. Changes in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0033] Figure 3 Depicted are software components of two computing devices representing data flows between components that represent aspects of using a group communication platform. For example, exclusive application 300A, shared application 302A, system application 304A, peer process 306A, and relayed network access 308A represent software code that can be executed on a first computing device. Similarly, exclusive application 300B, shared application 302B, system application 304B, peer process 306B, and relayed network access 308B represent software code that can be executed on a second computing device. Figure 2 Describes exclusive applications, shared applications, and system applications.

[0034] Peer processes 306A and 306B represent background processes executed on their respective computing devices to facilitate data communication between computing devices via a peer-to-peer communication channel. Relay network processes 308A and 308B represent background processes executed on their respective computing devices to facilitate communication via a peer-to-peer communication channel. Figure 3 A relay network communication channel represented by relay 310 in the figure is managed by a relay server to perform data transfer between computing devices. Computing devices can use one or both of these types of communication channels to communicate with each other. According to various aspects of the subject technology, multiple peer-to-peer communication channels connecting different corresponding computing devices can be maintained by multiple peer processes. A computing device can also communicate with multiple other computing devices via a relay network communication channel, wherein the relay server manages unicast connections between corresponding pairs of computing devices and / or multicast connections originating from one or more of the computing devices.

[0035] Figure 3The Group Communication Management (GCM) box in represents an application programming interface (API) provided by the group communication platform for executing on a computing device to access functional applications (such as exclusive applications, shared applications, and system applications) provided by the group communication platform. The API may include, for example, functions for initiating a group session, sharing an application with a group session, inviting another user to join a group session, and leaving a group session. The API may also include methods for registering XR objects to be copied and synchronized between two or more computing devices. The API may include various callback functions for notifying users when a group session has started or ended, notifying when a specific user has joined or left a group session, notifying when a specific application has been shared in or removed from a group session, providing user IDs for users participating in a group session, and identifying whether users in a group session are physically adjacent (e.g., in the same room) or geographically distant (e.g., located in different buildings, cities, countries, etc.). The API provides a handle to an out-of-process network connection to a communication channel established between computing devices. The API includes methods for transmitting data to each computing device in a group session and / or to multiple computing devices in a group session via an out-of-process network connection (e.g., fan-out communication).

[0036] Figure 3 The XR Engine box in represents the API of the low-level rendering system that manages ownership and synchronization of XR objects by applications within the XR environment. The rendering system may include simulation and physics engines for use by the rendering process. The XR Engine may facilitate communication between different applications on itself that are external to the group communication platform described herein. Alternatively, the XR Engine may use the group communication platform to connect to the initiated connection and transfer data between instances of the application.

[0037] Figure 3 The boxes in are shown as being connected by solid arrows or dashed arrows. Solid arrows represent actual data flow between software components, while dashed lines represent perceived data flow between software components. For example, exclusive application 300A may pass / transmit data to exclusive application 300B via the group communication platform to update the state of exclusive application 300B based on changes to the state of exclusive application 300A. Because the group communication platform establishes and maintains an out-of-process network connection to be used for such data transfer, exclusive applications 300A and 300B may appear as if they are directly connected through the group communication platform API without involving management of the actual network connection that enables data transfer. The solid arrows between shared application 302A and system application 304A and between shared application 302B and system application 304B represent data communication within the XR engine network to enable system applications to present XR objects generated by corresponding shared applications within the shared XR environment.

[0038] Figure 4 A flow chart of an exemplary process for managing a group session of a shared XR environment in accordance with aspects of the subject technology is shown. The blocks of process 400 are described herein as occurring serially or linearly. However, multiple blocks of process 400 may occur in parallel. Furthermore, the blocks of process 400 need not be executed in the order shown, and / or one or more blocks of process 400 need not be executed and / or may be replaced by other operations.

[0039] A group communication platform on a computing device receives a request to initiate a group session for a group of users (block 402). The request may be in response to a user interaction with a user interface element of an exclusive application, a shared application, a system application, or an operating system indicating that the user wishes to initiate a group session. The request may include a user ID of each user invited to participate in the group session and an identifier of each application to be shared within the group session.

[0040] In response to the request, the group communication platform on the computing device creates a group session and assigns a group ID to the session. The group communication platform further establishes an out-of-process network connection to a system communication channel for the group session (box 404). As described above, the out-of-process network connection is managed by the group communication platform rather than by individual applications. This configuration allows multiple applications to utilize the network connection without the burden of the overhead associated with managing the connection. The system communication channel can pass through a network and a relay server, where the application can rely on the identification service provided by the relay server to communicate with computing devices associated with other users in the same group session. The out-of-process network connection can provide access to the socket by exposing the handle of the socket to the application shared in the group session.

[0041] The group communication platform on the computing device can send the invitation via the out-of-process network connection and the system communication channel to other users invited to participate at the respective computing devices associated with those users. If the other users accept the invitation with an affirmative interaction with the user interface of the respective computing device, the group communication platform can provide a notification that the other users have joined the group session.

[0042] According to aspects of the subject technology, a group session may have been created for a group of users. In this case, a group communication platform may receive a request to share a new application with the users in the group session. An invitation to share the new application may be issued by the group communication platform to other participants in the group session. If the other participants in the group session accept the invitation, the group communication platform may provide an acceptance notification.

[0043] According to aspects of the subject technology, a group communication platform may receive a request to add a new user to an existing group session. The group communication platform may issue an invitation to the new user at a computing device associated with the new user to join the group session and share one or more applications shared within the group session. If the new user accepts the invitation, the group communication platform may provide an acceptance notification.

[0044] During the group session, the group communication platform may receive a system call to transfer data from a first instance of an application being shared to other instances of the sharing application on computing devices associated with other users participating in the group session via an out-of-process network connection (block 406). After receiving the system call, the group communication platform facilitates transferring the data to the system communication channel via the out-of-process network connection (block 408). According to aspects of the subject technology, the data transfer process may use a shared system memory to which the shared instances of the application write the data to be shared. The system call from the sharing application may be forwarded by the group communication platform to a peer process or a relay network process, depending on which system communication channel is being used. The process receiving the system call reads the data from the shared memory and transfers the data to a corresponding process on the other computing device for distribution to the corresponding application on the computing device.

[0045] The data passed between instances of a shared application may include state data for one instance of the shared application, which another instance of the shared application uses to update the state of the instance of the shared application. The state of each instance of the shared application may be updated to reflect changes in the corresponding state of the other instances of the shared application by passing the state data to each of the other instances. In this way, a multi-user state may be maintained for the shared application and maintained consistent across all instances of the shared application. Each instance of the shared application may manage coordination with other instances. Alternatively, one instance of the shared application may be responsible for collecting state data from all other instances of the shared application, and then combining the state data into a single set of multi-user state data for passing to other instances of the shared application.

[0046] According to aspects of the subject technology, the state of different instances of a shared application may not be maintained as the same state. For example, a user of one instance of a shared application may only want to share a portion of the application with other users in a group session. In this example, the data passed to the other instances of the shared application includes only the information intended to be shared, so that the other instances of the shared application are updated to reflect only the shared information.

[0047] A user may have an application installed on their associated computing device that is not already installed on the computing devices of other users participating in the group session. Upon receiving a request to share an application with other users in the group session, the group communication platform may send an invitation to the other computing devices. Upon receiving the invitation, the group communication platform on the other computing devices may generate an invitation to install the missing application on the other computing devices. If the other users accept the invitation and install the missing application on their respective computing devices, the sharing process may proceed as described above.

[0048] For another alternative scenario where the shared application is not installed on the computing devices of the other users in the group session, the application can be cast to the other computing devices to replicate the visual and / or audio aspects of the shared application without providing interactivity with the shared application for the other users. For example, if the shared application is not installed on the other computing devices, the system application that manages the shared application on the first computing device can pass along a scene graph that includes the visual interface of the shared application to the corresponding system application on the other computing devices. The corresponding system application presents the scene graph on the other computing devices so that the other users can see the visual representation of the shared application even if they cannot interact with the shared application.

[0049] As described above, aspects of the subject technology may include the collection and delivery of data from applications to other users' computing devices. The present disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include demographic data, location-based data, online identifiers, phone numbers, email addresses, home addresses, data or records related to a user's health or fitness level (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other personal information.

[0050] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, the personal information data can be used in a collaborative setting with multiple users. In addition, the present disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used according to the user's preferences to provide insights into their overall health, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0051] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities will be expected to implement and consistently apply privacy practices that are generally recognized as meeting or exceeding the requirements of the industry or government that maintain user privacy. Such information about the use of personal data should be prominently and easily accessible to users, and should be updated as the collection and / or use of data changes. The user's personal information should be collected only for legitimate use. In addition, such collection / sharing should only occur after receiving user consent or other legal basis specified in applicable law. In addition, such entities should consider taking any necessary steps to defend and safeguard access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted for specific types of personal information data collected and / or accessed, and applied to applicable laws and standards, including specific considerations that are exclusive to jurisdictions that can be used to impose higher standards. For example, in the United States, collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0052] Regardless of the foregoing, the present disclosure also contemplates specific implementations in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case of selectively sharing information from a specific application, the technology of the present invention may be configured to allow users to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for the service. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0053] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at a city level rather than at an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods such as differential privacy, where appropriate.

[0054] Therefore, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or part of such personal information data.

[0055] Figure 5 An exemplary computing device for implementing various aspects of the subject technology according to one or more specific implementations is shown. Computing device 500 can be any computing device or server and / or a part thereof for generating the features and processes described above, including but not limited to laptop computers, smart phones, tablet devices, wearable devices such as goggles or glasses, etc. Computing device 500 may include various types of computer-readable media and interfaces for various other types of computer-readable media. Computing device 500 includes permanent storage device 502, system memory 504 (and / or buffer), input device interface 506, output device interface 508, bus 510, ROM 512, one or more processing units 514, one or more network interfaces 516, image sensor 518 and / or their subsets and variations.

[0056] The bus 510 generally represents all system, peripheral, and chipset buses that communicatively connect the many internal devices of the computing device 500. In one or more implementations, the bus 510 communicatively connects the one or more processing units 514 with the ROM 512, the system memory 504, and the permanent storage device 502. The one or more processing units 514 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed herein. In different implementations, the one or more processing units 514 can be a single processor or a multi-core processor.

[0057] ROM 512 stores static data and instructions needed by one or more processing units 514 and other modules of computing device 500. On the other hand, permanent storage device 502 can be a read-write memory device. Permanent storage device 502 can be a non-volatile memory unit that stores instructions and data even when computing device 500 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) can be used as permanent storage device 502.

[0058] In one or more implementations, a removable storage device (such as a floppy disk, a flash drive, and its corresponding disk drive) can be used as the permanent storage device 502. Like the permanent storage device 502, the system memory 504 can be a read-write memory device. However, unlike the permanent storage device 502, the system memory 504 can be a volatile read-write memory, such as a random access memory. The system memory 504 can store any of the instructions and data that one or more processing units 514 may need at runtime. In one or more implementations, the processes disclosed in the subject matter are stored in the system memory 504, the permanent storage device 502, and / or the ROM 512. The one or more processing units 514 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes of one or more implementations.

[0059] Bus 510 is also connected to input device interface 506 and output device interface 508. Input device interface 506 enables a user to transmit information and select commands to computing device 500. Input devices that can be used with input device interface 506 may include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). Output device interface 508 may, for example, enable the display of images generated by computing device 500. Output devices that can be used with output device interface 508 may include, for example, a printer and a display device, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information.

[0060] One or more implementations may include a device that acts as both an input device and an output device, such as a touch screen. In these implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user may be received in any form, including acoustic, voice, or tactile input.

[0061] The bus 510 is also connected to an image sensor 520. In one or more implementations, the image sensor 520 can be used to capture image data, including but not limited to RGB image data or infrared image data.

[0062] Finally, if Figure 5As shown, bus 510 also couples computing device 500 to one or more networks and / or to one or more network nodes via one or more network interfaces 516. In this manner, computing device 500 may be part of a computer network such as a LAN, a wide area network ("WAN"), or an intranet, or may be part of one of a plurality of networks such as the Internet. Any or all components of computing device 500 may be used with the subject disclosure.

[0063] Implementations within the scope of the present disclosure may be implemented in part or in whole using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) programmed with one or more instructions. Tangible computer-readable storage media may also be non-transitory in nature.

[0064] Computer-readable storage media can be any storage media that can be read, written, or otherwise accessed by a general or special computing device, including any processing electronics and / or processing circuitry capable of executing instructions. For example, without limitation, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.

[0065] In addition, the computer-readable storage medium may include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more specific implementations, the tangible computer-readable storage medium may be directly coupled to the computing device, while in other specific implementations, the tangible computer-readable storage medium may be indirectly coupled to the computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0066] Instructions can be directly executable, or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or can be implemented as high-level language instructions that can be compiled to produce executable or non-executable machine code. In addition, instructions can also be implemented as data, or can include data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those skilled in the art, the details including but not limited to the number, structure, sequence and organization of instructions can be significantly different without changing the underlying logic, function, processing and output.

[0067] Although the above discussion mainly relates to a microprocessor or multi-core processor that executes software, one or more specific implementations are performed by one or more integrated circuits such as ASICs or FPGAs. In one or more specific implementations, such integrated circuits execute instructions stored on the circuit itself.

[0068] Those skilled in the art will recognize that the various illustrative frames, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, parts, methods and algorithms have been generally described above according to functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician can implement the described functionality in different ways for each specific application. Various parts and frames can be arranged differently (e.g., arranged in different orders, or divided in different ways) without departing from the scope of the subject technology.

[0069] It should be understood that the specific order or the hierarchical structure of the frame in the process disclosed by the present invention are the illustration of exemplary methods. Based on the design preferred requirements, it should be understood that the specific order or the hierarchical structure of the frame in the process can be rearranged or all the frames shown are executed. Any frame in these frames can be executed simultaneously. In one or more specific implementations, multitasking and parallel processing may be advantageous. In addition, the division of each system component in the above-mentioned specific implementation should not be understood as requiring such division in all specific implementations, and it should be understood that program components and systems can be generally integrated together in a single software product or encapsulated in multiple software products.

[0070] As used in this specification and any claims of this patent application, the terms "base station", "receiver", "computer", "server", "processor" and "memory" all refer to electronic devices or other technical devices. These terms exclude people or groups of people. For the purpose of this specification, the term "display" or "displaying" means displaying on an electronic device.

[0071] As used herein, the phrase "at least one of" following a list of items, any of which is separated by the terms "and" or "or", modifies the list as a whole, rather than each member (i.e., each item) of the list. The phrase "at least one of" does not require selection of at least one of each item listed; rather, the phrase allows for a meaning that includes at least one of any one item and / or at least one of any combination of items and / or at least one of each item. For example, the phrases "at least one of A, B, and C" or "at least one of A, B, or C" each refer to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0072] The predicate words "configured to," "operable to," and "programmed to" do not imply any particular tangible or intangible modification of a subject matter but are intended to be used interchangeably. In one or more specific implementations, a processor configured to monitor and control an operation or component may also mean that the processor is programmed to monitor and control the operation or that the processor is operable to monitor and control the operation. Likewise, a processor configured to execute code may be interpreted as a processor programmed to execute code or operable to execute code.

[0073] Phrases such as aspect, this aspect, on the other hand, some aspects, one or more aspects, a specific implementation, this specific implementation, another specific implementation, some specific implementations, one or more specific implementations, an embodiment, this embodiment, another embodiment, some embodiments, one or more embodiments, configuration, this configuration, other configurations, some configurations, one or more configurations, subject technology, disclosure, the present disclosure, other variations thereof, etc. are for convenience and do not mean that the disclosure involving such one or more phrases is essential to the subject technology, nor does it mean that such disclosure applies to all configurations of the subject technology. Disclosures involving such one or more phrases may apply to all configurations or one or more configurations. Disclosures involving such one or more phrases may provide one or more examples. Phrases such as aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to the other aforementioned phrases.

[0074] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" or as an "example" is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the terms "including," "having," and the like are used in the specification or claims, such terms are intended to be inclusive, in a manner similar to the way the term "comprising" is interpreted when used as a transitional word in a claim.

[0075] All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be made available to the public regardless of whether the disclosure is explicitly stated in the claims. No claim element shall be interpreted under the provisions of 35 U.S.C. §112(f) unless the element is explicitly stated using the phrase “means for…” or, in the case of a method claim, the element is stated using the phrase “step for…”

[0076] The previous description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications of these aspects are apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are intended to make the full scope consistent with the language claims, wherein reference to elements in singular values ​​is not intended to mean "only one", but refers to "one or more", unless specifically noted. Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are used only for convenience and do not limit the subject disclosure.

Claims

1. A method, comprising: establishing, by an operating system process of a local device associated with a local user, a system communication channel between a local device associated with the local user and a remote device associated with a remote user; providing, by the operating system process, to the remote device through the system communication channel, an invitation for the remote user to share the first application and the second application with the local user; as well as After receiving an acceptance of the invitation, first state data of the first application and second state data of the second application are provided by the operating system process to the remote device.

2. The method according to claim 1, wherein: Providing the first state data of the first application and the second state data of the second application to the remote device includes: For the operating system process, the first application program, and the second application program, the first state data and the second state data are obtained from a shared memory at the local device.

3. The method according to claim 2, wherein: The first state data and the second state data have been written into the shared memory by the first application and the second application, respectively.

4. The method according to claim 1, wherein: Providing the first state data of the first application and the second state data of the second application to the remote device includes: In response to a first system call from the first application to the operating system process to share the first state data, providing the first state data to the remote device through the system communication channel; and In response to a second system call from the first application to the operating system process to share the second state data, the second state data is provided to the remote device through the system communication channel.

5. The method according to claim 1, wherein: Establishing the system communication channel includes establishing the system communication channel for a group communication session between the local user and the remote user.

6. The method according to claim 5, wherein: Establishing the system communication channel for the group communication session of the local user and the remote user includes establishing the group communication session in response to a request to the operating system process to establish the group communication session, the request including a first identifier of the local user and a second identifier of the remote user.

7. The method of claim 6, further comprising generating, by the operating system process, a group identifier for the group communication session.

8. The method according to claim 1, further comprising: receiving, at the operating system process, a request to share a third application with the remote user; sending, by the operating system process via the system communication channel, an additional invitation to the remote device to share the third application; as well as After receiving an acceptance of the additional invitation, third state data of the third application is provided by the operating system process to the remote device.

9. The method according to claim 1, further comprising: receiving, at the operating system process, a request to add an additional remote user to a group communication session corresponding to the system communication channel; sending, by the operating system process, additional invitations to additional remote devices of the additional remote users to join the group communication session and share the first application and the second application; as well as After receiving an acceptance of the additional invitation, the first state data of the first application and the second state data of the second application are provided by the operating system process to the additional remote device.

10. A method comprising: Writing state data by a first application at a local device to a shared system memory at the local device to update a state of a remote instance of the first application at a remote device, wherein the shared system memory is write-accessible to the first and second applications at the local device; as well as A request is provided from the first application at the local device to the operating system process at the local device via a system communication channel established with the remote device through the operating system process to share the state data of the first application in the shared system memory with the remote instance of the first application at the remote device.

11. The method according to claim 10, wherein: The request comprises a system call from the first application to the operating system process.

12. The method according to claim 10, wherein: The system communication channel has been established by the operating system process for a group communication session between a local user of the local device and a remote user of the remote device.

13. The method according to claim 12, wherein: The group communication session includes sharing, by the operating system process via the system communication channel, additional state data of the second application at the local device.

14. An electronic device, comprising: Memory; and One or more processors configured to: establishing, by an operating system process of the electronic device, a system communication channel between a remote device associated with a remote user and the electronic device, wherein the electronic device is associated with a local user; providing, by the operating system process, to the remote device through the system communication channel, an invitation for the remote user to share the first application and the second application with the local user; as well as After receiving an acceptance of the invitation, first state data of the first application and second state data of the second application are provided by the operating system process to the remote device through the system communication channel.

15. The electronic device according to claim 14, wherein: The one or more processors are configured to provide the first state data of the first application and the second state data of the second application to the remote device in part by: For the operating system process, the first application program, and the second application program, the first state data and the second state data are acquired from a shared memory at the electronic device.

16. The electronic device according to claim 14, wherein: The one or more processors are configured to establish the system communication channel for a group communication session between the local user and the remote user.

17. The electronic device according to claim 16, wherein: The one or more processors are configured to establish the system communication channel for the group communication session of the local user and the remote user in response to a request to the operating system process to establish the group communication session, the request including a first identifier of the local user and a second identifier of the remote user.

18. The electronic device of claim 17, the one or more processors further configured to generate, by the operating system process, a group identifier for the group communication session.

19. The electronic device according to claim 14, wherein: The one or more processors are further configured to: receiving, at the operating system process, a request to share a third application with the remote user; sending, by the operating system process through the system communication channel, an additional invitation to the remote device via the system communication channel to share the third application; as well as After receiving an acceptance of the additional invitation, third state data of the third application is provided by the operating system process to the remote device via the system communication channel.

20. The electronic device according to claim 14, wherein: The one or more processors are further configured to: receiving, at the operating system process, a request to add an additional remote user to a group communication session corresponding to the system communication channel; sending, by the operating system process, additional invitations to additional remote devices of the additional remote users to join the group communication session and share the first application and the second application; as well as After receiving an acceptance of the additional invitation, the first state data of the first application and the second state data of the second application are provided by the operating system process to the additional remote device.

Images