Application sharing across endpoint devices

Abstract

An endpoint framework is implemented for sharing one or more applications between one or more endpoint devices. To initiate application sharing, the system includes a first endpoint device running an application that generates notifications based on a state or status of the running application. The first endpoint device provides these notifications to the second endpoint device, which are output by the second endpoint device. The second endpoint device receives interactions with the output notifications and provides these interactions to the first endpoint device. In response to the interaction indicating the initiation of application sharing, the first endpoint device generates and provides an application stream associated with the running application to the second endpoint device.

Description

[Background technology]

[0001] background In a device ecosystem, multiple devices, each having respective functions and running respective applications associated with those functions, are networked to provide those functions to one or more users. Traditionally, to add new functions to a device ecosystem, new devices are added to the network that perform the new functions by running the respective applications when requested by the network. However, adding new devices to a device ecosystem can be cumbersome and restrictive due to the limited operating systems and communication protocols supported by the device ecosystem, such as in a closed device ecosystem. Also, continuously adding devices to the device ecosystem increases the complexity of managing the functions of each device in the device ecosystem.

[0002] Additionally, devices that are added to a device ecosystem are often tied to user data, limiting the device ecosystems that the device is allowed to participate in. Additionally, many device ecosystems require that each device only run applications associated with its functionality.

[0003] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference symbols in different drawings indicates similar or identical items. [Brief description of the drawings]

[0004] [Figure 1] 1 is a block diagram of an endpoint device framework for sharing one or more applications according to some embodiments. [Diagram 2]2 is a block diagram of an endpoint device configured to share one or more applications or receive one or more shared applications within the endpoint device framework of FIG. 1 in accordance with some embodiments. [Diagram 3] 4 is a signal flow diagram illustrating an example application sharing initiation operation for an endpoint device according to some embodiments. [Figure 4] 4 is a signal flow diagram illustrating an example application sharing operation for an endpoint device according to some embodiments. [Diagram 5] FIG. 1 is a flow diagram illustrating a process for sharing one or more applications across endpoint devices according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Overview of the embodiment The techniques and systems described herein are directed to providing flexibility in a device ecosystem of endpoint devices through sharing of applications across one or more of the endpoint devices. According to an example embodiment, a method can include generating a notification associated with an application running on a first endpoint device. Further, the method can include receiving a signal from a second endpoint device representing an interaction with the notification. Further, the method can include initiating an application stream based on the interaction with the notification and transmitting, by the first endpoint device, the application stream to the second endpoint device.

[0006] In an embodiment, the method may further include determining a state of an application running on the first endpoint device (e.g., the application is expired, the application is running, the application is suspended, the application has terminated). A notification associated with the application may be generated based on the state of the application. Additionally, the state may include an incoming phone call to the application. The method may further include determining an application control based on an interaction with the notification. An application stream may also be initiated based on the application control. Additionally, the application control may include at least one of starting an application stream, terminating an application, ignoring an application, an application command, or any combination thereof.

[0007] Further, the method may include rendering a plurality of application frames based on an application executing on the first endpoint device. The method may further include encoding the plurality of application frames to generate at least a portion of the application stream. In an embodiment, the application stream may include at least a portion of a video call. Further, the method may include rendering audio data based on an application executing on the first endpoint device and encoding the audio data to generate at least a portion of the application stream. Further, the application stream may include at least a portion of an incoming phone call.

[0008] In another example embodiment, the method can include sending a notification associated with an application executing on the first endpoint device to a second endpoint device. Additionally, the method can include receiving data from the second endpoint device indicating an interaction with the notification. The method can also include generating an application stream representing at least a portion of the application executing on the first endpoint device in response to the data indicating the interaction with the notification, and sending the application stream to the second endpoint device.

[0009] In an embodiment, the method may further include determining a state of an application executing on the first endpoint device (e.g., the application is expired, the application is running, the application is suspended, the application has terminated) and generating a notification associated with the application in response to determining the state of the application. The method may further include rendering audio data based on the application executing on the first endpoint device and encoding the audio data to generate an encoded audio stream. In an embodiment, the application stream may include an audio stream. The method may also include rendering frames based on the application executing on the first endpoint device and encoding the frames to generate an encoded video stream. In an embodiment, the application stream may include an encoded video stream.

[0010] Generally, the notification may include a prompt to accept the call. Additionally, the notification may include a prompt to activate an input device of the second endpoint device. The method may also include receiving, at the first endpoint device, data representative of an interaction with at least a portion of the application stream from the second endpoint device. Additionally, the method may also include modifying an application running on the first endpoint device based on the interaction with at least a portion of the application stream. Additionally, the method may include generating a modified application stream based on the interaction with at least a portion of the application stream and transmitting the modified application stream to the second endpoint device.

[0011] In an additional exemplary embodiment, the method can include outputting a notification to the first endpoint device associated with an application executing on the second endpoint device. Further, the method can include receiving an interaction with the notification and transmitting data representing the interaction with the notification to the first endpoint device. Further, the method can include receiving an application stream from the second endpoint device associated with the application executing on the second endpoint device.

[0012] In an embodiment, the notification may be output by an output device including at least one of a display or a speaker. Further, the method may include outputting at least a portion of the application stream by the output device. Also, the interaction may be received by an input device including a touch screen, a mouse, a keyboard, a remote control, a microphone, or any combination thereof.

[0013] Further, the method may include receiving an interaction with the application stream by an input device and transmitting data representative of the interaction with the application stream to a second endpoint device. The method may also include receiving a modified application stream from the second endpoint device based on the interaction with the application stream. Further, the method may include decoding at least a portion of the application stream to generate a decoded video stream. The method may also include decoding at least a portion of the application stream to generate a decoded video stream. In an embodiment, the notification may include a prompt to accept the call. Further, the application stream may include at least a portion of an incoming phone call.

[0014] In general, a first endpoint device may have a first operating system and a second endpoint device may have a second, different operating system that may have a different interface, language, memory management technique, communications protocol, encryption, network stack, or any combination thereof, than the first operating system.

[0015] In an example embodiment, the system may include a first endpoint device that includes one or more processors and stores executable instructions configured to operate the one or more processors to perform one or more of the methods disclosed herein.

[0016] In other example embodiments, a system may include an endpoint device framework including one or more endpoint devices communicatively coupled to one or more other endpoint devices, the system configured to perform one or more methods disclosed herein.

[0017] Detailed Description The techniques and systems described herein provide flexibility in a device ecosystem of endpoint devices through the sharing of applications across one or more of the endpoint devices. Specifically, such techniques and systems described herein enable an endpoint device to execute an application and share the application with one or more other endpoint devices such that the application is operated, partially operated, interactable, or partially simulated on the other endpoint devices. To initiate application sharing, a first endpoint device running the application generates notifications based on a state or status of the running application. These notifications are provided to and output by the second endpoint device. The second endpoint device receives interactions with the output notifications and provides these interactions to the first endpoint device. In response to the interaction indicating the initiation of application sharing, the first endpoint device generates an application stream related to the running application and provides it to the second endpoint device. In this manner, application sharing is initiated between the endpoint devices through notifications related to the application being shared between the devices.

[0018] Further, to share an application between the endpoint devices, the first endpoint device generates an application stream based on the application running on the first endpoint device and provides it to the second endpoint device. The second endpoint device outputs the application stream and receives interactions with the output application stream. The second endpoint device provides the interactions with the application stream to the first endpoint device. The first endpoint device then modifies the application running on the first endpoint device based on the interactions with the application stream and generates a modified application stream based on the modified application. In this manner, the application running on the first endpoint device is shared with the second endpoint device such that the application is operated, partially operated, interactable, or partially simulated on the second endpoint device.

[0019] Additionally, the techniques and systems described herein enable operating system agnostic sharing of system applications between endpoint devices, i.e., an endpoint device having a first operating system shares applications with one or more endpoint devices having one or more different operating systems. Such a framework allows for increased flexibility in sharing applications within a device ecosystem.

[0020] 1 illustrates an endpoint device framework 100 for cross-device application sharing, according to some embodiments. The endpoint device framework 100 includes one or more shared endpoint devices 102, each communicatively coupled to at least one other endpoint device 110, and is configured to enable sharing of one or more applications between the one or more shared endpoint devices 102 and one or more other endpoint devices 110. Such applications include, for example, telecommunication applications (e.g., incoming phone calls, video calls, Voice over Internet Protocol (VoIP) applications), messaging applications, media presentation applications, virtual assistant applications, gaming applications, shopping applications, encryption applications, decryption applications, web browser applications, and navigation applications, to name a few. Communicatively coupled includes, for example, two or more devices, each configured to enable transmission of data, reception of data, or both, between them, via wired communication, wireless communication, network communication, bridge communication, etc., to name a few. As used herein, a "shared endpoint device" and an "endpoint device" each include hardware and software resources configured to run one or more applications, such as one or more desktop computers, servers, virtual servers, notebook computers, tablet computers, virtual reality systems, augmented reality systems, computer-enabled mobile phones (i.e., "smartphones"), computer-enabled wearable devices (i.e., "wearables"), communications hubs, vehicle integrated displays, computer-enabled displays (i.e., "smart displays"), computer-enabled televisions (i.e., "smart TVs"), computer-enabled glasses, computer-enabled speakers, computer-enabled cameras, computer-enabled alarms, drones, or any combination thereof.

[0021] In an embodiment, one or more shared endpoint devices 102 are configured to execute and share one or more applications with one or more other communicatively coupled endpoint devices 110 such that the shared application is operated, partially operated, interactable, or partially simulated on the endpoint device 110. For example, in the exemplary embodiment shown in FIG. 1, the sharing, i.e., first endpoint device 102 is configured to share one or more applications with the second endpoint device 110 such that the application is operated, partially operated, interactable, or partially simulated on the second endpoint device 110. To facilitate the sharing of applications to another endpoint device 110, the shared endpoint device 102 is configured to execute a corresponding application instance 104 for each application to be shared. The application instance 104 includes software and hardware for facilitating the execution of at least a portion of the application to be shared to another endpoint device 110. Such facilitation may include, for example, performing one or more operations of the application, such as rendering video frames, rendering audio data, controlling data streams, call processing, message processing, network management, communication management, authentication processing, notification generation, or any combination thereof, to name a few. According to an embodiment, data resulting from operations performed by application instance 104 is provided as application stream 108 to one or more communicatively coupled endpoint devices 110. As used herein, "application stream" includes one or more data streams that enable an application to be operated, partially operated, interactable, or partially simulated on one or more other endpoint devices.Such data streams may include, for example, a video stream, an audio stream, a control data stream, an encrypted data stream, or any combination thereof, to name a few. For example, the shared endpoint device 102 provides an application stream 108 to the endpoint device 110, which includes an encoded video stream and an encoded audio stream of a video call.

[0022] According to an embodiment, the one or more shared endpoint devices 102 each include a notification engine 106 configured to generate one or more application notifications related to applications running on the shared endpoint device 102. The application notifications generated by the notification engine 106 include, for example, a user prompt (e.g., a prompt to enter information, a prompt to enable a service, a prompt to activate hardware, a prompt to select, a prompt to confirm), an application status (e.g., data received by the application, application authorization, application activation, application deactivation), an application output (e.g., data generated by the application, a value calculated by the application, video data, audio data), or any combination thereof. In an embodiment, the application notifications generated by the notification engine 106 are configured to be output visually, audibly, or tactilely by the endpoint device. For example, the application notifications generated by the notification engine 106 are configured to be displayed on a screen. As another example, the application notifications generated by the notification engine 106 are configured to induce vibration of the endpoint device. According to an embodiment, the application notifications generated by the notification engine 106 include one or more push notifications. In an embodiment, the shared endpoint device 102 is configured to send one or more application notifications generated by the notification engine 106 to the communicatively coupled endpoint device 110. For example, the shared endpoint device 102 is configured to send an application notification to the endpoint device 110 indicating that a call has been received by an application running on the shared endpoint device 102. As another example, the shared endpoint device 102 is configured to send an application notification to the endpoint device 110 prompting a user to turn on the camera of the endpoint device 110.According to an embodiment, the shared endpoint device 102 is configured to output one or more notifications generated by the notification engine 106 of the shared endpoint device 102 and to receive one or more interactions with the output notifications using one or more input devices.

[0023] An endpoint device 110 that receives an application stream 108 or an application notification from a shared endpoint device 102 includes an output device 112 configured to output at least a portion of the received application stream 108 or the received application notification. The output device may include, for example, one or more displays (e.g., a cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), an organic light emitting diode (OLED), and a plasma display), audio devices (e.g., a full-range driver, a mid-range driver, a coaxial driver, a loudspeaker, a tweeter, a woofer, a subwoofer, and a crossover), and a haptic output device (e.g., a kinesthetic communication device, a haptic feedback device, or a vibration motor). As an example, the endpoint device 110 includes a display configured to output a video stream of the received application stream 108. As another example, the endpoint device 110 includes a speaker configured to output an audio stream of an incoming phone call from the received application stream 108. As yet another example, the endpoint device includes a display configured to output a notification including a user prompt to activate the camera of the endpoint device 110.

[0024] In an embodiment, the endpoint device 110 includes an input device 114 configured to receive one or more interactions. The input device may include one or more devices configured to receive user interactions, such as, for example, a touch screen, a keyboard, a remote control, a virtual keyboard, a mouse, a joystick, a controller pad, a microphone, an accelerometer, a proximity sensor, a camera, etc. The interactions may include, for example, gestures on a touch screen, mouse clicks, button presses, wheel clicks, keyboard strokes, speech into a microphone, movements of the endpoint device, hand gestures, or any combination thereof, to name a few. In an embodiment, the endpoint device 110 that outputs at least a portion of the received application stream 108 is configured to receive one or more interactions related to the portion of the output application stream 108. That is, the endpoint device 110 is configured to receive interactions with an application represented by the output received application stream 108. As an example, the endpoint device 110 receives an application stream 108 that is running on the shared endpoint device 102 and represents an application that includes a video stream. While the endpoint device 110 outputs a video stream of the application stream 108, the endpoint device 110 receives interactions related to the output video stream, such as gestures on a touch screen, mouse clicks, remote control button presses (e.g., interactions with buttons on a remote control), or any combination thereof. Similarly, an endpoint device 110 that outputs a receipt notification is configured to receive one or more interactions related to the output notification. In other words, the endpoint device 110 receives interactions with the notification being output by the endpoint device 110.As an example, an endpoint device 110 that displays a notification including a user prompt receives an interaction related to the user prompt, such as a gesture on a touch screen, a mouse click, a press of a remote control button (e.g., an interaction with a button on a remote control), or any combination thereof.

[0025] The endpoint device 110 is configured to send one or more application interactions 116 related to the application to the shared endpoint device 102 that shares the application. "Application interactions," as used herein, include data representing one or more interactions with a portion of the application stream 108 output by the endpoint device 110. According to an embodiment, the application instance 104 is configured to modify the execution of the shared application running on the application instance 104 based on the received application interactions 116. For example, the application instance 104 provides the application interactions 116 as one or more inputs to the shared application, determines one or more inputs for the shared application based on the application interactions, terminates the execution of the shared application, launches another application, runs another portion of the shared application, or any combination thereof. In this manner, the application running on the shared endpoint device 102 is operated, partially operated, interactable, or partially simulated on the endpoint device 110. As an example, application instance 104 on shared endpoint device 102 renders audio and video streams of a video call and provides the audio and video streams as application stream 108 to endpoint device 110. Endpoint device 110 outputs the audio and video of the video stream and receives audio user input via a microphone. Endpoint device 110 then transmits the user input as application interactions 116 to shared endpoint device 102 which provides application stream 108. Application instance 104 then modifies the shared application based on the received user input, for example, by providing audio of the user input to another device associated with the video call.

[0026] According to an embodiment, the endpoint device 110 is configured to send one or more notification interactions to the shared endpoint device 102, e.g., the shared endpoint device that generated the application notification. A "notification interaction," as used herein, includes data representing one or more interactions with a notification output by the endpoint device 110. In an embodiment, the application instance 104 of the shared endpoint device 102 is configured to initiate an application sharing session based on the one or more notification interactions. Initiating the application sharing session includes, for example, initiating an application stream, providing the notification interaction as an input to the application, determining an input to the application, launching the application, executing at least a portion of the application, generating one or more subsequent application notifications, or any combination thereof. For example, the shared endpoint device 102 generates a notification prompting the user to answer or reject an incoming call and sends the notification to the endpoint device 110. The endpoint device 110 displays the notification, receives the user's interaction with the prompt as a gesture on a touch screen, and sends the notification interaction to the shared endpoint device 102. In response to receiving the notification interaction, the application instance 104 initiates application sharing by initiating and sending an application stream 108 to the endpoint device 110 .

[0027] In an embodiment, each shared endpoint device 102 and endpoint device 110 of the endpoint device framework 100 includes one or more operating systems, each having, for example, one or more interfaces, languages, memory management techniques, communication protocols, encryption, network stacks, or any combination thereof, to name a few. According to an embodiment, the endpoint device framework 100 is configured to be operating system agnostic. "Operating system agnostic," as used herein, includes an endpoint device framework 100 configured to support at least one shared endpoint device 102 having a first operating system and at least one other endpoint device 110 having a second, distinct operating system. That is, within the endpoint device framework 100, a shared endpoint device 102 having a first operating system shares one or more applications with an endpoint device 110 having a second, distinct operating system that includes one or more interfaces, memory management techniques, communication protocols, network stacks, or any combination thereof that are different from the first operating system. For example, the shared endpoint device 102 includes a first operating system having a first interface and a first memory management system. Within the endpoint device framework 100, the shared endpoint device 102 shares one or more applications with an endpoint device 110 having a second different operating system with a second different interface and a second different memory management system. By initiating application sharing using one or more notifications sent between the endpoint devices within the endpoint device framework 100, the endpoint device framework 100 is operating system agnostic.For example, using one or more standard communication protocols (e.g., Bluetooth, Wi-Fi, LTE), the shared endpoint device 102 sends one or more notifications to another endpoint device 110 configured to interact with the notifications. Once data representing these interactions is sent to the shared endpoint device via the standard communication protocol, the shared endpoint device 102 determines one or more responses to the interactions, such as, for example, initiating an application stream 108, sending the application stream to the other endpoint device using the standard communication protocol, etc. In this manner, the shared endpoint device 102 within the endpoint device framework 100 shares one or more applications with one or more other endpoint devices in an operating system independent manner.

[0028] 2, an exemplary endpoint device 202 configured to share one or more applications is shown. In an embodiment, the endpoint device 202 implements aspects of the endpoint device framework 100 as described in FIG. 1. For example, the endpoint device 202 may be similar to or the same as the shared endpoint device 102 or the endpoint device 110 described in FIG. 1. According to some embodiments, the endpoint device 202 is part of an electronic device that supports application sharing, including, for example, a desktop computer, a notebook computer, a drone, a tablet computer, a virtual reality system, an augmented reality system, a smartphone, a communications hub, a smart display, or a smart television, to name a few.

[0029] In an embodiment, the endpoint device 202 includes a processing element 240 communicatively coupled to a memory 232. The memory 232 includes an electronic storage device, such as a solid state drive, a hard disk drive, a random access memory ("RAM"), a read only memory ("ROM"), an electronically erasable programmable ROM ("EEPROM"), an optical storage device, or any combination thereof. The memory 232 includes instructions and data related to the execution of one or more applications and the sharing of one or more applications, such as device identification data, codecs, messaging protocols, encryption data, decryption data, reference frames, or any combination thereof, to name a few.

[0030] According to an embodiment, the processing element 240 includes one or more control processing units (CPUs), microprocessors, field programmable gate arrays (FPGAs), graphics processing units (GPUs), application specific integrated circuits (ASICs), or any combination thereof, and is configured to import one or more system services to and export those services from one or more other endpoint devices. In an embodiment, the endpoint device 202 includes an application instance 204 similar or identical to the application instance 104, and includes hardware and software configured to execute at least a portion of one or more applications 242. According to an embodiment, the application instance 204 is configured to render one or more application frames 218 based on the one or more applications 242 executed by the application instance 204. In an embodiment, each rendered application frame 218 represents at least a portion of a visual environment associated with the application 242, such as a video call, a movie scene, a television scene, a heads-up display (HUD), a graphical user interface (GUI), or any combination thereof, to name a few. In an embodiment, application instance 204 is configured to render application audio 220 based on one or more applications executed by application instance 204. According to an embodiment, the rendered application audio 220 represents sound, music, or dialogue associated with application 242, such as, for example, audio from a telephone call, audio from a video call, a song, sound effects, or any combination thereof, to name a few.

[0031] In an embodiment, the processing element 240 includes one or more encoders 228, including hardware and software configured to encode the application frames 218 and application audio 220 rendered by the application instance 204 to generate respective video and audio streams. The encoders 228 are configured to encode the application frames 218 and application audio 220, for example, through compression, reorganization, and manipulation of the application frames 218 and application audio 220. According to an embodiment, each encoder 228 implements one or more codecs to encode the application frames 218 and application audio 220. In an embodiment, the video and audio streams generated by the encoders 228 are transmitted to another endpoint device as at least a portion of the application stream 108. In this manner, the video and audio of each application 242 is rendered on the endpoint device 202 and shared with another endpoint device, allowing each application 242 to be manipulated, partially manipulated, interactable, or partially simulated on the other endpoint device.

[0032] According to an embodiment, the endpoint device 202 includes or is communicatively coupled to one or more output devices 238, similar or identical to the output device 112, configured to output at least a portion of the received application stream 108, such as a display that outputs a video stream of the application stream 108, a speaker that outputs an audio stream of the application stream 108, or a touch screen that outputs a GUI, to name a few. In an embodiment, the endpoint device 202 includes or is communicatively coupled to one or more input devices 236, similar or identical to the input device 114, configured to receive interactions with the portion of the application stream 108 output by the output device 238. That is, the input device 236 is configured to receive interactions with one or more applications 242 shared with the endpoint device 202. For example, a touch screen for receiving interactions with the application stream 108 output on a display, a microphone for receiving interactions with the phone application output, or remote control buttons (e.g., buttons on a remote control) for receiving interactions related to the audio stream output by a speaker, to name a few. Although the exemplary embodiment shown in Figure 2 presents endpoint device 202 including or communicatively coupled to three input devices (236-1, 236-2, and 236-3) and three output devices (238-1, 238-2, and 238-3), in other embodiments endpoint device 202 may include or be communicatively coupled to any number of input devices 236 and any number of output devices 238.

[0033] In an embodiment, the endpoint device 202 includes one or more decoders 230, each including hardware and software configured to decode one or more portions of the application stream 108. For example, the decoders 230 are configured to decode the encoded video and audio streams of the application stream 108. The decoders 230 are configured to decode application frames and application audio, for example, through compression, reorganization, and manipulation of the application stream 108. In an embodiment, each decoder 230 implements one or more codecs to decode the application frames and application audio from the application stream 108. The decoders 230 are configured to provide any decoded application frames, video streams, and application audio data from the application stream 108 to one or more output devices 238 for output.

[0034] The endpoint device 202 includes a notification engine 206 similar or identical to the notification engine 106, which includes hardware and software configured to generate one or more application notifications 224 related to one or more applications 242 executed by the application instance 204. The application notifications 224 include, for example, a user prompt (e.g., a prompt to enter information, a prompt to enable a service, a prompt to activate hardware, a prompt to select, a prompt to confirm), an application status (e.g., data being received by the application, authorization of the application, activation of the application, deactivation of the application), an application output (e.g., data generated by the application, values ​​calculated by the application, video data, audio data), or any combination thereof. In an embodiment, the application notifications 224 are configured to be output visually, audibly, or tacitly by one or more output devices. For example, the application notifications 224 are configured to be displayed on an output device 238 including a screen. According to an embodiment, the notification engine 206 is configured to generate one or more application notifications 224 based on the state of the applications 242 executed on the application instance 204. As used herein, an "application state" or "application state" may refer to, for example, the status of an application 242 (e.g., application is stale, application is running, application is suspended, application has terminated), data received by an application 242 (e.g., incoming phone call, incoming video call, update data, file, video stream, audio stream), the progress of one or more operations of an application 242 (e.g., completion of an operation, output of an operation, start of an operation), or any combination thereof.Based on the state of the application (e.g., application state), the notification engine 206 is configured to determine and generate one or more application notifications 224. In an embodiment, the notification engine 206 is configured to generate the application notification 224 based on the state of the application 242, for example, using a lookup table, a relational database, or both. In an embodiment, the notification engine 206 is configured to generate one or more application notifications 224 to initiate application sharing with another endpoint device of the application 242 running on the application instance 204. That is, the notification engine 206 is configured to determine one or more user prompts, application status, or application outputs that are necessary, helpful, or assistive to sharing the application 242 with another endpoint device. For example, the notification engine 206 generates a notification that includes a user prompt to activate a camera of the other endpoint device to share a video calling application with the other endpoint device. As another example, the notification engine 206 generates a notification that includes an available video to share a video streaming application with the other endpoint device. In an embodiment, the endpoint device 202 is configured to send the one or more application notifications 224 to one or more other endpoint devices. According to an embodiment, the endpoint device 202 is configured to output one or more notifications generated by the endpoint device's notification engine 206 using one or more output devices 238, and to receive one or more interactions with the output notifications using one or more input devices 236. Based on the received interactions, the application instance 204 is configured to modify one or more applications 242 running on the endpoint device 202.

[0035] According to an embodiment, the endpoint device 202 receives one or more application notifications 224 related to an application 242 running on at least one other endpoint device. The endpoint device 202 is configured to output the received application notifications to one or more output devices 238. For example, the endpoint device 202 is configured to display the application notification 224 including a user prompt to activate a microphone of the endpoint device on the output device 238 including a touch screen. In an embodiment, the endpoint device 202 is configured to receive one or more notification interactions 226 with the application notification 224 output by the one or more output devices 238 via the one or more input devices 236. For example, the endpoint device 202 is configured to receive the notification interaction 226 including one or more mouse clicks while the application notification 224 including the user prompt is displayed. As another example, the endpoint device 202 is configured to receive the notification interaction 226 including one or more keyboard strokes while the application notification 224 including an application status is displayed. According to an embodiment, the endpoint device 202 sends one or more notification interactions 226 to one or more other endpoint devices. For example, the endpoint device 202 sends one or more notification interactions 226 to an endpoint device that is running an application 242 related to the notification interaction 226 (i.e., the application 242 to which the application notification 224 received by the endpoint device 202 is related).

[0036] In an embodiment, the notification engine 206 receives one or more notification interactions 226 from one or more other endpoint devices. Based on the received notification interactions 226, the notification engine 206 initiates sharing of one or more applications 242 running on the application instance 204. For example, the notification engine 206 compares the one or more received notification interactions 226 to one or more strings, lookup tables, or databases. In response to a comparison indicating the initiation of application sharing, the notification engine 206 initiates sharing of the one or more applications 242. The notification engine 206 initiates sharing of the one or more applications 242 by, for example, initiating an application stream 108, providing the notification interaction 226 as an input to the application 242, determining an input to the application 242, launching the application 242, executing at least a portion of the application 242, generating one or more subsequent application notifications 224, or any combination thereof. In this manner, the application notifications 224 are used to initiate sharing of one or more applications 242 between two or more endpoint devices.

[0037] 3 presents an exemplary application sharing initiation operation 300. In an embodiment, a first endpoint device 302 similar or identical to the shared endpoint device 102 is configured to share one or more applications running on an application instance 304 similar or identical to the application instance 104 with a second endpoint device 310 similar or identical to the endpoint device 110. The first endpoint device 302 also includes a notification engine 306 similar or identical to the notification engine 106 and is configured to generate one or more application notifications related to the applications running on the application instance 304. The initiation operation 300 includes the notification engine 306 of the first endpoint device 302 determining an application state 301 of one or more applications running on the application instance 304. The application state 301 indicates a status of the application running on the application instance 304, data received by the application running on the application instance 304, progress of one or more operations of the application running on the application instance 304, or any combination thereof. Based on the determined application state 301, the notification engine 306 is configured to generate one or more notifications 303 that are necessary, useful, or aiding in sharing the application running on the application instance 304 with the second endpoint device 310. For example, the notification engine 306 compares the determined application state 301 to one or more strings, lookup tables, databases, or any combination thereof to determine and generate one or more notifications 303 that are necessary, useful, or aiding in sharing the application running on the application instance 304 with the second endpoint device 310.As an example, based on the determined application state 301 of the incoming phone call, the notification engine 306 generates a notification 303 that includes a user prompt to accept or decline the call.

[0038] The first endpoint device 302 is configured to transmit the generated notification 303 to the second endpoint device 310. The second endpoint device 310 is configured to output any visual, audio, or tactile data of the notification 303 to one or more output devices. While the notification 303 is output by the second endpoint device 310, the second endpoint device 310 is configured to receive one or more interactions with the output notification 303 via one or more input devices. That is, the second endpoint device 310 is configured to receive one or more notification interactions 305. For example, the second endpoint device 310 receives one or more button presses, gestures on a touch screen, audio input to a microphone, mouse clicks, keystrokes, or any combination thereof, that interact with the output notification 303. The second endpoint device 310 is configured to transmit data representing the one or more received notification interactions 305 to the first endpoint device 302.

[0039] Based on the data representing the one or more received notification interactions 305 received from the second endpoint device 310, the notification engine 306 of the first endpoint device 302 determines whether application sharing should be initiated, i.e., whether an application running on the application instance 304 should be shared with the second endpoint device 310. For example, the notification engine 306 compares the data representing the one or more received notification interactions 305 to one or more strings, lookup tables, databases, or any combination thereof, and determines whether the comparison indicates that the application running on the application instance 304 should be shared with the second endpoint device 310. In response to determining that the one or more applications running on the application instance 304 should be shared with the second endpoint device 310, the notification engine 306 generates one or more application controls 307. The application control 307 includes one or more inputs, commands, messages, or any combination thereof that instruct the application instance 304 to initiate an application stream, provide a notification interaction 305 as input to an application running on the application instance 304, ignore an application running on the application instance 304, provide input to an application running on the application instance 304, launch an application, execute at least a portion of an application, or any combination thereof. In this manner, the notification 303 generated by the notification engine 306 is used to initiate an application sharing session between the endpoint devices.

[0040] 4 presents an exemplary application sharing operation 400. In an embodiment, a first endpoint device 402, similar or identical to the shared endpoint device 102, is configured to share one or more applications running on the first endpoint device 402 with a second endpoint device 410, similar or identical to the endpoint device 110. In an embodiment, the sharing operation 400 includes the first endpoint device 402 generating an application stream 401. The application stream 401 includes one or more encoded video streams, encoded audio streams, control data streams, encrypted data streams, or any combination thereof, associated with one or more applications running on the first endpoint device 402. The first endpoint device 402 transmits the application stream 401 to the second endpoint device 410.

[0041] The second endpoint device 410 includes one or more output devices configured to output at least a portion of the received application stream 401. For example, the second endpoint device 410 includes a display that outputs an encoded video stream of the application stream 401 and a speaker that outputs an encoded audio stream of the application stream 401. While the application stream 401 is being output by the second endpoint device 410, the second endpoint device 410 receives interactions on the output portion of the application stream 401 via one or more input devices. That is, the second endpoint device 410 receives one or more interactions with an application associated with the application stream 401 (e.g., application interactions 403). As an example, the second endpoint device 410 receives keyboard strokes that interact with the output video stream of the application stream 401 and touch screen gestures that interact with the GUI of the application stream 401. The second endpoint device 410 transmits data representing one or more received application interactions 403 to the first endpoint device 402 (eg, the endpoint device that transmitted the application stream 401).

[0042] Based on the received data representing the one or more received application interactions 403 from the second endpoint device 410, the first endpoint device 402 modifies one or more applications running on the first endpoint device 402. The first endpoint device 402 modifies the applications, for example, by providing the data representing the one or more received application interactions 403 to the one or more applications, starting the application, terminating the application, determining inputs for the application, or any combination thereof. In response to modifying the one or more applications running on the first endpoint device 402, a modified application stream 405 is generated based on the modified application. The modified application stream 405 is then provided to the second endpoint device. In this manner, one or more applications running on the first endpoint device 402 are shared with the second endpoint device 410 such that the one or more applications running on the first endpoint device 402 are operated, partially operated, interactable, or partially simulated on the second endpoint device 410.

[0043] 5, a flow diagram of an exemplary application sharing operation 500 is shown. In block 505, a notification engine similar or identical to the notification engine 106 of a first endpoint device similar or identical to the shared endpoint device 102 determines a state of one or more applications running on the first endpoint device, e.g., indicating a status of the application, data received by the application, progress of one or more operations of the application, or any combination thereof. In block 510, the notification engine determines whether a condition for notification is met. For example, the notification engine compares the determined state of the application to one or more strings, lookup tables, databases, or any combination thereof, and determines whether the comparison indicates that a condition for notification is met. In response to the condition for notification not being met, the system returns to block 505. In response to the condition for notification being met, the system moves to block 515.

[0044] In block 515, the notification engine generates one or more notifications associated with one or more applications running on the first endpoint device. As an example, the notification engine generates the one or more notifications based on a comparison of the determined state of the application with one or more strings, lookup tables, databases, or any combination thereof. Furthermore, the notification engine transmits the generated notifications to a second endpoint device similar or identical to the endpoint device 110. In block 520, the second endpoint device outputs the one or more received notifications using one or more output devices. Furthermore, in block 520, the second endpoint device receives one or more notification interactions with the one or more output notifications via one or more input devices. The second endpoint device transmits data representing the notification interactions to the first endpoint device.

[0045] In block 525, the first endpoint device determines whether one or more notification interactions represented in the received data indicate the start of application sharing. For example, the first endpoint device compares the data representing the notification interactions with one or more strings, lookup tables, databases, or any combination thereof, and determines whether the comparison indicates that application sharing is to be started. In response to there being no interaction indicating the start of application sharing, the system returns to block 505. In response to one or more notifications indicating that application sharing is to be started, the system moves to block 530. In block 530, the first endpoint device initiates an application stream representing at least a portion of one or more applications running on the first endpoint device. The application stream is then provided to the second endpoint device. In other words, the first endpoint device shares one or more applications with the second endpoint device.

[0046] In block 535, the second endpoint device outputs at least a portion of the received application stream using one or more output devices. While the portion of the application stream is being output, the second endpoint device is configured to receive one or more application interactions with the portion of the application stream being output via one or more input devices. That is, the second endpoint device receives one or more application interactions with the shared application from the first endpoint device. The second endpoint device then transmits data representing the application interactions to the first endpoint device. In block 540, the first endpoint device modifies one or more applications running on the first endpoint device based on the application interactions. For example, the first endpoint device compares the one or more application interactions with one or more strings, lookup tables, databases, or any combination thereof, and determines a modification from the comparison. Such modification may include, for example, providing data representing one or more application interactions, starting an application, terminating an application, determining an input for the application, or any combination thereof. From the modified application running on the first endpoint device, the first endpoint device generates a modified application stream and transmits it to the second endpoint device.

[0047] A computer-readable storage medium may include any storage medium, or combination of storage media, that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical system (MEMS)-based storage media. A computer-readable storage medium may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., magnetic hard drives), removably attached to a computing system (e.g., optical disks or universal serial bus (USB)-based flash memory), or coupled to a computer system via a wired or wireless network (e.g., network-accessible storage (NAS)).

[0048] It should be noted that not all of the activities or elements described above in the general description are required, some of the specific activities or devices may not be required, and one or more additional activities may be performed or elements may be included in addition to those described. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will appreciate that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the following claims. Thus, the present specification and figures should be regarded in an illustrative rather than restrictive sense, and all such modifications are intended to be within the scope of the present disclosure.

[0049] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature(s) that may cause or make any benefit, advantage, or solution more pronounced are not to be construed as key, essential, or essential features of any or all claims. Moreover, the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein, and the specific embodiments disclosed above are illustrative only. No limitations are intended to the details of construction or design shown herein, other than as set forth in the following claims. It is therefore apparent that the specific embodiments disclosed above are capable of alteration or modification, and all such variations are considered within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the following claims.

Claims

1. generating a notification associated with an application running on the first endpoint device; receiving a signal from a second endpoint device representing an interaction with the notification; Initiating an application stream based on the interaction with the notification; transmitting, by the first endpoint device, the application stream to the second endpoint device.

2. determining a state of the application running on the first endpoint device; The method of claim 1 , wherein the notification associated with the application is generated based on the state of the application.

3. The method of claim 2 , wherein the state includes an incoming call.

4. determining application control based on the interaction with the notification; The method according to any one of claims 1 to 3, wherein the application stream is further initiated based on the application control.

5. The method of claim 4 , wherein the application control includes at least one of: starting an application stream, ending an application, ignoring an application, or an application command.

6. Rendering a plurality of application frames based on the application running on the first endpoint device; The method of any one of claims 1 to 3, further comprising: encoding the plurality of application frames to generate at least a portion of the application stream.

7. The method of claim 6 , wherein the application stream comprises at least a portion of a video call.

8. Rendering audio data based on the application running on the first endpoint device; and The method of any one of claims 1 to 3, further comprising: encoding the audio data to generate at least a portion of the application stream.

9. The method of claim 8 , wherein the application stream comprises at least a portion of an incoming call.

10. sending a notification to a second endpoint device related to an application running on the first endpoint device; receiving data from the second endpoint device indicating an interaction with the notification; generating an application stream representing at least a portion of the application running on the first endpoint device in response to the data indicative of the interaction with the notification; transmitting the application stream to the second endpoint device.

11. determining a state of the application running on the first endpoint device; 11. The method of claim 10, further comprising: generating the notification associated with the application in response to determining the state of the application.

12. Rendering audio data based on the application running on the first endpoint device; and The method of claim 10 or claim 11, further comprising: encoding the audio data to generate an encoded audio stream.

13. The method of claim 12 , wherein the application stream comprises the encoded audio stream.

14. Rendering a frame based on the application running on the first endpoint device; The method of claim 10 or claim 11, further comprising: encoding the frames to generate an encoded video stream.

15. The method of claim 14 , wherein the application stream comprises the encoded video stream.

16. The method of any of claims 1 to 3, wherein the notification includes a prompt to accept the call.

17. The method of any of claims 1 to 3, wherein the notification includes a prompt that activates an input device of the second endpoint device.

18. The method of any of claims 1 to 3, further comprising receiving, at the first endpoint device, data representing an interaction with at least a portion of the application stream from the second endpoint device.

19. 20. The method of claim 18, further comprising modifying the application running on the first endpoint device based on the interaction with at least a portion of the application stream.

20. generating a modified application stream based on the interaction with at least a portion of the application stream; 20. The method of claim 18, further comprising: transmitting the modified application stream to the second endpoint device.

21. outputting a notification to the first endpoint device related to an application running on the second endpoint device; receiving an interaction with the notification; and sending data representing the interaction with the notification to the first endpoint device; receiving an application stream from the second endpoint device associated with the application running on the second endpoint device.

22. The method of claim 21 , wherein the notification is output by an output device including at least one of a display or a speaker.

23. 23. The method of claim 22, further comprising outputting at least a portion of the application stream by the output device.

24. The method of any of claims 21 to 23, wherein the interaction is received by an input device including at least one of a touch screen, a mouse, a keyboard, a remote control, or a microphone.

25. receiving an interaction with the application stream by the input device; 25. The method of claim 24, further comprising: transmitting data representing the interaction with the application stream to the second endpoint device.

26. 26. The method of claim 25, further comprising receiving from the second endpoint device a modified application stream based on the interaction with the application stream.

27. The method of any of claims 21 to 23, further comprising decoding at least a portion of the application stream to generate a decoded video stream.

28. The method of any of claims 21 to 23, further comprising decoding at least a portion of the application stream to generate a decoded audio stream.

29. The method of any of claims 21 to 23, wherein the notification includes a prompt to accept the call.

30. The method according to any one of claims 21 to 23, wherein the application stream comprises at least a part of an incoming call.

31. The method of any of claims 21 to 23, wherein the first endpoint device has a first operating system and the second endpoint device has a second, different operating system.

32. 32. The method of claim 31, wherein the second, different operating system has at least one interface, language, memory management technique, communication protocol, encryption, or network stack that is different from the first operating system.

33. A system comprising: a first endpoint device including one or more processors and storing executable instructions configured to operate the one or more processors to perform the method of any one of claims 1 to 3.

34. 1. A system including an endpoint device framework including one or more endpoint devices communicatively coupled to one or more other endpoint devices, A system configured to perform the method according to any one of claims 1 to 3.

Images