Streaming of External Applications

Abstract

The present technology is to pair a first client device (102, 602) with a second client device (302, 604) via at least one wireless connection, enabling the client devices to communicate directly and generating an encoded virtual remote display of an application by one or more processors of the second client device (402) to enable communication via at least one wireless connection, where the application is running on the second client computing device (302, 604), and pairing; and presenting, on a graphical user interface (GUI) (106, 1200, 1900, 2300, 2500), a virtual remote display (1502, 1902, 2302) of an application on the first client device (102, 602) based on the encoded virtual remote display to enable a user of the first client device (102, 602) to control an application running on the second client device (302, 604), including streaming an application on the first client device (102, 602).

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of the filing date and priority of U.S. Provisional Application No. 63 / 340,226, filed May 10, 2022, the disclosure of which is hereby incorporated by reference in its entirety.

Background Art

[0002] In a system, multiple devices can be network - connected to each other to enable a user to access the functions of different devices throughout the system. Such access may be limited in functionality because there is no compatibility between different types of devices. For example, a program designed to operate on a certain device or platform may not operate smoothly on a different device or platform. Even if a program is executable on multiple platforms, the technical problem may be that some functions are lost due to the incompatible user interfaces.

Summary of the Invention

[0003] Aspects of the present technology provide a technical solution that enables streaming of an application from one computing device to another computing device. By way of example, this includes wireless streaming of a messaging application, such as a chat application or a text application, from a user's mobile phone (e.g., a second client computing device) to the user's laptop (e.g., a first client computing device). Many other configurations using different client devices are possible. The graphical user interfaces (GUIs) on different client devices can help support a smooth user experience (UX), thereby facilitating more effective and efficient interactions between the user and others. The second client computing device may create an encoded virtual remote display for the currently running application. This encoded virtual remote display is sent to the first client computing device, where a virtual remote display based on the encoded virtual remote display is presented via the GUI of that device. The virtual remote display on the first device does not need to present the same information as shown in the GUI of the second device and can receive one or more user inputs relayed to the first device. This provides a technical advantage by enabling control of the application without the user having to directly use the first device.

[0004] According to one aspect, the method is to pair a first client device with a second client device via at least one wireless connection, wherein pairing enables the first client device and the second client device to communicate directly; receive, by one or more processors of the first client device, an encoded virtual remote display of an application by streaming from the second client device, wherein the encoded virtual display is generated by the second client device and the application runs on the second client device; and present, by one or more processors of the first device, a virtual remote display of the application based on the encoded virtual remote display on the first client device by controlling a graphical user interface (GUI), wherein the virtual remote display is configured to enable a user of the first client device to control an application running on the second client device.

[0005] The method may further include receiving, by the first client device, an indication of an event associated with an application running on the second client device; and generating, by one or more processors of the first client device, a notification regarding the event for presentation via the GUI based on the received indication. Here, the notification may be executable by a user of the first device such that when an input is received from the user, the method further includes sending a signal to the second client device, and the signal is configured to cause the second client device to perform an action associated with the application.

[0006] Alternatively or additionally, upon termination of the application streaming, the method may include storing an image of the virtual remote display of the application in a cache. Here, upon termination of the application streaming, the method may include designating the application as one of a set of one or more recent applications and generating a menu of one or more recent applications for presentation by a GUI of the first client device. Alternatively or additionally, the method may further include a first client device's one or more processors sending a signal to the second client device to terminate the stream when the first client device determines that it has become unpaired with the second client device.

[0007] Alternatively or additionally, the method may further include terminating the streaming after a certain period when at least one wireless connection is not maintained, re - establishing at least one wireless connection between the first client device and the second client device, and resuming the streaming following the re - establishment of at least one wireless connection. Alternatively or additionally, the method may further include generating, via a GUI of the first client device, at least one of an icon indicating that the second client device and the first client device are paired or an icon indicating that application streaming is in progress.

[0008] Alternatively or additionally, at least one wireless connection may include a first wireless connection and a second wireless connection different from the first wireless connection. Here, pairing is performed via the first wireless connection, and receiving the encoded virtual remote display of the application by streaming is performed via the second wireless connection. In this case, the method may further include transmitting, by the first client device via the second wireless connection to the second client device, one or more user inputs received in response to the virtual remote display.

[0009] Alternatively or additionally, receiving the encoded virtual remote display of the application by streaming includes decoding the encoded virtual remote display of the application by one or more processors of the first client device. Alternatively or additionally, the method may further include generating at least one prompt to facilitate pairing between the first client device and the second client device via the GUI of the first client device.

[0010] According to another aspect, a method for streaming an application comprises pairing a second client device with a first client device via at least one wireless connection, wherein pairing enables the first client device and the second client device to communicate directly; generating, by one or more processors of the second client device, an encoded virtual remote display of the application running on the second client device; transmitting, via the at least one wireless connection, the encoded virtual remote display of the application from the second client device to the first client device by streaming; receiving, by the second client device, one or more user inputs associated with the application based on presenting the application via the virtual remote display based on the encoded virtual remote display on the first client device; and performing, by one or more processors of the second client device, an action associated with the application based on the received one or more user inputs.

[0011] The method may further comprise, prior to transmitting, encoding, by one or more processors of the second client device, the virtual remote display of the application. Alternatively or additionally, performing an action associated with the application may comprise updating the encoded virtual remote display of the application. Alternatively or additionally, the method may further comprise generating, by one or more processors of the second client device, a graphical user interface (GUI) comprising at least one of a permission prompt to facilitate pairing or an actual display of the application for presentation to the user. Here, the encoded virtual remote display generated by one or more processors of the second client device may have a visual appearance different from the actual display of the application.

[0012] Alternatively or additionally for this method, an encoded virtual remote display may be generated by one or more processors of the second client device when the display device of the second client device is in sleep mode. Alternatively or additionally for this method, an encoded virtual remote display may be generated by one or more processors of the second client device while another application is being presented on the display device of the second client device. Alternatively or additionally, the method may further include, in response to the end of streaming, one or more processors of the second client device storing an image of the virtual remote display of the application in a cache. Alternatively or additionally, at least one wireless connection may include a first wireless connection and a second wireless connection different from the first wireless connection. Here, pairing may be performed via the first wireless connection, and transmitting the encoded virtual remote display of the application from the second client device is performed via the second wireless connection.

[0013] According to a further aspect, a computing device is configured to stream an application from a remote device. The computing device includes one or more processors, one or more user inputs operably coupled to the one or more processors, and a communication module operably coupled to the one or more processors. The communication module is configured to facilitate pairing the computing device with the remote device via at least one wireless connection, where pairing enables the computing device and the remote device to communicate directly, and to receive an encoded virtual remote display of the application from the remote device via a stream, where the application is being executed on the remote device. The one or more processors are configured to present the application, including generating a graphical user interface (GUI) to present a GUI virtual remote display of the application being executed on the remote device based on the encoded virtual remote display, receive input associated with the virtual remote display from the one or more user inputs, and transmit information regarding the received input to the remote device via the communication module to control the application on the remote device.

[0014] According to yet another aspect, a computing device is configured to stream an application to a remote device. The computing device includes one or more processors, one or more user inputs operably coupled to the one or more processors, and a communication module operably coupled to the one or more processors. The communication module is configured to facilitate pairing the computing device with the remote device via at least one wireless connection, and the pairing enables the computing device and the remote device to communicate directly. The one or more processors are configured to generate an encoded virtual remote display of an application of the computing device, where the application is executed on the computing device, to cause the communication module to transmit the encoded virtual remote display of the application to the remote display via streaming over at least one wireless connection, to receive from the remote device via the communication module one or more user inputs associated with the application based on a presentation of the encoded virtual remote display of the application on the remote display based on the virtual remote display, and to perform an action associated with the application based on the received one or more user inputs.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11A

Figure 11B

Figure 12A

Figure 12B

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26A

Figure 26B

Figure 27A

Figure 27B

Figure 28

Figure 29

Mode for Carrying Out the Invention

[0016] This technology provides for the streaming of an application on one computing device (the first device) of a user from another device (the second device) of that user. The user can control the operation of the streamed app (e.g., messaging app, streaming app, social media, game, news, etc.) via a virtual remote display presented via the GUI of the first device. The virtual remote display may be based on an encoded virtual remote display that is generated and transmitted from the second device. The encoded virtual remote display may include encoded app content (e.g., audio, video, etc.). The encoder used by the second device may be, for example, an H.264 or MPEG-4 AVC encoder to ensure the quality of the streamed content. This approach addresses the technical problem of supporting a given app on different user devices without the need to download the app to each device or, alternatively, run it on each device. In one scenario, the technical solution is to pair the user devices via a first type of communication link (e.g., a Bluetooth® connection), while the streamed app and user commands sent back to the second device are provided via a second type of communication link (e.g., a WiFi or NFC connection). The virtual remote display does not need to present the same information as shown on the GUI of the second device, and in some examples, the app can be run on the second device in the background (or when the display of that device is off). In this regard, the virtual remote display may provide a standardized view (e.g., a view compatible with the GUI of the first device, input, etc.) or representation of the app on the GUI of the first device, thereby making it easier for the user to control the app using input from the computing device. The input is transmitted to the second device on which the virtual remote display is generated.This provides a technical advantage by enabling control of the application without the user having to directly use the second device. In this regard, instead of an exact copy of the GUI of the second device, a standardized view or representation of the application is utilized, which may require less bandwidth / overhead and smaller memory requirements for application control. Thus, real-time or near-instantaneous control of the application on the second device using input from the first device can be supported.

[0017] Exemplary System Unless otherwise specified, the disclosed embodiments and scenarios are not mutually exclusive, but can be implemented in various combinations to achieve their respective advantages. These and other variations and combinations of the above-described features can be utilized without departing from the subject matter defined by the claims, and thus the description of the embodiments should be construed as illustrative rather than as a limitation of the subject matter defined by the claims. Additionally, the examples described herein and the provision of phrases such as "such as" and "including" should not be construed as limiting the subject matter of the claims to specific examples, but rather the examples are intended to illustrate only one of many possible embodiments. Further, the same reference numerals in different drawings can identify the same or similar elements.

[0018] Exemplary Computing Device FIG. 1 shows an example 100 depicting a client computing device 102 such as a laptop computer. In this example, there is a display 104 for visually presenting content via a GUI 106, and a user input section 108 having a keyboard 110 and a trackpad 112 as different user inputs. A microphone (not shown) may be provided for audible input. The display 104 may be configured as a user input (e.g., a touch screen). The keyboard 110 may be a virtual keyboard, for example, as part of another display. Although one trackpad 112 is shown, the device may have two or more trackpads disposed thereon. The device may include additional user inputs (e.g., a mouse, audio, and / or gesture-based input).

[0019] An integrated web camera 114 can be used for video conferencing, interactive games, etc. An indicator 116 such as an LED can be illuminated to warn the user whenever the integrated web camera is in use. The client device may also include one or more other sensors 118 that can be used to supplement the visual information obtained by the web camera 114. By way of example, the one or more other sensors may include additional imaging devices, RF-based or ultrasonic-based motion sensors, for example, to assist in identifying gestures by the user or to function as a presence detector.

[0020] FIG. 2 shows a block diagram of an exemplary client computing device 200, such as a home device such as a laptop (e.g., a tablet PC or a netbook), a desktop PC, a smart display, etc. shown in FIG. 1. As shown, the client computing device 200 includes a processing module 202 having one or more computer processors and / or graphics processors 206, such as a central processing unit 204, and a memory module 208 configured to store instructions 210 and data 212. The processor may operate in parallel or not, and may include ASICs, controllers, and other types of hardware circuits. The processor is configured to receive information from the user through a user interface module 214 and present information to the user on one or more display devices of a display module 216 having a display interface.

[0021] The user interface module 214 may receive commands or other input information from the user via user input and convert them for submission to a given processor. The user interface module may link to a web browser and other applications executed by the processing module 202 of the computing device. User input may include one or more of a touch screen, keypad, track pad, stylus, microphone, or other types of input devices. The display module 216 may include appropriate circuitry for driving a display device to present graphic information and other information to the user. By way of example, the graphic information may be generated by a graphics processor(s) 206, while the CPU 204 manages the overall operation of the client device 200. The graphic information may display an answer to a user query or other content on the display module 216. For example, the processing module may use instructions and data stored in the memory module 208 to execute a browser application, messaging application, or other service and present information associated with the browser application or other service to the user via the display module 216. The memory module may include a database or other storage for application-related information and the like.

[0022] The memory module 208 can be implemented as one or more of a computer-readable medium(s), volatile memory unit(s), or non-volatile memory unit(s). The memory module 208 can include, for example, flash memory and / or NVRAM, and can be embodied as a hard drive or a memory card. Alternatively or additionally, the memory module 208 can also include removable media (e.g., DVD, CD-ROM, or USB thumb drive). One or more regions of the memory module 208 can be writable, while other regions can include read-only (or otherwise write-protected) memory. In one embodiment, the computer program product is tangibly embodied in an information carrier. FIG. 2 functionally shows the processor(s), memory module, and other elements of the client computing device 200 as being within the same overall block, although such components may or may not be housed within the same physical enclosure. For example, some or all of the instructions and data can be stored on an information carrier that is a removable storage medium (e.g., an optical drive, a high-density tape drive, or a USB drive), and others can be stored within a read-only computer chip.

[0023] Data 212 can be retrieved, stored, or modified by the processor according to instructions 210. For example, the data can be stored in a relational database as a table having a plurality of different fields and records, an XML document, or a flat file, or in a computing device register. The data can also be formatted in any computing device-readable format.

[0024] Instruction 210 may be any set of instructions that are executed directly (such as machine code) or indirectly (such as a script) by one or more processors. For example, the instructions may be stored on a computer-readable medium as computer device code. In that regard, the terms “instruction” and “program” may be used interchangeably herein. The instructions may also be in object code form for direct processing by one or more processors, or may be stored in any other computer device language including a script or a collection of independent source code modules that are interpreted on demand or pre-compiled.

[0025] As also shown in FIG. 2, client device 200 includes a communication module 218 for communicating with other devices and systems, including other client devices, servers, and databases. Communication module 218 includes a wireless transceiver, or alternatively or additionally, the module may include a wired transceiver. Client device 200 may communicate with other remote devices via communication module 218 using various configurations and protocols, including short-range communication protocols such as near field communication (NFC), Bluetooth®, Bluetooth® Low Energy (BLE), or other ad hoc networks, the Internet, an intranet, a virtual private network, a wide area network, a local area network, a private network using a communication protocol unique to one or more companies, Ethernet®, WiFi, and HTTP, and combinations of the foregoing.

[0026] The illustrated exemplary client device 200 also includes one or more position and orientation sensors 220. The position and orientation sensors 220 are configured to determine the position and orientation of one or more portions of the client computing device 200. For example, these components can include a GPS receiver or other global positioning component that determines the device's latitude, longitude, and / or altitude, as well as another direction / velocity detection device such as an accelerometer, gyroscope, or inertial measurement unit (IMU). The client device 200 may also include one or more cameras (s) 222, speakers (s) 224, such as the integrated web camera described above, for capturing still images and recording video streams. A power module 226 provides power to the various system components. Further, user input includes a keyboard 228 and a microphone 230, which may include a microphone array having one or more transducers or other microphone elements distributed at various points along the housing of the computing device. The trackpad 232 may include either (or both) a capacitive touch sensor unit 234a or a piezoelectric sensor unit 234b.

[0027] Exemplary Mobile Device FIG. 3 shows an example 300 showing a handheld type client mobile device 302, such as a smartphone, personal digital assistant (PDA), or wearable (e.g., smartwatch). In this example, there is a display 304 for visually presenting content. The display 304 may also be configured as a user input (e.g., touch screen). A microphone 306 may be provided for audible input. The integrated camera 308 can be used for taking photos, video conferencing, interactive games, etc. An indicator 310, such as an LED, can be illuminated to warn the user whenever the integrated camera 308 is in use.

[0028] Figure 4 shows a block diagram of an exemplary handheld client mobile device 400, such as the smartphone shown in Figure 3. As shown, the client mobile device 400 includes a processing module 402 having one or more processors and / or a graphics processor 406, such as a central processing unit 404, and a memory module 408 configured to store instructions 410 and data 412. The processors may operate in parallel or not at all and may include ASICs, controllers, and other types of hardware circuits. The processors are configured to receive information from a user via a user interface module 414 and present information to the user on one or more display devices of a display module 416 having a display interface.

[0029] The user interface module 414 may receive commands or other input information from the user via user input and convert them for submission to a given processor. The user interface module may link to a web browser and other applications executed by the processing module 402 of the mobile device. User input may include one or more of a touch screen, a stylus, a microphone, or other types of input devices. The display module 416 may include appropriate circuitry for driving the display device to present graphic and other information to the user. As an example, the graphic information may be generated by the graphics processor(s) 406, while the CPU 404 manages the overall operation of the client mobile device 400. The graphic information may display an answer to a user query on the display module 416. For example, the processing module may use instructions and data stored in the memory module 408 to execute an application or other service and present information associated with a browser application or other service to the user via the display module 416. The memory module may include a database or other storage for application-related information and the like.

[0030] Memory module 408 can be implemented as one or more of a computer-readable medium(s), volatile memory unit(s), or non-volatile memory unit(s). Memory module 208 can include, for example, flash memory and / or NVRAM, and can be embodied as a hard drive or a memory card. One or more regions of memory module 408 may be writable, while other regions may include read-only (or otherwise write-protected) memory.

[0031] Data 412 can be retrieved, stored, or modified by a processor according to instructions 410. For example, the data can be stored in a relational database as a table having a plurality of different fields and records, an XML document, or a flat file, or in a computing device register. The data can also be formatted in any computing device-readable format.

[0032] Instructions 410 can be any set of instructions that are executed directly (such as machine code) or indirectly (such as a script) by a processor(s). For example, the instructions can be stored on a mobile device-readable medium as mobile device code. In that regard, the terms "instructions" and "program" can be used interchangeably herein. The instructions can also be stored in object code form for direct processing by a processor(s), or in any other computing device language that includes a script or a collection of independent source code modules that are interpreted on demand or pre-compiled.

[0033] As also shown in FIG. 4, the client mobile device 400 includes a communication module 418 for communicating with other devices and systems, including other client devices, servers, and databases. The communication module 418 includes a wireless transceiver. The client device 400 can communicate with other remote devices via the communication module 418 that uses various configurations and protocols, including short-range communication protocols such as Near Field Communication (NFC), Bluetooth™, Bluetooth™ Low Energy (BLE), or other ad-hoc networks, the Internet, an intranet, a virtual private network, a wide area network, a local area network, a private network using a communication protocol unique to one or more companies, WiFi and HTTP, and combinations of the foregoing.

[0034] The illustrated exemplary client device 400 also includes one or more position and orientation sensors 420. The position and orientation sensors 420 are configured to determine the position and orientation of one or more portions of the client mobile device 400. For example, these components can include a GPS receiver or other global positioning component that determines the latitude, longitude, and / or altitude of the device, as well as another direction / velocity detection device such as an accelerometer, gyroscope, or inertial measurement unit (IMU). The mobile device 400 can also include one or more cameras 422 for capturing still images and recording video streams, such as the integrated camera described above, and speakers 424. A power module 426 provides power to the various system components. Further, the user input can include a microphone 430, which can include a microphone array having one or more transducers or other microphone elements distributed at various points along the housing of the computing device.

[0035] Exemplary Network As further described herein, one or more devices may be connected to each other via a network to stream an application or app. FIGS. 5A and 5B are, respectively, a drawing and a functional diagram of an exemplary system 500 including a plurality of computing devices and a database connected via a network. For example, the computing device(s) 502 may be a cloud-based server system that provides or otherwise supports one or more apps, games, or other programs. The database 504 may store messaging app information, game data, user profile information, and / or other information. The server system may access the database via the network 506. The client devices may include one or more of home devices such as a desktop computer 508, a laptop or tablet PC 510, and a smart display 512. Other client devices may include personal communication devices such as a mobile phone or PDA 514, or wearable devices such as a smartwatch 516. Another exemplary client device is a large-screen display or interactive whiteboard 518 that may be used in a classroom, conference room, auditorium, or other shared collective space where multiple users may be present. Additional client devices may be a pair of headphones or earphones (not shown).

[0036] In one example, computing device 502 can include one or more server computing devices having a plurality of computing devices, such as, for example, a load-balanced server farm or a cloud computing system, which exchange information with different nodes of a network for the purpose of receiving data, processing it, and transmitting data between other computing devices. For example, computing device 502 can include one or more server computing devices that can communicate with any of computing devices 508-518 via network 506. This can be done as part of hosting one or more collaborative applications (e.g., a video conferencing program, an interactive spreadsheet application, or a multiplayer game) or services (e.g., a movie streaming service or an interactive game show where viewers can provide comments or other feedback).

[0037] As shown in FIG. 5B, each of computing devices 502 and 508 - 518 may include one or more processors, memory, data, and instructions. The memory stores information accessible by one or more processors, including instructions and data that may be executed or otherwise used by the processor(s). The memory may be any type of information storage accessible by the processor(s) that includes a computing device readable medium. The memory may be non - transient media such as a hard drive, memory card, optical disk, solid state, etc. The system may include different combinations as described above, whereby different portions of instructions and data are stored on different types of media. The instructions may be any set of instructions that are executed directly (such as machine code) or indirectly (such as a script) by the processor(s). For example, the instructions may be stored on a computing device readable medium as computing device code. In that regard, the terms "instructions", "modules", and "programs" may be used interchangeably herein. The instructions may also be stored in object code form for direct processing by the processor, or in any other computing device language including scripts or collections of independent source code modules that are interpreted on demand or pre - compiled.

[0038] The processor may be any conventional processor such as a commercially available CPU. Alternatively, each processor may be a dedicated device such as an ASIC, a graphics processing unit (GPU), a tensor processing unit (TPU), or other hardware-based processor. FIG. 4B functionally shows the processor, memory, and other elements of a given computing device as being within the same block, although such devices may in fact include multiple processors, computing devices, or memories that may or may not be housed within the same physical enclosure. Similarly, the memory may be, for example, a hard drive or other storage medium located within an enclosure different from that of the processor(s) within, for example, a cloud computing system of server 1402. Thus, a reference to a processor or computing device is understood to include a collection of processors, computers, computing devices, or memories that may or may not operate in parallel.

[0039] A computing device may include all of the components that are typically used in connection with a computing device, such as the processor and memory described above, as well as a user interface subsystem that receives input from a user and presents information (e.g., text, images, and / or other elements) to the user. The user interface subsystem may include one or more user inputs (e.g., at least one forward (user)-facing camera, mouse, keyboard, touch screen, and / or microphone) and one or more display devices operable to display information (e.g., text, images, and / or other graphic elements). Other output devices such as speakers may also provide information to the user.

[0040] Computing devices related to a user (e.g., 508 - 518) can communicate with a backend computing system (e.g., server 502) via one or more networks such as network 506. Network 506 and intervening nodes can include various configurations and protocols, including short - range communication protocols such as Bluetooth (trademark), Bluetooth LE (trademark), the Internet, the World Wide Web, intranets, virtual private networks, wide - area networks, local networks, private networks using a communication protocol unique to one or more companies, Ethernet, WiFi, and HTTP, as well as various combinations of the foregoing. Such communication can be facilitated by any device capable of transmitting data between other computing devices, such as modems and wireless interfaces.

[0041] In some embodiments, computing devices related to a user can also communicate with each other without communicating with the backend computing system. For example, FIG. 6 shows a system 600 in which a client computing device 602 (e.g., a laptop) and a client mobile device 604 (e.g., a cell phone) are wirelessly connected via a network 606. The client computing device 602 can be the client computing device described with respect to FIGS. 1 - 2. Similarly, the client mobile device 604 can be the client mobile device described with respect to FIGS. 3 - 4. Network 606, and intervening nodes, can include the same configuration as network 506. Alternatively or additionally, network 606 can include one or more ad - hoc communication links and / or peer - to - peer communication links between devices 602 and 604, where there would be no intervening nodes.

[0042] Exemplary Method Pairing While starting the streaming process of an application according to an aspect of the present technology, a client mobile device and a client computing device can be connected to each other using a first wireless connection such as, for example, a Bluetooth (trademark) connection or other ad hoc connection. When starting the connection, one of the user's computing devices (either a mobile device or a computing device) can generate a prompt to cause the user to pair the devices (a process that enables two or more devices to communicate directly when within range of each other), as shown in FIGS. 7-11. In FIG. 7, a first prompt 700 within the GUI (here, on the user computing device) can provide a message 708 to notify the user regarding the pairing function. The first prompt 700 can include a link included in the text "More Details" 702, enabling the user to access additional information such as user consent, pairing details, supported applications, etc. The first prompt 700 can further include a link to end the process included in the text "Cancel" 706 and a link to proceed to the next prompt included in the text "Next" 704.

[0043] After the user selects "Next" 704 using one or more user interfaces of a computing device (such as a touch screen, trackpad, mouse, keyboard, microphone, or gesture, etc.), the device may generate a second prompt 800 within the GUI as shown in FIG. 8. The second prompt 800 may provide a message 808 and may include one or more selections 810, 812 to enable the user to establish privacy protection. The user may choose to enable password protection, personal identification number protection, or both that need to be input by the user when the mobile device, computing device, or both are asleep (such as components like screen power off). The second prompt 800 may similarly include links to "More Details" 802, "Cancel" 806, and "Next" 804 that the user may be involved with. In some embodiments, the second prompt 800 may not be generated, for example, if the user has previously established a password, personal identification number, or both.

[0044] After the user makes selections 810, 812, the user may select "Next" 804. The computing device may then generate a third prompt 900 in the GUI as shown in FIG. 9. The third prompt may similarly include links to "More Details" 902 and "Cancel" 906. The third prompt 900 enables the user to continue the pairing process on the mobile device using message 908, and the message 908 may state, for example, "Please confirm some permissions as you move to the phone". The third prompt 900 also includes a link in the text "Done" 904 to end the pairing process upon completion. The link "Done" 904 may not be active until the user completes the permission process.

[0045] In this scenario, after a third prompt 900 is generated on the user computing device, at least one permission prompt 1000 is generated on the user mobile device as shown in FIGS. 10A - 10B. The user may either grant or deny authorization (e.g., access to photos and media, notifications, and applications) via the "Grant" 1002 or "Deny" 1004 link. A message 1006 may notify the user of what permissions are being requested. Here, as shown by 1006A in FIG. 10A, the message section may indicate the device type to which access is being granted, along with other information such as notifications and / or photos and media. As shown by 1006B in FIG. 10B, the message section may indicate the above device type, but may present different information such as an app notification regarding streaming the app of the device. Additional information about the streaming implementation may be found by clicking on icon 1008. If the user selects "Deny" 1004, the pairing process ends. If the user selects "Grant" 1002, a fourth prompt 1100 may be generated in the GUI on the computing device shown in FIG. 11A. The fourth prompt 1100 presents a message 1108 notifying the user that the mobile device and the computing device are paired, and includes a link contained in the text "Done" 1104 to end the pairing process upon completion. In some embodiments, if the user has previously given permission for the computing device to access the mobile device, the permission prompt may not be generated on the mobile device. When there is an active connection, the mobile device GUI may show a "Connected Devices" screen 1120 as shown in FIG. 11B. This screen may show options to pair a device previously connected at 1122, other devices at 1124, and new devices at 1126.

[0046] After the user selects "Done" or otherwise completes the process, the mobile device and the computing device may remain paired and, when within direct communication range of each other, may share content. This range may be within or more than 2 to 5 meters depending on the type of connection (e.g., Bluetooth (trademark) or Bluetooth LE (trademark) link). According to one aspect, the connection is maintained to keep the devices paired. The computing device, the client device, or both may periodically test for network connectivity. If the connection is not detected, the computing device, the client device, or both may display a "timeout" message. For example, the connection may not be detected because the devices are out of range of each other or due to interference on the communication channel. When the connection is re-established, the message may disappear. Otherwise, the connection ends and, in some scenarios, may automatically reconnect when the computing device and the mobile device come back within range of each other. A notification may be generated to communicate this to the user if the devices are out of range or otherwise unable to connect.

[0047] When a computing device and a mobile device are paired, the user can access the functions of the mobile device by streaming an app via the GUI on the computing device. As shown in FIG. 12A, the mobile device interface 1202 is located within the GUI on the display 1200 of the computing device. The interface 1202 may include functions such as app selection 1204, recent photo selection 1206, phone control functions 1208 (e.g., hotspot control, phone volume control, phone search, etc.), mobile device name 1209, mobile device status icon 1210 (e.g., power level, mobile network connection), and mobile device settings menu link 1212. The app selection 1204 may include links and corresponding icons 1205 that can direct the user to an app list. In some examples, the app selection 1204 may be a recent app selection. Alternatively, the app selection 1204 may be a selection of favorite apps. In this regard, the user can select the app selection 1204 to be one of the recent app selection or favorite app selection via the settings menu. In some embodiments, the computing device may also display an icon 1214 included in the task bar of the computing device indicating that the mobile device is paired and currently connected to the computing device. The icon 1214 may also indicate when the device is paired but not currently connected. Both the computing device and the mobile device may further include a list of paired devices within the settings menu. When paired, the computing device may indicate the mobile device as paired, and vice versa. Further, each list may indicate whether the device is currently connected to others.

[0048] In some embodiments, the link and corresponding icon 1205 may direct the user to an app list. An exemplary app list 1250 is shown in FIG. 12B. Although FIG. 12B shows an app list 1250 in a grid format, the app list may be in a line list (e.g., one icon or app per line) format. In some examples, the app list 1250 may enable the user to view personal apps and work apps separately. In this regard, the user may select or pin an app as a work app. When a work app is displayed, the selected or pinned work app may be displayed in the app list. When personal apps are displayed, apps that are not selected or pinned as work apps may be displayed in the app list. Additionally or alternatively, the app list may include a favorites list. The favorites list may include apps that the user has selected or pinned as favorites.

[0049] Streaming of Applications Following the pairing of a computing device and a mobile device via a first wireless connection, the user may start streaming an application as long as the pairing connection is maintained. Here, the system (the mobile device, the computing device, or both) may check periodically (e.g., every 10 - 30 seconds, or more, or less) to ensure that the pairing connection is maintained. According to one aspect, the streaming of the application may use a second wireless connection such as a Web Real - Time Communication (WebRTC) connection. This connection may use a different wireless connection such as a WiFi connection. In one embodiment, the first wireless connection may be used to ensure proximity between the computing device and the mobile device, while the second wireless connection may be used to communicate stream information between the devices. The system may notify the user that the pairing connection has been lost (e.g., the mobile device and the computing device are out of Bluetooth (trademark) range), and may set a timeout at which the streaming connection may be disconnected if it is not re - established before the timeout is reached. Here, the system may automatically re - establish the pairing connection when the devices come back within range of each other.

[0050] Streaming of an application on a user computing device can be initiated in several ways and can be managed by the device's operating system that executes one or more modules via the device's processor. In one example, an application notification 1302 can be generated in the GUI of a computing device display 1300 as shown in FIG. 13. In this example, the notification 1302 is a mirrored notification, meaning that the notification can be displayed on both the computing device and the mobile device. The mirrored notification may look the same on both displays or may have a different appearance (e.g., due to screen size and / or orientation). The user can interact with the notification 1302 using the user interface. For example, in addition to ignoring the notification, the user can dismiss it (and that dismissal may also apply to the connected phone or other devices running on the app), respond inline (i.e., never leave the notification), or tap on the notification to stream the full app from another device. For example, the app can be a messaging app. The notification for the messaging application shown in FIG. 13 includes options to select "Reply" 1304, "Mark as Read" 1306, or select the body of the notification. In this example, if the user selects "Mark as Read" 1306, the message can be marked as read, meaning that the message notification may no longer be displayed on the mobile device or the computing device.

[0051] When the user selects "Reply" 1304 or selects the body of the notification, the messaging application can be streamed on the computing device. As shown in example 1400 of FIG. 14, an icon 1402 having a circular progress bar 1404 is displayed on the computing device, indicating the load status of the application stream. The icon 1402 can be located in any part of the computing device display. In an embodiment that includes the reply inline function described below with respect to FIG. 24, when the user selects "Reply" 1304, the user can reply within the mirrored notification without streaming the app. In such an embodiment, the stream can be initiated from the notification by selecting the body of the notification. In one example, the notification can be executable by the user such that when an input is received from the user, a signal can be sent to the device running the app, and the signal can be configured to cause the device to perform an action associated with the application.

[0052] After the application stream is loaded, the user can interact with the stream via the user interface as shown in view 1500 of FIG. 15. Although FIG. 15 shows the stream in a portrait screen, the stream may also be in a landscape screen or full-screen display. Additionally or alternatively, the stream may be alternately displayed among portrait screen, landscape screen, or full-screen display. The stream 1502 on the computing device enables the same application functions as on the mobile device. For example, if the application is the messaging application shown in FIG. 15, the stream 1502 may include the ability to enter text in the text box 1506, the ability to attach a file or photo via the icon 1508 from either the computing device or the mobile device, the ability to capture a new photo via the icon 1510, the ability to enter a voice command via the input 1512, the ability to start a video call via the icon 1514, and / or the ability to start a call via the icon 1516. Such functions may be accessible through the user interface(s) of the computing device.

[0053] In some scenarios, after the stream is loaded, the corresponding icon may further display an application symbol to indicate which application is currently streaming. For example, as shown in view 1600 of FIG. 16, the icon 1602 includes a logo mark 1604 indicating a specific application streaming on the computing device. Similarly, view 1700 of FIG. 17 shows a scenario where a messenger application is streaming and the icon 1702 includes the messenger application 1704. Alternatively, the user may choose not to interact with the notification. When the user is not interacting with the notification, the notification may disappear after a certain period (e.g., 20 seconds or more or less). The user can access any notification in the notification center, which will be described in more detail below.

[0054] In another example, the user may manually select an application for streaming. The user may manually select an application, for example, by selecting an app from a list of mobile applications on a computing device or by selecting an app for streaming from a mobile device to a computing device.

[0055] In a further example, the system may implement smart audio source selection, and the system automatically determines whether to play audio directly from the mobile device or from the computing device. This may include providing smooth switching of headphones (or earphones) between audio sources. Here, the selection may be based on whether the headphones (or earphones) are currently paired with either (or both) the mobile device or the computing device. If there is pairing with both devices, the system may connect the headphones (or earphones) to a particular device based on, for example, physical proximity (which can be estimated by evaluating signal strength information from the pairing), the type(s) of the application(s) being streamed, etc.

[0056] In another example, the user may select an application for streaming from an application selection or application list. In one example, the application selection may be a selection of a favorite application. Since the application is a favorite application, the user may so specify the application. In one example, the application may be pinned or selected as a favorite application on an application list (e.g., application list 1250). A recent application may be selected via a user interface for streaming from, for example, the application selection 1204 list shown in FIG. 12A. After selection, the application may start streaming on the computing device. In an example, the user may select a recently used application for streaming. If the application is considered recently used, in one example, the application may have been streamed on the computing device or utilized on a mobile device within a particular time frame (e.g., 30 minutes, 4 hours, within the last day, or some other time frame). The time frame may be set by the user or may be a default time frame. A recent application may be selected via a user interface for streaming from, for example, the application selection 1204 list shown in FIG. 12A.

[0057] After selection, the application may start streaming on the computing device. Multiple applications may be streamed between devices at once. The stream may resume the streaming content when the user most recently accessed the application. For example, if the user streams an application on a computing device, ends the streaming, and then selects the application from the app selection 1204, the computing device may stream the application as if it were when the user ended the stream. In another example, if the user streams an application on a computing device, ends the stream, and starts using the application on a mobile device, the application may resume as if it were when the user ended the stream. In this regard, the stream may be paused when it is inactive, such as when the user ends the stream. However, the app may persist on the source device (e.g., mobile device, etc.) as a background operation. By persisting in this way as a background operation, it becomes possible for the user to resume the app.

[0058] Similarly, if the user next closes the application on the mobile device and then selects the application from the app selection 1204 or the app list 1250, the computing device will stream the application as it did when the user closed the application on the mobile device. In one example, the system (e.g., on the mobile device) may store a full-color image of the streamed app in a cache when the stream is closed to preserve the current stream state. The image stored in the cache can be used as a start image to be presented when the app is streamed again, such as when the user chooses to stream the app by interacting with the link or starting the stream in any way as described above. The image stored in the cache can be stored as long as the connection between the computing device and the mobile device is maintained.

[0059] When streaming an application on a computing device, the application need not be displayed on the mobile device. According to one scenario, to generate the stream, the mobile device creates a virtual remote display for presentation on the computing device. The mobile device encodes the virtual remote display that includes the application content to be streamed (e.g., audio, video, etc.). The mobile device may use, for example, an H.264 or MPEG-4 AVC encoder to ensure the quality of the streamed content. The encoded information including the application content can be sent to the computing device via the connection between the devices. Upon reception, the computing device decodes and renders the virtual remote display, thereby enabling the computing device to stream the application. The computing device may function as an input device for the virtual remote display. User input can be received from the computing device and the user input is then communicated to the mobile device via a wireless connection. The mobile device can then update the virtual remote display based on the user input. Thus, the mobile device not only acts on the received user input but also modifies the virtual remote display. In this regard, the streamed application can be hosted by the mobile device, meaning that the processing module and memory module of the mobile device are utilized to execute the application. The computing device renders the virtual remote display and functions as an input device, but does not need to install the streamed application on the computing device. The virtual remote display can be formatted as a web page, thereby enabling the computing device to set its display size as desired. Thus, the appearance of the virtual remote display (e.g., portrait mode or landscape mode or overall size) can be modified during streaming.Furthermore, there is no need to store application information (e.g., data, instructions) in the computing device. In some examples, the mobile device may display an indication or notification to the user that an app streaming is taking place. In this regard, the virtual remote display provides a standardized view of the app on the GUI of the computing device, which makes it easier for the user to control the app using the input from the computing device. The input is transmitted to the mobile device on which the virtual remote display is generated.

[0060] According to one aspect, the user interaction with the streamed app on the computing device is transient, and the information about the streaming may not be stored by the computing device when the computing device passes a signal (e.g., user control of an audio stream or a video stream) to the mobile device. As an example, input events (e.g., pausing a song, replying to a text) are converted in the computing device into a format that the mobile device can process. This can include processing touch input, keypad input, cursor input, audio input, or gesture input into signals that the app running on the mobile device can act on. For example, in a messaging app, the main action may be to press the enter key to send a message, but the app running on the mobile device may not support that function. In such an example, the mobile device may execute a machine learning (ML) model to determine which input(s) map to sending, and as a result, the mobile device may map the input(s) when such input(s) are received from the computing device.

[0061] In contrast to screen mirroring, app streaming does not require the mobile device to show the same content that is being streamed on the computing device. The user can freely leave the mobile device screen powered off, or keep it dimmed, or interact with another app while the stream is in progress. Thus, while the virtual remote display is being streamed on the computing device, the mobile device can perform other functions. For example, the mobile device may enter sleep mode (e.g., the mobile device display may be off or present a screen saver) and still generate the virtual remote display on the computing device. In this regard, a partial processing module of the mobile device, such as an application processor, may remain active to generate the stream while the mobile device display is inactive. Alternatively, the mobile device may enable the user to interact with additional applications while generating the stream to the computing device. In one scenario, the user can minimize the streamed app, and the streamed app can remain minimized for a selected time (e.g., 5 minutes or more or less), after which the streamed app can be automatically closed to conserve the mobile device battery. In this regard, the virtual remote display can be considered essentially a secondary display that uses a secondary set of inputs and outputs (e.g., the inputs and outputs of the computing device), and thus the inputs and outputs of the mobile device (e.g., display, microphone, speaker, etc.) may be powered off or utilized for different functions.

[0062] A user may stop an app stream in several ways. In one example, the stream may be closed due to user inactivity (e.g., no user interaction with the streaming application for a certain minimum time). As an example, when the stream is closed due to inactivity, the stream may be idle for a period of 10 - 15 minutes, or more, or less. As shown in view 1800 of FIG. 18, when the stream is stopped due to inactivity, a message 1802 may be generated for display on the computing device. Message 1802 may include text 1804 notifying the user that the connection or stream has ended due to inactivity. The message may also include a link included in text "Resume" 1806 for the user to resume the stream. As described above, the system may store an image (e.g., a full-color image) of the streaming app in the cache when the user is idle for a period of time to maintain the current stream state. The image stored in the cache may be used to resume the stream when the user chooses to resume the stream by interacting with the link or starting the stream in any of the ways described above.

[0063] In another example, the stream can be closed by the user selecting the app that is currently being streamed in app selection 1204 or app list 1250. Selecting app selection 1204 or app list 1250 can stop the stream on the computing device, transfer the app content to the main display of the mobile device, and enable the user to resume using the app on the mobile device from the point where the user closed the stream on the computing device. In a further example, the user can close the stream from within the stream itself. FIG. 19 shows a computing device display 1900 having an open stream 1902 and a streaming app task bar 1904. The virtual remote display 1902 includes a back button 1905, a minimize control 1906, and a close control 1908. The back button 1905 can end the stream and return to the app list and / or app selection screen. The minimize control 1906 can minimize the app stream while maintaining the active stream, while the close control 1908 can end the app stream. The minimized app stream can remain minimized for a certain period (e.g., 10 minutes, or more, or less), after which the stream can automatically end. In another example, the stream can close due to a specific error, such as when the computing device and the mobile device are no longer in range. The closing of the stream due to an error is described in more detail below.

[0064] In an additional example, the stream can be closed from the mobile device via a silent and persistent notification 2002, as shown in example 2000 of FIG. 20. While the app is streaming to the computing device, the mobile device can display a persistent notification 2002 that notifies the user that the application is currently being streamed. The notification 2002 can be located anywhere on the GUI of the mobile device. The notification 2002 can include a stream end control 2004 and a settings control 2006. The user can use the stream end control 2004 to end the stream. When the stream ends, the streamed app can be added to the most recent app section. For example, when app selection 1204 is the most recent app selection, the app can be added to app selection 1204 if the app is closed from the computing device or the mobile device. In one example, when an app is opened on the computing device, the app moves to the top of the most recent app stack on the mobile device side. In another example, when an instance of an app is opened from the most recent app stack on the mobile device, any active streaming connection associated with that app may also be disconnected. Here, even if the stream is stopped, the app may not close because the app is actively open on the mobile device. However, when the app stream is closed on the computing device, access to the app may become available at the top of the most recent app stack on the mobile device.

[0065] In some embodiments, a prompt may be displayed on a computing device that directs a user to install, update, and / or enter user information required to stream an app. Depending on the user's status, various messages may be generated. For example, the user may be a new user, an existing user, or an existing user with one or more new devices. In one embodiment, as shown in example 2100 of FIG. 21, a new user may receive a notification 2102 to install software on the computing device that can execute the methods described herein and may establish a user profile. Notification 2102 may include a link 2106 to initiate the installation process and a control 2104 to abort the installation process and delete notification 2102. An existing user with one or more new devices may receive the same or a similar message as a new user. As shown in view 2200 of FIG. 22, the user may receive a notification 2202 on the computing device to update existing software that can execute the methods described herein. Notification 2202 may also include a link 2206 to initiate the installation process and a control 2204 to abort the installation process and delete notification 2202.

[0066] Alternatively, the user may be prompted to update current software via a mobile device interface 2302 on a display 2300 of the computing device. FIG. 23 shows an interface 2302 with a prompt 2304 to update and / or install software. Prompt 2304 may include a link 2308 to initiate the installation process and a control 2306 to abort the installation process and dismiss the notification.

[0067] In some embodiments, the app may be a messaging app, and notifications for such a messaging app may include a reply inline feature. For example, as shown in view 2400 of FIG. 24, a notification 2402 regarding a messaging app displays text and app information 2404. The notification 2402 also includes a reply inline feature that allows a user to read the message text 2404 without streaming the app, enter a reply in a text box 2406 using one or more user inputs, and send the reply using a send control 2408 that uses one or more user inputs. In some embodiments, the notification and stream may enable sending a message from one or more user inputs using a specific input. For example, a user may send a message using the enter key (or other key) of a computing device keyboard. To do this, mobile device inputs may be mapped to computing device inputs, and when using a mobile device, if pressing the enter key sends a message, this functionality may be mapped to the computing device, meaning that the enter key is also made to enable sending a message.

[0068] In some embodiments, when the user does not engage with the notification (e.g., open the stream, reply inline, or dismiss the notification), the notification can be accessed in the notification center. For example, in FIG. 25, the user can access a quick settings menu 2502 on a display 2500 of a computing device. A notification center 2504 that includes one or more presence notifications can be displayed along with the quick settings menu 2502. In one scenario, only conversation notifications (e.g., chat or text strings) are mirrored from a mobile device onto the computing device. As an example, when responding to a notification or dismissing the notification on the computing device, the notification is cleared from the mobile device. Selecting an app notification (e.g., by tapping or clicking on the app notification) can open the streamed app and can add the app to the most recent app list on the mobile device.

[0069] In some embodiments, the mobile device generates a silent and persistent notification 2602 indicating that the mobile device is connected to a computing device. For example, as shown in FIG. 26A, the persistent notification 2602A may enable the user to end the connection via the disconnection control 2604A. The persistent notification may also enable the user to access the connection settings via the settings control 2606A. In some embodiments, the mobile device may generate a different silent and persistent notification 2602B when the mobile device is streaming to a computing device. FIG. 26B shows the persistent notification 2602B. The persistent notification 2602B notifies the user that the mobile device is currently streaming to the computing device and may enable the user to end the stream via the stream end control 2604B and access the connection settings via the settings control 2606B. Additionally or alternatively, the silent and persistent notification may be a connection indicator such as an icon for notifying the user of the connection between the mobile device and the computing device. When streaming, the icon may further indicate that the application is currently streaming. This may be done, for example, by including the icon of the streamed application. Such an indicator may be located anywhere on the display of the mobile device. Further, the indicator may be a light indicator (e.g., an LED) located on the mobile device. Additionally or alternatively, there may be multiple indicators at different locations.

[0070] In some embodiments, the computing device enables the user to zoom in and out of the displayed stream image using multi-touch interaction. The multi-touch interaction can be supported by one or more user inputs such as a trackpad and a touch screen. FIGS. 27A-27B illustrate ways in which a user can use multi-touch interaction. For example, to zoom in or enlarge, the user may touch the display with two fingers, namely, a first finger at a first point 2702A and a second finger at a second point 2704A. The user can then slide the fingers towards each other across the user interface. To zoom out or reduce, the user may touch the user interface with two fingers, namely, a first finger at a first point 2702B and a second finger at a second point 2704B. The user can then slide the fingers away from each other across the user interface. In an example where the user interface is a trackpad, the stream can zoom in and / or out at the location where the cursor is currently positioned on the display of the computing device. Alternatively, zooming in and out can also be performed using gestures or voice commands.

[0071] In some embodiments, the system may be configured to handle stream errors (e.g., connection timeouts) and respond to content errors (e.g., content playback not functioning) to maintain the system's integrity. In one example, when a stream is experiencing a long load time (e.g., about 10 - 30 seconds or more), the computing device may display a notification indicating that the mobile device is loading the stream and / or is connected to the computing device, assuring the user that the device is connected to the network and advising the user to ensure that the mobile device is within range of the computing device. In another example, when the computing device and the mobile device fail to connect, the notification may inform the user of the connection failure, prompt the user to ensure that the mobile device and the computing device are connected to the network, and prompt the user to try reconnecting the devices. This notification may further inform the user that the computing device and the mobile device are connected to different networks and prompt the user to connect the computing device and the mobile device to the same network. The notification may further inform the user that the computing device and the mobile device are connected to different networks and prompt the user to connect the computing device and the mobile device to the same network. Additionally or alternatively, the notification may inform the user that the network is not supported (e.g., not available for streaming) and prompt the user to stream using a different network. The notification may further instruct the user to seek assistance. In a further example, when an application is not supported (e.g., cannot be streamed), the notification may inform the user by displaying a message such as "Not supported", "Not supported by the app", or "Cannot stream the app".In this regard, the different notifications presented can be generated based on reasons for which the application is not supported (e.g., security, technical limitations, developer opt-out, etc.). In an additional example, when the connection via the network is weak and slows down the stream (e.g., the quality deteriorates), the notification can inform the user that the stream may become slow.

[0072] Similarly, when the computing device and the mobile device fail to pair, the prompt (on one or both devices) can notify the user of the pairing failure and prompt the user to learn more about the potential problem. The prompt can also enable the user to end the pairing process. In another example, when the connection between the computing device and the mobile device is lost, the notification can inform the user that the connection has been lost, prompt the user to ensure that the mobile device and the computing device are connected to the network, and prompt the user to try to reconnect the devices. In an additional example, when a stream or a notification fails to start, the notification can inform the user of the start-up failure. In another example, when the stream cannot be loaded because the application does not support the streaming of the application, the notification can inform the user. In such an embodiment, the computing device can still generate mirrored notifications for the application. In an additional example, when streaming is hindered by an unknown error, a general error message can be presented to the reader via the notification.

[0073] As described above, during the pairing process, the user may be prompted to accept permissions as shown in FIGS. 10A-10B. In some embodiments, different permissions are required for application streaming and notification mirroring. For example, for notification mirroring, permissions may be granted for photos, media, and notifications. Message 1006A can be used to convey the required permissions and prompt the user to "permit" 1002 such information that the computing device accesses. For application streaming, permissions may be granted for the application. Message 1006B can be used to convey the required permissions and prompt the user to "permit" 1002 such information that the computing device accesses. The user may revoke the permission at any time. If the user revokes the permissions required to receive notifications and / or stream an application, the user may be prompted to accept the revoked permissions again.

[0074] In some embodiments, the system verifies one or more lock screen settings before starting a stream. A mobile device or computing device enables a lock screen when the device goes to sleep, is powered off, or is locked and the user may enter a personal identification number or passcode to access the device. When the mobile device does not have the lock screen enabled, the computing device may start streaming when prompted to start streaming. When neither the mobile device nor the computing device has the lock screen enabled, the computing device may start streaming when prompted to start streaming. When the mobile device has the lock screen enabled but the computing device does not, streaming may not start and a notification may be presented to the user prompting the user to enable the lock screen of the computing device.

[0075] In some embodiments, the computing device may stream an application that can make and receive video calls and / or voice calls. The call may be maintained while the stream is open on the computing device. If a user attempts to close the stream during a call, the user may receive a notification informing the user that the stream is closed, and the call may end. The notification may enable the user to close the stream or cancel closing the stream and continue the call.

[0076] Alternatively or additionally, in some embodiments, an active stream may enable a user to import and export information between the computing device. In one example, the user may copy and paste text, images, etc. between a stream from a mobile device and the computing device. Inputs on the computing device via one or more user inputs trigger the mobile device to send additional information including the material to be copied and / or pasted over the network. In another example, the user may interact with a link to a web page. In such an example, the user may interact with the link within the stream via one or more user interfaces. In response, the mobile device may send information over the network and enable the web page to be loaded on the computing device. In some scenarios, the web page may require signing into a user account to access the page. If the mobile device includes the user credentials for the web page, the mobile device may send the credentials to the computing device in addition to the link information.

[0077] Alternatively or additionally, in some embodiments, the system may be configured to switch the connection of the audio output between the mobile device and the computing device to facilitate the stability of the audio during app streaming. The audio switch may be triggered by an application stream that includes audio and an audio output, such as a headset connected to the network via the computing device or the mobile device. When switching from a device with a network-connected audio output to another device, a notification may be generated to inform the user of the switch. In one example, the headset may be connected to the computing device, and if the started stream includes audio, the headset connection may switch to the mobile device while the application is streaming to the computing device. The stream audio may be output to the headset via the mobile device, and the audio to the computing device may be paused while the headset remains connected to the mobile device. In another example, the headset may be connected to the mobile device, and if the started stream includes audio, the headset connection may stay with the computing device while the application is streaming to the computing device. In such an example, the audio from the stream may be output from the headset via the computing device.

[0078] In one embodiment, the connection of the audio output may not be switchable between a computing device and a mobile device. For example, if a headset is connected to a mobile device and the started stream includes audio, the headset connection may remain with the mobile device while the application is streaming to the computing device. The streamed audio may be output to the headset via the mobile device, and the audio to the computing device may be paused while the headset remains connected to the mobile device. In another example, if a headset is connected to a computing device and the started stream includes audio, the headset connection may remain with the computing device while the application is streaming to the computing device. In such an example, the audio from the stream may be output from the headset via the computing device.

[0079] In addition to or instead of the above examples and scenarios, the streaming process of an app on an external device can support the use of a camera, microphone, and / or speaker in a computing device and provide the captured information to a mobile device. This may be beneficial for certain apps such as messaging apps or video conferencing apps. Also, copy / paste events can be supported by a virtual remote display, enabling the user to perform those actions on a computing device and send the information back to the mobile device. Further, the present technology can support native handoff with filtering of various notifications. Here, for example, when a user receives a notification that includes a response field, the system can use that as a signal to indicate that it is associated with a messaging app, and as a result, selectively display the notification on the computing device. As an example, the system can pass information indicating that there is a chat message for the user while passing data and user credentials to other devices. In this way, other devices (e.g., computing devices) can handle the message through their own app version without having to go through a streaming approach. Or, in another example, hyperlinks can be provided within a chat message, and the user may be able to directly open the link within the browser of a computing device.

[0080] Figure 28 shows an exemplary method 2800. As shown in block 2802, the method includes pairing a first client device with a second client device via at least one wireless connection, where pairing enables the first client device and the second client device to communicate directly. In block 2804, the method includes receiving, by one or more processors of the first client device, an encoded virtual remote display of an application, where the encoded virtual remote display is generated by the second client device and the application is executed on the second client device. And in block 2806, the method includes controlling, by one or more processors of the first device, a graphical user interface (GUI) to present a virtual remote display of the application based on the encoded virtual remote display on the first client device, where the virtual remote display is configured to enable a user of the first client device to control an application executed on the second client device.

[0081] Figure 29 shows an exemplary method 2900 for streaming an application. As shown in block 2902, pairing a second client device with a first client device via at least one wireless connection, the pairing enabling the first client device and the second client device to communicate directly. In block 2904, the method includes generating an encoded virtual remote display of an application to be executed on the second client device by one or more processors of the second client device. In block 2906, the method includes transmitting the encoded virtual remote display of the application from the second client device to the first client device via at least one wireless connection. In block 2908, the method includes receiving, by the second client device, one or more user inputs associated with the application based on streaming of the application via the virtual remote display based on the encoded virtual remote display on the first client device. And in block 2910, the method includes performing an action associated with the application based on the one or more received user inputs by one or more processors of the second client device.

[0082] Specific examples herein refer to computing devices and mobile devices performing certain operations, but the technology can be used with different types of client devices and is not limited to the specific types of devices identified in the examples.

[0083] Although the technology herein has been described with reference to specific embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the technology. Accordingly, it is to be understood that many changes may be made to the exemplary embodiments and other configurations may be devised without departing from the spirit and scope of the technology as defined by the appended claims.

Claims

Claim 1 A method comprising: pairing a first client device with a second client device via at least one wireless connection, said pairing enabling direct communication between said first client device and said second client device, and said method further comprising: receiving, by one or more processors of said first client device, an encoded virtual remote display of an application by streaming from said second client device, said encoded virtual remote display being generated by said second client device, said application being one that runs on said second client device, and said method further comprising: controlling, by said one or more processors of said first device, a graphical user interface (GUI) to present a virtual remote display of said application based on said encoded virtual remote display on said first client device, said virtual remote display being configured to enable a user of said first client device to control said application running on said second client device. Claim 2 receiving, by said first client device, an indication of an event associated with said application running on said second client device; and generating, by said one or more processors of said first client device, a notification regarding said event for presentation via said GUI based on said received indication. The method of claim 1 further comprising. Claim 3 The notification is executable by the user of the first client device such that the method further comprises transmitting a signal to the second client device upon receipt of an input from the user, said signal being configured to cause the second client device to perform an action associated with the application. The method of claim 2. Claim 4 The method according to claim 1, further comprising storing an image of the virtual remote display of the application in a cache when the streaming of the application ends.

5. designating the application as one of a set of one or more recent applications when the streaming of the application ends; The method according to claim 4, further comprising generating a menu of the one or more recent applications for presentation by the GUI of the first client device.

6. The method according to claim 4, wherein when the one or more processors of the first client device determine that the first client device has become unpaired with the second client device, the method further comprises sending a signal to the second client device to end the stream.

7. ending the streaming after a certain period when the at least one wireless connection is not maintained; re-establishing the at least one wireless connection between the first client device and the second client device; The method according to claim 1, further comprising resuming the streaming following the re-establishing of the at least one wireless connection.

8. The method according to claim 1, further comprising generating, via the GUI of the first client device, at least one of an icon indicating that the second client device and the first client device are paired or an icon indicating that the streaming of the application is in progress.

9. The at least one wireless connection includes a first wireless connection and a second wireless connection different from the first wireless connection; the pairing is performed via the first wireless connection; The method according to claim 1, wherein receiving the virtual remote display of the application by the streaming is performed via the second wireless connection.

10. The method according to claim 9, further comprising transmitting, by the first client device via the second wireless connection, one or more user inputs received in response to the virtual remote display to the second client device.

11. Receiving, by streaming, the encoded virtual remote display of the application includes decoding, by one or more processors of the first client device, the encoded virtual remote display of the application, according to the method of claim 1.

12. The method according to claim 1, further comprising generating at least one prompt via the GUI of the first client device to facilitate pairing between the first client device and the second client device.

13. A method for streaming an application, comprising pairing a second client device to a first client device via at least one wireless connection, the pairing enabling the first client device and the second client device to communicate directly, and the method further comprising generating, by one or more processors of the second client device, an encoded virtual remote display of an application running on the second client device; transmitting, by streaming via the at least one wireless connection, the encoded virtual remote display of the application from the second client device to the first client device; receiving, by the second client device, one or more user inputs associated with the application based on a presentation of the application via a virtual remote display based on the encoded virtual remote display on the first client device; performing, by one or more processors of the second client device, an action associated with the application based on the received one or more user inputs.

14. The method of claim 13, further comprising, before said transmitting, encoding, by one or more processors of said second client device, said virtual remote display of said application. **Claim 15** The method of claim 13, wherein performing said action associated with said application includes updating said encoded virtual remote display of said application. **Claim 16** The method of claim 13, further comprising generating, by one or more processors of said second client device, a graphical user interface (GUI) including at least one of a permission prompt to facilitate said pairing or an actual display of said application for presentation to a user. **Claim 17** The method of claim 16, wherein said virtual remote display generated by one or more processors of said second client device has a visual appearance different from an actual display of said application. **Claim 18** The method of claim 13, wherein said encoded virtual remote display is generated by one or more processors of said second client device when a display device of said second client device is in a sleep mode. **Claim 19** The method of claim 13, wherein said encoded virtual remote display is generated by one or more processors of said second client device while another application is presented on a display device of said second client device. **Claim 20** The method of claim 13, further comprising, in response to an end of said streaming, storing, by one or more processors of said second client device, an image of said virtual remote display of said application in a cache. **Claim 21** Said at least one wireless connection includes a first wireless connection and a second wireless connection different from said first wireless connection, wherein said pairing is performed via said first wireless connection. Transmitting the encoded virtual remote display of the application from the second client device is performed via the second wireless connection, the method according to claim 13.

22. A computing device configured to stream an application from a remote device, One or more processors; One or more user inputs operably coupled to the one or more processors; A communication module operably coupled to the one or more processors, The communication module, configured to facilitate pairing the computing device with the remote device via at least one wireless connection, the pairing enabling direct communication between the computing device and the remote device, and the communication module further, configured to receive an encoded virtual remote display of an application from the remote device via a stream, the application being one executed on the remote device, The one or more processors, generating a graphical user interface (GUI) to present the application, including presenting on the GUI virtual remote display of the application executed on the remote device based on the encoded virtual remote display; receiving input associated with the virtual remote display from the one or more user inputs; transmitting information regarding the received input to the remote device to control the application on the remote device via the communication module. A computing device configured to perform.

23. A computing device configured to stream an application to a remote device, One or more processors; One or more user inputs operably coupled to the one or more processors; A communication module operably coupled to the one or more processors, The communication module is configured to facilitate pairing of the computing device with the remote device via at least one wireless connection, and the pairing enables the computing device and the remote device to communicate directly. The one or more processors are configured to generate an encoded virtual remote display of an application of the computing device, the application being executed on the computing device, and the one or more processors further cause the communication module to transmit the encoded virtual remote display of the application to the remote device via the at least one wireless connection by streaming. receive, from the remote device via the communication module, one or more user inputs associated with the application, based on a presentation of the virtual remote display of the application on the remote device based on the encoded virtual remote display. A computing device configured to perform an action associated with the application based on the received one or more user inputs.

Images