External application streaming

Abstract

The present technology provides a system capable of pairing a first client device (102, 602) with a second client device (302, 604) via at least one wireless connection, thereby allowing the client devices to communicate directly; generating, by one or more processors of a second client device (402), an encoded virtualized remote display of an application, the application being running on a second client computing device (302, 604), and transmitting the encoded virtualized remote display of the application via at least one wireless connection; streaming the application on the first client device (102, 602) includes presenting on the first client device (102, 602) a virtualized remote display (1502, 1902, 2302) of the application based on the encoded virtualized remote display on a graphical user interface (GUI) (106, 1200, 1900, 2300, 2500) to enable a user of the first client device (102, 602) to control the application running on the second client device (302, 604).

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to the filing date of U.S. Provisional Patent Application No. 63 / 340,226, filed on May 10, 2022, the entire disclosure of which is incorporated herein by reference. Background Art

[0002] In a system, multiple devices can be networked together to allow users to access features of different devices throughout the system. Due to incompatibilities across different types of devices, such access may have limited functionality. For example, a program designed to operate on one device or platform may not operate smoothly on a different device or platform. Even if a program can run on multiple platforms, technical issues may arise in which a certain functionality is lost due to incompatible user interfaces. Summary of the invention

[0003] Aspects of the present technology provide a technical solution for implementing application streaming from one computing device to different computing devices. For example, this includes wireless transmission of a messaging app (such as a chat or text messaging app) from a user's mobile phone (e.g., a second client computing device) to their laptop computer (e.g., a first client computing device). For different client devices, many other configurations are also possible. Graphical user interfaces (GUIs) on different client devices can help support seamless user experiences (UX), which promote more effective and efficient communication between users and others. The second client computing device can create an encoded virtualized remote display for its currently running app. The encoded virtualized remote display is sent to the first client computing device, where a virtualized remote display based on the encoded virtualized remote display is presented via the GUI of the device. The virtualized remote display at the first device without presenting the same information as that shown in the GUI of the second device can receive one or more user inputs relayed back to the first device. This provides technical benefits by enabling control of the app without the user having to use the first device directly.

[0004] According to one aspect, a method includes: pairing a first client device with a second client device via at least one wireless connection, wherein the pairing allows the first client device and the second client device to communicate directly; receiving, by one or more processors of the first client device, an encoded virtualized remote display of an application from the second client device via streaming, the encoded virtualized remote display being generated by the second client device, and the application running on the second client device; and controlling, by one or more processors of the first device, a graphical user interface (GUI) to present, on the first client device, a virtualized remote display of the application based on the encoded virtualized remote display, the virtualized remote display being configured to enable a user of the first client device to control the application running on the second client device.

[0005] The method may further include: receiving, by the first client device, an indication of an event associated with an application running on the second client device; and generating, by one or more processors of the first client device, a notification regarding the event based on the received indication for presentation via the GUI. Here, the notification may be actionable by a user of the first device, such that upon receiving input from the user, the method further includes transmitting a signal to the second client device, wherein the signal is configured to cause the second client device to perform an action associated with the application.

[0006] Alternatively or additionally, upon terminating the streaming of the application, the method may include causing an image of the virtualized remote display of the application to be cached. Here, upon terminating the streaming of the application, the method may include designating the application as a recent application in a set of one or more recent applications, and generating a menu of one or more recent applications for presentation by the GUI of the first client device. Alternatively or additionally, the method may further include: upon determining that the first client device is no longer paired with the second client device, the one or more processors of the first client device sending a signal to the second client device for terminating the streaming.

[0007] Alternatively or additionally, the method may further include: causing the streaming to terminate after a period of time when the at least one wireless connection is not maintained; reestablishing the at least one wireless connection between the first client device and the second client device; and causing the streaming to resume after the at least one wireless connection is reestablished. Alternatively or additionally, the method may further include generating, via the GUI of the first client device, at least one of: an icon indicating that the second client device is paired with the first client device, or an icon indicating that streaming of the application is occurring.

[0008] Alternatively or additionally, at least one wireless connection may include a first wireless connection and a second wireless connection different from the first wireless connection. Here, the pairing occurs via the first wireless connection, and the encoded virtualized remote display of the application is received via streaming via the second wireless connection. In this case, the method may further include: transmitting, by the first client device, to the second client device via the second wireless connection, one or more user inputs received in response to the virtualized remote display.

[0009] Alternatively or additionally, receiving the encoded virtualized remote display of the application via streaming includes decoding the encoded virtualized remote display of the application by one or more processors of the first client device.Alternatively or additionally, the method may further include generating at least one prompt to facilitate pairing between the first client device and the second client device via a GUI of the first client device.

[0010] According to another aspect, an application streaming method includes: pairing a second client device with a first client device via at least one wireless connection, wherein the pairing allows the first client device to communicate directly with the second client device; generating, by one or more processors of the second client device, an encoded virtualized remote display of an application running on the second client device; streaming the encoded virtualized remote display of the application from the second client device to the first client device via at least one wireless connection; receiving, by the second client device, one or more user inputs associated with the application based on presentation of the application on the first client device via the virtualized remote display based on the encoded virtualized remote display; and taking, by one or more processors of the second client device, an action associated with the application based on the received one or more user inputs.

[0011] The method may further include encoding, by one or more processors of the second client device, the virtualized remote display of the application prior to transmission. Alternatively or additionally, taking an action associated with the application may include updating the encoded virtualized remote display of the application. Alternatively or additionally, the method may further include generating, by one or more processors of the second client device, a graphical user interface (GUI) comprising at least one of: a permission prompt for facilitating pairing, or an actual display of the application for presentation to the user. Here, the encoded virtualized remote display generated by the one or more processors of the second client device may have a visual appearance different from the actual display of the application.

[0012] Alternatively or additionally to the method, an encoded virtualized remote display may be generated by one or more processors of the second client device when the display device of the second client device is in sleep mode. Alternatively or additionally to the method, an encoded virtualized remote display may be generated by one or more processors of the second client device when another application is presented on the display device of the second client device. Alternatively or additionally, the method may further include, in response to termination of streaming, one or more processors of the second client device caching an image of the virtualized remote display of the application. Alternatively or additionally, at least one wireless connection may include a first wireless connection and a second wireless connection that is different from the first wireless connection. Here, the pairing may occur via the first wireless connection, and the transmission of the encoded virtualized remote display of the application from the second client device occurs via the second wireless connection.

[0013] According to a further aspect, a computing device is configured to stream an application from a remote device. The computing device includes one or more processors, one or more user inputs operatively coupled to the one or more processors, and a communication module operatively coupled to the one or more processors. The communication module is configured to: facilitate pairing of the computing device with the remote device via at least one wireless connection, wherein the pairing allows the computing device to communicate directly with the remote device, and receive an encoded virtualized remote display of the application from the remote device via streaming, the application being run on the remote device. The one or more processors are configured to: generate a graphical user interface (GUI) to present the application, including presenting a virtualized remote display based on the encoded virtualized remote display of the application running on the remote device on the GUI; receive input associated with the virtualized remote display from the one or more user inputs; and transmit information about the received input to the remote device via the communication module to control the application on the remote device.

[0014] According to another aspect, a computing device is configured to stream an application to a remote device. The computing device includes one or more processors, one or more user inputs operatively coupled to the one or more processors, and a communication module operatively coupled to the one or more processors. The communication module is configured to facilitate pairing of the computing device with the remote device via at least one wireless connection, wherein the pairing allows the computing device to communicate directly with the remote device. The one or more processors are configured to generate an encoded virtualized remote display of an application of the computing device, which is running on the computing device; cause the communication module to stream the encoded virtualized remote display of the application to the remote device via at least one wireless connection; based on the presentation of the virtualized remote display of the application based on the virtualized remote display on the remote device, receive one or more user inputs associated with the application from the remote device via the communication module; and take an action associated with the application based on the one or more user inputs received. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A client computing device in accordance with aspects of the present technology is shown.

[0016] Figure 2 is a block diagram of a client computing device according to aspects of the present technology.

[0017] Figure 3 A client mobile device in accordance with aspects of the present technique is shown.

[0018] Figure 4 is a block diagram of a client mobile device according to aspects of the present technology.

[0019] FIG. 5A to FIG. 5B A system for use with aspects of the present technique is shown.

[0020] Figure 6 A system for use with aspects of the present technique is shown.

[0021] Figure 7 Example prompts in accordance with aspects of the present technique are shown.

[0022] Figure 8 Example prompts in accordance with aspects of the present technique are shown.

[0023] Fig. 9 Example prompts in accordance with aspects of the present technique are shown.

[0024] FIG. 10A to FIG. 10B Example prompts in accordance with aspects of the present technique are shown.

[0025] FIG. 11A to FIG. 11B Example prompts and connections in accordance with aspects of the present technique are shown.

[0026] FIG. 12A to FIG. 12B An example scenario in accordance with aspects of the present technique is shown.

[0027] Fig.13 Example notifications in accordance with aspects of the present technology are shown.

[0028] Fig.14 An example scenario in accordance with aspects of the present technique is shown.

[0029] Fig.15 An example flow in accordance with aspects of the present technique is shown.

[0030] Fig.16 An example scenario in accordance with aspects of the present technique is shown.

[0031] Fig.17 An example scenario in accordance with aspects of the present technique is shown.

[0032] Fig.18 Example notifications in accordance with aspects of the present technology are shown.

[0033] Fig.19 An example flow in accordance with aspects of the present technique is shown.

[0034] Fig. 20 An example scenario in accordance with aspects of the present technique is shown.

[0035] Fig.21 Example notifications in accordance with aspects of the present technology are shown.

[0036] Fig. 22 Example notifications in accordance with aspects of the present technology are shown.

[0037] Fig.23 An example scenario in accordance with aspects of the present technique is shown.

[0038] Fig.24 Example notifications in accordance with aspects of the present technology are shown.

[0039] Fig.25 An example scenario in accordance with aspects of the present technique is shown.

[0040] FIG. 26A to FIG. 26B An example scenario in accordance with aspects of the present technique is shown.

[0041] FIG. 27A to FIG. 27B An example multi-touch interaction scenario in accordance with aspects of the present technology is shown.

[0042] Fig.28 Example methods according to aspects of the present technology are shown.

[0043] Fig.29 Example methods according to aspects of the present technology are shown. DETAILED DESCRIPTION

[0044] The technology can provide application streaming from different devices (second device) of the user on one computing device (first device) of the user. The user can control the operation of the streaming app (e.g., messaging app, streaming app, social media, game, news, etc.) through a virtualized remote display presented via the GUI of the first device. The virtualized remote display can be based on an encoded virtualized remote display generated and sent from the second device. The encoded virtualized remote display can include encoded app content (e.g., audio, video, etc.). The encoder used by the second device can be, for example, an H.264 or MPEG-4 AVC encoder to ensure the quality of the streaming content. This method solves the technical problem of supporting a given app on different user devices without the app having to be downloaded to each device or otherwise running on each device. In one scenario, the technical solution pairs the user device via a first type of communication link (e.g., a Bluetooth connection), and provides streaming app and user instructions back to the second device via a second type of communication link (e.g., a WiFi or NFC connection). The virtualized remote display need not present the same information as that shown in the GUI of the second device, and in some cases, the app can run in the background (or when the device's display is turned off) on the second device. In this regard, the virtualized remote display can provide a stylized view (e.g., a view compatible with the GUI, input, etc. of the first device) or representation of the app on the GUI of the first device, which enables the user to more easily control the app using input from the first device. The input is delivered to the second device on which the virtualized remote display is generated. This provides a technical benefit by enabling control of the app without the user having to use the second device directly. In this regard, since a stylized view or representation of the app is utilized instead of an identical copy of the GUI of the second device, less bandwidth / overhead and smaller memory requirements may be required when controlling the app. Therefore, real-time or near-instantaneous control of the app on the second device using input from the first device can be supported. Example System

[0045] Unless otherwise stated, the disclosed examples and scenarios are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the description of the implementation should be made in an illustrative manner rather than in a manner that limits the subject matter defined by the claims. In addition, the examples described herein and the provision of clauses expressed as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; on the contrary, the examples are intended to illustrate only one of many possible implementations. In addition, the same reference numerals in different figures may identify the same or similar elements. Example computing device

[0046] Figure 1 An example 100 showing a client computing device 102 (such as a laptop computer) is shown. In this example, there is a display 104 for visually presenting content via a GUI 106, and a user input portion 108 having a keyboard 110 and a touchpad 112 as different user inputs. A microphone (not shown) may be provided for audible input. The display 104 may be configured as a user input (e.g., a touch screen). The keyboard 110 may be a virtual keyboard, such as part of another display. Although one touchpad 112 is shown, the device may have two or more touchpads arranged thereon. The device may include additional user inputs (e.g., a mouse, audio, and / or gesture-based input).

[0047] The integrated webcam 114 can be used for video conferencing, interactive gaming, etc. An indicator 116 (such as an LED) can light up to alert the user whenever the integrated webcam is in use. The client device can also include one or more other sensors 118 that can be used to supplement the visual information obtained by the webcam 114. For example, the one or more other sensors can include additional imaging devices, RF or ultrasonic based motion sensors, such as to help recognize a user's gestures or act as a human presence detector.

[0048] Figure 2 A block diagram of an example client computing device 200 is shown, such as Figure 1The client computing device 200 includes a laptop computer (e.g., a tablet PC or a netbook), a desktop PC, a home device (such as a smart display), etc. As shown, the client computing device 200 includes a processing module 202 having one or more computer processors (such as a central processing unit 204 and / or a graphics processor 206) and a memory module 208 configured to store instructions 210 and data 212. The processors may or may not operate in parallel and may include ASICs, controllers, and other types of hardware circuits. The processor is configured to receive information from a user through a user interface module 214 and present information to the user on one or more display devices of a display module 216 having a display interface.

[0049] The user interface module 214 can receive commands or other input information from the user via the user input unit, and convert them to submit to a given processor. The user interface module can be linked to a web browser and other applications executed by the processing module 202 of the computing device. The user input unit can include one or more of a touch screen, a small keyboard, a touch pad, a stylus, a microphone, or other types of input devices. The display module 216 may include a suitable circuit system for driving a display device to present graphic information and other information to the user. For example, the graphic information can be generated by the graphics processor 206, and the CPU 204 manages the overall operation of the client device 200. The graphic information can be displayed on the display module 216. Responses or other content to user queries. For example, the processing module can use the instructions and data stored in the memory module 208 to run a browser application, a messaging application, or other services, and present information associated with the browser application or other services to the user via the display module 216. The memory module may include a database or other storage for application-related information, etc.

[0050] The memory module 208 may be implemented as one or more of: one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. The memory module 208 may include, for example, flash memory and / or NVRAM, and may be embodied as a hard drive or a memory card. Alternatively or in addition, the memory module 208 may also include removable media (e.g., a DVD, CD-ROM, or USB thumb drive). One or more areas of the memory module 208 may be writable, while other areas may include read-only (or otherwise write-protected) memory. In one implementation, the computer program product is tangibly embodied in an information carrier. Although Figure 2The processor, memory module, and other elements of the client computing device 200 are functionally shown as being within the same integral block, but such components may or may not be stored within the same physical housing. For example, some or all of the instructions and data may be stored on an information carrier that is a removable storage medium (e.g., an optical drive, a high-density tape drive, or a USB drive), and other instructions and data may be stored within a read-only computer chip.

[0051] Data 212 may be retrieved, stored, or modified by the processor in accordance with instructions 210. For example, the data may be stored in a computing device register, as a table with multiple different fields and records, in a relational database, as an XML document, or a flat file. The data may also be formatted in any computing device readable format.

[0052] Instructions 210 may be any set of instructions to be executed directly by a processor, such as machine code, or any set of instructions to be executed indirectly, such as a script. For example, the instructions may be stored as computing device code on a computing device readable medium. In this regard, the terms "instructions" and "programs" may be used interchangeably herein. The instructions may be stored in an object code format for direct processing by a processor, or in any other computing device language, including scripts or independent source code module collections that are interpreted on demand or compiled in advance.

[0053] like Figure 2 As shown, the client device 200 includes a communication module 218 for communicating with other devices and systems (including other client devices, servers, and databases). The communication module 218 includes a wireless transceiver; optionally, the module may alternatively or additionally include a wired transceiver. The client device 200 can communicate with other remote devices via the communication module 218 using various configurations and protocols, including short-range communication protocols such as near field communication (NFC), Bluetooth™, Bluetooth™ Low Energy (BLE) or other ad hoc networks, the Internet, an intranet, a virtual private network, a wide area network, a local network, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, and combinations of the foregoing.

[0054] The example client device 200 shown in the figure also includes one or more position and orientation sensors 220. The position and orientation sensor 220 is configured to determine the position and orientation of one or more parts of the client computing device 200. For example, these components may include a GPS receiver or other global positioning component for determining the latitude, longitude and / or altitude of the device, as well as an accelerometer, a gyroscope or another direction / speed detection device, such as an inertial measurement unit (IMU). The client device 200 may also include one or more cameras 222 (such as the integrated webcam discussed above), a speaker 224 for capturing still images and recording video streams. A power module 226 provides power to various system components. In addition, the user input unit includes a keyboard 228 and a microphone 230, which may include a microphone array having one or more transducers or other microphone elements distributed at various points along the housing of the computing device. The touch pad 232 may include a capacitive touch sensor unit 234a or a piezoelectric sensor unit 234b (or both). Example Mobile Device

[0055] Figure 3 An example 300 is shown, which shows a handheld client mobile device 302, such as a smart phone, a personal digital assistant (PDA), or a wearable device (e.g., a smart watch). In this example, there is a display 304 for visually presenting content. The display 304 can also be configured as a user input portion (e.g., a touch screen). A microphone 306 can be provided for audible input. An integrated camera 308 can be used for taking pictures, video conferencing, interactive games, etc. An indicator 310 (such as an LED) can light up to remind the user whenever the integrated camera 308 is used.

[0056] Figure 4 An example handheld client mobile device 400 (such as Figure 3 4. As shown in the figure, the client mobile device 400 includes a processing module 402 having one or more processors (such as a central processing unit 404 and / or a graphics processor 406), and a memory module 408 configured to store instructions 410 and data 412. The processors may or may not operate in parallel and may include ASICs, controllers, and other types of hardware circuits. The processor is configured to receive information from a user through a user interface module 414 and present information to the user on one or more display devices of a display module 416 having a display interface.

[0057] The user interface module 414 can receive commands or other input information from the user via the user input unit, and convert them to submit to a given processor. The user interface module can be linked to a web browser and other applications executed by the processing module 402 of the mobile device. The user input unit can include one or more of a touch screen, a stylus, a microphone or other types of input devices. The display module 416 may include a suitable circuit system for driving a display device to present graphic information and other information to the user. For example, the graphic information can be generated by a graphics processor 406, and the CPU 404 manages the overall operation of the client mobile device 400. The graphic information can be displayed on the display module 416 to respond to user queries. For example, the processing module can use the instructions and data stored in the memory module 408 to run an application or other services, and present information associated with the application or other services to the user via the display module 416. The memory module may include a database or other storage for application related information, etc.

[0058] The memory module 408 may be implemented as one or more of the following: one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. The memory module 208 may include, for example, flash memory and / or NVRAM, and may be embodied as a hard drive or a memory card. One or more regions of the memory module 408 may be writable, while other regions may include read-only (or otherwise write-protected) memory.

[0059] Data 412 may be retrieved, stored, or modified by the processor in accordance with instructions 410. For example, the data may be stored in a computing device register, as a table with multiple different fields and records, in a relational database, as an XML document, or a flat file. The data may also be formatted in any computing device readable format.

[0060] Instructions 410 may be any set of instructions to be executed directly by a processor, such as machine code, or any set of instructions to be executed indirectly, such as a script. For example, the instructions may be stored as mobile device code on a mobile device readable medium. In this regard, the terms "instructions" and "programs" may be used interchangeably herein. The instructions may be stored in an object code format for direct processing by a processor, or in any other computing device language, including scripts or independent source code module collections that are interpreted on demand or compiled in advance.

[0061] Also like Figure 4As shown, the client mobile device 400 includes a communication module 418 for communicating with other devices and systems (including other client devices, servers, and databases). The communication module 418 includes a wireless transceiver. The client device 400 can communicate with other remote devices via the communication module 418 using various configurations and protocols, including short-range communication protocols such as near field communication (NFC), Bluetooth™, Bluetooth™ low energy consumption (BLE) or other ad hoc networks, the Internet, an intranet, a virtual private network, a wide area network, a local network, a private network using one or more company-specific communication protocols, WiFi and HTTP, and combinations of the foregoing.

[0062] The example client device 400 shown in the figure also includes one or more position and orientation sensors 420. The position and orientation sensor 420 is configured to determine the position and orientation of one or more parts of the client mobile device 400. For example, these components may include a GPS receiver or other global positioning component for determining the latitude, longitude and / or altitude of the device, as well as an accelerometer, a gyroscope or another direction / speed detection device, such as an inertial measurement unit (IMU). The mobile device 400 may also include one or more cameras 422 (such as the integrated camera discussed above) for capturing still images and recording video streams, a speaker 424. A power module 426 provides power to various system components. In addition, the user input portion may include a microphone 430, which may include a microphone array having one or more transducers or other microphone elements distributed at various points along the housing of the computing device. Example Network

[0063] One or more devices may be connected to each other via a network to stream applications or apps, as discussed further herein. Figure 5A and Figure 5B502 are respectively an intuitive diagram and a functional diagram of an example system 500, which includes multiple computing devices and databases connected via a network. For example, computing device 502 may be a cloud-based server system that provides or otherwise supports one or more apps, games, or other programs. Database 504 may store messaging app information, game data, user profile information, and / or other information. The server system may access the database via network 506. Client devices may include one or more of a desktop computer 508, a laptop or tablet PC 510, and a home device such as a smart display 512. Other client devices may include a personal communication device such as a mobile phone or PDA 514 or a wearable device 516 such as a smart watch. Another example client device is a large screen display or interactive whiteboard 518 such as may be used in a classroom, conference room, auditorium, or other collaborative gathering space where multiple users may be present. Additional client devices may be a pair of headphones or earbuds (not shown).

[0064] In one example, computing device 502 may include one or more server computing devices having multiple computing devices, such as a load balancing server farm or a cloud computing system, that exchange information with different nodes of a network to receive, process, and transmit data to and from other computing devices. For example, computing device 502 may include one or more server computing devices capable of communicating with any of computing devices 508-518 via network 506. This may be done as part of hosting one or more collaborative apps (e.g., a video conferencing program, an interactive spreadsheet app, or a multiplayer game) or services (e.g., a movie streaming service or an interactive game show where viewers can provide comments or other feedback).

[0065] like Figure 5BAs shown, each of the computing devices 502 and 508-518 may include one or more processors, memory, data and instructions. The memory stores information accessible by one or more processors, including instructions and data that can be executed or otherwise used by the processor. The memory can be any type capable of storing information accessible by the processor, including computing device readable media. The memory is a non-temporary medium, such as a hard drive, a memory card, an optical disk, a solid state, etc. The system may include different combinations of the foregoing, thereby storing different parts of the instructions and data on different types of media. The instruction can be any instruction set (such as machine code) that will be directly executed by the processor or any instruction set (such as a script) that is indirectly executed. For example, the instruction can be stored as a computing device code on a computing device readable medium. In this regard, the terms "instruction", "module" and "program" can be used interchangeably herein. The instruction can be stored in an object code format for direct processing by the processor, or stored in any other computing device language, which includes a script or independent source code module set that is interpreted or compiled in advance on demand.

[0066] The processor may be any conventional processor, such as a commercially available CPU. Alternatively, each processor may be a dedicated device, such as an ASIC, a graphics processing unit (GPU), a tensor processing unit (TPU), or other hardware-based processor. Figure 4 B functionally illustrates the processor, memory, and other elements of a given computing device as being located within the same box, but such a device may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, the memory may be a hard drive or other storage medium located in a housing different from that of the processor (e.g., located in a cloud computing system such as server 1402). Therefore, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0067] The computing device may include all components that are typically used in connection with the computing device, such as the processor and memory described above, and a user interface subsystem for receiving input from a user and presenting information (e.g., text, images, and / or other graphical elements) to the user. The user interface subsystem may include one or more user input units (e.g., at least one front (user) camera, mouse, keyboard, touch screen, and / or microphone) and one or more display devices operable to display information (e.g., text, images, and / or other graphical elements). Other output devices, such as speakers, may also provide information to the user.

[0068] The user-related computing devices (e.g., 508-518) can communicate with the backend computing system (e.g., server 502) via one or more networks (such as network 506). Network 506 and intermediate nodes may include various configurations and protocols, including short-range communication protocols such as Bluetooth™, Bluetooth LE™, the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, and various combinations of the foregoing. Such communications may be facilitated by any device capable of transferring data to and from other computing devices, such as a modem and a wireless interface.

[0069] In some implementations, user-related computing devices may also communicate with each other without also communicating with a backend computing system. Figure 6 System 600 is shown in which a client computing device 602 (e.g., a laptop computer) and a client mobile device 604 (e.g., a mobile phone) are wirelessly connected via a network 606. The client computing device 602 may be a computer system that is connected to a server or a mobile device. Figure 1 to Figure 2 Similarly, the client mobile device 604 may be a client computing device of Figure 3 to Figure 4 The client mobile device discussed. Network 606 and the intermediate nodes may include the same configuration as network 506. Alternatively or in addition, network 606 may include one or more ad hoc and / or peer-to-peer communication links between devices 602 and 604. Here, there will be no intermediate nodes. Example Method pair

[0070] When initiating an application streaming process according to aspects of the present technology, the client mobile device and the client computing device may connect to each other using a first wireless connection (e.g., a Bluetooth™ connection or other ad hoc connection). When the connection is initiated, one of the user's computing devices (the mobile device or the computing device) may generate a prompt for the user to pair the devices (a process that allows two or more devices to communicate directly when within range of each other), such as Figure 7 to Figure 1 1. Figure 7, a first prompt 700 in a GUI (here, on a user computing device) may provide a message 708 to inform the user about the functionality of pairing. The first prompt 700 may include a link contained in the text "Learn more" 702, allowing the user to access additional information, such as user agreements, pairing details, supported applications, etc. The first prompt 700 may additionally include a link contained in the text "Cancel" 706 to terminate the process, and a link contained in the text "Next" 704 to proceed to the next prompt.

[0071] After the user selects "Next" 704 using one or more user interfaces of the computing device (e.g., a touch screen, a touchpad, a mouse, a keyboard, a microphone, or a gesture, etc.), the device may generate a second prompt 800 in the GUI, such as Figure 8 As shown. The second prompt 800 may provide a message 808 and include one or more selections 810, 812 for enabling the user to establish privacy protection measures. The user may choose to enable password protection, pin protection, or both, and the user may be required to enter password protection, pin protection, or both when the mobile device, computing device, or both enters sleep (e.g., components such as the screen are turned off). The second prompt 800 may similarly include "Learn more" 802, "Cancel" 806, and "Next" 804 links with which the user may interact. In some implementations, if the user has previously established a password, pin, or both, the second prompt 800 may not be generated.

[0072] After the user makes selections 810, 812, the user may select "Next" 804. The computing device may then generate a third prompt 900 in the GUI, such as Fig. 9 . The third prompt may similarly include “Learn more” 902 and “Cancel” 906 links. The third prompt 900 allows the user to continue the pairing process on the mobile device through a message 908, which may, for example, state “Go to your phone to review a few more permissions”. The third prompt 900 also includes a link contained in the text “Done” 904 for exiting the pairing process upon completion. The link “Done” 904 may be inactive until the user completes the permissions process.

[0073] In this scenario, after the third prompt 900 is generated on the user computing device, at least one permission prompt 1000 may be generated on the user mobile device, such as FIG. 10A to FIG. 10BThe user can allow or deny permissions (e.g., access to photos and media, notifications, and apps) via the “Allow” 1002 or “Don’t allow” 1004 links. Message 1006 can inform the user what permissions are being requested. Here, Fig. 10A As shown in 1006A in FIG. 1 , the message portion may indicate the type of device that is allowed to access, as well as other information, such as notifications and / or photos and media. Fig. 10B As shown in 1006B in FIG. 1 , the message portion may indicate the device type as described above, but also present different information, such as an App notification about the streaming device's app. Additional information about the streaming implementation may be found by clicking on icon 1008. If the user selects "Don't allow" 1004, the pairing process terminates. If the user selects "Allow" 1002, a fourth prompt 1100 may be generated in the GUI on the computing device, such as Fig.11A The fourth prompt 1100 presents a message 1108 informing the user that the mobile device and the computing device have been paired and includes a link contained in the text "Done" 1104 for exiting the pairing process upon completion. In some implementations, if the user has previously accepted permission for the computing device to access the mobile device, a permission prompt may not be generated on the mobile device. When there is an active connection, the mobile device GUI may show a "Connected Devices" screen 1120, such as Fig. 11B The screen may show previously connected devices at 1122, other devices at 1124, and an option for pairing a new device at 1126.

[0074] After the user selects "done" or otherwise completes the process, the mobile device and the computing device can remain paired and can share content within the direct communication range of each other. Depending on the type of connection (e.g., Bluetooth™ or Bluetooth LE™ link), the range can be within 2-5 meters or more. According to one aspect, the connection is maintained to keep the devices paired. The computing device, the client device, or both may test the network connection periodically. If no connection is detected, the computing device, the client device, or both may display a "timed out" message. The connection may not be detected due to, for example, the devices being out of range of each other, interference with the communication channel, etc. If the connection is reestablished, the message may be lost. Otherwise, the connection may be terminated, and in a scenario, the computing device and the mobile device may be automatically reconnected when they are within range of each other again. If the device is out of range or otherwise unable to connect, a notification may be generated to communicate this to the user.

[0075] Pairing of the computing device and the mobile device allows the user to access features of the mobile device through app streaming via a GUI on the computing device. Fig. 12A As shown, the mobile device interface 1202 is located in a GUI on the display 1200 of the computing device. The interface 1202 may include features such as app selection 1204, recent photo selection 1206, phone control features 1208 (e.g., hotspot control, phone volume control, locate phone, etc.), mobile device name 1209, mobile device status icon 1210 (e.g., power level, mobile network connection), and mobile device settings menu link 1212. App selection 1204 may include a link and corresponding icon 1205 that can direct the user to the app list. In some cases, app selection 1204 may be a recent app selection. Alternatively, app selection 1204 may be a favorite app selection. In this regard, the user may select app selection 1204 as one of the recent app selections or favorite app selections via the settings menu. In some implementations, the computing device may also display an icon 1214 contained in a taskbar of the computing device, which indicates that the mobile device has been paired with the computing device and is currently connected to the computing device. Icon 1214 may also indicate when the device is paired but not currently connected. Both the computing device and the mobile device may additionally include a paired device list in the settings menu. Once paired, the computing device may show the mobile device as paired, and vice versa. In addition, the corresponding list may indicate whether the device is currently connected to another device.

[0076] In some implementations, the link and corresponding icon 1205 can direct the user to a list of apps. Fig. 12B An example app list 1250 is shown in FIG. Fig. 12B An app list 1250 is shown in a grid format, but the app list can also be in the form of a row-by-row list (e.g., one icon or app per row). In some cases, the app list 1250 can allow a user to view personal apps and work apps separately. In this regard, a user can select or mark an app as a work app. When viewing work apps, the selected or marked work apps can be displayed in the app list. When viewing personal apps, apps that are not selected or marked as work apps can be displayed in the app list. Additionally or alternatively, the app list can include a favorites list. The favorites list can include apps selected or marked as favorites by the user. Application Streaming

[0077] After pairing the computing device with the mobile device via a first wireless connection, the user can start streaming the application as long as the paired connection is maintained. Here, the system (mobile device, computing device, or both) can check periodically to ensure that the paired connection is maintained (e.g., every 10-30 seconds, or longer, or shorter). According to one aspect, the streaming of the app can use a second wireless connection, such as a Web Real-Time Communication (WebRTC) connection. The connection can use a different wireless connection, such as a WiFi connection. In one implementation, the first wireless connection can be used to ensure proximity between the computing device and the mobile device, while the second wireless connection can be used to transmit stream information between the devices. The system can inform the user that the paired connection has been lost (e.g., the mobile device and the computing device are out of the Bluetooth™ range), and a timeout can be set, wherein if the streaming connection is not reestablished before the timeout is reached, the streaming connection can be cut off. Here, when the devices return to each other's range, the system can automatically reestablish the paired connection.

[0078] Application streaming on a user computing device may be initiated in a variety of ways and may be managed by the device's operating system via the device's processor executing one or more modules. In one example, an application notification 1302 may be generated on a GUI of a computing device display 1300, such as Fig.13 As shown. In this example, notification 1302 is a mirrored notification, which means that the notification can be displayed on both the computing device and the mobile device. The mirrored notification can look the same on both displays, or have different appearances (e.g., due to screen size and / or orientation). The user can interact with notification 1302 using the user interface. For example, in addition to ignoring the notification, the user can also dismiss (and this dismissal can also apply to a connected phone or other device running the app), reply in-line (i.e., without leaving the notification), or tap the notification to stream the complete app from the other device. For example, the app can be a messaging app. Fig.13 1304, "MARK AS READ" 1306, or select the body of the notification. In this example, if the user selects "MARK AS READ" 1306, the message may be marked as read, which means that the message notification may no longer appear on the mobile device or computing device.

[0079] If the user selects "REPLY" 1304 or selects the body of the notification, the messaging application may be streamed on the computing device. Fig.14As shown in example 1400 of FIG. 1 , an icon 1402 with a circular progress bar 1404 is displayed on a computing device, indicating the loading status of the application flow. Icon 1402 can be located on any portion of the computing device display. In an implementation that includes an online reply feature (described below with respect to Fig.24 ), if the user selects "REPLY" 1304, the user can reply within the mirrored notification without streaming the app. In such an implementation, streaming can be initiated from the notification by selecting the body of the notification. In one example, the notification can be actionable by the user, so that upon receiving input from the user, a signal can be transmitted to the device running the app, wherein the signal can be configured to cause the device to perform an action associated with the application.

[0080] After loading the application flow, the user can interact with the flow through the user interface, such as Fig.15 1500 is shown. Although Fig.15 The stream is shown in a portrait view, but the stream may alternatively be in a landscape view or a full screen view. Additionally or alternatively, the stream may be able to alternate between a portrait view, a landscape view, and a full screen view. The stream 1502 on the computing device allows the same application functionality as on the mobile device. For example, if the application is Fig.15 If a messaging application is shown, flow 1502 may include the ability to enter text in text box 1506, attach files or photos from a computing device or mobile device via icon 1508, capture a new photo via icon 1510, enter a voice command via input 1512, start a video call via icon 1514, and / or start a call via icon 1516. Such features may be accessible through a user interface of the computing device.

[0081] In some scenarios, after a stream is loaded, the corresponding icon may additionally display an application symbol to indicate which application is currently being streamed. Fig.16 As shown in view 1600 of FIG. 1 , icon 1602 includes a logo 1604 for indicating a specific app being streamed on the computing device. Similarly, Fig.17 The view 1700 of exemplifies a scene in which a messaging application is streaming, and the icon 1702 includes a messaging application logo 1704. Alternatively, the user can choose not to interact with the notification. When the user does not interact with the notification, the notification can be retracted after a period of time (e.g., 20 seconds, or longer, or shorter). The user can access any notification in the notification center discussed in further detail below.

[0082] In another example, the user can manually select an application for streaming. The user can manually select an application, for example, by selecting an app from a mobile device application list on the computing device or by selecting an app to be streamed to the computing device from the mobile device.

[0083] In a further example, the system can implement intelligent audio source selection, where the system automatically determines whether to play audio directly from the mobile device or from the computing device. This can include providing seamless headphone (or earbud) switching between audio sources. Here, the selection can be based on whether the headphone (or earbud) is currently paired with either (or both) the mobile device or the computing device. If paired with both devices, the system can connect the headphone (or earbud) to a specific device based on physical proximity (e.g., which can be estimated by evaluating signal strength information from the pairing), the type of app being streamed, etc.

[0084] In another example, the user can select an app to stream from an app selection or app list. In one case, the app selection can be a favorite app selection. To make an app a favorite app, the user can designate the app as a favorite app. In one example, the app can have been marked or selected as a favorite app in an app list (e.g., app list 1250). The app can be selected via a user interface, such as from Fig. 12A , select a recent application to stream from the app selection 1204 list shown in . After selection, the application can begin streaming on the computing device. For example, the user can select a recently used application to stream. In order for an application to be considered recently used, in one example, the app may have been streamed on the computing device or used on the mobile device within a certain time range (e.g., 30 minutes, 4 hours, within the past day, or some other time range). The time range can be set by the user, or it can be a default time range. The time range can be selected via a user interface, such as from Fig. 12A Select a recent app for streaming from the app selection 1204 list shown in FIG.

[0085] After selection, the application can begin streaming on the computing device. More than one app can be streamed between devices at a time. The stream can resume streaming content from the time the user last accessed the application. For example, if the user streams an application on a computing device, ends the stream, and then selects the application from app selection 1204, the computing device can stream the application in the state when the user ended the stream. In another example, if the user streams an application on a computing device, ends the stream, and begins using the application on a mobile device, the application can be restored to the state when the user ended the stream. In this regard, the stream can be paused when inactive (such as when the user ends the stream). However, the app can persist as a background operation on a source device (e.g., a mobile device). Such persistence as a background operation allows the user to resume the application.

[0086] Similarly, if the user then closes the app on the mobile device and selects the app from app selection 1204 or app list 1250, the computing device can stream the app in the state it was in when the user closed the app on the mobile device. In one example, the system (e.g., on the mobile device) can cache the full-color image of the streamed app when the stream is closed to save the current stream state. The cached image can be used as the starting image presented when the app is streamed again (e.g., when the user chooses to do so by interacting with a link or initiating a stream in any of the ways discussed above). The cached image can be stored as long as the connection between the computing device and the mobile device is maintained.

[0087] When streaming an application on a computing device, the application does not need to be displayed on the mobile device. According to a scenario, in order to generate a stream, the mobile device creates a virtualized remote display to be presented on the computing device. The mobile device encodes the virtualized remote display (including the application content (e.g., audio, video, etc.) to be streamed). The mobile device may, for example, use an H.264 or MPEG-4 AVC encoder to ensure the quality of the content being streamed. The encoded information (including the application content) may be sent to the computing device via a connection between devices. After receiving, the computing device decodes and renders the virtualized remote display, thereby allowing the computing device to stream the application. The computing device may act as an input device for the virtualized remote display. User input may be received from the computing device, which is then transmitted to the mobile device via a wireless connection. The mobile device may then update the virtualized remote display based on the user input. Therefore, the mobile device acts on the received user input and modifies the virtualized remote display. In this regard, the streaming application may be hosted by the mobile device, which means that the processing module and memory module of the mobile device are used to run the application. The computing device renders the virtualized remote display and acts as an input device, but there is no need to install the streaming application on the computing device. The virtualized remote display can be formatted as a web page, which enables the computing device to set its display size as needed. Therefore, the virtualized remote display appearance (e.g., portrait or landscape mode or total size) can be modified during streaming. In addition, there is no need to store any application information (e.g., data, instructions) in the computing device. In some cases, the mobile device may display an indication or notification to the user that the app stream is occurring. In this regard, the virtualized remote display provides a stylized view of the app on the GUI of the computing device, which makes it easier for the user to control the app using input from the computing device. The input is delivered to the mobile device that generates the virtualized remote display.

[0088] According to one aspect, user interactions with streaming apps on a computing device are transient, meaning that once the computing device passes a signal to the mobile device (e.g., user control of an audio or video stream), the computing device may not store information about the streaming. For example, input events (e.g., pausing a song, replying to a text) are converted at the computing device into a format that can be processed by the mobile device. This can include processing touch input, mouse pad input, cursor input, audio input, or gesture input into a signal that can be acted upon by an app running on the mobile device. For example, in a messaging app, the primary action may be to press the enter key to send a message; however, the app running on the mobile device may not support this feature. In this case, the mobile device may run a machine learning (ML) model that determines what input maps to sending so that the mobile device can map the input when such input is received from the computing device.

[0089] In contrast to screen mirroring, app streaming does not require the mobile device to show the same content as what is being streamed on the computing device. The user is free to keep his mobile device screen off or dimmed, or interact with another app midway through streaming. Therefore, when the virtualized remote display is streamed on the computing device, the mobile device can perform other functions. For example, the mobile device can enter a sleep mode (e.g., where the display of the mobile device can be turned off or a screen saver is presented), and the virtualized remote display can still be generated on the computing device. In this regard, when the display of the mobile device is inactive, part of the processing module of the mobile device (such as an application processor) can remain active to generate the stream. Alternatively, the mobile device can allow the user to interact with additional applications while generating a stream to the computing device. In one scenario, the user can minimize the streaming app, where it can remain minimized for a selected amount of time (e.g., 5 minutes, or longer, or shorter), after which it can automatically shut down to save the battery of the mobile device. In this regard, the virtualized remote display can essentially be viewed as a secondary display that uses a secondary set of inputs and outputs (e.g., those of the computing device) such that the mobile device inputs and outputs (e.g., display, microphone, speakers, etc.) can be turned off or used for different functions.

[0090] A user can stop an app stream in a number of ways. In one example, a stream can be closed due to user inactivity (e.g., no user interaction with the streaming application for some minimum amount of time). For example, for a stream to be closed due to inactivity, the stream can be idle for a period of time (such as 10-15 minutes, or longer, or less). When a stream is stopped due to inactivity, such as Fig.18 As shown in view 1800 of , a message 1802 may be generated for display on the computing device. Message 1802 may include text 1804 that informs the user that the connection or stream has been terminated due to inactivity. The message may also include a link contained in the text "Open again" 1806 for the user to reopen the stream. As described above, when the user is idle for a period of time, the system may cache an image (e.g., a full-color image) of the streaming app to save the current stream state. When the user chooses to do so by interacting with a link or initiating a stream in any of the ways discussed above, the cached image can be used to resume the stream.

[0091] In another example, the stream can be closed by the user selecting the currently streaming app in the app selection 1204 or app list 1250. Selecting the app selection 1204 or app list 1250 can stop the stream on the computing device and can transfer the app content to the primary display of the mobile device, allowing the user to resume using the app on the mobile device from the point in time when the user closed the stream on the computing device. In a further example, the user can close the stream from the stream itself. Fig.19 A computing device display 1900 is shown with an open stream 1902 and a streaming app taskbar 1904. The virtualized remote display 1902 includes a back button 1905, a minimize control 1906, and a close control 1908. The back button 1905 can end the stream and return to the app list and / or app selection screen. The minimize control 1906 can minimize the app stream while still maintaining the active stream, and the close control 1908 can terminate the app stream. The minimized app stream can remain minimized for a period of time (e.g., 10 minutes, or longer, or less), and the stream can then terminate automatically. In another example, the stream can be closed due to certain errors, such as when the computing device and the mobile device are no longer in range. Stream closure due to errors is discussed in further detail below.

[0092] In an additional example, the flow may be closed from the mobile device via a silent persistent notification 2002, such as Fig. 20 2000. When the app is being streamed to the computing device, the mobile device may display a persistent notification 2002 that informs the user that the application is currently being streamed. Notification 2002 may be located anywhere on the GUI of the mobile device. Notification 2002 may include an end stream control 2004 and a settings control 2006. The user may use the end stream control 2004 to end the stream. When the stream is ended, the streamed app may be added to the recent app section. For example, when app selection 1204 is a recent app selection, the app may be added to app selection 1204 if it is closed from the computing device or the mobile device. In one example, opening the app at the computing device moves it to the front of the recent app stack on the mobile device side. In another example, opening an instance of the app from the recent app stack on the mobile device may disconnect any active streaming connection associated with the app. Here, stopping the stream may not close the app because it is being actively opened on the mobile device. However, closing the app stream on the computing device may make the app accessible at the front of the recent app stack on the mobile device.

[0093] In some implementations, a prompt may be displayed on a computing device to induce a user to install, update, and / or enter user information required for streaming an app. Different messages may be generated depending on the user's status. For example, the user may be a new user, an existing user, or an existing user with one or more new devices. In one implementation, if Fig.21 As shown in example 2100 of FIG. 1 , a new user may receive a notification 2102 for installing software capable of performing the methods described herein on their computing device and establishing a user profile. The notification 2102 may include a link 2106 for starting the installation process and a control 2104 for interrupting the installation process and deleting the notification 2102. Existing users with one or more new devices may receive the same or similar messages as the new user. Fig. 22 As shown in the view 2200 of , a user may receive a notification 2202 for updating the existing software on his computing device that can execute the methods described herein. The notification 2202 may also include a link 2206 for starting the installation process and a control 2204 for interrupting the installation process and deleting the notification 2202.

[0094] Alternatively, the user may be prompted to update the current software through the mobile device interface 2302 on the display 2300 of the computing device. Fig.23 An interface 2302 is shown with a prompt 2304 for updating and / or installing software. The prompt 2304 may contain a link 2308 for starting the installation process and a control 2306 for interrupting the installation process and dismissing the notification.

[0095] In some implementations, the app may be a messaging app, and notifications for such messaging app may include an online reply feature. Fig.24 , a notification 2402 related to a messaging app displays text and app information 2404. Notification 2402 also includes an online reply feature, in which a user can read the message text 2404 without streaming the app, enter a reply into a text box 2406 using one or more user inputs, and send a reply using a send control 2408 using one or more user inputs. In some implementations, notifications and streams can allow messages to be sent using specific inputs from one or more user inputs. For example, a user can use the enter key (or other key) of a keyboard of a computing device to send a message. To do this, mobile device inputs can be mapped to computing device inputs; this means that if the enter key is pressed to send a message when using a mobile device, this feature can be mapped to the computing device and similarly allow the enter key to send a message.

[0096] In some implementations, when a user does not interact with a notification (e.g., starts a stream, responds online, or dismisses the notification), the notification can be accessed in the notification center. Fig.25 , a user may access a quick settings menu 2502 on a display 2500 of a computing device. A notification center 2504 containing one or more missed notifications may be displayed along with the quick settings menu 2502. In one scenario, only conversation notifications (e.g., chats or text strings) are mirrored from the mobile device to the computing device. For example, responding to a notification or dismissing a notification on the computing device clears the notification from the mobile device. Selecting (e.g., by tapping or clicking) an app notification may open the streamed app and may cause the app to be added to a recent apps list on the mobile device.

[0097] In some implementations, the mobile device generates a silent persistent notification 2602 indicating that the mobile device is connected to the computing device. Fig.26A In the example embodiment, the persistent notification 2602A can allow the user to terminate the connection through the disconnect control 2604A. The persistent notification can also allow the user to access the connection settings through the settings control 2606A. In some implementations, when the mobile device is streaming to the computing device, the mobile device can generate a different silent persistent notification 2602B. Fig.26B A persistent notification 2602B is shown. Persistent notification 2602B notifies the user that the mobile device is currently streaming to the computing device, and allows the user to terminate the stream through the end stream control 2604B and access the connection settings through the settings control 2606B. In addition or alternatively, the silent persistent notification can be a connection indicator, such as an icon for notifying the user of the connection between the mobile device and the computing device. When streaming, the icon can additionally indicate that the application is currently streaming. This can be achieved by, for example, including an icon of the streaming app. Such an indicator can be located anywhere on the display of the mobile device. In addition, the indicator can be a light indicator (e.g., an LED) located on the mobile device. In addition or alternatively, multiple indicators can be present in different locations.

[0098] In some implementations, the computing device allows a user to use multi-touch interaction to enlarge and reduce the displayed streaming image.Multi-touch interaction may be supported on one or more user inputs, such as a trackpad and a touch screen. FIG. 27A to FIG. 27B2702A and 2704A. The user can then slide their fingers towards each other on the user interface. To reduce or zoom out, the user can contact the user interface with two fingers, with the first finger at the first point 2702B and the second finger at the second point 2704B. The user can then slide their fingers away from each other on the user interface. In an example where the user interface is a touchpad, the flow can zoom in and / or zoom out at the location where the cursor is currently located on the display of the computing device. Alternatively, gestures or even voice commands can be used to perform the expansion and reduction.

[0099] In some implementations, the system may be configured to resolve stream errors (e.g., connection timeouts), resolve content errors (e.g., playing content does not work), and maintain system health. In one example, when a stream is experiencing a long loading time (e.g., approximately 10-30 seconds or longer), the computing device may display the following notification: indicating that the mobile device is in the process of loading the stream and / or connecting to the computing device, and advising the user to ensure that the device is connected to the network and that the mobile device is within the range of the computing device. In another example, when the computing device and the mobile device fail to connect, the notification may inform the user of the connection failure, prompt the user to ensure that the mobile device and the computing device are connected to the network, and prompt the user to try to connect the device again. The notification may additionally inform the user that the computing device and the mobile device are connected to different networks, and prompt the user to connect the computing device and the mobile device to the same network. Additionally or alternatively, the notification may inform the user that the network is not supported (e.g., cannot be used for streaming), and may prompt the user to use a different network for streaming. The notification may additionally guide the user to seek help. In a further example, when an application is not supported (e.g., cannot be used for streaming), the notification can inform the user by displaying a message such as "Not supported," "Not supported by app," or "Can't stream app." In this regard, different display notifications can be generated based on the reason why the application is not supported (e.g., security, technical limitations, developer opt-out, etc.). In an additional example, when the connection through the network is weak, resulting in a slow stream (e.g., reduced quality), the notification can inform the user that the stream may be slow.

[0100] Similarly, during device pairing, if the computing device and the mobile device fail to pair, a prompt (at one or both devices) can inform the user that the pairing failed and prompt the user to learn more about potential problems. The prompt may also allow the user to exit the pairing process. In another example, when the connection between the computing device and the mobile device is lost, the notification can inform the user that the connection is lost, prompt the user to ensure that the mobile device and the computing device are connected to the network, and prompt the user to try to connect the device again. In an additional example, when the stream or notification fails to start, the notification can inform the user of the startup failure. In another example, when the application does not support application streaming and therefore cannot load the stream, the notification can inform the user. In such an implementation, the computing device can still generate a mirror notification for the application. In an additional example, when an unknown error prevents streaming, a general error message can be displayed to the reader via a notification.

[0101] As discussed above, during the pairing process, the user may be prompted to accept permissions, such as FIG. 10A to FIG. 10B As shown. In some implementations, different permissions are required for application streaming and notification mirroring. For example, for notification mirroring, permissions can be granted for photos and media as well as notifications. Message 1006A can be used to communicate the required permissions and prompt the user to "Allow" 1002 such information to be accessed by the computing device. For application streaming, permissions can be granted for the application. Message 1006B can be used to communicate the required permissions and prompt the user to "Allow" 1002 such information to be accessed by the computing device. Users can revoke permissions at any time. If a user revokes permissions required to receive notifications and / or stream applications, the user can be prompted to accept the revoked permissions again.

[0102] In some implementations, the system verifies one or more lock screen settings before starting streaming. When the mobile device or computing device goes to sleep, is turned off, or is locked, the mobile device or computing device has the lock screen enabled, at which point the user can enter a pin or password to access the device. When the mobile device does not have the lock screen enabled, the computing device can start streaming when prompted. When both the mobile device and the computing device have the lock screen enabled, the computing device can start streaming when prompted. When the mobile device has the lock screen enabled but the computing device is not enabled, streaming may not start, and a notification may be presented to the user prompting the user to enable the lock screen of the computing device.

[0103] In some implementations, a computing device may stream an application that is capable of making and receiving video calls and / or audio calls. While the stream is open on the computing device, the call may be maintained. If a user attempts to close the stream during a call, the user may receive a notification informing them that the call may be terminated if the stream is closed. The notification may allow the user to close the stream, or cancel closing the stream and continue the call.

[0104] Alternatively or additionally, in some implementations, the activity stream may allow a user to import information to and export information from a computing device. In one example, a user may copy and paste text, images, etc. between streams from a mobile device and a computing device. An input via one or more user inputs on the computing device triggers the mobile device to send additional information containing the material to be copied and / or pasted over a network. In another example, a user may interact with a link to a web page. In such examples, a user may interact with a link in a stream via one or more user interfaces. In response, the mobile device sends link information over a network, allowing the web page to be loaded onto the computing device. In some scenarios, a web page may require logging into a user account to access the page. If the mobile device includes user credentials for the web page, it may send the credentials to the computing device in addition to sending the link information.

[0105] Alternatively or additionally, in some implementations, the system may be configured to switch the connection of the audio output between the mobile device and the computing device to facilitate audio stability during app streaming. The audio switch may be triggered by an application stream containing audio and an audio output (such as a headset) connected to a network via a computing device or a mobile device. When the audio output connected to the network is switched from one device to another, a notification informing the user of the switch may be generated. In one example, if a headset can be connected to a computing device and the stream initiated contains audio, the headset connection may be switched to a mobile device when the application is streamed to the computing device. The streaming audio may be output to the headset by a mobile device, and the audio to the computing device may be suspended while the headset remains connected to the mobile device. In another example, if a headset can be connected to a mobile device and the stream initiated contains audio, the headset connection to the computing device may be maintained when the application is streamed to the computing device. In such examples, the audio from the stream may be output from the headset via the computing device.

[0106] In one implementation, the connection for the audio output may not be switched between the computing device and the mobile device. For example, if a headset is connected to the mobile device and the stream initiated contains audio, the headset connection to the mobile device may remain while the application is streamed to the computing device. The streaming audio may be output to the headset by the mobile device, and the audio to the computing device may be suspended while the headset remains connected to the mobile device. In another example, if a headset is connected to the computing device and the stream initiated contains audio, the headset connection to the computing device may remain while the application is streamed to the computing device. In such examples, the audio from the stream may be output from the headset via the computing device.

[0107] In addition to or as an alternative to the above examples and scenarios, the external device app streaming process can support the use of cameras, microphones, and / or speakers at the computing device and provide the captured information to the mobile device. This can be beneficial to some apps (such as messaging apps and video conferencing apps). In addition, the virtualized remote display can support copy / paste events, so that the user can perform those actions at the computing device and send the information back to the mobile device. In addition, the present technology can support local switching involving filtering down different notifications. Here, for example, when a user receives a notification containing a reply field, the system can use it as a signal to indicate that it is associated with a messaging app so that the notification can be selectively shown on the computing device. For example, the system can pass information indicating that there is a chat message for the user, while also passing metadata and user credentials to other devices. In this way, other devices (e.g., computing devices) can act on the message via their own app version without having to do it by streaming. Alternatively, in another example, a hyperlink can be provided in the chat message, and the user may be able to open the link directly in the browser of the computing device.

[0108] Fig.28 An example method 2800 is shown. As shown in box 2802, the method includes pairing a first client device with a second client device via at least one wireless connection, wherein the pairing allows the first client device to communicate directly with the second client device. At box 2804, the method includes receiving, by one or more processors of the first client device, an encoded virtualized remote display of an application, the encoded virtualized remote display being generated by the second client device, and the application running on the second client device. And at box 2806, the method includes controlling, by one or more processors of the first device, a graphical user interface (GUI) to present a virtualized remote display of the application based on the encoded virtualized remote display on the first client device, the virtualized remote display being configured to enable a user of the first client device to control an application running on the second client device.

[0109] Fig.29An example method 2900 for application streaming is shown. As shown in box 2902, a second client device is paired with a first client device via at least one wireless connection, wherein the pairing allows the first client device to communicate directly with the second client device. At box 2904, the method includes generating, by one or more processors of the second client device, an encoded virtualized remote display of an application running on the second client device. At box 2906, the method includes transmitting the encoded virtualized remote display of the application from the second client device to the first client device via at least one wireless connection. At box 2908, the method includes receiving, by the second client device, one or more user inputs associated with the application based on the streaming of the virtualized remote display based on the encoded virtualized remote display on the first client device. At box 2910, the method includes taking, by one or more processors of the second client device, an action associated with the application based on the received one or more user inputs.

[0110] Although certain examples herein involve computing devices and mobile devices performing certain operations, the present technology can be used with different types of client devices and is not limited to the specific types of devices identified in the examples.

[0111] Although the present technology has been described herein with reference to specific implementations, it should be understood that these implementations are only illustrative of the principles and applications of the present technology. Therefore, it should be understood that numerous modifications may be made to the illustrative implementations, and other arrangements may be designed without departing from the spirit and scope of the present technology as defined by the appended claims.

Claims

1. A method comprising: pairing a first client device with a second client device via at least one wireless connection, wherein the pairing allows the first client device to communicate directly with the second client device; receiving, by one or more processors of the first client device, via streaming, an encoded virtualized remote display of an application from a second client device, the encoded virtualized remote display being generated by the second client device, the application being executed on the second client device; and A graphical user interface (GUI) is controlled by the one or more processors of the first device to present a virtualized remote display of the application based on the encoded virtualized remote display on the first client device, the virtualized remote display being configured to enable a user of the first client device to control the application running on the second client device.

2. The method of claim 1, further comprising: receiving, by the first client device, an indication of an event associated with the application running on the second client device; as well as A notification regarding the event is generated, by the one or more processors of the first client device, based on the received indication for presentation via the GUI.

3. The method of claim 2, wherein: The notification is actionable by the user of the first client device such that upon receiving input from the user, the method further comprises transmitting a signal to the second client device, the signal configured to cause the second client device to perform an action associated with the application.

4. The method of claim 1, further comprising: Upon terminating the streaming of the application, the virtualized remotely displayed image of the application is caused to be cached.

5. The method of claim 4, further comprising: upon terminating the streaming of the application, designating the application as a recent application in a set of one or more recent applications; as well as A menu of the one or more recent applications is generated for presentation by the GUI of the first client device.

6. The method of claim 4, further comprising: Upon determining that the first client device is no longer paired with the second client device, the one or more processors of the first client device send a signal to the second client device to terminate the flow.

7. The method of claim 1, further comprising: causing termination of the streaming after a period of time when the at least one wireless connection is not maintained; reestablishing the at least one wireless connection between the first client device and the second client device; and The streaming is caused to resume after the at least one wireless connection is reestablished.

8. The method of claim 1, further comprising generating, via the GUI of the first client device, at least one of: an icon indicating that the second client device is paired with the first client device, or an icon indicating that the streaming of the application is occurring.

9. The method of claim 1, wherein: The at least one wireless connection includes a first wireless connection and a second wireless connection different from the first wireless connection; The pairing occurs via the first wireless connection; and The virtualized remote display of the received application via streaming occurs via the second wireless connection.

10. The method of claim 9, further comprising: One or more user inputs received in response to the virtualized remote display are transmitted by the first client device to the second client device via the second wireless connection.

11. The method of claim 1, wherein: Receiving the encoded virtualized remote display of the application via streaming includes decoding, by the one or more processors of the first client device, the encoded virtualized remote display of the application.

12. The method of claim 1, further comprising generating, via the GUI of the first client device, at least one prompt to facilitate pairing between the first client device and the second client device.

13. An application streaming method, comprising: pairing a second client device with a first client device via at least one wireless connection, wherein the pairing allows the first client device to communicate directly with the second client device; generating, by one or more processors of the second client device, an encoded virtualized remote display of an application running on the second client device; transmitting the encoded virtualized remote display of the application from the second client device to the first client device via streaming via the at least one wireless connection; receiving, by the second client device, one or more user inputs associated with the application based on presentation of the application on the first client device via a virtualized remote display based on the encoded virtualized display; and An action associated with the application is taken by the one or more processors of the second client device based on the received one or more user inputs.

14. The method of claim 13, further comprising: Prior to the transmitting, the one or more processors of the second client device encode the virtualized remote display of the application.

15. The method of claim 13, wherein: Taking the action associated with the application includes updating the encoded virtualized remote display of the application.

16. The method of claim 13, further comprising generating, by the one or more processors of the second client device, a graphical user interface (GUI), the graphical user interface (GUI) comprising at least one of: a permission prompt for facilitating the pairing, or an actual display of the application for presentation to a user.

17. The method of claim 16, wherein: The virtualized remote display generated by the one or more processors of the second client device has a different visual appearance than the actual display of the application.

18. The method of claim 13, wherein: The encoded virtualized remote display is generated by the one or more processors of the second client device while the display device of the second client device is in a sleep mode.

19. The method of claim 13, wherein: The encoded virtualized remote display is generated by the one or more processors of the second client device while another application is presented on a display device of the second client device.

20. The method of claim 13, further comprising, in response to termination of the streaming, the one or more processors of the second client device caching an image of the virtualized remote display of the application.

21. The method of claim 13, wherein: The at least one wireless connection includes a first wireless connection and a second wireless connection different from the first wireless connection; The pairing occurs via the first wireless connection; and Transmitting the encoded virtualized remote display of the application from the second client device occurs via the second wireless connection.

22. A computing device configured to stream an application from a remote device, the computing device comprising: one or more processors; one or more user inputs operatively coupled to the one or more processors; as well as a communication module operatively coupled to the one or more processors, the communication module configured to: facilitating pairing of the computing device with the remote device via at least one wireless connection, wherein the pairing allows the computing device to communicate directly with the remote device, and receiving, via a stream, from the remote device an encoded virtualized remote display of an application, the application being executed on the remote device, Wherein, the one or more processors are configured to: generating a graphical user interface (GUI) to present the application, including presenting on the GUI a virtualized remote display of the application running on the remote device based on the encoded virtualized remote display; receiving input associated with the virtualized remote display from the one or more user inputs; and Information about the received input is transmitted to the remote device via the communication module to control the application on the remote device.

23. A computing device configured to stream an application to a remote device, the computing device comprising: one or more processors; one or more user inputs operatively coupled to the one or more processors; as well as a communication module operatively coupled to the one or more processors, the communication module configured to: facilitating pairing of the computing device with the remote device via at least one wireless connection, wherein the pairing allows the computing device to communicate directly with the remote device, and Wherein, the one or more processors are configured to: generating an encoded virtualized remote display of an application of the computing device, the application being executed on the computing device; causing the communication module to transmit, via streaming, the encoded virtualized remote display of the application to the remote device via the at least one wireless connection; Based on the presentation of the application on the remote device based on the encoded virtualized remote display, receiving from the remote device via the communication module one or more user inputs associated with the application; and An action associated with the application is taken based on the received one or more user inputs.

Images