Adaptable real-time communications plugin for virtual desktop infrastructure solutions

Abstract

To provide a method, system, and program for improving performance of a remote desktop client that is executed on a client computing device to present a user interface of a communications application that is remotely executed in a cloud computing environment.SOLUTION: In a client computing device, a plugin works with a remote desktop client that is executed on the client computing device to present a user interface of a communications application that is executed in a cloud computing environment. The plugin enables the remote desktop client to conduct audio and / or video communication with a remote computing device in a peer-to-peer manner rather than via the communications application. The plugin also enables the remote desktop client to determine a hardware-based media processing capability of the client computing device and leverage such capability in conducting the peer-to-peer audio and / or video communication with the remote computing device.SELECTED DRAWING: Figure 4

Description

[Background technology]

[0001] background

[0001] Virtualization technology abstracts a desktop's operating system (OS) and associated applications from the client computing devices used to access them. For example, a virtual desktop infrastructure solution (VDI) may involve hosting a desktop on a server in a data center and delivering an image of the desktop to a client computing device over a network. That desktop image can then be rendered on the client computing device, and a user of the client computing device can directly interact with the image as if the desktop and its applications were running locally on the client computing device.

[0002]

[0002] This approach allows customers to streamline management and costs by consolidating and centralizing user desktops. The benefits of centralization include optimization of hardware resources, reduced software maintenance, and improved security. For example, in some instances, software patching and OS migration can be applied and tested for all users. Furthermore, software assets are centralized, which makes them easier to monitor and protect, and the security of sensitive data is not compromised if a desktop is lost or stolen. In addition, desktop virtualization enhances user mobility and freedom to access a virtual desktop from any location and any device.

[0003]

[0003] Despite the many advantages offered by virtualization technology, some applications may not be optimized for or may not be well supported within a VDI environment. One such example is a communication application running within a VDI environment that allows real-time communication (RTC) between users (e.g., Microsoft® Teams, Slack®). RTC refers to the exchange of information (e.g., voice, instant messaging, video, etc.) from a sender to a receiver over a network, nearly simultaneously, with negligible latency. Summary of the Invention

[0004] overview

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005]

[0005] Methods, systems, and computer program products are described herein for improving the performance of a remote desktop client executing on a client computing device to present a user interface (UI) of a communication application executed remotely within a cloud computing environment. In embodiments, the methods, systems, and computer program products enable the remote desktop client to engage in voice and / or video communications with a remote computing device in a peer-to-peer manner rather than through a remotely executed communication application. Further according to such embodiments, the methods, systems, and computer program products enable the remote desktop client to determine one or more hardware-based media processing capabilities of the client computing device and to leverage such one or more hardware-based media processing capabilities when engaging in peer-to-peer voice and / or video communications with the remote computing device, thereby improving the quality of such voice and / or video communications and reducing the processing load on the client computing device. One or more hardware-based media processing capabilities can be used, for example, to process media received from a remote computing device for rendering by the client computing device, to process media captured from a media source on the client computing device for transmission to a remote computing device, or as the basis for negotiating media communication parameters with a remote computing device.

[0006]

[0006] Further features and advantages of the present invention, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings. It should be noted that the embodiments are not limited to the specific embodiments described herein. Such embodiments are shown in this specification for illustrative purposes only. Based on the teachings contained herein, further embodiments will be apparent to those skilled in the art.

[0007] BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present application and, together with the description, further serve to explain the principles of the embodiments and enable those skilled in the art to make and use the embodiments. [Brief description of the drawings]

[0008] [Figure 1]

[0008] A block diagram of an example of a system for enabling video and / or audio communication with communication applications executing in a cloud computing environment, according to an example embodiment. [Diagram 2]

[0009] A block diagram of an example of a system for enabling peer-to-peer voice and / or video communication between a client computing device and a remote computing device, as opposed to voice and / or video communication between the client computing device and the remote computing device via a communication application, according to an example embodiment, is shown. [Diagram 3]

[0010] FIG. 1 illustrates a block diagram of an example system including a plug-in that enables redirecting communications from a communications application to enable peer-to-peer voice and / or video communications between a client computing device and a remote computing device, according to an example embodiment. [Figure 4]

[0011] FIG. 1 illustrates a block diagram of a client computing device including a plug-in to a remote desktop client that determines the hardware-based media processing capabilities of the client computing device and uses the detected hardware-based media processing capabilities to process media, according to another example embodiment. [Diagram 5]

[0012] 1 illustrates a flow diagram of a method for determining hardware-based media processing capabilities of a client computing device and using the detected hardware-based media processing capabilities to process media, according to an example embodiment. [Figure 6]

[0013] 1 illustrates a flow diagram of a method for negotiating media communication parameters with a remote computing device based on determined hardware-based media processing capabilities of a client computing device, according to an example embodiment. [Figure 7]

[0014] 1 illustrates a flow diagram of a method for determining hardware-based media processing capabilities of a client computing device and using the hardware-based media processing capabilities to process media captured from a media source of the client computing device, according to an example embodiment. [Figure 8]

[0015] FIG. 1 is a block diagram of an example of a processor-based computer system that can be used to implement various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009]

[0016] Features and advantages of the present invention will become more apparent from the following detailed description when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout. In the drawings, like reference numbers generally indicate identical, functionally similar, and / or structurally similar elements. The drawing in which an element first appears is indicated by the most significant digit(s) in the corresponding reference number.

[0010] Detailed Description I. Introduction

[0017] This specification and the accompanying drawings disclose one or more embodiments incorporating the features of the present invention. The scope of the invention is not limited to the disclosed embodiments. The disclosed embodiments are merely exemplary of the invention, and modifications of the disclosed embodiments are also encompassed by the invention. The embodiments of the invention are defined by the appended claims.

[0011]

[0018] References herein to "one embodiment," "an embodiment," "an example embodiment," and the like mean that the embodiment being described may include a particular feature, structure, or characteristic, but all embodiments may not necessarily include that particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Moreover, when a particular feature, structure, or characteristic is described in connection with one embodiment, it is believed to be within the knowledge of one of ordinary skill in the art to affect such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described.

[0012]

[0019] In this description, unless otherwise specified, adjectives such as "approximately" and "about" modifying a condition or relationship characteristic of a feature of an embodiment of the present disclosure are understood to mean that the condition or characteristic is defined within an acceptable tolerance for the operation of the embodiment for the intended application.

[0013]

[0020] Numerous exemplary embodiments are described below. Please note that any section / subsection headings provided herein are not intended to be limiting. Embodiments are described throughout this specification, where any type of embodiment may be included under any section / subsection. Furthermore, an embodiment disclosed in any section / subsection may be combined in any manner with any other embodiment described in the same section / subsection and / or in a different section / subsection.

[0014] II. Example Embodiments

[0021] Virtualization technology abstracts a desktop's operating system (OS) and associated applications from the client computing devices used to access them. For example, a virtual desktop infrastructure solution (VDI) may involve hosting a desktop on a server in a data center and delivering an image of the desktop over a network to a client computing device. The desktop image can then be rendered on the client computing device, and a user of the client computing device can directly interact with the image as if the desktop and its applications were running locally on the client computing device.

[0015]

[0022] This approach allows customers to streamline management and costs by consolidating and centralizing user desktops. The benefits of centralization include optimization of hardware resources, reduced software maintenance, and improved security. For example, in some instances, software patching and OS migrations can be applied and tested for all users. Furthermore, software assets are centralized, and therefore easily monitored and protected, and security of sensitive data is not compromised if a desktop is lost or stolen. In addition, desktop virtualization enhances user mobility and freedom to access a virtual desktop from any location and any device.

[0016]

[0023] Despite the many advantages offered by virtualization technology, some applications may not be optimized for or may not be well supported within a VDI environment. One such example is a communication application running within a VDI environment that allows real-time communication (RTC) between users (e.g., Microsoft® Teams, Slack®). RTC as described herein refers to the nearly simultaneous exchange of information (e.g., voice, instant messaging, video, etc.) from a sender to a receiver over a network with negligible latency.

[0017]

[0024] For example, Figure 1 illustrates a block diagram of an example system 100 for enabling video and / or audio communications with communication applications executing within a cloud computing environment. As shown in Figure 1, the system 100 includes a cloud computing environment 102 that includes a virtual desktop host 104, a client device 112, and a remote computing device 114.

[0018]

[0025] For purposes of illustration, the cloud computing environment 102 is shown to include only one virtual desktop host 104, but may include any number of resources. For example, the cloud computing environment 102 may be comprised of resources (e.g., servers) running in one or more cloud datacenters and / or on-premises for an enterprise or organization. Additionally, in embodiments, the cloud computing environment 102 may include any number of and types of other resources, including resources (e.g., network switches, networks, etc.) that facilitate communication with and between the servers, storage by the servers (e.g., storage devices, etc.), resources that manage other resources (e.g., hypervisors that manage virtual machines to present a virtual operating platform for tenants of a multi-tenant cloud, etc.), and / or additional types of resources. In some embodiments, the virtual desktop host 104 may include a remote server or virtual machine accessible in a datacenter or on-premises server.

[0019]

[0026] Additionally, client computing device 112 and remote computing device 114 are illustrated as a laptop and a smartphone, respectively, but may be any type of mobile or fixed computing device. Examples of mobile computing devices include, but are not limited to, a Microsoft® Surface® device, a personal digital assistant (PDA), a thin client, a laptop computer, a notebook computer, a tablet computer such as the Apple iPad™, a netbook, a mobile phone, a handheld gaming device, or a wearable computing device. Examples of fixed computing devices include, but are not limited to, a desktop computer or PC (personal computer), a gaming console, or a server.

[0020]

[0027] 1, client computing device 112 is communicatively coupled to cloud computing environment 102 via network 108, and remote computing device 114 is communicatively coupled to cloud computing environment 102 via network 118. Networks 108 and 118 may include one or more networks, such as a local area network (LAN), a wide area network (WAN), an enterprise network, the Internet, etc., and may include one or more wired and / or wireless portions. In certain scenarios, network 108 and network 118 may include the same network.

[0021]

[0028] Client computing device 112, remote computing device 114, and virtual desktop host 104 may include at least one network interface that enables communication over networks 108 and 118. Examples of such network interfaces, either wired or wireless, include IEEE 802.11 Wireless LAN (WLAN) radio interfaces, Worldwide Interoperability for Microwave Access (Wi-MAX) interfaces, Ethernet interfaces, Universal Serial Bus (USB) interfaces, cellular network interfaces, Bluetooth™ interfaces, Near Field Communication (NFC) interfaces, etc. Further examples of network interfaces are described elsewhere herein.

[0022]

[0029] The client computing device 112 also includes a remote desktop client configured to enable video and / or audio communications between users of the communications applications and to present within a user interface of the client computing device 112 a user interface of the communications applications executing on the virtual desktop host 104. For example, data related to the user interface 106 of the communications applications may be transferred to the client computing device 112 and rendered as a user interface within a screen display of the client computing device 112. Any interaction (e.g., mouse and / or keyboard input) that the user 110 may have with the user interface on the client computing device 112 that presents the user interface 106 of the communications applications may be tracked and transmitted to the virtual desktop host 104.

[0023]

[0030] More specifically, user 110 can initiate video and / or audio communications with user 116 of remote computing device 114 by interacting with the user interface of client computing device 112 (e.g., by clicking on a "phone icon" next to a contact displayed within the user interface). This interaction can be provided to virtual desktop host 104, and a communications application executing on virtual desktop host 104 can initiate and conduct video and / or audio communications with user 116.

[0024]

[0031] In embodiments, during video and / or audio communication between user 110 and user 116, device sources (such as video cameras) on client computing device 112 and remote computing device 114 may capture images appearing in front of the device sources. As shown in FIG. 1, the captured video of users 110 and 116 may be transferred via virtual desktop host 104 to the other user's computing device for presentation on the other user's computing device's display screen. For example, video of user 116 captured by a device source on remote computing device 114 may be transferred to virtual desktop host 104, processed (e.g., decoded) on virtual desktop host 104, and rendered locally as part of a communication application's user interface 106. In some embodiments, to render video of user 116 within the user interface of client computing device 112, screen scraping may be used to gather screen display data related to the video of user 116 rendered within user interface 106, such that user 110 may simultaneously view the video of user 116 as it is captured by the device source.

[0025]

[0032] 1 may introduce latency issues (especially for high-resolution video and audio experiences) because the video and / or audio communication between the client computing device 112 and the remote computing device 114 is transmitted through the network 108, the virtual desktop host 104, and the network 118. Additionally, this approach may increase the amount of time and resources spent processing the media. For example, when the virtual desktop host 104 receives the media, it may have to decode the media, and when the virtual desktop host 104 transmits the media, it may have to encode the media.

[0026]

[0033] Implementing peer-to-peer video and / or audio communication techniques between client computing devices 112 and remote computing devices 114 can alleviate the latency issues discussed above, as well as reduce the processing workload on the virtual desktop host 104. For example, Figure 2 illustrates a block diagram of an example system 200 for enabling peer-to-peer audio and / or video communication between client computing devices and remote computing devices interacting with a communications application in the cloud.

[0027]

[0034] As described above with respect to Figure 1, a user 110 may initiate video and / or audio communications with a user 116 of a remote computing device 114 by interacting with a user interface of the client computing device 112. This interaction may be provided to the virtual desktop host 104 and may prompt a communications application running on the virtual desktop host 104 to initiate media communications with the remote computing device 114. In Figure 2, however, the media communications calls may be intercepted and redirected to the client computing device 112 to enable peer-to-peer media communications between the client computing device 112 and the remote computing device 114. The remote desktop client of the client computing device 112 is configured to receive the redirected communications from the communications application to enable peer-to-peer audio and / or video communications between the client computing device 112 and the remote computing device 114.

[0028]

[0035] When client computing device 112 and remote computing device 114 are connected for audio and / or video communication (a process described in more detail below with respect to Figures 4 and 5), client computing device 112 and remote computing device 114 can communicate video and / or audio with each other over network 202 (which may be the same as, similar to, or completely different from networks 108 and 118 of Figure 1).

[0029]

[0036] By enabling peer-to-peer audio and / or video communication between the client computing device 112 and the remote computing device 114, media communication via the virtual desktop host 104 is avoided, thereby reducing latency associated with such audio and / or video communication and reducing the processing load on the virtual desktop host 104. Furthermore, as part of moving the media communication connection between the client computing device 112 and the remote computing device 114 to a peer-to-peer channel, the embodiments described herein also enable local hardware-based media processing capabilities (e.g., hardware-based video or audio codecs) of the client computing device 112 to be discovered and leveraged by the remote desktop client to better perform such communication. As used herein, the term media should be understood to encompass at least video, audio, or a combination thereof. Because such embodiments can leverage the hardware-based media processing capabilities of the client computing device 112, the quality and efficiency of media processing by the client computing device 112 can be improved as compared to software-based media processing approaches. Furthermore, utilizing such hardware-based media processing capabilities of the client computing device 112 can reduce the load on the processor of the client computing device 112 without such processor having to instead implement corresponding software-based media processing capabilities.

[0030]

[0037] To help illustrate the process of redirecting communications from a communications application executing on the virtual desktop host 104 to a client computing device 112, Figure 3 will now be described with continued reference to Figures 1 and 2. Figure 3 shows a block diagram of an example system 300 for facilitating redirection of communications from a communications application to enable peer-to-peer voice and / or video communications between a client computing device and a remote computing device.

[0031]

[0038] As shown in Figure 3, the cloud computing environment 102 includes a virtual desktop host 104 and a plug-in 302. As further shown in Figure 3, the plug-in 302 includes a redirection shim 304. The redirection shim 304 is configured to intercept communications (e.g., API calls) initiated by a communications application running on the virtual desktop host 104 and redirect the calls over the network 108 to the client computing device 112. For example, when a communications application initiates a video and / or audio communication with a remote computing device 114, the redirection shim 304 intercepts the video and / or audio communication request, modifies the request as necessary (e.g., replacing certain script objects with different implementations), and redirects it over the communication channel 306 (over the network 108) to the client computing device 112.

[0032]

[0039] In one embodiment, the communications application may load the plug-in 302 at run time if certain conditions are met (e.g., if the communications application is run on a particular virtual desktop environment and if multi-session users are enabled). Once loaded, the plug-in 302 may attempt to establish a communication channel 306 (e.g., establish a dynamic virtual channel (DVC)) with the client computing device 112 to enable communication between the virtual desktop host 104 and the client computing device 112. In an embodiment, the plug-in 302 may make a remote procedure call (RPC) to the client computing device 112 through a WebSocket server process.

[0033]

[0040] To provide a more detailed overview of the client computing device 112, FIG. 4 is now described. FIG. 4 illustrates a client computing device 112 that includes a plug-in 404 to a remote desktop client 402 that enables peer-to-peer communication between the client computing device 112 and a remote computing device, determines the hardware-based media processing capabilities of the client computing device 112, and uses the detected hardware-based media processing capabilities to process media. The plug-in 404 includes a set of components 432 that are non-OS specific and adapt to any platform, and a set of components 430 that are OS specific and need to be reimplemented for each platform. The non-OS specific set of components 432 includes an RTC listener 406, an RTC manager 408, a window manager 410, and an RTC component 412.

[0034]

[0041] As further shown in Figure 4, the client computing device 112 also includes a remote desktop client 402. As previously described, the remote desktop client 402 of the client computing device 112 is configured to receive redirected communications over the communication channel 306 from a communications application (i.e., via the plug-in 302 of the cloud computing environment 102 of Figure 3) to enable peer-to-peer voice and / or video communications between the client computing device 112 and a remote computing device (e.g., the remote computing device 114 of Figures 1 and 2).

[0035]

[0042] In a process of enabling peer-to-peer communication between the client computing device 112 and the remote computing device 114, the RTC listener 406 is configured to listen for redirected communications on the communication channel 306 and provide a stream of redirected communications to the RTC manager 408 upon receipt. The RTC manager 408 is configured to receive the stream of redirected communications (e.g., serialized RPC call information) and convert such communications into a format that conforms to a framework (e.g., WebRTC) for enabling real-time communications. The RTC manager 408 then provides the converted communications (e.g., WebRTC API calls) to the RTC component 412. The RTC component 412 is configured to connect the client computing device 112 and the remote computing device 114 for voice and / or video communication based on the converted communications (e.g., WebRTC API calls) provided from the RTC manager 408. The RTC component 412 can establish a communication channel 436 between the client computing device 112 and the remote computing device 114 for voice and / or video communication.

[0036]

[0043] The set of components 430 of the OS specific plug-in 404 includes a media capture component 414, a media renderer component 416, and a window manager component 418. After the client computing device 112 and the remote computing device 114 are connected for audio and / or video communication, the media capture component 414 and the media renderer component 416 can be used to detect hardware-based media processing capabilities of the client computing device 112 and use the hardware-based media processing capabilities to optimize the audio and / or video communication between the client computing device 112 and the remote computing device 114.

[0037]

[0044] For example, the media capturing component 414 can determine the hardware-based media processing capabilities of the client computing device 112 by using OS-specific APIs to determine what is available on the client computing device 112. After determining what is available on the client computing device 112, the media capturing component 414 is configured to inform the source reader 420 of any information related to processing the captured media (such as formatting) and any configuration of hardware required to process the captured media using the hardware-based media processing capabilities. The source reader 420 is configured to capture media from the video camera 426 and other device sources (e.g., microphone) of the client computing device 112.

[0038]

[0045] To aid in explanation, assume that the media capture component 414 determines that the client computing device 112 has hardware-based media processing capabilities to transcode the captured video to a lower resolution / quality. The media capture component 414 can inform the source reader 420 that the original captured media needs to be decoded to an intermediate uncompressed format (e.g., PCM for audio or YUV for video) and then encoded to the target format. The source reader 420 can then perform the transcoding and provide the processed video to the media capture component 414. When capturing video, the source reader 420 can act like a frame server and stream the captured video to the media capture component 414.

[0039]

[0046] The media capture component 414 is further configured to receive processed and / or unprocessed media captured from the video camera 426 from the source reader 420 and provide the captured media to the RTC component 412. The RTC component 412 is configured to prepare the captured media for transmission to the remote computing device 114. As shown in FIG. 4, the RTC component 412 can send the captured media to the remote computing device 114 over a communication channel 436 via a socket API 424.

[0040]

[0047] The media renderer component 416 can determine the hardware-based media processing capabilities of the client computing device 112 by using OS-specific APIs to determine what is available on the client computing device 112. After determining what is available on the client computing device 112, the media renderer component 416 is configured to inform the media engine 422 of any information related to processing the received media (formatting, etc.) and any configuration of hardware required to process the received media using the hardware-based media processing capabilities.

[0041]

[0048] The media renderer component 416 is further configured to provide the particular source of the media received from the RTC component 412 to the media engine 422. The media engine 422 can then build the media pipeline and set it up for rendering. The media engine 422 is configured to provide the particular source (compressed or decompressed) to the media stream source 428. The media stream source 428 is configured to provide the media source to the hardware required to process the received media by leveraging hardware-based media processing capabilities. In one embodiment, the media stream source 428 can use a frame server approach to create video media.

[0042]

[0049] As shown in FIG. 4, the media stream source 428 is connected to a hardware decoder 438. In some embodiments, if the incoming media is compressed, the hardware decoder 438 can decode the incoming media. In addition, the media stream source 428 can provide the incoming media to other hardware-based media processing capabilities, such as hardware accelerated video processing (e.g., frame resizing, color conversion, etc.) or hardware accelerated audio processing. As shown in FIG. 4, the processed media can be provided to the remote desktop client 402 for rendering within a user interface 434 of the remote desktop client 402. Alternatively or additionally, the processed media can be rendered on a display 440. The window manager 410 and the window manager component 418 are configured to monitor and track any changes that need to be made to the video rendered within the user interface 434 (e.g., moving windows, resizing windows, pop-up menus, etc.).

[0043]

[0050] The client computing device 112 is described in more detail below with respect to Figures 5-7. Figure 5 illustrates a flow diagram 500 of a method for enabling peer-to-peer communication between a client computing device and a remote computing device, determining hardware-based media processing capabilities of the client computing device, and using the detected hardware-based media processing capabilities to process media.

[0044]

[0051] Flow diagram 500 begins at step 502, where a redirected communication is received from a communication application to enable peer-to-peer voice and / or video communications between the client computing device and the remote computing device as opposed to voice and / or video communications via the communication application. For example, and with continued reference to FIG. 4, remote desktop client 402 receives a redirected communication from a communication application over communication channel 306 to enable peer-to-peer voice and / or video communications between client computing device 112 and remote computing device 114 as opposed to voice and / or video communications via the communication application.

[0045]

[0052] At step 504, a framework for enabling real-time communications is used to connect the client computing device and the remote computing device for voice and / or video communications based on the redirected communications. For example, and continuing to FIG. 4, the RTC component 412 uses a framework for enabling real-time communications to connect the client computing device 112 and the remote computing device 114 for voice and / or video communications based on the redirected communications. In one embodiment, the redirected communications include WebRTC API calls intercepted from the communications application, and the WebRTC APIs and protocols are the framework used to connect the client computing device 112 and the remote computing device 114.

[0046]

[0053] At step 506, the hardware-based media processing capabilities of the client computing device are determined. For example and continuing to FIG. 4, the media renderer component 416 determines the hardware-based media processing capabilities of the client computing device 112 by using OS-specific APIs to determine what is available on the client computing device 112. Such hardware-based media processing capabilities of the client computing device 112 may include, for example, without limitation, a video codec (including a video decoder), an audio codec (including an audio decoder), hardware accelerated video processing, or hardware accelerated audio processing.

[0047]

[0054] At step 508, the media transmitted from the remote computing device is received. For example, and continuing to FIG.

[0048]

[0055] At step 510, the received media from the remote computing device is processed for rendering by the client computing device using hardware-based media processing capabilities. For example, and continuing to FIG. 4, the media stream source 428 processes the received media from the remote computing device 114 for rendering by the client computing device 112 using hardware-based media processing capabilities. To aid in explanation, the media stream source 428 is connected to a hardware decoder 438. In some embodiments, the hardware decoder 438 may decode the received media if it is compressed. Additionally, the media stream source 428 may subject the received media to other hardware-based media processing capabilities, such as hardware-accelerated video processing (e.g., frame resizing, color conversion, etc.) or hardware-accelerated audio processing.

[0049]

[0056] 6 illustrates a flow diagram 600 of a method for negotiating media communication parameters with a remote computing device based on a determined hardware-based media processing capability of the client computing device. Flow diagram 600 begins with step 602, which determines another hardware-based video processing capability of the client computing device. For example, and continuing with FIG. 4, the RTC manager 408 may determine another hardware-based video processing capability of the client computing device 112. The RTC manager 408 may determine the other hardware-based media processing capability through an API of a framework for enabling real-time communications.

[0050]

[0057] At step 604, media communication parameters are negotiated with the remote computing device based on other hardware-based media processing capabilities. For example, and continuing to FIG. 4, the RTC manager 408 negotiates media communication parameters with the remote computing device 114 based on other hardware-based media processing capabilities. For example, the RTC manager 408 may determine that the client computing device 112 has a particular decoder (e.g., a VP9 decoder) that the remote computing device 114 does not have. The RTC manager 408 can negotiate with the remote computing device 114 to provide the media in a format compatible with the particular decoder of the remote computing device 114.

[0051]

[0058] FIG. 7 illustrates a flow diagram 700 of a method for determining hardware-based media processing capabilities of a client computing device and using the hardware-based media processing capabilities to process media captured from a media source of the client computing device. Flow diagram 700 begins with step 702. In step 702, another hardware-based media processing capability of the client computing device is determined. For example and continuing to FIG. 4, the media capture component 414 determines another hardware-based media processing capability of the client computing device 112. The media capture component 414 can determine the available hardware-based media processing capabilities of the client computing device 112 by using an API specific to the OS of the client computing device 112. Such hardware-based media processing capabilities of the client computing device 112 may include, for example, without limitation, a video codec (including a video encoder), an audio codec (including an audio encoder), hardware-accelerated video processing, or hardware-accelerated audio processing.

[0052]

[0059] At step 704, other hardware-based media processing capabilities are used to process the media captured from the media source of the client computing device. For example, a hardware-based video or audio encoder may be used to encode the video or audio, respectively, captured from the media source of the client computing device 112. As another example, hardware-accelerated video or audio processing may be used to accelerate the processing of the video or audio, respectively, captured from the media source of the client computing device 112. In some embodiments and continuing to FIG. 4, the media capture component 414 is configured to inform the source reader 420 of any information related to the processing of the captured media (e.g., formatting) and any configuration of hardware required to process the captured media utilizing the hardware-based media processing capabilities.

[0053]

[0060] Step 706 transmits the processed media to the remote computing device. For example, and continuing to FIG.

[0054]

[0061] Although the above embodiments refer to communication applications running on a virtual desktop host, it should be understood that the techniques described herein may be utilized in conjunction with any communication application running remotely from the client computing device 112, whether such communication application runs on a virtual desktop or some other type of platform. That is, the embodiments described herein are not limited to communication applications running on a virtual desktop, but broadly encompass any type of communication application that may run on a server and that enables voice and / or video communications between remotely located users and user devices.

[0055]

[0062] Additionally, although the above embodiments refer to remote desktops and associated plug-ins executing on the client computing device 112, it should be understood that the functionality provided by those components may be encapsulated within any type of software component having any type of structure or architecture. For example, the functionality and features of plug-in 404 may be included within a software component that is not necessarily a plug-in, but rather some other type of software component operable to function with the remote desktop client 402. Furthermore, the functionality and features of plug-in 404 may also be integrated directly within the remote desktop client 402 such that no additional software components are required beyond the remote desktop client 402. Still further, the functionality and features of the remote desktop client 402 and plug-in 404 may be included within an application or operating system component of the client computing device 112 that is not referred to as a remote desktop client or plug-in, but that provides similar functionality to the components described herein.

[0056] III. Example of computer system implementation

[0063] The cloud computing environment 102, the virtual desktop host 104, the plug-in 302, the client computing device 112, the remote desktop client 402, the plug-in 404, the flow chart 500, the flow chart 600, and / or the flow chart 700 may be implemented by hardware or hardware in combination with one or both of software and / or firmware. For example, the cloud computing environment 102, the virtual desktop host 104, the plug-in 302, the client computing device 112, the remote desktop client 402, the plug-in 404, the flow chart 500, the flow chart 600, and / or the flow chart 700 may be implemented as computer program code / instructions configured to be executed in one or more processors and stored in a computer readable storage medium. In another embodiment, the cloud computing environment 102, the virtual desktop host 104, the plug-in 302, the client computing device 112, the remote desktop client 402, the plug-in 404, the flow chart 500, the flow chart 600, and / or the flow chart 700 may be implemented by hardware operating a Software as a Service (SaaS) or Platform as a Service (PaaS) system. Alternatively, the cloud computing environment 102, the virtual desktop host 104, the plug-in 302, the client computing device 112, the remote desktop client 402, the plug-in 404, the flow chart 500, the flow chart 600, and / or the flow chart 700 may be implemented as hardware logic / circuitry.

[0057]

[0064] For example, in one embodiment, one or more of any combination of cloud computing environment 102, virtual desktop host 104, plug-in 302, client computing device 112, remote desktop client 402, plug-in 404, flowchart 500, flowchart 600, and / or flowchart 700 may be implemented together by a system on a chip (SoC). The SoC may include an integrated circuit chip including one or more of a processor (e.g., a central processing unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or additional circuitry, and optionally include embedded firmware for executing received program code and / or performing functions.

[0058]

[0065] 8 illustrates an exemplary implementation of a computing device 800 on which embodiments may be implemented. For example, the components of the cloud computing environment 102, virtual desktop host 104, plug-in 302, client computing device 112, remote desktop client 402, and plug-in 404 may each be implemented by one or more computing devices similar to computing device 800 including one or more features and / or alternative features of computing device 800, in either a fixed or mobile computing embodiment. The description of computing device 800 provided herein is provided for illustrative purposes and is not intended to be limiting. As will be appreciated by one of ordinary skill in the art, embodiments may be implemented by additional types of computer systems.

[0059]

[0066] As shown in FIG. 8, the computing device 800 includes one or more processors, referred to as processor circuitry 802, a system memory 804, and a bus 806 that couples various system components, including the system memory 804, to the processor circuitry 802. The processor circuitry 802 is an electrical and / or optical circuit implemented by one or more physical hardware electrical circuitry elements and / or integrated circuit devices (semiconductor material chips or dies) as a central processing unit (CPU), microcontroller, microprocessor, and / or other physical hardware processor circuitry. The processor circuitry 802 can execute program code stored in a computer readable medium, such as program code for an operating system 830, application programs 832, other programs 834, etc. The bus 806 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. The system memory 804 includes a read only memory (ROM) 808 and a random access memory (RAM) 810. A basic input / output system 812 (BIOS) is stored in the ROM 808 .

[0060]

[0067] The computing device 800 also has one or more of the following drives: a hard disk drive 814 for reading from and writing to a hard disk, a magnetic disk drive 816 for reading from and writing to a removable magnetic disk 818, and an optical disk drive 820 for reading from and writing to a removable optical disk 822, such as a CD ROM, DVD ROM, or other optical medium. The hard disk drive 814, magnetic disk drive 816, and optical disk drive 820 are connected to the bus 806 by a hard disk drive interface 824, a magnetic disk drive interface 826, and an optical drive interface 828, respectively. The drives and their associated computer-readable media provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer. Although hard disks, removable magnetic disks, and removable optical disks are described, other types of hardware-based computer-readable storage media such as flash memory cards, digital video disks, RAM, ROM, and other hardware storage media may be used to store data.

[0061]

[0068] A number of program modules may be stored on the hard disk, magnetic disk, optical disk, ROM, or RAM. These programs include an operating system 830, one or more application programs 832, other programs 834, and program data 836. The application programs 832 or other programs 834 may include, for example, the virtual desktop host 104, the plug-in 302, the remote desktop client 402, the plug-in 404, the flowchart 500, the flowchart 600, and / or the flowchart 700 (including any suitable steps of the flowcharts 500, 600, and 700), and / or computer program logic (e.g., computer program code or instructions) for implementing further embodiments described herein.

[0062]

[0069] A user may enter commands and information into computing device 800 by input devices such as a keyboard 838 and a pointing device 840. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, touch screen and / or touch pad, a voice recognition system for accepting voice input, a gesture recognition system for accepting gesture input, etc. These and other input devices are often connected to the processor circuitry 802 by a serial port interface 842 coupled to bus 806, but may also be connected by other interfaces, such as a parallel port, game port, or universal serial bus (USB).

[0063]

[0070] A display screen 844 is also connected to the bus 806 by an interface, such as a video adapter 846. The display screen 844 may be external to the computing device 800 or may be incorporated into the computing device 800. The display screen 844 may not only display information but may also be a user interface for accepting user commands and / or other information (e.g., by touch, finger gestures, a virtual keyboard, etc.). In addition to the display screen 844, the computing device 800 may include other peripheral output devices (not shown), such as speakers and a printer. The display screen 844 and / or any other peripheral output devices (not shown) may be used to implement the user interfaces 106 and 434 and / or any further embodiments described herein.

[0064]

[0071] Computing device 800 is connected to a network 848 (e.g., the Internet) by an adapter or network interface 850, a modem 852, or other means for establishing communications over the network. The modem 852, which can be internal or external, can be connected to the bus 806 via a serial port interface 842 as shown in Figure 8, or can be connected to the bus 806 using another interface type, including a parallel interface.

[0065]

[0072] As used herein, the terms "computer program medium," "computer readable medium," and "computer readable storage medium" are used to refer to other physical hardware media, such as a hard disk associated with the hard disk drive 814, a removable magnetic disk 818, a removable optical disk 822, RAM, ROM, flash memory cards, digital video disks, zip disks, MEMs, nanotechnology-based storage devices, and additional types of physical / tangible hardware storage media. Such computer readable storage media are distinct from and do not overlap with (and do not include) communication media. Communication media embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example and not limitation, communication media include wireless media such as acoustic, RF, infrared and other wireless media, as well as wired media. The embodiments also cover such communication media, which are separate and do not overlap with the embodiments directed to computer readable storage media.

[0066]

[0073] As mentioned above, computer programs and modules (including application programs 832 and other programs 834) may be stored on a hard disk, magnetic disk, optical disk, ROM, RAM, or other hardware storage medium. Such computer programs may also be received via a network interface 850, a serial port interface 842, or any other type of interface. Such computer programs, when executed or loaded by an application, enable the computing device 800 to implement features of the embodiments discussed herein. Thus, such computer programs represent controllers of the computing device 800.

[0067]

[0074] Embodiments are also directed to computer program products that include computer code or instructions stored on any computer readable medium, including hard disk drives, optical disk drives, memory device packages, portable memory sticks, memory cards, and other types of physical storage hardware.

[0068] IV. Further Example Embodiments

[0075] In a first embodiment, a client computing device includes one or more processors and one or more memory devices storing computer program logic for execution by the one or more processors, the computer program logic including: a remote desktop client configured to present within a user interface of a communications application executing within the cloud computing environment within a user interface of the client computing device, the remote desktop client further configured to receive redirected communications from the communications application to enable peer-to-peer voice and / or video communications between the client computing device and the remote computing device as opposed to voice and / or video communications via the communications application; a real-time communications manager configured to receive the redirected communications from the remote desktop channel component and convert the redirected communications into a format that conforms with a framework for enabling real-time communications; a real-time communications component configured to connect the client computing device and the remote computing device for voice and / or video communications based on the converted communications; and a plug-in including a media capture component configured to determine hardware-based media processing capabilities of the client computing device, process media captured from a media source of the client computing device using the hardware-based media processing capabilities, and provide the processed media to the real-time communications component for transmission to the remote computing device.

[0069]

[0076] In one embodiment, the media includes one of video or audio.

[0070]

[0077] In one embodiment, the real-time communications manager is further configured to determine other hardware-based video processing capabilities of the client computing device and negotiate media communications parameters with the remote computing device based on the other hardware-based media processing capabilities.

[0071]

[0078] In one embodiment, the media capture component determines hardware-based media processing capabilities through an application programming interface (API) of an operating system of the client computing device, and the real-time communications manager determines other hardware-based media processing capabilities through an API of a framework for enabling real-time communications.

[0072]

[0079] In one embodiment, the plug-in further includes a media renderer component configured to receive media transmitted from the remote computing device by the real-time communication component and provide the received media to a media engine of the client computing device for rendering within a user interface of the client computing device, and the video renderer component is further configured to determine additional hardware-based media processing capabilities of the client computing device and use the other media processing capabilities to process the received media from the remote computing device.

[0073]

[0080] In one embodiment, the hardware-based media processing capabilities include one of a video codec or an audio codec.

[0074]

[0081] In one embodiment, the hardware-based media processing capabilities include one of hardware accelerated video processing or hardware accelerated audio processing.

[0075]

[0082] In another embodiment, a method performed by one or more processors of a client computing device for presenting a user interface of a communications application executing within a cloud computing environment includes receiving a redirected communication from the communications application to enable peer-to-peer voice and / or video communication between the client computing device and a remote computing device as opposed to voice and / or video communication via the communications application, using a framework to enable real-time communication, connecting the client computing device and the remote computing device for voice and / or video communication based on the redirected communication, determining hardware-based media processing capabilities of the client computing device, receiving media transmitted from the remote computing device, and processing the received media from the remote computing device for rendering by the client computing device using the hardware-based media processing capabilities.

[0076]

[0083] In one embodiment, the media includes one of video or audio.

[0077]

[0084] In one embodiment, the method further includes determining another hardware-based video processing capability of the client computing device, and negotiating media communication parameters with the remote computing device based on the other hardware-based media processing capabilities.

[0078]

[0085] In one embodiment, determining the hardware-based media processing capabilities includes determining the hardware-based media processing capabilities using an application programming interface (API) of an operating system of the client computing device, and determining the other hardware-based media processing capabilities includes determining the other hardware-based media processing capabilities using an API of a framework for enabling real-time communications.

[0079]

[0086] In one embodiment, the method further includes determining another hardware-based media processing capability of the client computing device, using the other hardware-based media processing capability to process the media captured from the media source of the client computing device, and transmitting the processed media to the remote computing device.

[0080]

[0087] In one embodiment, the hardware-based media processing capabilities include one of a video codec or an audio codec.

[0081]

[0088] In one embodiment, the hardware-based media processing capabilities include one of hardware accelerated video processing or hardware accelerated audio processing.

[0082]

[0089] In another embodiment, a computer-readable storage medium having recorded thereon program instructions that, when executed by at least one processor of a client computing device presenting a user interface of a communications application executing within a cloud computing environment, cause the at least one processor to execute a method including receiving a redirected communication from the communications application to enable peer-to-peer voice and / or video communications between the client computing device and a remote computing device in contrast to voice and / or video communications via the communications application, using a framework for enabling real-time communications, connecting the client computing device and the remote computing device for voice and / or video communications based on the redirected communication, determining hardware-based media processing capabilities of the client computing device, and negotiating media communication parameters with the remote computing device based on the hardware-based media processing capabilities.

[0083]

[0090] In one embodiment, the media includes one of video or audio.

[0084]

[0091] In one embodiment, the method further includes determining another hardware-based video processing capability of the client computing device, and receiving media transmitted from the remote computing device, and processing the received media from the remote computing device using the other hardware-based media processing capability for rendering by the client computing device.

[0085]

[0092] In one embodiment, the method further includes determining another hardware-based media processing capability of the client computing device, using the other hardware-based media processing capability to process the media captured from the media source of the client computing device, and transmitting the processed media to the remote computing device.

[0086]

[0093] In one embodiment, the hardware-based media processing capabilities include one of a video codec or an audio codec.

[0087]

[0094] In one embodiment, the hardware-based media processing capabilities include one of hardware accelerated video processing or hardware accelerated audio processing.

[0088] V. Conclusion

[0095] While various embodiments of the present invention have been described above, it should be understood that they are presented by way of example only, and not by way of limitation. Those skilled in the art will recognize that various changes in form and detail can be made therein without departing from the spirit and scope of the present invention as defined by the appended claims. Thus, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the appended claims and their equivalents.

Claims

1. one or more processors; one or more memory devices storing computer program logic for execution by said one or more processors; a client computing device comprising: a remote desktop client configured to present within a user interface of the client computing device a user interface of a communications application executing within a cloud computing environment, the remote desktop client further configured to receive redirected communications from the communications application to enable peer-to-peer voice and / or video communications between the client computing device and a remote computing device as opposed to voice and / or video communications via the communications application; a real-time communication manager configured to receive the redirected communication from the remote desktop client and convert the redirected communication into a format compatible with a framework for enabling real-time communications; a real-time communication component configured to connect the client computing device and the remote computing device for audio and / or video communication based on the converted communication; and a media capture component configured to determine a hardware-based media processing capability of the client computing device, process media captured from a media source of the client computing device using the hardware-based media processing capability, and provide the processed media to the real-time communication component for transmission to the remote computing device. Plugins including a client computing device comprising:

2. The client computing device of claim 1 , wherein the media includes one of video or audio.

3. The real-time communication manager, determining additional hardware-based video processing capabilities of the client computing device; Negotiating media communication parameters with the remote computing device based on the other hardware-based media processing capabilities. The client computing device of claim 1 , further configured to:

4. 4. The client computing device of claim 3, wherein the media capture component determines the hardware-based media processing capabilities through an application programming interface (API) of an operating system of the client computing device, and the real-time communications manager determines the other hardware-based media processing capabilities through an API of the framework for enabling real-time communications.

5. a media renderer component configured to receive media transmitted from the remote computing device by the real-time communication component and to provide the received media to a media engine of the client computing device for rendering within the user interface of the client computing device, the video renderer component being determining additional hardware-based media processing capabilities of the client computing device; using the other media processing capabilities to process the media received from the remote computing device The client computing device of claim 1 , further configured to:

6. The client computing device of claim 1 , wherein the hardware-based media processing capabilities include one of a video codec or an audio codec.

7. The client computing device of claim 1 , wherein the hardware-based media processing capabilities include one of hardware accelerated video processing or hardware accelerated audio processing.

8. 1. A method performed by one or more processors of a client computing device for presenting a user interface for a communications application executing within a cloud computing environment, the method comprising: receiving redirected communications from the communications application to enable peer-to-peer voice and / or video communications between the client computing device and a remote computing device as opposed to voice and / or video communications by the communications application; connecting said client computing device and said remote computing device for voice and / or video communication based on said redirected communication using a framework for enabling real-time communication; determining a hardware-based media processing capability of the client computing device; receiving media transmitted from the remote computing device; and processing the media received from the remote computing device for rendering by the client computing device using the hardware-based media processing capabilities; A method comprising:

9. The method of claim 8 , wherein the media includes one of video or audio.

10. determining additional hardware-based video processing capabilities of the client computing device; and negotiating media communication parameters with the remote computing device based on the other hardware-based media processing capabilities. The method of claim 8 further comprising:

11. determining the hardware-based media processing capabilities includes determining the hardware-based media processing capabilities using an application programming interface (API) of an operating system of the client computing device; determining the other hardware-based media processing capabilities includes determining the other hardware-based media processing capabilities using an API of the framework for enabling real-time communications. The method of claim 10.

12. determining additional hardware-based media processing capabilities of the client computing device; using the other hardware-based media processing capability to process media captured from a media source on the client computing device; and transmitting the processed media to the remote computing device. The method of claim 8 further comprising:

13. The method of claim 8 , wherein the hardware-based media processing capabilities include one of a video codec or an audio codec.

14. The method of claim 8 , wherein the hardware-based media processing capabilities include one of hardware accelerated video processing or hardware accelerated audio processing.

15. Computer program logic means for enabling a processor to carry out any of claims 8 to 14.

2. A computer program product comprising a computer readable medium having computer program logic recorded thereon, the computer program logic comprising:

Images