Web Browser Multimedia Redirection

Abstract

A computer system, method, and server system are provided that can reduce the workload and associated costs imposed on server hardware. In one embodiment, a computer system running a host virtual machine (34) that hosts a session with a client computing device (12) accesses media content from a media source (16), runs a web browser application (36), receives encoded media content (18) in a media container format (38) from the media source (16), and executes a multimedia redirection module (54) that intercepts the encoded media content (18) from being processed by a decoding module (38). The multimedia redirection module (54) redirects the encoded media content (18) to the client computing device (12).

Description

[Background technology]

[0001] In a virtual environment, end users may use client devices to connect to virtual machines running on server hardware, which may provide the end users with the benefits of additional computing power, application compatibility, security and regulatory compliance, and reduced overall costs for the end users. Typically, these end users may consume media such as video streaming, which can potentially increase the workload imposed on the server hardware and increase associated costs. Summary of the Invention

[0002] A computer system is provided that includes at least one processor configured to execute a host virtual machine configured to host a session with at least one client computing device. The at least one processor may be further configured to execute a web browser application configured to access media content from a remote media source, and a multimedia redirection module configured to receive encoded media content in a media container format from the remote media source and intercept the encoded media content from being processed by a decoding module of the web browser application. The multimedia redirection module may be configured to redirect the encoded media content to the at least one client computing device.

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Moreover, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a schematic diagram of an exemplary computer system for implementing the multimedia redirection described herein.

[0005] [Figure 2] FIG. 2 is a schematic diagram of an example server system of the computer system of FIG. 1.

[0006] [Figure 3] FIG. 2 is a schematic diagram of a web browser application implementing a multimedia redirection module to shim the rendering pipeline of the web browser application for the computer system of FIG. 1.

[0007] [Figure 4] 2 is a flowchart of a method for performing multimedia redirection implemented by the computer system of FIG. 1.

[0008] [Figure 5] Figure 1 is a schematic diagram of the software architecture for a web browser application implementing multimedia redirection for a computer system.

[0009] [Figure 6]FIG. 2 is a schematic diagram of the software architecture for a remote desktop client communicating with a multimedia redirection module implemented by the computer system of FIG. 1.

[0010] [Figure 7] FIG. 2 is a schematic diagram of an exemplary computing environment in which the computer system of FIG. 1 may be defined. DETAILED DESCRIPTION OF THE INVENTION

[0011] A cloud platform service may provide a virtualized desktop environment to end users who may utilize client computing devices to connect to virtual machines running on the cloud platform's server devices. Typically, the cloud platform's computing hardware and network capabilities are more powerful than the end users' client computing devices. Thus, by hosting these end users in a virtualized desktop environment, the cloud platform may provide these users with additional computing power, as well as other potential benefits such as improved application compatibility, security / regulatory compliance, and an overall reduction in associated computing costs.

[0012] In some examples, these cloud platform services may provide a virtual desktop infrastructure for enterprises with multiple end users and client devices. Users of these enterprises may consume media, such as video and / or audio media, that is stored locally on the enterprise's cloud platform or stored by a remote media source and streamed to a virtualized desktop environment running on the cloud platform. Typically, media content provided by the media source is decoded on a virtual machine hosting the end user, and then display frames of the virtualized desktop presenting the media content are re-encoded and streamed to the user's client computing device via a remote desktop protocol. The client computing device may then decode these display frames and present them to the user via the client computing device's display.

[0013] The decoding and subsequent encoding processes performed on a virtual machine can consume significant computational resources of the server hardware. In one particular example, the video rendering process for video streaming can potentially consume 20% to 40% of the central processing unit (CPU) resources of the server device running the virtual machine. These problems can be exacerbated in multi-session virtualized desktop environment deployments where multiple users share an operating system (OS) instance on a VM. For example, if two or more users attempt to stream video content through a shared host VM, potentially all of the CPU resources for that VM can be consumed by the associated decoding / encoding / rendering processes, thus potentially degrading the user experience for all users hosted by that VM.

[0014] To address the above-described problems, FIG. 1 illustrates an exemplary computer system 10 for a virtualized desktop environment that performs multimedia redirection to reduce the computational workload on the server hardware running the virtualized desktop environment, as described in more detail below. As illustrated in FIG. 1, computer system 10 includes one or more client computing devices 12, a server system 14, and one or more media sources 16. End users of one or more client computing devices 12 may connect to virtual machines running on server system 14. Using applications running within the VM's host operating system, such as a web browser application, the end users may interact with media stored by media source 16.

[0015] In one example, media source 16 may be local to or included in server system 14. In another example, media source 16 may be remote from both client computing device 12 and server system 14. For example, server system 14 may be configured to communicate with remote media sources over a communications network, such as a wide area network (WAN). As a specific example, an end user may provide input to a web browser application running within a host VM to navigate to a website associated with media source 16, such as a video streaming website. Using the associated website, the end user may initiate an experience that includes streaming video and / or audio. Typically, the streaming video and / or audio is sent to server system 14 as encoded media content 18 and processed by the host VM running the web browser application that requested the streaming video and / or audio.

[0016] As described above, a typical virtualization environment implementation would decode encoded media content 18, present the media content via a virtualized desktop environment, encode display / audio frames for the virtualized desktop environment, and then send the encoded display / audio frames to client computing device 12 for presentation to the end user. To reduce the computational load imposed on the hardware of server system 14 by the decoding / encoding process, server system 14 may implement multimedia redirection to pass encoded media content 18 received from media source 16 to client computing device 12 without performing the encoding or decoding process on server system 14.

[0017] FIG. 2 illustrates an example of a server system 14. The server system 14 may include a hardware plane 20, a virtual machine plane 22, a hypervisor plane 24, and a network infrastructure 26, collectively configured to operate a cloud platform. The hardware plane 20 includes a collection of nodes 28 (each represented by the symbol "N" in FIG. 2 ), which may include a processor, a graphics processing unit (GPU), volatile memory, and other computer components configured to run host server instances. The host server instances executed by the nodes 28 of the hardware plane 20 are configured to communicate with one or more hypervisors of the hypervisor plane 24. The one or more hypervisors of the hypervisor plane 24 may create, manage, and monitor multiple virtual machines 30 of the virtual machine plane 22 (each represented by the symbol "VM" in FIG. 2 ). Through the hypervisor plane 24, each virtual machine 30 of the virtual machine plane 22 may be hosted and executed by hardware components of one or more nodes 28 of the hardware plane 20. In this way, multiple virtual machines 30 in the virtual machine plane 22 may share virtualized hardware resources managed by the hypervisor plane 24. Each virtual machine 30 provides a virtualized desktop environment in which software such as web browser applications, media player applications, and other types of software may be executed.

[0018] In one example, the computer system 14 corresponds to a data center environment that communicatively couples multiple nodes 28 via a standard network infrastructure. For example, the network infrastructure 26 may include a typical network infrastructure, such as a server rack including top-of-rack (TOR) network switches. The computer system 14 may include multiple node clusters, each with an associated TOR network switch. The network infrastructure 26 may further include higher-level switching infrastructure 32 (L1) and (L2) that connects the TOR network switches together. The higher-level switching infrastructure 32 may take the form of any suitable networking architecture and may be driven by any suitable routing protocol. In the illustrated example, the higher-level switching infrastructure 32 includes a collection of aggregation switches L1 and core switches L2. However, it will be appreciated that the higher-level switching infrastructure may include any suitable number of levels of switches.

[0019] The virtual machines 30 of the virtual machine plane 22 provide a virtual computing environment in which users of the server system 14 may access and run applications such as web browsers, media player applications, etc. As discussed in more detail below, these virtual machines 30 of the virtual machine plane 22 may be configured to implement multimedia redirection to reduce the computational workload imposed on the nodes of the hardware plane 20 caused by the decoding and encoding processes typically performed for streaming media.

[0020] Additionally, it should be appreciated that server system 14 is not limited to the example data center / cloud platform illustrated in Figure 2. In other examples, server system 14 may take the form of one or more computing devices configured to execute a virtualized desktop environment. Using client computing device 12, an associated user may connect to and access software executing within the virtualized desktop environment executed by one or more computing devices, and such virtualized desktop environment may be configured to implement the multimedia redirection features and processes described herein.

[0021] 3 illustrates an example host virtual machine 34 configured to implement multimedia redirection techniques to address the above-described problems. The host virtual machine 34 may take the form of one of the virtual machines 30 of the virtual machine plane 22 of the cloud platform described in connection with FIG. 2. However, it should be appreciated that the host virtual machine 34 may also be implemented in the form of a non-cloud platform of the server system 14.

[0022] The host virtual machine 34 may be executed by at least one processor of the server system 14. The host virtual machine 34 may be configured to host sessions with at least one client computing device 12 and may host applications executing within the operating system of the host virtual machine 34. In one example, the host virtual machine 34 may include a web browser application 36 executed by at least one processor of the server system 14. As shown, the web browser application 36 may be configured to access media content from a media source 16. Using a graphical user interface (GUI) of the web browser application 36, a user may navigate to a target media source 16, such as a web page and associated server that provides the media content. In one example, the media source 16 may be local to the server system 14 executing the host virtual machine 34. In another example, the media source 16 may be remote from the server system 14 and configured to transmit encoded media content 18 to the server system 14 over a WAN.

[0023] The host virtual machine 34 may be configured to receive encoded media content 18 from a media source 16, which in the illustrated example may take the form of a remote media source. The encoded media content 18 may be encoded using any suitable algorithm. In one example, the encoded media content 18 is received in a media container format 38 that includes one or more of a video stream 40, an audio stream 42, and a text stream 44. As specific examples, the media container format 38 may be an MP4 format, a WAV format, an AVI format, or another type of media container format 38.

[0024] The encoded media content 18 received by the host VM 34 may be processed by a media source extensions (MSE) module 46 of the web browser application 36 executing on the host VM 34. The MSE module 46 may be configured to provide functionality for web-based media streaming. The MSE module 46 may be further configured to demultiplex the encoded media content 18 in the media container format 38 into separate video, audio, and text streams. Typically, as shown in FIG. 3, the separate encoded audio stream 42 and encoded video stream 40 will be processed by audio and video rendering pipelines of the web browser application 36. For example, the encoded audio stream 42 and the encoded video stream 40 may be processed by a decoder 48 that decompresses the audio and video streams, respectively. The decompressed audio and video streams may then be passed to an audio renderer 50 and a video renderer 52, respectively, to prepare the video and audio content for presentation. As mentioned above, these decoding and rendering processes performed on the host VM 34 consume hardware resources of the server device running the host VM 34, which can potentially degrade the user experience of other users hosted within the multi-session operating system of the host virtual machine 34.

[0025] To address these issues, the host virtual machine 34 may be configured to implement a multimedia redirection module 54 configured to redirect encoded media content 18, including encoded audio streams 42 and encoded video streams 40, to the client computing device 12. Figure 4 is a flowchart of a method 400 implemented by a server system 14 executing the host virtual machine 34 to perform multimedia redirection. The method 400 may be performed using the systems described above or utilizing other suitable hardware and software elements.

[0026] At 402, method 400 may include executing a host virtual machine configured to host a session with at least one client computing device. The host virtual machine 34 may be executed by at least one processor of a server system 14, which may take the form of, for example, a cloud platform data center as illustrated in FIG. 2. In one example, the host virtual machine 34 may implement a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices 12. The multiple sessions may be hosted on the same multi-session operating system instance. Each session may implement the techniques and processes for multimedia redirection described herein.

[0027] At 404, the method 400 may include executing a web browser application 36 configured to access media content from a remote media source. The web browser application 36 may execute within the host virtual machine 34. In an example multi-session operating system, each of multiple sessions may have an associated web browser application 36 instance executing in the multi-session operating system. The remote media source 16 may take the form of a third-party media service that streams media content such as video and / or audio. The web browser application 36 may be configured to access the remote media source 16 based on user web navigation input, such as a URL address of the remote media source 16.

[0028] At 406, the method 400 may include receiving encoded media content 18 from a remote media source in a media container format. The encoded media content 18 may be received by the host virtual machine 34 over a WAN. As shown in FIG. 3, the encoded media content 18 may include one or more of an encoded video stream 40, an encoded audio stream 42, and an encoded text stream 44. The encoded streams of media content may be bundled in a media container format 38, such as an MP4 format, a WAV format, or the like.

[0029] At 408, the method 400 may include demultiplexing the encoded media content in the media container format into independent streams of encoded media content. For example, the web browser application 36 may be configured to implement an MSE module 46 that includes processes for unpacking the media container format 38 and demultiplexing the encoded media content 18 into independent streams, such as an encoded video stream 40, an encoded audio stream 42, and / or an encoded text stream 44.

[0030] At 410, the method 400 may include executing a multimedia redirection module 54 configured to intercept encoded media content from being processed by a decoding module of the web browser application. The multimedia redirection module 54 may take the form of a shim that intercepts application programming interface (API) calls and modifies the arguments passed and the system behavior applied to those arguments. For example, the multimedia redirection module 54 may intercept API calls that would pass encoded audio / video / text streams to their respective decoders and renderers and redirect the encoded content to the associated client computing device 12 for the session. In this way, the encoded media content 18 is not decoded by the decoder 48 on the host virtual machine 34.

[0031] The encoded media content 18 being redirected to the client computing device 12 may be buffered and streamed over a WAN to the client computing device 12. At 412, the method 400 may include buffering the independent streams of encoded media content. The independent streams of encoded media content, such as the encoded audio stream 42 and the encoded video stream 40, may be temporarily stored in a buffer on the host virtual machine 34.

[0032] At 414, method 400 may include transmitting media samples of the buffered independent streams of encoded media content to at least one client computing device in response to receiving a request for media samples from the at least one client computing device. Client computing device 12 may request new media samples independently for each stream of encoded media content 18. For example, client computing device 12 may request media samples for encoded video stream 40 independently from requesting media samples for encoded audio stream 42. In this manner, client computing device 12 may request media samples at an independent rate for each independent stream of encoded media content (e.g., encoded audio stream, encoded video stream, etc.), and host virtual machine 34 may transmit these media samples for each independent stream of encoded media content at the requested rate. Video content is typically larger in size than audio content and, therefore, may require a higher rate of video samples being transmitted to client computing device 12.

[0033] 5 illustrates an exemplary server-side software architecture for implementing the multimedia redirection module 54 and the steps described in method 400. In the illustrated example, at least one processor 56 of a server device included in server system 14 may be configured to execute a host virtual machine 34 configured to host a session 58 with at least one client computing device 12. In the illustrated example, the host virtual machine 34 implements a multi-session operating system 60 configured to simultaneously host multiple sessions 58 for multiple client computing devices 12. Each session 58 may be associated with a different client computing device 12 and may run a separate instance of an application program on the host virtual machine 34. For example, each session 58 may run a separate instance of a web browser application 36 configured to access media content from remote media sources 16.

[0034] Each session 58 executing within the multi-session operating system 60 may be further configured to receive encoded media content 18 in a media container format 38 from a remote media source 16, as described above with reference to FIGS. 3 and 4. In one example, the encoded media content 18 may take the form of dynamic streaming media content. In another example, the encoded media content 18 may take the form of HTTP progressive streaming content. However, it should be appreciated that the encoded media content 18 may take other suitable forms. Additionally, the encoded media content 18 may be received in a media container format that includes one or more of a video stream, an audio stream, and a text stream. The video stream, the audio stream, and the text stream may be multiplexed into the media container format. It should be appreciated that the media container format may take the form of an MP4 format, a WAV format, or any other suitable type of media container format.

[0035] Execution of web browser application 36 may spin up multiple related processes executing within multi-session operating system 60. As a specific example, a web browser application process 62 may be executed for session 58 with client computing device 12. In addition, a separate renderer process 64 may be executed for each tab opened in web browser application 36. These separate renderer processes 64 may be sandboxed and may execute at a lower integrity level compared to web browser application process 62.

[0036] Each renderer process 64 may include a demultiplexer module 66, a renderer client 68, and a courier renderer 70. The demultiplexer module 66 may be configured to demultiplex the encoded media content 18 in the media container format 38 into independent streams of encoded media content 72. The demultiplexed independent streams of encoded media content 72 may include, for example, an encoded video stream, an encoded audio stream, and / or an encoded text stream.

[0037] The courier renderer 70 forms part of the MMR module 54 and may be configured to request and receive compressed media samples from the demultiplexed independent streams of encoded media content 72. In one example, the courier renderer 70 may also be configured to notify the renderer client 68 of changes in playback state 74 and playback times for the encoded media content 18.

[0038] In one example, when multimedia redirection (MMR) is enabled, the courier renderer 70 executes within the renderer process 64. If MMR is not currently enabled for the received encoded media content 18, then the renderer process 64 may be configured to execute a media renderer in place of the courier renderer 70, which may be configured to handle decoding and rendering of the encoded media content via the decoder 48, audio renderer 50, and video renderer 52. When MMR is enabled, the courier renderer 70 is loaded in place of a typical media renderer and is configured to intercept the demuxed stream of media content and perform the functions and processes described herein. In this manner, the demuxed stream of media content is not decoded and rendered on the host virtual machine 34.

[0039] The MMR module 54 may further include a remote desktop connection (RDC) remoting connector module 76. The courier renderer 70 may be further configured to communicate cross-process with the RDC remoting connector module 76. In one example, the RDC remoting connector module 76 may execute within the process of the web browser application process 62. In another example, the RDC remoting connector module 76 may execute within a separate process separate from the web browser application process 62. As shown in FIG. 5 , the courier renderer 70 may include a renderer controller 78 configured to handle communication with the RDC remoting connector module 76 via sending and receiving serialized messages 80. As a particular example, the serialized messages 80 may include commands serialized using a protocol such as extended GOOGLE® PROTOCOL BUFFERS, a language- and platform-neutral mechanism for serializing and deserializing structured data. However, it should be recognized that other serialization protocols may also be implemented.

[0040] The courier renderer 70 may be configured to send demultiplexed media samples for the demultiplexed independent streams 72 of encoded media content to the RDC remoting connector module 76. The courier renderer 70 may also be configured to convert renderer commands into serialized messages 80 that are also sent to the RDC remoting connector module 76. Furthermore, the courier renderer 70 may be configured to convert serialized messages 80 received from the RDC remoting connector module 76 into callbacks in the renderer client 68.

[0041] As described above, each tab opened in an instance of a web browser application may have an associated renderer process 64. The instance of the courier renderer 70 for each of these instances of the renderer process 64 may communicate with a single RDC remoting connector module 76 for that instance of the web browser application running in session 58 with the client computing device 12. The RDC remoting connector module 76 may include a remoting bridge 82 for each courier renderer 70 configured to exchange serialized messages with the associated courier renderer 70. In one example, the RDC remoting connector module 76 may be configured to function as a multiplexer, allowing multiple courier renderers to remoting to the client computing device 12 simultaneously.

[0042] 5, the RDC remoting connector module 76 may further include an RDC media remoting core 84 that includes logic for performing the functions and processes of the RDC remoting connector module 76. The RDC media remoting core 84 may also be configured to handle the transmission of the demultiplexed independent streams 72 of encoded media data received at the remoting bridge 82 to a dynamic virtual channel (DVC) plug-in implemented on the client computing device 12. The RDC remoting connector module 76 may also handle the communication of serialized messages 80 with the DVC plug-in on the client computing device 12.

[0043] As described above, the MMR module 54 may include an RDC remoting connector module 76 and a courier renderer 70. Using the processes described above, the MMR module 54 may be configured to intercept encoded media content 18 from being processed by the decoding module 48 of the web browser application 36. For example, when MMR is enabled, the courier renderer 70 may execute in place of a typical media renderer in the renderer process 64. That is, the MMR module 54 may cause media samples from the demultiplexed independent streams (e.g., video / audio / text streams) of the encoded media content to be sent to the courier renderer 70, which is configured to send these media samples to the associated remoting bridge 82 of the RDC remoting connector module 76. Using the functionality described for the RDC remoting connector module 76, the MMR module 54 may transmit these media samples to the client computing device 12 over a WAN. In this manner, the encoded media content is redirected by the MMR module 54 to at least one client computing device 12 such that the encoded media content is not decoded or rendered on the host virtual machine 34.

[0044] Additionally, the MMR module 54 may be configured to buffer the independent streams 72 of encoded media content and, in response to receiving a request for the media samples from the at least one client computing device 12, transmit media samples of the buffered independent streams 72 of encoded media content to the at least one client computing device 12. For example, the client computing device 12 may send a serialized message 80 including a request 88 for media samples for at least one of the streams of encoded media content to the RDC remoting connector module 76. The RDC remoting connector module 76 may request the courier renderer 70 to send the requested media samples to an associated remoting bridge 82. The RDC remoting connector module 76 may then transmit the requested encoded media samples 86 to the client computing device 12 via a dynamic virtual channel. In one example, the client computing device 12 may send a separate request 88 for each independent stream 72 of encoded media content 2. For example, a client computing device 12 may request media samples of an encoded video content stream at a higher rate than media samples of encoded audio content. Accordingly, MMR module 54 may be configured to transmit media samples 86 for each of the independent streams of encoded media content 72 at an independent rate based on requests 88 for media samples received from at least one client computing device 12.

[0045] As described above, the MMR module 54, which may include the RDC remoting connector module 76 and the courier renderer 70, is loaded and executed when MMR is enabled for the encoded media content 18. In one example, the at least one processor 56 may be configured to determine, based on one or more monitored parameters 90, whether to enable MMR and whether to execute the MMR module 54 to intercept the encoded media content 18 or allow it to be processed by the decoding module 48 of the web browser application 36. In one example, the host virtual machine 34 may be configured to monitor network parameters between the server system 14 and the at least one client computing device 12 and determine whether to enable MMR based on the monitored network parameters.

[0046] As another example, the host virtual machine 34 may be configured to monitor the current processing load of at least one processor 56 executing the host virtual machine 34 and determine whether to enable MMR based on the current processing load. As a specific example, if the current processing load on the at least one processor 56 exceeds a threshold level, the host virtual machine 34 may be configured to enable MMR and load the MMR module 54 and its associated media stack to intercept the encoded media content 18 from being processed by the standard rendering process that performs decoding and rendering. On the other hand, if the current processing load is below the threshold level and the monitored network parameters indicate low network bandwidth, the host virtual machine may be configured to disable MMR and decode / render the encoded media content. The rendered frames may then be encoded at a compression level appropriate for the network bandwidth between the server system and the client computing device.

[0047] As another example, the host virtual machine 34 may be configured to determine whether to enable MMR and load the MMR module 54 based on user settings for at least one client computing device 12. For example, an administrator may control user settings to enable or disable MMR for a client computing device. The user settings may be transmitted to the host virtual machine 34 and stored for that client computing device. In one specific example, MMR may be enabled or disabled independently via user settings for different media sources 16.

[0048] As described above, the host virtual machine 34 may implement a multi-session operating system 60 that hosts sessions 58 with multiple client computing devices 12. Activity with each client computing device 12 may increase the processing workload imposed on at least one processor 56 running the host virtual machine 34. Accordingly, in one example, the host virtual machine 34 may be configured to determine the aggregate activity of the multiple sessions 58 hosted by the multi-session operating system 60 and determine whether to execute the MMR module 54 based on the aggregate activity. As a specific example, if the aggregate activity of the multiple sessions 58 is low because only one session is currently accessing the encoded media content for streaming, the host virtual machine 34 may be configured not to enable MMR and allow the decoding and rendering process to run on the host virtual machine 34. On the other hand, if the aggregate activity of the multiple sessions 58 is high because multiple sessions are attempting to access the encoded media content, the host virtual machine 34 may be configured to enable MMR and load the MMR module 54.

[0049] The host virtual machine 34 may be configured to dynamically enable or disable MMR based on changes to the monitored parameters 90. Additionally, it should be appreciated that the example monitored parameters described above are merely exemplary, and that the host virtual machine 34 may dynamically determine whether to enable MMR based on other parameters, such as the size of the encoded media content, the source of the encoded media content, etc.

[0050] 6 illustrates an exemplary remote desktop client 92 that may be executed by the processor of each client computing device 12 to communicate with the host virtual machine 34. The remote desktop client 92 includes an MMR DVC plug-in 94 configured to interact with the MMR module 54 executed by the host virtual machine 34. When MMR is enabled by the host virtual machine 34, the MMR DVC plug-in 94 may be loaded into the remote desktop client 92. On the other hand, when MMR is not enabled, the remote desktop client 92 may be configured to use typical remoting techniques and processes.

[0051] The MMR DVC plug-in 94 includes a communication layer 96 and a media layer 98. The communication layer 96 is configured to handle communication between the MMR DVC plug-in 94 and the RDC remoting connector module 76 of the host virtual machine 34. The communication layer 96 may include an MMR virtual channel 100 configured to open a channel with the RDC remoting connector module 76 of the host virtual machine 34. An instance of the MMR virtual channel 100 may be created for each instance of the web browser application 36 opened for a session. A remote transceiver 102 is created when a new connection from the host virtual machine 34 is accepted, such as when an instance of the web browser application 36 on the host virtual machine 34 is opened and media content is accessed.

[0052] The remote transceiver 102 may be configured to send and receive messages 104 to and from the RDC remoting connector module 76 of the host virtual machine 34. As shown, the messages 104 are structured to include a main message handler 106, a source message handler 108, a renderer message handler 110, and a demux stream message handler 112. The main message handler 106 may be configured to include data for negotiating version information between the client and the host.

[0053] The messages 104 may include a source message handler 108 for each remoting bridge 82 of the RDC remoting connector module 76. Each source message handler 108 further includes a respective renderer message handler 110 for handling messages with the renderer process 64. The renderer message handlers 110 may include data regarding the playback state of the encoded media content being processed by the associated renderer process 64. For example, the renderer message handler 110 may communicate whether the media content should be paused, whether to seek to a target position, etc. The renderer message handler 110 may also be used to report the status of playback to the host virtual machine 34, such as the playback position, whether playback has finished, or any errors in playback.

[0054] Each renderer message handler 110 further includes a demux stream message handler 112 configured to contain data for processing demux-based operations, such as requesting media samples for each demuxed independent stream 72 of encoded media. That is, a client computing device 12 may request new media samples for a particular stream of encoded media content via the demux stream message handler 112. As described above, a client computing device 12 may request media content at an independent rate for each demuxed stream of the media content. For example, a client computing device 12 may request media samples for video media content at a higher rate than media samples for audio media content.

[0055] The media layer 98 includes a media renderer 114 that may send and receive messages 104 via a remote transceiver 102 of the communications layer 96. Using the messages 104, the media renderer 114 may request media samples for the stream of encoded media content 72, such as media samples of encoded video content, media samples of encoded audio content, and media samples of encoded text content. The media renderer 114 may then decode the received media samples and render the decoded media samples for presentation to a user via an output device of the client computing device 12, such as a display, speakers, etc. It should be appreciated that the media renderer 114 may use any appropriate rendering technique for the platform of the client computing device 12.

[0056] In the above-described manner, encoded media content may be redirected to client computing device 12 such that the media content is not decoded or rendered on host virtual machine 34. Rather, the encoded media content is streamed to client computing device 12, which is configured to decode and render the media content for display. Thus, by performing MMR and avoiding the decoding / rendering process on the host virtual machine, the processing workload on the hardware running the host virtual machine may be reduced.

[0057] In some embodiments, the methods and processes described herein may be coupled to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer application program or service, an application programming interface (API), a library, and / or other computer program product.

[0058] 7 schematically illustrates a non-limiting embodiment of a computing system 700 capable of defining one or more of the methods and processes described above. The computing system 700 is shown in simplified form. The computing system 700 may embody the client computing device 12 and the server device of the server system 14 described above and illustrated in FIGS. 1 and 2. The computing system 700 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices, as well as wearable computing devices such as smart watches and head-mounted augmented reality devices.

[0059] Computing system 700 includes a logical processor 702, a volatile memory 704, and a non-volatile storage device 706. Computing system 700 may optionally include a display subsystem 708, an input subsystem 710, a communication subsystem 712, and / or other components not shown in FIG.

[0060] Logical processor 702 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.

[0061] A logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, a logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processors of logical processor 702 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Individual components of a logical processor may optionally be distributed among two or more separate devices, which may be remotely located and / or configured for cooperative processing. Aspects of a logical processor may be virtualized and executed by remotely accessible networked computing devices arranged in a cloud computing configuration. In such cases, it will be understood that these virtualized aspects may execute on different physical logical processors of various different machines.

[0062] Non-volatile storage device 706 includes one or more physical devices configured to hold instructions executable by the logical processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of non-volatile storage device 706 may be transformed, for example, to hold different data.

[0063] The non-volatile storage device 706 may include removable and / or internal physical devices. The non-volatile storage device 706 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray® Disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH® memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.) or other mass storage device technologies. The non-volatile storage device 706 may include non-volatile, dynamic, static, read / write, read-only, sequential access, position-addressable, file-addressable, and / or content-addressable devices. It will be appreciated that the non-volatile storage device 706 is configured to retain instructions even when power to the non-volatile storage device 706 is interrupted.

[0064] Volatile memory 704 may include physical devices including random access memory. Volatile memory 704 is typically utilized by logical processor 702 to temporarily store information during the processing of software instructions. It will be appreciated that volatile memory 704 typically does not continue to store instructions when power to volatile memory 704 is interrupted.

[0065] Aspects of the logic processor 702, volatile memory 704, and non-volatile storage device 706 may be integrated together into one or more hardware logic components, which may include, for example, field programmable gate arrays (FPGAs), program specific and application specific integrated circuits (PASICs / ASICs), program specific and application specific standard products (PSSPs / ASSPs), systems on a chip (SOCs), and complex programmable logic devices (CPLDs).

[0066] The terms “module,” “program,” and “engine” may be used to describe aspects of computing system 700 that are implemented, typically in software, by a processor to perform a particular function using a portion of volatile memory, where the function includes variations that specifically configure the processor to perform that function. Thus, a module, program, or engine may be instantiated via logical processor 702 using a portion of volatile memory 704 to execute instructions held by non-volatile storage device 706. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” may include individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

[0067] When included, the display subsystem 708 may be used to present a visual representation of the data maintained by the non-volatile storage device 706. The visual representation may take the form of a graphical user interface (GUI). As the methods and processes described herein modify the data maintained by the non-volatile storage device, and thus transform the state of the non-volatile storage device, the state of the display subsystem 708 may likewise be transformed to visually represent the changes in the underlying data. The display subsystem 708 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the logic processor 702, the volatile memory 704, and / or the non-volatile storage device 706 in a shared enclosure, or such display devices may be peripheral display devices.

[0068] When included, the input subsystem 710 may include or interface with one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the transduction and / or processing of input actions may be handled on-board or off-board. Exemplary NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components and / or any other suitable sensors for assessing brain activity.

[0069] When included, the communications subsystem 712 may be configured to communicatively couple the various computing devices described herein with each other and with other devices. The communications subsystem 712 may include wired and / or wireless communication devices compatible with one or more different communications protocols. As a non-limiting example, the communications subsystem may be configured for communication over a wireless telephone network or over a wired or wireless local or wide area network, such as, for example, an HDMI over a Wi-Fi connection. In some embodiments, the communications subsystem may enable the computing system 700 to send and / or receive messages to and / or from other devices over a network, such as the Internet.

[0070] The following paragraphs provide additional support for the claims of the subject application. One aspect provides a computer system including at least one processor configured to execute a host virtual machine configured to host a session with at least one client computing device. The at least one processor executes a web browser application configured to access media content from a remote media source, and is further configured to execute a multimedia redirection module configured to receive encoded media content from the remote media source in a media container format and intercept the encoded media content from being processed by a decoding module of the web browser application. The multimedia redirection module is configured to redirect the encoded media content to the at least one client computing device. In this aspect, additionally or alternatively, the encoded media content may be dynamic streamed media content. In this aspect, additionally or alternatively, the encoded media content may be received in a media container format including one or more of a video stream, an audio stream, and a text stream, and the at least one processor may be configured to demultiplex the encoded media content from the media container format into independent streams of encoded media content. In this aspect, additionally or alternatively, the independent streams of encoded media content may include one or more of a video stream, an audio stream, and a text stream.Additionally or alternatively, in this aspect, the multimedia redirection module may be configured to buffer independent streams of encoded media content and, in response to receiving a request for media samples from the at least one client computing device, transmit media samples of the buffered independent streams of encoded media content to the at least one client computing device. Additionally or alternatively, in this aspect, the multimedia redirection module may be configured to transmit media samples for each independent stream of encoded media content at an independent rate based on the request for media samples received from the at least one client computing device. Additionally or alternatively, in this aspect, the at least one processor may be configured to execute the multimedia redirection module to determine whether to intercept the encoded media content or allow the encoded media content to be processed by a decoding module of a web browser application based on one or more monitored parameters. Additionally or alternatively, in this aspect, the one or more monitored parameters may be selected from the group consisting of network parameters between the computer system and the at least one client computing device, a current processing load of at least one processor of the computer system, a size of the encoded media content, and user settings of the at least one client computing device. In this aspect, additionally or alternatively, the host virtual machine executed by the at least one processor may include a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices.In this aspect, additionally or alternatively, the at least one processor may be configured to determine whether to execute the multimedia redirection module based on aggregate activity of multiple sessions hosted by the multi-session operating system.

[0071] Another aspect provides a method including, in a processor of a server device, executing a host virtual machine configured to host a session with at least one client computing device; executing a web browser application configured to access media content from a remote media source; receiving encoded media content in a media container format from the remote media source; and executing a multimedia redirection module configured to intercept the encoded media content from being processed by a decoding module of the web browser application. The multimedia redirection module is configured to redirect the encoded media content to the at least one client computing device. Additionally or alternatively, in this aspect, the encoded media content may be dynamic streaming media content. Additionally or alternatively, in this aspect, the encoded media content may be received in a media container format including one or more of a video stream, an audio stream, and a text stream, and the method may further include demultiplexing the encoded media content in the media container format into independent streams of encoded media content. Additionally or alternatively, in this aspect, the independent streams of encoded media content may include one or more of a video stream, an audio stream, and a text stream. In this aspect, additionally or alternatively, the method may further include buffering the independent stream of encoded media content, and transmitting media samples of the buffered independent stream of encoded media content to the at least one client computing device in response to receiving a request for the media samples from the at least one client computing device.In this aspect, additionally or alternatively, the method may further include transmitting media samples of each independent stream of encoded media content at an independent rate based on a request for the media samples received from the at least one client computing device. In this aspect, additionally or alternatively, the method may further include executing a multimedia redirection module to determine whether to intercept the encoded media content or enable the encoded media content for processing by a decoding module of a web browser application based on one or more monitored parameters. In this aspect, additionally or alternatively, the one or more monitored parameters may be selected from the group consisting of network parameters between the server device and the at least one client computing device, a current processing load of at least one processor of the server device, a size of the encoded media content, and user settings of the at least one client computing device. In this aspect, additionally or alternatively, the host virtual machine may include a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices.

[0072] Another aspect provides a server system including at least one processor configured to execute a host virtual machine implementing a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices. The at least one processor is further configured to execute an instance of a web browser application for one or more of the multiple sessions. The web browser application is configured to receive encoded media content from a remote media source. The processor is further configured to execute a multimedia redirection module configured to determine that aggregate activity of the multiple sessions for the host virtual machine exceeds a threshold level, enable multimedia redirection for one or more of the multiple sessions based on the determined aggregate activity, and intercept the encoded media content from being processed by a decoding module of the web browser application. The multimedia redirection module is configured to redirect the encoded media content to a respective client computing device of one or more of the multiple sessions.

[0073] It will be understood that the configurations and / or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be construed in a limiting sense, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Accordingly, various operations shown and / or described may be performed in the order shown and / or described, in other orders, in parallel, or omitted. Similarly, the order of the processes described above may also be changed.

[0074] The subject matter of the present disclosure includes all novel and non-obvious combinations and subcombinations of the various processes, systems, and configurations, as well as other features, functions, operations, and / or properties disclosed herein, and all equivalents thereof.

Claims

1. 1. A computer system comprising: at least one processor configured to execute a host virtual machine configured to host a session with at least one client computing device, the at least one processor comprising: running a web browser application configured to access encoded media content from a media source, wherein the encoded media content is formatted in a media container format that includes a video stream, an audio stream, a text stream, or a combination thereof; demultiplexing, by the web browser application, the encoded media content formatted in the media container format into independent streams; intercepting the independent stream before it is provided to a decoder of the web browser application, thereby preventing the independent stream from being decoded by the decoder of the web browser application; buffering, by the web browser application, a portion of the independent stream; transmitting, via the web browser application, media samples of the portion of the independent stream to the at least one client computing device; The computer system further comprises:

2. the encoded media content is dynamic stream media content; 10. The computer system of claim 1.

3. the media samples of the portion of the independent streams are transmitted to the at least one client computing device in response to receiving a request for the media samples from the at least one client computing device.

10. The computer system of claim 1.

4. the media samples for each of the independent streams are transmitted at an independent rate based on the request for the media samples received from the at least one client computing device.

4. The computer system of claim 3.

5. the at least one processor is configured to determine, based on one or more monitored parameters, whether to intercept the independent stream or allow the independent stream to be processed by the decoder of the web browser application.

10. The computer system of claim 1.

6. the one or more monitored parameters are selected from the group consisting of network parameters between the computer system and the at least one client computer device, a current processing load of the at least one processor of the computer system, a size of the encoded media content, or user settings of the at least one client computer device.

6. The computer system of claim 5.

7. the host virtual machine executed by the at least one processor includes a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices; 6. The computer system of claim 5.

8. the one or more monitored parameters include aggregate activity of the plurality of sessions hosted by the multi-session operating system; 8. The computer system of claim 7.

9. executing, by a processor, a host virtual machine configured to host a session with at least one client computing device; executing, by the processor, a web browser application configured to access media content from a media source; receiving encoded media content in a media container format from the media source; determining, by the processor, based on one or more monitored parameters, whether to redirect processing of the encoded media content from the web browser application to the at least one client computing device or allow the encoded media content to be processed by the web browser application; A method comprising:

10. the encoded media content is dynamic stream media content; 10. The method of claim 9.

11. the encoded media content is received in the media container format as a video stream, an audio stream or a text stream; The method further comprises demultiplexing the encoded media content in the media container format into independent streams.

10. The method of claim 9.

12. Redirecting the processing of the encoded media content from the web browser application to the at least one client computing device comprises: buffering a portion of the independent stream; transmitting media samples of the portion of the independent stream to the at least one client computing device; The method of claim 11 , comprising:

13. the media samples for each of the independent streams are transmitted at an independent rate based on requests for the media samples received from the at least one client computing device. The method of claim 12.

14. the one or more monitored parameters are selected from the group consisting of network parameters between a server device on which the processor is configured and the at least one client computer device, a current processing load of the processor, a size of the encoded media content, or user settings of the at least one client computer device.

10. The method of claim 9.

15. the host virtual machine includes a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices; 10. The method of claim 9.

16. the one or more monitored parameters include aggregate activity of the plurality of sessions hosted by the multi-session operating system; 16. The method of claim 15.

17. A server system, at least one processor configured to execute a host virtual machine implementing a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices, the at least one processor comprising: executing a web browser application for a session of the plurality of sessions, wherein the web browser application is configured to receive encoded media content from a media source for the session; determining that the aggregate activity of the plurality of sessions for the host virtual machine exceeds a threshold level; based on the aggregated activity, enabling multimedia redirection for the session by preventing the encoded media content from being processed by a decoder of the web browser application and redirecting the encoded media content to an associated client computing device of the plurality of client computing devices; The server system further comprises:

18. the encoded media content is formatted in a media container format that includes a video stream, an audio stream, or a text stream; the at least one processor is configured to demultiplex the encoded media content in the media container format into independent streams prior to the multimedia redirection.

18. The server system of claim 17.

19. The at least one processor buffering a portion of the independent stream; transmitting media samples of the portion of the independent stream to the associated client computing device; 20. The server system of claim 18, further configured to:

20. the media samples for each of the independent streams are transmitted at an independent rate based on requests for the media samples received from the associated client computing device.

20. The server system of claim 19.

Images