Multi-media redirection for media applications

Abstract

To provide a computer system and a method for a multi-media redirection for media applications.SOLUTION: A computer system includes a host virtual machine hosting a session with a client computer device. The host virtual machine on a server system including an operating system having a multi-media framework for rendering media content. The computer system executes a media application to access a media content from a media source, processes an encoded media content from the media application using a multi-media framework, and executes a multi-media redirect module to intercept the encoded media content. The multi-media redirect module redirects the encoded media content to the at least one client computer device.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] In a virtual environment, end users can use client devices to connect to virtual machines running on server hardware, which can provide the end users with the benefits of additional computing power, application compatibility, security and regulatory compliance, and overall lower costs for the end users. Typically, these end users may consume media through media player applications, which can potentially increase the workload imposed on the server hardware and increase associated costs. Summary of the Invention

[0002] A computer system is provided that may include at least one processor configured to execute a host virtual machine configured to host a session with at least one client computing device. The host virtual machine may include an operating system having a multimedia framework that renders media content. The at least one processor may be further configured to execute a media application configured to access media content from a media source, process encoded media content from the media application using the multimedia framework, and execute a multimedia redirection module configured to intercept the encoded media content from being processed by a decoding module of the multimedia framework. The multimedia redirection module may be configured to redirect the encoded media content to the at least one client computing device.

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Moreover, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of the disclosure. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a schematic diagram of an exemplary computer system that implements the multimedia redirection of the present description. [Figure 2] FIG. 2 is a schematic diagram of an exemplary server system of the computer system of FIG. 1. [Figure 3] FIG. 2 is a schematic diagram of a media application and multimedia framework of an operating system implementing a multimedia redirection module for the computer system of FIG. 1. [Figure 4] 2 is a flowchart of a method for performing multimedia redirection implemented by the computer system of FIG. 1. [Figure 5] FIG. 2 is a schematic diagram of the software architecture of a media application and multimedia framework that implements multimedia redirection for the computer system of FIG. 1. [Figure 6] FIG. 2 is a schematic diagram of the software architecture of a remote desktop client communicating with a multimedia redirection module implemented by the computer system of FIG. 1. [Figure 7] FIG. 2 is a schematic diagram of an exemplary computing environment in which the computer system of FIG. 1 may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0005] A cloud platform service may provide a virtualized desktop environment to end users, allowing the end users to utilize client computing devices to connect to virtual machines running on the cloud platform's server devices. Typically, the cloud platform's computing hardware and network capabilities are more powerful than the end users' client computing devices. Thus, by hosting these end users on a virtualized desktop environment, the cloud platform can provide these users with additional computing power and other potential benefits, such as improved application compatibility, security / regulatory compliance, and an overall reduction in associated computing costs.

[0006] In some examples, these cloud platform services may provide a virtual desktop infrastructure for enterprises with multiple end users and client devices. Users of these enterprises may consume media, such as video and / or audio media, that is stored locally on the cloud platform for the enterprise or stored by a remote media source and streamed to a virtualized desktop environment running on the cloud platform. Typically, media content provided by the media source is decoded on a virtual machine hosting the end user, and then display frames of the virtualized desktop presenting the media content are re-encoded and streamed to the user's client computing device over a remote desktop protocol. The client computing device can then decode these display frames and present them to the user via the client computing device's display.

[0007] The decoding and subsequent encoding processes performed on a virtual machine can consume significant computational resources of server hardware. In one particular example, the video rendering process for video streaming can potentially consume 20% to 40% of the central processing unit (CPU) resources of the server device running the virtual machine. These problems can be exacerbated in multi-session virtualized desktop environment deployments where multiple users share an operating system (OS) instance on a VM. For example, if multiple users attempt to stream video content through a shared host VM, all of the CPU resources for that VM can potentially be consumed by the associated decoding / encoding / rendering processes, thus potentially degrading the user experience for all users hosted by that VM.

[0008] To address the problems discussed above, Figure 1 illustrates an exemplary computer system 10 for a virtualized desktop environment that performs multi-media redirection to reduce the computational workload on the server hardware running the virtualized desktop environment, as discussed in more detail below. As shown in Figure 1, computer system 10 includes one or more client computing devices 12, a server system 14, and one or more media sources 16. End users of one or more client computing devices 12 can connect to virtual machines running on server system 14. Using applications running within the VM's host operating system, such as a media application that plays media content or a web browser application, the end users can interact with media stored by media source 16.

[0009] In one example, media source 16 may be local to or included in server system 14. In another example, media source 16 may be remote to both client computing device 12 and server system 14. For example, server system 14 may be configured to communicate with remote media sources over a communications network, such as a wide area network (WAN). As a particular example, an end user may interact with a media application executing within a host VM to trigger a media experience that views media stored on media source 16. The media application may take the form of, for example, a media streaming application that facilitates streaming and display of media content retrieved from media source 16. Typically, streaming video and / or audio is sent to server system 14 as encoded media 18 and processed by a host VM executing the media application that is playing the streaming video and / or audio.

[0010] As discussed above, a typical virtualized environment implementation performs a media rendering process that includes decoding encoded media 18, presenting media content via a virtualized desktop environment, encoding display / audio frames for the virtualized desktop environment, and then sending the encoded display / audio frames to client computing device 12 for presentation to the end user. To reduce the computational burden imposed on the hardware of server system 14 by the decoding / encoding process, server system 14 may implement multimedia redirection to pass encoded media 18 received from media source 16 to client computing device 12 without performing the encoding or decoding process on server system 14.

[0011] FIG. 2 illustrates an example of a server system 14. The server system 14 may include a hardware plane 20, a virtual machine plane 22, a hypervisor plane 24, and a network infrastructure 26, which collectively are configured to operate a cloud platform. The hardware plane 20 includes a collection of nodes 28 (each represented by the symbol "N" in FIG. 2 ), which may include processors, graphics processing units (GPUs), volatile memory, and other computer components configured to run host server instances. The host server instances executed by the nodes 28 of the hardware plane 20 are configured to communicate with one or more hypervisors of the hypervisor plane 24. The one or more hypervisors of the hypervisor plane 24 can create, handle, and monitor multiple virtual machines 30 of the virtual machine plane 22 (each represented by the symbol "VM" in FIG. 2 ). Via the hypervisor plane 24, each virtual machine 30 of the virtual machine plane 22 can be hosted and executed by hardware components of one or more nodes 28 of the hardware plane 20. In this way, multiple virtual machines 30 in the virtual machine plane 22 can share virtualized hardware resources managed by the hypervisor plane 24. Each virtual machine 30 provides a virtualized desktop environment within which software such as a web browser application, a media player application, and other types of software may run.

[0012] In one example, the computer system 14 corresponds to a data center environment that communicatively couples multiple nodes 28 via a standard network infrastructure. For example, the network infrastructure 20 may include typical network infrastructure such as server racks including top-of-rack (TOR) network switches. The computer system 14 may also include multiple node clusters, each with an associated TOR network switch. The network infrastructure 20 may further include higher-level switching infrastructure 32 (L1) and (L2) that connects the TOR network switches together. The higher-level switching infrastructure 32 may take the form of any suitable networking architecture and may be driven by any suitable routing protocol. In the illustrated example, the higher-level infrastructure 32 includes a collection of aggregation switches L1 and core switches L2. However, it will be understood that the higher-level switching infrastructure may include any suitable number of levels of switches.

[0013] The virtual machines 30 of the virtual machine plane 22 provide a virtual computing environment in which users of the server system 14 can access and run applications, such as web browsers, media applications, etc. As discussed in more detail below, these virtual machines 30 of the virtual machine plane 22 can be configured to perform multimedia redirection to reduce the computational workload imposed on the nodes of the hardware plane 20 caused by the decoding and encoding processes typically performed for streaming media.

[0014] Furthermore, it should be understood that server system 14 is not limited to the example data center / cloud platform shown in Figure 2. In other examples, server system 14 may take the form of one or more computing devices configured to execute a virtualized desktop environment. Using client computing devices 12, associated users can connect to and access software executing within the virtualized desktop environment executed by one or more computing devices, which may be configured to implement the multimedia redirection features and processes described herein.

[0015] Figure 3 illustrates an exemplary host virtual machine 34 configured to perform a multimedia redirection technique to address the problems discussed above. The host virtual machine 34 may take the form of one of the virtual machines 30 of the virtual machine plane 22 of the cloud platform described with reference to Figure 2. However, it should be understood that the host virtual machine 34 may also be implemented in a non-cloud platform form of the server system 14.

[0016] A host virtual machine 34 may be executed by at least one processor of the server system 14. The host virtual machine 34 may be configured to host sessions with at least one client computing device 12 and may host applications executing within the operating system of the host virtual machine 34. The host virtual machine 34 may include an operating system 60 having a multimedia framework 62 for rendering media content. The multimedia framework 62 may provide a runtime environment for media processing within the operating system 60 and includes application programming interfaces (APIs) that may be used by media applications 36 executing within the operating system 60 of the host virtual machine 34.

[0017] As shown, media application 36 may be configured to access media content from media source 16. Using a graphical user interface (GUI) of media application 36, an end user can trigger a media experience of the media content stored by media source 16. In one example, media source 16 may be local to server system 14 running host virtual machine 34. In another example, media source 16 may be remote to server system 14 and configured to transmit encoded media content 18 to server system 14 over a WAN.

[0018] After a media experience is triggered in the media application 36, the host virtual machine 34 may be configured to receive encoded media content 18 from the media source 16 for the triggered experience. The encoded media content 18 may be encoded using any suitable algorithm. In one example, the encoded media content 18 is received in a media container format 38 that includes one or more of a video stream 40, an audio stream 42, and a text stream 44. As a particular example, the media container format 38 may be an MP4 format, a WAV format, an AVI format, or another type of media container format 38.

[0019] The encoded media content 18 received by the host VM 34 may be received by a media application 36. The media application 36 may include program logic that makes API calls to a multimedia framework 62 to perform media processing on the encoded media content 18. The encoded media content 18 is then passed to the multimedia framework 62 and processed by the multimedia framework 62 of the operating system 60 of the host virtual machine 34. The multimedia framework 62 may be configured to provide a rendering pipeline and infrastructure for media rendering of the encoded media content 18. The multimedia framework 62 may load a multimedia framework source 46 based on the type of media container format 38 of the encoded media content 18. The multimedia framework source 46 may be configured to demultiplex (demux) the encoded media content 18 in the media container format 38 into separate video, audio, and text streams. Typically, as shown in FIG. 3 , the separate encoded audio stream 42 and encoded video stream 40 are processed by the audio and video rendering pipelines of the multimedia framework 62. For example, encoded audio stream 42 and encoded video stream 40 may be processed by media conversion modules of multimedia framework 62, such as decoder 48, which decompresses the audio and video streams, respectively. The decompressed audio and video streams may then be passed to streaming audio renderer 50 and streaming video renderer 52, respectively, to prepare the video and audio content for presentation. As discussed above, these decoding and rendering processes performed on host VM 34 consume hardware resources of the server device executing host VM 34, which can potentially degrade the user experience of other users hosted within the multi-session operating system of host virtual machine 34.

[0020] To address these issues, the host virtual machine 34 may be configured to implement a multimedia redirection module 54 configured to redirect encoded media content 18, including encoded audio streams 42 and encoded video streams 40, to the client computing device 12. For example, rather than building a complete multimedia framework topology using media conversion modules and renderers, the host virtual machine 34 may be configured to load the multimedia redirection module 54, which functions as a media sink, and render from the multimedia framework sources 46 directly to the multimedia redirection module 54. Figure 4 is a flowchart of a method 400 implemented by the server system 14 executing the host virtual machine 34 to perform multimedia redirection. The method 400 may be performed using the systems described above or other suitable hardware and software elements.

[0021] At 402, method 400 may include executing a host virtual machine configured to host a session with at least one client computing device. The host virtual machine 34 may be executed by at least one processor of a server system 14, which may take the form of, for example, a cloud platform data center as shown in FIG. 2. The host virtual machine 34 may include an operating system having a multimedia framework for rendering media content. In one example, the operating system may be a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices 12. The multiple sessions may be hosted within the same multi-session operating system instance. Each session may implement the techniques and processes for multimedia redirection described herein.

[0022] At 404, the method 400 may include executing a media application 36 configured to access media content from a media source. The media application 36 may execute within the host virtual machine 34. In an example of a multi-session operating system, each of multiple sessions may have an instance of an associated media application 36 executing in the multi-session operating system. The media source 16, in one example, may take the form of a third-party media service that streams media content such as video and / or audio. The media application 36 may be configured to access the media source 16 based on user input that triggers a media experience. In another example, the media source 16 may take the form of a storage device in the server system 14 that stores media content associated with the media application 36.

[0023] At 406, the method 400 may include processing the encoded media content from the media application using the multimedia framework. The encoded media content 18 may be received by the host virtual machine 34 over a WAN in an example remote media source. As shown in FIG. 3, the encoded media content 18 may include one or more of an encoded video stream 40, an encoded audio stream 52, and an encoded text stream 44. The encoded streams of the media content may be packaged in a media container format 38, such as, for example, an MP4 format, a WAV format, or the like.

[0024] At 408, method 400 may include demultiplexing the encoded media content in the media container format into independent streams of encoded media content. For example, multimedia framework 62 of operating system 60 may include a media conversion module that includes processes for unpacking media container format 38 and demultiplexing encoded media content 18 into independent streams, such as encoded video stream 40, encoded audio stream 42, and / or encoded text stream 44.

[0025] At 410, method 400 may include executing a multimedia redirection module configured to intercept encoded media content from being processed by a decoding module of the multimedia framework. The multimedia redirection module 54 may include a media sink, and the multimedia framework 62 of the operating system 60 may be configured to render directly from the source 46 to the media sink of the multimedia redirection module 54 rather than building a complete rendering topology. The multimedia redirection module 54 may receive the encoded content from the multimedia framework 62 and redirect the encoded content to the client computing device 12 associated with the session. In this manner, the encoded media content 18 is not decoded by the decoder 48 or processed by the renderers 50 and 52 on the host virtual machine 34.

[0026] The encoded media content 18 redirected to the client computing device 12 may be buffered and streamed to the client computing device 12 over a WAN. At 412, the method 400 may include buffering the independent streams of encoded media content. The independent streams of encoded media content, such as the encoded audio stream 42 and the encoded video stream 40, may be temporarily stored in buffers on the host virtual machine 34.

[0027] At 414, method 400 may include transmitting media samples of the buffered independent streams of encoded media content to at least one client computing device in response to receiving a request for the media samples from the at least one client computing device. Client computing device 12 may request new media samples independently for each stream of encoded media content 18. For example, client computing device 12 may request media samples for encoded video stream 40 independently of requesting media samples for encoded audio stream 40. In this manner, client computing device 12 may request media samples for each independent stream of encoded media content (e.g., encoded audio stream, encoded video stream, etc.) at an independent rate, and host virtual machine 34 may transmit such media samples for each independent stream of encoded media content at the requested rate. Typically, video content is larger in size than audio content and may therefore require higher-rate video samples to be sent to client computing device 12.

[0028] 5 illustrates an exemplary server-side software architecture for implementing the multimedia redirection module 54 and the steps described in method 400. In the illustrated example, at least one processor 56 of a server device included in server system 14 may be configured to execute a host virtual machine 34 configured to host a session 58 with at least one client computing device 12. In the illustrated example, the host virtual machine 34 implements a multi-session operating system 60 configured to simultaneously host multiple sessions 58 for multiple client computing devices 12. Each session 58 may be associated with a different client computing device 12 and may run a separate instance of an application program on the host virtual machine 34. For example, each session 58 may run a separate instance of a media application 36 configured to access media content from a media source 16.

[0029] Each session 58 executing within the multi-session operating system 60 may be further configured to receive encoded media content 18 from a media source 16 in a media container format 38, as described above with reference to FIGS. 3 and 4 . In one example, the encoded media content 18 may take the form of media files stored on storage associated with the host virtual machine 34. In this example, the media source 16 may be an associated storage device local to the server system 14. However, it should be understood that the encoded media content 18 may take other suitable forms, such as dynamic streaming media content, HTTP progressive streaming content, or other forms of streaming media content received from a remote media source. Furthermore, the encoded media content 18 may be received in a media container format that includes one or more of a video stream, an audio stream, and a text stream. The video stream, the audio stream, and the text stream may be muxed in the media container format. It should be understood that the media container format may take the form of an MP4 format, a WAV format, or any other suitable type of media container format.

[0030] Execution of the media application 36 may spin up multiple related processes executing within the multi-session operating system 60. As a particular example, a media application process 64 for a session 58 with a client computing device 58 and a multimedia framework process 66 may be executed. In one example, the functionality of the multimedia framework process 66 described herein may be included in a process of the multi-session operating system 60. In another example, the multimedia framework process 66 may be executed in a separate processing thread.

[0031] Application logic 68 for media application 36 may execute within media application process 64. Application logic 68 may include program logic that retrieves encoded media content 18 based on a user of media application 36 triggering a media viewing experience. Application logic 68 may further include program logic that makes API calls 70 to multimedia framework process 66 to render encoded media content 18. The encoded media content 18 retrieved by media application process 64 and / or a unique resource identifier for the encoded media content 18 may also be passed to multimedia framework process 66.

[0032] A multimedia framework process 66 executed by processor 56 may be configured to process encoded media content 18 received from media application process 64. In one example, multimedia process 66 may be configured to identify the format type of encoded media content 18, such as the type of media container format 38, and then select a corresponding multimedia framework source 72. Among other functions, multimedia framework source 72 may be configured to unpackage encoded media content 18 and demultiplex the encoded media content 18 into independent streams of encoded media content 74. The demultiplexed independent streams of encoded media content 74 may include, for example, encoded video streams, encoded audio streams, and / or encoded text streams.

[0033] If multimedia redirection (MMR) is not currently enabled for the received encoded media content 18, the multimedia framework process 66 may be configured to load rendering pipelines for the demultiplexed streams of video, audio, and / or text. For example, the multimedia framework process 66 may be configured to load multiple media transformation modules that process the encoded media content. These media transformation modules may include, for example, the decoder 48 shown in FIG. 3 and other modules that perform other types of data transformation. The multimedia framework process 66 may also load a streaming audio renderer 50, a streaming video renderer 52, etc. The multimedia framework process 66 may then use the loaded rendering pipelines to process the demultiplexed independent streams 74 of the encoded media content. In some examples, the rendering process may include other lower-level processes of the multi-session operating system 60. Display frames of the rendered video content and audio frames of the rendered audio content may then be encoded and streamed to the client computing device 12 over a WAN.

[0034] On the other hand, when MMR is enabled, the multimedia framework process 66 may be configured to intercept the demultiplexed stream of media content 74 and load the multimedia redirection module 54 configured to perform the functions and processes described herein. In this manner, the demultiplexed stream of media content 74 is not decoded and rendered on the host virtual machine 34, providing the potential advantage of reducing the computational workload imposed on the hardware running the host virtual machine 34.

[0035] In one example, the multimedia redirection module 54 may take the form of a media sink, and the multimedia framework source 72 may be configured to render directly to the multimedia redirection module 54 when MMR is enabled. Rather than processing the demultiplexed streams of encoded media content 74 using a media conversion module as described above, the multimedia redirection module 54 may be configured to redirect the demultiplexed streams of encoded media content 74 to the corresponding client computing device 12 for that session 58. To redirect the media content, the multimedia redirection module 54 may be configured to handle the transmission of the demultiplexed, independent streams of encoded media data to a dynamic virtual channel (DVC) plug-in implemented on the client computing device 12. The multimedia redirection module 54 may also handle the communication of serialized messages with the DVC plug-in on the client computing device 12. In one example, commands associated with playing and rendering the encoded media content 18 may be serialized using a protocol such as, for example, the extended GOOGLE® PROTOCOL BUFFERS, a language- and platform-neutral mechanism for serializing and deserializing structured data. However, it should be understood that other serialization protocols may be implemented. Multimedia redirection module 54 may be configured to send and receive these commands to and from client computing device 12 via serialized messages.

[0036] Additionally, when redirecting encoded media content 18 to client computing devices 12, MMR module 54 may be configured to buffer independent streams 74 of encoded media content and transmit media samples of the buffered independent streams 74 of encoded media content to at least one client computing device 12 in response to receiving a request for the media samples from at least one client computing device 12. For example, client computing device 12 may send a serialized message including a request 76 for media samples for at least one of the streams of encoded media content to MMR module 54. MMR module 54 may then transmit the requested buffered encoded media samples 78 to client computing device 12 through a dynamic virtual channel. In one example, client computing device 12 may send a separate request 76 for each independent stream 74 of encoded media content. For example, client computing device 12 may request media samples of an encoded video content stream at a higher rate than media samples of encoded audio content. Thus, the MMR module 54 can be configured to transmit media samples 78 for each independent stream 74 of encoded media content at an independent rate based on requests 76 for media samples received from at least one client computing device 12.

[0037] As discussed above, MMR module 54, which may take the form of a media sink of multimedia framework 62, is loaded and executed when MMR is enabled for encoded media content 18. In one example, at least one processor 56 may be configured to determine whether to enable MMR and whether to execute MMR module 54 to intercept encoded media content 18 or allow encoded media content 18 to be processed by decoding module 48 and other media conversion modules of multimedia framework 62 based on one or more monitored parameters 80. In one example, host virtual machine 34 may be configured to monitor network parameters between server system 14 and at least one client computing device 12 and determine whether to enable MMR based on the monitored network parameters.

[0038] As another example, the host virtual machine 34 may be configured to monitor the current processing load of at least one processor 56 executing the host virtual machine 34 and determine whether to enable MMR based on the current processing load. As a particular example, if the current processing load on the at least one processor 56 exceeds a threshold level, the host virtual machine 34 may be configured to enable MMR and load the MMR module 54 and its associated media stack to intercept the encoded media content 18 from being processed by the rendering pipeline, which may typically include, for example, multiple media transformation modules and a renderer. On the other hand, if the current processing load is below the threshold level and the monitored network parameters indicate low network bandwidth, the host virtual machine may be configured to disable MMR and decode / render the encoded media content. The rendered frames may then be encoded at a compression level appropriate for the network bandwidth between the server system and the client computing device.

[0039] As another example, the host virtual machine 34 may be configured to determine whether to enable MMR and load the MMR module 54 based on a user setting of at least one client computing device 12. For example, an administrator may control the user setting to enable or disable MMR for a client computing device. The user setting may be transmitted to the host virtual machine 34 and stored for that client computing device. In one particular example, MMR may be enabled or disabled independently for different media sources 16 via the user setting.

[0040] As discussed above, the host virtual machine 34 may implement a multi-session operating system 60 that hosts sessions 58 with multiple client computing devices 12. Activity with each client computing device 12 may increase the processing workload imposed on at least one processor 56 executing the host virtual machine 34. Accordingly, in one example, the host virtual machine 34 may be configured to determine the aggregate activity of the multiple sessions 58 hosted by the multi-session operating system 60 and determine whether to execute the MMR module 54 based on the aggregate activity. As a particular example, if the aggregate activity of the multiple sessions 58 is low because only one session is currently accessing the encoded media content for streaming, the host virtual machine 34 may be configured not to enable MMR and allow the decoding and rendering process to run on the host virtual machine 34. On the other hand, if the aggregate activity of the multiple sessions 58 is high because multiple sessions are attempting to access terminated media content, the host virtual machine 34 may be configured to enable MMR and load the MMR module 54.

[0041] Host virtual machine 34 may be configured to dynamically enable or disable MMR based on changes to monitoring parameters 80. Furthermore, it should be understood that the example monitoring parameters discussed above are merely exemplary, and that host virtual machine 34 may dynamically determine whether to enable MMR based on other parameters, such as the size of the encoded media content, the source of the encoded media content, etc.

[0042] 6 illustrates an exemplary remote desktop client 82 that may be executed by the processor of each client computing device 12 to communicate with the host virtual machine 34. The remote desktop client 82 includes an MMR DVC plug-in 84 configured to interact with the MMR module 54 executed by the host virtual machine 34. When MMR is enabled by the host virtual machine 34, the MMR DVC plug-in 84 may be loaded into the remote desktop client 82. On the other hand, when MMR is not enabled, the remote desktop client 82 may be configured to use typical remoting techniques and processes.

[0043] The MMR DVC plug-in 84 includes a communication layer 86 and a media layer 88. The communication layer 86 is configured to handle communication between the MMR DVC plug-in 84 and the MMR module 54 of the host virtual machine 34. The communication layer 86 may include an MMR virtual channel 90 configured to open a channel with the MMR module 54 of the host virtual machine 34. An instance of the MMR virtual channel 90 may be created for each instance of the media application 36 opened for a session. The remote transceiver 92 may be created when a new connection from the host virtual machine 34 is accepted, for example, when an instance of the media application 36 on the host virtual machine 34 is opened and media content is accessed.

[0044] The remote transceiver 92 may be configured to send and receive messages 94 to and from the MMR module 54 of the host virtual machine 34. As shown, the messages 94 are structured to include a main message handle 96, a source message handle 98, a renderer message handler 100, and a demuxer stream message handler 102. The main message handler 96 may be configured to include data for negotiating versioning information between the client and the host.

[0045] The messages 94 may include a source message handler 98 for each multimedia framework source 72. Each source message handler 98 further includes a respective renderer message handler 100 for handling messages that may include data regarding the playback state of the encoded media content 18. For example, the renderer message handler 100 may communicate whether the media content should be paused, whether to seek to a target position, etc. Additionally, the renderer message handler 100 may be used to report the status of playback to the host virtual machine 34, such as the playback position, whether playback has finished, or any errors in playback.

[0046] Each renderer message handler 100 further includes a demultiplexer stream message handler 102 configured to include data for handling demultiplexing-based operations, such as requesting media samples for each demultiplexed independent stream 74 of coded media. That is, a client computing device 12 can request new media samples for a particular stream of coded media content via the demultiplexer stream message handler 102. As discussed above, a client computing device 12 can request media content at an independent rate for each demultiplexed stream of the media content. For example, a client computing device 12 can request media samples of video media content at a higher rate than media samples of audio media content.

[0047] The media layer 88 includes a media renderer 104 that can send and receive messages 104 via the remote transceiver 92 of the communications layer 86. Using the messages 94, the media renderer 104 can request media samples of the stream of encoded media content 74, such as media samples of encoded video content, media samples of encoded audio content, and media samples of encoded text content. The media renderer 104 can then decode the received media samples and render the decoded media samples via an output device of the client computing device 12, such as a display, speakers, or the like, for presentation to a user. It should be understood that the media renderer 104 may use any appropriate rendering technique for the platform of the client computing device 12.

[0048] In the methods discussed above, encoded media content may be redirected to client computing device 12 such that the media content is not decoded or rendered on host virtual machine 34. Rather, the encoded media content is redirected and streamed to client computing device 12, which is configured to decode and render the media content for display. Thus, by performing MMR and avoiding the decoding / rendering process on the host virtual machine, the processing workload on the hardware running the host virtual machine may be reduced.

[0049] In some embodiments, the methods and processes described herein may be coupled to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer application programs or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0050] 7 schematically illustrates one non-limiting embodiment of a computing system 700 capable of implementing one or more of the methods and processes described above. The computing system 700 is shown in simplified form. The computing system 700 may embody the client computing device 12 and the server device of the server system 14 shown in FIGS. 1 and 2, described above. The computing system 700 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices, as well as wearable computing devices such as smart watches and head-mounted augmented reality devices.

[0051] Computing system 700 includes a logic processor 702, a volatile memory 704, and a non-volatile storage device 706. Computing system 700 may optionally include a display subsystem 708, an input subsystem 710, a communication subsystem 712, and / or other components not shown in FIG.

[0052] Logical processor 702 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.

[0053] A logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, a logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processors of logical processor 702 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Individual components of a logical processor may optionally be distributed among two or more separate devices, which may be remotely located and / or configured for cooperative processing. Aspects of a logical processor may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. In such cases, it will be understood that these virtualized aspects may execute on different physical logical processors of various different machines.

[0054] Non-volatile storage 706 includes one or more physical devices configured to hold instructions executable by the logical processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of non-volatile storage 706 may be transformed, for example, to hold different data.

[0055] The non-volatile storage 706 may include removable and / or internal physical devices. The non-volatile storage 706 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray® disk, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH® memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), or other mass storage technologies. The non-volatile storage 706 may include non-volatile, dynamic, static, read / write, read-only, sequential access, position-addressable, file-addressable, and / or content-addressable devices. It will be understood that the non-volatile storage 706 is configured to retain instructions even when power is interrupted to the non-volatile storage 706.

[0056] Volatile memory 704 may include physical devices including random access memory. Volatile memory 704 is typically utilized by logical processor 702 to temporarily store information during the processing of software instructions. It will be understood that volatile memory 704 typically does not continue to store instructions when power is interrupted to volatile memory 704.

[0057] Aspects of the logic processor 702, volatile memory 704, and non-volatile storage 706 may be integrated together into one or more hardware logic components, including, for example, field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASICs / ASICs), program- and application-specific standard products (PSSPs / ASSPs), systems-on-chips (SOCs), and complex programmable logic devices (CPLDs).

[0058] The terms “module,” “program,” and “engine” may be used to describe aspects of computing system 700 that are typically implemented in software by a processor to perform a particular function using a portion of volatile memory, the function including transformation processes that specifically configure the processor to perform that function. Thus, a module, program, or engine may be instantiated via logical processor 702 using a portion of volatile memory 704 to execute instructions held by non-volatile storage 706. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” may include individual or groupings of executable files, data files, libraries, drivers, scripts, database records, etc.

[0059] When included, the display subsystem 708 may be used to present a visual representation of the data maintained by the non-volatile storage device 706. The visual representation may take the form of a graphical user interface (GUI). When the methods and processes described herein modify the data maintained by the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 708 may likewise be transformed to visually represent the changes in the underlying data. The display subsystem 708 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the logic processor 702, the volatile memory 704, and / or the non-volatile storage device 706 in a shared enclosure, or such display devices may be peripheral display devices.

[0060] When included, the input subsystem 710 may include or interface with one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the introduction and / or processing of input actions may be handled on-board or off-board. Exemplary NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity; and / or any other suitable sensors.

[0061] When included, the communications subsystem 712 may be configured to communicatively couple the various computing devices described herein with each other and with other devices. The communications subsystem 712 may include wired and / or wireless communication devices compatible with one or more different communications protocols. As a non-limiting example, the communications subsystem may be configured for communication over a wireless telephone network, or a wired or wireless local or wide area network, e.g., HDMI over a Wi-Fi connection. In some embodiments, the communications subsystem may enable the computing system 700 to send and / or receive messages to and / or from other devices over a network, such as the Internet.

[0062] The following paragraphs provide further support for the claims of the subject application. One aspect provides a computer system including at least one processor configured to execute a host virtual machine configured to host a session with at least one client computing device. The host virtual machine includes an operating system having a multimedia framework that renders media content. The at least one processor is further configured to execute a media application configured to access media content from a media source, process encoded media content from the media application using the multimedia framework, and execute a multimedia redirection module configured to intercept the encoded media content from being processed by a decoding module of the multimedia framework. The multimedia redirection module is configured to redirect the encoded media content to the at least one client computing device. In this aspect, additionally or alternatively, the encoded media content may be dynamic streaming media content. In this aspect, additionally or alternatively, the encoded media content may be received in a media container format including one or more of a video stream, an audio stream, and a text stream, and the at least one processor may be configured to demultiplex the encoded media content in the media container format into independent streams of encoded media content. In this aspect, the separate streams of encoded media content may additionally or alternatively include one or more of a video stream, an audio stream, and a text stream.Additionally or alternatively, in this aspect, the multimedia redirection module may be configured to buffer independent streams of the encoded media content and transmit media samples of the buffered independent streams of the encoded media content to the at least one client computing device in response to receiving a request for the media samples from the at least one client computing device. Additionally or alternatively, in this aspect, the multimedia redirection module may be configured to transmit media samples for each independent stream of the encoded media content at an independent rate based on the request for the media samples received from the at least one client computing device. Additionally or alternatively, in this aspect, the at least one processor may be configured to determine whether to execute the multimedia redirection module to intercept the encoded media content or allow the encoded media content to be processed by a decoding module of the multimedia framework based on one or more monitoring parameters. Additionally or alternatively, in this aspect, the one or more monitoring parameters may be selected from the group including network parameters between the computer system and the at least one client computing device, a current processing load of the at least one processor of the computer system, a size of the encoded media content, and user settings of the at least one client computing device. Additionally or alternatively, in this aspect, the operating system may be a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices. In this aspect, additionally or alternatively, the at least one processor may be configured to determine whether to execute the multimedia redirection module based on the aggregate activity of multiple sessions hosted by the multi-session operating system.

[0063] Another aspect provides a method including executing, on a processor of a server device, a host virtual machine configured to host a session with at least one client computing device. The host virtual machine includes an operating system having a multimedia framework that renders media content. The method further includes executing a media application configured to access media content from a media source, processing the encoded media content from the media application using the multimedia framework, and executing a multimedia redirection module configured to intercept the encoded media content from being processed by a decoding module of the multimedia framework. The multimedia redirection module is configured to redirect the encoded media content to the at least one client computing device. In this aspect, additionally or alternatively, the encoded media content may be dynamic streaming media content. In this aspect, additionally or alternatively, the encoded media content may be received in a media container format including one or more of a video stream, an audio stream, and a text stream, and the method may further include demultiplexing the encoded media content in the media container format into independent streams of encoded media content. In this aspect, additionally or alternatively, the independent streams of encoded media content may include one or more of a video stream, an audio stream, and a text stream. In this aspect, additionally or alternatively, the method may further include buffering the independent stream of encoded media content and transmitting media samples of the buffered independent stream of encoded media content to at least one client computing device in response to receiving a request for the media samples from the at least one client computing device.In this aspect, the method may further include transmitting media samples for each of the independent streams of encoded media content at an independent rate based on a request for the media samples received from the at least one client computing device. In this aspect, the method may further include determining, based on one or more monitoring parameters, whether to execute a multimedia redirection module to intercept the encoded media content or allow the encoded media content to be processed by a decoding module of the multimedia framework. In this aspect, the one or more monitoring parameters may be selected from the group including network parameters between the server device and the at least one client computing device, a current processing load of at least one processor of the server device, a size of the encoded media content, and user settings of the at least one client computing device. In this aspect, the operating system may further or alternatively be a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices.

[0064] Another aspect provides a server system including at least one processor configured to execute a host virtual machine implementing a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices, the multi-session operating system having a multimedia framework for rendering media content. The at least one processor is further configured to execute an instance of a media application for one or more of the multiple sessions, the media application being configured to access media content from a media source. The processor is further configured to execute a multimedia redirection module configured to determine that aggregate activity of the multiple sessions of the host virtual machine exceeds a threshold level, enable multimedia redirection for one or more of the multiple sessions based on the determined aggregate activity, process encoded media content from the instance of the media application using the multimedia framework, and intercept the encoded media content from being processed by a decoding module of the multimedia framework. The multimedia redirection module is configured to redirect the encoded media content to each client computing device of one or more of the multiple sessions.

[0065] It will be understood that the configurations and / or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be construed in a limiting sense, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various operations shown and / or described may be performed in the sequence shown and / or described, in other sequences, in parallel, or omitted. Similarly, the order of the processing described above may also be changed.

[0066] The subject matter of the present disclosure includes all novel and non-obvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, operations, and / or properties disclosed herein, and any and all equivalents thereof.

Claims

1. 1. A computer system comprising: at least one processor configured to execute a host virtual machine configured to host a session with at least one client computing device, the host virtual machine including an operating system having a multimedia framework for decoding and rendering media content; Including, The at least one processor further comprises: accessing encoded media content from a media source, the encoded media content being formatted into a media container format that includes two or more of a video stream, an audio stream, a text stream, or a combination thereof; demultiplexing, by the multimedia framework, the encoded media content formatted in the media container format into independent streams; intercepting the independent stream before it is provided to a decoder of the multimedia framework, thereby preventing the independent stream from being decoded by the decoder; buffering, by the multimedia framework, portions of the independent streams; transmitting, by the multimedia framework, media samples of the portion of the independent stream to the at least one client computing device; A computer system configured to:

2. The computer system of claim 1 , wherein the encoded media content is dynamic streaming media content.

3. 10. The computer system of claim 1, further comprising: a computer system configured to transmit the media samples of the portion of the independent stream to the at least one client computing device in response to receiving a request for the media samples from the at least one client computing device.

4. 4. The computer system of claim 3, wherein the media samples for each of the independent streams are transmitted at an independent rate based on a request for the media samples received from the at least one client computing device.

5. 2. The computer system of claim 1, wherein the at least one processor is configured to determine whether to intercept the independent stream or allow the independent stream to be decoded by the decoder of the multimedia framework based on one or more monitoring parameters.

6. 6. The computer system of claim 5, wherein the one or more monitored parameters are selected from the group including network parameters between the computer system and the at least one client computer device, a current processing load of the at least one processor of the computer system, a size of the encoded media content, or user settings of the at least one client computer device.

7. 6. The computer system of claim 5, wherein the operating system is a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices.

8. The computer system of claim 7 , wherein the one or more monitored parameters include aggregate activity of the plurality of sessions hosted by the multi-session operating system.

9. a processor executing a host virtual machine configured to host a session with at least one client computing device, the host virtual machine including an operating system having a multimedia framework for decoding and rendering media content; executing, by the processor, a media application configured to access encoded media content from a media source; determining, based on one or more monitored parameters, whether to redirect processing of the encoded media content from the multimedia framework to the at least one client computing device or allow the encoded media content to be processed by a decoder of the multimedia framework; A method comprising:

10. The method of claim 9 , wherein the encoded media content is dynamic streaming media content.

11. 10. The method of claim 9, wherein the encoded media content is received in a media container format including a video stream, an audio stream, a text stream, or a combination thereof, the method further comprising demultiplexing the encoded media content in the media container format into independent streams.

12. Redirecting the processing of the encoded media content from the multimedia framework to the at least one client computing device comprises: buffering portions of said independent streams; transmitting media samples of the portion of the independent stream to the at least one client computing device; The method of claim 11 further comprising:

13. 13. The method of claim 12, further comprising transmitting media samples for each of the independent streams of the encoded media content at an independent rate based on a request for the media samples received from the at least one client computing device.

14. 10. The method of claim 9, wherein the one or more monitored parameters are selected from a group including network parameters between a server device on which the processor is configured and the at least one client computer device, a current processing load of the processor, a size of the encoded media content, or user settings of the at least one client computer device.

15. 10. The method of claim 9, wherein the operating system is a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices.

16. The method of claim 15 , wherein the one or more monitored parameters include aggregate activity of the multiple sessions hosted by the multi-session operating system.

17. A server system, a host virtual machine configured to execute a host virtual machine implementing a multi-session operating system configured to simultaneously host multiple sessions for multiple client computing devices, the multi-session operating system including at least one processor having a multimedia framework for decoding and rendering media content; The at least one processor further comprises: configured to execute a media application on a session of the plurality of sessions, the media application configured to access encoded media content from a media source; determining that the aggregate activity of the plurality of sessions of the host virtual machine exceeds a threshold level; enabling multimedia redirection for the session based on the aggregate activity by preventing the encoded media content from being processed by a decoder of the multimedia framework and redirecting the encoded media content to an associated one of the plurality of client computing devices; It is configured as follows: Server system.

18. 20. The server system of claim 17, wherein the encoded media content is formatted into a media container format including a video stream, an audio stream, a text stream, or a combination thereof, and the at least one processor is configured to demultiplex the encoded media content in the media container format into independent streams.

19. The at least one processor further comprises: buffering portions of said independent streams; 20. The server system of claim 18, configured to transmit media samples of the portions of the independent streams to the associated client computing devices.

20. 20. The server system of claim 19, further comprising: the media samples for each of the independent streams being transmitted at an independent rate based on requests for the media samples received from the associated client computing devices.

Images