Multimedia Redirect for Media Applications

Abstract

A computer system is provided that includes at least one processor configured to execute a host virtual machine configured to host a session with at least one client computing device. The host virtual machine includes an operating system having a multimedia framework that renders media content. The at least one processor is further configured to execute a media application configured to access media content from a media source, process encoded media content from the media application using the multimedia framework, and execute a multimedia redirection module configured to intercept the encoded media content from being processed by a decoding module of the multimedia framework. The multimedia redirection module is configured to redirect the encoded media content to the at least one client computing device.

Description

Background Art

[0001] In a virtual environment, an end user can use a client device to connect to a virtual machine operating on server hardware, which can provide the end user with the advantages of additional computing power, application compatibility, security and regulatory compliance, and overall cost reduction. Typically, these end users may consume media via a media player application, which can potentially increase the workload imposed on the server hardware and increase the associated costs.

Summary of the Invention

[0002] A computer system may include at least one processor configured to execute a host virtual machine configured to host a session with at least one client computer device. The host virtual machine can include an operating system having a multimedia framework for rendering media content. The at least one processor may further execute a media application configured to access media content from a media source, process the encoded media content from the media application using the multimedia framework, and execute a multimedia redirect module configured to intercept the encoded media content from being processed by the decoding module of the multimedia framework. The multimedia redirect module may be configured to redirect the encoded media content to at least one client computer device.

[0003] This summary is provided to introduce selected ones of the concepts further described in the detailed description below in a simplified form. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Further, the claimed subject matter is not limited to implementations that solve any or all disadvantages described in any part of this disclosure. **Brief Description of the Drawings**

[0004]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

[0005] Cloud platform services may provide an end user with a virtualized desktop environment, and the end user can use a client computer device to connect to a virtual machine operating on a server device of the cloud platform. Typically, the computer hardware and network capabilities of the cloud platform are more powerful than the end user's client computer device. Therefore, by hosting these end users on a virtualized desktop environment, the cloud platform can provide these users with additional computing power and other potential benefits such as improved application compatibility, compliance with security / regulations, and an overall reduction in related computing costs.

[0006] In some examples, these cloud platform services may provide a virtual desktop infrastructure to an enterprise having a plurality of end users and client devices. The users of these enterprises may consume media such as video and / or audio media, and this media is stored locally on the cloud platform for the enterprise or is stored by a remote media source and streamed to the virtualized desktop environment running on the cloud platform. Typically, the media content provided by the media source is decoded on the virtual machine hosting the end user, and then the display frame of the virtualized desktop presenting the media content is re-encoded and streamed to the user's client computer device through a remote desktop protocol. The client computer device can then decode these display frames and present them to the user via the display of the client computer device.

[0007] The decoding and subsequent encoding processes executed on a virtual machine may consume a large amount of computing resources of the server hardware. In one particular example, the video rendering process for video streaming may potentially consume 20% to 40% of the central processing unit (CPU) resources of the server device running the virtual machine. These problems may further deteriorate in a multi-session virtualized desktop environment deployment where multiple users share operating system (OS) instances on a VM. For example, when multiple users attempt to stream video content via a shared host VM, all of the CPU resources for that VM may potentially be consumed by the associated decoding / encoding / rendering processes, and thus potentially degrade the user experience of all users hosted by that VM.

[0008] To address the problems discussed above, FIG. 1 shows an exemplary computer system 10 for a virtualized desktop environment that performs multi-media redirection to reduce the computational workload on the server hardware running the virtualized desktop environment, as discussed in more detail below. As shown in FIG. 1, computer system 10 includes one or more client computer devices 12, a server system 14, and one or more media sources 16. End users of the one or more client computer devices 12 can connect to a virtual machine executed on the server system 14. Using an application executed within the host operating system of the VM, such as a media application that plays media content or a web browser application, the end user can interact with the media stored by the media source 16.

[0009] In one example, media source 16 may be local to server system 14 or may be included within server system 14. In another example, media source 16 may be remote to both client computer device 12 and server system 14. For example, server system 14 may be configured to communicate with a remote media source through a communication network such as a wide area network (WAN). As a specific example, an end user can interact with a media application running within a host VM to trigger a media experience of viewing media stored in media source 16. The media application may take the form of, for example, a media streaming application that facilitates streaming and display of media content retrieved from media source 16. Typically, streaming video and / or audio is sent to server system 14 as encoded media 18 and is processed by a host VM executing a media application that is playing the streaming video and / or audio.

[0010] As discussed above, a typical virtualized environment implementation performs a media rendering process that includes decoding encoded media 18, presenting media content through a virtualized desktop environment, encoding display / audio frames for the virtualized desktop environment, and then sending the encoded display / audio frames to client computer device 12 for presentation to an end user. To reduce the computational burden imposed on the hardware of server system 14 by the decoding / encoding process, server system 14 can implement multimedia redirection to pass encoded media 18 received from media source 16 to client computer device 12 without performing an encoding or decoding process on server system 14.

[0011] FIG. 2 shows an example of the server system 14. The server system 14 can include a hardware plane 20, a virtual machine plane 22, a hypervisor plane 24, and a network infrastructure 26, which are collectively configured to operate a cloud platform. The hardware plane 20 includes a set of nodes 28 (each denoted by the symbol "N" in FIG. 2) that can include a processor, a graphics processing unit (GPU), volatile memory, and other computer components configured to execute a host server instance. The host server instance executed by the nodes 28 of the hardware plane 20 is configured to communicate with one or more hypervisors of the hypervisor plane 24. One or more hypervisors of the hypervisor plane 24 can create, handle, and monitor a plurality of virtual machines 30 (each denoted by the symbol "VM" in FIG. 2) of the virtual machine plane 22. Through the hypervisor plane 24, each virtual machine 30 of the virtual machine plane 22 can be hosted and executed by the hardware components of one or more nodes 28 of the hardware plane 20. In this way, the plurality of virtual machines 30 of the virtual machine plane 22 can share virtualized hardware resources managed by the hypervisor plane 24. Each virtual machine 30 provides a virtualized desktop environment in which software such as a web browser application, a media player application, and other types of software can be executed.

[0012] In one example, computer system 14 corresponds to a data center environment that communicatively couples a plurality of nodes 28 via a standard network infrastructure. For example, network infrastructure 20 may include a typical network infrastructure such as a server rack that includes top-of-rack (TOR) network switches. Computer system 14 may include a plurality of node clusters each having an associated TOR network switch. Network infrastructure 20 can further include higher-level switching infrastructure 32 (L1) and (L2) that connects the TOR network switches together. The higher-level switching infrastructure 32 may take the form of any suitable networking architecture and may be driven by any suitable routing protocol. In the illustrated example, the higher-level infrastructure 32 includes a set of aggregation switches L1 and core switches L2. However, it will be understood that the higher-level switching infrastructure may include any suitable number of levels of switches.

[0013] The virtual machines 30 of virtual machine plane 22 provide a virtual computing environment in which a user of server system 14 can access and run an application, such as a web browser, media application, etc. As discussed in more detail below, these virtual machines 30 of virtual machine plane 22 can be configured to perform multimedia redirection to reduce the computational workload imposed on the nodes of hardware plane 20 that is typically caused by decoding and encoding processes performed for streaming media.

[0014] Furthermore, it should be understood that the server system 14 is not limited to the example of the data center / cloud platform shown in FIG. 2. In other examples, the server system 14 may take the form of one or more computer devices configured to execute a virtual desktop environment. Using the client computer device 12, a related user can connect to a virtual desktop environment executed by one or more computer devices to access software executed within the virtual desktop environment, and this virtual desktop environment can be configured to implement the multimedia redirection functions and processes described herein.

[0015] FIG. 3 shows an exemplary host virtual machine 34 configured to execute a multimedia redirection technique to address the problems discussed above. The host virtual machine 34 can take the form of one of the virtual machines 30 in the virtual machine plane 22 of the cloud platform described with reference to FIG. 2. However, it should be understood that the host virtual machine 34 may be implemented in a non-cloud platform form of the server system 14.

[0016] The host virtual machine 34 can be executed by at least one processor of the server system 14. The host virtual machine 34 can be configured to host a session with at least one client computer device 12 and can host an application executed within the operating system of the host virtual machine 34. The host virtual machine 34 may include an operating system 60 having a multimedia framework 62 for rendering media content. The multimedia framework 62 can provide a runtime environment for media processing within the operating system 60 and includes an application programming interface (API) that can be used by a media application 36 executed within the operating system 60 of the host virtual machine 34.

[0017] As shown, media application 36 can be configured to access media content from media source 16. Using the graphical user interface (GUI) of media application 36, an end user can trigger a media experience of the media content stored by media source 16. In one example, media source 16 may be local to server system 14 that executes host virtual machine 34. In another example, media source 16 may be remote to server system 14 and may be configured to transmit encoded media content 18 to server system 14 through a WAN.

[0018] After a media experience is triggered in media application 36, host virtual machine 34 can be configured to receive encoded media content 18 from media source 16 for the triggered experience. Encoded media content 18 may be encoded using any suitable algorithm. In one example, encoded media content 18 is received in a media container format 38 that includes one or more of video stream 40, audio stream 42, and text stream 44. As a specific example, media container format 38 may be an MP4 format, a WAV format, an AVI format, or another type of media container format 38.

[0019] The encoded media content 18 received by the host VM 34 can be received by the media application 36. The media application 36 can include program logic that makes API calls to the multimedia framework 62 to perform media processing on the encoded media content 18. The encoded media content 18 is then passed to the multimedia framework 62 and processed by the multimedia framework 62 of the operating system 60 of the host virtual machine 34. The multimedia framework 62 can be configured to provide a rendering pipeline and infrastructure for media rendering of the encoded media content 18. The multimedia framework 62 can load the multimedia framework source 46 based on the type of the media container format 38 of the encoded media content 18. The multimedia framework source 46 can be configured to demultiplex the encoded media content 18 in the media container format 38 into independent video, audio, and text streams. Typically, as shown in FIG. 3, the independent encoded audio stream 42 and the encoded video stream 40 are processed by the audio and video rendering pipelines of the multimedia framework 62. For example, the encoded audio stream 42 and the encoded video stream 40 may be processed by the media conversion module of the multimedia framework 62, such as a decoder 48 that decompresses the audio and video streams. The decompressed audio and video streams may then be passed to a streaming audio renderer 50 and a streaming video renderer 52, respectively, to prepare the video and audio content for presentation. As discussed above, these decoding and rendering processes executed on the host VM 34 consume the hardware resources of the server device executing the host VM 34, which can potentially degrade the user experience of other users hosted within the multi-session operating system of the host virtual machine 34.

[0020] To address these issues, host virtual machine 34 may be configured to implement a multimedia redirect module 54 configured to redirect encoded media content 18, including encoded audio stream 42 and encoded video stream 40, to client computer device 12. For example, rather than constructing a complete multimedia framework topology using a media conversion module and a renderer, host virtual machine 34 may load a multimedia redirect module 54 that functions as a media sink and be configured to render directly from multimedia framework source 46 to multimedia redirect module 54. FIG. 4 is a flowchart of a method 400 implemented by server system 14 executing host virtual machine 34 to perform multimedia redirect. Method 400 may be executed using the system described above or using other suitable hardware and software elements.

[0021] At 402, method 400 may include executing a host virtual machine configured to host a session with at least one client computer device. Host virtual machine 34 may be executed by at least one processor of server system 14, which may take the form of, for example, a cloud platform data center as shown in FIG. 2. Host virtual machine 34 may include an operating system having a multimedia framework for rendering media content. In one example, the operating system may be a multi-session operating system configured to host multiple sessions with multiple client computer devices 12 simultaneously. The multiple sessions may be hosted within the same multi-session operating system instance. Each session may implement the techniques and processes for multimedia redirect described herein.

[0022] At 404, method 400 can include executing a media application configured to access media content from a media source. The media application 36 can execute within the host virtual machine 34. In an example of a multi-session operating system, each of the plurality of sessions can have an instance of the associated media application 36 that executes in the multi-session operating system. The media source 16 can, in one example, take the form of a third-party media service that streams media content such as video and / or audio. The media application 36 can be configured to access the media source 16 based on user input that triggers a media experience. In another example, the media source 16 can take the form of a storage device of the server system 14 that stores media content associated with the media application 36.

[0023] At 406, method 400 can include processing encoded media content from the media application using a multimedia framework. The encoded media content 18 can be received by the host virtual machine 34 through the WAN in an example of a remote media source. As shown in FIG. 3, the encoded media content 18 can include one or more of an encoded video stream 40, an encoded audio stream 52, and an encoded text stream 44. The encoded streams of media content can be grouped in a media container format 38 such as, for example, the MP4 format, the WAV format, etc.

[0024] At 408, method 400 can include demultiplexing the encoded media content in the media container format into independent streams of the encoded media content. For example, the multimedia framework 62 of the operating system 60 can include a media conversion module that unpacks the media container format 38 and demultiplexes the encoded media content 18 into independent streams, such as an encoded video stream 40, an encoded audio stream 42, and / or an encoded text stream 44, etc.

[0025] At 410, method 400 can include executing a multimedia redirect module configured to intercept the encoded media content since it is to be processed by a decoding module of the multimedia framework. The multimedia redirect module 54 can include a media sink, and the multimedia framework 62 of the operating system 60 can be configured to render directly from the source 46 to the media sink of the multimedia redirect module 54 instead of constructing a complete rendering topology. The multimedia redirect module 54 can receive the encoded content from the multimedia framework 62 and redirect the encoded content to the associated client computer device 12 of the session. In this way, the encoded media content 18 is not decoded by the decoder 48 on the host virtual machine 34 and is not processed by the renderers 50 and 52.

[0026] The encoded media content 18 redirected to the client computer device 12 can be buffered and streamed to the client computer device 12 through the WAN. At 412, the method 400 can include buffering independent streams of the encoded media content. Independent streams of the encoded media content, such as the encoded audio stream 42 and the encoded video stream 40, can be temporarily stored in a buffer on the host virtual machine 34.

[0027] At 414, the method 400 can include transmitting media samples of the buffered independent streams of the encoded media content to at least one client computer device in response to receiving a request for media samples from the at least one client computer device. The client computer device 12 can independently request new media samples for each stream of the encoded media content 18. For example, the client computer device 12 can request media samples of the encoded video stream 40 independently of requesting media samples of the encoded audio stream 42. In this way, the client computer device 12 can request media samples of each independent stream of the encoded media content (e.g., the encoded audio stream, the encoded video stream, etc.) at an independent rate, and the host virtual machine 34 can transmit such media samples of each independent stream of the encoded media content at the requested rate. Typically, video content is larger in size than audio content and thus may require higher-rate video samples to be sent to the client computer device 12.

[0028] FIG. 5 shows an exemplary server-side software architecture for implementing the multimedia redirect module 54 and the steps described in method 400. In the illustrated example, at least one processor 56 of the server device included in server system 14 can be configured to execute a host virtual machine 34 configured to host a session 58 with at least one client computer device 12. In the illustrated example, host virtual machine 34 implements a multi-session operating system 60 configured to simultaneously host multiple sessions 58 for multiple client computer devices 12. Each session 58 can be associated with a different client computer device 12 and can execute a separate instance of an application program on host virtual machine 34. For example, each session 58 can execute a separate instance of a media application 36 configured to access media content from media source 16.

[0029] Each session 58 executed within the multi-session operating system 60 can be further configured to receive the encoded media content 18 from the media source 16 in the media container format 38, as described above with reference to FIGS. 3 and 4. In one example, the encoded media content 18 can take the form of a media file stored in a storage device associated with the host virtual machine 34. In this example, the media source 16 may be a related storage device that is local to the server system 14. However, it should be understood that the encoded media content 18 may take other suitable forms, such as, for example, dynamic streaming media content, HTTP progressive streaming content, or other forms of streaming media content received from a remote media source. Further, the encoded media content 18 may be received in a media container format that includes one or more of a video stream, an audio stream, and a text stream. The video stream, audio stream, and text stream can be multiplexed (muxed) in the media container format. It should be understood that the media container format may take the form of an MP4 format, a WAV format, or any other suitable type of media container format.

[0030] The execution of the media application 36 can spin up a plurality of related processes executed within the multi-session operating system 60. As a specific example, a media application process 64 for the session 58 with the client computer device 58 and a multimedia framework process 66 can be executed. In one example, the functions of the multimedia framework process 66 described herein may be included in the processes of the multi-session operating system 60. In another example, the multimedia framework process 66 may be executed in a separate processing thread.

[0031] The application logic 68 for the media application 36 can be executed within the media application process 64. The application logic 68 can include program logic that retrieves the encoded media content 18 based on a user of the media application 36 triggering a media viewing experience. The application logic 68 may further include program logic that makes an API call 70 to the multimedia framework process 66 to render the encoded media content 18. The encoded media content 18 retrieved by the media application process 64, and / or a unique resource identifier of the encoded media content 18, can also be passed to the multimedia framework process 66.

[0032] The multimedia framework process 66 executed by the processor 56 can be configured to process the encoded media content 18 received from the media application process 64. In one example, the multimedia process 66 can be configured to identify the format type of the encoded media content 18, such as the type of the media container format 38, and then select the corresponding multimedia framework source 72. Among other functions, the multimedia framework source 72 can be configured to unpack the encoded media content 18 and demultiplex the encoded media content 18 into independent streams 74 of the encoded media content. The demultiplexed independent streams 74 of the encoded media content can include, for example, an encoded video stream, an encoded audio stream, and / or an encoded text stream.

[0033] If multimedia redirection (MMR) is not currently enabled for the received encoded media content 18, the multimedia framework process 66 may be configured to load a rendering pipeline for the demultiplexed streams of video, audio, and / or text. For example, the multimedia framework process 66 may be configured to load a plurality of media conversion modules that process the encoded media content. These media conversion modules can include, for example, the decoder 48 shown in FIG. 3 and other modules that perform other types of data conversion. The multimedia framework process 66 can also load a streaming audio renderer 50, a streaming video renderer 52, etc. The multimedia framework process 66 can then use the loaded rendering pipeline to process the demultiplexed independent streams 74 of the encoded media content. In some examples, the rendering process may include other lower-level processes of the multi-session operating system 60. The display frames of the rendered video content and the audio frames of the rendered audio content can then be encoded and streamed to the client computer device 12 over the WAN.

[0034] On the other hand, when MMR is enabled, the multimedia framework process 66 may be configured to intercept the demultiplexed stream 74 of the media content and load a multimedia redirection module 54 configured to perform the functions and processes described herein. In this way, the demultiplexed stream 74 of the media content is not decoded and rendered on the host virtual machine 34, providing the potential advantage of reducing the computational workload imposed on the hardware executing the host virtual machine 34.

[0035] In one example, the multimedia redirect module 54 can take the form of a media sink, and the multimedia framework source 72 can be configured to render directly to the multimedia redirect module 54 when MMR is enabled. Instead of processing the demultiplexed stream 74 of the encoded media content using the media conversion module as described above, the multimedia redirect module 54 can be configured to redirect the demultiplexed stream 74 of the encoded media content to the corresponding client computer device 12 of its session 58. To redirect the media content, the multimedia redirect module 54 can be configured to handle the transmission of the independent demultiplexed streams of the encoded media data to the dynamic virtual channel (DVC) plugin implemented on the client computer device 12. The multimedia redirect module 54 can further handle the communication of serialized messages with the DVC plugin on the client computer device 12. In one example, the commands associated with the playback and rendering of the encoded media content 18 may be serialized using a protocol such as the Extensible GOOGLE (registered trademark) PROTOCOL BUFFERS, which is a language and platform-neutral mechanism for serializing and deserializing structured data. However, it should be understood that other serialization protocols may be implemented. The multimedia redirect module 54 can be configured to transmit and receive these commands to and from the client computer device 12 via the serialized messages.

[0036] Furthermore, when redirecting the encoded media content 18 to the client computer device 12, the MMR module 54 may be configured to buffer an independent stream 74 of the encoded media content and transmit media samples of the buffered independent stream 74 of the encoded media content to at least one client computer device 12 in response to receiving a media sample request from the at least one client computer device 12. For example, the client computer device 12 can send a serialized message including a media sample request 76 for at least one of the streams of the encoded media content to the MMR module 54. The MMR module 54 can then send the requested encoded media samples 78 stored in the buffer to the client computer device 12 through a dynamic virtual channel. In one example, the client computer device 12 may send a separate request 76 for each independent stream 74 of the encoded media content. For example, the client computer device 12 can request media samples of the encoded video content stream at a rate higher than the media samples of the encoded audio content. Thus, the MMR module 54 can be configured to transmit media samples 78 for each of the independent streams 74 of the encoded media content at an independent rate based on the media sample request 76 received from the at least one client computer device 12.

[0037] As discussed above, the MMR module 54, which can take the form of a media sink of the multimedia framework 62, is loaded and executed when MMR is enabled for the encoded media content 18. In one example, at least one processor 56 is configured to determine whether to enable MMR, execute the MMR module 54 to intercept the encoded media content 18, or enable the encoded media content 18 to be processed by the decoding module 48 of the multimedia framework 62 and other media conversion modules, based on one or more monitoring parameters 80. In one example, the host virtual machine 34 may be configured to monitor network parameters between the server system 14 and at least one client computer device 12 and determine whether to enable MMR based on the monitored network parameters.

[0038] As another example, the host virtual machine 34 may be configured to monitor the current processing load of at least one processor 56 that executes the host virtual machine 34 and determine whether to enable MMR based on the current processing load. As a specific example, when the current processing load on at least one processor 56 exceeds a threshold level, the host virtual machine 34 may enable MMR, load the MMR module 54 and its associated media stack, and intercept the encoded media content 18 to be processed, for example, by a rendering pipeline that may typically include a plurality of media conversion modules and a renderer. On the other hand, when the current processing load is lower than the threshold level and the monitored network parameters indicate a low network bandwidth, the host virtual machine may be configured to disable MMR and decode / render the encoded media content. The rendered frames can then be encoded at a compression level suitable for the network bandwidth between the server system and the client computer device.

[0039] As another example, the host virtual machine 34 may be configured to determine whether to enable MMR and load the MMR module 54 based on user settings of at least one client computer device 12. For example, an administrator can control the user settings to enable or disable MMR for a client computer device. The user settings may be sent to the host virtual machine 34 and stored for that client computer device. In one particular example, MMR may be independently enabled or disabled via user settings for different media sources 16.

[0040] As discussed above, the host virtual machine 34 can implement a multi-session operating system 60 that hosts sessions 58 with a plurality of client computer devices 12. Activity with each client computer device 12 can increase the processing workload imposed on at least one processor 56 that executes the host virtual machine 34. Thus, in one example, the host virtual machine 34 can determine the aggregate activity of the plurality of sessions 58 hosted by the multi-session operating system 60 and be configured to determine whether to execute the MMR module 54 based on the aggregate activity. As a specific example, if the aggregate activity of the plurality of sessions 58 is low because only one session is currently accessing encoded media content for streaming, the host virtual machine 34 may be configured not to enable MMR and to allow the decoding and rendering processes to be executed on the host virtual machine 34. On the other hand, if the aggregate activity of the plurality of sessions 58 is high due to the plurality of sessions attempting to access media content that has ended, the host virtual machine 34 may be configured to enable MMR and load the MMR module 54.

[0041] The host virtual machine 34 can be configured to dynamically enable or disable MMR based on changes to the monitoring parameter 80. Further, it should be understood that the monitoring parameters of the examples discussed above are merely illustrative, and the host virtual machine 34 may dynamically determine whether to enable MMR based on other parameters such as the size of the encoded media content, the source of the encoded media content, etc.

[0042] FIG. 6 shows an exemplary remote desktop client 82 that can be executed by a processor of each client computer device 12 to communicate with the host virtual machine 34. The remote desktop client 82 includes an MMR DVC plugin 84 configured to interact with the MMR module 54 executed by the host virtual machine 34. When MMR is enabled by the host virtual machine 34, the MMR DVC plugin 84 can be loaded into the remote desktop client 82. On the other hand, when MMR is not enabled, the remote desktop client 82 can be configured to use typical remote processing techniques and processes.

[0043] The MMR DVC plugin 84 includes a communication layer 86 and a media layer 88. The communication layer 86 is configured to handle communication between the MMR DVC plugin 84 and the MMR module 54 of the host virtual machine 34. The communication layer 86 can include an MMR virtual channel 90 configured to open a channel with the MMR module 54 of the host virtual machine 34. Instances of the MMR virtual channel 90 may be created for each instance of the media application 36 opened for a session. The remote transceiver 92 may be generated when a new connection from the host virtual machine 34 is received, for example, when an instance of the media application 36 on the host virtual machine 34 is opened and media content is accessed.

[0044] The remote transceiver 92 can be configured to transmit and receive messages 94 with the MMR module 54 of the host virtual machine 34. As shown, the message 94 is structured to include a main message handle 96, a source message handle 98, a renderer message handle 100, and a demultiplexer (demuxer) stream message handle 102. The main message handle 96 can be configured to include data for negotiating versioning information between the client and the host.

[0045] The message 94 can include a source message handle 98 for each multimedia framework source 72. Each source message handle 98 further includes a respective renderer message handle 100 for handling messages that can include data regarding the playback state of the encoded media content 18. For example, the renderer message handle 100 can communicate whether the media content should be paused or seeked to a target position. Further, the renderer message handle 100 may be used to report the status of playback, such as the playback position, whether playback has ended, or an error in playback, to the host virtual machine 34.

[0046] Each renderer message handler 100 further includes a demultiplexer stream message handler 102 configured to include data such as, for example, to request media samples of each demultiplexed independent stream 74 of the encoded media in order to handle the demultiplexing-based operation. That is, the client computer device 12 can request new media samples for a specific stream of the encoded media content via the demultiplexer stream message handler 102. As discussed above, the client computer device 12 can request media content at independent rates for each demultiplexed stream of the media content. For example, the client computer device 12 can request media samples of video media content at a higher rate than media samples of audio media content.

[0047] The media layer 88 includes a media renderer 104 that can transmit and receive messages 104 via the remote transceiver 92 of the communication layer 86. Using the message 94, the media renderer 104 can request media samples of the stream 74 of the encoded media content, such as, for example, media samples of the encoded video content, media samples of the encoded audio content, and media samples of the encoded text content. The media renderer 104 can then decode the received media samples and render them via an output device of the client computer device 12, such as a display, speaker, etc., for presentation to the user. It should be understood that the media renderer 104 may use any suitable rendering technique for the platform of the client computer device 12.

[0048] In the method discussed above, the encoded media content can be redirected to the client computer device 12 so that the media content is not decoded or rendered on the host virtual machine 34. Rather, the encoded media content is redirected and streamed to the client computer device 12, which is configured to decode and render the media content for display. Thus, by performing MMR and avoiding the decoding / rendering process on the host virtual machine, the processing workload on the hardware running the host virtual machine can be reduced.

[0049] In some embodiments, the methods and processes described herein can be associated with a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer application program or service, an application programming interface (API), a library, and / or other computer program products.

[0050] FIG. 7 schematically shows a non-limiting embodiment of a computing system 700 that can implement one or more of the methods and processes described above. The computing system 700 is shown in a simplified form. The computing system 700 can embody the server device of the client computer device 12 and the server system 14 shown in FIGS. 1 and 2 described above. The computing system 700 can take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices, as well as wearable computing devices such as smartwatches and head-mounted augmented reality devices.

[0051] The computing system 700 includes a logical processor 702, volatile memory 704, and a non-volatile storage device 706. The computing system 700 may optionally include a display subsystem 708, an input subsystem 710, a communication subsystem 712, and / or other components not shown in FIG. 7.

[0052] The logical processor 702 includes one or more physical devices configured to execute instructions. For example, the logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions can be implemented to perform tasks, implement data types, transform the state of one or more components, achieve a technical effect, or otherwise reach a desired result.

[0053] The logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute logic or firmware instructions implemented in hardware. The processors of the logical processor 702 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The individual components of the logical processor may optionally be distributed among two or more separate devices, which may be remotely located and / or configured for cooperative processing. Aspects of the logical processor may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. In such a case, it will be understood that these virtualized aspects are executed on different physical logical processors of various different machines.

[0054] The non-volatile memory device 706 includes one or more physical devices configured to hold instructions executable by a logical processor for implementing the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile memory device 706 can be transformed, for example, to hold different data.

[0055] The non-volatile memory device 706 may include removable and / or built-in physical devices. The non-volatile memory device 706 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray (registered trademark) disk, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH (registered trademark) memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), or other mass storage device technologies. The non-volatile memory device 706 can include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It will be appreciated that the non-volatile memory device 706 is configured to hold instructions even when power is removed from the non-volatile memory device 706.

[0056] The volatile memory 704 may include a physical device including random access memory. The volatile memory 704 is typically utilized by the logical processor 702 to temporarily store information during the processing of software instructions. It will be appreciated that the volatile memory 704 typically does not continue to store instructions when power is removed from the volatile memory 704.

[0057] Aspects of the logic processor 702, volatile memory 704, and non-volatile memory device 706 may be integrated together into one or more hardware logic components. Such hardware logic components can include, for example, field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC / ASICs), program- and application-specific standard products (PSSP / ASSPs), system-on-chips (SOCs), and complex programmable logic devices (CPLDs).

[0058] The terms "module", "program", and "engine" may be used to describe aspects of the computing system 700 that are typically implemented in software by a processor to perform a particular function using a portion of the volatile memory, and this function includes transformative processing that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated via a logic processor 702 that uses a portion of the volatile memory 704 to execute instructions held by the non-volatile memory device 706. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" can include individual or groups such as executable files, data files, libraries, drivers, scripts, database records, etc.

[0059] The display subsystem 708, when included, can be used to present a visual representation of data retained by the non-volatile memory device 706. The visual representation can take the form of a graphical user interface (GUI). When the methods and processes described herein change data retained by the non-volatile memory device and thus transform the state of the non-volatile memory device, the state of the display subsystem 708 can likewise be transformed to visually represent the change in the underlying data. The display subsystem 708 may include one or more display devices that utilize substantially any type of technology. Such display devices may be combined with the logical processor 702, volatile memory 704, and / or non-volatile memory device 706 within a shared enclosure, or such display devices may be peripheral display devices.

[0060] The input subsystem 710, when included, includes or can interface with one or more user input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the introduction and / or processing of input actions may be handled on-board or off-board. Exemplary NUI components can include a microphone for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; a head tracker, eye tracker, accelerometer, and / or gyroscope for motion detection and / or intent recognition; and an electric field sensing component for evaluating brain activity; and / or any other suitable sensor.

[0061] When included, the communication subsystem 712 can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 712 may include wired and / or wireless communication devices that are compatible with one or more different communication protocols. By way of non-limiting example, the communication subsystem may be configured for communication via a radio telephone network, or a wired or wireless local or wide area network, e.g., for HDMI (registered trademark) over a Wi-Fi connection. In some embodiments, the communication subsystem may enable the computing system 700 to send and / or receive messages to and / or from other devices via a network such as the Internet.

[0062] The following paragraphs provide further support for the claims of the subject application. One aspect provides a computer system including at least one processor configured to execute a host virtual machine configured to host a session with at least one client computer device. The host virtual machine includes an operating system having a multimedia framework for rendering media content. The at least one processor is further configured to execute a media application configured to access media content from a media source, process the encoded media content from the media application using the multimedia framework, and execute a multimedia redirect module configured to intercept the encoded media content since it is to be processed by the decoding module of the multimedia framework. The multimedia redirect module is configured to redirect the encoded media content to at least one client computer device. In this aspect, additionally or alternatively, the encoded media content may be dynamic streaming media content. In this aspect, additionally or alternatively, the encoded media content may be received in a media container format including one or more of a video stream, an audio stream, and a text stream, and the at least one processor may be configured to demultiplex the encoded media content in the media container format into independent streams of the encoded media content. In this aspect, additionally or alternatively, the independent streams of the encoded media content may include one or more of a video stream, an audio stream, and a text stream.In this aspect, additionally or alternatively, the multimedia redirect module may be configured to buffer independent streams of encoded media content and transmit media samples of the buffered independent streams of encoded media content to at least one client computer device in response to receiving a media sample request from the at least one client computer device. In this aspect, additionally or alternatively, the multimedia redirect module may be configured to transmit media samples for each of the independent streams of encoded media content at an independent rate based on a media sample request received from the at least one client computer device. In this aspect, additionally or alternatively, at least one processor may be configured to determine, based on one or more monitoring parameters, whether to execute the multimedia redirect module to intercept the encoded media content or to enable the encoded media content to be processed by the decoding module of the multimedia framework. In this aspect, additionally or alternatively, the one or more monitoring parameters may be selected from the group including network parameters between the computer system and the at least one client computer device, the current processing load of at least one processor of the computer system, the size of the encoded media content, and user settings of the at least one client computer device. In this aspect, additionally or alternatively, the operating system may be a multi-session operating system configured to simultaneously host multiple sessions for multiple client computer devices. In this aspect, additionally or alternatively, at least one processor may be configured to determine whether to execute the multimedia redirect module based on the aggregated activity of multiple sessions hosted by the multi-session operating system.

[0063] Another aspect provides a method including steps of a processor of a server device executing a host virtual machine configured to host a session with at least one client computer device. The host virtual machine includes an operating system having a multimedia framework for rendering media content. The method further includes steps of executing a media application configured to access media content from a media source, processing the encoded media content from the media application using the multimedia framework, and executing a multimedia redirect module configured to intercept the encoded media content since it is processed by a decoding module of the multimedia framework. The multimedia redirect module is configured to redirect the encoded media content to at least one client computer device. In this aspect, further or alternatively, the encoded media content may be dynamic streaming media content. In this aspect, further or alternatively, the encoded media content may be received in a media container format including one or more of a video stream, an audio stream, and a text stream, and the method may further include a step of demultiplexing the encoded media content in the media container format into independent streams of the encoded media content. In this aspect, further or alternatively, the independent streams of the encoded media content may include one or more of a video stream, an audio stream, and a text stream. In this aspect, further or alternatively, the method may further include steps of buffering the independent streams of the encoded media content, and transmitting the buffered media samples of the independent streams of the encoded media content to at least one client computer device in response to receiving a request for media samples from at least one client computer device.In this aspect, further or alternatively, the method may further include transmitting media samples for each independent stream of the encoded media content at an independent rate based on requests for media samples received from at least one client computer device. In this aspect, further or alternatively, the method may further include determining, based on one or more monitoring parameters, whether to execute a multimedia redirect module to intercept the encoded media content or to enable the encoded media content to be processed by a decoding module of a multimedia framework. In this aspect, further or alternatively, the one or more monitoring parameters may be selected from the group including network parameters between the server device and at least one client computer device, the current processing load of at least one processor of the server device, the size of the encoded media content, and user settings of at least one client computer device. In this aspect, further or alternatively, the operating system may be a multi-session operating system configured to simultaneously host multiple sessions for multiple client computer devices.

[0064] Another aspect provides a server system including at least one processor configured to execute a host virtual machine that implements a multi-session operating system configured to simultaneously host a plurality of sessions for a plurality of client computer devices, the multi-session operating system having a multimedia framework for rendering media content. The at least one processor is further configured to execute an instance of a media application for one or more of the plurality of sessions, the media application being configured to access media content from a media source. The processor is further configured to determine that an aggregate activity of the plurality of sessions of the host virtual machine exceeds a threshold level, enable multimedia redirection for one or more of the plurality of sessions based on the determined aggregate activity, process encoded media content from the instance of the media application using the multimedia framework, and execute a multimedia redirection module configured to intercept the encoded media content since it is processed by a decoding module of the multimedia framework. The multimedia redirection module is configured to redirect the encoded media content to each client computer device of one or more of the plurality of sessions.

[0065] The configurations and / or approaches described herein are exemplary in nature, and it should be understood that these specific embodiments or examples are not to be construed in a limiting sense as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, the various operations illustrated and / or described may be performed in the illustrated and / or described sequence, in other sequences, in parallel, or omitted. Similarly, the order of the processing described above may be changed.

[0066] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems, and configurations, and other features, functions, operations, and / or characteristics disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computer system, at least one processor configured to execute a host virtual machine configured to host a session with at least one client computer device, the host virtual machine including an operating system having a multimedia framework for rendering media content, at least one processor, comprising, the at least one processor further executes a media application configured to access media content from a media source, processes the encoded media content from the media application using the multimedia framework, the encoded media content being a media container format including two or more of a video stream, an audio stream, and a text stream, demultiplexes the encoded media content in the media container format into independent streams of the encoded media content, executes a multimedia redirect module configured to intercept the encoded media content since it is processed by a decoding module of the multimedia framework, the multimedia redirect module being configured to redirect the encoded media content to the at least one client computer device, the multimedia redirect module is configured to buffer independent streams of the encoded media content, the multimedia redirect module is configured to transmit media samples of the buffered independent streams of the encoded media content to the at least one client computer device in response to receiving a request for the media samples from the at least one client computer device configured as such, a computer system.

2. The computer system according to claim 1, wherein the encoded media content is dynamic streaming media content.

3. The multimedia redirect module of the computer system according to claim 1 is configured to transmit media samples for each independent stream of the encoded media content at independent rates based on requests for the media samples received from the at least one client computer device.

4. The computer system according to claim 1, wherein the at least one processor is configured to determine, based on one or more monitoring parameters, whether to execute the multimedia redirect module to intercept the encoded media content or to enable the encoded media content to be processed by the decoding module of the multimedia framework.

5. The computer system according to claim 4, wherein the one or more monitoring parameters are selected from the group including network parameters between the computer system and the at least one client computer device, the current processing load of the at least one processor of the computer system, the size of the encoded media content, and user settings of the at least one client computer device.

6. The computer system according to claim 4, wherein the operating system is a multi-session operating system configured to simultaneously host multiple sessions for multiple client computer devices.

7. The computer system according to claim 6, wherein the at least one processor is configured to determine whether to execute the multimedia redirect module based on the aggregated activity of the multiple sessions hosted by the multi-session operating system.

8. In a processor of a server device, executing a host virtual machine configured to host a session with at least one client computer device, the host virtual machine including an operating system having a multimedia framework for rendering media content; executing a media application configured to access media content from a media source; Processing the encoded media content from the media application using the multimedia framework, wherein the encoded media content is received in a media container format including two or more of a video stream, an audio stream, and a text stream; Demultiplexing the encoded media content in the media container format into independent streams of the encoded media content; Determining, based on one or more monitoring parameters, whether to execute a multimedia redirect module to intercept the encoded media content or to enable the encoded media content to be processed by a decoding module of the multimedia framework; The multimedia redirect module is configured to intercept the encoded media content from being processed by the decoding module of the multimedia framework; The multimedia redirect module is configured to redirect the encoded media content to the at least one client computer device; Buffering the independent streams of the encoded media content; Transmitting media samples of the buffered independent streams of the encoded media content to the at least one client computer device in response to receiving a request for the media samples from the at least one client computer device; A method comprising the above steps.

9. The method according to claim 8, further comprising transmitting media samples for each of the independent streams of the encoded media content at an independent rate based on the request for the media samples received from the at least one client computer device.

Images