Multimedia redirection for media applications

Abstract

A computer system is provided that includes at least one processor configured to execute a host virtual machine configured to host a session with at least one client computer device. The host virtual machine includes an operating system having a multimedia framework for drawing media content. The at least one processor is further configured to execute a media application configured to access media content from a media source, process encoded media content from the media application using a multimedia framework, and execute a multimedia redirection module, the multimedia redirection module is configured to intercept the encoded media content from being processed by a decoding module of the multimedia framework. The multimedia redirection module is configured to redirect the encoded media content to at least one client computer device.

Description

[0001] Description of the case

[0002] This application is a divisional application of the invention patent application with application date of November 24, 2020, application number "202080091636.5", and invention name "Multimedia Redirection for Media Applications". Background Art

[0003] In a virtual environment, end users can use client devices to connect to virtual machines running on server hardware, which can provide end users with more computing power, application compatibility, security and regulatory compliance, and reduce the total cost to the end user. Typically, these end users may consume media through media player applications, which can increase the workload on the server hardware and increase the associated costs. Summary of the invention

[0004] A computer system is provided, which may include at least one processor configured to execute a host virtual machine, the host virtual machine configured to host a session with at least one client computer device. The host virtual machine may include an operating system having a multimedia framework for rendering media content. The at least one processor may also be configured to execute a media application configured to access media content from a media source, process encoded media content from the media application using the multimedia framework, and execute a multimedia redirection module, the multimedia redirection module being configured to intercept the encoded media content so as not to be processed by a decoding module of the multimedia framework. The multimedia redirection module may be configured to redirect the encoded media content to at least one client computer device.

[0005] This summary is provided to introduce a selection of concepts in a simplified form that are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a schematic diagram of an example computer system for performing the multimedia redirection described herein.

[0007] Figure 2 yes Figure 1 Schematic diagram of an example server system of a computer system.

[0008] Figure 3 Is to achieve Figure 1A schematic diagram of a media application and multimedia framework of an operating system of a multimedia redirection module of a computer system.

[0009] Figure 4 Is Figure 1 A flowchart of a method for performing multimedia redirection implemented by a computer system.

[0010] Figure 5 It is used to achieve Figure 1 Schematic diagram of the software architecture of media applications and multimedia framework for multimedia redirection of a computer system.

[0011] Figure 6 is used with Figure 1 A schematic diagram of the software architecture of a computer system implementing a multimedia redirection module for communicating with a remote desktop client.

[0012] Figure 7 yes Figure 1 Schematic diagram of an example computing environment in which a computer system may be implemented. DETAILED DESCRIPTION

[0013] Cloud platform services can provide end users with a virtualized desktop environment that can utilize client computer devices to connect to virtual machines running on the cloud platform's server devices. Typically, the cloud platform's computer hardware and network capabilities are more powerful than the end user's client computer devices. Therefore, by hosting these end users in a virtualized desktop environment, the cloud platform can provide these users with more computing power, as well as other potential benefits, such as improved application compatibility, security / regulatory compliance, and an overall reduction in related computing costs.

[0014] In some examples, these cloud platform services can provide virtual desktop infrastructure for enterprises with multiple end users and client devices. Users of these enterprises can consume media (e.g., video and / or audio media) that is stored locally on the cloud platform for the enterprise or stored by a remote media source and streamed to a virtualized desktop environment running on the cloud platform. Typically, the media content provided by the media source is decoded on the virtual machine hosting the end user, and then the display frame of the virtualized desktop for presenting the media content is re-encoded and streamed to the user's client computer device via the remote desktop protocol. The client computer device can then decode these display frames and present them to the user via the display of the client computer device.

[0015] The decoding and subsequent encoding processes performed on a virtual machine can consume significant computing resources of the server hardware. In one specific example, the video rendering process for a video stream can potentially consume 20%-40% of the central processing unit (CPU) resources of the server device running the virtual machine. These problems can be further exacerbated in multi-session virtualized desktop environment deployments where multiple users share an operating system (OS) instance on a VM. For example, if more than one user attempts to stream video content through a shared host VM, all of the CPU resources of that VM may be consumed by the related decoding / encoding / rendering processes, potentially degrading the user experience of all users hosted by that VM.

[0016] In order to solve the problems discussed above, Figure 1 An example computer system 10 for a virtualized desktop environment is shown that performs multimedia redirection to reduce the computing workload on the server hardware running the virtualized desktop environment, as will be discussed in more detail below. Figure 1 As shown, computer system 10 includes one or more client computer devices 12, server system 14, and one or more media sources 16. End users of one or more client computer devices 12 can connect to a virtual machine running on server system 14. Using an application executed within a host operating system of the VM, such as a media application that plays media content, or a web browser application, the end user can interact with media stored by media source 16.

[0017] In one example, the media source 16 may be local to the server system 14 and may be included in the server system 14. In another example, the media source 16 may be remote to both the client computer device 12 and the server system 14. For example, the server system 14 may be configured to communicate with the remote media source via a communication network such as a wide area network (WAN). As a specific example, an end user may interact with a media application executed within a host VM to trigger a media experience for viewing media stored on the media source 16. The media application may, for example, take the form of a media streaming application, which facilitates streaming and display of media content retrieved from the media source 16. Typically, the streaming video and / or audio is sent to the server system 14 as encoded media 18 and is processed by the host VM executing the media application that is playing the streaming video and / or audio.

[0018] As described above, a typical virtualized environment implementation will perform a media rendering process, including decoding the encoded media 18, and presenting the media content via the virtualized desktop environment, encoding display / audio frames for the virtualized desktop environment, and then sending the encoded display / audio frames to the client computer device 12 for presentation to the end user. In order to reduce the computational burden on the hardware of the server system 14 caused by the decoding / encoding process, the server system 14 can implement multimedia redirection to pass the encoded media 18 received from the media source 16 to the client computer device 12 without performing the encoding or decoding process on the server system 14.

[0019] Figure 2 An example of a server system 14 is shown. The server system 14 may include a hardware plane 20, a virtual machine plane 22, a hypervisor plane 24, and a network infrastructure 26, which are collectively configured to operate a cloud platform. The hardware plane 20 includes a set of nodes 28 (each node 28 is Figure 2 The host server instance executed by the node 28 of the hardware plane 20 is configured to communicate with one or more hypervisors of the hypervisor plane 24. The one or more hypervisors of the hypervisor plane 24 can create, process and monitor multiple virtual machines 30 of the virtual machine plane 22 (each virtual machine 30 in Figure 2 Through the hypervisor plane 24, each virtual machine 30 of the virtual machine plane 22 can be hosted and run by hardware components of one or more nodes 28 of the hardware plane 20. In this way, multiple virtual machines 30 of the virtual machine plane 22 can share virtualized hardware resources managed by the hypervisor plane 24. Each virtual machine 30 provides a virtualized desktop environment in which software, such as a web browser application, a media player application, and other types of software, can execute.

[0020] In one example, the computer system 14 corresponds to a data center environment that communicatively couples multiple nodes 28 via a standard network infrastructure. For example, the network infrastructure 20 may include a typical network infrastructure, such as a server rack including a top-of-rack (TOR) network switch. The computer system 14 may include multiple node clusters, each node cluster having an associated TOR network switch. The network infrastructure 20 may also include a higher-level switching infrastructure 32 (L1) and (L2) that connects the TOR network switches together. The higher-level switching infrastructure 32 can take the form of any suitable network architecture and can be driven by (multiple) any suitable routing protocol. In the example shown, the higher-level infrastructure 32 includes a collection of aggregation switches L1 and core switches L2. However, it should be understood that the higher-level switching infrastructure may include any suitable number of levels of switches.

[0021] The virtual machines 30 of the virtual machine plane 22 provide a virtual computing environment in which users of the server system 14 can access and run applications, such as web browsers, media applications, etc. As will be discussed in more detail below, these virtual machines 30 of the virtual machine plane 22 can be configured to implement multimedia redirection to reduce the computing workload imposed on the nodes of the hardware plane 20 caused by the decoding and encoding processes typically performed for streaming media.

[0022] In addition, it should be understood that the server system 14 is not limited to Figure 2 The data center / cloud platform example shown. In other examples, the server system 14 can take the form of one or more computer devices configured to run a virtual desktop environment. Using the client computer device 12, the associated user can connect to the virtualized desktop environment run by one or more computer devices to access the software running within the virtualized desktop environment, and the computer device can be configured to implement the multimedia redirection functions and processes described herein.

[0023] Figure 3 An example host virtual machine 34 configured to perform multimedia redirection technology to solve the above problems is shown. The host virtual machine 34 can use reference Figure 2 The host virtual machine 34 is in the form of a virtual machine in the virtual machine plane 22 of the cloud platform. However, it should be understood that the host virtual machine 34 can also be implemented in the form of a non-cloud platform of the server system 14.

[0024] The host virtual machine 34 may be executed by at least one processor of the server system 14. The host virtual machine 34 may be configured to host a session with at least one client computer device 12, and may host applications running within an operating system of the host virtual machine 34. The host virtual machine 34 may include an operating system 60 having a multimedia framework 62 for rendering media content. The multimedia framework 62 may provide a runtime environment for media processing within the operating system 60, and include an application programming interface (API) that may be used by a media application 36 executing within the operating system 60 of the host virtual machine 34.

[0025] As shown, the media application 36 may be configured to access media content from the media source 16. Using a graphical user interface (GUI) of the media application 36, an end user may trigger a media experience for the media content stored by the media source 16. In one example, the media source 16 may be local to the server system 14 running the host virtual machine 34. In another example, the media source 16 may be remote to the server system 14 and may be configured to send the encoded media content 18 to the server system 14 over a WAN.

[0026] After the media experience is triggered in the media application 36, the host virtual machine 34 can be configured to receive the encoded media content 18 from the media source 16 for the triggered experience. The encoded media content 18 can be encoded using any suitable algorithm. In one example, the encoded media content 18 is received in a media container format 38 that includes one or more of the following: a video stream 40, an audio stream 42, and a text stream 44. As a specific example, the media container format 38 can be an MP4 format, a WAV format, an AVI format, or other types of media container formats 38.

[0027] The encoded media content 18 received by the host VM 34 may be received at the media application 36. The media application 36 may include program logic that makes API calls to the multimedia framework 62 to perform media processing on the encoded media content 18. The encoded media content 18 is then passed to the multimedia framework 62 and processed by the multimedia framework 62 of the operating system 60 of the host virtual machine 34. The multimedia framework 62 may be configured to provide a rendering pipeline and infrastructure for media rendering of the encoded media content 18. The multimedia framework 62 may load a multimedia framework source 46 based on the type of media container format 38 of the encoded media content 18. The multimedia framework source 46 may be configured to demultiplex (demux) the encoded media content 18 in the media container format 38 into separate video, audio, and text streams. Typically, as Figure 3As shown, the independent encoded audio stream 42 and the encoded video stream 40 will be processed by the audio and video rendering pipeline of the multimedia framework 62. For example, the encoded audio stream 42 and the encoded video stream 40 can be processed by the media transformation module of the multimedia framework 62, such as the decoder 48 that decompresses the audio and video streams. The decompressed audio and video streams can then be passed to the stream audio renderer 50 and the stream video renderer 52, respectively, to prepare the video and audio content for presentation. As described above, these decoding and rendering processes performed on the host VM 34 will consume hardware resources of the server device running the host VM 34, which may potentially reduce the user experience of other users in the multi-session operating system hosted in the host virtual machine 34.

[0028] To address these issues, the host virtual machine 34 may be configured to implement a multimedia redirection module 54 that is configured to redirect the encoded media content 18, including the encoded audio stream 42 and the encoded video stream 40, to the client computer device 12. For example, instead of building a complete multimedia framework topology with a media transformation module and a renderer, the host virtual machine 34 may be configured to load the multimedia redirection module 54 that acts as a media sink and renders directly to the multimedia redirection module 54 from the multimedia framework source 46. Figure 4 is a flow chart of a method 400 for performing multimedia redirection implemented by a server system 14 running a host virtual machine 34. The method 400 may be performed using the system described above or using other suitable hardware and software elements.

[0029] At 402, method 400 may include executing a host virtual machine configured to host a session with at least one client computer device. Host virtual machine 34 may be executed by at least one processor of server system 14, which may be implemented, for example, using Figure 2 The form of a cloud platform data center as shown. The host virtual machine 34 may include an operating system with a multimedia framework for rendering media content. In one example, the operating system may be a multi-session operating system configured to concurrently host multiple sessions for multiple client computer devices 12. Multiple sessions may be hosted in the same multi-session operating system instance. Each session may implement the techniques and processes for multimedia redirection described herein.

[0030] At 404, method 400 may include executing a media application configured to access media content from a media source. Media application 36 may be running within a host virtual machine 34. In a multi-session operating system example, each of a plurality of sessions may have an associated media application 36 instance running in the multi-session operating system. In one example, media source 16 may take the form of a third-party media service that streams media such as video and / or audio. Media application 36 may be configured to access media source 16 based on user input that triggers a media experience. In another example, media source 16 may take the form of a storage device of server system 14 that stores media content associated with media application 36.

[0031] At 406, method 400 may include processing the encoded media content from the media application using the multimedia framework. In the remote media source example, the encoded media content 18 may be received by the host virtual machine 34 via a WAN. Figure 3 As shown, the encoded media content 18 may include one or more of an encoded video stream 40, an encoded audio stream 52, and an encoded text stream 44. The encoded media content streams may be bundled in a media container format 38, such as, for example, MP4 format, WAV format, or the like.

[0032] At 408, the method 400 may include demultiplexing the encoded media content in the media container format into independent encoded media content streams. For example, the multimedia framework 62 of the operating system 60 may include a media transformation module that includes processes for unpacking the media container format 38 and demultiplexing the encoded media content 18 into independent streams (e.g., the encoded video stream 40, and the encoded audio stream 42, and / or the encoded text stream 44).

[0033] At 410, the method 400 may include executing a multimedia redirection module that is configured to intercept the encoded media content so as not to be processed by the decoding module of the multimedia framework. The multimedia redirection module 54 may include a media receiver, and the multimedia framework 62 of the operating system 60 may be configured to draw directly from the source 46 to the media receiver of the multimedia redirection module 54, rather than building a complete drawing topology. The multimedia redirection module 54 may receive the encoded content from the multimedia framework 62 and redirect the encoded content to the associated client computer device 12 of the session. In this way, the encoded media content 18 is not decoded by the decoder 48 or processed by the renderer 50 and the renderer 52 on the host virtual machine 34.

[0034] The encoded media content 18 redirected to the client computer device 12 can be buffered and streamed to the client computer device 12 over the WAN. At 412, the method 400 can include buffering independent encoded media content streams. Independent encoded media content streams such as the encoded audio stream 42 and the encoded video stream 40 can be temporarily stored in a buffer on the host virtual machine 34.

[0035] At 414, method 400 may include sending media samples to at least one client computer device in response to receiving a request for buffered media samples of independent encoded media content streams from at least one client computer device. Client computer device 12 may independently request new media samples for each stream of encoded media content 18. For example, client computer device 12 may request media samples for encoded video stream 40 independently of requesting media samples for encoded audio stream 40. In this way, client computer device 12 may request media samples for each independent encoded media content stream (e.g., encoded audio stream, encoded video stream, etc.) at an independent rate, and host virtual machine 34 may send these media samples for each independent encoded media content stream at the requested rate. Typically, the size of video content will be larger than audio content, and therefore a higher rate of video samples may be required to be sent to client computer device 12.

[0036] Figure 5 An example server-side software architecture for implementing the multimedia redirection module 54 and the steps described in the method 400 is shown. In the illustrated example, at least one processor 56 of a server device included in the server system 14 can be configured to execute a host virtual machine 34 that is configured to host a session 58 with at least one client computer device 12. In the illustrated example, the host virtual machine 34 implements a multi-session operating system 60 that is configured to concurrently host multiple sessions 58 for multiple client computer devices 12. Each session 58 can be associated with a different client computer device 12 and can run a separate instance of an application on the host virtual machine 34. For example, each session 58 can execute a separate instance of a media application 36 that is configured to access media content from a media source 16.

[0037] Each session 58 running within the multi-session operating system 60 may be further configured to receive encoded media content 18 from a media source 16 in a media container format 38, as described above with reference to Figure 3 and Figure 4In one example, the coded media content 18 can be in the form of a media file stored on a storage device associated with the host virtual machine 34. In this example, the media source 16 can be an associated storage device at the server system 14 locals. However, it should be understood that the coded media content 18 can adopt other suitable forms, such as dynamic streaming media content, HTTP progressive streaming media content, or other forms of streaming media content received from a remote media source. In addition, the coded media content 18 can be received in a media container format including one or more of the following: video stream, audio stream, and text stream. Video stream, audio stream, and text stream can be mixed in a media container format. It should be understood that the media container format can adopt the form of MP4 format, WAV format, or any other suitable type of media container format.

[0038] Execution of media application 36 may accelerate multiple associated processes running within multi-session operating system 60. As a specific example, media application process 64 and multimedia framework process 66 may be running for session 58 with client computer device 58. In one example, the functionality of multimedia framework process 66 described herein may be included in the processes of multi-session operating system 60. In another example, multimedia framework process 66 may run in a separate processing thread.

[0039] The application logic 68 of the media application 36 may run within the media application process 64. The application logic 68 may include program logic for retrieving the encoded media content 18 based on a user-triggered media viewing experience of the media application 36. The application logic 68 may also include program logic for making an API call 70 to the multimedia framework process 66 to render the encoded media content 18. The encoded media content 18 retrieved by the media application process 64 and / or a unique resource identifier for the encoded media content 18 may also be passed to the multimedia framework process 66.

[0040] The multimedia framework process 66 executed by the processor 56 can be configured to process the encoded media content 18 received from the media application process 64. In one example, the multimedia process 66 can be configured to identify the format type of the encoded media content 18, such as the type of the media container format 38, and then select the corresponding multimedia framework source 72. Among other functions, the multimedia framework source 72 can be configured to unpack the encoded media content 18 and demultiplex the encoded media content 18 into independent encoded media content streams 74. The demultiplexed independent encoded media content streams 74 can, for example, include an encoded video stream and an encoded audio stream and / or an encoded text stream.

[0041] If multimedia redirection (MMR) is not currently enabled for the received encoded media content 18, the multimedia framework process 66 can be configured to load a rendering pipeline for the demultiplexed video, audio, and / or text streams. For example, the multimedia framework process 66 can be configured to load a plurality of media transformation modules for processing the encoded media content. For example, these media transformation modules may include Figure 3 The multimedia framework process 66 may also load a streaming audio renderer 50, a streaming video renderer 52, etc. The multimedia framework process 66 may then use the loaded rendering pipeline to process the demultiplexed independent encoded media content streams 74. In some examples, the rendering process may include other lower-level processes of the multi-session operating system 60. Display frames for the rendered video content and audio frames for the rendered audio content may then be encoded and streamed to the client computer device 12 over the WAN.

[0042] On the other hand, when MMR is enabled, the multimedia framework process 66 can be configured to load the multimedia redirection module 54, which is configured to intercept the demultiplexed media content stream 74 and perform the functions and processes described herein. In this way, the demultiplexed media content stream 74 is not decoded and is not drawn on the host virtual machine 34 to provide the potential benefit of reducing the computational workload imposed on the hardware running the host virtual machine 34.

[0043] In one example, the multimedia redirection module 54 can take the form of a media receiver, and the multimedia framework source 72 can be configured to draw directly to the multimedia redirection module 54 when MMR is enabled. Instead of processing the demultiplexed encoded media content stream 74 using the media transformation module as described above, the multimedia redirection module 54 can be configured to redirect the demultiplexed encoded media content stream 74 to the corresponding client computer device 12 for the session 58. In order to redirect the media content, the multimedia redirection module 54 can be configured to handle the transmission of the demultiplexed independent encoded media data streams to a dynamic virtual channel (DVC) plug-in implemented on the client computer device 12. The redirection module 54 can also handle the communication of serialized messages with the DVC plug-in on the client computer device 12. In one example, commands associated with the playback and drawing of the encoded media content 18 can be serialized using a protocol, such as an extended PROTOCOL BUFFERS, which is a language and platform neutral mechanism for serializing and deserializing structured data. However, it should be understood that other serialization protocols may also be implemented. The multimedia redirection module 54 may be configured to send and receive these commands with the client computer device 12 via serialized messages.

[0044] In addition, when redirecting the encoded media content 18 to the client computer device 12, the MMR module 54 can be configured to buffer the independent encoded media content streams 74 and send the media samples to the at least one client computer device 12 in response to receiving a request for the media samples of the buffered independent encoded media content streams 74 from the at least one client computer device 12. For example, the client computer device 12 can send a serialized message to the MMR module 54, which includes a request 76 for the media samples of at least one of the encoded media content streams. The MMR module 54 can then send the requested encoded media samples 78 stored in the buffer to the client computer device 12 through the dynamic virtual channel. In one example, the client computer device 12 can send a separate request 76 for each independent encoded media content stream 74. For example, the client computer device 12 can request media samples for the encoded video content stream at a higher rate than the media samples for the encoded audio content. Thus, the MMR module 54 may be configured to send media samples 78 at independent rates for each of the independent encoded media content streams 74 based on a request 76 for media samples received from at least one client computer device 12 .

[0045] As described above, when MMR is enabled for the encoded media content 18, the MMR module 54, which may take the form of a media receiver of the multimedia framework 62, is loaded and executed. In one example, at least one processor 56 may be configured to determine whether to enable MMR and whether to execute the MMR module 54 to intercept the encoded media content 18 or allow the encoded media content 18 to be processed by the decoding module 48 and other media transformation modules of the multimedia framework 62 based on one or more monitoring parameters 80. In one example, the host virtual machine 34 may be configured to monitor network parameters between the server system 14 and at least one client computer device 12, and determine whether to enable MMR based on the monitored network parameters.

[0046] As another example, the host virtual machine 34 can be configured to monitor the current processing load of at least one processor 56 running the host virtual machine 34, and determine whether to enable MMR based on the current processing load. As a specific example, if the current processing load on at least one processor 56 exceeds a threshold level, the host virtual machine 34 can be configured to enable MMR, and load the MMR module 54 and its associated media stack to intercept the encoded media content 18 so as not to be processed by the rendering pipeline, which can typically include multiple media conversion modules and a renderer, for example. On the other hand, if the current processing load is lower than the threshold level and the monitored network parameters indicate that the network bandwidth is low, the host virtual machine can be configured to disable MMR and decode / draw the encoded media content. Then, the drawn frame can be encoded using a compression level suitable for the network bandwidth between the server system and the client computer device.

[0047] As another example, the host virtual machine 34 can be configured to determine whether to enable MMR and load the MMR module 54 based on the user settings of at least one client computer device 12. For example, an administrator can control the user settings for enabling or disabling MMR for a client computer device. The user settings can be sent to the host virtual machine 34 and stored for the client computer device. In a specific example, MMR can be independently enabled or disabled for different media sources 16 via user settings.

[0048] As described above, the host virtual machine 34 may implement a multi-session operating system 60 that hosts sessions 58 with multiple client computer devices 12. The activity of each client computer device 12 may increase the processing workload imposed on the at least one processor 56 running the host virtual machine 34. Thus, in one example, the host virtual machine 34 may be configured to determine the aggregate activity of the multiple sessions 58 hosted by the multi-session operating system 60, and determine whether to execute the MMR module 54 based on the aggregate activity. As a specific example, if the aggregate activity of the multiple sessions 58 is low because only one session is currently accessing the encoded media content for streaming, the host virtual machine 34 may be configured to not enable MMR and allow the decoding and rendering processes to be performed on the host virtual machine 34. On the other hand, if the aggregate activity of the multiple sessions 58 is high because multiple sessions are attempting to access the finished media content, the host virtual machine 34 may be configured to enable MMR and load the MMR module 54.

[0049] The host virtual machine 34 may be configured to dynamically enable or disable MMR based on changes to the monitoring parameters 80. Additionally, it should be understood that the exemplary monitoring parameters discussed above are merely exemplary, and the host virtual machine 34 may dynamically determine whether to enable MMR based on other parameters, such as the size of the encoded media content, the source of the encoded media content, etc.

[0050] Figure 6 An example remote desktop client 82 is shown that can be executed by the processor of each client computer device 12 to communicate with the host virtual machine 34. The remote desktop client 82 includes an MMR DVC plug-in 84 configured to interact with the MMR module 54 executed by the host virtual machine 34. When MMR is enabled by the host virtual machine 34, the MMR DVC plug-in 84 can be loaded on the remote desktop client 82. On the other hand, if MMR is not enabled, the remote desktop client 82 can be configured to use typical remote techniques and procedures.

[0051] The MMR DVC plug-in 84 includes a communication layer 86 and a media layer 88. The communication layer 86 is configured to handle communications between the MMR DVC plug-in 84 and the MMR module 54 of the host virtual machine 34. The communication layer 86 may include an MMR virtual channel 90 configured to open a channel with the MMR module 54 of the host virtual machine 34. An instance of the MMR virtual channel 90 may be created for each instance of the media application 36 opened for the session. A remote transceiver 92 may be created when a new connection from the host virtual machine 34 is accepted, such as when an instance of the media application 36 on the host virtual machine 34 is opened and media content is accessed.

[0052] The remote transceiver 92 may be configured to send and receive messages 94 to / from the MMR module 54 of the host virtual machine 34. As shown, the message 94 is structured to include a main message handler 96, a source message handler 98, a renderer message handler 100, and a demultiplexer stream message handler 102. The main message handler 96 may be configured to include data for negotiating version control information between the client and the host.

[0053] The message 94 may include a source message handler 98 for each multimedia framework source 72. Each source message handler 98 also includes a corresponding renderer message handler 100, which is used to process messages that may include data about the playback status of the encoded media content 18. For example, the renderer message handler 100 can convey whether the media content should be paused, seek to a target position, etc. The renderer message handler 100 can also be used to report the status of the playback to the host virtual machine 34, such as the playback position, whether the playback is over, or a playback error.

[0054] Each renderer message handler 100 also includes a demultiplexer stream message handler 102, which is configured to include data for processing demultiplexing-based operations, such as requesting media samples for each demultiplexed independent coded media stream 74. That is, the client computer device 12 can request new media samples for a specific coded media content stream via the demultiplexer stream message handler 102. As described above, the client computer device 12 can request media content at an independent rate for each demultiplexed media content stream. For example, the client computer device 12 can request media samples for video media content at a higher rate than for media samples for audio media content.

[0055] The media layer 88 includes a media renderer 104 that can send and receive messages 104 via the remote transceiver 92 of the communication layer 86. Using the messages 94, the media renderer 104 can request media samples for the encoded media content stream 74, such as media samples for encoded video content, media samples for encoded audio content, and media samples for encoded text content. The media renderer 104 can then decode the received media samples and render the decoded media samples for presentation to the user via an output device (such as a display, speaker, etc.) of the client computer device 12. It should be understood that the media renderer 104 can use any suitable rendering technology for the platform of the client computer device 12.

[0056] In the manner discussed above, the encoded media content can be redirected to the client computer device 12 so that the media content is not decoded or rendered on the host virtual machine 34. Instead, the encoded media content is redirected and streamed to the client computer device 12, which is configured to decode and render the media content for display. Therefore, by performing MMR on the host virtual machine and avoiding the decoding / rendering process, the processing workload on the hardware running the host virtual machine can be reduced.

[0057] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0058] Figure 7 A non-limiting embodiment of a computing system 700 is schematically shown, and the computing system 700 can implement one or more of the above methods and processes. The computing system 700 is shown in simplified form. The computing system 700 can embody the above-described and Figure 1 and Figure 21 and the server device of the server system 14. The computing system 700 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices, as well as wearable computing devices such as smart watches and head-mounted augmented reality devices.

[0059] The computing system 700 includes a logic processor 702, a volatile memory 704, and a non-volatile storage device 706. The computing system 700 may optionally include a display subsystem 708, an input subsystem 710, a communication subsystem 712, and / or Figure 7 Other components not shown.

[0060] Logical processor 702 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, implement technical effects, or otherwise achieve desired results.

[0061] The logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 702 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The individual components of the logical processor may optionally be distributed between two or more separate devices, which may be remotely located and / or configured for coordinated processing. Various aspects of the logical processor may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration. In this case, it is understandable that these virtualized aspects are run on different physical logical processors of various different machines.

[0062] The non-volatile storage device 706 includes one or more physical devices configured to store instructions executable by a logical processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 706 can be transformed—for example, to store different data.

[0063] The non-volatile storage device 706 may include removable and / or built-in physical devices. The non-volatile storage device 706 may include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor storage (e.g., ROM, EPROM, EEPROM, flash memory, etc.) and / or magnetic storage (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.) or other mass storage device technology. The non-volatile storage device 706 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable and / or content addressable device. It should be understood that the non-volatile storage device 706 is configured to save instructions even when the power to the non-volatile storage device 706 is cut off.

[0064] The volatile memory 704 may include a physical device that includes a random access memory. The volatile memory 704 is typically used by the logical processor 702 to temporarily store information during the processing of software instructions. It should be understood that when the power to the volatile memory 704 is cut off, the volatile memory 704 typically does not continue to store instructions.

[0065] Aspects of the logic processor 702, volatile memory 704, and non-volatile storage device 706 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0066] The terms "module," "program," and "engine" may be used to describe an aspect of a computing system 700 that is typically implemented in software by a processor to perform a specific function using a portion of volatile memory that involves a transformation process that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated via a logical processor 702 executing instructions stored by a non-volatile storage device 706 using a portion of volatile memory 704. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" may include a single or a group of executable files, data files, libraries, drivers, scripts, database records, etc.

[0067] When included, the display subsystem 708 can be used to present a visual representation of the data stored by the non-volatile storage device 706. The visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 708 can also be transformed to visually represent the changes in the underlying data. The display subsystem 708 can include one or more display devices using almost any type of technology. Such a display device can be combined with the logical processor 702, the volatile memory 704 and / or the non-volatile storage device 706 in a shared shell, or such a display device can be a peripheral display device.

[0068] When included, the input subsystem 710 may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be handled on-board or off-board. Example NUI components may include a microphone for speech and / or voice recognition; an infrared camera, color camera, stereo camera, and / or depth camera for machine vision and / or gesture recognition; a head tracker, eye tracker, accelerometer, and / or gyroscope for motion detection and / or intent recognition; and an electric field sensing component for assessing brain activity; and / or any other suitable sensor.

[0069] When included, the communication subsystem 712 can be configured to communicatively couple the various computing devices described herein to each other and to communicate with other devices. The communication subsystem 712 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via a wireless telephone network, or a wired or wireless local area network or wide area network (such as an HDMI connected via Wi-Fi). In some embodiments, the communication subsystem may allow the computing system 700 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.

[0070] The following paragraphs provide additional support for the claims of the subject application. One aspect provides a computer system comprising at least one processor configured to execute a host virtual machine configured to host a session with at least one client computer device. The host virtual machine includes an operating system having a multimedia framework for drawing media content. At least one processor is also configured to execute a media application configured to access media content from a media source, process encoded media content from the media application using the multimedia framework, and execute a multimedia redirection module configured to intercept the encoded media content so as not to be processed by a decoding module of the multimedia framework. The multimedia redirection module is configured to redirect the encoded media content to at least one client computer device. In this aspect, additionally or alternatively, the encoded media content can be dynamic streaming media content. In this aspect, additionally or alternatively, the encoded media content can be received in a media container format including one or more of the following: a video stream, an audio stream, and a text stream, and at least one processor can be configured to demultiplex the encoded media content in the media container format into independent encoded media content streams. In this aspect, additionally or alternatively, the independent encoded media content streams may include one or more of the following: a video stream, an audio stream, and a text stream. In this aspect, additionally or alternatively, the multimedia redirection module may be configured to buffer the independent encoded media content streams, and in response to receiving a request for media samples of the buffered independent encoded media content streams from at least one client computer device, send the media samples to at least one client computer device. In this aspect, additionally or alternatively, the multimedia redirection module may be configured to: send media samples at an independent rate based on a request for media samples for each independent encoded media content stream in the independent encoded media content streams received from at least one client computer device. In this aspect, additionally or alternatively, at least one processor may be configured to: determine whether to execute the multimedia redirection module to intercept the encoded media content based on one or more monitoring parameters, or to allow the encoded media content to be processed by the decoding module of the multimedia framework. In this aspect, additionally or alternatively, the one or more monitoring parameters may be selected from the group consisting of: network parameters between the computer system and at least one client computer device, a current processing load of at least one processor of the computer system, a size of the encoded media content, and a user setting of at least one client computer device. In this aspect, additionally or alternatively, the operating system may be a multi-session operating system configured to concurrently host multiple sessions for multiple client computer devices. In this aspect, additionally or alternatively, the at least one processor may be configured to determine whether to execute the multimedia redirection module based on aggregated activity of multiple sessions hosted by the multi-session operating system.

[0071] On the other hand, a method is provided, which includes executing a host virtual machine at a processor of a server device, and the host virtual machine is configured to host a session with at least one client computer device. The host virtual machine includes an operating system having a multimedia framework for drawing media content. The method also includes executing a media application configured to access media content from a media source, using the multimedia framework to process the encoded media content from the media application, and executing a multimedia redirection module, which is configured to intercept the encoded media content so as not to be processed by the decoding module of the multimedia framework. The multimedia redirection module is configured to redirect the encoded media content to at least one client computer device. In this aspect, additionally or alternatively, the encoded media content can be dynamic streaming media content. In this aspect, additionally or alternatively, the encoded media content can be received in a media container format including one or more of the following: video stream, audio stream and text stream, and the method can also include demultiplexing the encoded media content of the media container format into an independent encoded media content stream. In this aspect, additionally or alternatively, the independent encoded media content stream can include one or more of the following: video stream, audio stream and text stream. In this aspect, additionally or alternatively, the method may also include buffering independent encoded media content streams, and in response to receiving a request for media samples of the buffered independent encoded media content streams from at least one client computer device, sending the media samples to at least one client computer device. In this aspect, additionally or alternatively, the method may also include sending the media samples at an independent rate based on a request for media samples of each independent encoded media content stream in the independent encoded media content streams received from at least one client computer device. In this aspect, additionally or alternatively, the method may also include determining whether to execute a multimedia redirection module to intercept the encoded media content or to allow the encoded media content to be processed by a decoding module of the multimedia framework based on one or more monitoring parameters. In this aspect, additionally or alternatively, one or more monitoring parameters may be selected from a group comprising the following: network parameters between a server device and at least one client computer device, a current processing load of at least one processor of a server device, a size of the encoded media content, and a user setting of at least one client computer device. In this aspect, additionally or alternatively, the operating system may be a multi-session operating system configured to concurrently host multiple sessions for multiple client computer devices.

[0072] On the other hand, a server system is provided, which includes at least one processor configured to execute a host virtual machine, the host virtual machine implements a multi-session operating system, the multi-session operating system is configured to concurrently host multiple sessions for multiple client computer devices, and the multi-session operating system has a multimedia framework for drawing media content. At least one processor is also configured to execute an instance of a media application for one or more of the multiple sessions, and the media application is configured to access media content from a media source. The processor is also configured to determine that the aggregate activity of the multiple sessions of the host virtual machine is higher than a threshold level, enable multimedia redirection of one or more of the multiple sessions based on the determined aggregate activity, use the multimedia framework to process the encoded media content from the instance of the media application, and execute a multimedia redirection module, which is configured to intercept the encoded media content so that it is not processed by the decoding module of the multimedia framework. The multimedia redirection module is configured to redirect the encoded media content to the corresponding client computer device of one or more sessions.

[0073] It should be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered to have limiting significance, because many variations are possible. The specific routines or methods described herein can represent one or more of any number of processing strategies. Like this, the various actions shown and / or described can be performed in the order shown and / or described, performed in other orders, performed in parallel, or omitted. Likewise, the order of the above-mentioned processing can be changed.

[0074] The subject matter of the present disclosure includes all novel and nonobvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computer system comprising: at least one processor configured to execute a host virtual machine configured to host a session with at least one client computer device, the host virtual machine including an operating system having a multimedia framework for decoding and rendering media content; The at least one processor is further configured to: Accessing media encoded content from a media source, wherein the encoded media content is formatted into a media container format, the media container format comprising two or more of: a video stream, an audio stream, a text stream, or a combination thereof; Demultiplexing, by the multimedia framework, the encoded media content formatted into the media container format into independent streams; preventing the independent stream from being decoded by the decoder of the multimedia framework by intercepting the independent stream before the independent stream is provided to the decoder of the multimedia framework; buffering, by the multimedia framework, portions of the independent stream; as well as Media samples of the portion of the independent stream are sent by the multimedia framework to the at least one client computer device.

2. The computer system of claim 1, wherein the encoded media content is dynamic streaming media content.

3. The computer system of claim 1, wherein the media samples of the portion of the independent stream are sent to the at least one client computer device in response to receiving a request for the media samples from the at least one client computer device.

4. The computer system of claim 3, wherein the media samples for each of the independent streams are sent at independent rates based on the requests for the media samples received from the at least one client computer device.

5. The computer system of claim 1, wherein the at least one processor is configured to determine whether to intercept the independent stream or allow the independent stream to be decoded by the decoder of the multimedia framework based on one or more monitoring parameters.

6. The computer system of claim 5, wherein the one or more monitoring parameters are selected from the group consisting of: network parameters between the computer system and the at least one client computer device, a current processing load of the at least one processor of the computer system, a size of the encoded media content, or a user setting of the at least one client computer device.

7. The computer system of claim 5, wherein the operating system is a multi-session operating system configured to concurrently host multiple sessions for multiple client computer devices.

8. The computer system of claim 7, wherein the one or more monitoring parameters include aggregated activity of the plurality of sessions hosted by the multi-session operating system.

9. A method comprising: executing, by a processor, a host virtual machine configured to host a session with at least one client computer device, the host virtual machine including an operating system having a multimedia framework for decoding and rendering media content; executing, by the processor, a media application configured to access encoded media content from a media source; A determination is made by the processor based on one or more monitoring parameters whether to redirect processing of the encoded media content from the multimedia framework to the at least one client computer device or to allow the encoded media content to be processed by a decoder of the multimedia framework.

10. The method of claim 9, wherein the encoded media content is dynamic streaming media content.

11. The method of claim 9, wherein the encoded media content is received in a media container format, the media container format comprising a video stream, an audio stream, a text stream, or a combination thereof, and wherein the method further comprises demultiplexing the encoded media content in the media container format into independent streams.

12. The method of claim 11, wherein redirecting the processing of the encoded media content from the multimedia framework to the at least one client computer device comprises: buffering portions of the independent streams; as well as Media samples of the portion of the independent stream are sent to the at least one client computer device.

13. The method according to claim 12, further comprising: Media samples for each of the independent streams of the encoded media content are sent at an independent rate based on a request for the media samples received from the at least one client computer device.

14. The method of claim 9, wherein the one or more monitoring parameters are selected from the group consisting of: network parameters between a server device on which the processor is configured and the at least one client computer device, a current processing load of the processor, a size of the encoded media content, or user settings of the at least one client computer device.

15. The method of claim 9, wherein the operating system is a multi-session operating system configured to concurrently host multiple sessions for multiple client computer devices.

16. The method of claim 15, wherein the one or more monitoring parameters include aggregated activity of the plurality of sessions hosted by the multi-session operating system.

17. A server system comprising: at least one processor configured to execute a host virtual machine implementing a multi-session operating system configured to concurrently host multiple sessions for multiple client computer devices, the multi-session operating system having a multimedia framework for decoding and rendering media content; The at least one processor is further configured to: executing a media application for a session of the plurality of sessions, the media application configured to access encoded media content from a media source; determining that an aggregate activity of the plurality of sessions for the host virtual machine is above a threshold level; Multimedia redirection for the session is enabled based on the aggregate activity by preventing the encoded media content from being processed by a decoder of the multimedia framework and by redirecting the encoded media content to an associated client computer device of the plurality of client computer devices.

18. The server system of claim 17, wherein the encoded media content is formatted into a media container format, the media container format comprising a video stream, an audio stream, a text stream, or a combination thereof, and wherein the at least one processor is configured to demultiplex the encoded media content in the media container format into independent streams.

19. The server system of claim 18, wherein the at least one processor is further configured to: buffering portions of the independent streams; and Media samples of the portion of the independent stream are sent to the associated client computer device.

20. The server system of claim 19, wherein the media samples for each of the independent streams are sent at an independent rate based on requests for the media samples received from the associated client computer devices.

Images