Computer system, method for redirecting media content, and server system

Abstract

The embodiment of the invention relates to a computer system, a method for redirecting media content and a server system. A computer system comprising at least one processor configured to execute a hosted virtual machine configured to hosting a session with at least one client computer device is provided. The at least one processor is further configured to execute a Web browser application configured to access media content from a remote media source, receive encoded media content from the remote media source in a media container format, and execute a multimedia redirection module, the multimedia redirection module is configured to intercept the encoded media content to be processed by a decoding module of the Web browser application. The multimedia redirection module is configured to redirect the encoded media content to at least one client computer device.

Description

[0001] Divisional Application Instructions

[0002] This application is a divisional application of the Chinese invention patent application with an international application date of November 26, 2020, which entered the Chinese national stage on July 1, 2022, with national application number 202080091683.X and named “Computer system, method and server system for redirecting media content”. Technical Field

[0003] Embodiments of the present application relate to a computer system, a method for redirecting media content, and a server system. Background Art

[0004] In a virtual environment, end users can use client devices to connect to virtual machines running on server hardware, which can provide the following benefits to the end users: additional computing power, application compatibility, security and compliance, and reduce the overall cost to the end user. Typically, these end users may consume media such as video streaming, which can increase the workload on the server hardware and increase the associated costs. Summary of the invention

[0005] A computer system is provided, which may include at least one processor configured to: execute a hosted virtual machine configured to host a session with at least one client computer device. The at least one processor may also be configured to execute a web browser application, the web browser application configured to access media content from a remote media source, receive encoded media content from the remote media source in a media container format, and execute a multimedia redirection module, the multimedia redirection module configured to intercept the encoded media content from being processed by a decoding module of the web browser application. The multimedia redirection module may be configured to redirect the encoded media content to at least one client computer device.

[0006] This summary is provided to introduce in simplified form some of the concepts that will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that address any or all of the disadvantages mentioned in any part of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic diagram of an example computer system for performing multimedia redirection of the present specification.

[0008] Figure 2 yes Figure 1A schematic diagram of an example server system of a computer system.

[0009] Figure 3 It is to realize the multimedia redirection module Figure 1 A schematic diagram of a web browser application of a computer system shimming a rendering pipeline of the web browser application.

[0010] Figure 4 Is Figure 1 A flowchart of a method for performing multimedia redirection implemented by a computer system.

[0011] Figure 5 Is to achieve Figure 1 Schematic diagram of the software architecture of a multimedia redirection web browser application of a computer system.

[0012] Figure 6 is and by Figure 1 A schematic diagram of the software architecture of a remote desktop client communicating with a multimedia redirection module implemented by a computer system.

[0013] Figure 7 yes Figure 1 A schematic diagram of an example computing environment in which a computer system may be implemented. DETAILED DESCRIPTION

[0014] Cloud platform services can provide virtualized desktop environments for end users, who can use client computer devices to connect to virtual machines running on server devices of the cloud platform. Typically, the computer hardware and network capabilities of the cloud platform are more powerful than the client computer devices of the end users. Therefore, by hosting these end users on a virtualized desktop environment, the cloud platform can provide these users with additional computing power, as well as other potential benefits, such as improved application compatibility, security / compliance, and an overall reduction in associated computing costs.

[0015] In some examples, these cloud platform services can provide virtual desktop infrastructure for enterprises with multiple end users and client devices. Users of these enterprises can consume media, such as, for example, video and / or audio media stored locally on the cloud platform for the enterprise or stored by a remote media source and streamed to a virtualized desktop environment running on the cloud platform. Typically, the media content provided by the media source is decoded on the virtual machine hosting the end user, and then the frame for the virtualized desktop is displayed, and the virtualized desktop presents that the media content is re-encoded and streamed to the user's client computer device via the remote desktop protocol. The client computer device can then decode these display frames and present them to the user via the display of the client computer device.

[0016] The decoding and subsequent encoding processes performed on a virtual machine can consume significant computing resources of the server hardware. In one specific example, the video rendering process used for video streaming can potentially consume 20%-40% of the central processing unit (CPU) resources of the server device running the virtual machine. These problems can be further exacerbated in multi-session virtualized desktop environment deployments where multiple users share an operating system (OS) instance on a virtual machine. For example, if more than one user attempts to stream video content through a shared hosted virtual machine, all CPU resources for that virtual machine can potentially be consumed by the associated decoding / encoding / rendering processes, potentially reducing the user experience of all users hosted by that virtual machine.

[0017] In order to solve the above problems, Figure 1 An example computer system 10 is shown for a virtualized desktop environment that performs multimedia redirection to reduce the computing load on the server hardware running the virtualized desktop environment, which will be discussed in more detail below. Figure 1 As shown, computer system 10 includes one or more client computer devices 12, server system 14, and one or more media sources 16. End users of one or more client computer devices 12 can connect to a virtual machine running on server system 14. Using an application (e.g., a Web browser application) executed within a hosted operating system of the virtual machine, the end user can interact with media stored by media source 16.

[0018] In one example, media source 16 may be a local source of server system 14 and may be included in server system 14. In another example, media source 16 may be remote from client computer device 12 and server system 14. For example, server system 14 may be configured to communicate with a remote media source via a communication network such as a wide area network (WAN). As a specific example, an end user may enter input into a Web browser application executed within a hosted virtual machine to navigate to a Web site associated with media source 16, such as a video streaming Web site. Using the associated Web site, the end user may initiate an experience that includes streaming video and / or audio. Typically, streaming video and / or audio is sent to server system 14 as encoded media 18 and processed by a hosted virtual machine executing a Web browser application that requested the streaming video and / or audio.

[0019] As described above, a typical virtualized environment implementation will decode the encoded media 18, present the media content via the virtualized desktop environment, encode the display / audio frames for the virtualized desktop environment, and then send the encoded display / audio frames to the client computer device 12 for presentation to the end user. In order to reduce the computational burden imposed on the hardware on the server system 14 caused by the decoding / encoding process, the server system 14 can implement multimedia redirection to pass the encoded media 18 received from the media source 16 to the client computer device 12 without performing the encoding or decoding process on the server system 14.

[0020] Figure 2 An example of a server system 14 is shown. The server system 14 may include a hardware plane 20, a virtual machine plane 22, a virtual machine monitor (hypervisor) plane 24, and a network infrastructure 26, which are collectively configured to operate a cloud platform. The hardware plane 20 includes a collection of nodes 28 (each node is composed of Figure 2 The node 28 may include a processor, a graphics processing unit (GPU), a volatile memory, and other computer components configured to run a host server instance. The host server instance executed by the node 28 of the hardware plane 20 is configured to communicate with one or more virtual machine monitors of the virtual machine monitor plane 24. The one or more virtual machine monitors of the virtual machine monitor plane 24 can create, handle, and monitor multiple virtual machines 30 of the virtual machine plane 22 (each in Figure 2 Through the virtual machine monitor plane 24, each virtual machine 30 of the virtual machine plane 22 can be hosted and run by hardware components of one or more nodes 28 of the hardware plane 20. In this way, multiple virtual machines 30 of the virtual machine plane 22 can share virtualized hardware resources managed by the virtual machine monitor plane 24. Each virtual machine 30 provides a virtualized desktop environment in which software such as a web browser application, a media player application, and other types of software can be executed.

[0021] In one example, the computer system 14 corresponds to a data center environment that communicatively couples multiple nodes 28 via a standard network infrastructure. For example, the network infrastructure 26 may include a typical network infrastructure, such as a server rack including a top-of-rack (TOR) network switch. The computer system 14 may include multiple node clusters, each node cluster having an associated TOR network switch. The network infrastructure 26 may also include a higher-level switching infrastructure 32 (L1) and (L2) that connects the TOR network switches together. The higher-level switching infrastructure 32 can take the form of any suitable networking architecture and can be driven by any suitable (multiple) routing protocols. In the example shown, the higher-level infrastructure 32 includes a collection of aggregation switches L1 and core switches L2. However, it should be understood that the higher-level switching infrastructure may include any appropriate number of levels of switches.

[0022] The virtual machines 30 of the virtual machine plane 22 provide a virtual computing environment within which users of the server system 14 can access and run applications such as, for example, a web browser, a media player, etc. As will be discussed in more detail below, these virtual machines 30 of the virtual machine plane 22 can be configured to implement multimedia redirection to reduce the computational workload imposed on the nodes of the hardware plane 20 caused by the decoding and encoding processes typically performed for streaming media.

[0023] In addition, it should be understood that the server system 14 is not limited to Figure 2 The data center / cloud platform example shown. In some other examples, the server system 14 can take the form of one or more computer devices configured to run a virtualized desktop environment. Using the client computer device 12, an associated user can connect to the virtualized desktop environment run by one or more computer devices to access software running within the virtualized desktop environment, which can be configured to implement the multimedia redirection functions and processes of this article.

[0024] Figure 3 An example hosted virtual machine 34 is shown that is configured to perform multimedia redirection technology to solve the above problems. The hosted virtual machine 34 can take the reference Figure 2 The hosted virtual machine 34 is in the form of one of the virtual machines 30 of the virtual machine plane 22 of the cloud platform. However, it should be understood that the hosted virtual machine 34 can also be implemented in a non-cloud platform form of the server system 14.

[0025] The hosted virtual machine 34 may be executed by at least one processor of the server system 14. The hosted virtual machine 34 may be configured to host a session with at least one client computer device 12, and may host applications running within the operating system of the hosted virtual machine 34. In one example, the hosted virtual machine 34 may include a Web browser application 36 executed by at least one processor of the server system 14. As shown, the Web browser application 36 may be configured to access media content from a media source 16. Using a graphical user interface (GUI) of the Web browser application 36, a user may navigate to a target media source 16, such as a Web page and an associated server serving media content. In one example, the media source 16 may be a local source of the server system 14 running the hosted virtual machine 34. In another example, the media source 16 may be remote from the server system 14, and may be configured to send the encoded media content 18 to the server system 14 via a WAN.

[0026] The hosted virtual machine 34 may be configured to receive encoded media content 18 from a media source 16, which in the illustrated example may take the form of a remote media source. The encoded media content 18 may be encoded using any suitable algorithm. In one example, the encoded media content 18 is received in a media container format 38 that includes one or more of a video stream 40, an audio stream 42, and a text stream 44. As a specific example, the media container format 38 may be an MP4 format, a WAV format, an AVI format, or another type of media container format 38.

[0027] The encoded media content 18 received by the hosted VM 34 may be processed by a media source extension (MSE) module 46 of a web browser application 36 running on the hosted VM 34. The MSE module 46 may be configured to provide functionality for web-based media streaming. The MSE module 46 may also be configured to demultiplex the encoded media content 18 in the media container format 38 into separate video streams, audio streams, and text streams. In general, as shown in FIG. Figure 3 As shown, the independent encoded audio stream 42 and the encoded video stream 40 will be processed by the audio and video rendering pipelines of the web browser application 36. For example, the encoded audio stream 42 and the encoded video stream 40 can be processed by a decoder 48 that decompresses the audio stream and the video stream, respectively. The decompressed audio stream and the video stream can then be passed to an audio renderer 50 and a video renderer 52, respectively, to prepare the video content and the audio content for presentation. As described above, these decoding processes and rendering processes executed on the hosted VM 34 will consume hardware resources of the server device running the hosted VM 34, which may potentially reduce the user experience of other users hosted in the multi-session operating system of the hosted virtual machine 34.

[0028] To address these issues, the hosted virtual machine 34 may be configured to implement a multimedia redirection module 54 that is configured to redirect the encoded media content 18 , including the encoded audio stream 42 and the encoded video stream 40 , to the client computer device 12 . Figure 4 is a flow chart of a method 400 implemented by a server system 14 running a hosted virtual machine 34 to perform multimedia redirection. The method 400 may be performed using the system described above or using other suitable hardware and software elements.

[0029] At 402, method 400 may include executing a hosted virtual machine configured to host a session with at least one client computer device. Hosted virtual machine 34 may be executed by at least one processor of server system 14, which may take the form of, for example, Figure 2 In one example, the hosted virtual machine 34 may implement a multi-session operating system that is configured to concurrently host multiple sessions for multiple client computer devices 12. Multiple sessions may be hosted in the same multi-session operating system instance. Each session may implement the techniques and processes described herein for multimedia redirection.

[0030] At 404, method 400 may include executing a web browser application configured to access media content from a remote media source. Web browser application 36 may run within hosted virtual machine 34. In one multi-session operating system example, each of a plurality of sessions may have an associated web browser application 36 instance running in the multi-session operating system. Remote media source 16 may take the form of a third-party media service that streams media content such as video and / or audio. Web browser application 36 may be configured to access remote media source 16 based on user web navigation input such as, for example, a URL address for remote media source 16.

[0031] At 406, method 400 may include receiving encoded media content in a media container format from a remote media source. The encoded media content 18 may be received by the hosted virtual machine 34 over the WAN. Figure 3 As shown, the encoded media content 18 may include one or more of an encoded video stream 40, an encoded audio stream 42, and an encoded text stream 44. The encoded streams of media content may be bundled in a media container format 38, such as MP4 format, WAV format, etc.

[0032] At 408, the method 400 may include demultiplexing the encoded media content in the media container format into independent streams of encoded media content. For example, the web browser application 36 may be configured to implement the MSE module 46, which includes processes for disassembling the media container format 38 and demultiplexing the encoded media content 18 into independent streams (such as, for example, the encoded video stream 40, the encoded audio stream 42, and / or the encoded text stream 44).

[0033] In 410, method 400 may include executing a multimedia redirection module, which is configured to intercept the decoder module processing of the coded media content by the Web browser application. The multimedia redirection module 54 can take the form of separation, and it intercepts the application programming interface (API) call and changes the independent variables transmitted and the system behavior applied to these independent variables. For example, the multimedia redirection module 54 can intercept the API call that will pass the coded audio / video / text stream to the corresponding decoder and renderer, and redirect the coded content to the associated client computer device 12 for the session. In this way, the coded media content 18 is not decoded by the decoder 48 on the managed virtual machine 34.

[0034] The encoded media content 18 redirected to the client computer device 12 can be buffered and streamed to the client computer device 12 via the WAN. In 412, the method 400 can include buffering the separate streams of the encoded media content. The separate streams of the encoded media content (such as the encoded audio stream 42 and the encoded video stream 40) can be temporarily stored in a buffer on the hosted virtual machine 34.

[0035] In 414, method 400 may include sending media samples of the buffered independent streams of the encoded media content to at least one client computer device in response to receiving a request for the media samples from at least one client computer device. The client computer device 12 may independently request new media samples for each stream in the encoded media content 18. For example, the client computer device 12 may independently request media samples for the encoded video stream 40 from requesting media samples for the encoded audio stream 42. In this way, the client computer device 12 may request media samples for each independent stream of the encoded media content (e.g., the encoded audio stream, the encoded video stream, etc.) at an independent rate, and the hosted virtual machine 34 may send the media samples for each independent stream of the encoded media content at the requested rate. Typically, the size of video content is larger than audio content, and therefore a higher rate may be required to send video samples to the client computer device 12.

[0036] Figure 5An example server-side software architecture for implementing the multimedia redirection module 54 and the steps described in the method 400 is shown. In the example shown, at least one processor 56 of a server device included in the server system 14 can be configured to execute a hosted virtual machine 34 that is configured to host a session 58 with at least one client computer device 12. In the example shown, the hosted virtual machine 34 implements a multi-session operating system 60 that is configured to concurrently host multiple sessions 58 for multiple client computer devices 12. Each session 58 can be associated with a different client computer device 12 and can run a separate instance of an application on the hosted virtual machine 34. For example, each session 58 can execute a separate instance of a web browser application 36 configured to access media content from a remote media source 16.

[0037] Each session 58 running within the multi-session operating system 60 may also be configured to receive encoded media content 18 in a media container format 38 from a remote media source 16, as described above with reference to FIG. Figure 3 and Figure 4 The content described. In one example, the encoded media content 18 can take the form of dynamic streaming media content. In another example, the encoded media content 18 can take the form of HTTP progressive streaming content. However, it should be understood that the encoded media content 18 can take other suitable forms. In addition, the encoded media content 18 can be received in a media container format including one or more of a video stream, an audio stream and a text stream. The video stream, the audio stream and the text stream can be multiplexed in a media container format. It should be understood that the media container format can take the form of MP4 format, WAV format or any other suitable type of media container format.

[0038] Execution of the web browser application 36 may launch multiple associated processes running within the multi-session operating system 60. As a specific example, a web browser application process 62 may be running for a session 58 with a client computer device 58. Additionally, a separate renderer process 64 may be running for each tab opened in the web browser application 36. These separate renderer processes 64 may be sandboxed and run at a lower integrity level than the web browser application process 62.

[0039] Each renderer process 64 may include a demultiplexing module 66, a renderer client 68, and a messenger renderer 70. The demultiplexing module 66 may be configured to demultiplex the encoded media content 18 of the media container format 38 into independent streams of encoded media content 72. The demultiplexed independent streams 72 of encoded media content may include, for example, an encoded video stream, an encoded audio stream, and / or an encoded text stream.

[0040] The messenger renderer 70 forms part of the MMR module 54 and may be configured to request and receive compressed media samples from the demultiplexed independent streams 72 of the encoded media content. In one example, the messenger renderer 70 may also be configured to notify the renderer client 68 of changes in the playback state 74 of the encoded media content 18 and the playback time.

[0041] In one example, if multimedia redirection (MMR) is enabled, the messenger renderer 70 is run in the renderer process 64. If MMR is not enabled concurrently for the received encoded media content 18 at present, the renderer process 64 can be configured to run a media renderer to replace the messenger renderer 70, and the messenger renderer 70 can be configured to handle the decoding and rendering of the encoded media content via a decoder 48, an audio renderer 50 and a video renderer 52. When MMR is enabled, the messenger renderer 70 is loaded instead of a typical media renderer, and is configured to intercept the demultiplexed stream of media content and perform the functions and processes described herein. In this way, the demultiplexed stream of media content is not decoded and rendered on the hosted virtual machine 34.

[0042] The MMR module 54 may also include a remote desktop connection (RDC) connector module 76. The messenger renderer 70 may also be configured to communicate with the RDC remote connector module 76 across processes. In one example, the RDC remote connector module 76 may run within the process of the web browser application process 62. In another example, the RDC remote connector module 76 may be executed within another process separate from the web browser application process 62. Figure 5 As shown, the messenger renderer 70 may include a renderer controller 78 configured to handle communications with the RDC remote connector module 76 via sending and receiving serialized messages 80. As a specific example, the serialized message 80 may include using an extension such as Protocol buffers use the protocol to serialize commands, extending Protocol buffers are a language and platform neutral mechanism for serializing and deserializing structured data. However, it should be understood that other serialization protocols may also be implemented.

[0043] The messenger renderer 70 may be configured to send demultiplexed media samples for the demultiplexed independent stream 72 of the encoded media content to the RDC remote connector module 76. The messenger renderer 70 may also be configured to convert the renderer commands into serialized messages 80, which are also sent to the RDC remote connector module 76. In addition, the messenger renderer 70 may be configured to convert the serialized messages 80 received from the RDC remote connector module 76 into callbacks on the renderer client 68.

[0044] As described above, each tab opened in an instance of the Web browser application may have an associated renderer process 64. An instance of the messenger renderer 70 for each of these instances of the renderer process 64 may communicate with a single RDC remote connector module 76 for the instance of the Web browser application being executed with the session 58 of the client computer device 12. The RDC remote connector module 76 may include a remote bridge 82 for each messenger renderer 70 that is configured to exchange serialized messages with the associated messenger renderer 70. In one example, the RDC remote connector module 76 may be configured to act as a multiplexer that allows multiple messenger renderers to be remote to the client computer device 12 concurrently.

[0045] like Figure 5 As shown, the RDC remote connector module 76 may also include an RDC media remote core 84, which includes logic for executing the functions and processes of the RDC remote connector module 76. The RDC media remote core 84 may also be configured to handle the transmission of the demultiplexed individual streams 72 of the encoded media data received at the remote bridge 82 to a dynamic virtual channel (DVC) plug-in implemented on the client computer device 12. The RDC remote connector module 76 may also handle the communication of the serialized message 80 with the DVC plug-in on the client computer device 12.

[0046] As described above, the MMR module 54 may include an RDC remote connector module 76 and a messenger renderer 70. Using the above process, the MMR module 54 can be configured to intercept the processing of the decoder 48 of the encoded media content 18 by the Web browser application 36. For example, when MMR is enabled, the messenger renderer 70 can replace the typical media renderer execution within the renderer process 64. That is, the MMR module 54 can cause the media samples of the demultiplexed independent streams (e.g., video stream / audio stream / text stream) from the encoded media content to be sent to the messenger renderer 70, which is configured to send these media samples to the associated remote bridge 82 of the RDC remote connector module 76. Using the functions described for the RDC remote connector module 76, the MMR module 54 can send those media samples to the client computer device 12 via the WAN. In this way, the encoded media content is redirected to at least one client computer device 12 by the MMR module 54 so that the encoded media content is not decoded or rendered on the hosted virtual machine 34.

[0047] Additionally, the MMR module 54 may be configured to buffer the independent streams 72 of the encoded media content and, in response to receiving a request for the media samples from at least one client computer device 12, send the media samples of the buffered independent streams 72 of the encoded media content to at least one client computer device 12. For example, the client computer device 12 may send a serialized message 80 to the RDC remote connector module 76, the serialized message 80 including a request 88 for media samples of at least one of the streams of the encoded media content. The RDC remote connector module 76 may request the messenger renderer 70 to send the requested media samples to the associated remote bridge 82. The RDC remote connector module 76 may then send the requested encoded media samples 86 to the client computer device 12 via a dynamic virtual channel. In one example, the client computer device 12 may send a separate request 88 for each independent stream 72 of the encoded media content. For example, the client computer device 12 may request media samples of the encoded video content stream at a higher rate than media samples of the encoded audio content. Thus, the MMR module 54 may be configured to send media samples 86 at an independent rate for each of the independent streams 72 of encoded media content based on a request 88 for media samples received from at least one client computer device 12 .

[0048] As described above, when MMR is enabled for the encoded media content 18, the MMR module 54, which may include the RDC remote connector module 76 and the messenger renderer 70, is loaded and executed. In one example, the at least one processor 56 may be configured to determine whether to enable MMR based on one or more monitored parameters 90, and whether to execute the MMR module 54 to intercept or allow the encoded media content 18 to be processed by the decoder 48 of the web browser application 36. In one example, the hosted virtual machine 34 may be configured to monitor network parameters between the server system 14 and at least one client computer device 12, and determine whether to enable MMR based on the monitored network parameters.

[0049] As another example, the hosted virtual machine 34 can be configured to monitor the current processing load of at least one processor 56 running the hosted virtual machine 34, and determine whether to enable MMR based on the current processing load. As a specific example, if the current processing load on at least one processor 56 exceeds a threshold level, the hosted virtual machine 34 can be configured to enable MMR and load the MMR module 54 and its associated media stack to intercept the processing of the encoded media content 18 by the standard rendering process that performs decoding and rendering. On the other hand, if the current processing load is below the threshold level and the monitored network parameters indicate that the network bandwidth is low, the hosted virtual machine can be configured to disable MMR and decode / render the encoded media content. The rendered frame can then be encoded using a compression level suitable for the network bandwidth between the server system and the client computer device.

[0050] As another example, the hosted virtual machine 34 may be configured to determine whether to enable MMR and load the MMR module 54 based on user settings of at least one client computer device 12. For example, an administrator may control user settings to enable or disable MMR for a client computer device. The user settings may be sent to the hosted virtual machine 34 and stored for the client computer device. In a specific example, MMR may be independently enabled or disabled via user settings of different media sources 16.

[0051] As described above, the hosted virtual machine 34 may implement a multi-session operating system 60 that hosts sessions 58 with multiple client computer devices 12. The activity of each client computer device 12 may increase the processing workload applied to the at least one processor 56 running the hosted virtual machine 34. Thus, in one example, the hosted virtual machine 34 may be configured to determine the aggregate activity of the multiple sessions 58 hosted by the multi-session operating system 60, and determine whether to execute the MMR module 54 based on the aggregate activity. As a specific example, if the aggregate activity of the multiple sessions 58 is low because only one session is currently accessing the encoded media content for streaming, the hosted virtual machine 34 may be configured to not enable MMR and allow the decoding and rendering processes to be executed on the hosted virtual machine 34. On the other hand, if the aggregate activity of the multiple sessions 58 is high because multiple sessions are attempting to access the encoded media content, the hosted virtual machine 34 may be configured to enable MMR and load the MMR module 54.

[0052] The hosted virtual machine 34 may be configured to dynamically enable or disable MMR based on changes to the monitored parameters 90. Additionally, it should be understood that the exemplary monitored parameters discussed above are merely exemplary, and the hosted virtual machine 34 may dynamically determine whether to enable MMR based on other parameters (such as the size of the encoded media content, the source of the encoded media content, etc.).

[0053] Figure 6 An example remote desktop client 92 is shown that is executable by the processor of each client computer device 12 to communicate with the hosted virtual machine 34. The remote desktop client 92 includes an MMR DVC plug-in 94 configured to interact with the MMR module 54 executed by the hosted virtual machine 34. When MMR is enabled by the hosted virtual machine 34, the MMR DVC plug-in 94 can be loaded onto the remote desktop client 92. On the other hand, if MMR is not enabled, the remote desktop client 92 can be configured to use typical remote techniques and processes.

[0054] The MMR DVC plug-in 94 includes a communication layer 96 and a media layer 98. The communication layer 96 is configured to handle communications between the MMR DVC plug-in 94 and the RDC remote connector module 76 of the hosted virtual machine 34. The communication layer 96 may include an MMR virtual channel 100 configured to open a channel with the RDC remote connector module 76 of the hosted virtual machine 34. An instance of the MMR virtual channel 100 can be created for each instance of the Web browser application 36 opened for the session. When a new connection from a hosted virtual machine 34 is accepted, such as, for example, when an instance of the Web browser application 36 on the hosted virtual machine 34 is opened and media content is accessed, a remote transceiver 102 is created.

[0055] The remote transceiver 102 may be configured to send and receive messages 104 to / from the RDC remote connector module 76 hosting the virtual machine 34. As shown, the message 104 is structured to include a master message handler 106, a source message handler 108, a renderer message handler 110, and a demultiplexer stream message handler 112. The master message handler 106 may be configured to include data for negotiating version information between the client and the host.

[0056] The message 104 may include a source message handler 108 for each remote bridge 82 of the RDC remote connector module 76. Each source message handler 108 also includes a corresponding renderer message handler 110 for handling the message using the renderer process 64. The renderer message handler 110 may include data about the playback status of the encoded media content processed by the associated renderer process 64. For example, the renderer message handler 110 may communicate whether the media content should be paused, sought to a target location, etc. The renderer message handler 110 may also be used to report the status of the playback to the hosted virtual machine 34, such as the playback position, whether the playback has ended, or an error in the playback.

[0057] Each renderer message handler 110 also includes a demultiplexer stream message handler 112, which is configured to include a method for handling operation-based demultiplexing, such as, for example, requesting media samples for each demultiplexed independent stream 72 of the encoded media. That is, the client computer device 12 can request new media samples for a specific stream of the encoded media content via the demultiplexer stream message handler 112. As described above, the client computer device 12 can request media content at an independent rate for each demultiplexed stream of the media content. For example, the client computer device 12 can request media samples for video media content at a higher rate than media samples for audio media content.

[0058] The media layer 98 includes a media renderer 114 that can send and receive messages 104 via the remote transceiver 102 of the communication layer 96. Using the messages 104, the media renderer 114 can request media samples for the stream 72 of encoded media content, such as media samples for encoded video content, media samples for encoded audio content, and media samples for encoded text content. The media renderer 114 can then decode the received media samples and render the decoded media samples for presentation to the user via an output device (such as a display, speaker, etc.) of the client computer device 12. It should be understood that the media renderer 114 can use any suitable rendering technology for the platform of the client computer device 12.

[0059] In the manner discussed above, the encoded media content can be redirected to the client computer device 12 so that the media content is not decoded or rendered on the hosted virtual machine 34. Instead, the encoded media content is streamed to the client computer device 12, which is configured to decode and render the media content for display. Therefore, by performing MMR on the hosted virtual machine and avoiding the decoding / rendering process, the processing workload on the hardware running the hosted virtual machine can be reduced.

[0060] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0061] Figure 7 A non-limiting embodiment of a computing system 700 that can perform one or more of the above methods and processes is schematically shown. The computing system 700 is shown in simplified form. The computing system 700 can be described above and Figure 1 and Figure 2The computing system 700 may be implemented on the client computer device 12 and the server device of the server system 14 shown in FIG. The computing system 700 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices, as well as wearable computing devices such as smart watches and head-mounted augmented reality devices.

[0062] The computing system 700 includes a logic processor 702, a volatile memory 704, and a non-volatile storage device 706. The computing system 700 may optionally include a display subsystem 708, an input subsystem 710, a communication subsystem 712, and / or Figure 7 Other components not shown.

[0063] Logical processor 702 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or otherwise achieve desired results.

[0064] The logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or optionally, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 702 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The various components of the logical processor may be selectively distributed in two or more separate devices, which may be remotely located and / or configured for coordinated processing. Various aspects of the logical processor may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration. In this case, it should be understood that these virtualized aspects run on different physical logical processors of various different machines.

[0065] The non-volatile storage device 706 includes one or more physical devices that are configured to store instructions that can be executed by a logical processor to implement the methods and processes herein. When implementing such methods and processes, the state of the non-volatile storage device 706 may be transformed, for example, to store different data.

[0066] The non-volatile storage device 706 may include removable and / or built-in physical devices. The non-volatile storage device 706 may include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor storage (e.g., ROM, EPROM, EEPROM, flash memory, etc.) and / or magnetic storage (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.) or other mass storage device technology. The non-volatile storage device 706 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable and / or content addressable devices. It should be appreciated that the non-volatile storage device 706 can be configured to retain instructions even when power to the non-volatile storage device 706 is removed.

[0067] The volatile memory 704 may include a physical device having random access memory. The volatile memory 704 is typically used by the logical processor 702 to temporarily store information during the processing of software instructions. It should be understood that when power to the volatile memory 704 is cut off, the volatile memory 704 typically does not continue to store instructions.

[0068] Aspects of the logic processor 702, volatile memory 704, and non-volatile storage device 706 may be integrated together into one or more hardware-logic components. Such hardware-logic components may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0069] The terms "module", "program", and "engine" may be used to describe an aspect of a computing system 700 that is typically implemented in software by a processor to use portions of volatile memory to perform a specific function that involves a transformation process that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated via a logical processor 702 that executes instructions stored by a non-volatile storage device 706 using portions of volatile memory 704. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" may include individuals or groups of executable files, data files, libraries, drivers, scripts, database records, and the like.

[0070] When included, the display subsystem 708 can be used to present a visual representation of the data stored by the non-volatile storage device 706. The visual representation can take the form of a graphical user interface (GUI). As the methods and processes described herein change the data held by the non-volatile storage device, and thus transform the state of the non-volatile storage device, the state of the display subsystem 708 can also be transformed to visually represent the changes in the underlying data. The display subsystem 708 can include one or more display devices utilizing virtually any type of technology. Such display devices can be combined with the logical processor 702, the volatile memory 704, and / or the non-volatile storage device 706 in a shared cabinet, or such display devices can be peripheral display devices.

[0071] When input subsystem 710 is included, input subsystem 710 may include or interface with one or more user input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) component portions. Such component portions may be integrated or peripheral, and the conversion and / or processing of input actions may be handled on-board or off-board. Example NUI component portions may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity; and / or any other suitable sensors.

[0072] When included, the communication subsystem 712 can be configured to communicatively couple the various computing devices herein to each other, as well as to communicatively couple to other devices. The communication subsystem 712 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via a wireless telephone network, or a wired or wireless local area network or wide area network (such as HDMI on a Wi-Fi connection). In some embodiments, the communication subsystem can allow the computing system 700 to send messages to other devices and / or receive messages from other devices via networks such as the Internet.

[0073] The following paragraphs provide additional support for the claims of the present application. One aspect provides a computer system including at least one processor, at least one processor is configured to execute a hosted virtual machine, and the hosted virtual machine is configured to host a session with at least one client computer device. At least one processor is also configured to: execute a Web browser application configured to access media content from a remote media source, receive encoded media content in a media container format from the remote media source, and execute a multimedia redirection module, which is configured to intercept the encoded media content from being processed by a decoding module of the Web browser application. The multimedia redirection module is configured to redirect the encoded media content to at least one client computer device. In this aspect, additionally or alternatively, the encoded media content can be dynamic streaming media content. In this aspect, additionally or alternatively, the encoded media content can be received in the following media container format: including one or more of a video stream, an audio stream, and a text stream, and at least one processor can be configured to demultiplex the encoded media content in the media container format into independent streams of the encoded media content. In this aspect, additionally or alternatively, the independent streams of the encoded media content may include one or more of a video stream, an audio stream, and a text stream. In this aspect, additionally or alternatively, the multimedia redirection module may be configured to buffer the independent streams of the encoded media content, and in response to receiving a request for media samples from at least one client computer device, send media samples of the buffered independent streams of the encoded media content to at least one client computer device. In this aspect, additionally or alternatively, the multimedia redirection module may be configured to: based on the request for media samples received from at least one client computer device, send media samples for each independent stream of the independent streams of the encoded media content at an independent rate. In this aspect, additionally or alternatively, at least one processor may be configured to: determine whether to execute the multimedia redirection module to intercept the encoded media content or allow the encoded media content to be processed by the decoding module of the Web browser application based on one or more monitored parameters. In this aspect, additionally or alternatively, one or more monitored parameters may be selected from the group consisting of: network parameters between the computer system and at least one client computer device, a current processing load of at least one processor of the computer system, a size of the encoded media content, and a user setting of at least one client computer device. In this aspect, additionally or alternatively, a hosted virtual machine executed by at least one processor may include a multi-session operating system configured to concurrently host multiple sessions for multiple client computer devices. In this aspect, additionally or alternatively, at least one processor may be configured to determine whether to execute the multimedia redirection module based on aggregated activity of multiple sessions hosted by the multi-session operating system.

[0074] On the other hand, a method is provided, including: at a processor of a server device, executing a hosted virtual machine, the hosted virtual machine is configured to host a session with at least one client computer device; executing a Web browser application configured to access media content from a remote media source; receiving encoded media content in a media container format from a remote media source; and executing a multimedia redirection module, the multimedia redirection module is configured to intercept the encoded media content from being processed by a decoding module of the Web browser application. The multimedia redirection module is configured to redirect the encoded media content to at least one client computer device. In this aspect, additionally or alternatively, the encoded media content can be dynamic streaming media content. In this aspect, additionally or alternatively, the encoded media content can be received in the following media container format: including one or more of a video stream, an audio stream, and a text stream, and the method can also include demultiplexing the encoded media content in the media container format into independent streams of the encoded media content. In this aspect, additionally or alternatively, the independent streams of the encoded media content can include one or more of a video stream, an audio stream, and a text stream. In this aspect, additionally or alternatively, the method may further include buffering independent streams of the encoded media content, and in response to receiving a request for the media sample from at least one client computer device, sending the media sample of the buffered independent stream of the encoded media content to at least one client computer device. In this aspect, additionally or alternatively, the method may further include: based on the request for the media sample received from at least one client computer device, sending the media sample for each independent stream of the encoded media content at an independent rate. In this aspect, additionally or alternatively, the method may further include: determining whether to execute the multimedia redirection module to intercept the encoded media content or allow the encoded media content to be processed by the decoding module of the web browser application based on one or more monitored parameters. In this aspect, additionally or alternatively, the one or more monitored parameters may be selected from the group including: network parameters between the server device and at least one client computer device, the current processing load of at least one processor of the server device, the size of the encoded media content, and user settings of at least one client computer device. In this aspect, additionally or alternatively, the hosted virtual machine may include a multi-session operating system configured to concurrently host multiple sessions for multiple client computer devices.

[0075] Another aspect provides a server system comprising at least one processor configured to execute a hosted virtual machine implementing a multi-session operating system, the multi-session operating system being configured to concurrently host multiple sessions for multiple client computer devices. The at least one processor is also configured to: execute an instance of a web browser application for one or more of the multiple sessions. The web browser application is configured to receive encoded media content from a remote media source. The processor is also configured to determine that the aggregate activity of the multiple sessions for the hosted virtual machine is above a threshold level, enable multimedia redirection for one or more of the multiple sessions based on the determined aggregate activity, and execute a multimedia redirection module, the multimedia redirection module being configured to intercept the encoded media content from being processed by a decoding module of the web browser application. The multimedia redirection module is configured to redirect the encoded media content to the corresponding client computer devices of one or more of the multiple sessions.

[0076] It should be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered as restrictive, because many variations are possible. The specific routine or method described herein can represent one or more of any number of processing strategies. Therefore, the various actions shown and / or can be performed in the order shown and / or, in other orders, in parallel or omitted. Equally, the order of the above process can be changed.

[0077] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computer system comprising: at least one processor configured to execute a hosted virtual machine configured to host a session with at least one client computer device, wherein the at least one processor is further configured to: executing a web browser application configured to access encoded media content from a media source, wherein the encoded media content is formatted in a media container format that includes a video stream, an audio stream, a text stream, or a combination thereof; multiplexing, by the web browser application, the encoded media content formatted in the media container format into independent streams; preventing the independent stream from being decoded by the decoder of the web browser application by intercepting the independent stream before the independent stream is provided to the decoder of the web browser application; buffering, by the web browser application, portions of the independent streams; as well as Media samples of the portion of the independent stream are sent by the web browser application to the at least one client computer device.

2. The computer system of claim 1, wherein the encoded media content is dynamic streaming media content.

3. The computer system of claim 1, wherein the media samples of the portion of the independent stream are sent to the at least one client computer device in response to a request for the media samples received from the at least one client computer device.

4. The computer system of claim 3, wherein the media samples for each of the independent streams are sent at an independent rate based on the request for the media samples received from the at least one client computer device.

5. The computer system of claim 1, wherein the at least one processor is configured to determine whether to intercept the independent stream or allow the independent stream to be decoded by the decoder of the web browser application based on one or more monitored parameters.

6. The computer system of claim 5, wherein the one or more monitored parameters are selected from the group consisting of: network parameters between the computer system and the at least one client computer device, a current processing load of the at least one processor of the computer system, a size of the encoded media content, or user settings of the at least one client computer device.

7. The computer system of claim 5, wherein the hosted virtual machine executed by the at least one processor comprises a multi-session operating system configured to concurrently host multiple sessions for multiple client computer devices.

8. The computer system of claim 7, wherein the one or more monitored parameters include aggregated activity of the plurality of sessions hosted by the multi-session operating system.

9. A method comprising: executing, by a processor, a hosted virtual machine configured to host a session with at least one client computer device; executing, by the processor, a web browser application configured to access media content from a media source; receiving encoded media content in a media container format from the media source; A determination is made by the processor based on one or more monitored parameters whether to redirect processing of the encoded media content from the web browser application to the at least one client computer device or to allow the encoded media content to be processed by the web browser application.

10. The method of claim 9, wherein the encoded media content is dynamic streaming media content.

11. The method of claim 9, wherein the encoded media content is received in the media container format as a video stream, an audio stream, or a text stream, and wherein the method further comprises demultiplexing the encoded media content in the media container format into independent streams.

12. The method of claim 11, wherein redirecting the processing of the encoded media content from the web browser application to the at least one client computer device comprises: buffering portions of the independent streams; as well as Media samples of the portion of the independent stream are sent to the at least one client computer device.

13. The method of claim 12, wherein the media samples for each of the independent streams are sent at an independent rate based on a request for the media samples received from the at least one client computer device.

14. The method of claim 9, wherein the one or more monitored parameters are selected from the group consisting of: network parameters between a server device on which the processor is configured and the at least one client computer device, a current processing load of the processor, a size of the encoded media content, or user settings of the at least one client computer device.

15. The method of claim 9, wherein the hosted virtual machine comprises a multi-session operating system configured to concurrently host multiple sessions for multiple client computer devices.

16. The method of claim 15, wherein the one or more monitored parameters include aggregated activity of the plurality of sessions hosted by the multi-session operating system.

17. A server system comprising: at least one processor configured to execute a hosted virtual machine implementing a multi-session operating system configured to concurrently host multiple sessions for multiple client computer devices, wherein the at least one processor is further configured to: executing a web browser application for a session of the plurality of sessions, the web browser application configured to receive encoded media content from a media source for the session; determining that an aggregate activity of the plurality of sessions for the hosted virtual machine is above a threshold level; Based on the aggregate activity, multimedia redirection for the session is enabled by preventing the encoded media content from being processed by a decoder of the web browser application and redirecting the encoded media content to an associated client computer device of the plurality of client computer devices.

18. The server system of claim 17, wherein the encoded media content is formatted in a media container format comprising a video stream, an audio stream, or a text stream, and wherein the at least one processor is configured to demultiplex the encoded media content in the media container format into independent streams prior to the multimedia redirection.

19. The server system of claim 18, wherein the at least one processor is further configured to: buffering portions of the independent streams; and Media samples of the portion of the independent stream are sent to the associated client computer device.

20. The server system of claim 19, wherein the media samples for each of the independent streams are sent at an independent rate based on requests for the media samples received from the associated client computer devices.

Images