Remote desktop composition

Abstract

In response to a notification from one or more virtual machines (130, 150, 170), a host (110) capable of accessing a physical function (120) of a virtualized display device (199) utilizes the physical function to directly access a frame memory (144, 164, 184) of the virtualized display device associated with the notified virtual machine. The host generates a composite frame for display via the virtualized display device or otherwise by selecting a rendering frame from one or more of the notification virtual machines.

Description

Background Art

[0001] The present disclosure relates to capturing and selectively displaying rendered graphic frames and / or other audiovisual content using remote desktop composition. Previous solutions for such capture and selective display have often been associated with high-latency operations involving copying and transmitting selected frames. Summary of the Invention

[0002] A computing system provides a low-latency way by which rendered frames from multiple source applications can be composed and / or selected. In an example, a method includes a host virtual machine (VM) executing on a computing system receiving a notification from at least one guest VM that a rendered frame from one or more applications executing on the at least one guest VM is available for display. In response to receiving the notification, the host VM retrieves the rendered frame from the at least one guest VM and generates a composite frame for display based at least in part on the rendered frame retrieved from the at least one guest VM.

[0003] In some embodiments, retrieving the rendered frame from the at least one guest VM includes retrieving the rendered frame by accessing a frame memory associated with a virtual function of the at least one guest VM. In some examples, the method further includes retrieving multiple rendered frames from multiple guest VMs, and generating the composite frame for display by the host VM includes compositing at least a portion of two or more of the multiple rendered frames for simultaneous display.

[0004] In some embodiments, the method further includes retrieving multiple rendered frames from multiple guest VMs, and generating the composite frame for display by the host VM includes selecting one of the multiple rendered frames for display. The rendered frame from the at least one guest VM can include output from multiple applications composed by the at least one guest VM.

[0005] In some embodiments, the method further includes providing the composite frame to a remote client device for local display by the remote client device. The host VM can be associated with a first set of privileges that includes access to physical functions of the computing system for displaying the composite frame, and wherein the at least one guest VM is associated with a set of privileges that does not include access to the physical functions. Receiving the notification from the at least one guest VM can include processing an interrupt caused by a write operation to one or more registers of the host VM. Further, in some embodiments, the write operation to the one or more registers is at least partially based on one or more modifications to a swap chain associated with the one or more applications executing on the at least one guest VM.

[0006] The virtual function of the at least one guest VM can be a virtual function that virtualizes a physical display, and retrieving the rendered frame from the at least one guest VM by accessing the associated frame buffer can be performed by the host VM via the physical function of the virtualized physical display.

[0007] In another example, a computing system includes one or more processors and a non-transitory computer-readable medium storing a set of executable instructions that, when executed, cause the one or more processors to: receive, by a host virtual machine (VM) executing on the computing system, a notification from a guest VM that a rendered frame from one or more applications executing on the guest VM is available for display. In response to the notification, the one or more processors retrieve the rendered frame from the guest VM, and generate, by the host VM, a composite frame for display based at least in part on the rendered frame retrieved from the guest VM.

[0008] In some embodiments, the guest VM is one of a plurality of guest VMs executing on the computing system and managed by the host VM, and the host VM retrieves a plurality of rendered frames from the plurality of guest VMs by accessing corresponding frame buffers associated with each of the plurality of guest VMs. The composite frame can include at least a portion of two or more of the plurality of rendered frames for simultaneous display. Generating, by the host VM, the composite frame for display can include selecting one of the plurality of rendered frames for display.

[0009] The rendered frame from the guest VM can include outputs from a plurality of applications synthesized by the guest VM. In some embodiments, the host VM provides the composite frame to a remote client device for local display by the remote client device. The host VM can be associated with a first set of privileges that includes access to the physical function of the computing system for displaying the composite frame, and the guest VM can be associated with a set of privileges that does not include access to the physical function. In some embodiments, receiving the notification from the guest VM includes handling an interruption caused by a write operation to one or more registers of the host VM. The write operation to the one or more registers can be at least partially based on one or more modifications to a swap chain associated with the one or more applications executing on the guest VM.

[0010] In another example, a non-transitory computer-readable medium includes a set of executable instructions for manipulating at least one processor to instantiate a host virtual machine (VM) and one or more guest VMs, receive, by the host VM from one of the one or more guest VMs, a notification indicating that a rendered frame from one or more applications executing on the one guest VM is available for display, retrieve, by the host VM and in response to receiving the notification, a rendered frame from the guest VM by accessing a frame memory associated with a virtual function of the one guest VM, and generate, by the host VM, a composite frame for display based at least in part on the rendered frame retrieved from the guest VM. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present disclosure may be better understood by reference to the accompanying drawings, and its numerous features and advantages will be apparent to those skilled in the art. Identical reference numerals are used in the different drawings to denote similar or identical items.

[0012] Figure 1 Illustrates a local virtualization server configuration for remote desktop composition according to some embodiments.

[0013] Figure 2 Is an example of an operational flowchart of routines for execution by a server computing system suitable for implementing one or more embodiments.

[0014] Figure 3 Is an example of a component-level block diagram of a server computing system suitable for implementing one or more embodiments. DETAILED DESCRIPTION

[0015] A single root I / O virtualization (SR-IOV) interface is an extension of the Peripheral Component Interconnect Express (PCI Express or PCIe) serial computer expansion bus specification that allows devices such as network adapters or display adapters to broker isolated access to their resources among various PCIe hardware functions. These functions include both a physical function (PF) and a virtual function (VF), where the PF directly encompasses the primary functionality of the hardware device and typically requires system or supervisory-level privileges to access, and the VF is associated with the PF of the hardware device but utilizes a virtualized version of one or more physical resources of the device (e.g., memory, display adapter, network port, etc.).

[0016] The embodiments described herein provide a low-latency manner by which rendered frames from multiple source applications can be synthesized and / or selected. In some embodiments, a host virtual machine directly accesses the frame memory of virtual functions associated with each of one or more virtual machines communicatively coupled to or executing on a server computing system. The virtual functions enable access to a virtualized display device such that the host virtual machine executing on the server computing system can utilize the physical capabilities of the virtualized display device to directly access the frame memory of one or more associated virtual functions for low-latency retrieval and display.

[0017] Figure 1 An example of a local virtualization server configuration for remote desktop composition according to some embodiments is illustrated. Specifically, in the depicted embodiment, computing system 100 is executing a host VM 110 as well as multiple guest VMs 130, 150, 170. The host VM 110 includes a virtualization manager 111 that performs various VM management functions regarding the guest VMs 130, 150, 170 and also executes a compositor 113 to generate a composite frame for display based on respective rendered frames received as input, such as from one or more applications 112 executed by the host VM 110 or one or more applications 132, 152, 172 executed by the guest VMs 130, 150, 170 respectively, as discussed in more detail below. It should be understood that in various embodiments, the guest VMs 130, 150, 170 can represent any number n of guest VMs (e.g., guest VMs 0, 1,..., n - 1).

[0018] Each of the host VM 110 and the guest VMs 130, 150, 170 is communicatively coupled to a virtualized physical display device (virtualized physical display) 199 via a physical or virtual function representing the physical hardware capabilities of the virtualized physical display device 199 (e.g., a monitor or other display screen) and is supported by a graphics processing unit (GPU) 190 of the computing system 100. Specifically, the host VM 110 is communicatively coupled to the virtualized physical display 199 via a physical function 120; each of the guest VMs 130, 150, 170 is communicatively coupled to the virtualized physical display 199 via respective GPU drivers 139, 159,..., 179, each of which utilizes a respective virtual function 140 (VF 0), 160 (VF 1), 180 (VF n - 1) to access the capabilities provided by the virtualized physical display 199.

[0019] It should be understood that in certain embodiments and scenarios, the computing system 100 can be a server computing system without any attached physical display device, such that the operations described herein with respect to frames intended to be displayed via the virtualized physical display 199 are performed such that those frames are output via one or more network connections to one or more client devices for local display.

[0020] While the various embodiments can operate in a variety of scenarios, for the purposes of this example, the local virtualization server configuration of the depicted embodiment can be considered to operate as the head unit (control and display management system) of a vehicle such that the host VM 110 and the guest VMs 130, 150, 170 are all co-located on (and executed by) the computing system 100. In such a scenario, each of the guest VMs 130, 150, 170 is responsible for running one or more applications as respective subsystems of vehicle configuration and control. For example, the guest VM 130 (via application 132) operates the entertainment subsystem to control and display information regarding music, video, and other audio-visual entertainment selected by a user of the computing system 100; the guest VM 150 (via application 152) operates the seat and environmental controls; and the guest VM 170 (via application 172) operates the engine tuning and diagnostic subsystem. This utilization of the computing system 100 is merely exemplary, and it should be understood that the embodiments described herein can operate in a variety of contexts and scenarios.

[0021] In operation, each of the guest VMs 130, 150, 170 is respectively responsible for generating content via applications 132, 152, 172, rendering frames of the content for display on the virtualized physical display 199, and (via their respective GPU drivers 139, 159, 179) storing the rendered frames in respective frame memories 144, 164, 184 associated with the respective virtual functions 140, 160, 180 of the guest VMs. The host VM 110 is responsible for generating composite frames for presentation via, for example, the virtualized physical display 199. Specifically, the host VM 110 retrieves rendered frames from one or more of the guest VMs 130, 150, 170 to selectively provide composite frames that may include at least a portion of one or more of those retrieved rendered frames. For example, to continue the vehicle head unit example, the host VM 110 may determine to display in full each rendered frame of the entertainment subsystem retrieved from the guest VM 130 during normal driving operations. In various scenarios and situations, the host VM 110 may determine to generate composite frames for display using all of one or more rendered frames from one guest VM (e.g., to present in full the rendered frames of the seat and environmental control subsystem rendered by the guest VM 150), or only use a portion of the rendered frames retrieved from two or more of the guest VMs 130, 150, 170 to generate such composite frames in order to present information from multiple such subsystems simultaneously. The host VM 110 uses the compositor 113 to generate composite frames for display based on one or more rendered frames retrieved from one or more of the guest VMs 130, 150, 170.

[0022] The host VM 110 generally operates as a privileged system management partition of the computing system 100. In certain embodiments, the primary responsibility of the host VM 110 is to manage the computing system 100, including acting as a hypervisor on the guest VMs (via the virtualization manager 111) (e.g., initiating instantiation, suspension, and / or destruction of guest VMs, guest VM workload scheduling, etc.). Additionally, in some embodiments, the host VM 110 is responsible via the virtualization manager 111 for various operations regarding the abstraction and virtualization of hardware resources associated with the computing system 100, such as the virtualized physical display 199.

[0023] In the depicted embodiment, the virtualized physical display 199 is associated with physical and virtual functions that allow a VM executing on (or otherwise privileged by) the computing system 100 to access and invoke hardware functions associated with the virtualized physical display 199 (or otherwise display one or more rendered or composited frames). The virtualized physical display 199 is associated with virtual functions 140, 160, 180 (VF0, VF1, ..., VFn-1 respectively) of the display hardware for use by the respective guest VMs 130, 150, 170; the host VM 110 has privileged access to the virtualized physical display 199 via a physical function (PF) 120 that directly accesses the hardware functions of the virtualized physical display 199 via the host VM GPU driver 119. Each of the physical function 120 and the virtual functions 140, 160, 180 is associated with a PCI configuration block 122, 142, 162, 182; a frame memory 124, 144, 164, 184; and a set of memory-mapped I / O (MMIO) registers 126, 146, 166, 186. The frame memories 124, 144, 164, 184 are frame buffers for storing the rendered frames expected by their respective VMs for presentation on the virtualized physical display 199.

[0024] The host VM 110 is associated with a set of elevated privileges not enjoyed by the managed guest VMs 130, 150, 170, the set of elevated privileges including retrieving rendered frames directly from the virtualized display memories (frame memories 144, 164, 184) of the respective virtual functions of those guest VMs, and accessing the physical function 120 (and its frame memory 124) for displaying composited frames. In contrast, each guest VM 130, 150, 170 is associated with a set of privileges that does not include access to the physical function 120 or the frame memories associated with other guest VMs. Instead, the PF 120 includes a superset of the functions provided by the VFs 140, 160, 180, including the ability to access the frame memories 144, 164, 184 associated with the guest VMs 130, 150, 170 respectively. Generally, the PF 120 and the VFs 140, 160, 180 are each assigned a unique requester identifier (RID) that allows the I / O memory management unit (IOMMU) to distinguish different traffic flows and apply memory and interrupt translations between the PF 120 and the VFs 140, 160, 180. This allows traffic to be delivered directly to the appropriate host VM or guest VM, such as delivering unprivileged data traffic from the host VM 110 to one of the guest VMs 130, 150, 170 without affecting the other guest VMs.

[0025] In operation, and with specific reference to the operation of guest VM 130, assume that application 132 generates content frames for display. Application 132 presents the generated frames for display by storing the generated frames in swap chain 134. A swap chain is a series of (usually application-specific, such that each application 132 maintains a corresponding swap chain 134) physical or virtualized frame buffers for displaying graphical frames to the user, such that each time an application presents a new frame for display, the next frame buffer in the swap chain replaces the currently displayed frame buffer. In various embodiments and scenarios, each of the swap chains 134 may contain frames rendered based on content from a specific application, or frames that form the entire composite multi-application desktop of guest VM 130. With respect to host VM 110, swap chain 114 may contain frames rendered based on content from a specific application 112, or frames that form the entire composite multi-application desktop of host VM 110. Additionally, swap chain 114 of host VM 110 may contain frames generated from any one of the swap chains 134, 154, 174 of the respective guest VMs 130, 150, 170.

[0026] When application 132 stores the newly generated frame in the frame buffer of swap chain 134, it notifies GPU driver 139, and the GPU driver transfers the generated frame to be displayed into frame memory 144 via virtual function 140. In some embodiments and scenarios, guest VM 130 may determine to generate a rendered frame for display via virtual function 140 by generating a composite frame that includes portions of frames rendered and stored in multiple corresponding swap chains 134 by multiple applications 132. In such a scenario, the compositor facility of OS 138 generates the composite frame and stores it in frame memory 144, rather than a single generated frame from one application 132. In either case, when guest VM 130 renders and stores the rendered frame in frame memory 144, the host VM may directly access and read the rendered frame via PF 120.

[0027] Continuing with the example of the operation of the guest VM 130, when the GPU driver on the guest VM is notified to present a rendered frame for display (e.g., in response to the frame buffer of one of the swap chains 134 being modified), it notifies the GPU driver 119 of the host VM 110 that a rendered frame is available from the guest VM 130. In some embodiments, this communication is facilitated through the mailbox facility of the SR-IOV interface. Specifically, in such an embodiment, the GPU driver 139 sets the mailbox register (not shown) of the VF 140. This write operation to the mailbox register triggers an interrupt on the host VM 110, which is processed by the host GPU driver 119 to determine which guest VM has written to the mailbox register (and thus which guest VM has a rendered frame available for display). Upon identifying the guest VM 130, the host GPU driver 119 accesses the frame memory 144 and retrieves the rendered frame prepared by the guest VM 130 via its application 132, swap chain 134, and GPU driver 139. In some embodiments, retrieving the generated frame from the frame memory 144 in a low-latency manner may include setting the pointer value of the frame memory 124 for the host VM to the memory address of the accessed frame memory 144 of the guest VM 130.

[0028] Once the host VM 110 has retrieved the generated frame from the frame memory 144 (by copying its contents or redirecting the pointer value accordingly), the host VM 110 generates a composite frame for display based on the rendered frame from the guest VM 130. As noted above, in various scenarios and situations, the host VM 110 may determine to generate a composite frame for display using the entire rendered frame from the guest VM 130, or only using a portion of that rendered frame and portions of similar rendered frames retrieved from one or more of the guest VMs 150, 170, in order to present information from multiple guest VMs simultaneously. The host VM 110 uses the compositor 113 to generate a composite frame for display based on one or more rendered frames retrieved from one or more of the guest VMs 130, 150, 170. In various embodiments, the compositor 113 can be one or more of a software compositor executable by the host VM 110; a component of the operating system (OS) 118 of the host VM 110; a hardware compositor of the GPU 190 and controlled by the host VM 110 via the GPU driver 119; or some combination of these.

[0029] It should be understood that although the above describes specific operations with respect to the guest VM 130 and its various associated elements, similar operations are performed by each of the guest VMs 150, 170 and their respective associated elements.

[0030] Each of the host VM 110 and the guest VMs 130, 150, 170 executes a respective operating system (OS) 118, 138, 158, 178. In various embodiments, each of the OSs 118, 138, 158, 178 may be substantially the same or may be different such that each VM operates via a different OS independent of the OSs executing on the other VMs. Additionally, each of the host VM 110 and the guest VMs 130, 150, 170 includes a respective GPU driver 119, 139, 159, 179 that is communicatively coupled to the virtualized physical display 199 via a physical function 120 or virtual functions 140, 160, 180.

[0031] In the depicted embodiment, the GPU 190 also includes a communication crossbar 192 that provides cross-communication for the components of the GPU 190. In the depicted embodiment, the GPU 190 also includes functional blocks 196 (e.g., fixed function blocks, compute blocks, direct memory access (DMA) control blocks, etc.) and a video interface 198 that communicatively couples the virtualized physical display 199 to the computing system 100.

[0032] Figure 2 is an example flowchart illustrating an operation routine 200 for execution by a server (such as Figure 1 the computing system 100 or the server computing system 300 described below Figure 3 ). Specifically, in some embodiments, the operation routine is substantially executed by one or more hardware processors of the server (e.g., Figure 3 the hardware processor 302) and is associated with instructions (e.g., as Figure 1 at least some of the instructions 324) that are associated with one or more virtual machines executing on the server (e.g., Figure 3 the host VM 110 and the guest VMs 130, 150, 170).

[0033] The operation routine begins at block 205 where the host virtual machine (e.g., Figure 1 the host VM 110) receives an indication that a new frame has been rendered by a designated guest VM and is available for display. In certain embodiments, the indication includes a notification triggered by one or more modifications to a frame buffer of a swap chain associated with an application executing on the designated guest VM. In certain scenarios and embodiments, the rendering guest VM is one of one or more guest VMs managed by the host VM, such as may be executing on the same server. The operation routine proceeds to block 210.

[0034] At block 210, the host VM accesses the frame memory associated with the rendering guest VM. In some embodiments, the frame memory is associated with a virtual function of the rendering guest VM, such as to provide rendered frames to a virtualized display device. In some embodiments, accessing the frame memory is performed as part of processing an interrupt generated by a set register, such as a mailbox register. The routine proceeds to block 215, where the host VM retrieves available rendered frames from the accessed frame memory, and then proceeds to block 220.

[0035] At block 220, the host VM generates a composite frame for display based at least in part on the rendered frames retrieved from the accessed frame memory of the guest VM. In some embodiments, generating the composite frame is performed by a compositor facility of the host VM (e.g., Figure 1 compositor 113) or the operating system (e.g., OS118). As discussed elsewhere herein, the composite frame may include portions generated by a single application executing on a single guest VM, by multiple applications executing on a single guest VM (such as may have been composited by the guest VM as part of rendering the frames retrieved in block 215), or by compositing portions of multiple rendered frames retrieved from multiple guest VMs (and generated by one or more applications executing thereby) for simultaneous display. The operational routine proceeds to block 225.

[0036] At block 225, the host machine provides the composite frame for display. In some embodiments and scenarios, the host VM utilizes the physical function of a GPU of the server (e.g., Figure 1 GPU 190 or Figure 3 GPU 306) to provide the composite frame for local display, such as via a video interface of the GPU (e.g., Figure 1 video interface 198) and a coupled display (e.g., virtualized physical display 199). In some embodiments, providing the composite frame for display includes providing it (e.g., streaming, such as providing a series of such composite frames substantially in real time over a network, such as via Figure 3 network interface device 320 and communication network 326 below) to a remote client device (e.g., client device 399) for presentation via a communicatively coupled display device.

[0037] Figure 3FIG. 0 is a component-level block diagram illustrating an example of a server computing system 300 suitable for implementing one or more embodiments. In an alternative embodiment, the server computing system 300 may operate as a stand-alone device or may be connected (e.g., networked) to other systems. In various embodiments, one or more components of the server computing system 300 each include a collection of circuitry that, when operating in combination, perform the various operations of the server computing system 300 as described herein. In some embodiments, one or more components of the server computing system 300 are incorporated as or within one or more server computing systems to provide, as a non-limiting example, graphics rendering for display to one or more users via one or more remote client computing devices. It should be understood that such associated client computing devices may include some, but not necessarily all, of the components of the server computing system 300. In a networked deployment, the server computing system 300 may operate in a server-client network environment with the capabilities of a server machine, a client machine, or both. In at least one example, the server computing system 300 may act as a peer system in a peer-to-peer (P2P) (or other distributed) network environment. Thus, in certain embodiments, the server computing system 300 may operate as a server computer, a personal computer (PC), a tablet PC, a set-top box (STB), a mobile computing device, a web appliance, a network router, switch, or bridge, or any system capable of executing instructions (sequential or otherwise) specifying actions to be taken by the system. Further, while only a single system is illustrated, the term "system" should also be considered to include any collection of systems that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

[0038] As described herein, an example may include a logic component or components or mechanisms, or may be operated by a logic component or components or mechanisms. A circuit is a collection of circuits implemented in a tangible entity that includes hardware (e.g., simple circuits, gates, logic components, etc.). Circuit membership may vary flexibly over time and with underlying hardware variability. A circuit includes components that, when operating alone or in combination, perform a specified operation. In an example, the hardware of a circuit may be designed invariantly to perform a particular operation (e.g., hardwired). In an example, the hardware of a circuit may include physically components that are variably connected (e.g., execution units, transistors, simple circuits, etc.), which include a computer-readable medium that is physically modified (e.g., magnetic properties of invariantly aggregated particles, electrically movable placement, etc.) to encode instructions for a particular operation. When physically connecting the components, the underlying electrical properties of the hardware components change, for example, from an insulator to a conductor and vice versa. The instructions enable the embedded hardware (e.g., execution unit or loading mechanism) to create components of the circuit in the hardware via variable connections to perform a portion of a particular operation when operating. Thus, when the device operates, the computer-readable medium is communicatively coupled to other components of the circuit. In an example, any one of the physical components may be used in more than one component of more than one circuit. For example, in operation, an execution unit may be used in a first circuit in a first circuit at one point in time and reused by a second circuit in the first circuit or by a third circuit in a second circuit at another time.

[0039] Server computing system 300 includes one or more hardware processors 302 (e.g., central processing unit (CPU), hardware processor cores, or any combination thereof), main memory 304, and a graphics processing unit (GPU) 306 or other parallel processors, some or all of which may communicate with each other via an interconnect link (e.g., a bus) 308. In the depicted embodiment, a graphics driver 325 (which may be operationally similar to Figure 1 the GPU driver 119) executes within the main memory 304, such as to interface with the GPU 306 to perform the execution of the various operations described herein.

[0040] The server computing system 300 also includes a display unit 310 (such as a display monitor or other display device), an input device 312 (e.g., a keyboard or other physical or touch-based actuator), and a user interface (UI) navigation device 314 (e.g., a mouse or other pointing device, such as a touch-based interface). In one example, the display unit 310, the input device 312, and the UI navigation device 314 may include a touchscreen display. The server computing system 300 may additionally include a storage device (e.g., a drive unit) 316, a signal generation device 318 (e.g., a speaker), a network interface device 320, and one or more sensors 321, such as a Global Positioning System (GPS) sensor, a compass, an accelerometer, or other sensors. The server computing system 300 may include an output controller 328, such as a serial (e.g., Universal Serial Bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), Near Field Communication (NFC), etc.) connection to communicate with or control one or more peripheral devices (e.g., a printer, a card reader, etc.).

[0041] The storage device 316 may include a computer-readable medium 322 on which a set or multiple sets of data structures or instructions 324 (e.g., software) are stored, which embody or are utilized by any one or more of the techniques or functions described herein. The instructions 324 may also reside, in whole or at least in part, within the main memory 304, within the GPU 306, or within the hardware processor 302 during execution by the server computing system 300. In an example, one or any combination of the hardware processor 302, the main memory 304, the GPU 306, or the storage device 316 may constitute a computer-readable medium.

[0042] Although the computer-readable medium 322 is illustrated as a single medium, the term "computer-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 324. Additionally, in certain implementations and scenarios, the instructions 324 may include data, such as digital representations of one or more synthesized or otherwise generated frames for local display by one or more client devices 399.

[0043] The term "computer-readable medium" can include any medium that can store, encode, or carry instructions for execution by the server computing system 300 and cause the server computing system 300 to perform any one or more of the techniques of the present disclosure, or that can store, encode, or carry data structures used by or associated with such instructions. Non-limiting examples of computer-readable media can include solid state memories and optical and magnetic media. In an example, an aggregated computer-readable medium includes a computer-readable medium having a plurality of particles that have invariant (e.g., rest) mass. Thus, an aggregated computer-readable medium is not a transient propagated signal. Specific examples of aggregated computer-readable media can include non-volatile memories such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0044] Instruction 324 can also be sent or received via a network interface device 320 over a communication network 326 using a transmission medium by way of any one of a variety of transmission protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Example communication networks can include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile telephone networks (e.g., cellular networks), plain old telephone (POTS) networks, and wireless data networks (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard series known as and the IEEE 802.16 standard series known as , the IEEE 802.15.4 standard series, peer-to-peer (P2P) networks, and so on. In an example, the network interface device 320 can include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas to connect to the communication network 326. In an example, the network interface device 320 can include multiple antennas to perform wireless communication using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term "transmission medium" should be considered to include any intangible medium that can store, encode, or carry instructions for execution by the server computing system 300 and includes digital or analog communication signals or other intangible media to facilitate the communication of such software.

[0045] In some embodiments, the above-described apparatus and techniques are implemented in a system that includes one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips). Electronic design automation (EDA) and computer-aided design (CAD) software tools can be used in the design and manufacture of these IC devices. These design tools are typically represented as one or more software programs. The one or more software programs include code executable by a computer system to manipulate the computer system to operate on code representing a circuit of one or more IC devices in order to perform at least a portion of a process for designing or tuning a manufacturing system to manufacture the circuit. The code can include instructions, data, or a combination of instructions and data. Software instructions representing design tools or manufacturing tools are typically stored in a computer-readable storage medium accessible to the computing system. Similarly, code representing one or more stages of the design or manufacture of an IC device can be stored in and accessed from the same computer-readable storage medium or a different computer-readable storage medium.

[0046] A computer-readable storage medium can include any non-transitory storage medium or combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard disk drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical systems (MEMS)-based storage media. A computer-readable storage medium can be embedded in the computing system (e.g., system RAM or ROM), fixedly attached to the computing system (e.g., magnetic hard disk drive), removably attached to the computing system (e.g., optical disc or universal serial bus (USB)-based flash memory), or coupled to the computer system via a wired or wireless network (e.g., network-attached storage device (NAS)).

[0047] In some embodiments, certain aspects of the above techniques may be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions that are stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and certain data that, when executed by one or more processors, manipulate the one or more processors to perform one or more aspects of the above techniques. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as flash memory, a cache, a random access memory (RAM), or other one or more non-volatile memory devices, etc. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executed by one or more processors.

[0048] Note that not all activities or elements described above in the general description are required, that a portion of a particular activity or device may not be required, and that one or more additional activities may be performed or elements may be included in addition to those described. Further, the order of listing the activities is not necessarily the order in which they are to be performed. Additionally, these concepts have been described with reference to specific embodiments. However, those of ordinary skill in the art understand that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the following claims. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the present disclosure.

[0049] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature that may cause any benefit, advantage, or solution to occur or become more pronounced should not be construed as a critical, required, or essential feature of any or all of the claims. Moreover, the specific embodiments disclosed above are merely illustrative, as the disclosed subject matter may be modified and practiced in different but equivalent manners that are apparent to those skilled in the art having the benefit of the teachings herein. The details of the construction or design shown herein are not intended to be limiting other than as set forth in the following claims. Accordingly, it is evident that the specific embodiments disclosed above can be altered or modified, and all such variations are considered to be within the scope of the disclosed subject matter. Thus, the protection sought herein is as set forth in the following claims.

Claims

1. A method, the method comprising: receiving, by a host virtual machine (VM) executing on a computing system, from at least one guest VM, a notification that a rendered frame from one or more applications executing on the at least one guest VM is available for display; retrieving, in response to receiving the notification, the rendered frame from the at least one guest VM; and generating, by the host VM, a composite frame for display based at least in part on the rendered frame retrieved from the at least one guest VM.

2. The method according to claim 1, wherein retrieving the rendered frame from the at least one guest VM comprises retrieving the rendered frame by accessing a frame memory associated with a virtual function of the at least one guest VM.

3. The method according to claim 1 or claim 2, comprising retrieving a plurality of rendered frames from a plurality of guest VMs, wherein generating the composite frame for display by the host VM comprises compositing at least a portion of two or more of the plurality of rendered frames for simultaneous display.

4. The method according to claim 2, comprising retrieving a plurality of rendered frames from a plurality of guest VMs, wherein generating the composite frame for display by the host VM comprises selecting one of the plurality of rendered frames for display.

5. The method according to any one of claims 1 to 4, wherein the rendered frame from the at least one guest VM comprises an output from a plurality of applications composited by the at least one guest VM.

6. The method according to any one of claims 1 to 5, further comprising providing the composite frame to a remote client device for local display by the remote client device.

7. The method according to any one of claims 1 to 6, wherein the host VM is associated with a first set of privileges including access to a physical function of the computing system for displaying the composite frame, and wherein the at least one guest VM is associated with a set of privileges that does not include access to the physical function.

8. The method according to any one of claims 1 to 7, wherein receiving the notification from the at least one guest VM comprises processing an interruption caused by a write operation to one or more registers of the host VM.

9. The method according to claim 8, wherein the write operation to the one or more registers is at least partially based on one or more modifications to a swap chain associated with the one or more applications executing on the at least one guest VM.

10. The method according to any one of claims 1 to 9, wherein the virtual function of the at least one guest VM is a virtual function that virtualizes a physical display, and wherein retrieving the rendered frame from the at least one guest VM by accessing the associated frame memory is performed by the host VM via a physical function of the virtualized physical display.

11. A computing system, the computing system comprising: one or more processors; a non-transitory computer-readable medium storing a set of executable instructions that, when executed, cause the one or more processors to: A host virtual machine (VM) executing on the computing system receives a notification from a guest VM that a rendered frame from one or more applications executing on the guest VM is available for display; In response to the notification, retrieve the rendered frame from the guest VM; And The host VM generates a composite frame for display based at least in part on the rendered frame retrieved from the guest VM.

12. The computing system according to claim 11, wherein the guest VM is one of a plurality of guest VMs executing on the computing system and managed by the host VM, and wherein the host VM retrieves a plurality of rendered frames from the plurality of guest VMs by accessing a respective frame memory associated with each of the plurality of guest VMs.

13. The computing system according to claim 12, wherein the composite frame includes at least a portion of two or more of the plurality of rendered frames for simultaneous display.

14. The computing system according to claim 12 or claim 13, wherein generating the composite frame for display by the host VM includes selecting one of the plurality of rendered frames for display.

15. The computing system according to any one of claims 11 to 14, wherein the rendered frame from the guest VM includes an output from a plurality of applications synthesized by the guest VM.

16. The computing system according to any one of claims 11 to 15, wherein the host VM provides the composite frame to a remote client device for local display by the remote client device.

17. The computing system according to any one of claims 11 to 16, wherein the host VM is associated with a first set of privileges including access to physical functions of the computing system for displaying the composite frame, and wherein the guest VM is associated with a set of privileges that do not include access to the physical functions.

18. The computing system according to claim 11, wherein receiving the notification from the guest VM includes processing an interruption caused by a write operation to one or more registers of the host VM.

19. The computing system according to claim 18, wherein the write operation to the one or more registers is at least partially based on one or more modifications to a swap chain associated with the one or more applications executing on the guest VM.

20. A non-transitory computer-readable medium having an executable instruction set for causing at least one processor to perform the following operations: Instantiate a host virtual machine (VM) and one or more guest VMs The host VM receives a notification from one of the one or more guest VMs indicating that a rendered frame from one or more applications executing on the one guest VM is available for display; In response to receiving the notification, the host VM retrieves the rendered frame from the guest VM by accessing a frame memory associated with a virtual function of the one guest VM; and The host VM generates a composite frame for display at least partially based on the rendered frame retrieved from the guest VM.

Images