Sharing multimedia physical functions in a virtualized environment of processing units

Abstract

A processing unit is disclosed, comprising a kernel mode unit configured to execute a hypervisor and a guest virtual machine (VM) and a register set. The processing unit also comprises a fixed function hardware block configured to implement a physical function. A virtual function corresponding to the physical function is exposed to the guest VM. Subsets of the register set are allocated for storing information associated with the virtual function, and the fixed function hardware block executes one of the virtual functions for one of the guest VMs based on the information stored in a corresponding one of the subsets. Each subset comprises a frame buffer for storing frames operated by the virtual function associated with the subset, a context register for defining the operating state of the virtual function, and a doorbell register for signaling that the virtual function is ready to be scheduled for execution.

Description

Background Art

[0001] A traditional processing system includes a central processing unit (CPU) and a graphics processing unit (GPU) that implements audio, video, and graphics applications. In some cases, the CPU and GPU are integrated into an accelerated processing unit (APU). Multimedia applications are represented as statically programmed sequences of microprocessor instructions grouped in programs or as processes (containers) where a set of resources are allocated to the multimedia application during the lifetime of the application. For example, A process consists of a private virtual address space, an executable program, a handle set that maps and utilizes various system resources (such as semaphores, synchronization objects, and thread-accessible files in the process), a security context (consisting of user identification, permissions, access attributes, user account control flags, sessions, etc.), a process identifier that uniquely identifies the client application, and one or more execution threads. The operating system (OS) also supports multimedia. For example, the OS can open multimedia files encapsulated in a specific container. Examples of multimedia containers include .mov, .mp4, and .ts. The OS locates the audio or video container, retrieves the content, decodes the content in software on the CPU or an available multimedia accelerator, renders the content, and presents the rendered content on the display, for example, as alpha blended or color keyed graphics. In some cases, the CPU initiates graphics processing by issuing a draw call to the GPU. A draw call is a command generated by the CPU and transmitted to the GPU to instruct the GPU to render an object (or a portion of an object) in a frame. A draw call includes information such as a defined texture, state, shader, render object, buffer, etc. used by the GPU to render an object or a portion thereof. The GPU renders objects to produce pixel values ​​that are provided to a display, which uses the pixel values ​​to display an image representing the rendered objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The present disclosure may be better understood, and its numerous features and advantages become apparent to those skilled in the art by referencing the accompanying drawings.The use of the same reference numbers in different drawings indicates similar or identical items.

[0003] Figure 1 is a block diagram of a processing system including a graphics processing unit (GPU) that implements physical function sharing in a virtualized environment, according to some embodiments.

[0004] Figure 2 is a block diagram of a system on a chip (SOC) that integrates a central processing unit (CPU) and a GPU on a single semiconductor die, according to some embodiments.

[0005] Figure 3is a block diagram of a first implementation of a hardware architecture that supports multimedia virtualization on a GPU, according to some implementations.

[0006] Figure 4 is a block diagram of a second embodiment of a hardware architecture supporting multimedia virtualization on a GPU according to some embodiments.

[0007] Figure 5 is a block diagram of an operating system (OS) for supporting multimedia processing in a virtualized OS ecosystem, according to some embodiments.

[0008] Figure 6 is a block diagram of an OS architecture with virtualization support according to some embodiments.

[0009] Figure 7 is a block diagram of a multimedia software system for compressed video decoding, rendering, and presentation, according to some embodiments.

[0010] Figure 8 is a block diagram of a physical function configuration space identifying a base address register (BAR) for a physical function, according to some embodiments.

[0011] Fig. 9 is a block diagram of a portion of a single root I / O virtualization (SR-IOV) header that identifies a BAR for a virtual function, according to some embodiments.

[0012] Fig.10 is a block diagram of the life cycle of a host OS implementing physical functions and a guest virtual machine (VM) implementing virtual functions associated with the physical functions, according to some embodiments.

[0013] Fig.11 is a block diagram of a multimedia user mode driver and a kernel mode driver according to some embodiments.

[0014] Fig.12 is the first part of a message sequence that supports sharing of multimedia capabilities in a virtualized OS ecosystem according to some embodiments.

[0015] Fig.13 is the second part of a message sequence that supports sharing of multimedia capabilities in a virtualized OS ecosystem according to some embodiments. DETAILED DESCRIPTION

[0016] Processing units such as graphics processing units (GPUs) support virtualization, which allows multiple virtual machines to use the hardware resources of the GPU. Each virtual machine is executed as a separate process using the hardware resources of the GPU. Some virtual machines implement operating systems that allow virtual machines to simulate actual machines. Other virtual machines are designed to execute code in a platform-independent environment. The hypervisor creates and runs virtual machines, which are also called guest machines or guests. The virtual environment implemented on the GPU provides virtual functions for other virtual components implemented on the physical machine. A single physical function implemented in the GPU is used to support one or more virtual functions. The physical function assigns virtual functions to different virtual machines on the physical machine by time slice. For example, the physical function assigns a first virtual function to a first virtual machine in a first time interval and assigns a second virtual function to a second virtual machine in a subsequent second time interval. In some cases, the physical function in the GPU supports up to thirty-one virtual functions, but in other cases supports more or fewer virtual functions. The single root input / output virtualization (SR IOV) specification allows multiple virtual machines to share the GPU interface to a single bus (such as a peripheral component interconnect express (PCIe) bus). Components access virtual functions by transmitting requests on the bus.

[0017] Hardware acceleration functions are used to accelerate the processing of multimedia content, for example, by a virtual machine executed on a GPU. For example, hardware accelerated multimedia content processing can be implemented by using an application that is part of a specific OS distribution or provided by an independent software vendor. To use hardware acceleration, a multimedia application needs to query the hardware accelerated multimedia function of the GPU before starting audio, video or multimedia playback. The query includes a request for information such as supported codecs (encoders-decoders), maximum video resolution, and maximum supported source rate. Separate processes (e.g., separate host or guest virtual machines) are used to execute different instances of the same multimedia application, and multiple instances of the multimedia application executed by different virtual machines are unaware of each other. In some cases, the user mode driver does not know how many different instances are running simultaneously on the GPU. The user mode driver typically only allows a single instance of a hardware function (such as a codec) to be opened and assigned to a process (such as a virtual machine). Therefore, the first application that starts graphics processing on the GPU (e.g., in the first virtual machine) is assigned fixed-function hardware to decode the compressed video bitstream. Fixed-function hardware is not available for allocation to subsequent applications while the first application is executing, and therefore the second application executing on the second virtual machine is decoded (or encoded) using software executing on a general-purpose application processor, such as a central processing unit (CPU). Applications executing on other virtual machines are also decoded (or encoded) using software executing on the CPU until the resources (cores and threads) of the CPU are fully occupied. This situation is energy inefficient and typically slows down the processing system when higher source resolutions and higher refresh rates are required.

[0018] Figures 1 to 13Disclosed is an embodiment of a technology for increasing the execution speed of multimedia applications while reducing the power consumption of a processing system by allowing multiple virtual machines to share hardware functions provided by fixed-function hardware blocks in a GPU, rather than forcing all processes except one process to use hardware acceleration provided by software executed on a CPU. The hardware acceleration function is implemented as a physical function provided by a fixed-function hardware block. In some embodiments, the physical function performs encoding of a multimedia data stream, decoding of a multimedia data stream, encoding / decoding of audio or video data, or other operations. Multiple virtual functions corresponding to the physical function are exposed to a guest virtual machine (VM) executed on a GPU. The GPU includes a register set, and a subset of registers is allocated for storing information associated with different virtual functions. The number of subsets and the number of registers in the subset are set to a static value corresponding to the maximum amount of space used by each virtual function or an initial value corresponding to the minimum amount of space used by each virtual function, which is then dynamically modified based on the properties of the virtual function. In some embodiments, each subset of registers includes a frame buffer for storing frames operated by a virtual function, a context register for defining the operating state of the virtual function, and a doorbell for signaling that the virtual function is ready to be scheduled for execution by a GPU (e.g., using one or more computing units of a GPU).

[0019] The hypervisor grants or denies access to registers to one guest VM at a time. The guest VM that has access to the register performs graphics rendering on the frame stored in the frame buffer in the register subset of the guest VM. The fixed function hardware block on the GPU is configured to perform virtual functions for the guest VM based on the information stored in the context register in the register subset of the guest VM. In some embodiments, the configuration of the fixed function hardware block includes installing a user mode driver and firmware image for multimedia functions that implement virtual functions. The guest VM signals that it is ready to be scheduled for execution by writing information to the doorbell register in the subset. The scheduler in the GPU schedules the guest VM to execute the virtual function at the scheduled time. In some embodiments, the guest VM is scheduled based on the priority associated with the guest VM and other priorities associated with other guest VMs that are ready to be scheduled. At the scheduled time, a world switch is performed to switch the context from the context defined for the previously executed guest VM to the context for the current guest VM, for example, as defined in the context register in the register subset of the current guest VM. In some embodiments, the world switch includes installing a user mode driver and firmware image for multimedia functions that implement virtual functions on the GPU. After the world switch is completed, the current guest VM begins executing a virtual function to perform a hardware accelerated operation on the frame in the frame buffer register. As discussed herein, examples of hardware accelerated operations include multimedia decoding, multimedia encoding, video decoding, video encoding, audio decoding, audio encoding, etc. The scheduler schedules the guest VM in a time interval, and the guest VM has exclusive access to the virtual function and the register subset during the time interval. In response to completing execution during the time interval, the guest VM notifies the hypervisor that another virtual function can be loaded for another guest VM and that the doorbell of the guest VM has been cleared.

[0020] Figure 11 is a block diagram of a processing system 100 including a graphics processing unit (GPU) 105 that implements physical function sharing in a virtualized environment according to some embodiments. The GPU 105 includes one or more GPU cores 106 that independently execute instructions simultaneously or in parallel and one or more shader systems 107 that support 3D graphics or video rendering. For example, the shader system 107 can be used to improve visual presentation by increasing the fraction of frames per second of graphics rendering or patching areas of the rendered image where the graphics engine does not accurately render the scene. A memory controller 108 provides an interface to a frame buffer 109 that stores frames during the rendering process. Some embodiments of the frame buffer 109 are implemented as dynamic random access memory (DRAM). However, the frame buffer 109 can also be implemented using other types of memory including static random access memory (SRAM), non-volatile RAM, etc. Some embodiments of the GPU 105 include other circuits, such as an encoder format converter, a multi-format video codec, a display output circuit that provides an interface to a display or screen, and an audio coprocessor, an audio codec for encoding / decoding audio signals, etc.

[0021] The processing system 100 also includes a central processing unit (CPU) 115 for executing instructions. Some embodiments of the CPU 115 include multiple processor cores 120, 121, 122 (collectively referred to herein as "CPU cores 120-122"), which can independently execute instructions simultaneously or in parallel. In some embodiments, the GPU 105 operates as an independent GPU (dGPU) connected to the CPU 115 via a bus 125 (such as a PCI-e bus) and a north bridge 130. The CPU 115 also includes a memory controller 108 that provides an interface between the CPU 115 and the memory 140. Some embodiments of the memory 140 are implemented as DRAM, SRAM, non-volatile RAM, etc. The CPU 115 executes instructions such as program code 145 stored in the memory 140 and the CPU 115 stores information 150 such as the results of the executed instructions in the memory 140. The CPU 115 is also capable of initiating graphics processing by issuing a draw call to the GPU 105. A draw call is a command generated by the CPU 115 and transmitted to the GPU 105 to instruct the GPU 105 to render an object (or portion of an object) in a frame.

[0022] Northbridge 130 is connected to southbridge 155. Southbridge 155 provides one or more interfaces 160 to peripheral units associated with processing system 100. Some embodiments of interface 160 include interfaces for peripheral units such as universal serial bus (USB) devices, general purpose I / O (GPIO), SATA for hard drives, serial peripheral bus interfaces such as SPI, I2C, etc.

[0023] The GPU 105 includes a GPU virtual memory management unit with an address translation controller (GPU MMU ATC) 165, and the CPU 115 includes a CPU MMU ATC 170. The GPU MMU ATC 165 and the CPU MMU ATC 170 provide translation of virtual memory addresses (VA) to physical memory addresses (PA) by using a multi-level translation logic and a set of translation tables maintained by the operating system kernel mode driver (KMD). Therefore, each application process executed on the main OS or the guest OS has its own virtual address space for CPU operation and GPU rendering. Therefore, the GPU MMU ATC 165 and the CPU MMU ATC 170 support the virtualization of the GPU and the CPU core. The GPU 105 has its own memory management unit (MMU), which translates the GPU virtual address of each process into a physical address. Each process has a separate CPU and GPU virtual address space using different page tables. The video memory manager manages the GPU virtual address space of all processes, and supervises allocation, growth, update, ensures the residency of memory pages and releases page tables.

[0024] Some embodiments of the GPU 105 share address space and page tables / page directories with the CPU 115 and can therefore operate in a system virtual memory mode (IOMMu). In the GPU MMU model, the video memory manager (VidMM) in the OS kernel manages the GPU MMU ATC 165 and page tables, while exposing the device driver interface (DDI) services to the user mode driver (UMD) for GPU virtual address mapping. In the IOMMU model, the GPU 105 and the CPU 115 share a common address space, a common page directory, and a page table. This model is called a (full) system virtual memory (SVM). Some embodiments of the APU hardware support:

[0025] • A first MMU unit for GPU 105 access to GPU memory and CPU system memory.

[0026] • A second MMU unit for CPU 115 access to CPU memory and GPU system memory.

[0027] Similarly, in some embodiments, the independent GPU HW has its own GPU MMU ATC 165 and the independent CPU multi-core system has its own CPU MMU with ATC 170. The MMU unit with ATC maintains separate page tables for CPU and GPU accesses for each virtual machine / guest OS, resulting in each guest OS having its own set of system and graphics memory.

[0028] Some embodiments of the processing system 100 implement a desktop window manager (DWM) to perform decoding, encoding, computing and / or rendering jobs submitted directly from user mode to the GPU 105. The GPU 105 exposes and manages various user mode work queues, thereby eliminating the need for the video memory manager (VidMM) to check and patch each command buffer before submitting it to the GPU engine. As a positive result, packet-based scheduling can be batch-based (allowing more back-to-back jobs to be submitted via the queue system per unit time), thereby allowing the central processor unit (CPU) to operate at a low power level, consuming minimal power. Other benefits of implementing some embodiments of the GPU and ATC 165 and the CPU MMU ATC 170 include the ability to disperse virtual memory allocations, which can be divided in non-contiguous GPU or CPU memory space. In addition, there is no need for CPU memory address patching, nor is there a need to track memory references within the GPU command buffer through a list of allocation and patch locations, or to patch these buffers with the correct physical memory references before submitting them to the GPU engine.

[0029] The GPU 105 also includes one or more fixed function hardware blocks 175 that implement physical functions. In some embodiments, the physical functions implemented in the fixed function hardware blocks 175 are hardware accelerated functions such as multimedia decoding, multimedia encoding, video decoding, video encoding, audio decoding, and audio encoding. The virtual environment implemented in the memory 140 supports the physical functions and the set of virtual functions exposed to the guest VM. The GPU 105 also includes a register set (for clarity) Figure 1 The fixed function hardware block 175 may include a register set (not shown) that stores information associated with processing performed by the kernel mode unit. A subset of the register set is allocated for storing information associated with a virtual function. The fixed function hardware block 175 performs a virtual function for a guest VM based on the information stored in a corresponding subset, as discussed in detail herein.

[0030] Figure 2 2 is a block diagram of a system on a chip (SOC) 200 that integrates a CPU and a GPU on a single semiconductor die according to some embodiments. SOC 200 includes a multi-core processing unit 205 that enables sharing of physical functions in a virtualized environment, as discussed herein. Multi-core processing unit 205 includes a CPU core complex 208 formed by one or more CPU cores that independently execute instructions simultaneously or in parallel. For clarity, Figure 2 The individual CPU cores are not shown.

[0031] The multi-core processing unit 205 also includes circuits for encoding and decoding data such as multimedia data, video data, audio data, and combinations thereof. In some embodiments, the encoding / decoding (codec) circuitry includes a next generation video codec (VCN) 210 controlled by a dedicated video reduced instruction set computing processor (RISC). In other embodiments, the codec circuitry includes a universal video decoder (UVD) / video compression engine (VCE) 215, which is implemented as a fixed hardware IP controlled by a dedicated RISC processor, which may be the same or different from the RISC processor used to implement the VCN 210. The VCN 210 and UVD / VCE 215 are alternative implementations of the encoding / decoding circuitry, and the illustrated embodiment of the multi-core processing unit 205 is implemented using the VCN 210 and does not include the UVD / VCE 215, as shown by the dashed box representing the UVD / VCE 215. Firmware is used to configure the VCN 210 and the UVD / VCE 215. Different firmware configurations associated with different guest VMs are stored in a subset of registers associated with the guest VM to facilitate world switching between guest VMs, as discussed in detail below.

[0032] The multi-core processing unit 205 also includes a bridge 220, such as a south bridge, for providing an interface between the multi-core processing unit 205 and interfaces of peripheral devices. In some embodiments, the bridge 220 connects the multi-core processing unit 205 to one or more PCIe interfaces 225, one or more universal serial bus (USB) interfaces 230, and one or more serial AT attachment (SATA) interfaces 235. Slots 240, 241, 242, 243 are provided for attaching memory elements such as double data rate (DDR) memory integrated circuits that store information for the multi-core processing unit 205.

[0033] Figure 3 300 is a block diagram of a first embodiment of a hardware architecture 300 that supports multimedia virtualization on a GPU according to some embodiments. The hardware architecture 300 includes a graphics core 302 that includes a compute unit (or other processor) to execute instructions simultaneously or in parallel. In some embodiments, the graphics core 302 includes integrated address translation logic for virtual memory management. The graphics core 302 uses flexible data routing to perform rendering operations, such as using local memory or by accessing content in system memory for performance rendering of CPU / GPU cooperative graphics processing.

[0034] The hardware architecture 300 also includes one or more interfaces 304. Some embodiments of the interfaces 304 include platform component interfaces for platform components such as voltage regulators, pin straps, flash memory, embedded controllers, south bridges, fan control, etc. Some embodiments of the interfaces 304 include interfaces to a Joint Test Action Group (JTAG) interface, a boundary scan diagnostic (BSD) scan interface, and a debug interface. Some embodiments of the interfaces 304 include display interfaces for one or more external display panels. The hardware architecture 300 also includes a system management unit 306 that manages thermal and power conditions of the hardware architecture 300.

[0035] The interconnect network 308 is used to facilitate communications with the graphics core 302, the interface 304, the system management unit 306, and other entities attached to the interconnect network 308. Some embodiments of the interconnect network 308 are implemented as an extensible control structure or system management network that provides register access and access to local data and instruction memory of fixed hardware for initialization, firmware loading, runtime control, etc. The interconnect network 308 is also connected to the video compression engine (VCE) 312, the universal video decoder (UVD) 314, the audio coprocessor 316, and the display output 318, as well as other entities such as direct memory access, hardware semaphore logic, display controllers, etc., for clarity. Figure 3 These entities are not shown in FIG.

[0036] Some embodiments of the VCE 312 are implemented as a compressed bitstream video encoder controlled using firmware executing on a local video RISC. The VCE 312 is capable of supporting multiple formats, for example, the VCE 312 encodes H.264, H.265, AV1, and other encoding or compression formats using various profiles and levels. The VCE 312 encodes from a provided YUV surface or an RGB surface with color space conversion. In some embodiments, the color space conversion and video scaling are performed on a GPU core that executes a pixel shader or a compute shader. In some embodiments, the color space conversion and video scaling are performed in a fixed function hardware video preprocessing block (for clarity) Figure 3 Executed on (not shown).

[0037] Some embodiments of UVD 314 are implemented as a compressed bitstream video decoder controlled by firmware running on a local video RISC. UVD 314 is capable of supporting multiple formats, for example, UVD 314 decodes traditional MPEG-2, MPEG-3, and VC1 bitstreams, as well as newer H.264, H.265, VP9, ​​and AV1 formats of various profiles, levels, and bit depths.

[0038] Some embodiments of the audio coprocessor 316 use local and global audio capture and rendering to perform host audio offloading. For example, the audio coprocessor 316 can perform audio format conversion, sample rate conversion, audio equalization, volume control, and mixing. The audio coprocessor 316 can also implement algorithms for audio video conferencing and voice-controlled computers, such as keyword detection, echo cancellation, noise suppression, microphone beamforming, etc.

[0039] The hardware architecture 300 includes a hub 320 for controlling various fixed function hardware blocks. Some embodiments of the hub 320 include a local GPU virtual memory address translation cache (ATC) 322 for performing address translations from virtual addresses to physical addresses. The local GPU virtual memory ATC 322 supports CPU register access and data transfer to and from a local frame buffer 324 or a buffer array stored in system memory.

[0040] The multi-level ATC 326 stores virtual address to physical address translations to support performing address translation. In some embodiments, address translations are used to facilitate access to the local frame buffer 324 and system memory 328.

[0041] Figure 4 is a block diagram of a second embodiment of a hardware architecture 400 for supporting multimedia virtualization on a GPU according to some embodiments. The hardware architecture 400 includes some Figure 3 For example, hardware architecture 400 includes graphics core 302, interface 304, system management unit 306, interconnect network 308, audio coprocessor 316, display output 318, and system memory 328. These entities are similar to the first embodiment of hardware architecture 300 shown in FIG. Figure 3 Corresponding entities in the illustrated hardware architecture 300 operate in the same or similar manner.

[0042] The second embodiment of the hardware architecture 400 is Figure 3 The first embodiment of the hardware architecture 300 shown in FIG. 4 differs in that it includes a CPU core complex 405 , a VCN engine 410 , an image signal processor (ISP) 415 , and a multimedia hub 420 .

[0043] Some embodiments of the CPU core complex 405 are implemented as a multi-core CPU system with multiple levels of cache memory that has access to system memory 328. The CPU core complex 405 also includes functional blocks for performing initialization, setup, state servicing, interrupt handling, etc. (for clarity, Figure 4 not shown).

[0044] Some embodiments of the VCN engine 410 include a multimedia video subsystem that includes an integrated compressed video decoder and video encoder. The VCN engine 410 is implemented as a video RISC processor that is configured to use firmware to perform priority-based decoding and encoder scheduling. The firmware scheduler submits decoding and encoding jobs to the kernel mode driver using a set of hardware-assisted queues. For example, the firmware executed on the VCN engine 410 uses a decoding queue that runs with a normal priority queue and an encoding queue that runs with normal, real-time, and time-critical priority levels. Other parts of the VCN engine 410 include:

[0045] a. Traditional MPEG-2, MPEG-4 and VC-1 decoders with fixed hardware IP blocks for hardware accelerated inverse entropy, inverse transform, motion predictor, deblocker decoding process steps, and register interface for setup and control.

[0046] bH264, H.265, and VP9 encoder and decoder subsystem with fixed hardware IP blocks for hardware accelerated inverse entropy, integer motion estimation, entropy coding, inverse transform and interpolation, motion prediction and interpolation, and deblocking encoding and decoding processing steps, with a register interface for setup and control, and context management of the hardware state of the fixed hardware IP blocks and a memory data manager with a memory interface that supports transfer of compressed bitstreams to and from locally attached memory and graphics memory with a dedicated memory controller interface.

[0047] c. JPEG decoder and JPEG encoder, which are implemented as fixed hardware functions under the control of the video RISC processor.

[0048] d. Register sets used for JPEG decoding / encoding, video codecs, and video RISC processors.

[0049] e. Ring buffer controller with a set of circular buffers with hardware support for write transfers and read transfers supported by the video RISC processor. The circular buffers support JPEG decoding, video decoding, general encoding (for transcoding use cases), real-time encoding (for video conferencing use cases), and time-critical encoding for wireless display.

[0050] Some embodiments of ISP 415 capture individual frames or video sequences from a sensor via an interface such as the Mobile Industry Processor Interface (MIPI) Alliance Camera Interface (CSI-2). Thus, ISP 415 provides input video or input still pictures. ISP 415 performs image acquisition, processing, and scaling on the acquired YCbCr surface. Some embodiments of ISP 415 support multiple cameras to perform image processing simultaneously by switching the cameras connected via the MIPI interface to a single internal pipeline. In some cases, the RGB or YCbCr image surface processed by the graphics compute engine bypasses the functions of ISP 415. Some embodiments of ISP 415 use an internal direct memory access (DMA) engine to implement image processing functions such as demosaicing, noise reduction, scaling, and transferring the acquired image / video to and from the memory.

[0051] The multimedia hub 420 supports access to the system memory 328 and interfaces (such as I / O hub 430) to access peripheral input / output (I / O) devices, such as USB, SATA, general purpose I / O (GPIO), real-time clock, SMBUS interface, serial I2C interface for accessing external configurable flash memory, etc. Some embodiments of the multimedia hub 420 include a local GPU virtual memory ATC 425 for performing address translation from virtual addresses to physical addresses. The local GPU virtual memory ATC 425 supports CPU register access and data transfer into and out of the local frame buffer or buffer array stored in the system memory 322.

[0052] Figure 5 is a block diagram of an OS 500 for supporting multimedia processing in a virtualized operating system (OS) ecosystem according to some embodiments. OS 500 is a Figure 3 The first embodiment of the hardware architecture 300 shown in FIG. Figure 4 The second embodiment of the hardware architecture 400 shown in FIG.

[0053] OS 500 is divided into user mode 505, kernel mode 510, and a portion of kernel mode 515 for the hypervisor (HV) context. User mode threads are executing in a private process address space. Examples of user mode threads include system processes 520, service processes 521, user processes 522, and environment subsystems 523. System processes 520, service processes 521, and user processes 522 communicate with subsystem dynamic link libraries (DLLs) 525. As a process executes, it goes through different states (start, ready, running, waiting, and exiting or terminating). An OS process is defined as an entity that represents a basic unit of work implemented in the system for initializing and running OS 500. Operating system service processes are responsible for managing platform resources, including processors, memory, files, and input and output. OS processes typically isolate applications from the implementation details of a computer system. Operating system service processes run as:

[0054] • Kernel services, which create and manage processes and threads of execution, execute programs, define and communicate asynchronous events, define and handle system clock operations, implement security features, manage files and directories, and control input / output processing to and from peripheral devices.

[0055] Utility services for comparing, printing, and displaying file contents, editing files, searching for patterns, evaluating expressions, logging events and messages, moving files between directories, sorting data, executing command scripts, controlling printers, and accessing environment information.

[0056] Batch processing services for queuing work (jobs) and managing the order of processing based on job control commands and data instruction lists.

[0057] · File and directory synchronization services, which are used to manage local and remote copies of files and directories.

[0058] The user process runs user-defined programs and executes user code. The OS environment or integrated application environment is the environment in which the user runs application software. The OS environment is located between the OS and the application, and is composed of a user interface provided by the application manager and an application programming interface (API) of the application manager between the OS and the application. The OS environment variables are dynamic values ​​used by the operating system and other software to determine specific information such as the location on the computer, the version number of the file, the list of files or device objects, etc. Two types of environment variables are user environment variables (specific to user programs or user-supplied device drivers) and system environment variables. The NTDLL.DLL layer 530 exports the Windows native API interface used by the user mode components of the operating system, which run without the support of Win32 or other API subsystems.

[0059] The separation between user mode 505 and kernel mode 510 provides OS protection from erroneous or malicious user mode code. Kernel mode 510 includes window and graphics block 535, executive function 540, one or more device drivers 545, one or more kernel mode drivers 550, and hardware abstraction layer 555. The second dividing line separates kernel mode drivers 550 in kernel mode 510 from the OS hypervisor 560, which runs at the same privilege level as the kernel (level 0) but uses special CPU instructions to isolate itself from the kernel while monitoring the kernel and applications. This is called the hypervisor running at ring-1.

[0060] Figure 6 is a block diagram of an operating system (OS) architecture 600 with virtualization support according to some embodiments. Figure 5 The OS architecture 600 is divided into a user mode 605 including a NTDLL layer 610 (as described above with respect to Figure 5 ) and kernel mode 615. Some embodiments of OS architecture 600 implement kernel local inter-process communication or local procedure call or lightweight procedure call (LPC), which is an internal inter-process communication (IPC) facility implemented in the kernel for lightweight IPC between processes on the same computer. In some cases, LPC is replaced by asynchronous local inter-process communication with a high-speed, scalable communication mechanism for implementing the User Mode Driver Framework (UMDF), the user-mode portion of which requires an efficient communication channel with a UMDF component in the kernel.

[0061] The framework of kernel mode 615 includes one or more system threads 620, which interact with device hardware 625 such as CPU, BIOS / ACPI, buses, I / O devices, interrupts, timers, memory cache control, etc. The system service dispatcher 630 interacts with the NTDLL layer 610 in user mode 605. The framework also includes one or more callable interfaces 635.

[0062] Kernel mode 615 also includes functionality to implement caches, monitors, and managers 640. Examples of caches, monitors, and managers 640 include:

[0063] • A kernel configuration manager, which stores configuration values ​​in "INI" (initialization) files and manages a persistent registry.

[0064] • The Kernel Object Manager, which manages the lifecycle of OS resources (files, devices, threads, processes, events, mutexes, semaphores, registry keys, jobs, sections, access tokens, and symbolic links).

[0065] The kernel process manager, which handles the execution of all threads in a process.

[0066] Kernel Memory Manager, which provides a set of system services that allocate and release virtual memory, share memory between processes, map files into memory, flush virtual pages to disk, retrieve information about virtual page ranges, change the protection level of virtual pages, and lock / unlock virtual pages into memory. In user mode 605, most of these services are exposed as APIs for virtual memory allocation and reallocation, heap APIs, local and global APIs, and APIs for manipulating memory mapped files to map files as memory and share memory handles between processes.

[0067] The kernel Plug and Play (PnP) manager, which recognizes when a device is added to or removed from a running computer system and provides device detection and enumeration. Throughout its life cycle, the PnP manager maintains a device tree that tracks the devices in the system.

[0068] The kernel power manager, which manages power state changes for all devices that support power state changes. The power manager relies on power policy management to handle power management and coordinate power events, and then generates procedure calls based on power management events. The power manager collects requests to change power states, decides in which order devices must change their power states, and then sends appropriate requests for the appropriate drivers to make the changes. The policy manager monitors activity in the system and integrates user state, application state, and device driver state into power policies.

[0069] Kernel security reference monitor, which provides routines for device drivers to handle kernel access controls defined using access control lists (ACLs). It ensures that device driver requests do not violate system security policies.

[0070] Kernel mode 615 also includes a kernel I / O manager 645, which manages the communication between the application and the interface provided by the device driver. The communication between the operating system and the device driver is completed by I / O request packets (IRPs) passed from the operating system to a specific driver and from one driver to another. Some embodiments of the kernel I / O manager 645 implement file system drivers and device drivers 650. The kernel file system driver modifies the default behavior of the file system by filtering I / O operations (create, read, write, rename, etc.) of one or more file systems or file system volumes. The kernel device driver receives data from the application, filters the data and passes it to the low-level driver that supports the device function. Some embodiments of the kernel mode driver conform to the Windows Driver Model (WDM). The kernel device driver provides a software interface to the hardware device, so that the operating system and other user mode programs can access the hardware function without knowing the precise details about the hardware being used. The virtual device driver is a special variant of the device driver, which is used to simulate hardware devices in a virtualized environment. Throughout the simulation process, the virtual device driver allows the guest operating system and its drivers running in the virtual machine to access real hardware in a time-division multiplexing session. An attempt by the guest operating system to access the hardware is routed to the virtual device driver in the host operating system as, for example, a function call.

[0071] Kernel mode 615 also includes OS component 655, which provides core functions for building a simple user interface for window management (creation, resizing, repositioning, destruction), title bar and menu bar, message passing, input processing and standard controls (such as buttons, pull-down menus, edit boxes, shortcut keys, etc.). OS component 655 includes a graphics driver interface (GDI), which is based on a handle set for windows, messages and message loops. OS component 655 also includes a graphics driver kernel component that controls graphics output by implementing a graphics device driver interface (DDI). The graphics driver kernel component supports initialization and termination, floating-point operations, graphics driver functions, the creation of device-related bitmaps, graphics output functions for drawing lines and curves, drawing and filling, copying bitmaps, halftones, image color management, graphics DDI color and palette functions, and graphics DDI font and text functions. The graphics driver supports entry points (e.g., called by GDI) to enable and disable the driver.

[0072] Kernel mode 615 includes kernel and kernel mode driver 660. Graphics kernel driver does not directly manipulate hardware. Instead, graphics kernel driver calls functions in hardware abstraction layer (HAL) 665 to interface with hardware. HAL 665 supports OS porting to various hardware platforms. Some embodiments of HAL 665 are implemented as loadable kernel mode modules (Hal.dll), which enable the same operating system to run on different platforms with different processors. In the illustrated framework, hypervisor 670 is implemented between HAL 665 and device hardware 625.

[0073] Figure 7 is a block diagram of a multimedia software system 700 for compressed video decoding, rendering, and presentation according to some embodiments. The multimedia software system 700 is a Figure 3 The first embodiment of the hardware architecture 300 shown in FIG. Figure 4 The multimedia software system 700 is implemented in the second embodiment of the hardware architecture 400 shown in FIG.

[0074] The user mode 705 of the multimedia software system 700 includes an application layer 715. Some embodiments of the application layer 715 execute applications such as metro applications, modern applications, immersive applications, store applications, etc. The application layer 715 interacts with a runtime layer 720, which provides connections to other layers and drivers used to support multimedia processes, as discussed below.

[0075] The hardware Media Foundation Transform (MFT) 725 is implemented in user mode 705. The MFT 725 is an optional interface available to application programmers. In some embodiments, a separate MFT 725 instance is provided for each decoder and encoder. The MFT 725 provides a common model for processing media data and is used for decoders and encoders that have one input and one output stream in the MFT representation. Some embodiments of the MFT 725 implement a processing model based on a previously defined application programming interface (API) with complete underlying hardware abstraction.

[0076] The media foundation (MF) layer 730 implemented under user mode 705 is used to provide a media software development kit (SDK) for the multimedia software system 700. The media SDK defined by the MF layer 730 is a media application framework that allows application programmers to access the CPU and compute shaders implemented in the GPU, and the hardware accelerators (such as accelerator functions) for media processing are implemented as physical functions provided by fixed-function hardware blocks. Examples of accelerator functions implemented by physical functions include encoding of multimedia data streams, decoding of multimedia data streams, encoding / decoding of audio or video data, or other operations. In some embodiments, the media SDK includes programming examples that illustrate how to implement video playback, video encoding, video transcoding, remote display, wireless display, etc.

[0077] Multimedia user mode driver (MMD) 735 provides an internal, OS-independent API set for MF layer 730. Some embodiments of MMD 735 are implemented as C++-based drivers that abstract the hardware of the processing system for implementing the multimedia software system 700. MMD 735 is connected to one or more graphics pipelines (DX) 740 (such as DirectX9 and DirectX11 pipelines) interfaces, which include components for allocating memory, video services, or graphics surfaces with different attributes. In some cases, MMD 735 operates under a specific OS ecosystem because it includes an OS-specific implementation.

[0078] Kernel mode 710 includes kernel mode drivers 745 that support hardware acceleration and rendering of a 3D graphics pipeline. Some embodiments of a 3D graphics pipeline include, among other elements, an input assembler, a vertex shader, a tessellator, a geometry shader, a rasterizer, a pixel shader, and output merging of rendering memory resources such as surfaces, buffers, and textures. Elements of the 3D pipeline are implemented as software-based shaders and fixed-function hardware.

[0079] The firmware interface 750 is used to provide firmware for configuring hardware 755, which is used to implement the accelerator function. Some embodiments of the hardware 755 are implemented as a dedicated video RISC processor, which receives instructions and commands from the user mode 705 via the firmware interface 750. The firmware is used to configure one or more of the UVD, VCE and VCN, such as Figure 1 The fixed function hardware block 165 shown in Figure 2 The VCN 210 shown in Figure 2 UVD / VCE 215, Figure 3 VCE 312, shown in Figure 3 UVD 314, and Figure 4. The commands received through the firmware interface 750 are used to initialize and prepare the hardware 755 for video decoding and video encoding. Content information is passed from the MMD 735 to the kernel mode driver 745 as a decoding and / or encoding job through a circular or ring buffer system. Buffers and surfaces are passed with their virtual addresses, which are translated into physical addresses in the kernel mode driver 745. Examples of content information include information indicating the allocated compressed bitstream buffer, decoding surface (called decoding context), decoding picture buffer, decoding target buffer, encoding input surface, encoding context, and encoding output buffer.

[0080] Kernel mode 710 also includes 3D driver 760 and platform security processor (PSP) 765. PSP 765 is a kernel mode component that provides encryption API and methods for decrypting and / or encrypting surfaces at the input and output of compressed bitstream decoders. PSP 765 also provides encryption API and methods at the output of video encoders. For example, PSP 765 can enforce the use of HDCP 1.4 and 2.x standards for content protection at the display physical output or virtual display for AMD WiFi Display or Microsoft Miracast Session.

[0081] Virtualization is the separation of service requests from their physical delivery. It can be accomplished by using:

[0082] • Binary translation of OS requests between the guest OS and the hypervisor (or VMM) running on top of the host computer hardware layer.

[0083] • OS-assisted paravirtualization, where the guest OS passes all requests to the hypervisor to stress the hardware, and the hypervisor provides software interfaces for memory management, interrupt handling, and time management.

[0084] Hardware-assisted virtualization enabled by AMD-v technology, which allows the VMM to run at an elevated privilege level below that of kernel-mode drivers. A hypervisor or VMM that runs on top of the hardware layer is called a bare-metal type 1 hypervisor. If it runs on top of a native (host) OS, it is called a type 2 hypervisor.

[0085] Virtualization is used in computer client and server systems. Virtualization allows different OS (or guest VM) to share multimedia hardware resources (hardware IP) in a seamless and controlled manner. Each OS (or guest VM) is unaware of the presence of other OS (or guest VM) within the same computer system. To reduce the number of interrupts to the main CPU, the sharing and collaboration of workloads from different guest VMs is managed by a multimedia hardware scheduler. In client-based virtualization, the host OS shares the GPU and multimedia hardware between guest VMs and user applications. Server use cases include virtualized desktop sharing (screen data H.264 compressed to reduce network traffic), cloud gaming, virtual desktop interface (VDI), and compute engine sharing. Desktop sharing is closely related to the use of VCN video encoders.

[0086] Single root I / O virtualization (SR-IOV) is an extension of the PCI express specification that allows segmented access to hardware resources using a PCIe physical function (PF) and one or more virtual functions (VFs). The physical function is used under the native (host OS) and its driver. Some implementations of the physical function are implemented as PCI Express functions, which include SR-IOV capabilities for configuring and managing physical functions and associated virtual functions, which are associated with corresponding physical functions and enabled in a virtualized environment. Virtual functions allow sharing of system memory, graphics memory (frame buffer), and various devices (hardware IP blocks). Each virtual function is associated with a single physical function. The GPU exposes one physical function according to each PCIe standard, and the PCIe exposure depends on the type of OS environment.

[0087] In the native (host OS) environment, native user-mode and kernel-mode drivers use physical functions, and all virtual functions are disabled. All GPU registers are mapped to physical functions via trusted access.

[0088] In a virtual environment, physical functions are used by the hypervisor (host VM), and the GPU exposes a certain number of virtual functions, such as one virtual function per guest VM, according to the PCIe SR-IOV standard. Each virtual function is mapped to a guest VM by the hypervisor. Only a subset of registers is mapped to each virtual function. Register access is limited to only one guest VM at a time, i.e., only to the active guest VM, where access is granted by the hypervisor. The active guest VM that has been granted access by the hypervisor is called "focused". Each guest VM has access to a subset of the register set, which is divided into a frame buffer, context registers, and a doorbell aperture for VF–PF synchronization. At any given time, only one focused guest VM is allowed to perform graphics rendering on its own frame buffer partition. Other guest VMs are denied access. Each virtual function has its own system memory (SM) and GPU frame buffer (FB). Each guest VM has its own user-mode driver and firmware image (i.e., each guest VM runs its own copy of firmware for any multimedia function (camera, audio, video decoding and / or video encoding)). To enforce ownership and control of hardware resources, the hypervisor uses the CPU MMU and the device IOMMU.

[0089] Figure 8 8 is a block diagram of a physical function configuration space 800 that identifies base address registers (BARs) for a physical function according to some embodiments. The physical function configuration space 800 includes a set of physical function BARs 805, including a frame buffer BAR 810, a doorbell BAR 815, an I / O BAR 820, and a register BAR 825. The configuration space 800 maps the physical function BARs to specific registers. For example, the frame buffer BAR 810 maps to the frame buffer register 830, the doorbell BAR 815 maps to the doorbell register 835, the I / O BAR 820 maps to the I / O space 840, and the register BAR 825 maps to the register space 845.

[0090] Fig. 9 900 is a block diagram of a portion 900 of a single root I / O virtualization (SR-IOV) header that identifies a BAR for a virtual function according to some embodiments. Portion 900 of the SR-IOV header includes fields that store information identifying virtual function BARs that can be used to assign to corresponding guest VMs executing on a processing system. In the illustrated embodiment, portion 900 indicates virtual function BARs 901, 902, 903, 904, 905, 906, which are collectively referred to herein as virtual function BARs 901-906. The mapping indicated by the virtual function BARs 901-906 in portion 900 is used to divide a register set into subsets associated with different guest VMs.

[0091] In the illustrated embodiment, the information in portion 900 is mapped to BARs in an SR-IOV BAR set 910. The set includes a frame buffer BAR 911, a doorbell BAR 912, an I / O BAR 913, and a register BAR 914, which include information pointing to corresponding register subsets in a register set 920. The set 920 is divided into subsets that serve as frame buffers, doorbells, and context registers for corresponding guest VMs. In the illustrated embodiment, the frame buffer BAR 911 includes information identifying a subset of registers (also referred to as apertures), which include registers for holding frame buffers 921, 922 for the guest VM. The doorbell BAR 911 includes information identifying a subset of registers, which include registers for holding doorbells 923, 924 for the guest VM. The I / O BAR 913 includes information identifying a subset of registers, which include registers for holding I / O space 925, 926 for the guest VM. Register BAR 914 includes information identifying a subset of registers including registers used to hold context registers 927, 928 for a guest VM.

[0092] With respect to the frame buffer aperture including frame buffers 921, 922, in some embodiments, the actual size of the frame buffer is larger than that of the frame buffers provided by VF BARs 901-906 (or Figure 8 The size of the PF BAR 805 shown in FIG. 8 is exposed, and a private GPU-IOV capability structure is introduced in the PCI configuration space as a communication channel for the hypervisor to interact with the GPU to partition the frame buffer. Through the GPU-IOV structure, the hypervisor can allocate frame buffers of different sizes to each virtual function, which is referred to as frame buffer partitions in this article.

[0093] The GPU doorbell is a mechanism for an application or driver to indicate to the GPU engine that it has enqueued work into the active queue. The doorbell is issued by software running on the CPU or GPU. On the GPU, the doorbell can be issued by any client that can generate a memory write, such as by a CP (command processor), SDMA (system DMA engine), or CU (compute unit). In some embodiments, the 64-bit doorbell BAR 912 points to the starting address of the doorbell aperture of the virtual function associated with the physical function. Within the doorbell aperture, each ring for command submission has its own doorbell register 923, 924 to signal through an interrupt that the contents of the ring buffer have changed. The interrupt is serviced by the video CPU (VCPU), and the decoding or encoding job is removed from the ring buffer and processed by the CPU, which responds to the interrupt to start the video decoding or video encoding process on the dedicated decoding or encoding hardware.

[0094] Registers are divided into four categories:

[0095] Hypervisor-specific registers are accessible only by the hypervisor. These registers are mirror images of the GPU-IOV registers in the PCIe configuration space.

[0096] PF-specific registers are accessible only by physical functions. Any read from a virtual function returns zero; any write from a virtual function is discarded. Display controller and memory controller registers are PF-specific.

[0097] PF or VF registers can be accessed by both virtual and physical functions, but only the virtual function's dedicated physical function can access such registers when the virtual function or physical function becomes the active function and thus owns the GPU. Register settings for a physical function or virtual function are only effective when that function is the active function. When the physical function of a virtual function is active, the corresponding driver cannot access such registers.

[0098] PF and VF registers can be accessed by both physical and virtual functions; each virtual or physical function has its own copy of the registers. Register settings in different functions can be effective at the same time. Interrupt registers, VM registers, and index / data registers belong to the PF and VF copy categories.

[0099] Fig.10 is a block diagram of the life cycle 1000 of a host OS implementing a physical function and a guest VM implementing a virtual function associated with the physical function according to some embodiments. In some embodiments, the graphics driver carries embedded firmware images for the following entities:

[0100] SMU (System Management Unit)

[0101] MC (Memory Controller)

[0102] ME (Micro Engine – Replica Graphics)

[0103] PFP (Prefetch Parser – CPF)

[0104] CE (Constant Engine – CP)

[0105] · Calculation (computation engine)

[0106] System DMA (sDMA)

[0107] ·RLC_G

[0108] DMIF (Display Management Interface)

[0109] UVD, VCE, VCN and PSP / SAMU safety.

[0110] The firmware images for SMU, MC, and RLC_V are loaded at vBIOS power-on self-test (POST), while the other firmware images are loaded by the graphics driver during ASIC initialization and before any related firmware engines are used under SR-IOV virtualization.

[0111] The system BIOS stage 1005 includes a power-on frame 1010 and a POST frame 1015. During the power-on frame 1010, the GPU reads the corresponding fuse or strap to determine the BAR size of the virtual function. For example, the GPU can read the size REG_BAR (32b), FB_BAR (64b), DOORBELL_BAR (64b). In this case, the virtual function does not support IO_BAR. During the POST frame 1015, the system BIOS identifies the SR-IOV capability of the GPU and shakes hands with the GPU to determine the BAR size of each virtual function. In response to determining the size requirement, the system BIOS allocates enough continuous MMIO (memory mapped I / O) space to accommodate the total BAR size of the virtual function, as well as the normal PCI configuration space range requirements of the physical function. Next, the system BIOS enables the ARI capability in the root port and the ARI Capable Hierarchy bit in the SR-IOV cap for the physical function.

[0112] The hypervisor, OS startup and driver initialization phase 1020 includes a hypervisor initialization / startup box 1025 and a host OS startup box 1030. In box 1025, the hypervisor begins to initialize the virtualized environment before loading the host OS as its user interface. When the host OS (or part of the hypervisor) starts, it will load into the GPUV driver that controls the hardware virtualized GPU. In response to loading the GPUV driver, the GPUV driver performs POST VBIOS at box 1030 to initialize the GPU. During VBIOS POST, the driver loads firmware (FW), including PSP FW, SMU FW, RLC_V FW, RLC_G FW, RLC save / restore list, SDMA FW, scheduler FW and MC FW. The video BIOS reserves its own space in the frame buffer at the end of the frame buffer for the PSP to copy and verify the firmware. After VBIOS POST, the GPUV driver can enable SR-IOV and configure the resources and corresponding virtual function stages 1035, 1040 of one or more virtual functions.

[0113] In the first virtual function stage 1035, the hypervisor assigns the first virtual function to the first guest VM at box 1045. Once SR-IOV is enabled, the location of the first frame buffer is programmed for the first virtual function. For example, a first subset of the register set is assigned to the first frame buffer of the first virtual function. At box 1050, the first guest VM is initialized and the guest graphics driver initializes the first virtual function. The first virtual function responds to PCIe requests to access the frame buffer and other activities. In the last stage, when the first guest VM is assigned the first virtual function as a directly assigned device, the guest VM identifies the virtual function as a GPU device. The graphics driver shakes hands with the GPUV driver and completes the GPU initialization of the virtual function. Once the initialization is completed, the first guest VM is booted to the predefined desktop at box 1055. The end user can now log in to the first guest VM through the remote desktop protocol and start performing the desired work on the first guest VM.

[0114] In the second virtual function stage 1040, the hypervisor assigns the second virtual function to the second guest VM at box 1060, initializes the second guest VM at box 1065, and starts the second guest VM at box 1070. At this point, multiple virtual functions and corresponding guest VMs are running on the GPU simultaneously. The hypervisor schedules the time slice to the VM-VF running on the GPU. The selection of the guest VM running after the currently executing guest VM (i.e., GPU switching) is implemented by the hypervisor or by the GPU scheduling switch. When the virtual function obtains its time slice on the GPU, the corresponding guest VM owns the GPU resources, and the graphics driver running in the guest VM behaves as if it owns the GPU alone. The guest VM responds to all command submissions and register accesses during its allocated time slice.

[0115] In processing units that do not contain a multimedia scheduler (MMSCH), programming of the multimedia engine and its lifecycle control is done by the main x64 or x86 CPU. In such modes, video encoding and / or video decoding firmware loading and initialization is done by the virtual function driver at its initial load. At runtime, each loaded virtual function instance has its own firmware image and performs firmware and register context restore, retrieves only one job from its own queue, encodes a complete frame, and performs context save. When a virtual function instance reaches idle time, it notifies the hypervisor that the hypervisor can load the next virtual function.

[0116] If present, the MMSCH assumes and takes over the CPU role in the management multimedia engine. It performs initialization and setup of virtual functions, context save / restore, submits jobs in the guest VM to the virtual function through doorbell programming, and performs reset of physical and virtual functions, as well as error recovery processing. Some implementations of the MMSCH are implemented as firmware on a low-power VCPU. The firmware for MMSCH and MMSCH initialization is executed by the Platform Security Processor (PSP), whose firmware is contained in the video BIOS (vBIOS). The PSP downloads the MMSCH firmware image by using the ADDRESS / DATA register pair with auto-increment function, programs its configuration registers, and brings the MMSCH firmware image out of reset. As soon as the MMSCH is running, the hypervisor performs the setup of multimedia virtual functions by programming SR-IOV and GPU-IOV capabilities. The hypervisor configures BARs for physical and virtual functions, performs multimedia initialization in the guest VM, and enables the guest VM to run sequentially. Multimedia initialization requires allocating memory in each guest VM to save the VCE and UVD (or VCN) virtual registers and corresponding firmware. The hypervisor then programs registers for the VCE / UVD or VCN hardware by setting the address and size of the aperture into which the firmware is loaded. The hypervisor also sets registers that define the address start and stack size for the firmware engine and its instruction and data caches. The hypervisor then programs the local memory interface (LMI) configuration registers and removes reset from the corresponding VCPU.

[0117] Some implementation plans of MMSCH perform the following activities:

[0118] · For PF and VF functions. With bare metal platforms, the driver initializes the VCE or UVD engine via direct MMIO register reads / writes. Under virtualization, the MM engine virtualization has the ability to execute one function while another function is being initialized. This capability is supported by submitting the initialization memory descriptor to the MMSCH, which will schedule and trigger the VF's multimedia engine initialization at a later time when the first command submission occurs.

[0119] ·For PF and VF functions. With a bare metal platform, command submission for VCE and UVD (or VCN) is done through the MMIO WPTR register such as VCE_RB_WPTR. Under virtualization, command submission switches to doorbell writes, which are similar to GFX, SDMA, and Compute command submission. To submit a command packet to a ring / queue, the GFX driver needs to write to the corresponding doorbell location. When writing to the doorbell location, the MMSCH receives notification for that VF and ring / queue. The MMSCH saves such information internally for each function and ring / queue. When the function becomes the active function, the MMSCH notifies the corresponding engine to start processing the accumulated command packets of the ring / queue.

[0120] Multimedia world switching means switching from the currently running multimedia VF instance to the next multimedia VF instance. Multimedia world switching is accomplished using several command exchanges between the MMSCH firmware and the UVD / VCE / VCN firmware of the currently running and next to run multimedia firmware instances. Commands are exchanged via a simple INDEX / DATA common register set found in the MMSCH and VCE / UVD / VCN. In some embodiments, there are the following commands:

[0121] gpu_idle(fcn_id) – requests the MM engine to stop processing any commands for the current function. If the MM engine is currently executing the function, the MMSCH waits until the MMSCH receives the current job completion from the MM engine, stopping any further processing of any additional commands for the function; otherwise, the MMSCH returns the command completion immediately.

[0122] gpu_save_state(fcn_id) – MMSCH saves the engine state of the current function fcn_id to the context save area.

[0123] gpu_load_state(fcn_id) – MMSCH loads the engine state of function (fcn_id) from the context SRAM area to the engine registers.

[0124] gpu_run(fcn_id)—MMSCH notifies the MM engine to start processing of the function (VFID=fcn_id).

[0125] gpu_context_switch(fcn_id, nxt_fcn_id)—MMSCH waits for the MM engine to finish processing jobs on the supplied function VFID=fcn_id and switches to processing jobs on the next function specified by the nxt_fcn_id parameter.

[0126] gpu_enable_hw_autoscheding(active_functions) – This command tells the MMSCH to perform world switches between the VM functions listed in the register array. During a MM engine world switch, each function in the list remains active for the time slice specified by the register.

[0127] gpu_init(fcn_id) – This command informs the MMSCH that the engine of a specific function (fcn_id) will undergo initialization.

[0128] gpu_disable_hw_autoscheduling(active_functions) – This command notifies MMSCH to stop performing MM engine world switching for the listed functions. After receiving this command, MMSCH waits for the current active function to complete its work (frame), then executes gpu_idle and gpu_save_state commands and stays at the current active function for further operations.

[0129] gpu_disable_hw_scheduling_and_context_switch – This command asks MMSCH to stop executing world switching. After receiving this command, MMSCH waits for the current active function to complete its work, and then executes the gpu_context_switch command to switch to the next function for further operation.

[0130] Bare metal multimedia page fault handling, when UVD or VCE command execution encounters a page fault, MC / VM notifies the UVD / VCE HW block of the page fault and raises an interrupt to the host. After that, UVD / VCE and KMD perform the following operations:

[0131] When UVD receives a page fault notification, it notifies the ring / queue that caused the page to the UVD firmware via an internal interrupt.

[0132] • The UVD firmware drains (drops) all requests for this ring / queue.

[0133] The UVD firmware then resets the engine and restarts the VCPU.

[0134] After the VCPU restarts, the UVD firmware polls any new commands in its own ring buffer.

[0135] When KMD receives a page fault interrupt, KMD will read the multimedia status register to find out which ring / queue has the page fault. After retrieving the page fault ring information, KMD resets the read / write pointers of the ring / queue that has the fault to zero and indicates to the UVD / VCE / VCN firmware that the page fault has been handled so that the FW can continue / start processing the submitted commands again.

[0136] In the above solution, the handshake between UVD / VCE firmware and KMD driver is done through UVD_PF_STATUS and VCE_PAGE_FAULT_STATUS registers.

[0137] Under SR-IOV virtualization, the page fault handshaking scheme is based on memory location, since there are no other PF and VF registers to rely on.

[0138] Fig.11 1 is a block diagram of a multimedia user mode driver 1100 and a kernel mode driver 1105 according to some embodiments. The decoding and encoding bandwidth of hardware accelerators such as VCE / UVD / VCN engines is limited, and therefore the hardware accelerator is not always able to properly serve all enabled virtual functions during runtime. Some embodiments of processing units such as video GPUs schedule or allocate VCE / UVD / VCN encoding or decoding engine bandwidth to specific virtual functions based on the configuration file of the corresponding guest VM. If the configuration file of the guest VM indicates that video encoding bandwidth is required, the GPU generates a message that is passed down to the virtual function through the mailbox register before the graphics driver begins to initialize the virtual function. In addition, the GPU also notifies the scheduler of the virtual function bandwidth requirements before the virtual function begins any job submission. For example, the VCE is capable of H.264 video encoding, where the maximum bandwidth is approximately 2M MB per second - one MB is equal to 16x16 pixels. The maximum bandwidth information is stored in the video BIOS table along with the maximum surface width and height (e.g., 4096x2160). During initialization, the GPU driver retrieves the bandwidth information as the initial total available bandwidth to manage the encoding engine bandwidth allocation. Some implementations of the GPU convert bandwidth information into profiles / partitions.

[0139] In the illustrated embodiment, the multimedia user mode driver 1100 and the kernel mode driver 1105 are multi-layered and consist of functional blocks. In operation, the multimedia user mode driver 1100 includes an interface 1110 of an operating system (OS) ecosystem 1115. Some embodiments of the interface 1110 include software components, such as interfaces for different graphics pipeline calls. For example, the multimedia user mode driver 1100 uses the UDX and DXX interfaces implemented in the interface 1110 when allocating surfaces of various sizes, in various color spaces, and in various tile formats. In some cases, the multimedia user mode driver 1100 also has direct DX9 and DX11 video DDI interface displays implemented in the interface 1110. The multimedia user mode driver 1100 also implements interfaces for communicating with media foundations such as Figure 7 The multimedia user mode driver 1100 is a private API set connected via an interface (MF layer 730 shown in ), which provides interactive interfaces to other media APIs and frameworks, such as in Windows, Linux and Android OS ecosystems. Some embodiments of the multimedia user mode driver 1100 use events replaced from external components (e.g., AMF and AMDUI CCC control panel). The multimedia user mode driver 1100 also implements a set of utility programs and helper functions that allow the OS to use synchronization objects (flags, semaphores, mutexes), timers, networking socket interfaces, video security, etc. independently. Some embodiments of the underlying internal structure of the multimedia user mode driver 1100 are organized around core base class objects written in C++. The multimedia core implements a set of OS and hardware-independent base classes and provides support for the following:

[0140] Compressed bitstream video decoding supporting multiple codecs and video resolutions

[0141] Video encoding from surfaces in YUV or RGB color space to H.264, H.265, VP9, ​​and AV1 compressed bitstreams

[0142] Video rendering that supports color space conversion and upscaling / downscaling of received or generated surfaces. There are other video rendering features such as gamut correction, de-interlacing, face detection, skin tone correction, which are automatically enabled by the AMD Multimedia Feature Selector (AFS) and Capability Manager (CM) and run as shaders on the Graphics Compute Engine.

[0143] The class derived for the multimedia user mode driver 1100 is OS specific. For example, Core Vista (for supporting the Windows OS ecosystem of all variants from Windows XP via Windows 7 to Windows 10), Core Linux, and Core Android all have multimedia core functions. These cores provide the portability of the multimedia software stack to other OS environments. Device portability is ensured by automatically detecting the multimedia hardware layer of the underlying device. Communication with the kernel mode driver 1105 is achieved through IOCTL (escape) calls.

[0144] The kernel mode driver 1105 includes a kernel interface 1120 of the OS kernel, which receives all kernel-related device-specific calls (such as DDI calls). The kernel interface 1120 includes a dispatcher that dispatches calls to appropriate modules of the kernel mode driver 1105 that abstract different functions. The kernel interface 1120 includes an OS manager that controls the interaction with OS-based service calls in the kernel. The kernel mode driver 1105 also includes a kernel mode module 1125, such as an engine node (UVD engine node) for multimedia decoding, an engine node (VCE engine node) for multimedia encoding, and an engine node for multimedia video codecs (VCN nodes for APU SOCs). The kernel mode module 1125 provides hardware initialization and allows decoding or encoding jobs to be submitted to the hardware-controlled ring buffer system. The topology translation layer 1130 isolates nodes from services and provides interface connections for software modules 1135 in the kernel mode driver 1105. Examples of software modules 1135 include swUVD, swVCE, and swVCN, which are hardware-specific modules that provide access to the ring buffer to receive and process decode or encode jobs, control tiling, control power gating, and respond to IOCTL messages received from user-mode drivers. Kernel-mode drivers 1105 also provide access to hardware IP 1140 through the hypervisor in kernel-HV mode 1145.

[0145] Fig.12 is the first part 1200 of a message sequence supporting multimedia capability sharing in a virtualized OS ecosystem according to some embodiments. The message sequence Figure 112 is implemented in some embodiments of the processing system 100 shown in . The first part 1200 shows the messages exchanged between the video BIOS (VBIOS), the hypervisor (HV), the kernel mode driver topology translation layer (TTL-PF) for the physical function, the multimedia UMD for the virtual function, the kernel mode driver TTL (TTL-VF) for the virtual function, and the kernel mode driver (KMD) for the virtual function. The communication between the physical function and the virtual function is completed via a mailbox message exchange protocol with doorbell signaling. In some embodiments, the mailbox operates via a common register set, and the doorbell signaling allows interrupt-based notifications in the physical function or virtual function to occur. In other embodiments, communication is achieved via a local shared memory with doorbell signaling.

[0146] The VBIOS determines whether the system can support SR-IOV, and if so, the VBIOS provides information indicating frame buffer segmentation to the hypervisor (at message 1202). The information may include a feature flag indicating frame buffer segmentation for UVD / VCE / VCN. Each supported instance of a virtual function associated with a physical function obtains (at message 1204) a record specific to the automatic identification device in its own frame buffer. The record indicates the maximum multimedia capability, such as 1080p60 or 4K30 or 4K60 or 8K24 or 8K60, which is the sum of all activities that can be sustained on a given device. In some embodiments, the bandwidth is exhausted by only one virtual function using decoding or encoding or both functions. For example, if the total multimedia capability is 4K60, it can support four virtual functions, each performing 1080p60 decoding, or up to ten virtual functions, each performing 1080p24 decoding, or two virtual functions, each performing 1080p60 decoding, and two virtual functions, each performing 1080p60 video encoding.

[0147] When an application on a guest OS / VM running on a virtual function loads a multimedia driver for a decoding or encoding use case, the loaded multimedia driver learns about the current encoding or decoding profile and sends a request to the TTL layer of the KMD driver (in message 1206). The request can express one of the following:

[0148] 1) The current resolution of the decoding or encoding operation, which indicates the horizontal and vertical size and refresh rate of the source (e.g. 720p24, 108030, etc.) or

[0149] 2) The total number of macroblocks that need to be decoded in the coded frame or compressed bitstream content

[0150] The TTL-VF in the current virtual function receives the request and forwards it to the TTL layer of the physical function (message 1208). The TTL-PF knows the maximum decoding or encoding bandwidth and keeps track of the multimedia usage of each virtual function.

[0151] If encoding or decoding capabilities are not available, the PF TTL notifies the TTL-VF (via message 1210), which then notifies the UMD in the same virtual function (via message 1212). In response to message 1212, the UMD fails the application's request to load the multimedia driver in the virtual function, and the application closes at activity 1214.

[0152] If the encoding or decoding capability is available, the PF TTL updates its bookkeeping record and notifies the TTL-VF (via message 1216), which sends a request to the KMD (via message 1218) to download the firmware, open and configure the UVD / VCE or VCN multimedia engine (at message 1218). The KMD then becomes able to run, and the KMD node in the virtual function then notifies the TTL-VF that it can accept the first job submission (at message 1220). In response to message 1220, the TTL-VF notifies the UMD for the virtual function that its configuration process has been completed (at message 1222).

[0153] Fig.13 is a second portion 1300 of a message sequence supporting multimedia capability sharing in a virtualized OS ecosystem according to some embodiments. The second portion 1300 of the message sequence is Figure 1 In some embodiments of the processing system 100 shown in FIG. Fig.12 1200. The second portion 1300 shows messages exchanged between the video BIOS (VBIOS), the hypervisor (HV), the kernel mode driver topology translation layer for physical functions (TTL-PF), the multimedia UMD for virtual functions, the kernel mode driver TTL for virtual functions (TTL-VF), and the kernel mode driver (KMD) for virtual functions.

[0154] During normal run-time operation, a multimedia application (eg, UMD) submits encoding or decoding job requests to the TTL-VF (via message 1305) at selected time intervals, and the TTL-VF notifies the appropriate node to submit and execute the requested job by transmitting message 1310 to the KMD.

[0155] During the last step of the application life cycle on the guest VM, the application issues a shutdown request to the multimedia driver at the TTL-VF. The TTL-VF forwards the request to the TTL-VF via message 1315. The TTL-VF issues a shutdown request (via message 1320) to the corresponding multimedia node, which notifies the TTL-VF (via message 1325) that the node has been shut down. After successfully deactivating the multimedia node, the TTL-VF signals the TTL-PF (via message 1330), which then reclaims the encoding or decoding bandwidth and updates its bookkeeping records (at activity 1335).

[0156] After completing a job submitted for a virtual function, the TTL-VF signals the multimedia scheduler that the job has been executed on the virtual function. The multimedia scheduler deactivates the virtual function. The multimedia scheduler then performs a world switch to the next active virtual function. Some embodiments of the multimedia scheduler use a cyclic scheduler to activate and service virtual functions. Other embodiments of the multimedia scheduler use a dynamic priority-based scheduler, in which the priority is evaluated based on the queue type used by the corresponding virtual function. In other embodiments, the multimedia scheduler implements a rate monotonic scheduler that serves a guest VM with a decoding or encoding job with a lower resolution (e.g., a shorter job interval) than a guest VM using a priority-based queue system (e.g., a time-critical queue for encoding jobs of a Skype application with minimal latency, or a real-time queue for encoding jobs of a wireless display session, a general encoding queue for non-real-time video transcoding, or a general decoding queue).

[0157] Fig.12 and Fig.13 Some embodiments of the message sequence disclosed in support sharing a multimedia hardware engine between many virtual functions serving each guest OS / VM. This creates an impression that each guest OS / VM has its own dedicated multimedia hardware, but shares one hardware instance to serve many virtual clients. In the simplest case, the number of virtual functions that allow the host and guest OS to run hardware accelerated video decoding or hardware accelerated video encoding simultaneously is two. In yet another embodiment, up to sixteen virtual functions are supported, but other embodiments support more or fewer virtual functions.

[0158] Fig.12 and Fig.13Some embodiments of the message sequences disclosed in are used in various computer client and server systems. In client-based virtualization, the host OS shares GPU and multimedia hardware intellectual property (IP) blocks between virtual machines (VMs) and user applications. Server use cases include desktop sharing (the captured screen data is H.264, which is compressed to reduce network traffic), cloud gaming, virtual desktop interface (VDI), and compute engine sharing.

[0159] The present application can be further understood with reference to the following examples:

[0160] Embodiment 1: A processing unit, comprising:

[0161] a kernel mode unit configured to execute a hypervisor and a guest virtual machine (VM);

[0162] a fixed function hardware block configured to implement a physical function, wherein a virtual function corresponding to the physical function is exposed to the guest VM; and

[0163] A register set, wherein a subset of the register set is allocated for storing information associated with the virtual function, and wherein the fixed function hardware block performs one of the virtual functions for one of the guest VMs based on the information stored in a corresponding one of the subsets.

[0164] Embodiment 2: The processing unit of embodiment 1, wherein the register set is divided into a number of subsets corresponding to a maximum amount of space allocated to the virtual function.

[0165] Embodiment 3: The processing unit of embodiment 1, wherein the register set is initially divided into a number of subsets corresponding to a minimum amount of space allocated to the virtual function, and wherein the number of subsets is subsequently modified based on attributes of the virtual function.

[0166] Embodiment 4: A processing unit according to any one of embodiments 1 to 3, wherein each subset of the register set includes a frame buffer for storing frames operated by the virtual function associated with the subset, a context register for defining an operating state of the virtual function, and a doorbell for signaling that the virtual function is ready to be scheduled for execution.

[0167] Embodiment 5: According to the processing unit described in Embodiment 4, the processing unit further includes:

[0168] A scheduler is configured to schedule a first one of the guest VMs to execute a first one of the virtual functions within a first time interval in response to signaling from the first guest VM.

[0169] Embodiment 6: The processing unit of Embodiment 5, wherein the hypervisor permits the first guest VM to access a first subset of the register set during the first time interval, and wherein the hypervisor denies unscheduled guest VMs access to the register set during the first time interval.

[0170] Embodiment 7: The processing unit of Embodiment 6, wherein the fixed function hardware block is configured to perform the first virtual function based on information stored in a first context register in the first subset of the register set.

[0171] Embodiment 8: The processing unit according to embodiment 7, wherein at least one of a user mode driver and a firmware image for implementing the multimedia function of the first virtual function is installed on the fixed function hardware block.

[0172] Embodiment 9: The processing unit of embodiment 7, wherein the first guest VM writes information to a doorbell register in the first subset to signal to the scheduler that the first guest VM is ready to be scheduled for execution.

[0173] Embodiment 10: The processing unit of Embodiment 9, wherein the first guest VM is scheduled based on a priority associated with the guest VM and other priorities associated with other guest VMs that are ready to be scheduled.

[0174] Embodiment 11: The processing unit of Embodiment 9, wherein the first guest VM performs graphics rendering on frames stored in frame buffers in the first subset using the first virtual function during the first time interval.

[0175] Embodiment 12: The processing unit of Embodiment 11, wherein the first guest VM notifies the hypervisor in response to completing execution during the first time interval, and wherein the doorbell registers in the first subset are cleared in response to completing execution during the first time interval.

[0176] Embodiment 13: A method comprising:

[0177] receiving, at a hypervisor and from a first guest virtual machine (VM) executing in a processing unit, a request to access a first virtual function corresponding to a physical function implemented on a fixed function hardware block in the processing unit;

[0178] granting, from the hypervisor and to the first guest VM, access to a first subset of a set of registers in the processing unit, wherein the first subset stores information associated with the first virtual function;

[0179] configuring the fixed function hardware block to perform the first virtual function of the first guest VM based on the information stored in the first subset; and

[0180] Using the first guest VM, graphics rendering is performed on the frames stored in the first subset using the fixed-function hardware blocks configured to implement the first virtual function.

[0181] Embodiment 14: The method according to embodiment 13, further comprising:

[0182] The register set is divided into a number of subsets corresponding to a maximum amount of space allocated to the virtual function.

[0183] Embodiment 15: The method according to embodiment 13, further comprising:

[0184] dividing the register set into a number of subsets corresponding to a minimum amount of space allocated to the virtual function; and

[0185] The number of the subsets is modified based on a property of the virtual function.

[0186] Embodiment 16: A method according to any one of embodiments 13 to 15, wherein the first subset of the register set includes a frame buffer for storing the frame operated by the first virtual function, a context register for defining the operating state of the virtual function, and a doorbell register for signaling that the virtual function is ready to be scheduled for execution.

[0187] Embodiment 17: The method according to embodiment 16, further comprising:

[0188] The first guest VM is scheduled to execute the first virtual function within a first time interval in response to signaling from the first guest VM.

[0189] Embodiment 18: The method according to embodiment 17, further comprising:

[0190] The first guest VM is granted access to the first subset of the set of registers from the hypervisor during the first time interval, and wherein the hypervisor denies access to the subset of the set of registers to unscheduled guest VMs during the first time interval.

[0191] Embodiment 19: The method according to embodiment 18, wherein configuring the first virtual function comprises: installing at least one of a user mode driver and a firmware image for implementing a multimedia function of the first virtual function on the fixed function hardware block.

[0192] Embodiment 20: The method according to embodiment 18, further comprising:

[0193] Information is written from the first guest VM to the doorbell registers in the first subset to signal that the first guest VM is ready to be scheduled for execution.

[0194] Embodiment 21: The method of Embodiment 20, wherein scheduling the first guest VM comprises: scheduling the first guest VM in response to reading the information from the doorbell register.

[0195] Embodiment 22: The method according to embodiment 21, wherein scheduling the first guest VM comprises: scheduling the first guest VM based on a priority associated with the first guest VM and other priorities associated with other guest VMs that are ready to be invoked.

[0196] Embodiment 23: The method of Embodiment 21, wherein performing the graphics rendering on the frame comprises: performing graphics rendering on the frame stored in the frame buffer in the first subset using the first virtual function during the first time interval.

[0197] Embodiment 24: A method according to embodiment 21, wherein the first guest VM notifies the hypervisor that another virtual function can be loaded for another guest VM in response to completing execution during the first time interval, and wherein the doorbell registers in the first subset are cleared in response to completing execution during the first time interval.

[0198] Embodiment 25: A method comprising:

[0199] performing graphics rendering on a frame stored in a first subset of a register set implemented in the processing unit using a first guest virtual machine (VM) executing on the processing unit, wherein the graphics rendering is performed using a first virtual function corresponding to a physical function implemented on a fixed-function hardware block, the fixed-function hardware block being configured to implement the first virtual function based on first context information stored in the first subset;

[0200] detecting, at the hypervisor, a request from a second guest VM to access a second virtual function corresponding to the physical function; and

[0201] A world switch is performed at the hypervisor and in response to the request to configure the fixed function hardware block to perform the second virtual function.

[0202] Embodiment 26: A method according to embodiment 25, wherein the second guest VM writes information to a doorbell register in a second subset of the register set to indicate that the second guest VM is ready to be scheduled, and wherein detecting the request includes reading the information from the doorbell register.

[0203] Embodiment 27: The method according to embodiment 26, further comprising:

[0204] The second guest VM is scheduled for execution during a time interval beginning at a scheduled time in response to detecting the request.

[0205] Embodiment 28: The method of Embodiment 27, wherein scheduling the second guest VM for execution during the time interval comprises: granting the second guest VM exclusive access to the register set during the time interval.

[0206] Embodiment 29: The method of Embodiment 27, wherein performing the world switch comprises performing the world switch at the scheduled time.

[0207] Embodiment 30: The method of Embodiment 29, wherein performing the world switch comprises configuring the fixed function hardware block based on second context information stored in the second subset of the register set.

[0208] Embodiment 31: The method according to embodiment 30, wherein configuring the fixed function hardware block comprises: installing at least one of a user mode driver and a firmware image for implementing the multimedia function of the second virtual function.

[0209] Embodiment 32: The method according to embodiment 30, further comprising:

[0210] Using the second guest VM, graphics rendering is performed using the second virtual function on the frame stored in the second subset of the register set.

[0211] Computer-readable storage media may include any non-transitory storage media or combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact disk (CD), digital versatile disk (DVD), Blu-ray disc), magnetic media (e.g., floppy disk, tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical system (MEMS)-based storage media. Computer-readable media may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disk or flash memory based on a universal serial bus (USB)), or coupled to a computer system via a wired or wireless network (e.g., a network accessible storage device (NAS)).

[0212] In some embodiments, certain aspects of the techniques described above may be implemented by one or more processors of a processing system that executes software. The software includes one or more executable instruction sets stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and certain data that manipulate one or more processors to perform one or more aspects of the techniques described above when executed by one or more processors. A non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, a cache, a random access memory (RAM), or other one or more non-volatile memory devices, etc. Executable instructions stored on a non-transitory computer-readable storage medium may present source code, assembly language code, object code, or other instruction formats interpreted or otherwise executed by one or more processors.

[0213] It should be noted that not all activities or elements described above in the general description are required, a portion of a particular activity or device may not be required, and one or more other activities may be performed, or elements other than those described may be included. In addition, the order in which the activities are listed is not necessarily the order in which the activities are performed. In addition, the corresponding concepts have been described with reference to specific embodiments. However, it should be understood by those of ordinary skill in the art that various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the appended claims. Therefore, this specification and the drawings will be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the present disclosure.

[0214] The benefits, other advantages, and solutions to the problems have been described above with reference to specific embodiments. However, the benefits, advantages, solutions to the problems, and any features that may make any benefit, advantage, or solution to the problem appear or become more prominent should not be interpreted as being the key, necessary, or essential features of any or all claims. In addition, the specific embodiments disclosed above are merely illustrative, because the disclosed subject matter can be modified and practiced in different but equivalent ways that are obvious to those skilled in the art who benefit from the teachings of this article. Except as described in the appended claims, it is not intended to be limited to the details of the construction or design shown herein. Therefore, it is obvious that the specific embodiments disclosed above may be changed or modified, and all such changes are considered to be within the scope of the disclosed subject matter. Therefore, the protection sought herein is as set forth in the appended claims.

Claims

1. A processing unit, comprising: a kernel mode unit configured to execute a hypervisor and to execute a plurality of guest virtual machines in parallel; a fixed function hardware block configured to implement a hardware acceleration function of the processing unit as a physical function, wherein a plurality of virtual functions corresponding to the hardware acceleration function implemented as the physical function are exposed to a plurality of guest virtual machines executed in parallel; as well as a register set, wherein subsets of the register set are each allocated for storing information associated with a corresponding virtual function of the plurality of virtual functions, and The fixed-function hardware block represents a first guest virtual machine among a plurality of guest virtual machines executed in parallel, executes a first virtual function based on information stored in a first subset associated with the first virtual function, and represents a second guest virtual machine among a plurality of guest virtual machines executed in parallel, executes a second virtual function based on information stored in a second subset associated with the second virtual function. 2 . The processing unit of claim 1 , wherein the register set is divided into a number of subsets corresponding to a maximum amount of space allocated to the plurality of virtual functions.

3. The processing unit of claim 1 , wherein the register set is initially divided into a number of subsets corresponding to a minimum amount of space allocated to the plurality of virtual functions, and wherein the number of subsets is subsequently modified based on attributes of the virtual functions.

4. The processing unit of claim 1 , wherein each subset of the register set comprises a frame buffer for storing frames operated by a corresponding virtual function of a plurality of virtual functions associated with the subset, a context register for defining an operating state of the corresponding virtual function, and a doorbell for signaling that the corresponding virtual function is ready to be scheduled for execution.

5. The processing unit according to claim 4, further comprising: A scheduler is configured to schedule the first guest virtual machine to execute a first virtual function of the plurality of virtual functions within a first time interval in response to signaling from the first guest virtual machine.

6. The processing unit of claim 5 , wherein the hypervisor permits the first guest virtual machine to access the first subset during the first time interval, and wherein the hypervisor denies access to the register set to an unscheduled guest virtual machine among a plurality of guest virtual machines executing in parallel during the first time interval.

7. The processing unit according to claim 6, wherein: For the first guest virtual machine, at least one of a user mode driver and a firmware image for implementing a multimedia function of the first virtual function is installed on the fixed function hardware block.

8. The processing unit of claim 5, wherein the first guest virtual machine writes information to a doorbell register in the first subset to signal to the scheduler that the first guest virtual machine is ready to be scheduled for execution.

9. The processing unit of claim 5, wherein the first guest virtual machine is scheduled based on a priority associated with the first guest virtual machine and other priorities associated with other guest virtual machines in a plurality of guest virtual machines that are ready to be scheduled and executed in parallel.

10. The processing unit of claim 8, wherein the first guest virtual machine performs graphics rendering on frames stored in a frame buffer in the first subset using the first virtual function during the first time interval.

11. The processing unit of claim 10, wherein the first guest virtual machine notifies the hypervisor in response to completing execution during the first time interval, and wherein the doorbell registers in the first subset are cleared in response to completing execution during the first time interval.

12. A method comprising: receiving, at a hypervisor and from a first guest virtual machine of a plurality of guest virtual machines executing in parallel in a processing unit, a request to access a first virtual function of a plurality of virtual functions exposed to the plurality of guest virtual machines executing in parallel, wherein each of the plurality of virtual functions corresponds to a physical function implemented on a fixed function hardware block in the processing unit, and wherein the physical function corresponds to a hardware accelerated function performed by the fixed function hardware block; granting, from the hypervisor and to the first guest virtual machine, access to a first subset of a plurality of subsets of a set of registers in the processing unit, wherein the first subset of the plurality of subsets stores information associated with corresponding virtual functions of the plurality of virtual functions; configuring the fixed function hardware block to execute the first virtual function of the first guest virtual machine on behalf of the first guest virtual machine based on the information associated with the first virtual function stored in the first subset; as well as Using the first guest virtual machine, graphics rendering is performed on the frames stored in the first subset using the fixed-function hardware blocks configured to implement the first virtual function.

13. The method according to claim 12, further comprising: The register set is divided into a number of subsets corresponding to a maximum amount of space allocated to the virtual function.

14. The method according to claim 12, further comprising: dividing the register set into a number of subsets corresponding to a minimum amount of space allocated to the plurality of virtual functions; as well as The number of the subsets is modified based on attributes of the plurality of virtual functions.

15. The method of claim 12, wherein the first subset of the plurality of subsets of the register set comprises a frame buffer for storing the frames operated by the first virtual function, a context register for defining an operating state of the virtual function, and a doorbell register for signaling that the first virtual function is ready to be scheduled for execution.

16. The method according to claim 15, further comprising: The first guest virtual machine is scheduled to execute the first virtual function within a first time interval in response to signaling from the first guest virtual machine.

17. The method of claim 16, wherein granting access to the first subset from the hypervisor to the first guest virtual machine occurs during the first time interval, and wherein the hypervisor denies access to the subset of the register set to an unscheduled guest virtual machine among a plurality of guest virtual machines executing in parallel during the first time interval.

18. The method according to claim 17, further comprising: At least one of a user mode driver and a firmware image for implementing a multimedia function of the first virtual function is installed on the fixed function hardware block.

19. The method according to claim 17, further comprising: Information is written from the first guest virtual machine to the doorbell registers in the first subset to signal that the first guest virtual machine is ready to be scheduled for execution.

20. The method of claim 19, wherein scheduling the first guest virtual machine comprises: The first guest virtual machine is scheduled in response to reading the information from the doorbell register.

21. The method of claim 20, wherein scheduling the first guest virtual machine comprises: The first guest virtual machine is scheduled based on a priority associated with the first guest virtual machine and other priorities associated with other guest virtual machines in the plurality of guest virtual machines executing in parallel that are ready to be invoked.

22. The method of claim 20, wherein performing the graphics rendering on the frames stored in the first subset using the first guest virtual machine comprises: Graphics rendering is performed on frames stored in the frame buffer in the first subset using the first virtual function during the first time interval.

23. The method of claim 20, wherein the first guest virtual machine notifies the hypervisor that another virtual function can be loaded for another guest virtual machine in the plurality of guest virtual machines executing in parallel in response to completion of execution during the first time interval, and wherein the doorbell register in the first subset is cleared in response to completion of execution during the first time interval.

24. A method comprising: Performing graphics rendering on a frame stored in a first subset of a plurality of subsets of register sets implemented in a processing unit using a first guest virtual machine of a plurality of guest virtual machines executing in parallel on the processing unit, wherein: performing the graphics rendering using a first virtual function of a plurality of virtual functions exposed to the plurality of guest virtual machines executing in parallel, Each virtual function of the plurality of virtual functions corresponds to a physical function implemented on a fixed function hardware block configured to perform a hardware accelerated function of the processing unit, and Each subset of the plurality of subsets stores respective context information for implementing a corresponding virtual function of the plurality of virtual functions; detecting, at the hypervisor, a request from a second guest virtual machine in a plurality of guest virtual machines executing in parallel to access a second virtual function in the plurality of virtual functions; and A world switch is performed at the hypervisor and in response to the request to configure the fixed function hardware block to perform the second virtual function based on context information associated with the second virtual function stored in a second subset of the plurality of subsets associated with the second virtual function.

25. The method of claim 24, wherein the second guest virtual machine writes information to a doorbell register in the second subset to indicate that the second guest virtual machine is ready to be scheduled, and wherein detecting the request comprises reading the information from the doorbell register.

26. The method according to claim 25, further comprising: The second guest virtual machine is scheduled for execution during a second time interval beginning at the scheduled time in response to detecting the request.

27. The method of claim 26, wherein scheduling the second guest virtual machine for execution during the second time interval comprises: The second guest virtual machine is granted exclusive access to the set of registers during the second time interval.

28. The method of claim 25, wherein configuring the fixed function hardware block comprises: At least one of a user mode driver and a firmware image for implementing the multimedia function of the second virtual function is installed.

29. The method according to claim 25, further comprising: Using the second guest virtual machine, graphics rendering is performed on the frames stored in the second subset using the second virtual function.

Images