Hardware Capability Negotiation Mechanism between Virtual Function Driver and Physical Function Driver in Virtualized Environment

Abstract

A computing device is configured to: receive one or more first message types transmitted from a virtual function (VF) driver to a physical function (PF) driver, wherein the one or more first message types are in a dynamic message format, the one or more first message types include a first dynamic indication representing a quantity of capabilities and a second dynamic indication associated with a corresponding capability, the second dynamic indication representing a size of data stored in a corresponding message associated with the corresponding capability, the one or more first message types include a set of requested capabilities supported by the VF driver on or enabled by the I / O device; transmit one or more message types from the PF driver to the VF driver, wherein the one or more second message types include a set of supported capabilities selected from the requested capabilities supported by the I / O device and the PF driver; enable a set of supported capabilities supported by the I / O device and the PF driver for use by the VF driver.

Description

Background Art

[0001] In some embodiments, the present invention relates to virtual machines (VMs), and more particularly, but not limited to, the communication between VMs and virtualized resources of physical hardware devices.

[0002] Hardware resources can be virtualized, especially I / O devices such as storage devices, accelerator hardware, and network interfaces. The virtualization of hardware resources enables multiple VMs to share the same hardware resources. Summary of the Invention

[0003] The object of the present invention is to provide a computing device, a method, and a computer-readable storage medium for virtualized hardware resources of input / output (I / O) devices.

[0004] The above and other objects are achieved by the features of the independent claims. Other implementations are apparent from the dependent claims, the description, and the drawings.

[0005] According to a first aspect, there is provided a computing device for communicating with an input / output (I / O) device that provides virtualized hardware resources for use by a plurality of virtual machines (VMs). The computing device is configured to: execute a virtual machine manager (VMM), wherein the VMM implements a physical function (PF) driver corresponding to the I / O device and a plurality of virtual function (VF) drivers implemented by the plurality of VMs; receive at least one first message type transmitted from a virtual function (VF) driver of a VM to the PF driver, wherein the at least one first message type adopts a dynamic message format, the at least one first message type includes a first dynamic indication and a plurality of second dynamic indications, the first dynamic indication represents the number of a plurality of capabilities, each second dynamic indication is associated with a corresponding capability, each corresponding second dynamic indication represents the size of data stored in a corresponding message associated with the corresponding capability, the at least one first message type includes a set of requested capabilities supported by the VF driver on or enabled by the I / O device; transmit at least one second message type from the PF driver to the VF driver, wherein the at least one second message type includes a set of supported capabilities selected from the I / O device and the requested capabilities implemented thereon by the PF driver; enable a set of supported capabilities supported by the I / O device and the PF driver for use by the VF driver.

[0006] According to a second aspect, a method for providing virtualized hardware resources of an input / output (I / O) device for use by multiple virtual machines (VMs) is provided. The method includes: executing a virtual machine manager (VMM), wherein the VMM implements a physical function (PF) driver corresponding to the I / O device and multiple virtual function (VF) drivers implemented by the multiple VMs; receiving at least one first message type transmitted from a virtual function (VF) driver of a VM to the PF driver, wherein the at least one first message type adopts a dynamic message format, the at least one first message type includes a first dynamic indication and multiple second dynamic indications, the first dynamic indication represents the number of multiple capabilities, each second dynamic indication is associated with a corresponding capability, each corresponding second dynamic indication represents the size of data stored in a corresponding message associated with the corresponding capability, the at least one first message type includes a set of requested capabilities supported by the VF driver on or enabled by the I / O device; transmitting at least one second message type from the PF to the VF, wherein the at least one second message type includes a set of supported capabilities selected from the I / O device and the requested capabilities implemented thereon by the PF driver; enabling a set of supported capabilities supported by the I / O device and the PF driver for use by the VF driver.

[0007] The method provided by the second aspect can be executed by a computer-readable storage medium storing program code. The program code includes instructions; when the instructions are executed on a computer, the instructions cause the computer to execute the method provided by the second aspect.

[0008] The dynamic message format is scalable to facilitate the expansion of capabilities. The first indication is dynamic and does not lock the dynamic message format to a specific size, which enables new / old drivers to communicate dynamically. The size of the dynamic message format can be changed to any size required to define any number of supported capabilities. Additional capabilities can be added to the VF driver and / or PF driver, but it is not necessarily required to adjust the driver code. Each corresponding capability is associated with its own variable data structure (i.e., the second dynamic indication), which stores data on which details of the corresponding capability are supported and / or which details are not supported.

[0009] In another implementation of the first aspect and the second aspect, each of the multiple capabilities defined by the dynamic message format is independently processed sequentially until the number of capabilities defined by the first dynamic indication is reached.

[0010] Each capability can be negotiated individually, providing greater flexibility. The first dynamic indication can detect when all capabilities in one or more received first message types have been negotiated.

[0011] In another implementation of the first aspect and the second aspect, the computing device is further configured to: when the processing of the capabilities defined in the received first message type is completed and the number of processed capabilities is less than the number of capabilities defined by the first dynamic indication, poll other first message types.

[0012] When a capability does not fit a message, multiple messages may be sent. The total capabilities can be tracked to activate these capabilities. When one or more messages in the total messages are lost, the lost messages including the lost capabilities can be requested to activate the lost capabilities.

[0013] When all capabilities in one or more received first message types have been received but the total number of capabilities defined in the first dynamic indication has not been reached, it indicates that some capabilities are lost.

[0014] In another implementation of the first aspect and the second aspect, the dynamic message format further includes multiple third dynamic indications, each third dynamic indication being associated with a corresponding capability, and each corresponding third dynamic indication representing the version of the corresponding capability.

[0015] The capability versions of the third dynamic indications can assist in developing internal functions and / or fixing bugs, for example, enabling the code to bypass on the PF and / or VF. The capability versions of the third dynamic indications can provide easier backward compatibility, specifically in the case of upgrading only one side, i.e., when only one of the VF and PF is upgraded. New capabilities can be defined by newer versions, which can have unique fields for settings. Each capability package can be defined with a version, which can facilitate negotiating the supported version between the new side and the old side. Capabilities can be individually selected for activation, where the corresponding capabilities are new versions of older existing capabilities, and so on. Capabilities that are version-independent and / or invariant may not necessarily need to be activated (since they are already activated and / or irrelevant).

[0016] In another implementation of the first aspect and the second aspect, the dynamic message format is implemented according to the type-version-length-value (TVLV) format. The type includes an opcode indicating that the corresponding message is a capability message. The version represents the third dynamic indication. The length represents the first dynamic indication. The value represents the second dynamic indication and the corresponding capability data.

[0017] In another implementation of the first aspect and the second aspect, the dynamic message format is implemented according to the type-length-value (TLV) format. The type includes an opcode indicating that the corresponding message is a capability message. The length represents the first dynamic indication. The value represents the second dynamic indication and the corresponding capability data.

[0018] The TLV and / or TVLV format supports one or more of the following: a search sequence for messages using a generalized parsing function, the data of the capabilities in each message, including the second and / or third dynamic implementations, and can be placed in any order in the message. The data in the message can be in binary format, which enables faster parsing and / or smaller data.

[0019] In another implementation of the first aspect and the second aspect, the at least one message type includes multiple messages. The first indication of the dynamic message format represents the total number of capabilities included in the multiple messages. The dynamic message format further includes a fourth dynamic indication, and the fourth dynamic indication represents the number of capabilities in the total number of capabilities of the first dynamic indication included in the corresponding message.

[0020] The SW / HW can prepare to receive certain data according to the remaining messages using the dynamic message format. The lost messages using the dynamic message format and / or the remaining messages using the dynamic message format can be polled until all messages are received.

[0021] In another implementation of the first aspect and the second aspect, the starting position within the corresponding message of the data associated with each corresponding capability is set to a fixed alignment value.

[0022] In another implementation of the first aspect and the second aspect, when the data associated with a specific corresponding capability is less than the fixed alignment value, the data associated with the specific corresponding capability is padded to reach the fixed alignment value.

[0023] The fixed alignment value optimizes the processing of the capabilities of the corresponding messages using the dynamic message format. For example, the capabilities can be divided according to the fixed alignment value to achieve parallel negotiation of activation.

[0024] In another implementation of the first and second aspects, the dynamic message format further includes an operation code representing a capability message and an indication selected from the group including a first message, an intermediate message, a last message, and a unique message.

[0025] In another implementation of the first and second aspects, the operation code is located in the starting area of the dynamic message format.

[0026] In another implementation of the first and second aspects, at least one capability of the dynamic message format includes an operation code for exchanging at least one user register of the I / O device, and one or more of the following: a register name of the at least one user register, a register offset of the at least one user register, a mask of the at least one user register, a field size of the at least one user register, and a bit position of the at least one user register.

[0027] The dynamic message format can be used to change existing user registers.

[0028] In another implementation of the first and second aspects, the dynamic message format includes a fifth dynamic indication that represents an overall version of a packet of the plurality of capabilities included in the at least one first message.

[0029] The capability version of the fifth dynamic indication can assist in developing internal functions and / or fixing bugs, for example, enabling code to bypass on the PF and / or VF. The capability version of the fifth dynamic indication can provide easier backward compatibility, specifically in the case of upgrading only one side, that is, when only one of the VF and PF is upgraded. New capabilities can be defined by newer versions, which can have unique fields for setting. Each capability packet can be defined with a version, which can facilitate negotiating supported versions between the new side and the old side.

[0030] In another implementation of the first and second aspects, the VF driver is implemented as an adaptive VF (AVF) and / or an assignable device interface (ADI), a plurality of instances of the AVF and / or ADI are implemented on the plurality of VMs, the plurality of instances of the AVF and / or ADI VF are mapped to a single common PF driver, and the plurality of instances of the AVF and / or ADI are adapted simultaneously and similarly.

[0031] In another implementation of the first aspect and the second aspect, the computing device is further configured to and / or the method further includes: enabling a basic mode including a set of basic capabilities on the PF driver and the I / O device for use by the VF driver; in response to the enabled basic mode, receiving the at least one first message type for enabling advanced capabilities.

[0032] In another implementation of the first aspect and the second aspect, the computing device is further configured to and / or the method further includes: receiving at least one third message type transmitted from the VF driver to the PF driver, wherein the at least one third message type adopts the dynamic message format, and the at least one third message type includes a set of enabled capabilities selected from the supported capabilities and enabled by the I / O device and the PF driver to be implemented thereon.

[0033] In another implementation of the first aspect and the second aspect, the at least one second message type adopts the dynamic message format.

[0034] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present invention, exemplary methods and / or materials are described below. In case of conflict, the present patent specification (including definitions) shall prevail. In addition, these materials, methods, and examples are illustrative only and not necessarily restrictive. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Some embodiments of the present invention are described herein only by way of example in conjunction with the accompanying drawings. Now, specifically in conjunction with the accompanying drawings, it should be emphasized that the details shown are only for purposes of illustration and for the purpose of an illustrative discussion of embodiments of the present invention. In this regard, it will be apparent to those skilled in the art how to practice the embodiments of the present invention based on the description of the drawings.

[0036] In the drawings:

[0037] Figure 1 is a block diagram of components in a computing device provided by some embodiments, the computing device implementing a dynamic message format for negotiating capabilities in a virtual environment;

[0038] Figure 2 is provided by some embodiments in combination with Figure 1 a data flow diagram of an exemplary data flow between components in the computing device described;

[0039] Figure 3 is a schematic diagram of an exemplary message adopting the dynamic message format;

[0040] Figure 4 is a flowchart of a method for negotiating capabilities between a VF driver and a PF driver by using messages in a dynamic message format provided by some embodiments;

[0041] Figure 5 is a schematic diagram of an exemplary virtualized environment that implements messages in a dynamic message format provided by some embodiments. Detailed Implementation Manner

[0042] Some embodiments of the present invention relate to a virtual machine (VM), and more specifically, but not limited to, relate to communication between a VM and virtualized resources of a physical hardware device.

[0043] One aspect of some implementations of the methods, apparatuses, and / or code instructions described herein relates to a dynamic message format of messages transmitted between a virtual function (VF) driver implemented on a VM and a physical function (PF) function driver corresponding to a hardware device (optionally corresponding to an input / output (I / O) device such as a network interface card (NIC), a graphics adapter card, and a storage accelerator device). The dynamic message format includes a first dynamic indication and a plurality of second dynamic indications. The first dynamic indication represents the number of capabilities included in the corresponding message, and each second dynamic indication is associated with a corresponding capability. The number of second dynamic indications may correspond to the number of capabilities included in the corresponding message, that is, each capability corresponds to a second dynamic indication. Each corresponding second dynamic indication represents the size of the data stored in the corresponding message associated with the corresponding capability.

[0044] During the negotiation phase of the capabilities supported by the PF driver and the I / O device and / or the capabilities that the VF driver desires to implement on the I / O device, messages are transmitted between the VF driver and the PF driver. In an exemplary negotiation session, one or more messages in a dynamic message format of a first message type are transmitted from the VF driver to the PF driver. The first message type includes a set of requested capabilities supported by the VF driver that are enabled on the I / O device or enabled by the I / O device. A second message type is transmitted from the PF to the VF. One or more second message types may be in a dynamic message format. One or more second message types include a set of supported capabilities selected from the capabilities supported by the I / O device and implemented on the PF driver. This set of supported capabilities is enabled by the I / O device and the PF driver for use by the VF driver.

[0045] The support capabilities may include support protocols, support functions, and / or support technologies, such as, for example, remote direct memory access (RDMA), receive side scaling (RSS).

[0046] At least some implementations of the methods, apparatuses, and / or code instructions described herein solve the following technical problems: the ability to discover physical hardware devices for use by virtual machines, particularly the ability of a physical function driver to provide to a virtual function driver associated with a VM. In a standard implementation, the structure of the messages transmitted between the VF driver and the PF driver is fixed or semi-fixed, which limits the number of capabilities that can be negotiated and also limits the parameters of the capabilities. The fixed structure of the messages lacks flexibility and / or scalability. This problem is particularly evident in data centers such as cloud services that execute a large number (e.g., thousands) of VMs. Additionally, the code executed on a VM can be imported by external customers using the cloud service. By maximizing the capabilities and / or the parameters of the capabilities that a VM can execute on the underlying hardware device, maximizing the compatibility between the capabilities available to the VM and the capabilities required by the VM helps to improve performance.

[0047] By providing a dynamic message structure that accommodates any number of negotiated capabilities and / or any size and / or any number of capability parameters, at least some implementations of the systems, methods, and code instructions described herein provide a technical solution to the above-described technical problems. The dynamic message structure is flexible and / or scalable to include an increasing number of candidate capabilities and / or candidate parameters of the capabilities.

[0048] Before explaining in detail at least one embodiment of the invention, it is to be understood that the invention is not necessarily limited in its application to the details of construction and arrangement of components and / or methods set forth in the following description and / or illustrated in the drawings and / or exemplified. The invention is capable of other embodiments or of being practiced or carried out in various ways.

[0049] The invention may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon that cause a processor to execute various aspects of the invention.

[0050] The computer-readable storage medium may be a tangible device that is capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, by way of example and not limitation, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing devices.

[0051] The computer-readable program instructions described herein can be downloaded to a corresponding computing / processing device from a computer-readable storage medium or downloaded to an external computer or external storage device through a network such as the Internet, a local area network, a wide area network, and / or a wireless network.

[0052] The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer; executed partially on the user's computer and partially on a remote computer as a stand-alone software package, or executed entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network connection, including a local area network (LAN) or a wide area network (WAN); it may also be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, an electronic circuit, including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), etc., may personalize the electronic circuit by utilizing the status information of the computer-readable program instructions and execute the computer-readable program instructions to perform various aspects of the present invention.

[0053] Aspects of the present invention are described herein in connection with the flowchart illustrations and / or block diagrams of the methods, apparatuses (systems), and computer program products provided in the embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and the combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0054] The flowcharts and block diagrams in the figures illustrate the structure, functionality, and operation of the possible implementations of the systems, methods, and computer program products provided by various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment, or a portion of instructions, which includes one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions described in the blocks may not be implemented in the order illustrated in the figures. For example, in fact, two consecutive blocks shown may be executed almost simultaneously, or sometimes in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combinations of blocks in the block diagram and / or flowchart, can be implemented by a system based on dedicated hardware, which performs specific functions or actions, or a combination of dedicated hardware and computer instructions.

[0055] Now refer to Figure 1 ,Figure 1 is a block diagram of components in a computing device 104 provided by some embodiments. The computing device 104 implements a dynamic message format for negotiating capabilities in a virtual environment. Also refer to Figure 2 , Figure 2 is an exemplary data flow between components in the computing device 104 provided by some embodiments in conjunction with Figure 1 described. Also refer to Figure 3 , Figure 3 is a schematic diagram of an exemplary message 302 that employs a dynamic message format. The dynamic message format is scalable to facilitate the expansion of capabilities. Messages transmitted by components in the computing device in conjunction with Figure 1 and Figure 2 described can adopt the dynamic message format described in conjunction with Figure 3 described. Also refer to Figure 4 , Figure 4 is a flowchart of a method for negotiating capabilities between a VF driver and a PF driver via a message that employs a dynamic message format. Also refer to Figure 5 , Figure 5 is an exemplary virtualization environment 502 that implements a message that employs a dynamic message format. Features described in conjunction with Figure 4 and / or Figure 5 described can be implemented by components in the computing device 104 described in conjunction with Figure 1 and / or Figure 2 described using the dynamic message format described in conjunction with Figure 3 described. For example, it can be implemented by one or more processors 102 in the computing device 104 that execute code instructions (e.g., code 106A) stored in the memory 106.

[0056] The computing device 104 can be implemented as one or more of a computing cloud, a single computing device (e.g., a client terminal), a set of computing devices side by side, a network server, a local server, a remote server, a client terminal, a mobile device, a fixed device, a self-service kiosk, a smartphone, a laptop computer, a tablet computer, a wearable computing device, a glasses computing device, a watch computing device, and a desktop computer, and so on.

[0057] One or more processors 102 may be implemented as one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more field programmable gate arrays (FPGAs), one or more digital signal processors (DSPs), one or more application specific integrated circuits (ASICs), one or more custom circuits, processors connected to other units, and / or dedicated hardware accelerators. One or more processors 102 may be implemented as a single processor, a multi-core processor, and / or a cluster of processors set up for parallel processing (which may include homogeneous processor architectures and / or heterogeneous processor architectures).

[0058] Memory 106 stores code instructions that may be implemented by one or more processors 102. Memory 106 may be implemented as random access memory (RAM), read-only memory (ROM), and / or storage devices, such as non-volatile memory, magnetic media, semiconductor memory devices, hard disk drives, removable memory, and optical media (e.g., DVDs, CD-ROMs).

[0059] Memory 106 may store a virtual machine manager (VMM) 108 that manages and / or runs one or more virtual machines (VMs) 110. VMM 108 may be implemented as a hypervisor. VMM 108 may be implemented in hardware, software, firmware, and / or a combination of the above.

[0060] Each VM 110 executes one or more virtual function (VF) drivers 112.

[0061] The computing device 104 includes one or more I / O devices 114 and / or communicates with one or more I / O devices 114. Each I / O device 114 is associated with one or more physical function (PF) drivers 116. The I / O device 114 can be an input / output (I / O) device, such as, for example, a universal serial bus, a network adapter (for wired and / or wireless networks), a network interface card, an accelerator, a graphics device, and a storage device.

[0062] The computing device 104 can include one or more data storage devices 118 and / or communicates with one or more data storage devices 118. The data storage device 118 can store different versions of VMs, VFs, PFs, and / or VMMs, and / or other data, etc. It should be noted that the code instructions can be selectively loaded from the data storage device 118 into the memory 106 for execution by one or more processors 102. One or more data storage devices 118 can be implemented as a memory, a local hard disk drive, a removable storage unit, an optical disc, a storage device, etc., and / or implemented as a remote server and / or a computing cloud (accessed, for example, through a network connection).

[0063] The computing device 104 can communicate with a network 120, and the network 120 can be the Internet, a local area network, a virtual network, a wireless network, a cellular network, a local bus, a (wired, for example) point-to-point link, and / or a combination of the above.

[0064] The computing device 104 can include a network interface for communicating with the network 120. The network interface can be implemented as one or more of the I / O devices 114 having an associated one or more PF drivers 116. The network interface can be virtualized for use by multiple VMs 110 through the corresponding executed VF drivers 112. For example, different VMs 110 can access the network 120 through one or more VF drivers 112 of one or more PF drivers 116 for accessing the network interface.

[0065] The computing device 104 can include one or more physical user interfaces 122 and / or communicates with one or more physical user interfaces 122, and the one or more physical user interfaces 122 include user interaction mechanisms, such as, for example, to input data (such as selecting a version of a VF to load) and / or view data (such as viewing a message indicating which capabilities have been negotiated between a VF and a PF).

[0066] The exemplary physical user interface 122 may include one or more of a touch screen, a display, a gesture activation device, a keyboard, a mouse, voice activation software using a speaker and a microphone, and a coordinator that sends data via a network interface.

[0067] Now return to the reference Figure 2 , the I / O device 114 is virtualized by the PF driver 116. The physical resources of the I / O device 114 are partitioned, and these physical resources are available for multiple VMs 110A to 110D to access and use through the corresponding executed VF drivers 112A to 112D. Each VF driver 112A to 112D communicates with the PF 116 associated with the I / O device 114. The VF drivers 112A to 112D may be independent of the actual underlying I / O device 114 in use.

[0068] As described herein, messages are transmitted between the VF drivers 112A to 112D and the PF driver 116 for capability negotiation. Each VF driver 112A to 112D can negotiate its own set of desired capabilities and the capabilities supported by the PF 116 and / or the I / O device (e.g., hardware) 114.

[0069] The VF driver 112 and the PF driver 116 may adopt the express PCI (PCI express, PCIe) format, specifically the single root I / O virtualization (SR-IOV) function of PCIe. SR-IOV partitions the resources of the I / O device 114 (e.g., network adapter) for use by multiple VMs 110A to 110D. For example, the memory and / or network ports (e.g., network adapter) of the I / O device 114 are partitioned and shared by the VF drivers 112A to 112D. The network traffic flowing between each VF 112A to 112D and the PF 116 can be distinguished and identified, which enables the application of storage transformation and / or interrupt transformation between the corresponding VF 112A to 112D and the PF 116. The network traffic can flow between the PF 116 and each corresponding VF 112A to 112D without affecting the VF. The network traffic can achieve performance close to that of a bare metal (i.e., non-virtualized) machine.

[0070] Optionally, VF drivers 112A through 112D may be implemented as Adaptive Virtual Function (AVF) drivers and / or assignable device interfaces (ADIs). The term "AVF" as used herein may sometimes refer to the term "ADI" and / or may be interchangeable with the term "ADI". Multiple instances of AVF and / or ADI are implemented on multiple VMs 110. Instances of AVF and / or ADI VFs are mapped to a single common PF driver 116. Instances of AVF and / or ADI are adapted simultaneously and similarly. AVF adapts to the changing capabilities of the associated physical function (PF) driver. The PF can be updated using AVF without having to update all associated VF drivers. AVF has a single common device ID and branding string. Adaptive Virtual Function (AVF) may be implemented as an SR-IOV virtual function having the same device ID on different vendor I / O devices 114. In some implementations, the AVF driver is a VF driver that supports all future vendor devices without VM updates. For the AVF driver, as the underlying hardware device supports more and more advanced capabilities that can be activated in the VM, each update of the VF driver adds these advanced capabilities in a device-independent manner without affecting the basic functions.

[0071] Returning now to Figure 3 , the dynamic message format includes a first dynamic indication 304 ("total number of capabilities") indicating the number of capabilities included in the corresponding one or more messages using the dynamic message format. The total number of capabilities may represent a packet of multiple capabilities.

[0072] The first indication is dynamic and does not lock the dynamic message format to a specific size, which allows new / old drivers to communicate dynamically. The size of the dynamic message format can be changed to any size required to define any number of supported capabilities. Additional capabilities can be added to the VF driver and / or PF driver without necessarily adjusting the driver code.

[0073] The dynamic message format includes one or more second dynamic indicators 306 ("Capability #N Data Length"), each of which is associated with a corresponding capability. The number of second dynamic indicators 306 corresponds to the number of capabilities 304, and optionally corresponds to the number of capabilities in the current message. Each corresponding second dynamic indicator 306 represents the size of the data stored in the corresponding message associated with the corresponding capability. Each corresponding capability is associated with its own variable data structure (i.e., the second dynamic indicator), which stores data on which details of the corresponding capability are supported and / or which details are not supported.

[0074] Each second dynamic indicator 306 is associated with a dynamic capability data indicator 308 ("Capability #N Data") that stores the data for the corresponding capability. The size of the data in the dynamic capability data indicator 308 depends on the corresponding second dynamic indicator 306.

[0075] Optionally, the dynamic message format includes one or more third dynamic indicators 310 ("Capability #N Version"), each of which is associated with a corresponding capability. The number of third dynamic indicators 310 corresponds to the number of capabilities 304, and optionally corresponds to the number of capabilities in the current message. Each corresponding third dynamic indicator 310 represents the version of the corresponding capability. The version can be one or more of a major version, a minor version, and a sub-minor version, etc.

[0076] Optionally, the dynamic message format includes a fifth dynamic indicator 316 that represents the overall version of the capability package included in one or more corresponding messages. The fifth dynamic indicator 316 can have the same or a similar format as the third dynamic indicator 310.

[0077] The capability versions of the third dynamic indicator and / or the fifth dynamic indicator can assist in developing internal functions and / or fixing bugs, for example, enabling the code to bypass on the PF and / or VF.

[0078] The capability versions of the third dynamic indicator and / or the fifth dynamic indicator can provide easier backward compatibility, specifically in the case of only upgrading one side, i.e., when only one of the VF and PF is upgraded. New capabilities can be defined by newer versions, which can have unique fields for settings. Each capability package can be defined with a version, which can facilitate negotiating the supported version between the new side and the old side.

[0079] Capabilities can be individually selected for activation, where the corresponding capabilities are new versions of older existing capabilities, etc. Capabilities that are version-irrelevant and / or unchanged do not necessarily need to be activated (since they are already activated and / or irrelevant).

[0080] Optionally, multiple capabilities in the first dynamic indication 304 are included in a single message. Alternatively, all capabilities in the first dynamic indication 304 are divided among multiple separate messages. In such an implementation, the first dynamic indication 304 represents the total number of capabilities included in the multiple messages, and the fourth dynamic indication 312 represents the number of capabilities included in each respective message among the multiple messages (within the total number of capabilities of the first dynamic indication 304). When capabilities do not fit into one message, multiple messages may be sent. The total capabilities can be tracked to activate these capabilities. When one or more messages among the total messages are lost, the lost messages including the lost capabilities can be requested to activate the lost capabilities.

[0081] Optionally, the dynamic message format 302 includes an opcode 314 ("capability message OPCODE") indicating that the respective message is a capability message. The opcode 314 can act on the entire message, i.e., affect a set of capabilities associated with the respective message. The opcode 314 can include an indication that the respective message is the first message among multiple messages, an intermediate message among multiple messages, the last message among multiple messages, or the only (i.e., single) message. The opcode 314 can be located in the starting region of each respective message.

[0082] Optionally, the dynamic message format 302 includes multiple capability opcodes 318 ("capability #N opcode"), each associated with a respective capability. The value of each opcode 318 can be set for different functions. For example, at least one capability opcode 318 includes a value indicating the exchange of one or more user registers of an I / O device, and one or more of the following: the register name of one or more user registers, the register offset of one or more user registers, the mask of one or more user registers, the field size of one or more user registers, and the bit position of one or more user registers. The dynamic message format can be used to change existing user registers. An example of changing user registers is provided below:

[0083] Capability opcode: 5 (register offset handshake)

[0084] Capability version: 1.5.0

[0085] Capability data length: 4,096

[0086] Capability data:

[0087] <Enum_Reg_yy_reg=1>123

[0088] <Enum_Reg_my_other_reg=2>987

[0089] <Enum_Reg_my_another_reg=3>456

[0090] and so on.

[0091] The above data can be stored in a message implemented using a dynamic message as follows: 5 1.5.0 4,096 1123 2 987 3456...

[0092] The dynamic message format 302 can be implemented according to the type-length-value (TLV) format. The type can be implemented as an operation code 314, indicating that the corresponding message is a capability message; and / or the type can be implemented as a capability operation code 318 for each corresponding capability. The length can be implemented as a first dynamic indicator 304. The value can be implemented as a second dynamic indicator 306 and a corresponding capability data field 308. Optionally or additionally, the dynamic message format 302 is implemented according to the type-version-length-value (TVLV) format. The version can be implemented as a third dynamic indicator 310 and / or a fifth dynamic indicator 316.

[0093] The TLV and / or TVLV formats can be nested and / or implemented in sequence. For example, the TLV and / or TVLV formats can act on the entire message, and the TLV and / or TVLV formats can act on each capability.

[0094] The TLV and / or TVLV formats support one or more of the following: a search sequence for messages using a generalized parsing function, data of capabilities in each message, including second and / or third dynamic implementation methods, and can be placed in any order in the message. The data in the message can be in binary format, which makes parsing faster and / or the data smaller.

[0095] Optionally, the dynamic message format 302 is set to fixed alignment. For example, it is fixed to bytes and / or words and / or double words, and / or other alignment formats, such as 8 bytes. The starting position within the corresponding message of the data stored in the field 308 associated with each corresponding capability can be set to a fixed alignment value. When the data associated with a specific corresponding capability is less than the fixed alignment value, the data associated with the specific corresponding capability (i.e., stored in the field 308) can be padded to reach the fixed alignment value. The padding can use irrelevant data, such as zeros or other characters that do not represent actual data.

[0096] The fixed alignment value optimizes the processing of the capabilities of the corresponding message using the dynamic message format. For example, capabilities can be divided according to the fixed alignment value to achieve parallel negotiation for activation.

[0097] Now return to reference Figure 4, in 402, initialize the VF driver and / or ADI (the terms "VF driver" and "ADI" may be used interchangeably). For example, the VM may initialize the VF driver corresponding to the desired capabilities of the I / O device. For example, load a new version of the VF driver to enable additional capabilities on the I / O device. The initialization may occur upon reset and / or startup of the VM, VF driver, PF driver, and / or hardware device.

[0098] In 404, initiate communication between the VF driver and the PF driver corresponding to the I / O device. Optionally, the VF driver initiates communication with the PF driver.

[0099] In 406, the VF driver sends a first message type in a dynamic message format (e.g., as described herein in connection with Figure 3 etc.) to the PF driver, where the first message type has request parameters for requested capabilities and / or capabilities supported by the VF driver for use on or enabled by the I / O device. Alternatively, the message with request parameters for requested capabilities and / or capabilities is in a standard fixed and / or semi-fixed format. The VF may request that the capabilities in the first message type be enabled on the hardware device used by the VF and / or that the virtual interface of the hardware device used by the VF be opened.

[0100] These capabilities may include a set of basic capabilities and / or another set of advanced capabilities. Some implementations of the VF driver (e.g., AVF) define a minimum set of functions called "basic mode", but may provide additional advanced capabilities based on the capabilities available in the PF associated with the AVF. The basic mode includes a set of basic capabilities for use by the VF driver on the PF driver and the I / O device. The basic capabilities may be assumed to be available on the hardware device as a standard. Examples of the basic capabilities of a wired network adapter include basic receive and transmit queues for data transfer, signaling events sent to software, basic stateless operations such as checksum, segmentation offload, jumbo frames, RSS, etc.

[0101] In 408, the VF driver and the PF driver may enable a set of basic capabilities to operate in basic mode. In this implementation, as described in connection with 410, the advanced capabilities are enabled. Alternatively, the basic mode is not initially set. In this implementation, there is no necessary distinction between the basic capabilities and the advanced capabilities, and both types of capabilities are enabled, as described in connection with 410.

[0102] In 410, the VF driver sends a first message type in a dynamic message format to the PF driver, where the first message type has request parameters for advanced capabilities and / or advanced capabilities supported by the VF driver on or enabled by the I / O device. When the basic mode is enabled, the capabilities in the first message type are advanced capabilities. When the basic mode is not enabled, 406 and 408 are not necessarily executed. The capabilities in the first message type are request capabilities.

[0103] There can be multiple first message types including request capabilities, which can be set as a sequence. In this case, the operation code (e.g., 314 in combination with Figure 3 described) can be set accordingly for each message. The operation code of the first message type in the sequence is set to a value representing FIRST. The operation code of the middle messages in the sequence is set to a value indicating MIDDLE. The operation code of the last message in the sequence is set to a value representing LAST. When the request capabilities are included in a single first message type, the operation code is set to a value representing ONLY. The total number of capabilities (e.g., field 304 in Figure 3 and the number field of capabilities in the current message (e.g., field 312 in Figure 3 are set according to the capabilities included in each corresponding message. The SW / HW can prepare to receive certain data based on the remaining messages in the dynamic message format. The lost messages in the dynamic message format and / or the remaining messages in the dynamic message format can be polled until all messages are received.

[0104] In 412, a sequence of the first message type is received (e.g., by the PF driver) and processed. The value of the operation code field 314, the total number field 304 of capabilities, and the number field 312 of capabilities in the current message can be evaluated to determine whether all messages in the sequence have been received, or whether one or more messages have not been received. For example, the loss of one or more messages can be detected when the FIRST message or the LAST message has not been received yet. Also, for example, the loss of one or more messages can be detected when the number of capabilities in the received messages (i.e., the sum of the capabilities in field 412 in Figure 4 is less than the total number of capabilities in the received messages (i.e., the value in field 404 of Figure 4 ).

[0105] When all the capabilities in one or more received first message types have been received but the total number of capabilities defined in the first dynamic indication has not been reached, it indicates that some capabilities are missing. When one or more messages in the sequence have not been received, polling for additional first message types can be generated to help ensure that all capabilities have been received. For example, after a timeout expires (or other conditions are met), some messages are lost. Or, an error is generated when not all messages are received and / or when the received size in the message (e.g., according to the defined number of bytes) is invalid. A message indicating the error can be sent to the VF driver. The error message can indicate one or more missing messages, capability errors, and size errors, etc.

[0106] In 414, each capability in the received first type of message is processed independently. These capabilities can be processed in sequence until the total number of capabilities is reached.

[0107] The first dynamic indication can detect when all capabilities in one or more received first message types have been negotiated.

[0108] One or more second message types are transferred from the PF to the VF. The second message type includes a set of capabilities supported by the I / O device and / or PF driver for implementation thereon. Capabilities not supported by the I / O device and / or PF driver may be rejected. This set of capabilities can be selected from the requested capabilities provided by the VF in the first message type.

[0109] One or more second message types can adopt a dynamic message format. Alternatively, one or more second message types adopt another format, such as an array, where each element in the array corresponds to a capability of the first message type. The value of each element in the array is binary, indicating whether the PF and / or hardware supports the capability corresponding to the array element.

[0110] In 416, the I / O device and PF driver enable (e.g., activate) each capability (e.g., advanced function) for use by the VF driver. The capabilities can depend on the initial version (when the initial version exists).

[0111] Optionally, these capabilities are enabled according to one or more third message types transferred from the VF to the PF. The third message type can adopt a dynamic message format. The third message type includes a set of enabling capabilities selected from the supported capabilities (e.g., in the second message type) for the I / O device and / or PF driver to enable for implementation thereon.

[0112] In 418, the functions described in combination with 414 and / or 416 are iteratively executed. The iteration can enable each corresponding capability or not enable the corresponding capability.

[0113] Optionally, each capability is processed independently to be enabled and / or disabled. The enabling or disabling of one capability does not necessarily depend on the enabling or disabling of another capability.

[0114] In 420, the capability enabling is completed. The VF driver can use the enabled capabilities.

[0115] Now return to the reference Figure 5 , the scalable IOV setting (S-IOV) can be implemented in the virtualization environment 502.

[0116] The virtualization environment 502 includes one or more of the following exemplary components:

[0117] User space 504, including one or more VMs 506, where each VM implements a virtual network interface controller (vNIC) driver 508 for communicating with the ADI 516; dynamic messages using the dynamic message format described herein can be transmitted between the vNIC 508 and the PF 512 through the ADI 516, for example, as described in connection with the VF driver and the PF driver;

[0118] Hypervisor / host kernel space 510, including a PF driver 512, a virtual device (VDEV) 514 that implements one or more assignable device interfaces (ADIs) 516, and a virtual device composition module (VDCM) 518, where the virtual device composition module 518 can provide a software management resource mapping between the VDEV 514 and the ADI 516.

[0119] It should be noted that no VF is implemented in the scalable IOV virtualization environment 502, but instead the ADI is used.

[0120] An input / output memory management unit (IOMMU) 550 with a Process Address Space ID (PASID) remapping code 520 and a direct memory access (DMA) implement one or more ADIs 516 and PF 520. Each ADI 516 is labeled with a unique PASID.

[0121] The I / O device may be implemented as a network card (NETC) 522. The NETC 522 may be PCIe compliant. The hardware resources of the NETC 522 are virtualized by the ADI 516 such that multiple VMs 506 can each access their own allocated ADI 516 to receive and / or transmit traffic over a connected network (e.g., via one or more corresponding queues (Q) 524).

[0122] The NETC 522 includes a set of PF base address registers (BARs) 526 mapped to the ADI 516. The NETC 522 may include other registers such as a PF configuration (Conf) register 528 and an MSI-X register 530.

[0123] Using software to virtualize the resources of the NETC 522 incurs a significant overhead, which reduces the overall efficiency of the host.

[0124] The virtual device combination module 518 may provide slow path enumeration (represented by the dashed line 532) and fast path enumeration (represented by the solid line 534) pass-through. The slow path enumeration (represented by the dashed line 532) is performed by software. The fast path enumeration (represented by the solid line 534) is performed by hardware configured for direct access. The DMA provides isolation between the fast paths. Each vNIC driver 508 is provided with an isolated fast path to the resources of the NETC 522 via the DMA, and these resources are partitioned by the ADI 516 allocated by the NETC 522 using a corresponding unique PASID.

[0125] Messages (e.g., of the first message type, second message type, and / or third message type described herein) are transmitted between the vNIC driver 508 and the PF driver 512 to enable capabilities supported by the NETC 522 and the PF driver 512 requested by the vNIC driver 508 for use by the vNIC driver 508.

[0126] After studying the following drawings and detailed description, other systems, methods, features, and advantages of the present invention will be or become apparent to those skilled in the art. It is intended that all such other systems, methods, features, and advantages be included within this specification, be within the scope of the present invention, and be protected by the appended claims.

[0127] The description of the embodiments of the present invention is for illustrative purposes only and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein are chosen to best explain the principles, practical applications, or technical advancements of the embodiments, or to enable other skilled artisans in the art to understand the embodiments disclosed herein, as compared to the techniques available in the market.

[0128] It is expected that many related drivers will be developed during the lifespan of this application as it grows into a patent, and the scope of the term "driver" is intended to a priori include all such new technologies.

[0129] The term "about" as used herein means ±10%.

[0130] The terms "comprising", "having", and their variations mean "including but not limited to". This term encompasses the terms "consisting of" and "consisting essentially of".

[0131] The phrase "consisting essentially of" means that a composition or method may include additional ingredients and / or steps, provided that the additional ingredients and / or steps do not substantially alter the basic and novel characteristics of the claimed composition or method.

[0132] Unless the context clearly dictates otherwise, the singular forms "a" and "the" as used herein include plural meanings. For example, the term "a complex" or "at least one complex" may include multiple complexes, including mixtures thereof.

[0133] The word "exemplary" as used herein means "serving as an example, instance, or illustration". Any embodiment described as "exemplary" is not necessarily to be construed as more preferred or advantageous than other embodiments, and / or excludes combinations of features of other embodiments.

[0134] The word "optionally" as used herein means "provided in some embodiments and not provided in other embodiments". Any particular embodiment of the present invention may include multiple "optional" features, unless these features are in conflict.

[0135] In the present application, various embodiments of the present invention may be presented in a range format. It should be understood that the description in range format is only for convenience and brevity, and should not be construed as a fixed limitation on the scope of the present invention. Thus, the description of a range should be considered to have specifically disclosed all possible sub-ranges as well as the individual numerical values within that range. For example, a description of a range such as from 1 to 6 should be considered to have specifically disclosed sub-ranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., as well as the individual numbers within that range, e.g., 1, 2, 3, 4, 5, and 6. This applies regardless of how broad the range is.

[0136] When a numerical range is indicated herein, it is meant to include any recited number (fractional or integral) within the indicated range. The phrases "range between the first indicated number and the second indicated number" and "range from the first indicated number to the second indicated number" are used interchangeably herein to mean including the first and second indicated numbers and all fractional and integral numbers therebetween.

[0137] It should be understood that, for the sake of brevity of description, certain features of the present invention described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, for the sake of brevity of description, the various features of the present invention described in the context of a single embodiment may also be provided separately or in any suitable sub-combination or as any suitable other embodiment of the present invention. Certain features described in the context of various embodiments are not considered to be essential features of those embodiments unless the embodiment is inoperable without those elements.

[0138] Herein, all publications, patents, and patent applications mentioned in this specification are incorporated herein by reference in their entirety, and likewise, each individual publication, patent, or patent application is specifically and individually incorporated herein by reference. Additionally, the citation or identification of any reference to this application should not be taken as allowing such reference to be prior art to the present invention. With respect to the use of section headings, the section headings should not be construed as necessarily limiting.

Claims

1. A computing device that communicates with an input / output (I / O) device that provides virtualized hardware resources for use by multiple virtual machines (VMs), characterized in that, The computing device is configured to: Execute a virtual machine manager (VMM), where the VMM implements a physical function (PF) driver corresponding to the I / O device and multiple virtual function (VF) drivers implemented by the multiple VMs; Receive at least one first message type transmitted from a virtual function (VF) driver of a VM to the PF driver, where the at least one first message type adopts a dynamic message format, the at least one first message type includes a first dynamic indication and multiple second dynamic indications, the first dynamic indication represents the number of multiple capabilities, each second dynamic indication is associated with a corresponding capability, each corresponding second dynamic indication represents the size of data stored in a corresponding message associated with the corresponding capability, the first dynamic indication does not lock the dynamic message format to a specific size, and the size of the dynamic message format is applicable to be changed to any size required to define any number of supported capabilities; The at least one first message type includes a set of requested capabilities supported by the VF driver on or enabled by the I / O device; Transmit at least one second message type from the PF driver to the VF driver, where The at least one second message type includes a set of supported capabilities selected from the I / O device and the requested capabilities implemented thereon and supported by the PF driver; Enable a set of supported capabilities supported by the I / O device and the PF driver for use by the VF driver.

2. The computing device according to claim 1, wherein Each of the multiple capabilities defined by the dynamic message format is independently processed sequentially until the number of the multiple capabilities defined by the first dynamic indication is reached.

3. The computing device according to claim 2, wherein The computing device is further configured to: when the capabilities defined in the received first message type have been processed and the number of the processed capabilities is less than the number of capabilities defined by the first dynamic indication, poll other first message types.

4. The computing device according to claim 2, wherein The dynamic message format further includes multiple third dynamic indications, each third dynamic indication is associated with a corresponding capability, and each corresponding third dynamic indication represents the version of the corresponding capability.

5. The computing device according to claim 4, wherein The dynamic message format is implemented according to the type-version-length-value (TVLV) format, where the type includes an opcode indicating that the corresponding message is a capability message, the version represents the third dynamic indication, the length represents the first dynamic indication, and the value represents the second dynamic indication and the corresponding capability data.

6. The computing device according to any one of claims 1, 2, 4, and 5, characterized in that, The dynamic message format is implemented according to the type-length-value (TLV) format, where the type includes an opcode indicating that the corresponding message is a capability message, the length represents the first dynamic indication, and the value represents the second dynamic indication and the corresponding capability data.

7. The computing device according to any one of claims 1, 2, 4, and 5, characterized in that, The at least one first message type includes a plurality of messages, the first dynamic indication of the dynamic message format represents the total number of capabilities included in the plurality of messages, the dynamic message format further includes a fourth dynamic indication, and the fourth dynamic indication represents the number of capabilities in the total number of capabilities of the first dynamic indication included in the corresponding message.

8. The computing device according to claim 1, wherein The starting position within the corresponding message of the data associated with each corresponding capability is set to a fixed alignment value.

9. The computing device according to claim 8, wherein When the data associated with a specific corresponding capability is less than the fixed alignment value, the data associated with the specific corresponding capability is padded to reach the fixed alignment value.

10. The computing device according to any one of claims 1, 2, 4, and 5, characterized in that The dynamic message format further includes an opcode representing a capability message and an indication selected from the group including a first message, an intermediate message, a last message, and a unique message.

11. The computing device according to claim 10, wherein The opcode is located in the starting area of the dynamic message format.

12. The computing device according to any one of claims 1, 2, 4, and 5, characterized in that, At least one capability of the dynamic message format includes an opcode for exchanging at least one user register of the I / O device, and one or more of the following: the register name of the at least one user register, the register offset of the at least one user register, the mask of the at least one user register, the field size of the at least one user register, and the bit position of the at least one user register.

13. The computing device according to any one of claims 1, 2, 3, 4, and 5, characterized in that The dynamic message format includes a fifth dynamic indication, and the fifth dynamic indication represents the overall version of the packet of the plurality of capabilities included in the at least one first message type.

14. The computing device according to claim 1, wherein The VF driver is implemented as an adaptive VF (AVF) and / or an assignable device interface (ADI), a plurality of instances of the AVF and / or ADI are implemented on the plurality of VMs, the plurality of instances of the AVF and / or ADI VF are mapped to a single common PF driver, and the plurality of instances of the AVF and / or ADI are adapted.

15. The computing device according to claim 1 or 14, characterized in that, The computing device is further configured to: Enable a basic mode including a set of basic capabilities on the PF driver and the I / O device for use by the VF driver; In response to the enabled basic mode, receive the at least one first message type for enabling advanced capabilities.

16. The computing device according to claim 1 or 14, characterized in that, The computing device is further configured to: Receive at least one third message type transmitted from the VF driver to the PF driver, wherein the at least one third message type adopts the dynamic message format, and the at least one third message type includes a set of enabled capabilities selected from the supported capabilities and enabled on the I / O device and the PF driver for implementation thereon.

17. The computing device according to claim 1, wherein The at least one second message type adopts the dynamic message format.

18. A method for providing virtualized hardware resources of an input / output (I / O) device for use by multiple virtual machines (VMs), characterized in that, The method includes: Execute a virtual machine manager (VMM), wherein the VMM implements a physical function (PF) driver corresponding to the I / O device and multiple virtual function (VF) drivers implemented by the multiple VMMs; Receive at least one first message type transmitted from a virtual function (VF) driver of a VM to the PF driver, wherein the at least one first message type adopts a dynamic message format, the at least one first message type includes a first dynamic indication and multiple second dynamic indications, the first dynamic indication represents the number of multiple capabilities, each second dynamic indication is associated with a corresponding capability, each corresponding second dynamic indication represents the size of data stored in a corresponding message associated with the corresponding capability, the first dynamic indication does not lock the dynamic message format to a specific size, and the size of the dynamic message format is applicable to be changed to any size required to define any number of supported capabilities; The at least one first message type includes a set of requested capabilities supported by the VF driver on or enabled by the I / O device; Transmit at least one second message type from the PF to the VF, wherein, The at least one second message type includes a set of supported capabilities selected from the I / O device and the requested capabilities implemented thereon supported by the PF driver; Enable a set of supported capabilities supported by the I / O device and the PF driver for use by the VF driver.

19. A computer-readable storage medium for storing program code, characterized in that, The program code includes instructions; when the instructions are executed on a computer, the instructions cause the computer to execute the method according to claim 18.

Images