Method and device for adding GPU resources in a virtual machine
By obtaining and writing GPU resource device identification information to the virtual machine device list, the GPU resources are transmitted to the operating system when the virtual machine is powered on and started, solving the problems of insufficient memory and BIOS restrictions on GPU resources added in the virtual machine. Ensures effective addition of GPU resources in K8S environment.
Patent Information
- Application Number
- CN202111101655.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-18
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-09-18
AI Technical Summary
When adding GPU resources to virtual machines, the existing technology faces the problems of insufficient memory and failure of hot plugging of multiple GPU devices. Especially in the K8S environment, it cannot effectively solve the problems of BIOS memory reservation restrictions and GPU resource information transfer.
By obtaining the soft connection file, obtaining the GPU resource device identification information in the device plug-in cache file of the K8S container cluster, writing it to the device list when the virtual machine is created, and passing it through OCI-SPEC data packets, solving the BIOS Memory limits and GPU resource information transfer issues.
It realizes the transmission of GPU resources to the operating system when the virtual machine is powered on and started, solves the problem of hot-swap GPU memory usage, and removes the BIOS limit when inserting GPU resources into the virtual machine in the Kata process.
Smart Images

Figure CN113849269B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cloud services, and in particular to a method and device for adding GPU resources in a virtual machine. Background Art
[0002] In the related technology, in the community version of the KATA component (a container component of a lightweight virtual machine), by using the PCIE (peripheral component interconnect express, a high-speed serial computer expansion bus) hot plug method, GPU (graphics processing unit) resources are added to the corresponding container instance, thereby realizing the application of GPU resources in a virtualized environment.
[0003] However, this solution of adding GPU resources to a virtualized environment through PCIE hot-plugging has the following defects: when the memory occupied by the GPU is too large, for example, when the community version of KATA component attempts to hot-plug a Nvidia Tesla V100 GPU, it will fail due to insufficient memory reserved by the BIOS. Moreover, even if the BIOS source code is modified and recompiled to reserve enough memory, it cannot cope with GPU models that may occupy more memory in the future. At the same time, in the current hot-plug solution, when multiple GPU devices need to be hot-plugged, there will be problems, resulting in the failure of the GPU to be successfully hot-plugged.
[0004] In addition, for the K8S environment. Due to its own architecture, K8S will not pass the relevant information of GPU resources to the KATA component during the boot process. In other words, in the K8S processing flow, the relevant information of GPU resources will be passed to the KATA component only after the boot process is completed. At this time, since the boot process has been completed, only the hot plug solution can be used to add GPU resources. Therefore, it is currently difficult to overcome the above defects of the hot plug solution in the K8S environment.
[0005] With respect to the above-mentioned problems existing in the related technologies, no effective solutions have been found so far. Summary of the invention
[0006] Embodiments of the present invention provide a method and device for adding GPU resources in a virtual machine.
[0007] According to one aspect of an embodiment of the present invention, a method for adding GPU resources in a virtual machine is provided, comprising: obtaining a soft connection file, wherein the soft connection file is used to point to a cache file of a device plug-in of a K8S container cluster corresponding to a target virtual machine, and the cache file includes device identification information of the GPU resources; when creating a container group of the target virtual machine, obtaining the device identification information of the GPU resources from the cache file according to the soft connection file; writing the device identification information corresponding to the currently created container group into a device list of the target virtual machine; and adding the GPU resources corresponding to the device identification information in the device list to the target virtual machine.
[0008] Further, obtaining the device identification information of the GPU resource from the cache file according to the soft link file includes: determining the location of the cache file according to the soft link file; storing the device identification information in the cache file into a preset data structure of the KATA component corresponding to the target virtual machine, wherein the preset data structure is used to store the device identification information.
[0009] Furthermore, the writing of the device identification information corresponding to the currently created container group into the device list of the target virtual machine includes: obtaining a first OCI-SPEC data packet transmitted between the K8S component and the KATA component; parsing a first Pod ID of the currently created container group from the first OCI-SPEC data packet; searching for the device identification information matching the first Pod ID in the preset data structure; and writing the device identification information matching the first Pod ID into the device list in the format of the OCI standard, wherein the device list is stored in the first OCI-SPEC data packet.
[0010] Furthermore, the device identification information includes overall information of the Pod device, and searching the preset data structure for the device identification information matching the first Pod ID includes: enumerating the overall information of the Pod device of each of the GPU resources in the preset data structure; extracting the second Pod ID of each of the overall information of the Pod device; determining whether the first Pod ID is consistent with the second Pod ID; if the first Pod ID is consistent with the second Pod ID, determining that the second Pod ID matches the first Pod ID, and extracting the device identification information of the GPU resource corresponding to the second Pod ID from the preset data structure.
[0011] Furthermore, after the step of adding the GPU resources corresponding to the device identification information in the device list to the target virtual machine, the method also includes: in the process of creating a Pod container of the target virtual machine, obtaining a second OCI-SPEC data packet transmitted between the K8S component and the KATA component; and removing the device identification information matching the first Pod ID in the second OCI-SPEC data packet.
[0012] Further, removing the device identification information matching the first Pod ID in the second OCI-SPEC data packet includes: obtaining pre-recorded device identification information matching the first Pod ID; searching for device identification information matching the first Pod ID in a device list in the second OCI-SPEC data packet; and removing the device identification information matching the first Pod ID in the device list in the second OCI-SPEC data packet.
[0013] Further, adding the GPU resources corresponding to the device identification information in the device list to the target virtual machine includes: obtaining the device identification information of the GPU resources in the device list, wherein the target virtual machine is used to load the container instance; determining the PCIE identification information corresponding to the GPU resources according to the device identification information; starting the target virtual machine, and adding the corresponding GPU resources to the target virtual machine according to the PCIE identification information.
[0014] According to another aspect of an embodiment of the present invention, a device for adding GPU resources in a virtual machine is provided, comprising: a first acquisition module, used to acquire a soft connection file, wherein the soft connection file is used to point to a cache file of a device plug-in of a K8S container cluster corresponding to a target virtual machine, and the cache file includes device identification information of the GPU resources; a second acquisition module, used to acquire the device identification information of the GPU resources from the cache file according to the soft connection file when creating a container group of the target virtual machine; a writing module, used to write the device identification information corresponding to the currently created container group into a device list of the target virtual machine; and a loading module, used to add the GPU resources corresponding to the device identification information in the device list to the target virtual machine.
[0015] Furthermore, the second acquisition module includes: a determination unit, used to determine the location of the cache file based on the soft link file; a storage unit, used to store the device identification information in the cache file into a preset data structure of the KATA component corresponding to the target virtual machine, wherein the preset data structure is used to store the device identification information.
[0016] Furthermore, the writing module includes: an acquisition unit, used to acquire a first OCI-SPEC data packet transmitted between the K8S component and the KATA component; a parsing unit, used to parse out the first Pod ID of the currently created container group from the first OCI-SPEC data packet; a search unit, used to search for the device identification information matching the first PodID in the preset data structure; a writing unit, used to write the device identification information matching the first Pod ID into the device list in accordance with the OCI standard format, wherein the device list is stored in the first OCI-SPEC data packet.
[0017] Furthermore, the device identification information includes overall information of the Pod device, and the search unit includes: an enumeration subunit, used to enumerate the overall information of the Pod device of each of the GPU resources in the preset data structure; an extraction subunit, used to extract the second Pod ID of each of the overall information of the Pod device; a judgment subunit, used to judge whether the first Pod ID is consistent with the second Pod ID; an extraction subunit, used to determine that the second Pod ID matches the first Pod ID when the first Pod ID is consistent with the second Pod ID, and extract the device identification information of the GPU resource corresponding to the second Pod ID from the preset data structure.
[0018] Furthermore, the device also includes: a third acquisition module, which is used to obtain a second OCI-SPEC data packet transmitted between the K8S component and the KATA component during the process of creating a Pod container of the target virtual machine after the loading module adds the GPU resources corresponding to the device identification information in the device list to the target virtual machine; and a removal module, which is used to remove the device identification information matching the first Pod ID in the second OCI-SPEC data packet.
[0019] Further, the elimination module includes: a first acquisition unit, used to obtain pre-recorded device identification information matching the first PodID; a search unit, used to search for device identification information matching the first Pod ID in the device list of the second OCI-SPEC data packet; and a elimination unit, used to eliminate the device identification information matching the first Pod ID in the device list of the second OCI-SPEC data packet.
[0020] Furthermore, the loading module includes: an acquisition unit, used to obtain device identification information of the GPU resource in the device list, wherein the target virtual machine is used to load the container instance; a determination unit, used to determine the PCIE identification information corresponding to the GPU resource according to the device identification information; a loading unit, used to start the target virtual machine and add the corresponding GPU resource to the target virtual machine according to the PCIE identification information.
[0021] According to another aspect of an embodiment of the present invention, a storage medium is further provided. The storage medium includes a stored program. When the program is run, the above steps are executed.
[0022] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the program stored in the memory.
[0023] The embodiment of the present invention also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps in the above method.
[0024] Through the present invention, a soft connection file is obtained, wherein the soft connection file is used to point to a cache file of a device plug-in of a K8S container cluster corresponding to a target virtual machine, the cache file includes device identification information of a GPU resource, and when creating a container group of a target virtual machine, the device identification information of the GPU resource is obtained from the cache file according to the soft connection file, the device identification information corresponding to the currently created container group is written into a device list of the target virtual machine, and the GPU resource corresponding to the device identification information in the device list is added to the target virtual machine, and by creating a soft connection file, the device identification information of the GPU resource is obtained from the cache file of K8S through the soft connection file, and then the GPU resource corresponding to the current container group is determined from the obtained device identification information of the GPU resource, and added to the device list of the target virtual machine, the GPU resource can be inserted into the startup command of the operating system when the virtual machine is powered on, and then the GPU resource is transparently transmitted to the target virtual machine, which solves the technical problem that hot-plugging a GPU in a virtual machine requires occupying memory in the related technology, and removes the limitation of BIOS when inserting GPU resources into a virtual machine in the Kata process. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0026] Figure 1 is a hardware structure block diagram of a server according to an embodiment of the present invention;
[0027] Figure 2 is a flow chart of a method for adding GPU resources in a virtual machine according to an embodiment of the present invention;
[0028] Figure 3 is a flow chart of transparent transmission of GPU resources according to an embodiment of the present invention;
[0029] Figure 4 is a schematic diagram of a process of using a GPU in a virtual machine according to an embodiment of the present invention;
[0030] Figure 5 is an inner structure diagram of a virtual machine according to an embodiment of the present invention;
[0031] Figure 6 is a structural block diagram of a device for adding GPU resources in a virtual machine according to an embodiment of the present invention;
[0032] Figure 7 It is a structural block diagram of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.
[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0035] Example 1
[0036] The method embodiment provided in the first embodiment of the present invention can be executed in a computer, a VR device, a server or a similar data storage device. Taking running on a server as an example, Figure 1 FIG. 1 is a hardware structure diagram of a server according to an embodiment of the present invention. Figure 1 As shown, the server may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the server may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0037] The memory 104 can be used to store server programs, for example, software programs and modules of application software, such as a server program corresponding to a method for adding GPU resources in a virtual machine in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the server program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0038] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0039] In this embodiment, a method for adding GPU resources to a virtual machine is provided. Figure 2 is a flow chart of a method for adding GPU resources in a virtual machine according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0040] Step S202, obtaining a soft connection file, wherein the soft connection file is used to point to a cache file of a device plug-in of a K8S container cluster corresponding to the target virtual machine, and the cache file includes device identification information of a GPU resource;
[0041] In this embodiment, by inserting GPU resources in a physical machine, or adding a virtualized GPU, the GPU resources to be loaded can be added to the cache file of the device plug-in of the K8S container cluster. In addition to adding GPU resources, other types of logical devices can also be added. This embodiment only describes the penetration and use of GPU resources in the target virtual machine.
[0042] The target virtual machine of this embodiment may include K8S related components, KATA components, operating system components, etc.
[0043] Step S204, when creating a container group of the target virtual machine, obtaining device identification information of the GPU resource from the cache file according to the soft link file;
[0044] By parsing the file cache mechanism of the K8S component on the target virtual machine and linking the cache file in a soft link manner, the KATA component corresponding to the target virtual machine can correctly find the corresponding cache file. If the version or configuration of the K8S component changes, resulting in a change in the location of the cache file, the soft link file can be used to maintain the KATA component of the target virtual machine's access to the cache file by adjusting the soft link file, even in the process stage where the KATA configuration file has not yet been read, without modifying the source code.
[0045] Step S206, writing the device identification information corresponding to the currently created container group into the device list of the target virtual machine;
[0046] Step S208: Add the GPU resources corresponding to the device identification information in the device list to the target virtual machine.
[0047] Among them, the GPU resources can be abstracted as PCIE devices on the physical machine, so that the GPU resources can be inserted into the startup command of the operating system when the virtual machine is powered on, and then the GPU resources can be transparently transmitted to the target virtual machine.
[0048] Through the above steps, a soft connection file is obtained, wherein the soft connection file is used to point to the cache file of the device plug-in of the K8S container cluster corresponding to the target virtual machine, and the cache file includes the device identification information of the GPU resource. When creating the container group of the target virtual machine, the device identification information of the GPU resource is obtained from the cache file according to the soft connection file, and the device identification information corresponding to the currently created container group is written into the device list of the target virtual machine, and the GPU resource corresponding to the device identification information in the device list is added to the target virtual machine. By creating a soft connection file, the device identification information of the GPU resource is obtained from the cache file of K8S through the soft connection file, and then the GPU resource corresponding to the current container group is determined from the obtained device identification information of the GPU resource, and added to the device list of the target virtual machine, the GPU resource can be inserted into the startup command of the operating system when the virtual machine is powered on, and then the GPU resource is transparently transmitted to the target virtual machine, which solves the technical problem that hot-plugging a GPU in a virtual machine requires occupying memory in the related technology, and removes the limitation of BIOS when inserting GPU resources into a virtual machine in the Kata process.
[0049] Optionally, in the process of obtaining the soft link file, you can create a new soft link file in a fixed directory, which points to the cache file of the device plug-in under the current K8S configuration. Taking the K8S version and configuration file in the test environment as an example, the soft link will point to the file: / var / lib / kubelet / device-plugins / kubelet_internal_checkpoint. The location of this file will change with the change of the K8S configuration file. If the K8S configuration file changes, modify the corresponding soft link file.
[0050] In one implementation of the embodiment, based on the original process of the KATA community, an improvement is made. When creating the deployment unit Pod of the target virtual machine, before downloading the Runtime configuration (load Runtime Config), the device identification information of the GPU resource is obtained from the cache file according to the soft link file. The process of obtaining the device identification information of the GPU resource from the cache file according to the soft link file may include:
[0051] S11, determining the location of the cache file according to the soft link file;
[0052] S12, storing the device identification information in the cache file into a preset data structure of the KATA component corresponding to the target virtual machine, wherein the preset data structure is used to store the device identification information.
[0053] In the process of creating a Pod sandbox of a target virtual machine, configuring a preset data structure, wherein the preset data structure is used to indicate a data item of cache data storing a cache file;
[0054] In this embodiment, the preset data structure is used to define the preset data structure for receiving cached data in the K8S cache file, which is a data format for indicating which data items of the cached data need to be transmitted and stored. Read the soft link file, parse the read content, and store the information in the newly defined preset data structure.
[0055] The cache data of the preset data structure is written into the first OCI-SPEC data packet, wherein the cache data includes device information of the GPU device. In one example, the first OCI-SPEC data packet is a data packet transmitted when the K8S component communicates with the KATA component. Of course, it can also be other data packets transmitted between the K8S component and the KATA component, or data packets transmitted between other components.
[0056] In one example, writing the device identification information corresponding to the currently created container group into the device list of the target virtual machine includes: obtaining the first open container standard OCI-SPEC (OpenContainer Initiative SPEC) data packet transmitted between the K8S component and the KATA component; parsing the first Pod ID of the currently created container group from the first OCI-SPEC data packet; searching for the device identification information matching the first Pod ID in a preset data structure; and writing the device identification information matching the first PodID into the device list in the format of the OCI standard, wherein the device list is stored in the first OCI-SPEC data packet.
[0057] In some examples, the device identification information includes overall Pod device information, and searching for device identification information matching the first Pod ID in a preset data structure includes: enumerating the overall Pod device information of each GPU resource in the preset data structure; extracting the second Pod ID of each Pod device overall information; determining whether the first Pod ID is consistent with the second Pod ID; if the first Pod ID is consistent with the second Pod ID, determining that the second Pod ID matches the first Pod ID, and extracting the device identification information of the GPU resource corresponding to the second Pod ID from the preset data structure.
[0058] Parse the PodID information of the container currently being processed from the OCI-SPEC data packet, then enumerate the Pod device overall information in the preset data structure stored before, that is, the Pod Device Entire device information, and find the device data consistent with the PodID currently being processed from the Pod device overall information. If the PodID on the device information is consistent with the PodID of the container instance currently being processed, it means that it is the device that needs to be processed (in this embodiment, it refers to the GPU resources that need to be transparently transmitted). The Pod device overall information in this embodiment is a device description information, including the POD information and file path information corresponding to the device currently enumerated in the device list.
[0059] Optionally, in the process of writing the device information into the first OCI-SPEC data packet in the format of the OCI standard, the GPU resources that need to be transparently transmitted are written into the OCI-SPEC data packet device information data structure in the format of the OCI standard, thereby recording the GPU resources that need to be added and transparently transmitted.
[0060] In one implementation of the embodiment, after adding the GPU resources corresponding to the device identification information in the device list to the target virtual machine, it also includes: in the process of creating a Pod container of the target virtual machine, obtaining a second OCI-SPEC data packet transmitted between the K8S component and the KATA component; and removing the device identification information matching the first Pod ID in the second OCI-SPEC data packet.
[0061] In one example, removing device identification information matching the first Pod ID from the second OCI-SPEC data packet includes: acquiring pre-recorded device identification information matching the first Pod ID; searching for device identification information matching the first Pod ID in a device list in the second OCI-SPEC data packet; and removing device identification information matching the first Pod ID from the device list in the second OCI-SPEC data packet.
[0062] Read the device information of the previously processed devices (including GPU resources that have been transparently transmitted), read the device information in the OCI-SPEC data packet transmitted in the current process (creating the Pod container of the target virtual machine), and match the above device information. From the device information data structure of the OCI-SPEC data packet transmitted by the current process, remove the relevant devices that can be successfully matched. These devices are the devices that have been processed in the process of creating the Pod sandbox. When starting up, these devices have completed the process of transparent transmission through the QEMU (virtual operating system simulator, Quick EMUlator) startup parameters. By deleting the relevant device information in OCI-SPEC here, it is avoided that it is hot-plugged again in the KATA community process, avoiding repeated transparent transmission and improving the boot speed.
[0063] Figure 3 This is a flow chart of transparent transmission of GPU resources in an embodiment of the present invention. First, the K8S component sends a pause request to the OCI-SPEC preprocessing module. The OCI-SPEC preprocessing module obtains the device cache data in the cache file through a soft connection and stores it. In the subsequent process, the K8S component initiates a GPU container request to the OCI-SPEC preprocessing module. The OCI-SPEC preprocessing module reads the previously processed device, removes it from the OCI-SPEC data packet, and then continues to boot and run the container.
[0064] In one implementation of the embodiment, adding the GPU resource corresponding to the device identification information in the device list to the target virtual machine includes:
[0065] S21, obtaining device identification information of a GPU resource in a device list, wherein the target virtual machine is used to load a container instance;
[0066] Optionally, a new KATA configuration option is added: named VFIOWithLaunchParam (VFIO (Virtual Function IO) with launch parameters). The specified configuration option can select two flags, such as 0 and 1, corresponding to VFIO with launch parameters and VFIO without launch parameters respectively. The KATA component of this embodiment is a component running between the container and Docker.
[0067] S22, determining PCIE identification information corresponding to the GPU resource according to the device identification information;
[0068] In one of the examples, determining the PCIE identification information corresponding to the GPU resources based on the device identification information includes: obtaining a specified configuration option in the KATA component corresponding to the target virtual machine, wherein the specified configuration option is used to indicate a virtual machine interface with startup parameters; detecting a flag bit of the specified configuration option, wherein the flag bit is used to indicate whether to add GPU resources; in the case of determining to add GPU resources, determining the PCIE identification information based on the device identification information; storing the PCIE identification information in a specified data structure of the hypervisor configuration corresponding to the target virtual machine, wherein the specified data structure is used to store the PCIE identification information.
[0069] In one of the examples, determining PCIE identification information based on device identification information includes: enumerating all device identification information in a device list of a target virtual machine, and selecting a GPU resource whose device identification information meets preset conditions in the device list; judging whether the GPU resource is a valid device based on a kernel directory of a physical machine, wherein the physical machine is used to host the target virtual machine; and if the GPU resource is a valid device, extracting a PCIE number corresponding to the GPU resource from the kernel directory, wherein the PCIE identification information includes the PCIE number.
[0070] In some embodiments, the device identification information includes prefix characters and all characters of a device file path, and selecting a GPU resource in a device list whose device identification information meets preset conditions includes: extracting the prefix characters and all characters of the device file path for each device identification information in the device list; comparing the prefix characters with a first preset character, and comparing all characters with a second preset character; and determining that the device corresponding to the device identification information is a GPU resource that meets the preset conditions when the first preset character is the same as the prefix character and the second preset character is different from all characters.
[0071] Optionally, the data structure of Hyper visor Config in the KATA community source code can be modified by adding a Data.PCIEDevice data structure as a new data item in the data structure. The structure supports the input of a set of string data, which represents the number of the PCIE device.
[0072] In an example, the first preset character is / dev / VFIO, the second preset character is / dev / VFIO / VFIO, and each device in the device list is enumerated. For each enumerated device, it is determined whether its prefix is / dev / VFIO and the full name of the device is not / dev / VFIO / VFIO. If the conditions are met, the device is a VFIO device, that is, a GPU resource that meets the preset conditions.
[0073] Optionally, determining whether the GPU resource is a valid device according to the kernel directory of the physical machine includes: extracting the VFIO number of the virtual machine interface corresponding to the GPU resource; querying the kernel directory and determining whether the VFIO number exists in the kernel directory; and determining that the GPU resource is a valid device if the VFIO number exists in the kernel directory.
[0074] In one example, the kernel directory is / sys / kernel / iommu_groups / [VFIO number] / devices / , by extracting the VFIO number of the GPU resource and querying the related kernel directory / sys / kernel / iommu_groups / [VFIO number] / devices / . If the directory does not exist, the device is not a valid VFIO-PCIE device, and if it exists, it is a valid device.
[0075] In one implementation of this embodiment, after adding the identification information of the GPU resource to the specified data structure according to the specified configuration options, the method further includes: deduplicating the identification information in the specified data structure.
[0076] S23, start the target virtual machine, and add the corresponding GPU resources to the target virtual machine according to the PCIE identification information.
[0077] In this embodiment, the PCIE identification information includes a PCIE number, and adding the corresponding GPU resources to the target virtual machine according to the PCIE identification information includes: parsing the PCIE number to convert the PCIE number into a startup parameter of a virtual operating system simulator (QEMU, Quick EMUlator), and the startup parameter is used to add GPU resources during the process of starting the target virtual machine; adding the startup parameter to the startup command of the virtual operating system simulator; and executing the startup command to start the target virtual machine based on the image of the target virtual machine.
[0078] Optionally, the image of the target virtual machine includes a driver and a transparent transmission tool corresponding to the GPU resources, and the GPU resources and the driver are added to the container instance of the target virtual machine through the transparent transmission tool.
[0079] In a scenario, the above process of abstracting GPU resources is set in the create sandbox process of the original community process. It is executed before the community's new sandbox is called. In a complete example, the following operations are included:
[0080] S31, by checking the VFIOWithLaunchParam configuration option, determine whether the process is enabled through the flag bit of the configuration option.
[0081] S32, obtain a device list of the virtual machine, and enumerate all devices in the device list.
[0082] S33, for each enumerated device, determine whether its prefix is / dev / VFIO and the full name of the device is not / dev / VFIO / VFIO. If the conditions are met, the device is a GPU resource that meets the preset conditions.
[0083] S34, extract the VFIO number of the device, and query the kernel directory / sys / kernel / iommu_groups / [VFIO number] / devices / of the physical machine. If the kernel directory does not contain the above VFIO number, the device is not a valid VFIO-PCIE device.
[0084] S25, enumerate the device file name of each device from the above directory, extract its file name, and determine the file name as the PCIE number of the device. If a legal PCIE number is obtained through the above method, it is determined that the device is a valid VFIO-PCIEE device and needs to be processed.
[0085] S26, for a valid VFIO-PCIE device, erase it from the data structure of the device information in the device list, and at the same time, add the PCIE number parsed from the device to the data structure of the previously configured HypervisorConfig.
[0086] S27, deduplicate the PCIE numbers added to HyperviConfig.
[0087] In this embodiment, transparently transmitting the PCIE device to the target virtual machine includes: parsing the PCIE number of the PCIE device and converting the PCIE number into QEMU startup parameters; adding the QEMU startup parameters to the QEMU startup command; adding a driver for GPU resources in the image of the target virtual machine, and transparently transmitting the driver to the Docker container of the target virtual machine.
[0088] Figure 4The present invention is a flowchart of using a GPU in a virtual machine according to an embodiment of the present invention. First, the K8Scontainerd component issues a create pause container instruction, which is passed to the KATA component to execute a start virtual machine instruction. When the virtual machine is powered on, the GPU is transparently transmitted to the virtual machine, and the instruction is transmitted to QEMU (including KATA-agent). After the KATA-agent is successfully started, it is returned to the KATA component. The KATA component starts the pause container instruction to QEMU. After the pause container instruction is successfully started, QEMU notifies the KATA component and sends a message that the create pause container instruction is completed to the K8S containerd component. The K8Scontainerd component issues a create GPU container instruction to the KATA component. The KATA component starts the container instruction, and there is no need to hot-plug GPU resources. A virtual machine image with a GPU driver is used in QEMU.
[0089] Figure 5 It is an inner structure diagram of the virtual machine of the embodiment of the present invention, the bottom layer is the KATA-runtime component and the QEMU virtual machine kernel, the kernel includes the KATA-agent, the upper layer is the OCI-SPCE preprocessing module, the VFIO-PCIE hot plug rejection module, and the mirror driver module.
[0090] In this embodiment, in order to enable the GPU container to pass through the GPU under the architecture of K8S component + KATA component + QEMU, an additional module is added to the original architecture of the virtual machine, the OCI-SPEC preprocessing module, which adds corresponding processing procedures in creating a Pod Sandbox and creating a Pod container, respectively, so that the relevant device information required for starting the container instance can be obtained before starting. By means of soft links, the corresponding cache file is linked so that the KATA component can correctly find the corresponding cache file. If the version or configuration of the K8S component changes, resulting in a change in the location of the cache file, the solution of this embodiment can be used to maintain KATA's access to the cache file by adjusting the soft link without modifying the source code, even in the process stage where the KATA configuration file has not yet been read. A new preset data structure has been added. This data structure is used to store the content read and parsed from the K8S cache file. The process added in create Pod Sandbox is responsible for reading device information from the K8S cache file, and by analyzing its own OCI-SPCE data packet, it parses the specific device information related to the container instance currently being processed from the device information read from the cache, and appends the GPU resource information to the data structure of the OCI-SPCE data packet in the standard format of OCI-SPEC, and records the appended device information. In the process of create Pod Container, according to the recorded device information, the duplicate related device information is removed from the data structure of the OCI-SPEC data packet transmitted by the current process to prevent repeated processing of the GPU that has been transparently transmitted in the process of Create Container.
[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0092] Example 2
[0093] In this embodiment, a device for adding GPU resources to a virtual machine is also provided to implement the above embodiments and preferred implementations, which have been described and will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0094] Figure 6 is a structural block diagram of a device for adding GPU resources in a virtual machine according to an embodiment of the present invention, such as Figure 6 As shown, the device includes: a first acquisition module 60, a second acquisition module 62, a writing module 64, and a loading module 66, wherein:
[0095] A first acquisition module 60 is used to acquire a soft connection file, wherein the soft connection file is used to point to a cache file of a device plug-in of a K8S container cluster corresponding to a target virtual machine, and the cache file includes device identification information of a GPU resource;
[0096] A second acquisition module 62 is used to acquire the device identification information of the GPU resource from the cache file according to the soft link file when creating the container group of the target virtual machine;
[0097] A writing module 64, configured to write the device identification information corresponding to the currently created container group into a device list of the target virtual machine;
[0098] The loading module 66 is used to add the GPU resources corresponding to the device identification information in the device list to the target virtual machine.
[0099] Optionally, the second acquisition module includes: a determination unit, used to determine the location of the cache file based on the soft link file; a storage unit, used to store the device identification information in the cache file into a preset data structure of the KATA component corresponding to the target virtual machine, wherein the preset data structure is used to store device identification information.
[0100] Optionally, the writing module includes: an acquisition unit, used to acquire a first OCI-SPEC data packet transmitted between the K8S component and the KATA component; a parsing unit, used to parse the first Pod ID of the currently created container group from the first OCI-SPEC data packet; a search unit, used to search the device identification information matching the first PodID in the preset data structure; and a writing unit, used to write the device identification information matching the first Pod ID to the device list in accordance with the OCI standard format, wherein the device list is stored in the first OCI-SPEC data packet.
[0101] Optionally, the device identification information includes overall Pod device information, and the search unit includes: an enumeration subunit, used to enumerate the overall Pod device information of each of the GPU resources in the preset data structure; an extraction subunit, used to extract the second Pod ID of each of the overall Pod device information; a judgment subunit, used to judge whether the first Pod ID is consistent with the second Pod ID; an extraction subunit, used to determine that the second Pod ID matches the first Pod ID when the first Pod ID is consistent with the second Pod ID, and extract the device identification information of the GPU resource corresponding to the second Pod ID from the preset data structure.
[0102] Optionally, the device also includes: a third acquisition module, used to obtain a second OCI-SPEC data packet transmitted between the K8S component and the KATA component during the process of creating a Pod container of the target virtual machine after the loading module adds the GPU resources corresponding to the device identification information in the device list to the target virtual machine; and a removal module, used to remove the device identification information matching the first Pod ID in the second OCI-SPEC data packet.
[0103] Optionally, the elimination module includes: a first acquisition unit, used to obtain pre-recorded device identification information matching the first PodID; a search unit, used to search for device identification information matching the first Pod ID in a device list of the second OCI-SPEC data packet; and a elimination unit, used to eliminate the device identification information matching the first Pod ID in the device list of the second OCI-SPEC data packet.
[0104] Optionally, the loading module includes: an acquisition unit, used to obtain device identification information of the GPU resource in the device list, wherein the target virtual machine is used to load the container instance; a determination unit, used to determine the PCIE identification information corresponding to the GPU resource according to the device identification information; and a loading unit, used to start the target virtual machine and add the corresponding GPU resource to the target virtual machine according to the PCIE identification information.
[0105] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0106] Example 3
[0107] The embodiment of the present application also provides an electronic device, Figure 7 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Figure 7 As shown, it includes a processor 71, a communication interface 72, a memory 73 and a communication bus 74, wherein the processor 71, the communication interface 72, and the memory 73 communicate with each other through the communication bus 74, and the memory 73 is used to store computer programs; when the processor 71 is used to execute the program stored in the memory 73, the following steps are implemented: obtaining a soft connection file, wherein the soft connection file is used to point to a cache file of a device plug-in of a K8S container cluster corresponding to a target virtual machine, and the cache file includes device identification information of a GPU resource; when creating a container group of the target virtual machine, obtaining the device identification information of the GPU resource from the cache file according to the soft connection file; writing the device identification information corresponding to the currently created container group into the device list of the target virtual machine; adding the GPU resource corresponding to the device identification information in the device list to the target virtual machine.
[0108] Optionally, obtaining the device identification information of the GPU resources from the cache file according to the soft link file includes: determining the location of the cache file according to the soft link file; storing the device identification information in the cache file into a preset data structure of the KATA component corresponding to the target virtual machine, wherein the preset data structure is used to store the device identification information.
[0109] Optionally, the step of writing the device identification information corresponding to the currently created container group into the device list of the target virtual machine includes: obtaining a first OCI-SPEC data packet transmitted between the K8S component and the KATA component; parsing a first Pod ID of the currently created container group from the first OCI-SPEC data packet; searching the preset data structure for the device identification information matching the first Pod ID; and writing the device identification information matching the first Pod ID into the device list in the format of the OCI standard, wherein the device list is stored in the first OCI-SPEC data packet.
[0110] Optionally, the device identification information includes overall Pod device information, and searching the preset data structure for the device identification information matching the first Pod ID includes: enumerating the overall Pod device information of each GPU resource in the preset data structure; extracting the second Pod ID of each overall Pod device information; determining whether the first Pod ID is consistent with the second Pod ID; if the first Pod ID is consistent with the second Pod ID, determining that the second Pod ID matches the first Pod ID, and extracting the device identification information of the GPU resource corresponding to the second Pod ID from the preset data structure.
[0111] Optionally, after the step of adding the GPU resources corresponding to the device identification information in the device list to the target virtual machine, the method further includes: in the process of creating a Pod container of the target virtual machine, obtaining a second OCI-SPEC data packet transmitted between the K8S component and the KATA component; and removing the device identification information matching the first Pod ID in the second OCI-SPEC data packet.
[0112] Optionally, removing the device identification information matching the first Pod ID in the second OCI-SPEC data packet includes: obtaining pre-recorded device identification information matching the first Pod ID; searching for device identification information matching the first Pod ID in a device list in the second OCI-SPEC data packet; and removing the device identification information matching the first Pod ID in the device list in the second OCI-SPEC data packet.
[0113] Optionally, adding the GPU resources corresponding to the device identification information in the device list to the target virtual machine includes: obtaining the device identification information of the GPU resources in the device list, wherein the target virtual machine is used to load the container instance; determining PCIE identification information corresponding to the GPU resources according to the device identification information; starting the target virtual machine, and adding the corresponding GPU resources to the target virtual machine according to the PCIE identification information.
[0114] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0115] The communication interface is used for communication between the above terminal and other devices.
[0116] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0117] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0118] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer executes the method of adding GPU resources in a virtual machine as described in any of the above embodiments.
[0119] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the method of adding GPU resources in a virtual machine described in any one of the above embodiments.
[0120] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0121] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
[0122] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.
Claims
1. A method for adding GPU resources in a virtual machine, characterized in that: include: Obtain a soft connection file, wherein the soft connection file is used to point to a cache file of a device plug-in of a K8S container cluster corresponding to a target virtual machine, wherein the cache file includes device identification information of a GPU resource, wherein the target virtual machine includes a K8S component and a KATA group; When creating the container group of the target virtual machine, obtaining device identification information of the GPU resource from the cache file according to the soft connection file; Writing the device identification information corresponding to the currently created container group into a device list of the target virtual machine; Adding the GPU resources corresponding to the device identification information in the device list to the target virtual machine; Among them, the writing of the device identification information corresponding to the currently created container group into the device list of the target virtual machine includes: obtaining a first OCI-SPEC data packet transmitted between the K8S component and the KATA component; parsing the first PodID of the currently created container group from the first OCI-SPEC data packet; searching for the device identification information matching the first PodID in a preset data structure; and writing the device identification information matching the first Pod ID into the device list in the format of the OCI standard, wherein the device list is stored in the first OCI-SPEC data packet.
2. The method according to claim 1, characterized in that The acquiring the device identification information of the GPU resource from the cache file according to the soft link file includes: Determine the location of the cache file according to the soft link file; The device identification information in the cache file is stored in a preset data structure of the KATA component corresponding to the target virtual machine, wherein the preset data structure is used to store the device identification information.
3. The method according to claim 1, characterized in that The device identification information includes overall information of the Pod device, and searching the preset data structure for the device identification information matching the first Pod ID includes: Enumerate the overall Pod device information of each of the GPU resources in the preset data structure; Extract the second Pod ID of the overall information of each Pod device; Determine whether the first Pod ID is consistent with the second Pod ID; When the first Pod ID is consistent with the second Pod ID, it is determined that the second Pod ID matches the first PodID, and device identification information of the GPU resource corresponding to the second PodID is extracted from the preset data structure.
4. The method according to claim 1, characterized in that: After the step of adding the GPU resources corresponding to the device identification information in the device list to the target virtual machine, the method further includes: In the process of creating the Pod container of the target virtual machine, obtaining a second OCI-SPEC data packet transmitted between the K8S component and the KATA component; The device identification information matching the first Pod ID is removed from the second OCI-SPEC data packet.
5. The method according to claim 4, characterized in that The removing the device identification information matching the first Pod ID from the second OCI-SPEC data packet includes: Obtaining pre-recorded device identification information matching the first Pod ID; Searching for device identification information matching the first Pod ID in a device list of the second OCI-SPEC data packet; The device identification information matching the first Pod ID is removed from the device list of the second OCI-SPEC data packet.
6. The method according to claim 1, characterized in that The adding the GPU resource corresponding to the device identification information in the device list to the target virtual machine includes: Obtaining device identification information of a GPU resource in the device list, wherein the target virtual machine is used to load a container instance; Determine the PCIE identification information corresponding to the GPU resource according to the device identification information; The target virtual machine is started, and corresponding GPU resources are added to the target virtual machine according to the PCIE identification information.
7. A device for adding GPU resources to a virtual machine, characterized in that: include: A first acquisition module is used to acquire a soft connection file, wherein the soft connection file is used to point to a cache file of a device plug-in of a K8S container cluster corresponding to a target virtual machine, and the cache file includes device identification information of a GPU resource, wherein the target virtual machine includes a K8S component and a KATA group; A second acquisition module is used to acquire the device identification information of the GPU resource from the cache file according to the soft connection file when creating the container group of the target virtual machine; A writing module, used to write the device identification information corresponding to the currently created container group into a device list of the target virtual machine; A loading module, used to add the GPU resources corresponding to the device identification information in the device list to the target virtual machine; Among them, the writing module includes: an acquisition unit, used to acquire the first OCI-SPEC data packet transmitted between the K8S component and the KATA component; a parsing unit, used to parse the first Pod ID of the currently created container group from the first OCI-SPEC data packet; a search unit, used to search for the device identification information matching the first PodID in a preset data structure; a writing unit, used to write the device identification information matching the first Pod ID into the device list in accordance with the OCI standard format, wherein the device list is stored in the first OCI-SPEC data packet.
8. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program executes the method steps of any one of claims 1 to 6 when running.
9. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface, and the memory communicate with each other via a communication bus; wherein: Memory, used to store computer programs; A processor, configured to execute the method steps of any one of claims 1 to 6 by running a program stored in a memory.
