Method and apparatus for updating GPU driver software, host and storage medium

Abstract

The application provides a GPU driver software updating method and device, a host and a storage medium, and relates to the technical field of computers.The method comprises the following steps: detecting GPU calling information of a GPU application program running in a plurality of containers of a target virtual machine, and whether a GPU driver software version in a target guest system image deployed by the target virtual machine is compatible; the target virtual machine is any virtual machine in a plurality of virtual machines; if the GPU driver software version is incompatible, updating the GPU driver software version deployed in the target guest system image; and the GPU application program keeps running in the plurality of containers of other virtual machines. The GPU driver software version deployed in the target guest system image is updated, so that the consumption of additional resources is avoided, and the GPU application program running in the plurality of containers of other virtual machines can still keep running based on the GPU driver software version in the guest system image of other virtual machines, without suspending all services, so that the service stability is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a method, apparatus, host, and storage medium for updating GPU driver software. Background Technology

[0002] With the large-scale development of machine learning, the demand for computing power from GPUs (Graphics Processing Units) is also increasing. Container technology is currently a popular and cutting-edge technology, and its emergence has made the deployment of application software more convenient.

[0003] In related technologies, GPU applications are deployed to nodes via containers. The architecture in the container performs calculations through the GPU driver software on the host machine. Different GPU applications correspond to different architecture versions, meaning that different versions of the architecture and the GPU driver software on the host machine may have compatibility issues. When upgrading the GPU driver software on the node, all services on the node need to be paused and migrated before the upgrade can be performed.

[0004] However, in related technologies, all services on the node need to be paused and migrated before the GPU driver software on the host machine can be upgraded, which consumes additional machine resources and can easily lead to service instability. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by providing a method, apparatus, host, and storage medium for updating GPU driver software, thereby resolving the aforementioned technical problems in the related art.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0007] In a first aspect, embodiments of the present invention provide a method for updating GPU driver software, applied to a host, wherein multiple virtual machines are deployed on the host, each virtual machine deploys multiple containers and a guest system image, and the multiple containers run GPU applications, including:

[0008] The GPU call information of GPU applications running in multiple containers of the target virtual machine is detected, and the compatibility of the GPU driver software version in the target guest system image deployed on the target virtual machine is verified; the target virtual machine can be any virtual machine among the multiple virtual machines.

[0009] If incompatible, update the GPU driver software version deployed in the target guest system image;

[0010] Among the plurality of virtual machines, GPU applications are kept running in multiple containers of the other virtual machines besides the target virtual machine; the GPU call information of the GPU applications running in the multiple containers of the other virtual machines is compatible with the GPU driver software version in the guest system image deployed by the other virtual machines.

[0011] Secondly, embodiments of the present invention also provide a GPU driver software update device, applied to a host, wherein multiple virtual machines are deployed on the host, each virtual machine deploys multiple containers and a guest system image, and the multiple containers run GPU applications, including:

[0012] The detection module is used to detect the GPU call information of GPU applications running in multiple containers of the target virtual machine and whether it is compatible with the GPU driver software version in the target guest system image deployed on the target virtual machine; the target virtual machine is any virtual machine among the multiple virtual machines;

[0013] An update module is used to update the GPU driver software version deployed in the target visitor system image if incompatible.

[0014] Among the plurality of virtual machines, GPU applications are kept running in multiple containers of the other virtual machines besides the target virtual machine; the GPU call information of the GPU applications running in the multiple containers of the other virtual machines is compatible with the GPU driver software version in the guest system image deployed by the other virtual machines.

[0015] Thirdly, embodiments of the present invention also provide a host computer, including a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to implement the GPU driver software update method described in any of the first aspects above.

[0016] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein when the computer program is read and executed, it implements the GPU driver software update method described in any of the first aspects above.

[0017] The beneficial effects of this invention are as follows: This invention provides a method for updating GPU driver software, applied to a host machine. The host machine deploys multiple virtual machines, each containing multiple containers and a guest system image. GPU applications run within the containers. The method includes: detecting whether the GPU call information of the GPU applications running in the containers of the target virtual machine is compatible with the GPU driver software version in the target guest system image deployed on the target virtual machine; the target virtual machine can be any virtual machine among the multiple virtual machines; if incompatible, the GPU driver software version deployed in the target guest system image is updated; wherein, among the multiple virtual machines, GPU applications continue to run in the containers of the other virtual machines besides the target virtual machine; the GPU call information of the GPU applications running in the containers of the other virtual machines is compatible with the GPU driver software version in the guest system image deployed on the other virtual machines. The GPU call information of the GPU application running in the target virtual machine is incompatible with the GPU driver software version in the target guest system image. Only the GPU driver software version deployed in the target guest system image needs to be updated, avoiding additional resource consumption. Meanwhile, the GPU applications running in multiple containers in other virtual machines can still continue to run based on the GPU driver software version in the guest system image of other virtual machines without pausing all services, thus improving service stability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of a virtual machine deployed on a host according to an embodiment of the present invention;

[0021] Figure 3 A flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention;

[0022] Figure 4 A flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention;

[0023] Figure 5 A flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention;

[0024] Figure 6 A flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention;

[0025] Figure 7 A flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention;

[0026] Figure 8 A flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention;

[0027] Figure 9 A schematic diagram of a GPU driver software update device provided in an embodiment of the present invention;

[0028] Figure 10 This is a schematic diagram of the structure of a host provided in an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0030] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0031] In the description of this application, it should be noted that if the terms "upper", "lower", etc. appear to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship that the product of this application is usually placed in, it is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0032] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0034] With the large-scale development of machine learning, the demand for computing power from GPUs (Graphics Processing Units) is also increasing. Container technology is currently a popular and cutting-edge technology, and its emergence has made the deployment of application software more convenient.

[0035] In related technologies, GPU applications are deployed to nodes via containers. The architecture within the containers performs computations using GPU driver software on the host machine. Different GPU applications correspond to different architecture versions, meaning that compatibility issues can arise between different architecture versions and the GPU driver software on the host machine. Upgrading the GPU driver software on a node requires pausing and migrating all services on that node before the upgrade can proceed. However, this process of pausing and migrating all services on the node before upgrading the GPU driver software on the host machine consumes additional machine resources and can easily lead to service instability.

[0036] To address the aforementioned technical problems in related technologies, this application provides a method for updating GPU driver software. Multiple virtual machines are deployed on a host, each containing multiple containers and a guest system image. GPU applications run within the containers. If the GPU call information of the GPU applications running in the containers of a target virtual machine is incompatible with the GPU driver software version in the target guest system image deployed on the target virtual machine, updating the GPU driver software version deployed in the target guest system image is sufficient, avoiding additional resource consumption. Meanwhile, GPU applications running in the containers of other virtual machines can still continue running based on the GPU driver software versions in the guest system images of those other virtual machines, without needing to pause other services, thus improving service stability.

[0037] The following is an explanation of the terms used in the embodiments of this application.

[0038] GPU (Graphics Processing Unit): A microprocessor used to perform graphics calculations.

[0039] OCI (Open Container Initiative, an open standard for operating system-level virtualization containers): defines the container runtime standard.

[0040] Kata Containers: An open-source secure container project that provides a container runtime compatible with OCI (Open Container Initiative) through virtualization technology. It is a novel implementation of a lightweight virtual machine that can be seamlessly integrated into the container ecosystem.

[0041] k8s (Kubernetes): An open-source resource orchestration system for managing containerized applications.

[0042] runc: The default OCI-compliant container runtime command-line tool for Kubernetes.

[0043] Containerd: An industry-standard open-source container runtime.

[0044] Pod: The smallest deployable unit that can create and manage Kubernetes computing. It consists of one or more containers and is also the basic granularity of Kubernetes scheduling. It can request a certain amount of resources, such as 2 CPU cores and 4GB of memory.

[0045] PCI: short for Peripheral Component Interconnect, is an interconnect bus standard that connects computer motherboards and peripheral components.

[0046] PCIe (Peripheral Component Interconnect Express): A higher-speed serial communication system standard was built on the PCI (Peripheral Component Interconnect) bus.

[0047] NVIDIA GPU Driver: The driver software for NVIDIA GPUs, used to provide the low-level interface for the device.

[0048] GPU Device Plugin: A device plugin compatible with the k8s device interface.

[0049] CUDA (Compute Unified Device Architecture): A unified computing architecture

[0050] CDI (Container Device Interface): A standard interface specification for using third-party devices during container runtime.

[0051] Guest Image: KVM (Kernel-based Virtual Machine, an open-source system virtualization module) guest system image.

[0052] VFIO: A user-space driver framework that can safely expose device I / O, interrupts, DMA, etc. to user space, allowing user-space processes to directly access hardware using the VFIO driver.

[0053] VFIO-PCI Driver: A wrapper around the VFIO device driver, providing the ability to access devices for the upper-layer VFIO interface.

[0054] Host machine: The server in the Kubernetes cluster, also known as a node.

[0055] The following explains a method for updating GPU driver software provided in the embodiments of this application.

[0056] Figure 1 This is a flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method may include:

[0057] S101. Detect the GPU call information of GPU applications running in multiple containers of the target virtual machine and whether they are compatible with the GPU driver software version in the target guest system image deployed on the target virtual machine.

[0058] The host machine has multiple virtual machines deployed on it. Each virtual machine has multiple containers and a guest system image deployed on it. The multiple containers run GPU applications. The target virtual machine can be any virtual machine among the multiple virtual machines. In this embodiment of the application, the number of target virtual machines is not specifically limited.

[0059] It should be noted that a GPU application refers to an application that requires the use of GPU computing power during its operation, and GPU call information can be used to identify the GPU version that needs to be called.

[0060] In some implementations, GPU applications run in multiple containers of the target virtual machine. During the operation of the GPU applications, GPU computing power needs to be called. The GPU call information and the GPU driver software version in the target guest system image deployed on the target virtual machine are checked for compatibility.

[0061] In addition, each virtual machine has a corresponding Pod (root container), and the Pod contains multiple containers.

[0062] S102. If incompatible, update the GPU driver software version deployed in the target guest system image;

[0063] Among these, GPU applications are kept running in multiple containers of virtual machines other than the target virtual machine; the GPU call information of the GPU applications running in multiple containers of other virtual machines is compatible with the GPU driver software version in the guest system image deployed on other virtual machines.

[0064] It should be noted that if the GPU call information of the GPU applications running in multiple containers of the target virtual machine is incompatible with the GPU driver software version in the target guest system image deployed on the target virtual machine, then the GPU applications cannot call the GPU driver software version in the target guest system image. It is necessary to update the GPU driver software version in the target guest system image according to the GPU call information so that the target guest system image has the updated GPU driver software deployed.

[0065] In this embodiment of the application, when the GPU driver software in the target guest system image is updated, the GPU driver software in the guest system images of other virtual machines is not affected and can work normally. Whether the GPU driver software in the guest system images of other virtual machines is updated depends on the compatibility of the GPU driver software version in the guest system images of other virtual machines and the GPU call information of the GPU applications running in other virtual machines.

[0066] In summary, this invention provides a method for updating GPU driver software, applied to a host machine. The host machine deploys multiple virtual machines, each containing multiple containers and a guest system image. GPU applications run within the containers. The method includes: detecting whether the GPU call information of the GPU applications running in the containers of the target virtual machine is compatible with the GPU driver software version in the target guest system image deployed on the target virtual machine; the target virtual machine can be any virtual machine among the multiple virtual machines; if incompatible, the GPU driver software version deployed in the target guest system image is updated; wherein, among the multiple virtual machines, GPU applications continue to run in the containers of the other virtual machines besides the target virtual machine; the GPU call information of the GPU applications running in the containers of the other virtual machines is compatible with the GPU driver software version in the guest system image deployed on the other virtual machines. The GPU call information of the GPU application running in the target virtual machine is incompatible with the GPU driver software version in the target guest system image. Only the GPU driver software version deployed in the target guest system image needs to be updated, avoiding additional resource consumption. Meanwhile, the GPU applications running in multiple containers in other virtual machines can still continue to run based on the GPU driver software version in the guest system image of other virtual machines without pausing all services, thus improving service stability.

[0067] Figure 2 This is a schematic diagram of a virtual machine deployed on a host according to an embodiment of the present invention, as shown below. Figure 2 As shown, the host machine has three virtual machines: A, B, and C. Virtual machine A contains process A, virtual machine B contains process B, and virtual machine C contains process C. Each process runs a GPU application as a container. Virtual machines A, B, and C each have their own namespaces and hardware virtualization. Guest system A is deployed on virtual machine A, guest system B on virtual machine B, and guest system C on virtual machine C. The guest system can also be referred to as a guest system image.

[0068] Optional, Figure 3 This is a flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention, as shown below. Figure 3 As shown, prior to the process in S101 above of detecting the GPU call information of GPU applications running in multiple containers of the target virtual machine and whether the GPU driver software version in the target guest system image deployed on the target virtual machine is compatible, the method may further include:

[0069] S201. Compile the target kernel according to the preset kernel configuration parameters.

[0070] The target kernel can be the Kata kernel, and the target kernel can support GPU devices, which can be NVIDIA GPUs.

[0071] It should be noted that the preset kernel configuration parameters may include: configuration parameters that support hot-plugging of PCI / PCIe devices, configuration parameters that support loading kernel modules, and configuration parameters that enable MMIO (Memory-mapped I / O) methods for PCIe devices.

[0072] The various kernel configuration parameters are represented as follows:

[0073] CONFIG_HOTPLUG_PCI_PCIE = y / / Supports hot-plugging of PCI / PCIe devices;

[0074] CONFIG_MODULES = y / / Supports loading kernel modules (NVIDIA drivers);

[0075] CONFIG_MODULE_UNLOAD = y;

[0076] CONFIG_PCI_MMCONFIG = y / / Enables the MMIO method for PCIe devices.

[0077] The commands for compiling the kernel are as follows:

[0078] $. / build-kernel.sh -v 5.15.23-g nvidia-f setup

[0079] $. / build-kernel.sh-v 5.15.23-g nvidia build

[0080] $sudo-E. / build-kernel.sh-v 5.15.23-g nvidia install

[0081] And build the deb package: make deb-pkg

[0082] S202. Based on the target kernel and GPU driver software, build the target guest system image for the target virtual machine.

[0083] Specifically, based on the target kernel, the GPU driver software is installed in the target guest system image to ensure that the target guest system image contains the GPU driver software. Optionally, the GPU driver software can be an NVIDIA GPU Driver.

[0084] Additionally, after installing the GPU driver software in the target guest system image, the target kernel in the target guest system image can be deleted.

[0085] S203. During the startup of the target virtual machine, load the target guest system image.

[0086] The target visitor system image can be a guest image.

[0087] In this embodiment of the application, a target virtual machine can be created. After the target virtual machine is created, during the process of starting the target virtual machine, a target guest system image can be loaded.

[0088] In summary, the target kernel is compiled based on preset kernel configuration parameters; and a target guest system image is built for the target virtual machine based on the target kernel and GPU driver software. This allows for the quick and efficient construction and deployment of the target guest system image within the target virtual machine.

[0089] Optional, Figure 4 This is a flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention, as shown below. Figure 4 As shown, the process of building a target guest system image for the target virtual machine based on the target kernel and GPU driver software in S202 above may include:

[0090] S401. Based on the target kernel and root file system, install the GPU driver software in the target guest system image.

[0091] S402. Based on the GPU driver software, root file system, and preset toolset, build the target guest system image for the target virtual machine.

[0092] In some implementations, a root file system (rootfs) can be created and multiple preset objects can be mounted into the root file system. The root file system is then entered, and the GPU driver software and preset toolset are installed according to the root file system and the header files of the target kernel, and a pre-boot hook is set. Subsequently, the multiple preset objects mounted in the root file system are unmounted and the header files of the target kernel are deleted. Based on the GPU driver software, the root file system, and the preset toolset, a target guest system image is packaged.

[0093] Optionally, the aforementioned preset objects may include: sysfs, proc, tmpfs, and dev (device). Among them, sysfs is a memory-based file system that provides kernel information to users in the form of files, proc is a process and system file system, and tmpfs is a temporary file system, which is a memory-based file system.

[0094] Optionally, the GPU driver software can be NVIDIA GPU Driver, and the default toolset can be NVIDIA Container Toolkit.

[0095] The command to create the root file system is as follows:

[0096] $sudo-E USE_DOCKER=true EXTRA_PKGS="apt kmod gcc make curl gnupglibc-dev libc6-dev g++ca-certificates" SECCOMP=no. / rootfs.sh Ubuntu.

[0097] The commands for installing the GPU driver and preset toolset are as follows:

[0098] $. / nvidia-installer-k 5.15.23-nvidia-gpu-aq--ui=none

[0099] $apt install nvidia-container-toolkit

[0100] The command to set the pre-startup hook is as follows:

[0101] OCI Pre Hook:

[0102] vim / usr / share / oci / hooks / prestart / nvidia-container-toolkit.sh:

[0103] #! / bin / bash-x / usr / bin / nvidia-container-toolkit-debug$@

[0104] The command to package the target visitor system image is as follows:

[0105] $image_builder.sh PATH_TO_ROOTFS / rootfs-ubuntu

[0106] In this embodiment, the target guest system image not only contains GPU driver software and root file system, but also a preset toolset, which allows hooks to be started according to the contents of the preset toolset in subsequent processes, so as to associate the GPU device with multiple containers.

[0107] Optional, Figure 5 This is a flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention, as shown below. Figure 5 As shown, prior to the process in S101 above of detecting GPU call information of GPU applications running in multiple containers of the target virtual machine, and whether the GPU driver software version in the target guest system image deployed on the target virtual machine is compatible, the method may further include:

[0108] S501. Based on the bus address of the GPU device, unbind the GPU device from the host's GPU driver software.

[0109] In some implementations, the GPU device is unbound from the host's GPU driver software based on its bus address, and then bound to the VFIO-PCI Driver (a wrapper for the VFIO device driver that provides device access capabilities for the upper-layer VFIO interface; VFIO is a hardware driver framework).

[0110] For example, the bus address of the GPU device can be 0000:1a:00.0, and the implementation code for the above process is as follows:

[0111] $echo 0000:1a:00.0> / sys / bus / pci / drivers / nvidia / unbind

[0112] $echo"vfio-pci"> / sys / bus / pci / devices / 0000:1a:00.0 / driver_override

[0113] $echo 0000:1a:00.0> / sys / bus / pci / drivers / vfio-pci / bind

[0114] S502: Mount the GPU device to multiple containers in the target virtual machine.

[0115] In this scenario, the GPU device is associated with multiple containers within the target virtual machine, enabling the GPU device to directly access multiple containers within the target virtual machine.

[0116] It should be noted that you can first associate the GPU device with the target virtual machine, and then mount the GPU device to multiple containers on the target virtual machine.

[0117] In summary, based on the GPU device's bus address, the GPU device is unbound from the host's GPU driver software; and then associated with multiple containers within the target virtual machine. This achieves the transformation from binding the GPU device to the host to associating it with multiple containers, facilitating the containers' access to the GPU device while running GPU applications.

[0118] Optional, Figure 6 This is a flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention, as shown below. Figure 6 As shown, the process of mounting the GPU device onto multiple containers in the target virtual machine in S502 above may include:

[0119] S601 uses a preset plugin to bind the various device functions of the GPU device to the target virtual machine according to the preset configuration file.

[0120] The default configuration file can be a CDI configuration file. The CDI configuration file can describe the attributes of various device functions on the GPU device, mainly including: device file path, whether it is attached to PCI or PCIe, etc.

[0121] Optionally, the default plugin can be the GPU Device Plugin.

[0122] In some implementations, a GPU Device Plugin is used to bind the various device functions on the GPU device to the target virtual machine according to the CDI configuration file.

[0123] S602. Mount the various device functions of the GPU device into multiple containers of the target virtual machine.

[0124] In this embodiment of the application, binding each device function of the GPU device to multiple containers of the target virtual machine can enable the various device functions of the GPU device to pass through to multiple containers of the target virtual machine.

[0125] In summary, by using pre-defined plugins and configuration files, GPU devices are bound to the target virtual machine and then to multiple containers within that virtual machine. This method makes the binding process more efficient and reliable.

[0126] Optionally, the target virtual machine also deploys the target kernel. The kernel can be pre-compiled into each virtual machine, and the kernel in the target virtual machine can be the target kernel.

[0127] Figure 7 This is a flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention, as shown below. Figure 7 As shown, the process of binding the various device functions of the GPU device to the target virtual machine in S601 above may include:

[0128] S701 uses a preset plugin to associate the various device functions of the GPU device with the container runtime through container annotations based on the preset configuration file.

[0129] In some implementations, a GPU Device Plugin is used to pass through the various device functions of the GPU device to the underlying container runtime through container annotations based on the CDI configuration file. This container runtime can be Kata Containers that implement the OCI standard.

[0130] In this embodiment of the application, a container runtime can be used to run the container. When running a container runtime, the target kernel needs to be applied.

[0131] S702 uses a container runtime, a guest system image, and a target kernel to bind the various device functions of the GPU device to the PCIe bus, thereby binding the GPU device to the target virtual machine.

[0132] In some implementations, if multiple containers requiring GPU resources are created, a container runtime (KataContainers), a guest system image, and a target kernel are used to associate the various device functions in the GPU device with the PCI bus, PCIe bus, or PCIe switch through preset parameters, thereby binding the various device functions in the GPU device to the target virtual machine.

[0133] Among them, the preset parameters can be qemu (emulated processor software) parameters.

[0134] For example, multiple containers that require GPU resources can be:

[0135] requests:nvidia.com / TU102_GEFORCE_RTX_2080_TI:"1".

[0136] In summary, by using pre-defined plugins and configuring pre-defined settings, the various functions of the GPU device are associated with the container runtime through container annotations. Then, using the container runtime, guest system image, and target kernel, the various functions of the GPU device are bound to the PCIe bus, thereby binding the GPU device to the target virtual machine. Binding each device of the GPU device to the target virtual machine provides a more detailed association, facilitating effective scheduling of the GPU device.

[0137] Optionally, the process of mounting the GPU device into multiple containers of the target virtual machine in S602 above may include:

[0138] Before running multiple containers on the target virtual machine, hooks are initiated based on the preset toolset in the target guest system image to mount the various device functions of the GPU device into multiple containers on the target virtual machine.

[0139] In some implementations, an OCI pre-start hook is used before multiple containers of the target virtual machine run to mount the GPU device-related object files into multiple containers of the target virtual machine. This allows the GPU device's various functions to be mounted into multiple containers of the target virtual machine, enabling the management of GPU driver software within the guest system image.

[0140] In summary, this method enables the direct pass-through of various device functions of the GPU device to multiple containers of the target virtual machine.

[0141] Optional, Figure 8 This is a flowchart illustrating a method for updating GPU driver software according to an embodiment of the present invention, as shown below. Figure 8 As shown, the method may further include:

[0142] S801: Use a preset plugin to collect the resource usage of the GPU device.

[0143] S802 uses a scheduler to obtain resource usage information and allocates GPU resources to multiple containers based on the resource usage information.

[0144] In some implementations, a GPU Device Plugin is used to collect the resource usage of GPU devices and report the resource usage of GPU devices to an API server. A scheduler is used to obtain the resource usage from the API server and, based on the resource usage, allocates GPU device resources to multiple containers using a preset resource allocation strategy.

[0145] In the embodiments of this application, GPU resources can be scheduled and allocated at the Kubernetes level. For users, GPU resources can be used just like ordinary CPU / MEM (memory display program) resources, and GPU devices can be managed in a more granular way.

[0146] Optionally, the scheduler can be Kube-scheduler.

[0147] In summary, this application provides a method for updating GPU driver software, applied to a host machine. The host machine deploys multiple virtual machines, each containing multiple containers and a guest system image. GPU applications run within the containers. The method includes: detecting GPU call information of the GPU applications running in the containers of the target virtual machine, and whether the GPU driver software version in the target guest system image deployed on the target virtual machine is compatible; the target virtual machine can be any of the multiple virtual machines; if incompatible, the GPU driver software version deployed in the target guest system image is updated. Since the GPU call information of the GPU applications running in the target virtual machine is incompatible with the GPU driver software version in the target guest system image, only the GPU driver software version deployed in the target guest system image needs to be updated, avoiding additional resource consumption. Meanwhile, the GPU applications running in the containers of other virtual machines can still continue running based on the GPU driver software versions in the guest system images of those other virtual machines, without needing to pause all services, thus improving service stability.

[0148] Furthermore, it enables the management of NVIDIA GPU Drivers within the Guest Image and allows GPU devices on the host machine to pass through to virtual machine containers, ensuring that the use of GPUs in virtual machines and containers is independent of the host machine. This way, version management of the NVIDIA GPU Drivers only requires updating the Guest Image, without affecting the nodes or the services on them, and without requiring additional machine resources or manual labor costs.

[0149] The following describes the GPU driver software update apparatus, host, and storage medium used to execute the GPU driver software update method provided in this application. For the specific implementation process and technical effects, please refer to the relevant content of the GPU driver software update method above, which will not be repeated below.

[0150] Figure 9 This is a schematic diagram of the structure of a GPU driver software update device provided in an embodiment of the present invention, as shown below. Figure 9 As shown, the device is applied to a host computer on which multiple virtual machines are deployed. Each virtual machine contains multiple containers and a guest system image. The multiple containers run GPU applications, including:

[0151] The detection module 901 is used to detect the GPU call information of GPU applications running in multiple containers of the target virtual machine and whether it is compatible with the GPU driver software version in the target guest system image deployed by the target virtual machine; the target virtual machine is any virtual machine among the multiple virtual machines;

[0152] The update module 902 is used to update the GPU driver software version deployed in the target guest system image if incompatible; wherein, among the plurality of virtual machines, GPU applications are kept running in multiple containers of the other virtual machines besides the target virtual machine; the GPU call information of the GPU applications running in the multiple containers of the other virtual machines is compatible with the GPU driver software version in the guest system image deployed on the other virtual machines.

[0153] Optionally, the device further includes:

[0154] The compilation module is used to compile the target kernel according to preset kernel configuration parameters;

[0155] A build module is used to build the target guest system image for the target virtual machine based on the target kernel and the GPU driver software;

[0156] The loading module is used to load the target guest system image during the startup of the target virtual machine.

[0157] Optionally, the building module is further configured to install the GPU driver software in the target guest system image based on the target kernel and root file system; and to build the target guest system image for the target virtual machine based on the GPU driver software, the root file system, and a preset toolset.

[0158] Optionally, the device further includes:

[0159] The unbinding module is used to unbind the GPU device from the GPU driver software of the host according to the bus address of the GPU device;

[0160] The mounting module is used to mount the GPU device to multiple containers in the target virtual machine.

[0161] Optionally, the mounting module is further configured to use a preset plugin to bind each device function of the GPU device to the target virtual machine according to a preset configuration file; and to mount each device function of the GPU device to multiple containers of the target virtual machine.

[0162] Optionally, the target virtual machine also deploys the target kernel. The mounting module is further specifically used to use the preset plugin to associate the various device functions of the GPU device with the container runtime through container annotations according to the preset configuration file; and to use the container runtime, the guest system image, and the target kernel to bind the various device functions of the GPU device to the PCIe bus of the target virtual machine, so as to bind the GPU device to the target virtual machine.

[0163] Optionally, the mounting module is further configured to, before the multiple containers of the target virtual machine run, activate hooks according to the preset toolset in the target guest system image to mount the various device functions of the GPU device to the multiple containers of the target virtual machine.

[0164] Optionally, the device further includes:

[0165] The reporting module is used to collect the resource usage of the GPU device using the preset plugin;

[0166] The allocation module is used to obtain the resource usage information using a scheduler, and allocate the GPU device resources to the multiple containers based on the resource usage information.

[0167] In summary, the GPU call information of the GPU application running in the target virtual machine is incompatible with the GPU driver software version in the target guest system image. Only the GPU driver software version deployed in the target guest system image needs to be updated, avoiding additional resource consumption. Meanwhile, the GPU applications running in multiple containers in other virtual machines can still continue to run based on the GPU driver software version in the guest system images of other virtual machines without pausing all services, thus improving service stability.

[0168] The above-described apparatus is used to execute the method provided in the foregoing embodiments, and its implementation principle is similar, so it will not be described again here.

[0169] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0170] Figure 10 This is a schematic diagram of the structure of a host provided in an embodiment of the present invention, such as... Figure 10 As shown, the host includes: processor 1001 and memory 1002.

[0171] The memory 1002 is used to store programs. The processor 1001 calls the programs stored in the memory 1002 to execute the above-described embodiment of the GPU driver software update method. The specific implementation and technical effects are similar, and will not be described in detail here.

[0172] For example, this GPU driver software update method, applied to a host on which multiple virtual machines are deployed, each virtual machine deploying multiple containers and a guest system image, the multiple containers running GPU applications, may include:

[0173] The GPU call information of GPU applications running in multiple containers of the target virtual machine is detected, and the compatibility of the GPU driver software version in the target guest system image deployed on the target virtual machine is verified; the target virtual machine can be any virtual machine among the multiple virtual machines.

[0174] If incompatible, the GPU driver software version deployed in the target guest system image will be updated; wherein, among the plurality of virtual machines, GPU applications continue to run in multiple containers of the other virtual machines besides the target virtual machine; the GPU call information of the GPU applications running in the multiple containers of the other virtual machines is compatible with the GPU driver software version in the guest system image deployed on the other virtual machines.

[0175] Optionally, before detecting the GPU call information of GPU applications running in multiple containers of the target virtual machine and whether they are compatible with the GPU driver software version in the target guest system image deployed by the target virtual machine, the method further includes:

[0176] Compile the target kernel according to the preset kernel configuration parameters;

[0177] Based on the target kernel and the GPU driver software, construct the target guest system image for the target virtual machine;

[0178] During the startup of the target virtual machine, the target guest system image is loaded.

[0179] Optionally, the step of building the target guest system image for the target virtual machine based on the target kernel and the GPU driver software includes:

[0180] Based on the target kernel and root file system, install the GPU driver software in the target guest system image;

[0181] Based on the GPU driver software, the root file system, and the preset toolset, build the target guest system image for the target virtual machine.

[0182] Optionally, before detecting GPU call information of GPU applications running in multiple containers of the target virtual machine, and whether the GPU driver software version in the target guest system image deployed by the target virtual machine is compatible, the method further includes:

[0183] Based on the GPU device's bus address, the GPU device is unbound from the host's GPU driver software;

[0184] The GPU device is mounted onto multiple containers within the target virtual machine.

[0185] Optionally, associating the GPU device with multiple containers in the target virtual machine includes:

[0186] Using a preset plugin and a preset configuration file, the various device functions of the GPU device are bound to the target virtual machine;

[0187] The various device functions of the GPU device are mounted into multiple containers of the target virtual machine.

[0188] Optionally, the target virtual machine also deploys the target kernel, and the step of binding the various device functions of the GPU device to the target virtual machine includes:

[0189] Using the aforementioned preset plugin, the various device functions of the GPU device are associated with the container runtime through container annotations according to the preset configuration file;

[0190] Using the container runtime, the guest system image, and the target kernel, the various device functions of the GPU device are bound to the PCIe bus of the target virtual machine, thereby binding the GPU device to the target virtual machine.

[0191] Optionally, mounting the various device functions of the GPU device into multiple containers of the target virtual machine includes:

[0192] Before the multiple containers of the target virtual machine run, hooks are activated according to the preset toolset in the target guest system image to mount the various device functions of the GPU device into the multiple containers of the target virtual machine.

[0193] Optionally, the method further includes:

[0194] The preset plugin is used to collect the resource usage of the GPU device;

[0195] The scheduler obtains the resource usage information and allocates the GPU device resources to the multiple containers based on the resource usage information.

[0196] In summary, the host machine deploys multiple virtual machines, each containing multiple containers and a guest system image. GPU applications run within these containers. If the GPU call information of the GPU applications running in the containers of the target virtual machine is incompatible with the GPU driver software version in the target guest system image, updating the GPU driver software version in the target guest system image is sufficient, avoiding additional resource consumption. Meanwhile, the GPU applications running in the containers of other virtual machines can still continue running based on the GPU driver software versions in the guest system images of those other virtual machines, without interrupting other services. This improves service stability, has no impact on the nodes or services on them, and requires no additional machine resources or labor costs.

[0197] Optionally, the present invention also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, is used to perform the above-described method embodiments.

[0198] For example, this GPU driver software update method, applied to a host on which multiple virtual machines are deployed, each virtual machine deploying multiple containers and a guest system image, the multiple containers running GPU applications, may include:

[0199] The GPU call information of GPU applications running in multiple containers of the target virtual machine is detected, and the compatibility of the GPU driver software version in the target guest system image deployed on the target virtual machine is verified; the target virtual machine can be any virtual machine among the multiple virtual machines.

[0200] If incompatible, the GPU driver software version deployed in the target guest system image will be updated; wherein, among the plurality of virtual machines, GPU applications continue to run in multiple containers of the other virtual machines besides the target virtual machine; the GPU call information of the GPU applications running in the multiple containers of the other virtual machines is compatible with the GPU driver software version in the guest system image deployed on the other virtual machines.

[0201] Optionally, before detecting the GPU call information of GPU applications running in multiple containers of the target virtual machine and whether they are compatible with the GPU driver software version in the target guest system image deployed by the target virtual machine, the method further includes:

[0202] Compile the target kernel according to the preset kernel configuration parameters;

[0203] Based on the target kernel and the GPU driver software, construct the target guest system image for the target virtual machine;

[0204] During the startup of the target virtual machine, the target guest system image is loaded.

[0205] Optionally, the step of building the target guest system image for the target virtual machine based on the target kernel and the GPU driver software includes:

[0206] Based on the target kernel and root file system, install the GPU driver software in the target guest system image;

[0207] Based on the GPU driver software, the root file system, and the preset toolset, build the target guest system image for the target virtual machine.

[0208] Optionally, before detecting GPU call information of GPU applications running in multiple containers of the target virtual machine, and whether the GPU driver software version in the target guest system image deployed by the target virtual machine is compatible, the method further includes:

[0209] Based on the GPU device's bus address, the GPU device is unbound from the host's GPU driver software;

[0210] The GPU device is mounted onto multiple containers within the target virtual machine.

[0211] Optionally, associating the GPU device with multiple containers in the target virtual machine includes:

[0212] Using a preset plugin and a preset configuration file, the various device functions of the GPU device are bound to the target virtual machine;

[0213] The various device functions of the GPU device are mounted into multiple containers of the target virtual machine.

[0214] Optionally, the target virtual machine also deploys the target kernel, and the step of binding the various device functions of the GPU device to the target virtual machine includes:

[0215] Using the aforementioned preset plugin, the various device functions of the GPU device are associated with the container runtime through container annotations according to the preset configuration file;

[0216] Using the container runtime, the guest system image, and the target kernel, the various device functions of the GPU device are bound to the PCIe bus of the target virtual machine, thereby binding the GPU device to the target virtual machine.

[0217] Optionally, mounting the various device functions of the GPU device into multiple containers of the target virtual machine includes:

[0218] Before the multiple containers of the target virtual machine run, hooks are activated according to the preset toolset in the target guest system image to mount the various device functions of the GPU device into the multiple containers of the target virtual machine.

[0219] Optionally, the method further includes:

[0220] The preset plugin is used to collect the resource usage of the GPU device;

[0221] The scheduler obtains the resource usage information and allocates the GPU device resources to the multiple containers based on the resource usage information.

[0222] In summary, the host machine deploys multiple virtual machines, each containing multiple containers and a guest system image. GPU applications run within these containers. If the GPU call information of the GPU applications running in the containers of the target virtual machine is incompatible with the GPU driver software version in the target guest system image, updating the GPU driver software version in the target guest system image is sufficient, avoiding additional resource consumption. Meanwhile, the GPU applications running in the containers of other virtual machines can still continue running based on the GPU driver software versions in the guest system images of those other virtual machines, without interrupting other services. This improves service stability, has no impact on the nodes or services on them, and requires no additional machine resources or labor costs.

[0223] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0224] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0225] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0226] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0227] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for updating GPU driver software, characterized in that, This applies to a host machine, on which multiple virtual machines are deployed. Each virtual machine contains multiple containers and a guest system image. The multiple containers run GPU applications, including: The GPU call information of GPU applications running in multiple containers of the target virtual machine is detected, and the compatibility of the GPU driver software version in the target guest system image deployed on the target virtual machine is verified; the target virtual machine can be any virtual machine among the multiple virtual machines. If incompatible, update the GPU driver software version deployed in the target guest system image; Among the plurality of virtual machines, GPU applications are kept running in multiple containers of the other virtual machines besides the target virtual machine; the GPU call information of the GPU applications running in the multiple containers of the other virtual machines is compatible with the GPU driver software version in the guest system image deployed by the other virtual machines. Before detecting GPU call information of GPU applications running in multiple containers of the target virtual machine and whether they are compatible with the GPU driver software version in the target guest system image deployed by the target virtual machine, the method further includes: Compile the target kernel according to the preset kernel configuration parameters; Based on the target kernel and the GPU driver software, construct the target guest system image for the target virtual machine; During the startup of the target virtual machine, the target guest system image is loaded.

2. The method according to claim 1, characterized in that, The step of building the target guest system image for the target virtual machine based on the target kernel and the GPU driver software includes: Based on the target kernel and root file system, install the GPU driver software in the target guest system image; Based on the GPU driver software, the root file system, and the preset toolset, build the target guest system image for the target virtual machine.

3. The method according to claim 1 or 2, characterized in that, Before detecting GPU call information of GPU applications running in multiple containers of the target virtual machine and whether they are compatible with the GPU driver software version in the target guest system image deployed by the target virtual machine, the method further includes: Based on the GPU device's bus address, the GPU device is unbound from the host's GPU driver software; The GPU device is mounted onto multiple containers within the target virtual machine.

4. The method according to claim 3, characterized in that, The step of mounting the GPU device onto multiple containers in the target virtual machine includes: Using a preset plugin and a preset configuration file, the various device functions of the GPU device are bound to the target virtual machine; The various device functions of the GPU device are mounted into multiple containers of the target virtual machine.

5. The method according to claim 4, characterized in that, The target virtual machine also deploys a target kernel, and the step of binding the various device functions of the GPU device to the target virtual machine includes: Using the aforementioned preset plugin, the various device functions of the GPU device are associated with the container runtime through container annotations according to the preset configuration file; Using the container runtime, the guest system image, and the target kernel, the various device functions of the GPU device are bound to the PCIe bus of the target virtual machine, thereby binding the GPU device to the target virtual machine.

6. The method according to claim 4, characterized in that, The step of mounting the various device functions of the GPU device into multiple containers of the target virtual machine includes: Before the multiple containers of the target virtual machine run, according to the preset toolset in the target guest system image, the various device functions of the GPU device are mounted into the multiple containers of the target virtual machine.

7. The method according to claim 4, characterized in that, The method further includes: The preset plugin is used to collect the resource usage of the GPU device; The scheduler obtains the resource usage information and allocates the GPU device resources to the multiple containers based on the resource usage information.

8. A GPU driver software update device, characterized in that, This applies to a host machine, on which multiple virtual machines are deployed. Each virtual machine contains multiple containers and a guest system image. The multiple containers run GPU applications, including: The detection module is used to detect the GPU call information of GPU applications running in multiple containers of the target virtual machine and whether it is compatible with the GPU driver software version in the target guest system image deployed on the target virtual machine; the target virtual machine is any virtual machine among the multiple virtual machines; An update module is used to update the GPU driver software version deployed in the target visitor system image if incompatible. Among the plurality of virtual machines, GPU applications are kept running in multiple containers of the other virtual machines besides the target virtual machine; the GPU call information of the GPU applications running in the multiple containers of the other virtual machines is compatible with the GPU driver software version in the guest system image deployed by the other virtual machines. The device further includes: The compilation module is used to compile the target kernel according to preset kernel configuration parameters; A build module is used to build the target guest system image for the target virtual machine based on the target kernel and the GPU driver software; The loading module is used to load the target guest system image during the startup of the target virtual machine.

9. A host computer, characterized in that, include: A memory and a processor, the memory storing a computer program executable by the processor, the processor executing the computer program to implement the GPU driver software update method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when read and executed, implements the GPU driver software update method according to any one of claims 1-7.

Images