A GPU device mounting and unmounting method, system, device, medium and product in a container

Abstract

The application discloses a GPU device mounting and unmounting method, system, device, medium and product in a container, relates to the field of computer software application, and comprises the following steps: determining a cgroup version according to a mounting list of an operating system, determining a cgroup directory of the container, and determining that a device file corresponding to a GPU that needs to be mounted by a user is located in a / dev directory of a host computer, and a device file corresponding to a GPU that needs to be unmounted by the user is located in a / dev directory of the container; based on user demand, traversing the two / dev directories, determining a GPU device name that needs to be mounted for the container and a GPU device name that needs to be mounted for the container, and configuring access permissions; and based on the cgroup version, mounting or unmounting the GPU device according to the GPU device file and the access permissions, so that the application does not need to stop and restart the container, avoids the problem of data loss caused by re-running of a program, and improves system performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer software applications, and in particular to a GPU device mounting and unmounting method, system, device, medium and product in a container. BACKGROUND

[0002] Container technology is the basis of cloud computing, allowing applications and their dependencies, such as libraries, environment variables, configuration files, etc., to be packaged into an independent, portable running unit, i.e., a container. Unlike traditional virtual machines, containers directly share the host's operating system kernel, and achieve resource allocation and process isolation through the namespace and cgroup isolation mechanisms.

[0003] If a running container needs to switch GPU devices, it usually needs to go through the following steps.

[0004] 1. Stop the currently running container.

[0005] 2. Start a new container, and specify the new GPU device to be used in the container through the start parameter or option.

[0006] In the above steps, the container needs to go through a stop, i.e., restart process, which will stop the application in the container, and then restart with the new container. In this process, the application runtime data will inevitably go through a discard, i.e., reload process.

[0007] For some programs that do not have instant saving, such as fluent, FEKO, this will cause the loss of program running data.

[0008] Another solution is to add a series of infrastructure, such as GPU virtualization, to form a GPU device abstraction layer, such as the most commonly used nvidia GPU operator with MIG in actual engineering, which will form a virtual GPU device layer, such as a vGPU layer, and then reassign the vGPU to the container, which requires restarting the container.

[0009] This type of solution needs to meet the following conditions.

[0010] 1. A virtualized GPU device abstraction needs to be added to simulate GPU hardware.

[0011] 2. It depends on a specific platform as a component / plug-in of the platform, such as kubernetes (k8s) etc.

[0012] 3. If the GPU device changes, this virtualization / abstraction layer needs to be reinitialized / restarted.

[0013] This solution requires a large amount of infrastructure, applications, plug-ins, integration with specific platforms, design / implementation difficulties, virtualization / device abstraction / monitoring, and consumption of certain system resources, which can easily lead to performance problems. SUMMARY

[0014] The purpose of the present application is to provide a GPU device mounting and unmounting method, system, device, medium and product in a container to solve the problems of program running data loss and large system resource consumption.

[0015] For container mounting and unmounting GPU devices, the general method is to restart the container, that is, to stop the container, create a new container, specify the GPU device to be mounted for the container when starting the new container, and the container manager creates a GPU device file for the container and configures the correct access permission configuration for the created GPU. The present application dynamically creates device files and access permission configurations for the container, so there is no need to stop and restart.

[0016] To achieve the above purpose, the present application provides the following solutions.

[0017] In a first aspect, the present application provides a GPU device mounting and unmounting method in a container, comprising the following steps.

[0018] Determine the cgroup version according to the mounting list of the operating system.

[0019] Based on the cgroup version, determine the cgroup directory of the container, and according to user needs, place the device file corresponding to the GPU that the user needs to mount in the / dev directory of the host machine, and place the device file corresponding to the GPU that the user needs to unmount in the / dev directory of the container.

[0020] Based on the user needs, traverse the / dev directory of the host machine to determine the GPU device name that needs to be mounted for the container, and traverse the / dev directory of the container to determine the GPU device name that needs to be unmounted for the container.

[0021] According to the GPU device name that needs to be mounted for the container, create the GPU device file that needs to be mounted for the container and configure the access permission, and according to the GPU device name that needs to be unmounted for the container, configure the access permission for the existing device in the container.

[0022] Based on the cgroup version, mount or unmount the GPU device according to the GPU device file and the access permission.

[0023] In a second aspect, the present application provides a GPU device mounting and unmounting system in a container, comprising the following modules.

[0024] A cgroup version determining module is configured to determine the cgroup version according to a mounting list of the operating system.

[0025] A container cgroup directory determining module is configured to determine a cgroup directory of the container according to the cgroup version and the user demand, and place the device file corresponding to the GPU that needs to be mounted by the user in the / dev directory of the host computer and place the device file corresponding to the GPU that needs to be unmounted by the user in the / dev directory of the container.

[0026] A GPU device name determining module is configured to traverse the / dev directory of the host computer to determine the GPU device name that needs to be mounted for the container and traverse the / dev directory of the container to determine the GPU device name that needs to be unmounted for the container according to the user demand.

[0027] A GPU device file creating and access permission configuring module is configured to create the GPU device file that needs to be mounted for the container and configure the access permission according to the GPU device name that needs to be mounted for the container, and configure the access permission for the existing device in the container according to the GPU device name that needs to be unmounted for the container.

[0028] A mounting and unmounting module is configured to mount or unmount the GPU device according to the cgroup version, the GPU device file and the access permission.

[0029] In a third aspect, the present application provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the GPU device mounting and unmounting method in the container.

[0030] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the GPU device mounting and unmounting method in the container.

[0031] In a fifth aspect, the present application provides a computer program product, which comprises a computer program, wherein the computer program is executable on a processor to implement the GPU device mounting and unmounting method in the container.

[0032] According to the specific embodiments provided in the application, the application has the following technical effects: based on the cgroup version, the application determines the cgroup directory of the container, and according to the user demand, the device node of the / dev directory of the host machine is traversed to obtain the GPU device name that needs to be mounted into the container, based on which the GPU device file with the GPU device in the container is determined, and the access permission is configured to mount the GPU device; the device node of the / dev directory of the container is traversed to obtain the GPU device name that needs to be mounted for the container, based on which the access permission of the container to the device is configured under the cgroup directory of the container, so as to achieve the effect of unmounting the GPU device for the container; based on the above steps, the GPU device file and the access permission are dynamically created in the running state of the container, without stopping and restarting the container, the change of the GPU device can be adapted, the continuity of the program running is maximally ensured, and the problem of data loss caused by re-running of the program is avoided. In addition, the cgroup version is directly determined according to the mounting list of the operating system, the mounting and unmounting are performed on the operating system, no GPU abstraction layer or resource abstraction layer is needed, no corresponding system resources such as virtualization / device abstraction / monitoring need to be consumed, and the system performance is improved. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0034] Figure 1 A flowchart of a GPU device mounting and unmounting method in a container according to an embodiment of the application.

[0035] Figure 2 A flowchart of a GPU device mounting and unmounting method in a container according to another embodiment of the application.

[0036] Figure 3 A device structure tree corresponding to a GPU0 device according to an embodiment of the application.

[0037] Figure 4 A structure diagram of a computer device according to an embodiment of the application. DETAILED DESCRIPTION

[0038] With reference to the drawings and specific embodiments, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0039] In order to make the purposes, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0040] As shown in the figure, the embodiment of the present application provides a mounting and unmounting method of GPU device in a container, which includes the following steps. Figure 1

[0041] S1: determining the cgroup version according to the mounting list of the operating system.

[0042] S2: based on the cgroup version, determining the cgroup directory of the container, and according to the user demand, placing the device file corresponding to the GPU that the user needs to mount under the / dev directory of the host, and placing the device file corresponding to the GPU that the user needs to unmount under the / dev directory of the container.

[0043] S3: based on the user demand, traversing the / dev directory of the host to determine the GPU device name that needs to be mounted for the container, and traversing the / dev directory of the container to determine the GPU device name that needs to be unmounted for the container.

[0044] S4: according to the GPU device name that needs to be mounted for the container, creating the GPU device file that needs to be mounted for the container and configuring the access permission, and according to the GPU device name that needs to be unmounted for the container, configuring the access permission for the existing device in the container.

[0045] S5: based on the cgroup version, mounting or unmounting the GPU device according to the GPU device file and the access permission.

[0046] In an exemplary embodiment, S1 specifically includes: obtaining the mounting list of the operating system through the "mount" command; determining the cgroup version according to the mounting list; the cgroup version includes cgroup v1 system and cgroup v2 system.

[0047] In actual application, in the mounting item of cgroup v1 system, type is cgroup; in the mounting item of cgroup v2 system, type is cgroup2.

[0048] ​The specific mounting path of the cgroup in the mounting item can also be obtained, for example, the cgroup file system is mounted under the / sys / fs / cgroup directory.

[0049] In an exemplary embodiment, S2 specifically comprises: based on the cgroup version, traversing all subdirectories of the cgroup mounting directory according to the user demand, and screening the cgroup directory of the container from all subdirectories; the user demand is to mount or unmount the GPU device.

[0050] In practical application, all subdirectories of the cgroup mounting directory are traversed to find the cgroup directory of the container, for example, the cgroup directory of the container is as follows.

[0051] cgroup v1 system: / sys / fs / cgroup / devices / docker-cidxxx.

[0052] cgroup v2 system: / sys / fs / cgroup / docker-cidxxx.

[0053] In an exemplary embodiment, S3 specifically comprises: determining the device file name according to the manufacturer's device manual; determining the device detailed information corresponding to the device file name in the / dev directory of the host or the / dev directory of the container by using the "ls -l" command; determining the GPU device name according to the device detailed information; the GPU device name is the GPU device name that needs to be mounted for the container or the GPU device name that needs to be unmounted for the container.

[0054] In practical application, the GPU device name of the GPU device that needs to be newly mounted for the container is found in the / dev directory of the host: GPU0, major_id and minor_id, for example, the device that needs to be mounted is GPU0, major_id: 195, and minor_id: 0.

[0055] When setting the access permission of the device through the file, the control file of the device access permission is located in the cgroup directory of the container.

[0056] The GPU device name of the GPU device that needs to be unmounted is found in the / dev directory in the container: major_id and minor_id, for example, the device that needs to be unmounted is GPU0, major_id: 195, and minor_id: 1.

[0057] In practical applications, first find the device file name provided by the GPU manufacturer according to the manual, such as the device name of nvidia is the file name starting with nvidiaX, and the device name of Huawei Ascott is the file name starting with davinciX, wherein X is a number from 0 to 7,

[0058] Find the detailed information of the device under the / dev directory of the host computer by the "ls -l" command, which contains the major / minor id of the device.

[0059] Taking the nvidia GPU as an example, execute the "ls -l / dev / nvidia*" command, and the output similar to the following can be seen, that is, there are three GPUs in the operating system: nvidia0, nvidia1, and nvidia2, as shown in Table 1.

[0060] Table 1: Schematic diagram of three GPUs in the operating system

[0061]

[0062] The corresponding major / minor id are 195 / 0, 195 / 1, and 195 / 2, respectively.

[0063] In an exemplary embodiment, S5 specifically comprises: when the cgroup version is a cgroup v1 system, determining a device access control file of the container under the cgroup directory, and creating a device node under the / dev directory of the container according to the GPU device file; based on the device node, configuring the access permission of the GPU under the cgroup directory of the container according to the access permission, thereby completing mounting or unmounting the GPU device.

[0064] In an exemplary embodiment, the access permission of the GPU to be mounted is determined, r: read, w: write, m: modify, and the default can be rwm.

[0065] In an exemplary embodiment, S5 specifically comprises: when the cgroup version is a cgroup v2 system, creating a device node under the / dev directory of the container according to the GPU device file; based on the device node, creating a bpf program according to the access permission; the bpf program returns "1" for read, write and modification requests of the GPU device that needs to be mounted, and returns "0" for read, write and modification requests of the GPU device that needs to be unmounted; compiling the bpf program into a bpf kernel module; loading the bpf kernel module in the operating system and unbinding the device type bpf program that the container has bound; binding the newly compiled bpf kernel module in the operating system to the cgroup where the container is located to mount or unmount the GPU device; wherein, in the binding process, the bpf type is specified as a device type.

[0066] As shown in Figure 2 Another embodiment of the present application provides a GPU device mounting and unmounting method in a container, the specific steps are as follows.

[0067] Step 1: The mounting list of the operating system can be obtained by the mount command, and the version of the cgroup is determined by the list.

[0068] Step 2: Traverse all subdirectories of the cgroup mounting directory in step 1 above to find the cgroup directory of the container, for example, the cgroup directory of the container is as follows.

[0069] cgroup v1 system: / sys / fs / cgroup / devices / docker-cidxxx.

[0070] cgroup v2 system: / sys / fs / cgroup / docker-cidxxx.

[0071] Step 3: Find the device name of the GPU that needs to be newly mounted for the container under the / dev directory of the host: gpu0, major_id and minor_id, for example, the device that needs to be mounted is gpu0, major_id: 195, minor_id: 0

[0072] Step 4: Find the major_id and minor_id of the GPU that needs to be unmounted in the / dev directory of the container, for example, the device that needs to be unmounted is gpu0, major_id: 195, minor_id: 1

[0073] Step 5: Determine the access permission of the GPU that needs to be mounted, r: read, w: write, m: modify, and the default can be rwm.

[0074] In practical applications, the steps for uninterrupted mounting / demounting of GPU devices in the cgroup v1 system are as follows.

[0075] cgv1 step 1: Find the device access control files of the container under the cgroup directory of the container (obtained in step 2): devices.allow and devices.deny. cgv1 step 1 determines the location of the device access control files in the cgroup v1 system.

[0076] cgv1 step 2: Create a device node for the device to be mounted under the / dev directory of the container, with the node type being c (character) and the character type device. cgv1 step 2 creates a device file for the device to be mounted in the container.

[0077] To create a device file, you can use the following command: "mknod -m 666 / dev / gpu0 c 195 0".

[0078] Where "666" is the access permission of the device file, / dev / gpu0 is the device file name in the container, and 195 / 0 is the major / minor id of the device.

[0079] cgv1 step 3: Write the device to be mounted (obtained in step 3) into the file devices.allow in the following format.

[0080] That is: c major_id:minor_id rwm

[0081] For example, if you need to mount a GPU device with major_id 195 and minor_id 0, the content to be written is: c195:0 rwm.

[0082] If you need to mount multiple devices, organize the file content as one line per device, and then write the file content into devices.allow.

[0083] In cgv1 step 3, configure appropriate access permissions for the device created in cgv1 step 2, so that the processes in the container can access the device file created in cgv1 step 2.

[0084] In the operating system (e.g., Linux system), everything is a file. If there is no device file, even if the device access permission is configured, the process has no means to access the device; if there is no appropriate permission or the device access permission file is written in the wrong location, the processes in the container cannot correctly access the device file, which is equivalent to being unable to access the device.

[0085] Two conditions are required for the container to access the GPU device correctly. If the following two conditions are met, the container does not need to be stopped / restarted.

[0086] 1. The container has a device file corresponding to the device.

[0087] 2. The device is configured with the correct access permission (file).

[0088] The above two conditions are the device access mechanism specified by the Linux kernel and the cgroup v1 system.

[0089] cgv1 step 4: Write the GPU device that needs to be unloaded in the format of the file devices.deny.

[0090] For example, if a GPU device with major_id 195 and minor_id 1 needs to be unloaded, the content written is: c195:1 rwm.

[0091] If multiple devices need to be unloaded, organize the file content as one device per line, and then write the file content to devices.deny.

[0092] Complete steps 1-5 above and cgv1 steps 1-cgv1 step 4 to mount / unload / modify GPU devices on the cgroup v1 system.

[0093] In practical applications, the steps for uninterrupted mounting / unloading of GPU devices in the cgroup v2 system are as follows.

[0094] The cgroup v1 system and the cgroup v2 system have very different access control methods for GPU devices. Cgropuv2 abandons the black and white list (devices.allow / devices.deny) file of cgroup v1 and instead uses the bfp method to control the access of processes in the cgroup to the device.

[0095] Considering that a considerable portion of older operating systems, especially those before 2020, use the cgroup v1 system, while operating systems after 2020 use the cgroup v2 system, it can be seen that the present application covers all existing cgroup versions.

[0096] cgv2 step 1: Create a device node for the device that needs to be mounted under the container's / dev directory, with the node type being c (character), a character device. For example, for device: GPU0, major_id: 194, minor_id: 0.

[0097] Create a device file, which can be done by the following command: "mknod -m 666 / dev / gpu0 c 195 0", where "666" is the access permission of the device file, " / dev / gpu0" is the device file name in the container, and 195 / 0 is the major / minor id of the device.

[0098] If multiple devices need to be mounted, create device files one by one.

[0099] cgv2 step 2: create a bpf program according to the above-mentioned mounted and unmounted devices, and the bpf program returns 1 for read / write / modify requests for devices to be mounted and 0 for read / write / modify requests for devices to be unmounted.

[0100] cgv2 step 3: compile the bpf program in cgv2 step 2 into a bpf kernel module.

[0101] cgv2 step 4: load the bpf kernel module generated in cgv2 step 3 into the operating system

[0102] cgv2 step 5: unbind the device type bpf that the container has bound.

[0103] cgv2 step 6: bind the bpf kernel module that has been loaded into the operating system in cgv2 step 4 to the cgroup (step 2) where the container is located, and specify the type of bpf as device type during the binding process.

[0104] Complete the above steps 1-5 and cgv2 steps 1-cgv2 steps 6, which can mount / unmount / modify GPU devices for containers on cgroup v2 system without stopping / restarting the container.

[0105] Modern GPU devices allow GPUs to be divided into smaller instances due to the need for resource utilization, that is, GPU slicing functions, such as nvidia's MIG (Multi instance GPU).

[0106] For a GPU device, a tree relationship is formed between the GPU device itself and the slices of the GPU, and the following rules are followed during mounting / unmounting.

[0107] Rule 1: Among the multiple nodes of the same level in a GPU device tree, some nodes need to be mounted to the container and some nodes need to be unmounted from the container. From the root node of the tree to the common parent node of the nodes that need to be mounted / unmounted, the nodes need to be mounted to the container, and then according to the cgroup version, follow the above steps.

[0108] 1) Unmounted child GPU devices.

[0109] 2) Mounted child GPU devices.

[0110] Rule 2: All sibling child nodes of a GPU device tree need to be unmounted from the container. Depending on the cgroup version, all device nodes from the root node to the child node need to be unmounted from the container according to the above steps.

[0111] Rule 3: All nodes of a GPU device tree are not mounted in the container. If it is needed to mount the nodes on the device tree to the container, all device nodes from the root node to the node need to be mounted in the container according to the above steps depending on the cgroup version.

[0112] The technical solutions of the present application are further illustrated below with specific examples.

[0113] The initial device situation is as follows.

[0114] Suppose there are two GPUs on a host.

[0115] GPU0 (name: gpu0, major_id: 195, minor_id: 0), abbreviated as gpu0.

[0116] GPU1 (name: gpu0, major_id: 195, minor_id: 1), abbreviated as gpu1.

[0117] gpu0 is further subdivided as follows.

[0118] GPU instance0 (name: gi0, major_id: 239, minor_id: 39), abbreviated as gi0.

[0119] GPU instance1 (name: gi1, major_id: 239, minor_id: 42), abbreviated as gi1.

[0120] And GPU instance0 is further divided as follows.

[0121] Compute Instance0 (name: ci0, major_id: 239, minor_id: 40), abbreviated as ci0.

[0122] Compute instance1 (name: ci1, major_id: 239, minor_id: 41), abbreviated as ci1.

[0123] From the above, the device structure tree corresponding to the GPU0 device can be drawn as shown in Figure 3

[0124] GPU1 has only one device node because it is not divided. The following illustrates how to mount the GPU device for the container without interruption.

[0125] The pre-preparation steps for the container to unload the GPU device are as follows.

[0126] Precondition: Assume that the container cidxxx has mounted the gpu0 / gi4 / ci0 mig slice, and the file location of the system cgroup is / sys / fs / cgroup. The following are the specific steps to unload the GPU device.

[0127] Step 1: Get the type of cgroup and the mounting location of cgroup through the mount command.

[0128] Step 2: Find the file location of the container group.

[0129] cgroup v1 system: / sys / fs / cgroup / devices / docker / cidxxx / .

[0130] cgroup v2 system: / sys / fs / cgroup / docker / cidxxx / .

[0131] Step 3: Enter the container and get which GPU devices and GPU mig device slices have been mounted in the container. The container-mounted GPU devices can be obtained, as shown in Table 2.

[0132] Table 2. Partial code table of container-mounted GPU devices

[0133]

[0134] From the output, it can be seen that three GPU devices are mounted in the system, as shown in Table 3.

[0135] Table 3. Illustration table of three mounted GPU devices

[0136]

[0137] Step 4: The permission of the device to be unloaded is "rwm".

[0138] The following discusses the cgroup v1 system and the cgroup v2 system.

[0139] ​The cgroup v1 system is as follows.

[0140] Step 5, find devices.allow and devices.deny from the directory / sys / fs / cgroup / devices / docker / cidxxx.

[0141] Step 6, write the device type, major_id, and minor_id into the devices.deny file, and the written content is shown in Table 4.

[0142] Table 4 devices.deny file writing content table

[0143]

[0144] After completing the above 6 steps, the GPU device in the container can be unloaded in the cgroup v1 system.

[0145] The cgroup v2 system is as follows.

[0146] The cgroup v2 system and the cgroup v1 system are both prepared in the above steps 1-4, and the difference starts from step 5, so the following step 5 is the next step in the cgroup v2 system.

[0147] Step 5, prepare a bpf program, and when there is a request to access the following GPU device node, return a value of 0, i.e., prohibit access, as shown in Table 5.

[0148] Table 5 Partial table for unloading GPU device in cgroup v2 system

[0149]

[0150] Step 6, compile the bpf program in step 5 into a bpf kernel module.

[0151] Step 7, load the bpf module in step 6 into the system.

[0152] Step 8, unbind the bpf kernel module that the container cidxxx has bound from the original.

[0153] Step 9, bind the bpf module that has been loaded into the system in step 7 to the container cidxxx, and specify the bpf module type as a device type.

[0154] After completing the above 9 steps, the GPU device in the container can be unloaded in the cgroup v2 system.

[0155] The pre-preparation steps for the container to mount a new GPU device are as follows.

[0156] Precondition: Assuming that the container cidxxx does not mount any GPU device, now it is necessary to mount the gpu0 / gi0 / ci0 device into the container cidxxx, the following are the specific mounting steps of the GPU device.

[0157] Step 1, get the type of cgroup and the mounting location of cgroup through the mount command.

[0158] Step 2, find the file location of the container's group.

[0159] cgroup v1 system: / sys / fs / cgroup / devices / docker / cidxxx / .

[0160] cgroup v2 system: / sys / fs / cgroup / docker / cidxxx / .

[0161] Step 3, get the GPU device to be mounted and the GPU mig device slice information in the operating system as shown in Table 6.

[0162] Table 6 GPU device to be mounted and GPU mig device slice information under the premise of not mounting any GPU device

[0163]

[0164] From the output, the following GPU devices need to be mounted into the container, as shown in Table 7.

[0165] Table 7 GPU devices that need to be mounted into the container under the premise of not mounting any GPU device

[0166]

[0167] Step 4, the permission of the device to be mounted is "rwm".

[0168] The following are classified discussions for cgroup v1 system and cgroup v2 system.

[0169] cgroup v1 system is as follows.

[0170] Step 5, find devices.allow and devices.deny from the directory / sys / fs / cgroup / devices / docker / cidxxx.

[0171] Step 6, create device nodes for gpu device in container: gpu0, gi0, ci0 according to step 3 above.

[0172] Step 7, write device type, major_id, minor_id into devices.allow file, the content written is shown in Table 8.

[0173] Table 8. devices.allow file writing content table without mounting any GPU device

[0174]

[0175] After completing the above 7 steps, the GPU device gpu0 / gi0 / ci0 can be mounted in the cgroup v1 system.

[0176] The cgroup v2 system and the cgroup v1 system are prepared in the above steps 1-4, and the difference starts from step 5, so the following step 5 is the next step in the cgroup v2 system.

[0177] Step 5, create device nodes for GPU device in container: gpu0, gi0, ci0 according to step 3

[0178] Step 6, prepare bpf program according to the output of step 3, for the following GPU device nodes, when there is a request to access, the return value is 1, that is, access is allowed, as shown in Table 9.

[0179] Table 9. Partial table of mounting GPU device in cgroup v2 system without mounting any GPU device

[0180]

[0181] Step 7, compile the bpf program in step 6 into a bpf kernel module.

[0182] Step 8, load the bpf module in step 6 into the system.

[0183] Step 9, unbind the bpf kernel module that the container cidxxx has bound from the original container.

[0184] Step 10, bind the bpf module loaded into the system in step 7 to the container cidxxx, and specify the bpf module type as device type.

[0185] After completing the above 10 steps, the GPU device gpu0 / gi0 / ci0 can be mounted in the cgroup v2 system.

[0186] The preparation step for modifying the existing device of the container is as follows.

[0187] Precondition: It is assumed that the device gpu1 has been mounted in the container, and now the gpu0 / gi0 / ci0 device needs to be mounted in the container cidxxx. The following are the specific operation steps of mounting / unmounting the GPU device.

[0188] Step 1: Get the type of cgroup and the mounting location of cgroup through the mount command.

[0189] Step 2: Find the file location of the group of the container.

[0190] cgroup v1 system: / sys / fs / cgroup / devices / docker / cidxxx / .

[0191] cgroup v2 system: / sys / fs / cgroup / docker / cidxxx / .

[0192] Step 3: Get the GPU device to be mounted and the GPU mig device slice information in the operating system as shown in Table 10.

[0193] Table 10: GPU device to be mounted and GPU mig device slice information under the premise of mounting device gpu1

[0194]

[0195] From the output, the following GPU devices need to be mounted in the container as shown in Table 11.

[0196] Table 11: GPU device to be mounted in the container under the premise of mounting device gpu1

[0197]

[0198] Step 4: Find the name, major_id, and minor_id of the GPU device to be unmounted in the / dev directory of the container as shown in Table 12.

[0199] Table 12: Name of GPU device to be unmounted under the premise of mounting device gpu1

[0200]

[0201] Step 5: The permission of the device to be mounted is "rwm".

[0202] The following is a classification discussion of the cgroup v1 system and the cgroup v2 system.

[0203] The cgroup v1 system is as follows.

[0204] The modification is a combination of mounting and unmounting, and requires separate operations on devices.deny and devices.allow.

[0205] Step 6, find devices.allow and devices.deny from the directory / sys / fs / cgroup / devices / docker / cidxxx.

[0206] Step 7, create device nodes gpu0, gi0, and ci0 for GPU devices in the container.

[0207] Step 8, write the device type, major_id, and minor_id of gpu0 / gi0 / ci0 into the devices.allow file, and the written content is as shown in Table 13.

[0208] Table 13 shows the content of the devices.allow file before mounting the device gpu1.

[0209]

[0210] Step 9, write the device type, major_id, and minor_id of gpu1 into the devices.deny file, and the written content is as follows: c 195:1 rwm.

[0211] After completing the above 9 steps, the GPU device gpu0 / gi0 / ci0 can be mounted in the container and the GPU device gpu1 can be unmounted from the container in the cgroup v1 system.

[0212] The cgroup v2 system and the cgroup v1 system both have the above steps 1-5, and the difference starts from step 6, so the following step 6 is the next step in the cgroup v2 system.

[0213] Step 6, create device nodes gpu0, gi0, and ci0 for GPU devices in the container.

[0214] Step 7, prepare a bpf program for the following GPU device nodes gpu0, gi0, and ci0, and when there is a request for access, return a value of 1, i.e., allow access, as shown in Table 14.

[0215] Table 14 Partial schematic table of the process of mounting GPU device in cgroup v2 system under the premise of mounting device gpu1

[0216]

[0217] For the following GPU device node gpu1: c 195:1 rwm, when there is a request to access, the return value is 0, that is, access is prohibited.

[0218] Step 8, compile the bpf program in step 7 into a bpf kernel module.

[0219] Step 9, load the bpf module in step 8 into the system.

[0220] Step 10, unbind the bpf kernel module that the container cidxxx has bound from the original container.

[0221] Step 11, bind the bpf module loaded into the system in step 7 to the container cidxxx, and specify the bpf module type as device type.

[0222] After the above 11 steps are completed, the GPU device gpu0 / gi0 / ci0 can be mounted in the container in the cgroup v2 system.

[0223] The application solves the problem of container restart during hot mounting / unmounting of GPU devices in the container, thereby maintaining the continuity of the container and the application running therein, and effectively improving the work efficiency.

[0224] Based on the same inventive concept, the embodiments of the application also provide a GPU device mounting and unmounting system in a container. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme described in the above method, so the specific limitations in one or more GPU device mounting and unmounting system embodiments provided below can refer to the limitations of the GPU device mounting and unmounting method in the above text, and will not be repeated here.

[0225] In one exemplary embodiment, a GPU device mounting and unmounting system in a container is provided, comprising:

[0226] A cgroup version determination module is configured to determine the cgroup version according to the mounting list of the operating system.

[0227] a container cgroup directory determination module configured to determine a cgroup directory of the container according to a user demand based on the cgroup version, and place device files corresponding to GPUs that the user needs to mount under a / dev directory of the host machine and place device files corresponding to GPUs that the user needs to unmount under a / dev directory of the container according to the user demand;

[0228] a GPU device name determination module configured to traverse the / dev directory of the host machine to determine GPU device names that need to be mounted for the container and traverse the / dev directory of the container to determine GPU device names that need to be unmounted for the container based on the user demand;

[0229] a GPU device file creation and access permission configuration module configured to create GPU device files that need to be mounted for the container and configure access permissions according to the GPU device names that need to be mounted for the container, and configure access permissions for existing devices in the container according to the GPU device names that need to be unmounted for the container.

[0230] a mounting and unmounting module configured to mount or unmount GPU devices based on the cgroup version, the GPU device files and the access permissions.

[0231] In an exemplary embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program. The processor implements the steps in the above-mentioned method embodiments when executing the computer program. The computer device can be a server or a terminal, and its internal structure diagram can be as shown in Figure 4 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus. The communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store to-be-processed data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a GPU device mounting and unmounting method in a container.

[0232] Those skilled in the art can understand that, Figure 4The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above method embodiments.

[0233] In an exemplary embodiment, a computer readable storage medium is provided, storing a computer program, the computer program being executed by a processor to implement the steps in the above method embodiments.

[0234] In an exemplary embodiment, a computer program product is provided, including a computer program, the computer program being executed by a processor to implement the steps in the above method embodiments.

[0235] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0236] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to a memory, a database or other medium used in the embodiments provided in the present application can include at least one of a non-volatile and a volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive random access memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc.

[0237] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0238] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0239] The principles and implementation modes of the present application are described by applying specific examples in the present application. The above-mentioned embodiments are only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed. In conclusion, the content of the present application should not be understood as a limitation.

Claims

1. A method for mounting and unmounting a GPU device in a container, characterized in that, include: Determine the cgroup version based on the operating system's mount list; Based on the cgroup version, determine the container's cgroup directory, and according to user requirements, place the device files corresponding to the GPUs that the user needs to mount in the / dev directory of the host machine, and place the device files corresponding to the GPUs that the user needs to unmount in the / dev directory of the container. Based on the user requirements, the host machine's / dev directory is traversed to determine the GPU device name that needs to be mounted to the container, and the container's / dev directory is traversed to determine the GPU device name that needs to be unmounted to the container. As needed, create the GPU device file that needs to be mounted to the container and configure access permissions for the GPU device to be unmounted to the container. As needed, configure access permissions for the GPU device that needs to be unmounted to the container and configure access permissions for the existing devices in the container. Based on the cgroup version, and according to the GPU device file and access permissions, mount or unmount the GPU device.

2. The method for mounting and unmounting a GPU device in a container according to claim 1, characterized in that, The cgroup version is determined based on the operating system's mount list, specifically including: The "mount" command is used to obtain the mount list of the operating system. The cgroup version is determined based on the mount list; the cgroup version includes cgroup v1 and cgroup v2 systems.

3. The method for mounting and unmounting a GPU device in a container according to claim 1, characterized in that, Based on the cgroup version, the container's cgroup directory is determined, specifically including: Based on the cgroup version, and according to the user requirements, all subdirectories of the cgroup mount directory are traversed, and the container's cgroup directory is selected from all subdirectories; the user requirements are to mount or unmount the GPU device.

4. The method for mounting and unmounting a GPU device in a container according to claim 1, characterized in that, Based on the user requirements, the host machine's / dev directory is traversed to determine the GPU device name that needs to be mounted to the container, and the container's / dev directory is traversed to determine the GPU device name that needs to be unmounted from the container. Specifically, this includes: Determine the device file name according to the manufacturer's equipment manual; Use the "ls -l" command to determine the detailed device information corresponding to the device file name in the / dev directory of the host machine or the / dev directory of the container; The GPU device name is determined based on the device details; the GPU device name is either the name of the GPU device to be mounted to the container or the name of the GPU device to be unmounted from the container.

5. The method for mounting and unmounting a GPU device in a container according to claim 1, characterized in that, Based on the cgroup version, and according to the GPU device file and access permissions, the GPU device is mounted or unmounted, specifically including: When the cgroup version is cgroup v1, the device access control file of the container is determined in the cgroup directory, and a device node is created in the / dev directory of the container according to the GPU device file name; Based on the device node and according to the access permissions, the access permissions for the GPU are configured in the cgroup directory of the container, thereby completing the mounting or unmounting of the GPU device.

6. The method for mounting and unmounting a GPU device in a container according to claim 1, characterized in that, Based on the cgroup version, and according to the GPU device file and access permissions, the GPU device is mounted or unmounted, specifically including: When the cgroup version is cgroup v2, a device node is created in the / dev directory of the container according to the GPU device file; Based on the device node, and according to the access permissions, a BPF program is created; the BPF program returns "1" for read, write, and modify requests for GPU devices that need to be mounted, and returns "0" for read, write, and modify requests for GPU devices that need to be unmounted; Compile the bpf program into a bpf kernel module; Load the BPF kernel module into the operating system and unbind the container from the device type BPF program; The newly compiled BPF kernel module in the operating system is bound to the cgroup where the container is located, and the GPU device is mounted or unmounted; wherein, during the binding process, the BPF type is specified as the device type.

7. A GPU device mounting and unmounting system in a container, characterized in that, include: The cgroup version determination module is used to determine the cgroup version based on the operating system's mount list. The container cgroup directory determination module is used to determine the container's cgroup directory based on the cgroup version and according to user requirements, and to place the device files corresponding to the GPUs that the user needs to mount in the / dev directory of the host machine, and place the device files corresponding to the GPUs that the user needs to unmount in the / dev directory of the container. The GPU device name determination module is used to traverse the / dev directory of the host machine to determine the GPU device name that needs to be mounted for the container based on the user requirements, and to traverse the / dev directory of the container to determine the GPU device name that needs to be unmounted for the container. The GPU device file creation and access permission configuration module is used to create GPU device files that need to be mounted to containers and configure access permissions for containers as needed, and to configure access permissions for GPU device names that need to be unmounted from containers and existing devices in containers as needed. The mount / unmount module is used to mount or unmount the GPU device based on the cgroup version, the GPU device file, and the access permissions.

8. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method for mounting and unmounting a GPU device in a container as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method for mounting and unmounting a GPU device in a container as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the method for mounting and unmounting a GPU device in a container as described in any one of claims 1-6.

Images