Device management method and device

Abstract

An embodiment of the present application provides a device management method and apparatus. The method includes: mounting N GPUs on each of a plurality of containers, with a preset link existing between the N GPUs, where N is an integer greater than 1; virtualizing the GPUs that can be called by each container to obtain one or more vGPU instances corresponding to each container; and providing the virtualized vGPU instances to the corresponding container for use. By mounting N GPUs on each container and virtualizing the GPUs that can be called by each container, the isolation of GPUs between containers is ensured while avoiding blocking the preset links between GPUs, thereby allowing communication between GPUs using the preset links.

Description

Technical Field

[0001] The present application relates to the field of computers, and more particularly, to a device management method and apparatus. Background Art

[0002] With the continuous development of computer technology, more and more artificial intelligence (AI) deep training tasks are deployed and run in the form of containers. AI deep training tasks rely heavily on graphics processing units (GPUs).

[0003] Currently, when attaching GPUs to containers, the system or user configures the GPUs available to each container. The GPU runtime, the software responsible for managing GPUs, attaches the appropriate GPU to each container based on the system or user configuration. Each container can only use the GPU attached to that container, ensuring isolation between containers.

[0004] In some scenarios, such as distributed training, the same tenant may use multiple containers to perform the same task to improve execution efficiency. Since multiple containers may need to share data, high-speed data transmission is required between multiple containers. High-speed data transmission can be achieved through GPU high-speed interconnection technology with communication bandwidth far higher than that of ordinary networks. However, different containers are isolated from each other, which means that high-speed interconnection cannot be used between GPUs in different containers. Therefore, data transmission between containers is currently mainly achieved through shared memory or network transmission. However, this may affect overall performance. For example, the use of shared memory and other methods requires data to be transferred through the system main memory, and multiple data copies lead to low communication efficiency and poor communication performance, which limits execution efficiency and restricts the scalability of training. Summary of the Invention

[0005] This application provides a device management method and apparatus to achieve container isolation while improving communication speed and execution efficiency.

[0006] In a first aspect, the present application provides a device management method, which includes: mounting a GPU on each of N containers, where a preset link exists between the N GPUs, and N is an integer greater than 1; virtualizing the GPU that can be called by each container to obtain one or more vGPU instances corresponding to each container; and providing the virtualized vGPU instances to the corresponding containers for use.

[0007] In a second aspect, the present application provides a device management device, which includes: a control module and a virtualization module; wherein the control module is used to mount N graphics processing units (GPUs) on each of a plurality of containers, and there is a preset link between the N GPUs, where N is an integer greater than 1; the virtualization module is used to virtualize the GPU that can be called by each container to obtain one or more vGPU instances corresponding to each container; the control module is also used to provide the virtualized vGPU instances to the corresponding containers for use.

[0008] It should be understood that each module can implement corresponding functions by executing computer programs.

[0009] In a third aspect, the present application provides a device management apparatus, which includes a processor for executing program code so that the apparatus implements the method in the first aspect.

[0010] In a fourth aspect, the present application provides a chip comprising at least one processor for implementing the functions involved in the first aspect above, such as virtualizing a GPU.

[0011] In a fifth aspect, the present application provides a computing device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method in the first aspect when executing the computer program.

[0012] In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor implements the method in the first aspect above.

[0013] Seventh method, the present application provides a computer program product, including a computer program, which implements the method in the above-mentioned first aspect when the computer program is executed.

[0014] Based on the above solution, on the one hand, virtualization technology is used to provide virtualized vGPU instances to containers based on the GPUs that each container can call, so that each container can only access the GPUs it can call, ensuring GPU isolation between containers. On the other hand, N GPUs are mounted separately in each container to avoid invalidating the preset links between GPUs mounted in different containers when the container starts. In other words, it can avoid blocking the preset links between GPUs, thereby allowing the GPUs to communicate using high-speed interconnects. Because the communication efficiency of shared memory or network transmission is far lower than that of preset links, this can greatly improve communication speed, improve execution efficiency, and ensure good communication performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1This is a schematic diagram of communication between GPUs provided in an embodiment of the present application;

[0016] Figure 2 is a schematic flow chart of the device management method provided in an embodiment of the present application;

[0017] Figure 3 is another schematic diagram of communication between GPUs provided in an embodiment of the present application;

[0018] Figure 4 It is a schematic block diagram of the device management device provided in an embodiment of the present application;

[0019] Figure 5 This is another schematic block diagram of the device management apparatus provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The technical solution in this application will be described below with reference to the accompanying drawings.

[0021] The technical solutions provided in this application can be applied to the fields of artificial intelligence (AI) and deep learning (DL). AI is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. DL is a new research direction in machine learning, introduced into machine learning to bring it closer to the original goals of AI.

[0022] Figure 1 This is a schematic diagram of communication between GPUs applicable to the embodiments of the present application.

[0023] exist Figure 1 In the communication scenario shown, two containers are started, namely container 0 and container 1. Each container contains multiple worker processes and corresponding communication libraries. Among them, container 0 contains worker processes 0, 1, and 2, and container 1 contains worker processes 3 and 4. It should be understood that multiple worker processes can form a collective process group to complete the same task in parallel. A GPU is mounted on each container. Figure 1As shown, GPU0, GPU1, and GPU2 are mounted on container 0, while GPU3 and GPU4 are mounted on container 1. Each GPU can be used by a corresponding worker process. For example, GPU0 can be used by worker process 0, GPU1 can be used by worker process 1, GPU2 can be used by worker process 2, GPU3 can be used by worker process 3, and GPU4 can be used by worker process 4. When a worker process in a container uses a GPU, the worker process can call the GPU to use it, so the worker process using the GPU may need to frequently communicate with each other.

[0024] It should be understood that a container can be a collection of processes that isolates other system resources and has its own independent resource view.

[0025] It should also be understood that mounting is the process of exposing certain GPUs of the host machine to the container so that the container can access and use these GPUs.

[0026] It should also be understood that collective communication is communication between a group of processes. It differs from point-to-point communication in that it requires all processes within a specific group to participate simultaneously. This communication can occur in one-to-many, many-to-one, or many-to-many modes. The communication library referred to in this application is a library for collective communication. For more information on collective communication and communication libraries, please refer to existing technologies and will not be elaborated on here.

[0027] Currently, when mounting GPUs for Container 0 and Container 1, the system or user configures the GPUs available to Container 0 and Container 1, respectively. For example, Container 0 is configured to use GPU0, GPU1, and GPU2, and Container 1 is configured to use GPU3 and GPU4. During GPU runtime, GPU0 is mounted to GPU2 for Container 0, and GPU3 and GPU4 are mounted to Container 1. Container 0 can then access GPUs 0 to GPU2, and Container 1 can access GPUs 3 and GPU4. Because Container 0 and Container 1 are isolated from each other, the GPU runtime ensures that Container 0 cannot access GPUs 3 and GPU4 in Container 1, and Container 1 cannot access GPUs 0 to GPU2 in Container 0.

[0028] However, researchers found that because container 0 and container 1 are isolated from each other, GPU0, GPU1, and GPU2 in container 0 cannot use GPU high-speed interconnect technology to communicate with GPU3 and GPU4 in container 1. In other words, high-speed communication between GPUs in containers cannot be achieved. Although data transmission between containers can currently be achieved through shared memory, network transmission, and other means, such as interconnecting GPU1 and GPU4 through shared memory, this may affect overall performance. For example, the use of shared memory and other methods requires data to be transferred through the system main memory, and multiple copies of data lead to low communication efficiency and poor communication performance, limiting the scalability of training.

[0029] In view of this, the present application provides a device management method. On the one hand, according to the GPUs that each container can call, the virtualized vGPU instance is provided to the container through virtualization technology, so that each container can only access the GPUs it can call, which can ensure GPU isolation between containers; on the other hand, N GPUs are mounted in each container respectively, so as to avoid invalidating the preset links between GPUs mounted in different containers when the container is started, that is, it can avoid blocking the preset links between GPUs, allowing GPUs to communicate using high-speed interconnection. If the preset link is designed as a high-speed communication link for communication between GPUs, then because the communication efficiency of shared memory or network transmission is far less than that of the high-speed communication link, it can greatly improve communication efficiency, execution efficiency, and overall performance.

[0030] The device management method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0031] See Figure 2 , Figure 2 This is a schematic flowchart of the device management method provided in an embodiment of the present application. Figure 2 The method 200 shown can be applied to a CPU, or to system software on the CPU. The system software may include an operating system and a resource scheduling system, wherein the resource scheduling system can be used for resource scheduling of a GPU.

[0032] Figure 2 The method 200 shown may include steps 201 to 203. The following describes the method 200 in detail by taking the application of the method 200 to the CPU as an example. Figure 2 The various steps in the method 200 are shown.

[0033] Step 201: Mount N GPUs on each of a plurality of containers, where a preset link exists between the N GPUs, where N is an integer greater than 1.

[0034] Among them, the preset link can be used for communication between GPUs. In an embodiment of the present application, the preset link can be designed as a high-speed communication link for communication between GPUs. The transmission bandwidth of the high-speed communication link is much higher than the transmission bandwidth of the ordinary network, so it can provide a higher communication speed. As an example and not a limitation, the high-speed communication link can be NVIDIA's NVLink, or it can also be a high-speed communication link for communication between GPUs provided by other GPU manufacturers, and the present application includes but is not limited to this.

[0035] When the CPU starts a container, it can mount N GPUs on each container. Here, N GPUs refers to all GPUs configured for multiple containers. These multiple containers can be, for example, multiple containers used by the same tenant to perform the same task. Because all GPUs are mounted on multiple containers, the high-speed communication links between the mounted GPUs will not be disabled when each container is started. Since the high-speed communication links between the GPUs are not blocked, they can be used for data transmission between the GPUs.

[0036] like Figure 3 As shown in the figure, the CPU has started two containers, container 0 and container 1, and both containers 0 and 1 have GPUs 0 through 3 mounted. For both containers 0 and 1, there is no isolation between GPUs 0, 1, and 1 and 3, and high-speed communication links exist between the GPUs. Therefore, GPUs 0, 1, 2, and 3 can be interconnected at high speed.

[0037] Optionally, the GPU mounted on each container in the multiple containers can be configured by the system or by the user.

[0038] Specifically, the system or user can configure a mountable GPU for a container. The system can be a resource scheduling system on a CPU, which can be used to configure a mountable GPU for a container. It should be understood that the resource scheduling system can also be set up in the CPU's system software.

[0039] The following example shows how to configure a mountable GPU for a container.

[0040] For example, the resource scheduling system or the user can uniformly configure the mountable GPUs for multiple containers. For example, the resource scheduling system on the CPU can configure Container 0 and Container 1 to mount GPU0, GPU1, GPU2, and GPU3. Therefore, the CPU mounts GPU0 through GPU3 for both Container 0 and Container 1 based on the configured GPUs.

[0041] Step 202: Virtualize the GPU that can be called by each container to obtain one or more vGPU instances corresponding to each container.

[0042] The GPU that each container can call, that is, the GPU that each container can actually use. For example, Figure 3 Container 0 in the example can call GPU0, GPU1, and GPU2, and container 1 can call GPU3. The GPUs that each container can call can be configured through a mapping relationship. This mapping relationship can be a mapping relationship between containers and GPUs configured by the resource scheduling system or manually by the user. This mapping relationship can be configured separately for each container, and the mapping relationship configured for each container is used to indicate the GPUs that this container can call; this mapping relationship can also be configured uniformly for all containers, and this mapping relationship can be used to indicate the GPUs that each container in multiple containers can call.

[0043] When configuring a mapping relationship for each container separately, the resource scheduling system or the user can generate a mapping relationship for each container. For example, the resource scheduling system generates mapping relationship #1 for the GPUs that container 0 can call, and mapping relationship #1 indicates that the GPUs that container 0 can call are GPU0, GPU1, and GPU2; the resource scheduling system also generates mapping relationship #2 for the GPUs that container 1 can call, and mapping relationship #2 indicates that the GPU that container 1 can call is GPU3. It should be understood that mapping relationship #1 and mapping relationship #2 are specific examples of mapping relationships. When configuring a mapping relationship for each container separately, the mapping relationships configured for each container are different.

[0044] When configuring all containers uniformly, the resource scheduling system or user generates a unified mapping for all containers. For example, this mapping indicates that container 0 can access GPU0, GPU1, and GPU3, while container 1 can access GPU3. In other words, this mapping is the complete set of mappings #1 and #2.

[0045] After the CPU learns the GPU that each container can call based on the mapping relationship, it can determine the GPU that can be called from the N GPUs mounted in each container and virtualize the GPU that can be called.

[0046] It should be understood that GPU virtualization refers to encapsulating a single GPU device into several logical vGPU instances for concurrent use by different work processes.

[0047] Optionally, each of the multiple containers includes one or more worker processes, and each worker process is provided with one or more vGPU instances.

[0048] When virtualizing a GPU, you can decide whether to virtualize it into one vGPU instance or multiple vGPU instances based on the GPU configuration. If the GPU configuration is high enough to meet the needs of multiple work processes simultaneously, you can virtualize the GPU into multiple vGPU instances. If the GPU configuration is low enough to meet the needs of multiple work processes simultaneously, you can virtualize the GPU into one vGPU instance.

[0049] For example, in Figure 3 In the example, if the CPU learns from the mapping relationship that the GPUs that container 0 can call are GPU0, GPU1, and GPU2, and the GPU that container 1 can call is GPU3, although GPU0 to GPU3 are mounted in both container 0 and container 1, for container 0, the CPU can virtualize GPU0 to GPU2 but not GPU3; for container 1, the CPU can virtualize GPU3 but not GPU0 to GPU2. Assuming that the configurations of GPU0, GPU1, and GPU2 are relatively low and cannot meet the needs of multiple working processes at the same time, GPU0, GPU1, and GPU2 can be virtualized into vGPU instance-0, vGPU instance-1, and vGPU instance-2, respectively. Assuming that the configuration of GPU3 is relatively high and can meet the needs of multiple working processes at the same time, GPU3 can be virtualized into vGPU instance-3 and vGPU instance-4. Thus, worker process 0 in container 0 may be provided with vGPU instance-0, worker process 1 may be provided with vGPU instance-1, worker process 2 may be provided with vGPU instance-2, and worker process 3 in container 1 may be provided with vGPU instance-3, and worker process 4 may be provided with vGPU instance-4.

[0050] As you can see, although the GPUs are not isolated from each other in the container, the use of virtualization technology ensures that the container's calls to resources are still isolated from each other.

[0051] The vGPU instance can be generated by virtualizing each GPU based on the above mapping relationship by the vGPU runtime. The vGPU runtime can be understood as the software used to virtualize the GPU and manage the virtualized vGPU instances.

[0052] One possible implementation is to inject a vGPU runtime into each container. The vGPU runtime injected into each container is used to virtualize the GPU that can be called.

[0053] For example, when launching a container, the CPU can inject the vGPU runtime into each container. The CPU can then call the vGPU runtime to virtualize the GPUs available to each container into one or more vGPU instances based on the aforementioned mapping. In a specific implementation, the CPU can inject the vGPU runtime into each container by mounting a host volume.

[0054] Optionally, the method further includes: providing the above-mentioned mapping relationship to the vGPU runtime of each container, where the mapping relationship is used to indicate the GPU that each container can call.

[0055] For example, after generating a mapping relationship, the resource scheduling system in the CPU can provide the mapping relationship to the vGPU runtime, which then virtualizes the GPU callable by the container based on the mapping relationship. The resource scheduling system can provide the mapping relationship to the vGPU runtime in the form of configuration files, environment variables, command line parameters, etc.

[0056] As mentioned above, the mapping relationship can be configured individually for each container or uniformly for all containers. When configuring each container individually, the resource scheduling system provides each vGPU runtime with the GPUs that can be called by the container to which it belongs. In other words, the mapping relationship content obtained by each vGPU runtime is different. When configuring all containers uniformly, the resource scheduling system provides each vGPU runtime with the GPUs that can be called by each container. In other words, the mapping relationship content obtained by each vGPU runtime is the same. The vGPU runtime simply searches the mapping relationship for the GPU that can be called by its container.

[0057] It should be understood that the mapping relationship may also be provided by the user.

[0058] Step 203: Provide the virtualized vGPU instance to the corresponding container for use.

[0059] After the CPU virtualizes the GPU available to the container by invoking the vGPU runtime, the container can use the vGPU instance in the following manner. It should be understood that the container's use of the vGPU instance can specifically be the use of the vGPU instance by a worker process within the container.

[0060] It should be understood that since the vGPU instance is obtained by GPU virtualization, the use of the vGPU instance corresponding to the GPU by the worker process in the container is equivalent to the use of the GPU by the worker process in the container.

[0061] Optionally, step 203 may specifically include: hijacking a call to the first application programming interface (API) by the vGPU runtime injected into each container, and providing a second API.

[0062] The first API is provided by the GPU manufacturer, specifically a GPU user-mode API or a GPU kernel driver API. The second API is provided by the vGPU runtime and is identical in name and appearance to the API provided by the GPU manufacturer. This second API is used to call the vGPU instance in each container.

[0063] It should be understood that hijacking an API call can be understood as modifying the original API entry point to redirect it to another API. In this embodiment of the present application, by hijacking the call to the first API and providing a second PAI, the call to the first API is redirected to the second API. Specifically, when a worker process in the container calls the first API, the vGPU runtime can block the worker process's call to the first API and provide the second API to the worker process.

[0064] Specifically, when a worker process in a container wants to use the GPU, it typically calls the manufacturer-provided API, known as the first API. At this point, the CPU controls the vGPU runtime in the container to block the worker process's call to the first API. Furthermore, the vGPU runtime provides the worker process with the second API, which is used to call the vGPU instance in the container. Because the first and second APIs look identical, the worker process in the container can be induced to call the second API, which in turn allows the corresponding vGPU to be used. When hijacking the manufacturer-provided API, the vGPU runtime can hijack it at either the GPU user-mode API or the GPU kernel driver API.

[0065] Optionally, the vGPU runtime provides a function of injecting a mapping relationship or a function of modifying GPU runtime environment variables.

[0066] Among them, GPU runtime can be understood as the software that manages the GPU. Environment variables are generally parameters in the operating system used to specify the operating environment of the operating system. The environment variables involved in the embodiments of the present application can originally be used to describe the GPU mounted on each container, for example, they can be "CUDE_VISIBLE_DEVICES", "HIP_VISIBLE_DEVICES", etc. It can be understood that in this embodiment, the GPUs mounted on each container are the N GPUs mentioned above. After modifying the environment variables, the environment variables present the GPUs that can be called by each container.

[0067] In one example, when the vGPU is running, the mapping relationship can be injected into the second API, and the working process in the container can use the corresponding vGPU instance by calling the second API.

[0068] For example, when container 0's vGPU runtime injects the mapping relationship indicating that container 0 can call GPU0 to GPU2 into the second API, worker process 0 can use vGPU instance 0 by calling the second API, worker process 1 can use vGPU instance 1 by calling the second API, and worker process 2 can use vGPU instance 2 by calling the second API. That is, by calling the second API, worker process 0 can use GPU0, worker process 1 can use GPU1, and worker process 2 can use GPU2. When container 1's vGPU runtime injects the mapping relationship indicating that container 1 can call GPU3 into the second API, worker process 3 can use vGPU instance 3 by calling the second API, and worker process 4 can use vGPU instance 4 by calling the second API. In other words, worker processes 3 and 4 reuse GPU 3.

[0069] Another example is the assumption that all configured GPUs are mounted in each container, and only the GPUs available to the container are virtualized during vGPU runtime. Since all configured GPUs are mounted in the container, the container can access all mounted GPUs. For non-virtualized GPUs, it is possible for the container to bypass the second API to access the non-virtualized GPU. For example, GPUs 0 to 3 are mounted in Container 0. Although only GPUs 0 to 2 are virtualized, Container 0 can still access GPU 3 and could potentially use illegal methods to bypass the second API to use GPU 3, which should be used by Container 1. For another example, Container 1 also mounts GPUs 0 to 3. Although only GPUs 3 are virtualized, Container 1 can still access GPUs 0 to 2 and could potentially use illegal methods to bypass the second API to use GPU 3, which should be used by Container 0. If this occurs, isolation between Container 0 and Container 1 cannot be guaranteed.

[0070] To avoid this situation, the vGPU runtime can modify GPU runtime environment variables, changing the container's access to non-virtualized GPUs from "accessible" to "inaccessible." For example, by modifying the GPU runtime environment variables describing the GPUs mounted on Container 0, making GPU3 "inaccessible," Container 0 would be unable to use GPU 3. Similarly, by modifying the GPU runtime environment variables describing the GPUs mounted on Container 1, making GPUs 0 through 2 "inaccessible," Container 1 would be unable to use GPUs 0 through 2.

[0071] It can be seen that although both container 0 and container 1 have GPU0 to GPU3 mounted, due to the isolation between containers during the vGPU runtime, container 0 and container 1 can only use the vGPU instances corresponding to the GPUs they can call. Container 0 cannot use GPU3 in container 1, and container 1 cannot use GPU0 to GPU2 in container 0. Therefore, the isolation between container 0 and container 1 is guaranteed at the vGPU runtime level.

[0072] As mentioned earlier, when GPU configurations are high, the GPU can be virtualized into multiple vGPU instances for container use. However, deadlock can occur when the communication library logic is dominated by the GPU. Specifically, when the communication library logic is dominated by the GPU, and GPU resources are scarce or GPU utilization is high (i.e., when GPU resources are almost non-existent), deadlock is very likely to occur if multiple nodes in the container reuse a single GPU and there are dependencies between the nodes.

[0073] It should be understood that mutual dependence can mean that a working process (for example, denoted as working process a) needs to wait for the signal of another working process (for example, denoted as working process b) before it can continue to run on the GPU, and working process b needs working process a to release the resources of the GPU before it can be scheduled. However, working process b can only run on the GPU after being scheduled, and send a signal to working process a, and the long-term persistent kernel mode of the GPU determines that working process a must release resources only after completing the task. In this way, working process a is waiting for the signal of working process b, while working process b is waiting for working process a to release resources. The two are waiting for each other, and a deadlock phenomenon occurs. Among them, the long-term persistent kernel mode can be specifically manifested as the working process running on the GPU needs to release resources after execution is completed.

[0074] It should also be understood that deadlock will not occur when the GPU has available resources.

[0075] For example, Figure 3During the vGPU runtime in container 1, GPU 3 is virtualized into vGPU instance 3 and vGPU instance 4, respectively, for use by worker processes 3 and 4. Therefore, both worker processes 3 and 4 essentially use GPU 3. Assuming that worker process 3 runs first on GPU 3, when it reaches a node on GPU 3, worker process 3 requires a signal from worker process 4 to continue. However, if GPU 3's resources are already fully occupied, worker process 4 cannot run on GPU 3. Since worker process 4 cannot run, it cannot signal worker process 3 to continue, leaving worker process 3 in a waiting state. Worker process 4, on the other hand, can only run on GPU 3 when GPU 3 releases the resources it occupied. However, worker process 3 is waiting for a signal from worker process 4, preventing GPU 3 from releasing resources, leaving worker process 4 in a waiting state. Consequently, worker processes 3 and 4 are waiting for each other, resulting in a deadlock.

[0076] Therefore, in order to avoid a deadlock phenomenon that may occur when multiple work processes reuse a GPU, method 200 may further include:

[0077] The work process is scheduled based on the control logic in the communication library so that the work process responds to the CPU's scheduling and calls the resources in the CPU for calculation.

[0078] Specifically, the control logic of the communication library is offloaded from the GPU to the CPU, and the CPU's operating system can be responsible for scheduling work processes based on the control logic of the communication library. Because the CPU runs an operating system, the operating system can ensure that the possibility of resource exhaustion is very small. Therefore, interdependent work processes can be scheduled to the CPU for calculation. On the other hand, unlike the GPU's persistent kernel model, the CPU controlling the communication logic does not load complex logic. For example, a work process that depends on external conditions can be completed when the GPU has available resources. There will be no situation where GPU resources are occupied but waiting for another unscheduled work process, and there will be no deadlock problem. Instead, the CPU can schedule interdependent work processes in turn when resources are exhausted. This can avoid deadlock caused by interdependent work processes waiting for each other.

[0079] In one possible implementation, all work processes are handed over to the CPU for processing. That is, both interdependent work processes and independent work processes can be scheduled by the CPU.

[0080] For interdependent worker processes, the CPU can use a round-robin scheduling mechanism. For example, assuming worker processes 3 and 4 in container 1 are interdependent, the CPU's operating system can schedule worker process 3 first. When worker process 3 reaches a certain node and requires a signal from worker process 4 to continue, and the CPU's resources are currently occupied, the CPU can de-schedule worker process 3 and bring up worker process 4, allowing worker process 4 to run. At this point, the scheduled worker process 4 can signal worker process 3 to continue, and the CPU can de-schedule worker process 4 and bring up worker process 3, allowing worker process 3 to continue running. This cycle allows worker processes 3 and 4 to take turns being scheduled and run, eliminating the problem of a worker process occupying resources and not releasing them, as can occur in GPUs, and thus avoiding deadlock.

[0081] For independent worker processes, the CPU can schedule them based on resource usage. For example, if worker processes 3 and 4 in container 1 are independent, the CPU's operating system can schedule worker process 3 first and allow it to run. If the CPU has remaining resources, it can schedule worker process 4. If the CPU has no remaining resources, the CPU can de-schedule worker process 3 and bring up worker process 4. Alternatively, the CPU can schedule worker process 4 after worker process 3 finishes running and releases resources.

[0082] Another possible implementation involves assigning interdependent processes to the CPU and independent processes to the GPU. The CPU's handling of interdependent processes is similar to the previous implementation and will not be further elaborated here. Regarding the GPU's handling of independent processes, since the GPU lacks the ability to schedule processes in a round-robin fashion, it can wait until the previous process completes and releases resources before allowing the next process to run.

[0083] It should also be understood that the control logic is determined by the CPU, and the specific executors of data transmission, etc. are not limited in this application and may be a direct memory access (DMA) engine, a network card, a GPU program, or the CPU itself.

[0084] Based on the above solution, on the one hand, based on the GPUs that each container can call, the virtualized vGPU instance is provided to the container through virtualization technology, so that each container can only access the GPUs it can call, which can ensure GPU isolation between containers; on the other hand, N GPUs are mounted separately in each container, so as to avoid invalidating the preset links between GPUs mounted in different containers when the container is started. In other words, it can avoid blocking the preset links between GPUs, thereby allowing GPUs to communicate using the preset links. Because the preset links between GPUs can be designed as high-speed communication links, GPUs can be allowed to communicate using high-speed communication links. Because the communication efficiency of shared memory or network transmission is far lower than that of high-speed links, this can greatly improve the communication speed, improve execution efficiency, and ensure good communication performance. In addition, by offloading the control logic of the collective communication library from the GPU to the CPU, it also avoids the deadlock phenomenon that may occur when multiple working processes reuse the same GPU.

[0085] Above, combined Figures 2 to 3 The method provided in the embodiment of the present application is described in detail. Figures 4 and 5 The device provided in the embodiments of the present application is described in detail.

[0086] Figure 4 This is a schematic block diagram of the device provided in the embodiment of the present application. Figure 4 As shown, the device 400 may include: a control module 410 and a virtualization module 420. Each module in the device 400 may be used to implement Figure 2 The corresponding process of the CPU in the method 200 is shown. For example, the control module 410 can be used to execute step 201 and step 203 in the method 200, and the virtualization module 420 can be used to execute step 202 in the method 200.

[0087] Specifically, the control module 410 can be used to mount N GPUs on each of multiple containers, where a preset link exists between the N GPUs, and N is an integer greater than 1; the virtualization module 420 can be used to virtualize the GPUs that can be called by each container to obtain one or more vGPU instances corresponding to each container; the control module 410 is also used to provide the virtualized vGPU instances to the corresponding containers for use.

[0088] Optionally, the control module 410 may also be used to inject a vGPU runtime into each container, and the vGPU runtime injected into each container is used to virtualize the callable GPU.

[0089] Optionally, the control module 410 can be specifically used to hijack calls to a first application programming interface API through the vGPU runtime injected into each container, and provide a second API, where the first API is a GPU user-mode API or a GPU kernel driver API provided by the GPU manufacturer, and the second API is used to call the vGPU instance in each container.

[0090] Optionally, the vGPU runtime provides a function of injecting a mapping relationship or a function of modifying GPU runtime environment variables.

[0091] Optionally, the control module 410 may also be configured to provide a mapping relationship to the vGPU runtime of each container, where the mapping relationship is used to indicate the GPU that each container can call.

[0092] Optionally, the GPU mounted on each of the multiple containers is configured by the system or by the user.

[0093] Optionally, each of the multiple containers includes one or more worker processes, and each worker process is provided with one or more vGPU instances.

[0094] Optionally, the control module 410 may also be used to schedule a work process based on the control logic in the communication library, so that the work process responds to the CPU's scheduling and calls resources in the CPU for calculation.

[0095] It should be understood that the division of modules in the embodiments of the present application is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the present application may be integrated into a single processor, or may exist physically as separate modules, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0096] Figure 5 is another schematic block diagram of an apparatus provided in an embodiment of the present application. The apparatus 500 can be used to implement the functions of the CPU in the above-described method 200. The apparatus 500 can be a chip system. In the embodiments of the present application, the chip system can be composed of a chip, or can include a chip and other discrete devices.

[0097] like Figure 5 As shown, the device 500 may include at least one processor 510, which is used to implement the functions of the CPU in the method 200 provided in the embodiment of the present application.

[0098] For example, when the apparatus 500 is used to implement the CPU functionality of the method 200 provided in an embodiment of the present application, the processor 510 may be configured to mount N GPUs on each of a plurality of containers, with a preset link existing between the N GPUs, where N is an integer greater than 1; virtualize the GPUs that can be invoked by each container to obtain one or more vGPU instances corresponding to each container; and provide the virtualized vGPU instances to the corresponding containers for use. For details, please refer to the detailed description in the method example and will not be repeated here.

[0099] The device 500 may also include at least one memory 520 for storing program instructions and / or data. The memory 520 is coupled to the processor 510. Coupling in the embodiments of the present application is an indirect coupling or communication connection between devices, units, or modules, which can be electrical, mechanical, or other forms, and is used for information exchange between devices, units, or modules. The processor 510 may operate in conjunction with the memory 520. The processor 510 may execute program instructions stored in the memory 520. At least one of the at least one memory may be included in the processor.

[0100] The apparatus 500 may also include a communication interface 530 for communicating with other devices via a transmission medium, so that the apparatus 500 can communicate with other devices. The communication interface 530 may be, for example, a transceiver, an interface, a bus, a circuit, or a device capable of implementing transceiver functions. The processor 510 may utilize the communication interface 530 to transmit and receive data and / or information, and to implement Figure 2 The method executed by the CPU in the corresponding embodiment.

[0101] The specific connection medium between the processor 510, the memory 520 and the communication interface 530 is not limited in the embodiment of the present application. Figure 5 The processor 510, the memory 520 and the communication interface 530 are connected via a bus. Figure 5 The connections between the other components are shown in bold lines, which are only for illustration and not intended to be limiting. The bus can be divided into address bus, data bus, control bus, etc. Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0102] It should be understood that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above-mentioned method embodiment can be completed by hardware integrated logic circuits in the processor or by software instructions. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above-mentioned method.

[0103] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0104] The present application also provides a chip, which includes at least one processor for implementing the above Figure 2 The functions involved in the CPU in the embodiment shown.

[0105] In one possible design, the chip further includes a memory, which is used to store program instructions and data. The memory is located inside or outside the processor.

[0106] The present application also provides a computing device, which includes: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, Figure 2 The method of the embodiment shown.

[0107] The present application also provides a computer-readable storage medium, which stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the following Figure 2 The method of the embodiment shown.

[0108] The present application also provides a computer program product, including a computer program, which, when enabled, implements the following Figure 2 The method of the embodiment shown.

[0109] The terms "unit," "module," and the like used in this specification may be used to refer to a computer-related entity, hardware, firmware, a combination of hardware and software, software, or software in execution.

[0110] Those skilled in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the several embodiments provided in this application, it should be understood that the disclosed devices, equipment, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed. In addition, the coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms.

[0111] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0112] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0113] In the above-described embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (program) are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0114] If this function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0115] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A device management method, characterized in that: The method comprises: Mounting N graphics processing units (GPUs) corresponding to all containers on each of the multiple containers, wherein a preset link exists between the N GPUs, and the GPUs communicate with each other using a high-speed interconnect. Each container corresponds to the same GPU, and N is an integer greater than 1; Virtualizing the GPU that can be called by each container to obtain one or more vGPU instances corresponding to each container; The vGPU instances obtained by virtualizing each GPU are provided to corresponding different containers for use; the vGPU runtime provides the function of injecting mapping relationships or modifying GPU runtime environment variables.

2. The method according to claim 1, wherein The method further comprises: A vGPU runtime is injected into each container, and the vGPU runtime injected into each container is used to virtualize a callable GPU.

3. The method according to claim 2, wherein Providing the virtualized vGPU instance to the corresponding container for use includes: The vGPU runtime injected into each container hijacks the call to the first application program interface API and provides a second API, where the first API is a GPU user-mode API or a GPU kernel driver API provided by the GPU manufacturer, and the second API is used to call the vGPU instance in each container.

4. The method according to claim 2 or 3, wherein: The method further comprises: A mapping relationship is provided to the vGPU runtime of each container, where the mapping relationship is used to indicate the GPU that can be called by each container.

5. The method according to claim 1, wherein The GPU mounted on each container in the multiple containers is configured by the system or by the user.

6. The method according to claim 1, wherein Each of the multiple containers includes one or more worker processes, and each worker process is provided with one or more vGPU instances.

7. The method according to claim 6, wherein Applied to a central processing unit (CPU), the method further comprises: The work process is scheduled based on the control logic in the communication library, so that the work process responds to the scheduling of the CPU and calls the resources in the CPU for calculation.

8. A device management device, characterized in that: Comprising means for performing the method according to any one of claims 1 to 7.

9. A device management device, characterized in that: The device comprises a processor configured to execute program code so as to enable the device to implement the method according to any one of claims 1 to 7.

10. A chip, characterized in that: include: At least one processor, configured to implement the functions involved in the method according to any one of claims 1 to 7.

11. A computing device, characterized in that include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

12. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, implements the method according to any one of claims 1 to 7.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images