A Method and System for Training Artificial Intelligence Tasks Based on Dynamic Reconfiguration Virtualization of GPU Resources

Abstract

This application relates to an AI task training method and system based on dynamic reconfiguration virtualization of GPU resources. The method includes real-time acquisition and identification of the current training stage of each training task, and matching the current computational pressure and memory pressure values ​​for each training task. When the computational pressure value of a training task is greater than or equal to a first threshold, a first vGPU to be borrowed is selected, and computational resources are temporarily authorized to the vGPU of the current training task. When the memory pressure value of a training task is greater than or equal to a second threshold, a second vGPU to be borrowed is selected, a memory access channel is constructed to read and write the memory resources of the second vGPU to be borrowed, and the current memory resource pool is updated. When the computational pressure value is less than the first threshold or the first vGPU to be borrowed needs to restore computational power, the computational resources are reclaimed. When the memory pressure value is less than the second threshold, the data stored in the second vGPU to be borrowed is migrated back. This application improves the overall efficiency of the training task.

Description

Technical Field

[0001] This application relates to the technical field of GPU virtualization, and in particular to an artificial intelligence task training method and system based on dynamic reconfiguration virtualization of GPU resources. Background Technology

[0002] To improve GPU utilization and reduce the cost of AI training tasks, mainstream cloud computing platforms widely adopt GPU virtualization technologies, such as NVIDIA's vGPU technology. These technologies allow a single physical GPU to be divided into multiple independent logical instances for different training tasks to use in parallel, achieving basic resource isolation and pooling.

[0003] Artificial intelligence training tasks are not constant workloads. A typical training iteration includes multiple stages: data loading (I / O intensive), forward and backward propagation (computation intensive), and gradient synchronization (communication intensive). The computational power and memory requirements fluctuate dynamically during training. However, current allocation of computing power and memory resources for vGPU virtual training platforms is generally fixed, meaning resources are allocated based on the peak demand of the training task. This results in a large amount of idle resources in vGPU slices during off-peak periods, reducing the efficiency of AI task training and thus requiring improvement. Summary of the Invention

[0004] In order to improve the overall efficiency of artificial intelligence training tasks by maximizing the utilization of computing power and GPU memory resources during the training process, this application provides an artificial intelligence task training method and system based on dynamic reconfiguration virtualization of GPU resources.

[0005] The above-mentioned objective of this application is achieved through the following technical solution:

[0006] A method for training artificial intelligence tasks based on dynamic reconfiguration virtualization using GPU resources, comprising the following steps:

[0007] Real-time acquisition and identification of the current training stage of each training task, and matching the current computational pressure and memory pressure values ​​for the training task;

[0008] When the computational pressure of the training task is greater than or equal to the first threshold, the first idle vGPU to be borrowed is selected from the pre-built computing resource pool. The computing resources of the first vGPU to be borrowed are temporarily authorized to the vGPU of the current training task, and the current computing resource pool is updated.

[0009] When the memory pressure of the training task is greater than or equal to the second threshold, the second vGPU with free memory resources in the pre-built memory resource pool is selected to be borrowed during the current training phase. A memory access channel is built to enable the vGPU of the current training task to read and write the memory resources of the second vGPU to be borrowed, and the current memory resource pool is updated.

[0010] When the computational pressure of the training task is less than the first threshold or the first borrowed vGPU needs to restore computing power, the computing power resources will be recovered to the first borrowed vGPU.

[0011] When the memory pressure of the training task is less than the second threshold, the data stored in the second borrowed vGPU is moved back, and the corresponding memory access channel is removed.

[0012] By adopting the above technical solution, the training process of the artificial intelligence training task is predefined into several training stages, each requiring different computing power and GPU memory. This effectively segments the fluctuating state of the training task, allowing real-time acquisition of the current training stage for each task. The current computing and GPU memory pressure values ​​for each task are then determined based on the current training stage. This determines whether the current training task requires computing power borrowing, GPU memory expansion, and which vGPUs have available computing power slices and GPU memory that can be borrowed. Furthermore, when the computing pressure value is greater than or equal to a first threshold, it is determined that the current training task requires computing power borrowing. The first vGPU with idle computing power resources is selected from the pre-abstracted computing resource pool and a portion of its computing power is temporarily authorized. The system allocates computing power to the current training task and reclaims borrowed computing power slices in real time when the computational pressure value is less than a first threshold or when the first borrowed vGPU needs computing power resources. This enables flexible borrowing of computing power resources during the training process. Similarly, memory resource borrowing compares the memory pressure value with a second threshold, selects and constructs a remote access channel for the second borrowed vGPU from a preset memory resource pool, and realizes remote storage of the training task without CPU memory copying. When the memory pressure value is less than the second threshold, the access channel is deleted to reclaim the memory. The borrowing method is flexible and low-cost. Therefore, it can cope with the fluctuation of computing power and memory demand during the training process, improve the utilization rate of computing power and memory resources during the training process, and thus improve the overall efficiency of the training task.

[0013] Optionally, the step of acquiring and identifying the current training stage of each training task in real time includes:

[0014] The system intercepts the instruction stream sent from the guest operating system to the physical GPU in real time and extracts instruction feature information from the instruction stream, including CUDA / ROCm API call sequences and kernel function execution information.

[0015] The instruction stream is compared with a preset instruction feature library to obtain the feature matching degree, and the training stage of the current training task is determined based on the feature matching degree.

[0016] By adopting the above technical solution, the computation modes of different training stages, such as forward propagation, back propagation, and gradient synchronization, are different, and the generated kernel types, execution order, and resource consumption characteristics are also completely different, that is, the corresponding instruction feature information is different. By intercepting and analyzing these underlying instruction feature information, the current training stage can be inferred in reverse.

[0017] Optionally, the step of intercepting the instruction stream sent from the guest operating system to the physical GPU in real time and extracting instruction feature information from the instruction stream includes the following steps:

[0018] Intercept the instruction stream and identify the instruction strings in the instruction stream in real time. Identify the start string and end string of each instruction from the instruction strings and divide them into several different instruction information.

[0019] Identify characteristic characters in instruction information to determine the name information and characteristics of each instruction;

[0020] The name information and corresponding instruction characteristics of multiple consecutive instructions are used as the instruction characteristic information of the instruction stream.

[0021] By adopting the above technical solution, the splitting and identification of multiple instructions in the instruction stream can be achieved by pre-storing and comparing the start and end strings in the instruction string. The name and feature information of each instruction in the instruction stream can be obtained, forming instruction feature information. This can reveal the loop characteristics and computational features of the instruction stream, thereby identifying the training stage of the current training task. This lightweight detection and identification method has low overhead.

[0022] Optionally, in the step of matching the current computational pressure value and memory pressure value for the training task, the criteria for determining the computational pressure value include the number of active cores of the vGPU per unit time, the core computational density, and the thread grid size; the criteria for determining the memory pressure value include the current memory allocation request frequency, memory release frequency, and the current rate of change of memory usage of the vGPU.

[0023] By adopting the above technical solution, the calculation of computational pressure and memory pressure values ​​includes multi-dimensional information data, making the obtained computational pressure and memory pressure values ​​more accurate and reliable.

[0024] Optionally, the step of selecting the first idle vGPU to be borrowed from the pre-built computing resource pool when the computational pressure of the training task is greater than or equal to the first threshold, temporarily authorizing the computing resources of the first vGPU to be borrowed to the vGPU of the current training task, and updating the current computing resource pool includes:

[0025] The computational pressure value is used to determine the appropriate computing power range for the current training stage of the training task, and the current computing power gap of the training task is calculated based on the existing computing power value of the current training task.

[0026] Based on the preset communication cost awareness rules, idle vGPUs with low communication costs and that meet the computing power gap are selected as the first vGPUs to be borrowed.

[0027] Based on the computing power gap, the corresponding computing power resources are allocated to the vGPU of the current training task; and the task scheduling of the first vGPU to be borrowed is paused and updated to the resource mapping table of the computing resource pool.

[0028] By adopting the above technical solution, in the process of selecting the first vGPU to be borrowed, the computing power gap is calculated based on the computing power range adapted to the current training task, i.e. the required computing power range, and the computing power value currently possessed by the training task. The size of the computing power resources to be borrowed is accurately assessed. After the pre-assessment of the borrowed computing power, based on the communication cost awareness rule, the vGPU with the lowest communication cost is selected as the first vGPU to be borrowed, so that the cost of borrowing computing power resources is lower. After the computing power resources are borrowed, the computing power resources of each vGPU are updated to the computing resource pool, which facilitates the accurate borrowing of vGPU resources in the future.

[0029] Optionally, the step of selecting a second vGPU with idle memory resources in the pre-built memory resource pool when the memory pressure value of the training task is greater than or equal to the second threshold, constructing a memory access channel to enable the vGPU of the current training task to read and write the memory resources of the second vGPU to be borrowed, and updating the current memory resource pool when the memory pressure value of the training task is greater than or equal to the second threshold includes:

[0030] Once the second vGPU to be borrowed is selected, a preset memory area is drawn on the physical memory of the second vGPU to be borrowed, and the memory area is used as the far-end extended memory of the vGPU where the current training task is located.

[0031] By establishing an RDMA access mapping channel through NVLink's DMA engine, the vGPU where the current training task is located can directly use the RDMA access mapping channel to dynamically migrate temporary or infrequently used data to the memory area allocated by the second vGPU to be borrowed.

[0032] By adopting the above technical solution, the borrowing of video memory area is carried out by constructing a remote access channel to remotely borrow and store video memory resources. The video memory resources of the second vGPU to be borrowed are used as the remote extension video memory of the current training task, which reduces the cost of video memory borrowing. When borrowing storage, temporary or infrequently used data are dynamically migrated. This ensures that the core and frequently used data of the training task are still stored in the current vGPU, reducing the overhead of context switching during training.

[0033] Optionally, the step of recovering computing resources to the first borrowed vGPU when the computational pressure of the training task is less than the first threshold or when the first borrowed vGPU needs to restore computing power specifically includes:

[0034] When the computational pressure of the training task is less than the first threshold, all the computing resources borrowed by the current training task vGPU will be recovered to the first vGPU to be borrowed.

[0035] When the training phase of the first vGPU to be borrowed changes and needs to increase computing power, and the current computing pressure value is still greater than or equal to the first threshold, determine the computing power value required after the first vGPU to be borrowed changes its training phase.

[0036] If the required computing power is greater than or equal to the borrowed computing power resources, then all the borrowed computing power resources will be recycled to the first vGPU to be borrowed.

[0037] If the required computing power is less than the borrowed computing power, the borrowed computing power will be re-allocated to the first vGPU to be borrowed, with an amount equal to the required computing power. The remaining computing power will still be authorized for use by the vGPU of the current training task.

[0038] By adopting the above technical solution, when the computational pressure of the training task is less than the first threshold or the first borrowed vGPU needs to restore computing power, the proportion of computing power restoration needs to be further determined according to different judgment conditions. When the computational pressure is less than the first threshold or the computing power required by the first borrowed vGPU is greater than the borrowed computing power resources, all computing power resources borrowed by the training task are restored. When the computing power required by the first borrowed vGPU is less than the borrowed computing power resources, computing power of equal size to the required computing power needs to be allocated and recovered to the first borrowed vGPU. This ensures that the current training task still enjoys the improved computing power service, and the training task run by the first borrowed vGPU can also be satisfied with the computing power, thus improving the overall running efficiency of multiple training tasks.

[0039] The second objective of this invention is achieved through the following technical solution:

[0040] An AI task training system based on dynamic reconfiguration virtualization using GPU resources, comprising:

[0041] The task identification module is used to acquire and identify the current training stage of each training task in real time, and match the current computational pressure value and memory pressure value for the training task.

[0042] The first judgment module is used to select the first idle vGPU to be borrowed from the pre-built computing resource pool when the computing pressure value of the training task is greater than or equal to the first threshold, temporarily authorize the computing resources of the first vGPU to be borrowed to the vGPU of the current training task, and update the current computing resource pool when the computing pressure value of the training task is greater than or equal to the first threshold.

[0043] The second judgment module is used to filter out the second available vGPU that is available in the current training phase from the pre-built memory resource pool when the memory pressure value of the training task is greater than or equal to the second threshold, build a memory access channel so that the vGPU of the current training task can read and write the memory resources of the second available vGPU, and update the current memory resource pool.

[0044] The first recovery module is used to recover computing resources to the first vGPU to be borrowed when the computational pressure of the training task is less than the first threshold or when the first vGPU to be borrowed needs to recover computing power.

[0045] The second recovery module is used to migrate the data stored in the second borrowed vGPU back and remove the corresponding memory access channel when the memory pressure value of the training task is less than the second threshold.

[0046] The above-mentioned objective three of this application is achieved through the following technical solution:

[0047] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for training artificial intelligence tasks based on GPU resource-driven dynamic reconfiguration virtualization.

[0048] The fourth objective of this application is achieved through the following technical solution:

[0049] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described AI task training method based on GPU resource dynamic reconfiguration virtualization.

[0050] In summary, this application includes at least one of the following beneficial technical effects:

[0051] 1. From the pre-abstracted computing resource pool, a first vGPU with idle computing power is selected and temporarily authorized for use by the current training task. The borrowed computing power slice is reclaimed in real time when the computing pressure value is less than a first threshold or when the first vGPU needs computing power. This realizes flexible borrowing of computing power resources during the training process. Similarly, the borrowing of video memory resources is achieved by comparing the video memory pressure value with a second threshold, selecting and constructing a remote access channel for the second vGPU to be borrowed from the preset video memory resource pool, realizing remote storage of the training task without CPU memory copying. The access channel is deleted when the video memory pressure value is less than the second threshold to realize video memory reclamation. The borrowing method is flexible and low cost. Therefore, it realizes the response and allocation of computing power and video memory demand fluctuations during the training process, improves the utilization rate of computing power and video memory resources during the training process, and thus improves the overall efficiency of the training task.

[0052] 2. Different training stages, such as forward propagation, back propagation, and gradient synchronization, have different computation modes, resulting in distinct kernel types, execution orders, and resource consumption characteristics. In other words, the corresponding instruction feature information is different. By intercepting and analyzing these underlying instruction feature information, the current training stage can be inferred in reverse.

[0053] 3. By knowing the name and feature information of each instruction in the instruction stream, instruction feature information can be formed. This allows us to know the loop characteristics and computational features of the instruction stream, thereby identifying the training stage of the current training task. This lightweight detection and recognition method has low overhead.

[0054] 4. Calculate the computing power gap, accurately assess the size of the computing power resources to be borrowed, and after the pre-assessment of the computing power to be borrowed, select the vGPU with the lowest communication cost as the first vGPU to be borrowed based on the communication cost awareness rule, so that the cost of borrowing computing power resources is lower. After the computing power resources are borrowed, update the computing power resources of each vGPU to the computing resource pool to facilitate the accurate borrowing of vGPU resources in the future. Attached Figure Description

[0055] Figure 1 This is a flowchart of an implementation of an embodiment of an artificial intelligence task training method based on dynamic reconfiguration virtualization of GPU resources according to this application;

[0056] Figure 2 This is a flowchart illustrating the implementation of step S11 in an embodiment of an AI task training method based on dynamic reconfiguration virtualization using GPU resources, as described in this application.

[0057] Figure 3 This is a flowchart of step S40 in an embodiment of an AI task training method based on dynamic reconfiguration virtualization of GPU resources according to this application.

[0058] Figure 4 This is a schematic block diagram of a computer device according to this application. Detailed Implementation

[0059] The following is in conjunction with the appendix Figure 1-4 This application will be described in further detail.

[0060] In one embodiment, such as Figure 1 As shown, this application discloses an artificial intelligence task training method based on dynamic reconfiguration virtualization using GPU resources, specifically including the following steps:

[0061] S10: Real-time acquisition and identification of the current training stage of each training task, and matching the current computational pressure value and memory pressure value for the training task;

[0062] In this embodiment, the training phase includes a data loading phase, such as an I / O-intensive training task; a forward propagation phase, such as a computationally intensive training task; a backpropagation phase, such as a computationally intensive training task; and a gradient synchronization phase, such as a communication-intensive training task. Especially in MoE hybrid expert models or Transformer models with dynamic sparsity, the number of neurons activated in different batches varies greatly, causing computational load and memory usage to fluctuate drastically within milliseconds. The I / O waiting period during the data loading phase results in idle computational power, while the computational power required for the forward or backpropagation phases increases.

[0063] The criteria for determining the computational stress value include the number of active cores of the vGPU per unit time, the core computation density, and the thread grid size; in this embodiment, the gradient setting for the computational stress value is as follows:

[0064] The calculated stress value range is 0 ~ 0.3, which is considered a light load. The response strategy is to actively lend resources, that is, to actively report available resources to the scheduler and become a resource provider.

[0065] The calculated stress value range is 0.3 ~ 0.6, which is considered a moderate load. The response strategy is to maintain the status quo, meaning that local resources are sufficient and neither borrowing nor lending.

[0066] The calculated stress value range is 0.6 ~ 0.8, which is considered a heavy load. The response strategy is light borrowing, that is, attempting to borrow a small amount (e.g., 20%) of neighbor resources to smooth out performance bottlenecks.

[0067] The computational stress value range is 0.8 ~ 0.95, which is considered an extreme load. The response strategy is emergency borrowing, that is, actively searching for available resources and borrowing as much computing power as possible.

[0068] The calculation stress value range is 0.95 ~ 1.0, which is considered overload. The response strategy is circuit breaker protection, which triggers QoS degradation, suspends non-core computing, and ensures that the main task does not fail.

[0069] The criteria for determining the video memory pressure value include the vGPU's current memory allocation request frequency, memory release frequency, and the rate of change of the vGPU's current memory usage. In this embodiment, the gradient setting for the video memory pressure value is as follows:

[0070] The memory stress value range is 0 ~ 0.4, which is a safe zone. The response strategy is normal lending, so you can act as a memory lender.

[0071] The memory stress value range is 0.4 ~ 0.7, which is in the warning zone. The response strategy is to prepare for borrowing, that is, to start searching for potential memory lenders and establish an access channel.

[0072] The memory stress value range is 0.7 ~ 0.9, which is a dangerous area. The corresponding strategy is to actively borrow, that is, to initiate data migration and unload cold data or intermediate active values ​​to a remote location.

[0073] The memory pressure range is 0.9 ~ 1.0, which is in the critical zone. The response strategy is emergency handling, which triggers compression and discards non-critical data; for example, certain activation value checkpoints.

[0074] Specifically, step S10, concerning the real-time acquisition and identification of the current training stage of each training task, includes:

[0075] S11: Real-time interception of instruction streams sent from the guest operating system to the physical GPU, extraction of instruction feature information from the instruction stream, including CUDA / ROCm API call sequences and kernel function execution information;

[0076] S12: Compare the instruction stream with the preset instruction feature library to obtain the feature matching degree, and determine the training stage of the current training task based on the feature matching degree.

[0077] In this embodiment, a computational feature awareness module is embedded in the virtualization monitor or GPU driver layer. This module intercepts and analyzes the instruction stream sent from the guest operating system to the physical GPU.

[0078] The instruction feature information differs across different training stages in the instruction stream, including:

[0079] The instruction stream during the data loading phase is characterized by the presence of few or no computational kernels. This phase primarily executes the CPU-side DataLoader, loading data from disk into memory and then copying it from memory to GPU memory. The instruction stream mainly consists of cudaMemcpyAsync (Host->Device), accompanied by a small number of data preprocessing kernels (such as image decoding nvJPEG and data augmentation DALI). Streaming multiprocessors (SMs) are mostly idle or under low load. Identification indicators: API call sequence = [memcpyH2D] + [preprocessing kernel] + [memcpyH2D], and SM activity is below a threshold (e.g., <10%).

[0080] The instruction flow during the forward propagation phase is characterized by a specific pattern of kernel sequences, executed sequentially according to the network structure. For example, for CNNs, it's a loop of Conv2D -> BatchNorm -> ReLU -> Pooling; for Transformers, it's a loop of MHA (Multi-Head Attention) -> LayerNorm -> FFN (Feed-Forward Network). Each kernel has a long execution time, consuming a large number of signal processing units (SMs); it mainly reads weight parameters and the input from the previous layer and writes them to the output of the current layer.

[0081] Identification markers: Specific kernel name prefixes are identified, such as volta_fp16_s884gemm, which represents matrix multiplication and is often used in fully connected layers; and cudnn_convolution, which represents convolution. Furthermore, the kernel execution intervals are short, forming a pipeline.

[0082] The instruction flow during the backpropagation phase is characterized by a kernel sequence and its forward mirror image, executed in reverse order of the computation graph. For example, the loss function kernel is executed first, followed by Conv2D backward and BatchNorm backward, with kernel names containing specific suffixes such as dgrad (data gradient), wgrad (weight gradient), and backward. Its memory read / write pattern is complex, requiring not only reading the weights but also the intermediate activation values ​​saved during forward propagation for gradient calculation.

[0083] Identification markers: The kernel name contains keywords such as backward, dgrad, and wgrad, and the instruction flow order is reversed compared to the forward propagation phase.

[0084] The gradient synchronization phase is crucial for distributed training. Its instruction flow is characterized by dense communication kernels, including the execution of collective communication operations such as AllReduce, ReduceScatter, and AllGather. Kernel names include a communication library prefix, such as the NVIDIA Collective Communications Library kernel. During communication, computation kernels pause, and SMs (Session Managers) remain idle, waiting for gradient data to arrive from other nodes.

[0085] Identification markers: A large number of kernels starting with nccl, rccl (ROCm) were detected, and network interface (NIC) traffic surged while GPU computing cores remained idle.

[0086] Reference Figure 2 Furthermore, step S11 includes the following steps:

[0087] S111: Intercept the instruction stream and identify the instruction strings in the instruction stream in real time. Identify the start string and end string of each instruction from the instruction strings and divide them into several different instruction information.

[0088] S112: Identify the characteristic characters in the instruction information to determine the name information and characteristics of each instruction;

[0089] S113: The name information and corresponding instruction characteristics of multiple consecutive instructions are used as the instruction characteristic information of the instruction stream.

[0090] In this embodiment, different instructions are distinguished by start and end characters to obtain the name of the instruction, the characteristics of each instruction, and the cyclic characteristics of multiple consecutive instructions, such as the kernel sequence of a specific pattern in the forward ship phase, thereby forming accurate instruction feature information. By comparing with the instruction features pre-stored in the database and the cyclic characteristics of multiple instructions in the instruction stream, the training phase represented by the current instruction stream can be identified.

[0091] S20: When the computational pressure of the training task is greater than or equal to the first threshold, select the first idle vGPU to be borrowed from the pre-built computing resource pool, temporarily authorize the computing resources of the first vGPU to be borrowed to the vGPU of the current training task, and update the current computing resource pool.

[0092] In this embodiment, the first threshold refers to the computational pressure range of the aforementioned heavy load. The degree of urgency for borrowing varies depending on whether the load range is heavy or extreme. After the borrowing is completed, the current resource occupancy of all vGPUs is updated to the computing resource pool. Temporary authorization of computing resources means temporarily allocating a portion of the time slice to the vGPU for the current training task.

[0093] Specifically, step S20 includes the following steps:

[0094] S21: Based on the computational pressure value matching, determine the computing power range that the training task is suitable for in the current training stage, and calculate the current computing power gap of the training task based on the existing computing power value of the current training task.

[0095] S22: Based on the preset communication cost awareness rules, prioritize the selection of idle vGPUs with low communication costs and that meet the computing power gap as the first vGPUs to be borrowed;

[0096] S23: Based on the computing power gap, allocate the corresponding computing power resources to the vGPU of the current training task; and suspend the task scheduling of the first vGPU to be borrowed, and update the resource mapping table of the computing resource pool.

[0097] In this embodiment, the computing power range adapted to the current training stage refers to the range of computing power required by the training task in the current training stage. The required computing power range is different for different training stages. Based on the computing power of the vGPU where the training task is currently located and the required computing power range, the computing power gap is calculated. When screening the first vGPU to be borrowed, under the premise of combining the communication cost awareness rule, the vGPU that can adapt to the computing power gap is selected as the first vGPU to be borrowed. In the communication cost awareness rule, the communication cost from low to high is as follows: different MIG slices within the same chip; different physical GPUs under the same PCIe switch and within the same NVLink domain; GPUs under the same CPU socket but across different PCIe switches; GPUs across CPU sockets and GPUs across physical servers; and the first vGPU to be borrowed takes the highest priority in terms of communication cost.

[0098] S30: When the memory pressure value of the training task is greater than or equal to the second threshold, select the second vGPU with free memory resources in the pre-built memory resource pool, build a memory access channel so that the vGPU of the current training task can read and write the memory resources of the second vGPU to be borrowed, and update the current memory resource pool.

[0099] In this embodiment, the screening of the second vGPU to be borrowed also takes into account communication costs, and vGPUs with low communication costs are selected first to allocate the borrowed video memory space.

[0100] Specifically, once the second vGPU to be borrowed is selected, a pre-defined memory region is allocated on the physical memory of the second vGPU to be borrowed, and this memory region is used as the remote extended memory of the vGPU where the current training task is located. An RDMA access mapping channel is established through NVLink's DMA engine, and the vGPU where the current training task is located directly uses the RDMA access mapping channel to dynamically migrate temporary or infrequently used data to the memory region allocated by the second vGPU to be borrowed, without having to go through CPU memory copying.

[0101] S40: When the computational pressure of the training task is less than the first threshold or the first borrowed vGPU needs to restore computing power, the computing power resources are recovered to the first borrowed vGPU.

[0102] In this embodiment, the computational pressure value is less than the first threshold, that is, the computational pressure value is less than 0.6 of the critical value of the heavy load range. The recovery of the computing power of the first borrowed vGPU includes the recovery of all borrowed computing power resources and the recovery of part of the borrowed computing power resources. The recovery of computing power resources refers to the recovery of the borrowed computing time slices.

[0103] Reference Figure 3 Specifically, step S40 includes the following steps:

[0104] S41: When the computational pressure of the training task is less than the first threshold, all the computing resources borrowed by the current training task vGPU are recovered to the first vGPU to be borrowed.

[0105] S42: When the training phase of the first vGPU to be borrowed changes and the computing power needs to be increased, but the current computing pressure value is still greater than or equal to the first threshold, determine the computing power value required after the first vGPU to be borrowed changes its training phase.

[0106] S43: If the required computing power is greater than or equal to the borrowed computing power resources, then all the borrowed computing power resources will be returned to the first borrowed vGPU.

[0107] S44: If the required computing power is less than the borrowed computing power, the borrowed computing power will be re-allocated to the first vGPU to be borrowed, with an amount equal to the required computing power. The remaining computing power will still be authorized to the vGPU of the current training task.

[0108] In this embodiment, when the training phase of the first borrowed vGPU changes and requires increased computing power, the computing power recovery value needs to be analyzed and determined. When the required computing power is greater than or equal to the borrowed computing power resources, all borrowed computing power resources should be returned. Specifically, when the required computing power is less than the borrowed computing power resources, only the computing power required for the current training phase of the first borrowed vGPU needs to be recovered, and the remaining unrecovered computing power resources are still used for the current training task. Subsequently, when the computing power resource requirement of the first borrowed vGPU rises to exceed the borrowed computing power resources, all remaining borrowed computing power resources will be recovered.

[0109] Preferably, if the currently borrowed computing resources are insufficient, computing resources are obtained from other vGPUs to make up for the fluctuating computing power gap.

[0110] S50: When the memory pressure of the training task is less than the second threshold, the data stored in the second borrowed vGPU is moved back and the corresponding memory access channel is removed.

[0111] In this embodiment, when the second vGPU to be borrowed has data storage requirements, that is, when the video memory demand increases, the second vGPU to be borrowed will reclaim video memory or reclaim part of the video memory resources, just like computing resources. When the second vGPU to be borrowed does not meet the video memory requirements to be borrowed for the current training task, the second vGPU to be borrowed will be replaced.

[0112] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0113] In one embodiment, an AI task training system based on GPU resource dynamic reconfiguration virtualization is provided. This system corresponds to the AI ​​task training method based on GPU resource dynamic reconfiguration virtualization described in the previous embodiment. The GPU resource-based AI task training system includes:

[0114] The task identification module is used to acquire and identify the current training stage of each training task in real time, and match the current computational pressure value and memory pressure value for the training task.

[0115] The first judgment module is used to select the first idle vGPU to be borrowed from the pre-built computing resource pool when the computing pressure value of the training task is greater than or equal to the first threshold, temporarily authorize the computing resources of the first vGPU to be borrowed to the vGPU of the current training task, and update the current computing resource pool when the computing pressure value of the training task is greater than or equal to the first threshold.

[0116] The second judgment module is used to filter out the second available vGPU that is available in the current training phase from the pre-built memory resource pool when the memory pressure value of the training task is greater than or equal to the second threshold, build a memory access channel so that the vGPU of the current training task can read and write the memory resources of the second available vGPU, and update the current memory resource pool.

[0117] The first recovery module is used to recover computing resources to the first vGPU to be borrowed when the computational pressure of the training task is less than the first threshold or when the first vGPU to be borrowed needs to recover computing power.

[0118] The second recovery module is used to migrate the data stored in the second borrowed vGPU back and remove the corresponding memory access channel when the memory pressure value of the training task is less than the second threshold.

[0119] Optionally, the task recognition module includes:

[0120] The interception submodule is used to intercept the instruction stream sent from the guest operating system to the physical GPU in real time, and extract instruction feature information from the instruction stream. The instruction feature information includes CUDA / ROCm API call sequence and kernel function execution information.

[0121] The feature matching submodule is used to compare the instruction stream with a preset instruction feature library to obtain the feature matching degree, and determine the training stage of the current training task based on the feature matching degree.

[0122] Optionally, the interception submodule includes:

[0123] The character recognition unit is used to intercept the instruction stream and recognize the instruction strings in the instruction stream in real time. It identifies the start string and end string of each instruction from the instruction string and divides them into several different instruction information.

[0124] The character analysis unit is used to identify characteristic characters in instruction information and determine the name information and characteristics of each instruction.

[0125] The judgment unit is used to treat the name information and corresponding instruction characteristics of multiple consecutive instructions as instruction characteristic information of the instruction stream.

[0126] Optionally, the first judgment module includes:

[0127] The gap calculation submodule is used to determine the appropriate computing power range for the current training stage of the training task based on the computing pressure value, and to calculate the current computing power gap of the training task based on the existing computing power value of the current training task.

[0128] The cost-aware submodule is used to prioritize the selection of idle vGPUs with low communication costs and that meet the computing power gap based on preset communication cost awareness rules, and to select them as the first vGPUs to be borrowed.

[0129] The borrowing submodule is used to allocate corresponding computing resources to the vGPU of the current training task based on the computing power gap; and to pause the task scheduling of the first vGPU to be borrowed and update the resource mapping table of the computing resource pool.

[0130] Optionally, the second judgment module includes:

[0131] The video memory partitioning submodule is used to partition a preset video memory area on the physical video memory of the second vGPU to be borrowed after the second vGPU to be borrowed is selected, and to use the video memory area as the far-end extended video memory of the vGPU where the current training task is located.

[0132] The channel establishment submodule is used to establish an RDMA access mapping channel through NVLink's DMA engine. The vGPU where the current training task is located directly uses the RDMA access mapping channel to dynamically migrate temporary or infrequently used data to the memory area allocated by the second vGPU to be borrowed.

[0133] Optionally, the first recovery module includes:

[0134] The recovery judgment submodule, when the computational pressure of the training task is less than the first threshold, will recover all the computing resources borrowed by the current training task vGPU to the first vGPU to be borrowed.

[0135] The computing power calculation submodule is used to determine the computing power required after the first vGPU to be borrowed changes its training phase and needs to increase computing power when the training phase of the first vGPU to be borrowed changes and the current computing pressure value is still greater than or equal to the first threshold.

[0136] The recycling judgment submodule is used to recycle all the borrowed computing resources to the first borrowed vGPU if the required computing power value is greater than or equal to the borrowed computing power resources.

[0137] If the required computing power is less than the borrowed computing power, the borrowed computing power will be re-allocated to the first vGPU to be borrowed, with an amount equal to the required computing power. The remaining computing power will still be authorized for use by the vGPU of the current training task.

[0138] Specific limitations regarding the GPU-based dynamically reconfigurable virtualization AI task training system can be found in the above section on the limitations of the GPU-based dynamically reconfigurable virtualization AI task training method, and will not be repeated here. Each module in the aforementioned GPU-based dynamically reconfigurable virtualization AI task training system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0139] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an artificial intelligence task training method based on GPU resource-driven dynamic reconfiguration virtualization.

[0140] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements an artificial intelligence task training method based on GPU resource dynamic reconfiguration virtualization.

[0141] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements an artificial intelligence task training method based on dynamic reconfiguration virtualization using GPU resources.

[0142] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0143] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0144] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for training artificial intelligence tasks based on dynamic reconfiguration virtualization using GPU resources, characterized in that: Real-time acquisition and identification of the current training stage of each training task, and matching the current computational pressure and memory pressure values ​​for the training task; When the computational pressure of the training task is greater than or equal to the first threshold, the first idle vGPU to be borrowed is selected from the pre-built computing resource pool. The computing resources of the first vGPU to be borrowed are temporarily authorized to the vGPU of the current training task, and the current computing resource pool is updated. When the memory pressure of the training task is greater than or equal to the second threshold, the second vGPU with free memory resources in the pre-built memory resource pool is selected to be borrowed during the current training phase. A memory access channel is built to enable the vGPU of the current training task to read and write the memory resources of the second vGPU to be borrowed, and the current memory resource pool is updated. When the computational pressure of the training task is less than the first threshold or the first borrowed vGPU needs to restore computing power, the computing power resources will be recovered to the first borrowed vGPU. When the memory pressure of the training task is less than the second threshold, the data stored in the second borrowed vGPU is moved back, and the corresponding memory access channel is removed.

2. The method for training artificial intelligence tasks based on dynamic reconfiguration virtualization using GPU resources according to claim 1, characterized in that, The steps for real-time acquisition and identification of the current training stage of each training task include: The system intercepts the instruction stream sent from the guest operating system to the physical GPU in real time and extracts instruction feature information from the instruction stream, including CUDA / ROCm API call sequences and kernel function execution information. The instruction stream is compared with a preset instruction feature library to obtain the feature matching degree, and the training stage of the current training task is determined based on the feature matching degree.

3. The method for training artificial intelligence tasks based on dynamic reconfiguration virtualization using GPU resources according to claim 2, characterized in that, The step of intercepting the instruction stream sent from the guest operating system to the physical GPU in real time and extracting instruction feature information from the instruction stream includes the following steps: Intercept the instruction stream and identify the instruction strings in the instruction stream in real time. Identify the start string and end string of each instruction from the instruction strings and divide them into several different instruction information. Identify characteristic characters in instruction information to determine the name information and characteristics of each instruction; The name information and corresponding instruction characteristics of multiple consecutive instructions are used as the instruction characteristic information of the instruction stream.

4. The method for training artificial intelligence tasks based on dynamic reconfiguration virtualization using GPU resources according to claim 1, characterized in that, In the step of matching the current computational pressure value and memory pressure value for the training task, the criteria for determining the computational pressure value include the number of active cores of the vGPU per unit time, the core computation density, and the thread grid size; the criteria for determining the memory pressure value include the current memory allocation request frequency, memory release frequency, and the current rate of change of memory usage of the vGPU.

5. The method for training artificial intelligence tasks based on dynamic reconfiguration virtualization using GPU resources according to claim 1, characterized in that, The step of selecting the first idle vGPU (virtual GPU) from the pre-built computing resource pool when the computational pressure of the training task is greater than or equal to a first threshold, temporarily allocating the computing resources of the first idle vGPU to the vGPU of the current training task, and updating the current computing resource pool when the computational pressure of the training task is greater than or equal to a first threshold includes: The computational pressure value is used to determine the appropriate computing power range for the current training stage of the training task, and the current computing power gap of the training task is calculated based on the existing computing power value of the current training task. Based on the preset communication cost awareness rules, idle vGPUs with low communication costs and that meet the computing power gap are selected as the first vGPUs to be borrowed. Based on the computing power gap, the corresponding computing power resources are allocated to the vGPU of the current training task; and the task scheduling of the first vGPU to be borrowed is paused and updated to the resource mapping table of the computing resource pool.

6. The method for training artificial intelligence tasks based on dynamic reconfiguration virtualization using GPU resources according to claim 1, characterized in that, The step of selecting a second available vGPU for borrowing from the pre-built memory resource pool when the memory pressure of the training task is greater than or equal to the second threshold, constructing a memory access channel to enable the vGPU of the current training task to read and write the memory resources of the second available vGPU, and updating the current memory resource pool when the memory pressure of the training task is greater than or equal to the second threshold includes: Once the second vGPU to be borrowed is selected, a preset memory area is drawn on the physical memory of the second vGPU to be borrowed, and the memory area is used as the far-end extended memory of the vGPU where the current training task is located. By establishing an RDMA access mapping channel through NVLink's DMA engine, the vGPU where the current training task is located can directly use the RDMA access mapping channel to dynamically migrate temporary or infrequently used data to the memory area allocated by the second vGPU to be borrowed.

7. The method for training artificial intelligence tasks based on dynamic reconfiguration virtualization using GPU resources according to claim 5, characterized in that, The step of recovering computing resources to the first borrowed vGPU when the computational pressure of the training task is less than the first threshold or when the first borrowed vGPU needs to restore computing power is specifically as follows: When the computational pressure of the training task is less than the first threshold, all the computing resources borrowed by the current training task vGPU will be recovered to the first vGPU to be borrowed. When the training phase of the first vGPU to be borrowed changes and needs to increase computing power, and the current computing pressure value is still greater than or equal to the first threshold, determine the computing power value required after the first vGPU to be borrowed changes its training phase. If the required computing power is greater than or equal to the borrowed computing power resources, then all the borrowed computing power resources will be recycled to the first vGPU to be borrowed. If the required computing power is less than the borrowed computing power, the borrowed computing power will be re-allocated to the first vGPU to be borrowed, with an amount equal to the required computing power. The remaining computing power will still be authorized for use by the vGPU of the current training task.

8. An artificial intelligence task training system based on dynamic reconfiguration virtualization using GPU resources, characterized in that: The task identification module is used to acquire and identify the current training stage of each training task in real time, and match the current computational pressure value and memory pressure value for the training task. The first judgment module is used to select the first idle vGPU to be borrowed from the pre-built computing resource pool when the computing pressure value of the training task is greater than or equal to the first threshold, temporarily authorize the computing resources of the first vGPU to be borrowed to the vGPU of the current training task, and update the current computing resource pool when the computing pressure value of the training task is greater than or equal to the first threshold. The second judgment module is used to filter out the second available vGPU that is available in the current training phase from the pre-built memory resource pool when the memory pressure value of the training task is greater than or equal to the second threshold, build a memory access channel so that the vGPU of the current training task can read and write the memory resources of the second available vGPU, and update the current memory resource pool. The first recovery module is used to recover computing resources to the first vGPU to be borrowed when the computational pressure of the training task is less than the first threshold or when the first vGPU to be borrowed needs to recover computing power. The second recovery module is used to migrate the data stored in the second borrowed vGPU back and remove the corresponding memory access channel when the memory pressure value of the training task is less than the second threshold.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the AI ​​task training method based on dynamic reconfiguration virtualization of GPU resources as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the AI ​​task training method based on GPU resource dynamic reconfiguration virtualization as described in any one of claims 1 to 7.

Images