A method for scheduling GPU task resources based on a microservice architecture
By combining microservice task dependency analysis and CUDA Checkpoint technology, GPU task resources are dynamically configured, solving the problems of GPU task interruption and low resource utilization, and achieving efficient GPU task resource scheduling.
Patent Information
- Application Number
- CN202611295895.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-25
- Publication Date
- 2026-11-17
AI Technical Summary
In microservice GPU task execution scenarios with dependencies, the GPU memory and computing resource requirements change continuously at different stages. Existing methods cannot uniformly consider the physical location of the target MIG instance, the occupancy status of existing MIG instance slices, and the running task status, resulting in task interruption and low resource utilization.
By combining microservice task dependency analysis, dynamic configuration of MIG resources, shortest path search and CUDA Checkpoint technology, the lowest cost configuration conversion path is determined. Through the deletion, creation and reorganization of MIG instances, dynamic reconstruction of GPU task resources and task recovery are achieved.
It improves resource utilization, reduces the impact of task interruptions, lowers configuration conversion overhead, and enhances scheduling flexibility and continuity.
Smart Images

Figure CN123114747A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cloud computing resource scheduling technology, and in particular to a method for scheduling GPU task resources based on a microservice architecture. Background Technology
[0002] With the development of AI inference services, microservice architectures, and GPU shared computing, using MIG technology to divide physical GPUs into multiple independent instances and scheduling GPU resources according to task resource requirements has become an important way to improve GPU utilization. Existing technologies have disclosed related solutions. For example, CN120892216A generates a configuration template containing multiple combinations of MIG configuration instances based on GPU hardware data to achieve adaptive allocation of different GPU resources; CN121704956A constructs a service topology based on microservice call relationships and forms independent computing units by splitting them into MIGs according to task memory requirements; CN121833233A further predicts future resource requirements based on GPU resource fragmentation and adjusts the GPU layout through task checkpoint migration and resource reconstruction; CN121455631A also discloses a scheduling method for task migration and recovery based on GPU task checkpoint data.
[0003] However, in microservice GPU task execution scenarios with dependencies, the memory and computing resource requirements change continuously at different stages. Although the target MIG instance can be determined according to resource specifications, its required physical slice location may still conflict with the running MIG instance. Existing methods mainly focus on scheduling based on MIG specification combinations, resource utilization, fragmentation risk, or task migration. They lack a configuration conversion mechanism that uniformly considers the physical location of the target MIG instance, the slice occupancy status of existing MIG instances, and the running task status. This can easily expand the task pause range when reconstructing GPU resources, causing tasks in unrelated MIG instances to be interrupted and increasing the overhead of task checkpoint saving, instance reconstruction, and downstream task waiting.
[0004] Therefore, how to provide a scheduling method for GPU task resources based on a microservice architecture is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] One objective of this invention is to propose a scheduling method for GPU task resources based on a microservice architecture. This invention combines microservice task dependency analysis, dynamic configuration of MIG resources, shortest path search, and CUDA Checkpoint technology. It determines the lowest-cost configuration conversion path based on GPU task resource requirements and physical slice occupancy conflicts, thereby achieving dynamic reconstruction of MIG resources and task recovery under running task protection conditions. It has the advantages of high resource utilization, minimal impact of task interruption, low configuration conversion overhead, and high scheduling flexibility.
[0006] A method for scheduling GPU task resources based on a microservice architecture according to an embodiment of the present invention includes the following steps: Step 1: Collect microservice call information and GPU task execution information, and construct GPU task dependency chains based on request identifiers and data dependencies; Step 2: Determine the target MIG instance specifications based on the execution status of the preceding graphics card tasks and the memory and computing resource requirements of the downstream graphics card tasks; Step 3: Obtain the physical slice occupancy relationship of the physical graphics card's MIG, map the target MIG instance specification to the physical slice position, and generate candidate instance layouts based on slice coverage conflicts with the current MIG instance; Step 4: Using the current MIG instance configuration and candidate instance layout as state nodes, and MIG instance deletion, creation, and reorganization as state transition edges, search for the lowest cost configuration transition path based on the number of affected GPU tasks, CUDA checkpoint state quantity, and configuration transition count. Step 5: When the preceding GPU task is completed and the downstream GPU task is ready to execute, perform a CUDA Checkpoint to save and release the corresponding MIG instance for the incomplete GPU task occupying the conflicting physical slice in the lowest cost configuration conversion path; Step 6: Reconstruct conflicting physical slices according to the lowest cost configuration conversion path, form target MIG instances and assign them to downstream GPU tasks, while MIG instances that have not experienced slice overwrite conflicts continue to run; Step 7: After the downstream graphics card task releases the target MIG instance, it restores the saved graphics card task and updates the state transition edge according to the actual configuration transition result.
[0007] Optionally, step one specifically includes: Collect microservice call information and GPU task execution information, and group microservice call information and GPU task execution information belonging to the same business request according to the request identifier; The microservice call order is determined based on the microservice identifier of the caller and the microservice identifier of the callee. The output data identifier of the previous graphics card task is matched with the input data identifier of the next graphics card task. When the output data identifier of the previous graphics card task is the same as the input data identifier of the next graphics card task, a data dependency relationship from the previous graphics card task to the next graphics card task is established. By treating GPU tasks as nodes and data dependencies as directed edges, a GPU task dependency chain is constructed. GPU tasks that are in running and have direct data dependencies are identified as preceding GPU tasks. GPU tasks that directly depend on the output of preceding GPU tasks and are in an unstarted state are identified as downstream GPU tasks.
[0008] Optionally, step two specifically includes: Based on the microservice type, model version, and input data scale of the downstream GPU task, filter similar historical GPU tasks, and determine the GPU memory and computing resource requirements of the downstream GPU task based on the peak GPU memory usage and computing resource usage level of similar historical GPU tasks. Filter MIG instance specifications with video memory capacity not less than video memory requirements and computing resource scale not less than computing resource requirements, and sort them in order of the number of MIG physical slices occupied from least to most, and select the MIG instance specification with the fewest MIG physical slices occupied as the target MIG instance specification. When there are multiple MIG instance specifications that occupy the same number of MIG physical slices, the video memory capacity of each MIG instance specification is reduced by the video memory requirement to obtain the video memory margin, and the computing resource scale of each MIG instance specification is reduced by the computing resource requirement to obtain the computing resource margin. The video memory margin and computing resource margin are normalized and weighted, and the MIG instance specification with the smallest weighted result is selected as the target MIG instance specification.
[0009] Optionally, step three specifically includes: Based on the instance start position and the number of MIG physical slices occupied by each current MIG instance, the current physical slice set corresponding to each current MIG instance is determined, forming the MIG physical slice occupancy relationship; The corresponding set of target physical slices is determined based on each legal placement location of the target MIG instance specification; The target physical slice set is compared with each current physical slice set one by one. When the target physical slice set and the current physical slice set contain at least one MIG physical slice with the same MIG, the corresponding current MIG instance is determined to be a conflicting MIG instance. When the target physical slice set and the current physical slice set do not contain any MIG physical slices with the same MIG, the corresponding current MIG instance is determined to be a non-conflicting MIG instance. For each legal placement location, conflicting MIG instances are collected separately, and the target MIG instance, conflicting MIG instances, and non-conflicting MIG instances that maintain the original MIG physical slice occupancy relationship are combined to generate a candidate instance layout.
[0010] Optionally, the step of collecting conflicting MIG instances for each legal placement location, and combining the target MIG instance, conflicting MIG instances, and non-conflicting MIG instances that maintain the original MIG physical slice occupancy relationship to generate a candidate instance layout, specifically includes: A physical slice occupancy table is established according to the MIG physical slice arrangement order. The corresponding current MIG instance is recorded in the MIG physical slice position occupied by the current MIG instance, and the MIG physical slice position not occupied by the current MIG instance is recorded as free. For each legal placement location of the target MIG instance specification, starting from the corresponding instance start position, continuously select MIG physical slices with the same number of MIG physical slices occupied by the target MIG instance specification to form a target physical slice set; Based on the physical slice occupancy table, determine the current MIG instance occupying the target physical slice, add the current MIG instance occupying at least one target physical slice to the corresponding conflicting MIG instance set, and establish the correspondence between the target physical slice set, the conflicting MIG instance, and the graphics card task currently being carried by the conflicting MIG instance. Remove the corresponding conflicting MIG instance from the current MIG instance configuration, keep the instance specifications, instance starting position and MIG physical slice occupancy relationship of the non-conflicting MIG instance unchanged, and add the target MIG instance to the target physical slice set to form a candidate instance layout for the corresponding legal placement position; For different legal placement locations, corresponding candidate instance layouts are retained, so that different legal placement locations of the same target MIG instance specification correspond to different sets of conflicting MIG instances and sets of graphics card tasks currently carried by the conflicting MIG instances.
[0011] Optionally, step four specifically includes: The current MIG instance configuration is used as the initial state node, each candidate instance layout is used as the target state node, and intermediate state nodes are established based on the legal MIG instance configurations formed during the transformation from the current MIG instance configuration to each candidate instance layout. When two state nodes can complete the transition through deletion, creation, or reorganization of MIG instances, a state transition edge is established between the two state nodes; The number of affected GPU tasks is determined based on the current MIG instance that needs to be released by the state transition edge. The CUDA checkpoint state quantity is determined based on the pending state of the CUDA process corresponding to the affected GPU task. The number of configuration transitions is determined based on the number of MIG instance deletion operations and creation operations actually performed by the state transition edge. Among them, the number of MIG instance reorganization operations is counted as the number of deletion operations and creation operations actually performed to complete the MIG instance reorganization. The number of affected GPU tasks, CUDA checkpoint state quantities, and configuration transition counts are normalized and weighted to obtain the state transition cost. The state transition cost is then used as the edge weight of the state transition edge to search for the lowest cost configuration transition path.
[0012] Optionally, determining the lowest-cost configuration conversion path specifically includes: For state transition edges involving the deletion or reorganization of MIG instances, the current MIG instances that need to be released are determined based on the changes in the MIG physical slices corresponding to the state transition edges, and the number of incomplete GPU tasks in the current MIG instances that need to be released is counted to obtain the number of affected GPU tasks. The amount of data to be saved corresponding to the device memory content, CUDA context, and task running status of each affected GPU task is summarized to obtain the CUDA checkpoint state quantity. The number of affected GPU tasks, CUDA checkpoint state quantity, and configuration transition count are all Min-Max normalized separately, and then weighted and summed according to the weight of the number of affected GPU tasks, the weight of CUDA checkpoint state quantity, and the weight of the configuration transition count to obtain the state transition cost. Dijkstra's shortest path algorithm is used to accumulate the state transition costs from the initial state node to each target state node, and the configuration transition path with the minimum accumulated state transition cost is determined as the lowest cost configuration transition path. When the cumulative state transition cost of two or more configuration transition paths is the same, the first screening is performed according to the total number of CUDA checkpoint state quantities from smallest to largest. If there are still two or more configuration transition paths after the first screening, the second screening is performed according to the total number of affected GPU tasks from smallest to largest. If there are still two or more configuration transition paths after the second screening, the third screening is performed according to the number of configuration transitions from smallest to largest. Based on the lowest-cost configuration conversion path, determine the physical slice position of the target MIG instance in the corresponding candidate instance layout, the current MIG instance that needs to be released, and the incomplete GPU tasks that need to be saved by executing CUDA Checkpoint.
[0013] Optionally, step five specifically includes: When the preceding graphics card task has finished executing and the input data required by the downstream graphics card task has been written, it is determined that the downstream graphics card task is ready to execute. Based on the lowest cost configuration conversion path, identify the MIG instances that occupy conflicting physical slices and need to be released, and remove the completed graphics card tasks from the graphics card tasks currently carried by the MIG instances that occupy conflicting physical slices and need to be released, and identify the remaining unfinished graphics card tasks as graphics card tasks that need to be saved by CUDA Checkpoint. Stop the CUDA process corresponding to the GPU task that needs to be saved by CUDA Checkpoint from submitting new GPU work, and save the device memory contents, CUDA context and task running status of the corresponding CUDA process; Once all incomplete GPU tasks in the same MIG instance that needs to be released have completed the CUDA Checkpoint save, the runtime binding between all saved GPU tasks and the MIG instance that needs to be released is released, and the MIG instance that needs to be released is released. MIG instances that have not experienced slice overwrite conflicts do not perform CUDA Checkpoint saving, do not unbind GPU tasks, and remain running.
[0014] Optionally, step six specifically includes: Release the MIG instances that occupy conflicting physical slices and need adjustment in sequence according to the lowest cost configuration conversion path, so that the MIG physical slices corresponding to the MIG instances that occupy conflicting physical slices and need adjustment are released from occupation. Each time a MIG instance is released or reorganized, the MIG physical slice occupancy relationship is updated; if the MIG physical slices required to form the target MIG instance are still occupied by the current MIG instance, the next configuration operation in the lowest cost configuration conversion path is executed; if none of the MIG physical slices required to form the target MIG instance are occupied by the current MIG instance, the release of other MIG instances is stopped. Create a target MIG instance based on the physical slice location and target MIG instance specifications determined by the lowest cost configuration conversion path, and assign the target MIG instance to the downstream GPU task; MIG instances that have not experienced slice overwrite conflicts and have not entered the lowest-cost configuration conversion path maintain their original instance specifications, instance starting position, and MIG physical slice occupancy relationships. The GPU tasks carried by MIG instances that have not experienced slice overwrite conflicts and have not entered the lowest-cost configuration conversion path continue to run.
[0015] Optionally, step seven specifically includes: After the downstream GPU task releases the target MIG instance, the saved GPU task is restored based on the device memory content, CUDA context and task running status saved by the CUDA Checkpoint. Collect the actual number of affected GPU tasks, the actual CUDA checkpoint status, and the actual number of configuration transitions corresponding to each actual execution state transition edge in the lowest cost configuration transition path; The updated number of affected GPU tasks is obtained by performing an exponentially weighted moving average of the actual number of affected GPU tasks and the number of affected GPU tasks recorded in the corresponding state transition edge; the updated number of affected CUDA checkpoint states is obtained by performing an exponentially weighted moving average of the actual number of CUDA checkpoint states and the number of CUDA checkpoint states recorded in the corresponding state transition edge; the updated number of configuration transitions is obtained by performing an exponentially weighted moving average of the actual number of configuration transitions and the number of configuration transitions recorded in the corresponding state transition edge. Based on the updated number of affected GPU tasks, the updated CUDA checkpoint state count, and the updated number of configuration transitions, normalize and weight the data again, and update the state transition cost of the corresponding state transition edge. When a state transition edge fails to delete a MIG instance, fails to create a MIG instance, fails to reassemble a MIG instance, or fails to save a CUDA Checkpoint, the failed state transition edge will be marked as an abnormal edge, and the abnormal edge will be prohibited from participating in the search for the lowest cost configuration transition path until the physical slice occupancy relationship of the MIG that caused the failure changes. When a new downstream graphics card task is formed, the lowest cost configuration conversion path is searched again based on the updated state transition cost.
[0016] The beneficial effects of this invention are: This invention determines the resource requirements of downstream GPU tasks based on GPU task dependency chains and further maps the target MIG instance specification to specific physical slice locations. Candidate instance layouts are generated based on slice coverage conflicts between the target physical slice and the current MIG instance. Simultaneously, using the current MIG instance configuration and candidate instance layouts as state nodes, MIG instance deletion, creation, and reorganization are treated as state transition edges. The number of affected GPU tasks, CUDA checkpoint state count, and configuration transition count are all included in the state transition cost to search for the lowest-cost configuration transition path. This solves the problem that existing MIG scheduling mainly focuses on specification combination and struggles to simultaneously consider physical slice conflicts and running task states, reducing task disturbances and configuration transition overhead during resource refactoring.
[0017] This invention further identifies incomplete GPU tasks occupying conflicting physical slices based on the lowest-cost configuration conversion path. Only CUDA Checkpoints are performed on the relevant tasks to save and release the corresponding MIG instances, forming the target MIG instance on the conflicting physical slice. Simultaneously, MIG instances without slice overlay conflicts continue to run. After the downstream GPU tasks are completed, the state conversion edges are updated based on the actual configuration conversion results. This mechanism avoids interruptions to irrelevant tasks due to overall reconstruction and allows subsequent scheduling to continuously correct path selection based on actual execution results, thereby improving the continuous resource supply capability, scheduling efficiency, and GPU resource utilization in a shared GPU environment. Attached Figure Description
[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a method for scheduling graphics card task resources based on a microservice architecture proposed in this invention; Figure 2This is a schematic diagram of the MIG instance layout in a microservice-based graphics card task resource scheduling method proposed in this invention. Figure 3 This is a data flow diagram for configuration transformation in a microservice-based graphics card task resource scheduling method proposed in this invention. Detailed Implementation
[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0020] refer to Figures 1-3 A method for scheduling GPU task resources based on a microservice architecture includes the following steps: Step 1: Collect microservice call information and GPU task execution information, and construct GPU task dependency chains based on request identifiers and data dependencies; Step 2: Determine the target MIG instance specifications based on the execution status of the preceding graphics card tasks and the memory and computing resource requirements of the downstream graphics card tasks; Step 3: Obtain the physical slice occupancy relationship of the physical graphics card's MIG, map the target MIG instance specification to the physical slice position, and generate candidate instance layouts based on slice coverage conflicts with the current MIG instance; Step 4: Using the current MIG instance configuration and candidate instance layout as state nodes, and MIG instance deletion, creation, and reorganization as state transition edges, search for the lowest cost configuration transition path based on the number of affected GPU tasks, CUDA checkpoint state quantity, and configuration transition count. Step 5: When the preceding GPU task is completed and the downstream GPU task is ready to execute, perform a CUDA Checkpoint to save and release the corresponding MIG instance for the incomplete GPU task occupying the conflicting physical slice in the lowest cost configuration conversion path; Step 6: Reconstruct conflicting physical slices according to the lowest cost configuration conversion path, form target MIG instances and assign them to downstream GPU tasks, while MIG instances that have not experienced slice overwrite conflicts continue to run; Step 7: After the downstream graphics card task releases the target MIG instance, it restores the saved graphics card task and updates the state transition edge according to the actual configuration transition result.
[0021] A method for scheduling GPU task resources based on a microservice architecture includes the following steps: Step 1: Collect microservice call information and GPU task execution information, and construct GPU task dependency chains based on request identifiers and data dependencies; Step 2: Determine the target MIG instance specifications based on the execution status of the preceding graphics card tasks and the memory and computing resource requirements of the downstream graphics card tasks; Step 3: Obtain the physical slice occupancy relationship of the physical graphics card's MIG, map the target MIG instance specification to the physical slice position, and generate candidate instance layouts based on slice coverage conflicts with the current MIG instance; Step 4: Using the current MIG instance configuration and candidate instance layout as state nodes, and MIG instance deletion, creation, and reorganization as state transition edges, search for the lowest cost configuration transition path based on the number of affected GPU tasks, CUDA checkpoint state quantity, and configuration transition count. Step 5: When the preceding GPU task is completed and the downstream GPU task is ready to execute, perform a CUDA Checkpoint to save and release the corresponding MIG instance for the incomplete GPU task occupying the conflicting physical slice in the lowest cost configuration conversion path; Step 6: Reconstruct conflicting physical slices according to the lowest cost configuration conversion path, form target MIG instances and assign them to downstream GPU tasks, while MIG instances that have not experienced slice overwrite conflicts continue to run; Step 7: After the downstream graphics card task releases the target MIG instance, it restores the saved graphics card task and updates the state transition edge according to the actual configuration transition result.
[0022] In this embodiment, step one specifically includes: Collect microservice call information and GPU task execution information. The microservice call information includes the request identifier, the caller microservice identifier, the callee microservice identifier, the call start time, the call completion time, the input data identifier, and the output data identifier. The GPU task execution information includes the GPU task identifier, the microservice identifier, the request identifier, the task submission time, the task start time, the task completion time, the task execution status, the physical GPU identifier, the MIG instance identifier, and the input data identifier. A unified clock is used in the microservice cluster to record the start time, completion time, task submission time, task start time, and task completion time. Microservice call information and GPU task execution information belonging to the same business request are aggregated according to the request identifier. The microservice call order is determined based on the microservice identifier of the caller and the microservice identifier of the callee. The output data identifier of the previous GPU task is matched with the input data identifier of the next GPU task. When the output data identifier of the previous GPU task is the same as the input data identifier of the next GPU task, a data dependency relationship is established from the previous GPU task to the next GPU task. When there are multiple GPU tasks within the same microservice, data dependencies are established according to the input data identifier and output data identifier of each GPU task. The GPU task is treated as a node, and the data dependency relationship is treated as a directed edge. The GPU task dependency chain is constructed according to the request identifier. Based on the task execution status of each graphics card task, graphics card tasks that are in running and have direct data dependencies are identified as preceding graphics card tasks, and graphics card tasks that directly depend on the output of preceding graphics card tasks and whose task execution status is not started are identified as downstream graphics card tasks. Write the preceding GPU task, the downstream GPU task, and the data dependencies between the preceding and downstream GPU tasks into the current scheduling record, as input for step two to determine the memory and computing resource requirements of the downstream GPU task.
[0023] In this embodiment, step two specifically includes: Based on the microservice type, model version, and input data size of the downstream GPU task, filter historical GPU tasks with the same microservice type and model version from the historical GPU tasks. Calculate the deviation between the input data size of each historical graphics card task and the input data size of the downstream graphics card task, and identify historical graphics card tasks whose deviation does not exceed 10% of the input data size of the downstream graphics card task as the same type of historical graphics card task. Historical GPU tasks of the same type are arranged in ascending order of peak video memory usage and computing resource usage. When there are more than 20 such historical GPU tasks, the peak video memory usage at the 95th percentile is determined as the video memory requirement, and the computing resource usage at the 95th percentile is determined as the computing resource requirement. When there are fewer than 20 such historical GPU tasks, the maximum peak video memory usage is determined as the video memory requirement, and the maximum computing resource usage is determined as the computing resource requirement. When the preceding graphics card task has generated the actual input data for the downstream graphics card task, the deviation between the actual input data size and the original input data size is calculated. If the deviation exceeds 10% of the original input data size, similar historical graphics card tasks are re-selected based on the actual input data size, and the video memory requirements and computing resource requirements are re-determined. If the deviation does not exceed 10% of the original input data size, the original video memory requirements and computing resource requirements are maintained. Obtain the MIG instance specifications supported by the physical graphics card, including video memory capacity, computing resource scale, number of MIG physical slices occupied, and legal placement location; Filter MIG instance specifications with video memory capacity not less than video memory requirements and computing resource scale not less than computing resource requirements, and sort them in order of the number of MIG physical slices occupied from least to most, and select the MIG instance specification with the fewest MIG physical slices occupied as the target MIG instance specification. When there are two or more MIG instance specifications that occupy the same number of MIG physical slices, the video memory capacity of each MIG instance specification is reduced by the video memory requirement to obtain the video memory margin, and the computing resource scale of each MIG instance specification is reduced by the computing resource requirement to obtain the computing resource margin. Min-Max normalization is performed on the video memory margin and computing resource margin respectively, transforming them to the range of 0 to 1, and they are weighted with the same weight. The MIG instance specification with the smallest weighted result is selected as the target MIG instance specification. When the maximum and minimum values of the same normalization index are the same, the corresponding normalization result is set to 0. Bind the target MIG instance specifications, the number of MIG physical slices occupied, and the legal placement location to the downstream GPU task.
[0024] In this embodiment, step three specifically includes: Obtain the current MIG instance configuration, which includes the instance specifications, instance starting position, number of MIG physical slices occupied, and current GPU tasks of each current MIG instance; A physical slice occupancy table is established according to the MIG physical slice arrangement order. The corresponding current MIG instance is recorded in the MIG physical slice position occupied by the current MIG instance, and the MIG physical slice position not occupied by the current MIG instance is recorded as free. Based on the instance start position and the number of MIG physical slices occupied by each current MIG instance, the corresponding number of MIG physical slices are continuously determined starting from the instance start position to form the current physical slice set corresponding to each current MIG instance, and the MIG physical slice occupancy relationship is formed by each current physical slice set. Based on each legal placement location of the target MIG instance specification, determine the corresponding target physical slice set; for each legal placement location, starting from the corresponding instance start position, continuously select MIG physical slices with the same number of MIG physical slices occupied by the target MIG instance specification to form the target physical slice set; The target physical slice set is compared with each current physical slice set one by one. When the target physical slice set and the current physical slice set contain at least one identical MIG physical slice, the corresponding current MIG instance is determined to be a conflicting MIG instance; when the target physical slice set and the current physical slice set do not contain any identical MIG physical slices, the corresponding current MIG instance is determined to be a non-conflicting MIG instance. For each legal placement location, conflicting MIG instances are collected separately. The current MIG instance that occupies at least one target physical slice is added to the corresponding conflicting MIG instance set, and the correspondence between the target physical slice set, conflicting MIG instances, and the graphics card tasks currently carried by the conflicting MIG instances is established. For each legal placement location, remove conflicting MIG instances from the current MIG instance configuration, keep the instance specifications, instance starting positions, and MIG physical slice occupancy relationships of non-conflicting MIG instances unchanged, and add the target MIG instance to the target physical slice set to form a candidate instance layout for the corresponding legal placement location; after the conflicting MIG instance is released, the MIG physical slices not occupied by the target MIG instance remain idle; The candidate instance layout is validated for legality, including whether each MIG instance specification is supported by the physical graphics card, whether the starting position of each instance is a legal placement position for the corresponding MIG instance specification, and whether the MIG physical slices occupied by different MIG instances overlap; if any validation fails, the corresponding candidate instance layout is deleted. For different legal placement locations, candidate instance layouts that have passed the legality check are retained, so that different legal placement locations of the same target MIG instance specification correspond to different sets of conflicting MIG instances and sets of graphics card tasks currently carried by the conflicting MIG instances.
[0025] In this embodiment, step four specifically includes: A configuration state diagram is constructed using the current MIG instance configuration and candidate instance layout as state nodes. The current MIG instance configuration is used as the initial state node, each candidate instance layout is used as the target state node, and intermediate state nodes are established based on the legal MIG instance configurations formed during the transformation from the current MIG instance configuration to each candidate instance layout. Each status node records the MIG instance specifications, instance start position, and MIG physical slice occupancy relationship according to the MIG physical slice arrangement order; when two status nodes record the same MIG instance specifications, instance start position, and MIG physical slice occupancy relationship, the two status nodes are merged into the same status node. When two state nodes can complete the transition through deletion, creation, or reorganization of MIG instances, a state transition edge is established between the two state nodes; MIG instance deletion means releasing the MIG physical slice occupied by the current MIG instance; MIG instance creation means creating a MIG instance on a continuous free MIG physical slice that meets the requirements for a legal placement location; MIG instance reorganization means releasing one or more current MIG instances and then recreating a MIG instance according to another MIG instance specification using the released MIG physical slice. For the state transition edge corresponding to the deletion or reorganization of MIG instances, the current MIG instances that need to be released are determined based on the changes in the MIG physical slices corresponding to the state transition edge, and the number of unfinished GPU tasks in the current MIG instances that need to be released is counted to obtain the number of affected GPU tasks; for MIG instances that do not release the current MIG instances, a state transition edge is created, and the number of affected GPU tasks is set to 0. For each affected GPU task, determine the amount of data corresponding to the device memory content, CUDA context, and task running state that the corresponding CUDA process needs to save. Sum the amounts of data corresponding to the device memory content, CUDA context, and task running state to obtain the amount of data to be saved for the corresponding affected GPU task. Add up the amounts of data to be saved for each affected GPU task involved in the same state transition edge to obtain the CUDA checkpoint state amount corresponding to the state transition edge. The number of configuration transitions is determined based on the number of MIG instance deletion operations and creation operations actually performed on the state transition edge. For MIG instance reorganization, the number of deletion operations and creation operations actually performed to complete the MIG instance reorganization is included in the configuration transition count, and MIG instance reorganization is not counted as an additional configuration transition count. Min-Max normalization is performed on the number of affected GPU tasks, CUDA checkpoint state quantity, and configuration transition count for each state transition edge in the configuration state diagram. When the maximum and minimum values of the same metric are the same across all state transition edges, the normalization result of the corresponding metric is set to 0. Multiply the normalized number of affected GPU tasks by 0.4, multiply the normalized CUDA checkpoint state by 0.4, multiply the normalized number of configuration transitions by 0.2, and add the three products together to get the state transition cost. Using the state transition cost as the edge weight of the state transition edge, Dijkstra's shortest path algorithm is used to search for and configure transition paths from the initial state node to each target state node; Set the cumulative state transition cost of the initial state node to 0, and sequentially select the state node with the smallest cumulative state transition cost that has not yet been expanded. Calculate the cumulative state transition cost to reach each adjacent state node through the current state node. When the newly calculated cumulative state transition cost is less than the cumulative state transition cost already recorded by the adjacent state node, update the adjacent state node with the newly calculated cumulative state transition cost, and record the current state node as the predecessor state node of the adjacent state node. After obtaining the configuration transition paths from the initial state node to each target state node, the state transition costs of all state transition edges in each configuration transition path are accumulated, and the configuration transition path with the minimum accumulated state transition cost is determined as the lowest cost configuration transition path. When there are two or more configuration transition paths with the same cumulative state transition cost, the first screening is performed according to the total number of CUDA checkpoint state quantities from smallest to largest; if there are still two or more configuration transition paths after the first screening, the second screening is performed according to the total number of affected GPU tasks from smallest to largest; if there are still two or more configuration transition paths after the second screening, the third screening is performed according to the number of configuration transitions from smallest to largest. Based on the lowest-cost configuration conversion path, determine the physical slice position of the target MIG instance in the corresponding candidate instance layout, the current MIG instance that needs to be released, and the incomplete GPU tasks that need to be saved by executing CUDA Checkpoint.
[0026] In this embodiment, step five specifically includes: Continuously acquire the execution status of the preceding graphics card task; when the execution status of the preceding graphics card task changes to "execution completed", check whether the input data required by the downstream graphics card task has been written; when the input data required by the downstream graphics card task has been written, determine that the downstream graphics card task has the conditions for execution. Based on the lowest cost configuration conversion path, identify the MIG instances that occupy conflicting physical slices and need to be released, and remove the completed graphics card tasks from the graphics card tasks currently carried by the MIG instances that occupy conflicting physical slices and need to be released, and identify the remaining unfinished graphics card tasks as graphics card tasks that need to be saved by CUDA Checkpoint. For GPU tasks that require CUDA Checkpoint saving, stop the corresponding CUDA process from submitting new GPU work and wait for the already submitted GPU work to be executed to a state where saving is allowed; Once the corresponding CUDA process enters the state where saving is allowed, CUDA Checkpoint saving is performed, saving the device's video memory content, CUDA context and task running state, and recording the graphics card task identifier, CUDA process identifier, original MIG instance identifier, original MIG instance specifications, original instance starting position, original MIG physical slice occupancy relationship and actual CUDA checkpoint state quantity. Once all incomplete GPU tasks in the same MIG instance that needs to be released have completed the CUDA Checkpoint save, the runtime binding between all saved GPU tasks and the MIG instance that needs to be released is released, and the MIG instance that needs to be released is released. If any GPU task that requires CUDA Checkpoint saving fails to save, the MIG instance carrying the failed GPU task will not be released, and subsequent state transition edges in the lowest-cost configuration transition path that depend on the release of the MIG instance carrying the failed GPU task will be stopped. MIG instances that have not experienced slice overwrite conflicts do not perform CUDA Checkpoint saving, do not unbind GPU tasks, and remain running.
[0027] In this embodiment, step six specifically includes: Execute the MIG instance deletion, creation, and reorganization sequentially according to the lowest-cost configuration conversion path; For MIG instances that occupy conflicting physical slices and need adjustment, release the MIG instance after confirming that all incomplete GPU tasks have been completed and saved with CUDA Checkpoint, so that the corresponding MIG physical slice is no longer occupied. After each MIG instance release or reorganization is completed, the MIG physical slice occupancy relationship is updated, and the updated MIG physical slice occupancy relationship is compared item by item with the MIG physical slice occupancy relationship corresponding to the next state node in the lowest cost configuration conversion path. If the MIG physical slice occupancy relationships are consistent, the next configuration operation is executed; if the MIG physical slice occupancy relationships are inconsistent, the subsequent configuration conversion is stopped. If the MIG physical slices required to form the target MIG instance are still occupied by the current MIG instance, continue to execute the next configuration operation in the lowest cost configuration conversion path; if none of the MIG physical slices required to form the target MIG instance are occupied by the current MIG instance, stop releasing other MIG instances. Create a target MIG instance based on the physical slice location and target MIG instance specifications determined by the lowest-cost configuration conversion path; After the target MIG instance is created, obtain the instance specifications, instance starting position, and MIG physical slice occupancy relationship of the target MIG instance, and verify them item by item with the target MIG instance specifications, target MIG instance physical slice position, and MIG physical slice occupancy relationship in the target status node corresponding to the lowest cost configuration conversion path; if all are consistent, set the target MIG instance to an allocatable state; if any item is inconsistent, delete the target MIG instance created this time and stop subsequent configuration conversion. Assign the target MIG instance to the downstream graphics card task, load the input data required by the downstream graphics card task, and start the downstream graphics card task. MIG instances that have not experienced slice overwrite conflicts and have not entered the lowest cost configuration conversion path maintain their original instance specifications, instance starting position, and MIG physical slice occupancy relationship. The GPU tasks carried by MIG instances that have not experienced slice overwrite conflicts and have not entered the lowest cost configuration conversion path continue to run. After the downstream graphics card task is completed, the binding between the downstream graphics card task and the target MIG instance is released and the target MIG instance is released.
[0028] In this embodiment, step seven specifically includes: Before performing the CUDA Checkpoint save, save the original MIG instance configuration corresponding to the MIG instance to be released. The original MIG instance configuration includes the instance specifications, instance starting position, MIG physical slice occupancy relationship and corresponding saved GPU tasks. After the downstream graphics card task releases the target MIG instance, it restores the released MIG instance according to the original MIG instance configuration, recreates the MIG instance according to the original instance start position and original MIG instance specifications, and verifies the instance specifications, instance start position and MIG physical slice occupancy relationship of the recreated MIG instance with the original MIG instance configuration item by item. After verification, the saved graphics card task is bound to the newly created MIG instance. CUDA Restore is performed based on the device memory content, CUDA context and task running status saved by CUDACheckpoint. After CUDA Restore is completed, the corresponding CUDA process is restored to run, so that the saved graphics card task can continue to run. Collect the actual number of affected GPU tasks, the actual CUDA checkpoint status, and the actual number of configuration transitions corresponding to each actual execution state transition edge in the lowest cost configuration transition path; The updated number of affected GPU tasks is obtained by performing an exponentially weighted moving average of the actual number of affected GPU tasks and the number of affected GPU tasks recorded in the corresponding state transition edge; the updated number of affected CUDA checkpoint states is obtained by performing an exponentially weighted moving average of the actual number of CUDA checkpoint states and the number of CUDA checkpoint states recorded in the corresponding state transition edge; the updated number of configuration transitions is obtained by performing an exponentially weighted moving average of the actual number of configuration transitions and the number of configuration transitions recorded in the corresponding state transition edge. In this embodiment, the exponentially weighted moving average uses the actual parameter weight of the current round (0.2) and the original recorded parameter weight (0.8); for the first execution and where there is no state transition edge with the original recorded parameter, the actual parameter of the current round is directly used as the recorded parameter of the corresponding state transition edge. Based on the updated number of affected GPU tasks, the updated CUDA checkpoint state quantity, and the updated number of configuration transitions, normalize again according to the Min-Max normalization method in step four, and recalculate the state transition cost with a weight of 0.4 for the number of affected GPU tasks, a weight of 0.4 for the CUDA checkpoint state quantity, and a weight of 0.2 for the number of configuration transitions. Write the recalculated state transition cost into the corresponding state transition edge. When a state transition edge fails to delete a MIG instance, fails to create a MIG instance, fails to reassemble a MIG instance, or fails to save a CUDA Checkpoint, the failed state transition edge will be marked as an abnormal edge, and the abnormal edge will be prohibited from participating in the search for the lowest cost configuration transition path until the physical slice occupancy relationship of the MIG that caused the failure changes. After detecting a change in the MIG physical slice occupancy relationship, the system re-evaluates whether the deletion, creation, or reorganization of the MIG instance corresponding to the abnormal edge meets the execution conditions based on the current MIG instance configuration. If the execution conditions are met, the abnormal edge is removed from the mark, and the state transition cost is re-determined based on the number of affected GPU tasks, CUDA checkpoint state quantity, and configuration transition count. When a new downstream graphics card task is formed, the candidate instance layout is regenerated based on the updated MIG physical slice occupancy relationship, and the lowest cost configuration conversion path is re-searched based on the updated state transition cost.
[0029] Example 1: To verify the feasibility of this invention in practice, it was applied to a microservice GPU resource scheduling scenario of an artificial intelligence inference platform. The platform deployed four 80GB physical graphics cards supporting MIG mode. Each physical graphics card was divided into seven MIG physical slices. The services included image preprocessing, object detection, feature extraction, and result rearrangement microservices. Graphics card tasks within the same service request were executed consecutively according to data dependencies. The test lasted 2 hours, processing 12,000 service requests. The number of concurrent graphics card tasks per physical graphics card ranged from 4 to 9, the peak memory requirement of downstream graphics card tasks ranged from 8 to 36GB, and the computing resource requirement ranged from 18% to 62%.
[0030] During the test, the deviation of the input data size of historical tasks was set to 10%. When there were no less than 20 similar historical GPU tasks, the 95th percentile position was used to determine the memory and computing resource requirements. The weights of the number of affected GPU tasks, CUDA checkpoint state quantity and configuration transition number were set to 0.4, 0.4 and 0.2 respectively. When updating the state transition, the weight of the actual parameters in this round was set to 0.2 and the weight of the original recorded parameters was set to 0.8.
[0031] Comparison Method A uses a fixed MIG instance configuration, pre-partitions instances based on historical average resource requirements, and does not search for new MIG instance layouts during operation. Its scheduling logic is simple, with a single scheduling decision calculation time of only 2.1ms in testing. Comparison Method B uses global dynamic reconstruction, releasing MIG instances on the target physical GPU and reconfiguring resources when resources are insufficient. It has a high degree of freedom in adjusting physical slices, achieving a configuration success rate of 98.9% in large-scale MIG instance creation requests. Comparison Method C generates candidate instance layouts based on the target MIG instance specifications and legal placement locations, but directly selects the first candidate instance layout that meets the configuration conditions without searching for the lowest-cost configuration transition path. Its single layout selection calculation time is 6.8ms. This invention performs complete slice coverage conflict judgment, state transition cost calculation, and lowest-cost configuration transition path search. The single scheduling decision calculation time is 12.6ms, but this calculation is completed on the CPU side and does not occupy the computing resources of the MIG instance to be scheduled.
[0032] Table 1. Comprehensive Test Results of Different Graphics Card Task Resource Scheduling Methods
[0033] The advantage of method A is that it eliminates the need for candidate instance layout search during runtime, resulting in the lowest computational overhead for scheduling decisions. However, the fixed MIG instance specifications cannot consistently match the changing memory and computing resource requirements of different microservice stages. When a MIG instance that meets the requirements of a downstream GPU task is already occupied, the downstream GPU task can only wait for existing tasks to release resources, resulting in an average waiting time of 236ms and an average GPU utilization rate of only 74.5%.
[0034] Method B, by globally releasing and reconfiguring the MIG instance, can obtain a larger space for physical slice adjustment. Therefore, the configuration success rate of large-scale MIG instances is higher, and the average effective GPU utilization reaches 89.1%. However, the global reconstruction does not limit the adjustment range based on the slice coverage conflict between the target physical slice and the current MIG instance. A single reconstruction affects an average of 6.6 GPU tasks, requires saving 16.9 GiB of CUDA checkpoint state, and the average configuration transition time reaches 279ms.
[0035] Comparison method C does not perform the lowest-cost configuration transformation path search, therefore its layout decision time is shorter than that of this invention, making it suitable for scenarios with a small number of GPU tasks and simple MIG physical slice occupancy relationships. However, directly selecting the first executable candidate instance for layout cannot distinguish the impact of different legal placement positions on running tasks, still affecting an average of 3.2 GPU tasks and generating 8.2 GiB of CUDA checkpoint state.
[0036] Although this invention adds CPU-side computation processes for slice coverage conflict analysis and minimum-cost configuration transition path search, it achieves optimal results for all core performance metrics by incorporating conflicting MIG instances corresponding to different legal placement locations, the number of affected GPU tasks, CUDA checkpoint state quantities, and configuration transition counts into a unified decision. Compared to comparative method C, this invention reduces the average downstream task latency from 147ms to 82ms, a reduction of 44.2%; the number of affected GPU tasks from 3.2 to 1.4, a reduction of 56.3%; the CUDA checkpoint state quantity from 8.2 GiB to 3.6 GiB, a reduction of 56.1%; the configuration transition time from 136ms to 71ms, a reduction of 47.8%; and the average effective GPU utilization from 87.0% to 92.6%.
[0037] Therefore, each comparative method has its own advantages in terms of fixed configuration simplicity, global resource adjustment capability, or candidate layout selection speed. However, this invention achieves synchronous optimization of five core business indicators—task waiting, running task disturbance, state saving overhead, configuration conversion time, and GPU resource utilization—in scenarios where the resource requirements of microservice GPU tasks are constantly changing, through a minimum cost configuration conversion mechanism driven by physical slice conflicts.
[0038] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for scheduling GPU task resources based on a microservice architecture, characterized in that, Includes the following steps: Step 1: Collect microservice call information and GPU task execution information, and construct GPU task dependency chains based on request identifiers and data dependencies; Step 2: Determine the target MIG instance specifications based on the execution status of the preceding graphics card tasks and the memory and computing resource requirements of the downstream graphics card tasks; Step 3: Obtain the physical slice occupancy relationship of the physical graphics card's MIG, map the target MIG instance specification to the physical slice position, and generate candidate instance layouts based on slice coverage conflicts with the current MIG instance; Step 4: Using the current MIG instance configuration and candidate instance layout as state nodes, and MIG instance deletion, creation, and reorganization as state transition edges, search for the lowest cost configuration transition path based on the number of affected GPU tasks, CUDA checkpoint state quantity, and configuration transition count. Step 5: When the preceding GPU task is completed and the downstream GPU task is ready to execute, perform a CUDA Checkpoint to save and release the corresponding MIG instance for the incomplete GPU task occupying the conflicting physical slice in the lowest cost configuration conversion path; Step 6: Reconstruct conflicting physical slices according to the lowest cost configuration conversion path, form target MIG instances and assign them to downstream GPU tasks, while MIG instances that have not experienced slice overwrite conflicts continue to run; Step 7: After the downstream graphics card task releases the target MIG instance, it restores the saved graphics card task and updates the state transition edge according to the actual configuration transition result.
2. The method for scheduling GPU task resources based on a microservice architecture according to claim 1, characterized in that, Step one specifically includes: Collect microservice call information and GPU task execution information, and group microservice call information and GPU task execution information belonging to the same business request according to the request identifier; The microservice call order is determined based on the microservice identifier of the caller and the microservice identifier of the callee. The output data identifier of the previous graphics card task is matched with the input data identifier of the next graphics card task. When the output data identifier of the previous graphics card task is the same as the input data identifier of the next graphics card task, a data dependency relationship from the previous graphics card task to the next graphics card task is established. By treating GPU tasks as nodes and data dependencies as directed edges, a GPU task dependency chain is constructed. GPU tasks that are in running and have direct data dependencies are identified as preceding GPU tasks. GPU tasks that directly depend on the output of preceding GPU tasks and are in an unstarted state are identified as downstream GPU tasks.
3. The method for scheduling GPU task resources based on a microservice architecture according to claim 1, characterized in that, Step two specifically includes: Based on the microservice type, model version, and input data scale of the downstream GPU task, filter similar historical GPU tasks, and determine the GPU memory and computing resource requirements of the downstream GPU task based on the peak GPU memory usage and computing resource usage level of similar historical GPU tasks. Filter MIG instance specifications with video memory capacity not less than video memory requirements and computing resource scale not less than computing resource requirements, and sort them in order of the number of MIG physical slices occupied from least to most, and select the MIG instance specification with the fewest MIG physical slices occupied as the target MIG instance specification. When there are multiple MIG instance specifications that occupy the same number of MIG physical slices, the video memory capacity of each MIG instance specification is reduced by the video memory requirement to obtain the video memory margin, and the computing resource scale of each MIG instance specification is reduced by the computing resource requirement to obtain the computing resource margin. The video memory margin and computing resource margin are normalized and weighted, and the MIG instance specification with the smallest weighted result is selected as the target MIG instance specification.
4. The method for scheduling GPU task resources based on a microservice architecture according to claim 1, characterized in that, Step three specifically includes: Based on the instance start position and the number of MIG physical slices occupied by each current MIG instance, the current physical slice set corresponding to each current MIG instance is determined, forming the MIG physical slice occupancy relationship; The corresponding set of target physical slices is determined based on each legal placement location of the target MIG instance specification; The target physical slice set is compared with each current physical slice set one by one. When the target physical slice set and the current physical slice set contain at least one MIG physical slice with the same MIG, the corresponding current MIG instance is determined to be a conflicting MIG instance. When the target physical slice set and the current physical slice set do not contain any MIG physical slices with the same MIG, the corresponding current MIG instance is determined to be a non-conflicting MIG instance. For each legal placement location, conflicting MIG instances are collected separately, and the target MIG instance, conflicting MIG instances, and non-conflicting MIG instances that maintain the original MIG physical slice occupancy relationship are combined to generate a candidate instance layout.
5. The method for scheduling graphics card task resources based on a microservice architecture according to claim 4, characterized in that, The process involves collecting conflicting MIG instances for each legal placement location, and combining the target MIG instance, conflicting MIG instances, and non-conflicting MIG instances that maintain the original MIG physical slice occupancy relationship to generate a candidate instance layout. Specifically, this includes: A physical slice occupancy table is established according to the MIG physical slice arrangement order. The corresponding current MIG instance is recorded in the MIG physical slice position occupied by the current MIG instance, and the MIG physical slice position not occupied by the current MIG instance is recorded as free. For each legal placement location of the target MIG instance specification, starting from the corresponding instance start position, continuously select MIG physical slices with the same number of MIG physical slices occupied by the target MIG instance specification to form a target physical slice set; Based on the physical slice occupancy table, determine the current MIG instance occupying the target physical slice, add the current MIG instance occupying at least one target physical slice to the corresponding conflicting MIG instance set, and establish the correspondence between the target physical slice set, the conflicting MIG instance, and the graphics card task currently being carried by the conflicting MIG instance. Remove the corresponding conflicting MIG instance from the current MIG instance configuration, keep the instance specifications, instance starting position and MIG physical slice occupancy relationship of the non-conflicting MIG instance unchanged, and add the target MIG instance to the target physical slice set to form a candidate instance layout for the corresponding legal placement position; For different legal placement locations, corresponding candidate instance layouts are retained, so that different legal placement locations of the same target MIG instance specification correspond to different sets of conflicting MIG instances and sets of graphics card tasks currently carried by the conflicting MIG instances.
6. The method for scheduling GPU task resources based on a microservice architecture according to claim 1, characterized in that, Step four specifically includes: The current MIG instance configuration is used as the initial state node, each candidate instance layout is used as the target state node, and intermediate state nodes are established based on the legal MIG instance configurations formed during the transformation from the current MIG instance configuration to each candidate instance layout. When two state nodes can complete the transition through deletion, creation, or reorganization of MIG instances, a state transition edge is established between the two state nodes; The number of affected GPU tasks is determined based on the current MIG instance that needs to be released by the state transition edge. The CUDA checkpoint state quantity is determined based on the pending state of the CUDA process corresponding to the affected GPU task. The number of configuration transitions is determined based on the number of MIG instance deletion operations and creation operations actually performed by the state transition edge. Among them, the number of MIG instance reorganization operations is counted as the number of deletion operations and creation operations actually performed to complete the MIG instance reorganization. The number of affected GPU tasks, CUDA checkpoint state quantities, and configuration transition counts are normalized and weighted to obtain the state transition cost. The state transition cost is then used as the edge weight of the state transition edge to search for the lowest cost configuration transition path.
7. The method for scheduling GPU task resources based on a microservice architecture according to claim 6, characterized in that, The determination of the lowest-cost configuration conversion path specifically includes: For state transition edges involving the deletion or reorganization of MIG instances, the current MIG instances that need to be released are determined based on the changes in the MIG physical slices corresponding to the state transition edges, and the number of incomplete GPU tasks in the current MIG instances that need to be released is counted to obtain the number of affected GPU tasks. The amount of data to be saved corresponding to the device memory content, CUDA context, and task running status of each affected GPU task is summarized to obtain the CUDA checkpoint state quantity. The number of affected GPU tasks, CUDA checkpoint state quantity, and configuration transition count are all Min-Max normalized separately, and then weighted and summed according to the weight of the number of affected GPU tasks, the weight of CUDA checkpoint state quantity, and the weight of the configuration transition count to obtain the state transition cost. Dijkstra's shortest path algorithm is used to accumulate the state transition costs from the initial state node to each target state node, and the configuration transition path with the minimum accumulated state transition cost is determined as the lowest cost configuration transition path. When the cumulative state transition cost of two or more configuration transition paths is the same, the first screening is performed according to the total number of CUDA checkpoint state quantities from smallest to largest. If there are still two or more configuration transition paths after the first screening, the second screening is performed according to the total number of affected GPU tasks from smallest to largest. If there are still two or more configuration transition paths after the second screening, the third screening is performed according to the number of configuration transitions from smallest to largest. Based on the lowest-cost configuration conversion path, determine the physical slice position of the target MIG instance in the corresponding candidate instance layout, the current MIG instance that needs to be released, and the incomplete GPU tasks that need to be saved by executing CUDA Checkpoint.
8. The method for scheduling graphics card task resources based on a microservice architecture according to claim 1, characterized in that, Step five specifically includes: When the preceding graphics card task has finished executing and the input data required by the downstream graphics card task has been written, it is determined that the downstream graphics card task is ready to execute. Based on the lowest cost configuration conversion path, identify the MIG instances that occupy conflicting physical slices and need to be released, and remove the completed graphics card tasks from the graphics card tasks currently carried by the MIG instances that occupy conflicting physical slices and need to be released, and identify the remaining unfinished graphics card tasks as graphics card tasks that need to be saved by CUDA Checkpoint. Stop the CUDA process corresponding to the GPU task that needs to be saved by CUDA Checkpoint from submitting new GPU work, and save the device memory contents, CUDA context and task running status of the corresponding CUDA process; Once all incomplete GPU tasks in the same MIG instance that needs to be released have completed the CUDA Checkpoint save, the runtime binding between all saved GPU tasks and the MIG instance that needs to be released is released, and the MIG instance that needs to be released is released. MIG instances that have not experienced slice overwrite conflicts do not perform CUDA Checkpoint saving, do not unbind GPU tasks, and remain running.
9. The method for scheduling graphics card task resources based on a microservice architecture according to claim 1, characterized in that, Step six specifically includes: Release the MIG instances that occupy conflicting physical slices and need adjustment in sequence according to the lowest cost configuration conversion path, so that the MIG physical slices corresponding to the MIG instances that occupy conflicting physical slices and need adjustment are released from occupation. Each time a MIG instance is released or reorganized, the MIG physical slice occupancy relationship is updated; if the MIG physical slices required to form the target MIG instance are still occupied by the current MIG instance, the next configuration operation in the lowest cost configuration conversion path is executed; if none of the MIG physical slices required to form the target MIG instance are occupied by the current MIG instance, the release of other MIG instances is stopped. Create a target MIG instance based on the physical slice location and target MIG instance specifications determined by the lowest cost configuration conversion path, and assign the target MIG instance to the downstream GPU task; MIG instances that have not experienced slice overwrite conflicts and have not entered the lowest-cost configuration conversion path maintain their original instance specifications, instance starting position, and MIG physical slice occupancy relationships. The GPU tasks carried by MIG instances that have not experienced slice overwrite conflicts and have not entered the lowest-cost configuration conversion path continue to run.
10. The method for scheduling GPU task resources based on a microservice architecture according to claim 1, characterized in that, Step seven specifically includes: After the downstream GPU task releases the target MIG instance, the saved GPU task is restored based on the device memory content, CUDA context and task running status saved by the CUDA Checkpoint. Collect the actual number of affected GPU tasks, the actual CUDA checkpoint status, and the actual number of configuration transitions corresponding to each actual execution state transition edge in the lowest cost configuration transition path; The updated number of affected GPU tasks is obtained by performing an exponentially weighted moving average of the actual number of affected GPU tasks and the number of affected GPU tasks recorded in the corresponding state transition edge; the updated number of affected CUDA checkpoint states is obtained by performing an exponentially weighted moving average of the actual number of CUDA checkpoint states and the number of CUDA checkpoint states recorded in the corresponding state transition edge; the updated number of configuration transitions is obtained by performing an exponentially weighted moving average of the actual number of configuration transitions and the number of configuration transitions recorded in the corresponding state transition edge. Based on the updated number of affected GPU tasks, the updated CUDA checkpoint state count, and the updated number of configuration transitions, normalize and weight the data again, and update the state transition cost of the corresponding state transition edge. When a state transition edge fails to delete a MIG instance, fails to create a MIG instance, fails to reassemble a MIG instance, or fails to save a CUDACheckpoint, the failed state transition edge will be marked as an abnormal edge, and the abnormal edge will be prohibited from participating in the search for the lowest cost configuration transition path until the physical slice occupancy relationship of the MIG that caused the failure changes. When a new downstream graphics card task is formed, the lowest cost configuration conversion path is searched again based on the updated state transition cost.
Citation Information
Patent Citations
Graphic processor resource allocation method
CN120892216A
Task scheduling method and device and storage medium
CN121455631A
GPU cluster load balancing system based on micro-service architecture
CN121704956A
GPU resource optimization method and system based on dynamic prediction and topological thermodynamic diagram
CN121833233A
