Expert resource allocation method and device in MoE, storage medium and equipment
By monitoring the popularity level of experts in MoE and dynamically adjusting and expanding vGPU resources, the problems of low resource utilization and large fluctuations in inference latency in MoE are solved, achieving more efficient resource allocation and inference throughput.
Patent Information
- Application Number
- CN202510998794.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-21
- Publication Date
- 2025-11-21
AI Technical Summary
In existing technologies, MoE allocates the same GPU resources to all experts, resulting in low resource utilization, large fluctuations in inference latency, and limited concurrency capabilities.
By monitoring the popularity levels of various experts in MoE, the vGPU instance resources of target experts can be dynamically adjusted and expanded, including vertical and horizontal expansion, and vGPU instances can be bound or unbound according to changes in the popularity level of the experts.
It improves resource utilization, provides real-time performance and flexibility, enhances inference throughput, and solves the problem of uneven resource allocation.
Smart Images
Figure CN120994364A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, storage medium and device for allocating resources for experts in MoE. Background Technology
[0002] The Mixture of Experts (MoE) model consists of a gating network and multiple specialized experts (i.e., sub-models). After receiving a token, the gating network assigns it to an expert who is proficient in handling problems in that domain, based on the domain corresponding to the token. The expert then processes the token.
[0003] In related technologies, computer equipment allocates the same Graphics Processing Unit (GPU) resources to all experts, who then use these GPU resources to process tokens. However, some experts are frequently invoked, while the majority of the remaining experts are in a long-term cold start or low-load state, resulting in problems such as low resource utilization, large fluctuations in inference latency, and limited concurrency capabilities. Summary of the Invention
[0004] This application provides a method, apparatus, storage medium, and device for allocating resources to experts in MoE, to solve the problems of low resource utilization, large inference latency fluctuations, and limited concurrency caused by allocating the same GPU resources to all experts. The technical solution is as follows:
[0005] According to a first aspect of this application, a method for allocating resources to experts in a MoE is provided, the method comprising:
[0006] Monitor the popularity level of each expert in MoE, which is calculated based on the resource usage of each expert within a predetermined period;
[0007] When the popularity level of the target expert increases, obtain the resource description information of the first vGPU instance bound to the target expert on the corresponding first GPU node;
[0008] If it is determined from the resource description information that the resources of the first vGPU instance can be adjusted, then on the first GPU node, the resources of the first vGPU instance are expanded according to the changed heat level.
[0009] If it is determined from the resource description information that the resources of the first vGPU instance cannot be adjusted, then a second vGPU instance is bound to the target expert on the first GPU node or the second GPU node, and resources are allocated to the second vGPU instance.
[0010] In one possible implementation, the method further includes: detecting whether there are remaining resources on the graphics card where the first vGPU instance is located based on the resource description information; if there are remaining resources, determining that the resources of the first vGPU instance can be adjusted.
[0011] Expanding the resources of the first vGPU instance according to the changed heat level includes: allocating some or all of the remaining resources to the first vGPU instance according to the changed heat level.
[0012] In one possible implementation, the method further includes: if there are no remaining resources, detecting whether there are other experts in the graphics card with a heat level lower than the target expert; if there are other experts, determining that the resources of the first vGPU instance can be adjusted.
[0013] Expanding the resources of the first vGPU instance according to the changed popularity level includes: allocating a portion of the resources of the third vGPU instance corresponding to the other experts to the first vGPU instance according to the changed popularity level.
[0014] In one possible implementation, the method further includes: if no other experts are available, determining that the resources of the first vGPU instance cannot be adjusted;
[0015] The step of binding a second vGPU instance to the target expert on the first GPU node or the second GPU node includes: detecting whether there is an idle graphics card in the first GPU node; if there is an idle graphics card, binding a second vGPU instance to the target expert on the first GPU node; if there is no idle graphics card, binding a second vGPU instance to the target expert on the second GPU node.
[0016] In one possible implementation, binding a second vGPU instance to the target expert on the second GPU node includes:
[0017] Locate GPU nodes with available GPUs among all GPU nodes;
[0018] If a GPU node is found, the found GPU node is identified as the second GPU node, and a second vGPU instance is bound to the target expert on the second GPU node;
[0019] If at least two GPU nodes are found, then select the GPU node that matches the changed heat level from the at least two GPU nodes, determine the selected GPU node as the second GPU node, and bind the second vGPU instance to the target expert on the second GPU node.
[0020] In one possible implementation, the method further includes:
[0021] When the popularity level of the target expert decreases, the resources of the first vGPU instance are reduced according to the changed popularity level.
[0022] In one possible implementation, the method further includes:
[0023] If the resources required by the first vGPU instance are less than the remaining resources of the target graphics card, then a third vGPU instance is bound to the target expert on the target graphics card, and resources are allocated to the third vGPU instance. The target graphics card is a graphics card located on the first GPU node or the third GPU node and bound to other vGPU instances.
[0024] According to a second aspect of this application, a resource allocation apparatus for experts in MoE is provided, the apparatus comprising:
[0025] The monitoring module is used to monitor the popularity level of each expert in MoE. The popularity level is calculated based on the resource usage of each expert within a predetermined period.
[0026] The acquisition module is used to acquire the resource description information of the first vGPU instance bound to the corresponding first GPU node when the popularity level of the target expert increases.
[0027] The first allocation module is configured to, if it is determined from the resource description information that the resources of the first vGPU instance can be adjusted, expand the resources of the first vGPU instance on the first GPU node according to the changed heat level.
[0028] The second allocation module is configured to, if it is determined from the resource description information that the resources of the first vGPU instance cannot be adjusted, bind a second vGPU instance to the target expert on the first GPU node or the second GPU node, and allocate resources to the second vGPU instance.
[0029] According to a third aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the resource allocation method for experts in MoE as described above.
[0030] According to a fourth aspect of this application, a computer device is provided, the computer device including the resource allocation device for experts in the above-described MoE.
[0031] The beneficial effects of the technical solution provided in this application include at least the following:
[0032] By acquiring the changes in the popularity level of each expert in MoE, when the popularity level increases and the resources of the first vGPU instance corresponding to the target expert are insufficient, the resources of the first vGPU instance can be expanded according to the changed popularity level of the target expert, thus achieving vertical resource expansion; or, a second vGPU instance can be bound to the target expert and resources can be allocated to the second vGPU instance, thus achieving horizontal resource expansion. This achieves the goal of adjusting resources in conjunction with the popularity level of the expert, improving resource utilization, and providing real-time, elastic, and inference throughput capabilities. Attached Figure Description
[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0034] Figure 1 This is a flowchart of a method for allocating expert resources in MoE according to an embodiment of this application;
[0035] Figure 2 This is a flowchart of a resource allocation method for experts in MoE provided in another embodiment of this application;
[0036] Figure 3 This is a structural block diagram of the expert resource allocation device in MoE provided in another embodiment of this application. Detailed Implementation
[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0038] The system in this application includes multiple GPU nodes, each configured with at least one graphics card. The number of graphics cards on different GPU nodes can be the same or different; this application does not limit the number of graphics cards. The models of the graphics cards on different GPU nodes can be the same or different; this application does not limit the model of the graphics cards. For example, if each GPU node is configured with four graphics cards of the same model, then GPU node 0 can be configured with four A100 graphics cards, and GPU node 1 can be configured with four H100 graphics cards.
[0039] When deploying the MoE model, each expert can be deployed on a single GPU on a GPU node. However, allocating a dedicated GPU to less frequently used experts would be wasteful of GPU resources. Therefore, we can virtualize a single GPU into multiple vGPU instances, allocate resources to each vGPU instance, and then bind each expert to one vGPU instance. This way, when multiple vGPU instances are created on a single GPU, the experts bound to these vGPU instances can share the resources of that GPU, improving GPU utilization. These resources can be any GPU resource, including but not limited to VRAM size and computing power quota.
[0040] like Figure 1 The diagram illustrates a flowchart of a method for allocating resources for experts in a Machine of Electronics (MoE) according to an embodiment of this application. This method can be applied to computer devices. The method may include:
[0041] Step 101: Monitor the popularity level of each expert in MoE. This popularity level is calculated based on the resource usage of each expert within a predetermined period.
[0042] After receiving a token, the gating network in MoE assigns it to an expert who is proficient in handling problems in that field, based on the domain corresponding to the token. The expert then uses the resources of the corresponding vGPU instance to process the token.
[0043] During the token processing process by experts, the computer device can statistically analyze the experts' resource usage and use a predetermined algorithm to determine the popularity level of each expert. The more times an expert is accessed and the more resources are used, the higher their popularity; conversely, the fewer times an expert is accessed and the less resources are used, the lower their popularity. Based on business needs, the computer device can categorize expert popularity into multiple levels, such as low, medium, and high. This embodiment does not limit the number of levels or the method of categorization.
[0044] Step 102: When the popularity level of the target expert increases, obtain the resource description information of the first vGPU instance bound to the target expert on the corresponding first GPU node.
[0045] For ease of distinction, in this embodiment, the expert whose popularity level changes is called the target expert, the vGPU instance bound to the target expert is called the first vGPU instance, and the GPU node where the first vGPU instance is located is called the first GPU node.
[0046] When the popularity level of the target expert increases, the resources originally allocated to the first vGPU instance may be insufficient. Therefore, it is necessary to expand the resources of the first vGPU instance to ensure the normal operation of the target expert.
[0047] Specifically, the computer device needs to obtain the resource description information of the first vGPU instance, and then determine the specific expansion method based on the resource description information. For example, when the graphics card corresponding to the first vGPU instance has remaining resources, the remaining resources can be allocated to the first vGPU instance, i.e., step 103 is executed to achieve vertical expansion; when the graphics card corresponding to the first vGPU instance has no remaining resources, a second vGPU instance on another graphics card can be bound to the target expert on the first GPU node, or a second vGPU instance on any graphics card can be bound to the target expert on the second GPU node, and then resources can be allocated to the second vGPU instance, i.e., step 104 is executed to achieve horizontal expansion.
[0048] Step 103: If it is determined from the resource description information that the resources of the first vGPU instance can be adjusted, then on the first GPU node, the resources of the first vGPU instance are expanded according to the changed heat level.
[0049] When allocating resources, computer equipment also needs to obtain the changed heat level and expand the resources accordingly. That is, the higher the changed heat level, the more resources are allocated.
[0050] For example, when the popularity level changes from low to medium, the video memory size is adjusted from 4 GiB to 8 GiB, and the computing power quota is adjusted from 20% to 60%; when the popularity level changes from low to high, the video memory size is adjusted from 4 GiB to 16 GiB, and the computing power quota is adjusted from 20% to 100%.
[0051] Step 104: If it is determined from the resource description information that the resources of the first vGPU instance cannot be adjusted, then bind the second vGPU instance to the target expert on the first GPU node or the second GPU node, and allocate resources to the second vGPU instance.
[0052] The computer device can migrate data from the first vGPU instance to the second vGPU instance, detaching the target expert from the first vGPU instance and providing services to the target expert through the second vGPU instance. Typically, the second vGPU instance has more resources than the first vGPU instance. For example, the first vGPU instance shares a graphics card with other vGPU instances, while the second vGPU instance has its own dedicated graphics card. Alternatively, the computer device can retain the first vGPU instance and create a new second vGPU instance, allowing both instances to provide concurrent services to the target expert.
[0053] It should be noted that this embodiment only uses resource allocation for one MoE as an example. In actual implementation, resource allocation can be performed on multiple MoEs, which will not be elaborated here.
[0054] In summary, the resource allocation method for experts in MoE provided in this application, by acquiring the changes in the popularity level of each expert in MoE, can expand the resources of the first vGPU instance according to the changed popularity level of the target expert when the popularity level increases and the resources of the first vGPU instance corresponding to the target expert are insufficient, thereby achieving vertical resource expansion; or, it can bind the target expert to a second vGPU instance and allocate resources to the second vGPU instance, thereby achieving horizontal resource expansion. This achieves the purpose of adjusting resources in conjunction with the popularity level of the expert, improving resource utilization, and possessing real-time performance, elasticity, and inference throughput capabilities.
[0055] like Figure 2 The diagram illustrates a flowchart of a method for allocating resources to experts in a Machine of Experience (MoE) according to an embodiment of this application. This method can be applied to computer devices. The method may include:
[0056] Step 201: Monitor the popularity level of each expert in MoE. This popularity level is calculated based on the resource usage of each expert within a predetermined period.
[0057] After receiving a token, the gating network in MoE assigns it to an expert who is proficient in handling problems in that field, based on the domain corresponding to the token. The expert then uses the resources of the corresponding vGPU instance to process the token.
[0058] During the token processing process by experts, the computer device can statistically analyze the experts' resource usage and use a predetermined algorithm to determine the popularity level of each expert. The more times an expert is accessed and the more resources are used, the higher their popularity; conversely, the fewer times an expert is accessed and the less resources are used, the lower their popularity. Based on business needs, the computer device can categorize expert popularity into multiple levels, such as low, medium, and high. This embodiment does not limit the number of levels or the method of categorization.
[0059] Step 202: When the popularity level of the target expert increases, obtain the resource description information of the first vGPU instance bound to the target expert on the corresponding first GPU node.
[0060] For ease of distinction, in this embodiment, the expert whose popularity level changes is called the target expert, the vGPU instance bound to the target expert is called the first vGPU instance, and the GPU node where the first vGPU instance is located is called the first GPU node.
[0061] When the popularity level of the target expert increases, the resources originally allocated to the first vGPU instance may be insufficient. Therefore, it is necessary to expand the resources of the first vGPU instance to ensure the normal operation of the target expert.
[0062] Specifically, the computer device needs to obtain the resource description information of the first vGPU instance, and then determine the specific expansion method based on the resource description information. For example, when the graphics card corresponding to the first vGPU instance has remaining resources, the remaining resources can be allocated to the first vGPU instance to achieve vertical scaling; when the graphics card corresponding to the first vGPU instance has no remaining resources, a second vGPU instance on another graphics card can be bound to the target expert on the first GPU node, or a second vGPU instance on any graphics card can be bound to the target expert on the second GPU node, and then resources can be allocated to the second vGPU instance to achieve horizontal scaling.
[0063] Step 203: Detect whether there are any remaining resources on the graphics card where the first vGPU instance is located, based on the resource description information.
[0064] The resource description information describes the resource allocation. Specifically, it describes the memory size and computing power quota allocated to the first vGPU instance; for example, the memory size is 4 GiB and the computing power quota is 20%.
[0065] When the graphics card containing the first vGPU instance contains only the first vGPU instance, check whether the video memory size of the first vGPU instance is equal to the video memory size of the graphics card, and check whether the computing power quota of the first vGPU instance is equal to the computing power quota of the graphics card; if either of them is not equal, it is determined that the graphics card has remaining resources, and step 204 is executed; otherwise, it is determined that the graphics card does not have remaining resources, and step 205 is executed.
[0066] When there are multiple vGPU instances on the graphics card where the first vGPU instance is located, calculate whether the video memory size of all vGPU instances is equal to the video memory size of the graphics card, and check whether the computing power quota of all vGPU instances is equal to the computing power quota of the graphics card; if either of them is not equal, it is determined that the graphics card has remaining resources and step 204 is executed; otherwise, it is determined that the graphics card does not have remaining resources and step 205 is executed.
[0067] Of course, if the graphics card has too few remaining resources, it can also be assumed that the graphics card has no remaining resources, and step 205 can be executed.
[0068] Step 204: If there are remaining resources, determine the resources that can be adjusted for the first vGPU instance, and allocate some or all of the remaining resources to the first vGPU instance according to the changed heat level.
[0069] When allocating resources, computer equipment also needs to obtain the changed heat level and expand the resources accordingly. That is, the higher the changed heat level, the more resources are allocated.
[0070] For example, when the popularity level changes from low to medium, the video memory size is adjusted from 4 GiB to 8 GiB, and the computing power quota is adjusted from 20% to 60%; when the popularity level changes from low to high, the video memory size is adjusted from 4 GiB to 16 GiB, and the computing power quota is adjusted from 20% to 100%.
[0071] Step 205: If there are no remaining resources, check if there are other experts on the graphics card with a lower heat level than the target expert.
[0072] If there is at least one other expert in the graphics card, compare the popularity level of the target expert with that of each other expert; if the popularity level of the target expert is higher than that of at least one other expert, proceed to step 206; if the popularity level of the target expert is lower than that of all other experts, bind a second vGPU instance to the target expert on the first GPU node or the second GPU node, and allocate resources to the second vGPU instance.
[0073] Step 206: If other experts exist, determine that the resources of the first vGPU instance can be adjusted. Based on the changed popularity level, allocate some resources of the third vGPU instance corresponding to the other experts to the first vGPU instance.
[0074] Assuming the third vGPU instance has 16 GiB of video memory and a computing power quota of 60%, and the first vGPU instance has 4 GiB of video memory and a computing power quota of 40%, then after adjustment, the third vGPU instance will have 8 GiB of video memory and a computing power quota of 40%, and the first vGPU instance will have 12 GiB of video memory and a computing power quota of 60%.
[0075] Step 207: If there are no other experts, it is determined that the resources of the first vGPU instance cannot be adjusted. The second vGPU instance is bound to the target expert on the first GPU node or the second GPU node, and resources are allocated to the second vGPU instance.
[0076] The computer device can migrate data from the first vGPU instance to the second vGPU instance, detaching the target expert from the first vGPU instance and providing services to the target expert through the second vGPU instance. Typically, the second vGPU instance has more resources than the first vGPU instance. For example, the first vGPU instance shares a graphics card with other vGPU instances, while the second vGPU instance has its own dedicated graphics card. Alternatively, the computer device can retain the first vGPU instance and create a new second vGPU instance, allowing both instances to provide concurrent services to the target expert.
[0077] Specifically, binding a second vGPU instance to the target expert on the first GPU node or the second GPU node includes: detecting whether there is an idle GPU in the first GPU node; if there is an idle GPU, binding a second vGPU instance to the target expert on the first GPU node; if there is no idle GPU, binding a second vGPU instance to the target expert on the second GPU node.
[0078] If at least two idle GPUs exist on the first GPU node, the computer can also select an idle GPU based on its type. That is, the higher the heat level, the higher the performance of the GPU. For example, when the heat level is high, a second vGPU instance can be created on a high-performance h100 GPU; when the heat level is medium, a second vGPU instance can be created on a slightly lower-performance a100 GPU.
[0079] The process of binding a second vGPU instance to the target expert on the second GPU node can include: searching for GPU nodes with idle GPUs among the various GPU nodes; if a GPU node is found, then the found GPU node is identified as the second GPU node, and the second vGPU instance is bound to the target expert on the second GPU node; if at least two GPU nodes are found, then the GPU nodes that match the changed heat level are selected from the at least two GPU nodes, the selected GPU nodes are identified as the second GPU node, and the second vGPU instance is bound to the target expert on the second GPU node.
[0080] If at least two GPU nodes are found, the computer can also select the GPU node based on the type of graphics card. That is, the higher the heat level, the higher the GPU node with higher graphics card performance is selected. For example, if GPU node 0 has an idle a100 graphics card and GPU node 1 has an idle h100 graphics card, when the changed heat level is high, GPU node 1 can be selected as the second GPU node; when the changed heat level is medium, GPU node 0 can be selected as the second GPU node.
[0081] Optionally, if no free graphics cards exist on either the first or second GPU node, a graphics card with remaining resources greater than the current resources of the first vGPU instance can be found. Then, the second vGPU instance is bound to the target expert on that graphics card, and resources are allocated to the second vGPU instance. In this way, although the target expert shares a graphics card with other experts on that graphics card, resource expansion can still be achieved.
[0082] Step 208: When the popularity level of the target expert decreases, reduce the resources of the first vGPU instance according to the changed popularity level.
[0083] When allocating resources, computer equipment also needs to obtain the changed heat level and reduce the amount of resources accordingly. That is, the lower the changed heat level, the fewer resources are allocated.
[0084] For example, when the popularity level changes from medium to low, the video memory size is adjusted from 8 GiB to 4 GiB, and the computing power quota is adjusted from 60% to 20%; when the popularity level changes from high to low, the video memory size is adjusted from 16 GiB to 4 GiB, and the computing power quota is adjusted from 100% to 20%.
[0085] In addition to reducing the resources of the first vGPU instance, it can also aggregate experts with low popularity levels on different graphics cards to the same graphics card to free up full graphics card resources.
[0086] Specifically, if the resources required by the first vGPU instance are less than the remaining resources of the target graphics card, then a third vGPU instance is bound to the target graphics card and resources are allocated to the third vGPU instance. The target graphics card is a graphics card located on the first GPU node or the third GPU node and bound to other vGPU instances.
[0087] For example, if there is a low-population expert vGPU instance on both GPU 0 and GPU 1 of the first GPU node, these two vGPU instances can be merged onto GPU 0, freeing up resources on GPU 1. Similarly, if there is a low-population expert vGPU instance on both GPU 0 and GPU 2 of the first GPU node, these two vGPU instances can be merged onto GPU 0 of the first GPU node, freeing up resources on GPU 2 of the third GPU node.
[0088] It should be noted that this embodiment only uses resource allocation for one MoE as an example. In actual implementation, resource allocation can be performed on multiple MoEs, which will not be elaborated here.
[0089] In summary, the resource allocation method for experts in MoE provided in this application, by acquiring the changes in the popularity level of each expert in MoE, can expand the resources of the first vGPU instance according to the changed popularity level of the target expert when the popularity level increases and the resources of the first vGPU instance corresponding to the target expert are insufficient, thereby achieving vertical resource expansion; or, it can bind the target expert to a second vGPU instance and allocate resources to the second vGPU instance, thereby achieving horizontal resource expansion. This achieves the purpose of adjusting resources in conjunction with the popularity level of the expert, improving resource utilization, and possessing real-time performance, elasticity, and inference throughput capabilities.
[0090] like Figure 3 The diagram illustrates a structural block diagram of a resource allocation device for experts in a Mobile Expert Array (MoE) according to an embodiment of this application. This resource allocation device for experts in a MoE can be applied to a computer device, and the System-on-Chip (SOC) includes:
[0091] Monitoring module 310 is used to monitor the popularity level of each expert in MoE. The popularity level is calculated based on the resource usage of each expert within a predetermined period.
[0092] The acquisition module 320 is used to acquire the resource description information of the first vGPU instance bound to the corresponding first GPU node of the target expert when the popularity level of the target expert increases.
[0093] The first allocation module 330 is used to expand the resources of the first vGPU instance on the first GPU node according to the changed heat level if it is determined from the resource description information that the resources of the first vGPU instance can be adjusted.
[0094] The second allocation module 340 is used to bind a second vGPU instance to the target expert on the first GPU node or the second GPU node and allocate resources to the second vGPU instance if it is determined from the resource description information that the resources of the first vGPU instance cannot be adjusted.
[0095] In an optional embodiment, the first allocation module 330 is further configured to:
[0096] Based on the resource description information, detect whether there are remaining resources on the graphics card where the first vGPU instance is located. If there are remaining resources, determine that the resources of the first vGPU instance can be adjusted.
[0097] Based on the changed heat level, allocate some or all of the remaining resources to the first vGPU instance.
[0098] In an optional embodiment, the first allocation module 330 is further configured to:
[0099] If there are no remaining resources, check if there are other experts on the graphics card with a lower heat level than the target expert. If there are other experts, determine if the resources of the first vGPU instance can be adjusted.
[0100] Based on the changed popularity level, some resources of the third vGPU instance corresponding to other experts are allocated to the first vGPU instance.
[0101] In an optional embodiment, the second allocation module 340 is further configured to:
[0102] If no other experts are available, it is determined that the resources of the first vGPU instance cannot be adjusted;
[0103] Check if there is an idle GPU in the first GPU node; if there is an idle GPU, bind a second vGPU instance to the target expert on the first GPU node; if there is no idle GPU, bind a second vGPU instance to the target expert on the second GPU node.
[0104] In an optional embodiment, the second allocation module 340 is further configured to:
[0105] Locate GPU nodes with available GPUs among all GPU nodes;
[0106] If a GPU node is found, it is designated as the second GPU node, and a second vGPU instance is bound to the target expert on the second GPU node.
[0107] If at least two GPU nodes are found, then filter out the GPU nodes that match the changed heat level from the at least two GPU nodes, determine the selected GPU node as the second GPU node, and bind the second vGPU instance to the target expert on the second GPU node.
[0108] In an optional embodiment, the device further includes a third allocation module for:
[0109] When the popularity level of a target expert decreases, the resources of the first vGPU instance are reduced according to the changed popularity level.
[0110] In an optional embodiment, the third allocation module is further configured to:
[0111] If the resources required by the first vGPU instance are less than the remaining resources of the target graphics card, then a third vGPU instance is bound to the target graphics card and resources are allocated to the third vGPU instance. The target graphics card is a graphics card located on the first GPU node or the third GPU node and bound to other vGPU instances.
[0112] In summary, the resource allocation device for experts in MoE provided in this application embodiment can, by acquiring the changes in the popularity level of each expert in MoE, expand the resources of the first vGPU instance according to the changed popularity level of the target expert when the popularity level increases and the resources of the first vGPU instance corresponding to the target expert are insufficient, thereby achieving vertical expansion of resources; or, it can bind the target expert to a second vGPU instance and allocate resources to the second vGPU instance, thereby achieving horizontal expansion of resources. This achieves the purpose of adjusting resources in conjunction with the popularity level of the expert, improving resource utilization, and possessing real-time performance, elasticity, and inference throughput capabilities.
[0113] One embodiment of this application provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the resource allocation method for experts in MoE as described above.
[0114] One embodiment of this application provides a computer device that includes a resource allocation device for experts in any of the above-described MoEs.
[0115] It should be noted that the resource allocation device for experts in MoE provided in the above embodiments is only illustrated by the division of the above functional modules when allocating resources for experts in MoE. In actual applications, the above functional allocation can be completed by different functional modules as needed, that is, the internal structure of the resource allocation device for experts in MoE can be divided into different functional modules to complete all or part of the functions described above. In addition, the resource allocation device for experts in MoE provided in the above embodiments and the resource allocation method embodiments for experts in MoE belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.
[0116] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0117] The above description is not intended to limit the embodiments of this application. Any adjustments, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of the embodiments of this application.
Claims
1. A method for allocating resources to experts in MoE, characterized in that, The method includes: Monitor the popularity level of each expert in MoE, which is calculated based on the resource usage of each expert within a predetermined period; When the popularity level of the target expert increases, obtain the resource description information of the first vGPU instance bound to the target expert on the corresponding first GPU node; If it is determined from the resource description information that the resources of the first vGPU instance can be adjusted, then on the first GPU node, the resources of the first vGPU instance are expanded according to the changed heat level. If it is determined from the resource description information that the resources of the first vGPU instance cannot be adjusted, then a second vGPU instance is bound to the target expert on the first GPU node or the second GPU node, and resources are allocated to the second vGPU instance.
2. The method for allocating expert resources in MoE according to claim 1, characterized in that, The method further includes: detecting whether there are remaining resources on the graphics card where the first vGPU instance is located based on the resource description information; if there are remaining resources, determining that the resources of the first vGPU instance can be adjusted. Expanding the resources of the first vGPU instance according to the changed heat level includes: allocating some or all of the remaining resources to the first vGPU instance according to the changed heat level.
3. The method for allocating expert resources in MoE according to claim 2, characterized in that, The method further includes: if there are no remaining resources, detecting whether there are other experts in the graphics card with a lower heat level than the target expert; if there are other experts, determining that the resources of the first vGPU instance can be adjusted. Expanding the resources of the first vGPU instance according to the changed popularity level includes: allocating a portion of the resources of the third vGPU instance corresponding to the other experts to the first vGPU instance according to the changed popularity level.
4. The method for allocating expert resources in MoE according to claim 3, characterized in that, The method further includes: if no other experts are available, determining that the resources of the first vGPU instance cannot be adjusted; The step of binding a second vGPU instance to the target expert on the first GPU node or the second GPU node includes: detecting whether there is an idle graphics card in the first GPU node; if there is an idle graphics card, binding a second vGPU instance to the target expert on the first GPU node; if there is no idle graphics card, binding a second vGPU instance to the target expert on the second GPU node.
5. The method for allocating expert resources in MoE according to claim 4, characterized in that, The step of binding a second vGPU instance to the target expert on the second GPU node includes: Locate GPU nodes with available GPUs among all GPU nodes; If a GPU node is found, the found GPU node is identified as the second GPU node, and a second vGPU instance is bound to the target expert on the second GPU node; If at least two GPU nodes are found, then select the GPU node that matches the changed heat level from the at least two GPU nodes, determine the selected GPU node as the second GPU node, and bind the second vGPU instance to the target expert on the second GPU node.
6. The method for allocating expert resources in MoE according to claim 1, characterized in that, The method further includes: When the popularity level of the target expert decreases, the resources of the first vGPU instance are reduced according to the changed popularity level.
7. The method for allocating expert resources in MoE according to claim 6, characterized in that, The method further includes: If the resources required by the first vGPU instance are less than the remaining resources of the target graphics card, then a third vGPU instance is bound to the target expert on the target graphics card, and resources are allocated to the third vGPU instance. The target graphics card is a graphics card located on the first GPU node or the third GPU node and bound to other vGPU instances.
8. A resource allocation device for experts in MoE, characterized in that, The device includes: The monitoring module is used to monitor the popularity level of each expert in MoE. The popularity level is calculated based on the resource usage of each expert within a predetermined period. The acquisition module is used to acquire the resource description information of the first vGPU instance bound to the corresponding first GPU node when the popularity level of the target expert increases. The first allocation module is configured to, if it is determined from the resource description information that the resources of the first vGPU instance can be adjusted, expand the resources of the first vGPU instance on the first GPU node according to the changed heat level. The second allocation module is configured to, if it is determined from the resource description information that the resources of the first vGPU instance cannot be adjusted, bind a second vGPU instance to the target expert on the first GPU node or the second GPU node, and allocate resources to the second vGPU instance.
9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the resource allocation method for experts in MoE as described in any one of claims 1 to 7.
10. A computer device, characterized in that, The computer device includes: the resource allocation device for experts in MoE as described in claim 8.