Scheduling method and device of resource nodes corresponding to inference model, medium and equipment

Abstract

The invention discloses a scheduling method and device for resource nodes corresponding to an inference model, a medium and equipment, and belongs to the technical field of deep learning. Obtaining model information of the inference model, wherein the model information at least comprises a model type and a model scale; a target weight combination matched with the model information is determined, and the target weight combination is a weight combination set for sensitivity of different resource parameters according to the reasoning model; various resource parameters in each resource node are obtained, and scores of the resource nodes are calculated according to the various resource parameters and the target weight combination; and selecting a target resource node according to the score, and scheduling the target resource node to execute a reasoning task of the reasoning model. According to the method, the scheduling logic can be suitable for inference models of different types and different scales, the scheduling intelligence and accuracy are improved, low-efficiency scheduling and resource waste are reduced, and the overall stability of the system is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a scheduling method and device for resource nodes corresponding to an inference model, a medium and equipment. BACKGROUND

[0002] An inference model is a model for performing various inference tasks. There are many types of inference models, including but not limited to: a large language model (LLM), a diffusion model, and a retrieval-augmented generation (RAG) model. Before using an inference model, the inference model needs to be deployed on a resource node so that the inference task can be performed by the resources allocated by the resource node.

[0003] Large-scale inference models have very high requirements for computing power, video memory, network bandwidth, and other resources. The load conditions of different resource nodes differ significantly, resulting in uneven resource utilization efficiency. Existing resource scheduling is mostly based on a single or simple indicator for decision-making, which is difficult to fully reflect the real load status of the resource nodes, resulting in unsatisfactory scheduling results. SUMMARY

[0004] The present application provides a scheduling method, device, medium and equipment for resource nodes corresponding to an inference model, to solve the problem that resource scheduling based on a single or simple indicator for decision-making is difficult to fully reflect the real load status of the resource nodes, resulting in unsatisfactory scheduling results. The technical solution is as follows:

[0005] According to a first aspect of the present application, a scheduling method for resource nodes corresponding to an inference model is provided, the method comprising:

[0006] obtaining model information of the inference model, the model information including at least a model type and a model size;

[0007] determining a target weight combination matching the model information, wherein the target weight combination is a weight combination set according to the sensitivity of the inference model to different resource parameters;

[0008] obtaining each resource parameter in each resource node, and calculating a score of the resource node according to each resource parameter and the target weight combination;

[0009] selecting a target resource node according to the score, and scheduling the target resource node to perform an inference task of the inference model.

[0010] In a possible implementation, the calculation of the score of the resource node according to each resource parameter and the target weight combination comprises:

[0011] Using a scoring formula Calculate the score of the resource node;

[0012] Where C i represents the i-th resource node, j represents the j-th resource parameter, n represents the total number of resource parameters, f j (C i ) represents the normalized score of the j-th resource parameter on the i-th resource node, T represents the model type, represents the weight of the j-th resource parameter under the model type T.

[0013] In a possible implementation, the determination of the target weight combination matched with the model information comprises:

[0014] Searching for a target weight combination matched with the model information in a plurality of preset weight combinations, wherein different model information corresponds to different weight combinations.

[0015] In a possible implementation, the resource parameters include at least two of GPU utilization, video memory fragmentation rate, queue length, inference delay, throughput and task failure rate.

[0016] In a possible implementation, when the model type is a large language model (LLM),

[0017] The size of the LLM is negatively correlated with the weight corresponding to the GPU utilization;

[0018] The size of the LLM is positively correlated with the weight corresponding to the video memory fragmentation rate;

[0019] The size of the LLM is negatively correlated with the weight corresponding to the queue length;

[0020] The size of the LLM is negatively correlated with the weight corresponding to the throughput;

[0021] The size of the LLM is positively correlated with the weight corresponding to the task failure rate.

[0022] In a possible implementation, when the model type is a Diffusion model,

[0023] The size of the Diffusion model is positively correlated with the weight corresponding to the GPU utilization;

[0024] The size of the Diffusion model is positively correlated with the weight corresponding to the video memory fragmentation rate;

[0025] The size of the Diffusion model is negatively correlated with the weight corresponding to the queue length.

[0026] The size of the Diffusion model is negatively correlated with the weight corresponding to the throughput.

[0027] The size of the Diffusion model is positively correlated with the weight corresponding to the task failure rate.

[0028] In a possible implementation, when the model type is a retrieval augmentation generation (RAG) model,

[0029] The size of the RAG model is positively correlated with the weight corresponding to the GPU utilization rate.

[0030] The size of the RAG model is negatively correlated with the weight corresponding to the queue length.

[0031] The size of the RAG model is positively correlated with the weight corresponding to the inference delay.

[0032] The size of the RAG model is positively correlated with the weight corresponding to the task failure rate.

[0033] According to a second aspect of the present application, a scheduling device of a resource node corresponding to an inference model is provided, and the device comprises:

[0034] An acquisition module is configured to acquire model information of an inference model, wherein the model information at least includes a model type and a model size.

[0035] A determination module is configured to determine a target weight combination matched with the model information, wherein the target weight combination is a weight combination set according to the sensitivity of the inference model to different resource parameters.

[0036] A calculation module is configured to acquire each resource parameter in each resource node, and calculate a score of the resource node according to each resource parameter and the target weight combination.

[0037] A scheduling module is configured to select a target resource node according to the score, and schedule the target resource node to perform an inference task of the inference model.

[0038] According to a third aspect of the present application, a computer readable storage medium is provided, and the storage medium stores at least one instruction, which is loaded and executed by a processor to implement the scheduling method of a resource node corresponding to an inference model as described above.

[0039] According to a fourth aspect of the present application, a computer device is provided, which comprises a scheduling apparatus of a resource node corresponding to the inference model.

[0040] The beneficial effects of the technical solutions provided by the present application at least include:

[0041] Different types of inference models have different sensitivities to resource parameters, and even inference models of the same type but different scales have different sensitivities to resource parameters. Therefore, a target weight combination that matches can be dynamically selected according to the model type and model scale of the inference model, and the score of each resource node can be calculated according to each resource parameter and the target weight combination. Finally, the target resource node is selected according to the score, and the target resource node is scheduled to perform the inference task. In this way, the scheduling logic can be applied to inference models of different types and different scales, and the intelligence and accuracy of scheduling can be improved.

[0042] When the resource parameters include GPU utilization and queue depth, the inference model with high GPU utilization requirement can no longer be allocated to the resource node with low GPU utilization, and the inference model with low delay requirement can no longer be allocated to the resource node with long queue length through weight adjustment. This can significantly reduce inefficient scheduling and resource waste.

[0043] The deployment cost of large-scale inference models is high, and the cost of failure is large. Therefore, weights with high memory fragmentation rate and task failure rate can be selected to perceive potential risk resource nodes in advance to avoid these risk resource nodes, thereby improving the overall stability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 is a flowchart of a scheduling method of a resource node corresponding to an inference model provided by an embodiment of the present application;

[0046] Figure 2 is a flowchart of a scheduling method of a resource node corresponding to an inference model provided by another embodiment of the present application;

[0047] Figure 3 is a structural block diagram of a scheduling apparatus of a resource node corresponding to an inference model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0049] The present application relates to a scheduling method for resource nodes corresponding to an inference model. The inference model can be any model capable of performing an inference task. Each resource node is configured with multiple acceleration chips supporting inference tasks. The inference model can be deployed on these acceleration chips and allocated resources so that the inference model can perform inference tasks using the allocated resources. Under different architectures, the acceleration chips can be graphics processing units (GPUs), neural processing units (NPUs), scalable processing units (XPU), etc.

[0050] We need to use multiple resource parameters to evaluate the state of the resource node and determine whether the inference task can be performed according to the state. The resource parameters in the present application include at least two of GPU utilization, GPU memory fragmentation rate, queue length, inference delay, throughput, and task failure rate.

[0051] (1) GPU utilization

[0052] GPU utilization is a core indicator of the efficiency of the use of computing resources in the acceleration chip. GPU utilization = (GPU active time / total observation time) x 100%.

[0053] (2) GPU memory fragmentation rate

[0054] GPU memory fragmentation rate is an indicator of the efficiency of the use of GPU memory in the acceleration chip, reflecting the proportion of "fragmented" space in the GPU memory that is not effectively utilized. High fragmentation rate can cause the GPU memory to have free space but unable to allocate large contiguous memory, thereby affecting the performance of tasks such as deep learning training and graphics rendering.

[0055] (3) Queue length

[0056] Queue length is a key indicator of the degree of congestion of the system, referring to the number of tasks or entities waiting for processing at a certain time.

[0057] (4) Inference delay (P95)

[0058] Inference delay is a core indicator of the time required for a model to process a single input data and generate an output, directly affecting applications with high real-time requirements (such as autonomous driving and voice interaction).

[0059] (5) Throughput

[0060] Throughput is a core indicator of measuring the system's ability to process tasks per unit of time, usually expressed as the number of tasks per second (such as FPS, QPS) or the amount of data per second (such as GB / s), unlike latency which focuses on the time consumption of a single task, throughput focuses on the overall processing efficiency of the system. Throughput = number of completed tasks / time window.

[0061] (6) Task failure rate

[0062] Task failure rate is a core indicator of measuring system reliability, representing the proportion of failed tasks to the total number of tasks within a certain time range. High failure rate will directly affect user experience and resource utilization (such as retry overhead). Task failure rate = (failed task number / total task number) x 100%.

[0063] The following takes the inference model as an example of LLM, Diffusion model and RAG model to analyze the sensitivity of various inference models to resource parameters.

[0064] For LLM, it has a higher requirement for the latency of token inference. Therefore, the core resource bottleneck of LLM is video memory, context cache and scheduling latency. It is more sensitive to three resource parameters: video memory fragmentation rate, inference delay and queue length.

[0065] For Diffusion model, each inference task takes a long time, and throughput priority needs to be ensured. Therefore, the core resource bottleneck of Diffusion model is GPU computing power and execution time. It is more sensitive to three resource parameters: GPU utilization, throughput and video memory fragmentation rate.

[0066] For RAG model, it is extremely dependent on external documents and has large fluctuations. Therefore, the core resource bottleneck of RAG model is input and output IO, retrieval chain and concurrent fluctuation. It is more sensitive to three resource parameters: queue length, task failure rate and inference delay.

[0067] The following takes the model size of the inference model as an example, including small (Small), medium (Medium), large (Large) and ultra-large (Ultra-Large) to analyze the sensitivity of various sizes of inference models to resource parameters.

[0068] The parameter number of small (Small) inference model is less than or equal to 7B (Billion), the video memory occupation is small, it is usually deployed on a single acceleration chip or lightweight startup, the cost of scheduling failure is low, and the score should focus on response delay and queue pressure.

[0069] Medium inference models have between 13B and 30B parameters, moderate memory usage, are typically deployed on multiple accelerator chips, and have a moderate cost for scheduling failures. Scoring should focus on memory fragmentation rate and queue length.

[0070] Large inference models have 65 bytes or more of parameters, high memory usage, and are typically deployed on multiple acceleration chips or resource nodes. The cost of scheduling failure is high, so the evaluation should focus on memory fragmentation rate and task failure rate.

[0071] Ultra-large inference models have more than or equal to 300 bytes of parameters, extremely high GPU memory usage, and are typically deployed in a distributed, parallel manner. The cost of scheduling failures is extremely high, so the evaluation should focus on GPU memory fragmentation rate and task failure rate.

[0072] like Figure 1 The diagram illustrates a flowchart of a method for scheduling resource nodes corresponding to an inference model according to an embodiment of this application. This method for scheduling resource nodes corresponding to the inference model can be applied to computer devices. The method for scheduling resource nodes corresponding to the inference model may include:

[0073] Step 101: Obtain the model information of the inference model, which includes at least the model type and model size.

[0074] The model type and model size can be set according to actual business needs, and are not limited in this embodiment.

[0075] In one example, the model type could include LLM, Diffusion, and RAG models, and the model size could include Small, Medium, Large, and Ultra-Large.

[0076] Step 102: Determine the target weight combination that matches the model information. The target weight combination is a weight combination set according to the sensitivity of the inference model to different resource parameters.

[0077] The above analysis shows that different inference models have varying sensitivities to different resource parameters. Therefore, we can set weights for each resource parameter based on its sensitivity, and combine these weights into a target weight combination. Generally speaking, the more sensitive the inference model is to a particular resource parameter, the larger its corresponding weight value.

[0078] Step 103: Obtain the resource parameters of each resource node, and calculate the score of the resource node based on the combination of the resource parameters and the target weight.

[0079] For each resource node, the resource parameters of the resource node can be weighted and calculated with the corresponding weight in the target weight combination to obtain the score of the resource node.

[0080] In step 104, a target resource node is selected according to the score, and the target resource node is scheduled to perform an inference task of the inference model.

[0081] Specifically, a target resource node with the highest score can be selected from all resource nodes, and then the target resource node is scheduled to perform an inference task in the inference model.

[0082] In summary, the scheduling method of the resource node corresponding to the inference model provided by the embodiments of the present application can dynamically select a matching target weight combination according to the model type and model size of the inference model, calculate the scores of each resource node according to each resource parameter and the target weight combination, and finally select a target resource node according to the score, and schedule the target resource node to perform an inference task. In this way, the scheduling logic can be applied to inference models of different types and different sizes, and the intelligence and accuracy of scheduling can be improved.

[0083] As shown in FIG. 1, Figure 2 The scheduling method of the resource node corresponding to the inference model can be applied to a computer device. The scheduling method of the resource node corresponding to the inference model can include:

[0084] In step 201, model information of an inference model is obtained, and the model information at least includes a model type and a model size.

[0085] In one example, the model type can include LLM, Diffusion model and RAG model, and the model size can include Small, Medium, Large and Ultra-Large.

[0086] In step 202, a target weight combination matching the model information is searched from a plurality of preset weight combinations, wherein the target weight combination is a weight combination set according to the sensitivity of the inference model to different resource parameters, and different model information corresponds to different weight combinations.

[0087] As can be known from the above analysis, different inference models have different sensitivities to different resource parameters, so we can set the weight of each resource parameter according to the sensitivity, and combine these weights into a target weight combination. Generally speaking, the more sensitive the inference model is to a certain resource parameter, the greater the corresponding weight value.

[0088] In this embodiment, the corresponding weight template can be configured for the model type and the model size for ease of management. When scheduling failure or delay occurs, the weight value in the weight template can be adjusted reversely to optimize the weight and improve the accuracy of scheduling.

[0089] The relationship between the model type, model size and resource parameters of the inference model and the corresponding weight is described below.

[0090] I. The model type is LLM:

[0091] (1) The size of the LLM is negatively correlated with the weight corresponding to the GPU utilization rate.

[0092] That is, the overall trend is that the larger the model size, the lower the weight corresponding to the GPU utilization rate; the smaller the model size, the higher the weight corresponding to the GPU utilization rate.

[0093] (2) The size of the LLM is positively correlated with the weight corresponding to the memory fragmentation rate.

[0094] That is, the overall trend is that the larger the model size, the higher the weight corresponding to the memory fragmentation rate; the smaller the model size, the lower the weight corresponding to the memory fragmentation rate.

[0095] (3) The size of the LLM is negatively correlated with the weight corresponding to the queue length.

[0096] That is, the overall trend is that the larger the model size, the lower the weight corresponding to the queue length; the smaller the model size, the higher the weight corresponding to the queue length.

[0097] (4) The size of the LLM is negatively correlated with the weight corresponding to the throughput.

[0098] That is, the overall trend is that the larger the model size, the lower the weight corresponding to the throughput; the smaller the model size, the higher the weight corresponding to the throughput.

[0099] (5) The size of the LLM is positively correlated with the weight corresponding to the task failure rate.

[0100] That is, the overall trend is that the larger the model size, the higher the weight corresponding to the task failure rate; the smaller the model size, the lower the weight corresponding to the task failure rate.

[0101] It should be noted that the size of the LLM has low sensitivity to inference delay, so the size of the LLM has no effect on the weight corresponding to the inference delay, i.e., the weight corresponding to the inference delay can be a fixed value.

[0102] II. Model type is Diffusion model:

[0103] (1) The size of the Diffusion model is positively correlated with the weight corresponding to GPU utilization.

[0104] That is, the overall trend is: the larger the model size, the higher the weight corresponding to GPU utilization; the smaller the model size, the lower the weight corresponding to GPU utilization.

[0105] Compared with LLM, the Diffusion model is more sensitive to GPU utilization, so the weight corresponding to GPU utilization in the Diffusion model scenario is greater than the weight corresponding to GPU utilization in the LLM scenario.

[0106] (2) The size of the Diffusion model is positively correlated with the weight corresponding to memory fragmentation rate.

[0107] That is, the overall trend is: the larger the model size, the higher the weight corresponding to memory fragmentation rate; the smaller the model size, the lower the weight corresponding to memory fragmentation rate.

[0108] The sensitivity of Diffusion model and LLM to memory fragmentation rate is similar, so the weight corresponding to memory fragmentation rate in the Diffusion model scenario is similar to the weight corresponding to memory fragmentation rate in the LLM scenario.

[0109] (3) The size of the Diffusion model is negatively correlated with the weight corresponding to queue length.

[0110] That is, the overall trend is: the larger the model size, the lower the weight corresponding to queue length; the smaller the model size, the higher the weight corresponding to queue length.

[0111] Compared with LLM, the Diffusion model is less sensitive to queue length, so the weight corresponding to queue length in the Diffusion model scenario is less than the weight corresponding to queue length in the LLM scenario.

[0112] (4) The size of the Diffusion model is negatively correlated with the weight corresponding to throughput.

[0113] That is, the overall trend is: the larger the model size, the lower the weight corresponding to throughput; the smaller the model size, the higher the weight corresponding to throughput.

[0114] Compared with LLM, the Diffusion model is more sensitive to throughput, so the weight corresponding to throughput in the Diffusion model scenario is greater than the weight corresponding to throughput in the LLM scenario.

[0115] (5) The size of the Diffusion model is positively correlated with the weight corresponding to the task failure rate.

[0116] That is, the overall trend is: the larger the model size, the higher the weight corresponding to the task failure rate; the smaller the model size, the lower the weight corresponding to the task failure rate.

[0117] The sensitivity of the Diffusion model and the LLM to the task failure rate is similar, so the weight corresponding to the task failure rate in the Diffusion model scenario is similar to the weight corresponding to the task failure rate in the LLM scenario.

[0118] It should be noted that the Diffusion model generates images through step-by-step reasoning, so the reasoning delay of the Diffusion model is mainly determined by the number of reasoning steps, and has little to do with the model size, so for Diffusion models of different sizes, the weight corresponding to the reasoning delay can be a fixed value.

[0119] Three, the model type is RAG model:

[0120] (1) The size of the RAG model is positively correlated with the weight corresponding to the GPU utilization rate.

[0121] That is, the overall trend is: the larger the model size, the higher the weight corresponding to the GPU utilization rate; the smaller the model size, the lower the weight corresponding to the GPU utilization rate.

[0122] Compared with the Diffusion model, the RAG model has lower sensitivity to GPU utilization rate, so the weight corresponding to the GPU utilization rate in the RAG model scenario is less than the weight corresponding to the GPU utilization rate in the Diffusion model scenario.

[0123] (2) The size of the RAG model is negatively correlated with the weight corresponding to the queue length.

[0124] That is, the overall trend is: the larger the model size, the lower the weight corresponding to the queue length; the smaller the model size, the higher the weight corresponding to the queue length.

[0125] The RAG model and the LLM have similar sensitivity to the queue length, so the weight corresponding to the queue length in the RAG model scenario is similar to the weight corresponding to the queue length in the LLM scenario.

[0126] (3) The size of the RAG model is positively correlated with the weight corresponding to the reasoning delay.

[0127] That is, the overall trend is: the larger the model size, the higher the weight corresponding to the inference delay; the smaller the model size, the lower the weight corresponding to the inference delay.

[0128] (4) The size of the RAG model is positively related to the weight corresponding to the task failure rate.

[0129] That is, the overall trend is: the larger the model size, the higher the weight corresponding to the task failure rate; the smaller the model size, the lower the weight corresponding to the task failure rate.

[0130] The inference of the RAG model involves the cooperation of multiple system components. As the model size expands, the construction of each component becomes more complex, the cross-component call chain becomes longer, and the risk of task failure rate increases. Therefore, compared with LLM and Diffusion model, the RAG model is more sensitive to the task failure rate, and the weight corresponding to the task failure rate in the RAG model scenario is greater than that in the LLM and Diffusion model scenarios, and the weight corresponding to the task failure rate increases with the increase of the model size.

[0131] The larger the parameter size of the RAG model, the higher the memory occupation, and the more stringent the requirement for resource matching. However, its inference structure is relatively stable, and the module scheduling is relatively concentrated. Therefore, compared with LLM and Diffusion model, the RAG model is less sensitive to the memory fragmentation rate. In other words, the RAG model pays more attention to whether there is enough total memory, rather than the continuity and fragmentation degree of the memory. Therefore, the weight corresponding to the memory fragmentation rate in the RAG model scenario can be a fixed value, rather than dynamically adjusted with the model size.

[0132] It should be noted that the size of the RAG model is not sensitive to the throughput, so the size of the RAG model has no effect on the weight corresponding to the throughput, that is, the weight corresponding to the throughput can be a fixed value.

[0133] In one example, Table 1 shows the target weight combination of LLM, Diffusion model and RAG model of different sizes.

[0134] Table 1

[0135]

[0136]

[0137] Step 203, obtaining each resource parameter in each resource node, calculating the score of the resource node according to each resource parameter and the target weight combination.

[0138] For each resource node, the resource parameters of that resource node can be weighted and summed with the corresponding weights in the target weight combination to obtain the score of that resource node.

[0139] Specifically, the score of a resource node is calculated based on a combination of various resource parameters and target weights, which may include:

[0140] Using the scoring formula Calculate the score of resource nodes;

[0141] Among them, C i Let f represent the i-th resource node, j represent the j-th resource parameter, n represent the total number of resource parameters, and f j (C i ) represents the normalized score of the j-th resource parameter at the i-th resource node, and T represents the model type. This represents the weight of the j-th resource parameter under model type T.

[0142] In this embodiment, various resource parameters and target weights can be combined and input into the scoring formula, and the scoring formula can be used to calculate the score of each resource node.

[0143] Step 204: Select the target resource node based on the score, and schedule the target resource node to execute the inference task of the inference model.

[0144] Specifically, the target resource node with the highest score can be selected from all resource nodes, and then that target resource node can be scheduled to execute the inference task in the inference model.

[0145] Right now,

[0146] Experimental results show that if the corresponding weights are selected based on the model type and model size, and then these weights are used to calculate the score, and finally the resource nodes to be scheduled are selected based on the score, the scheduling matching degree can be improved by 20-40%, the GPU utilization rate can be improved by 10-25%, and the scheduling failure rate can be reduced by more than 30%.

[0147] In summary, the resource node scheduling method corresponding to the inference model provided in this application embodiment addresses the fact that different types of inference models have different sensitivities to resource parameters. Even inference models of the same type but different scales have different sensitivities to resource parameters. Therefore, a matching target weight combination can be dynamically selected based on the model type and model scale of the inference model. Then, the score of each resource node is calculated based on the resource parameters and the target weight combination. Finally, the target resource node is selected based on the score, and the target resource node is scheduled to execute the inference task. In this way, the scheduling logic can be applied to inference models of different types and scales, improving the intelligence and accuracy of scheduling.

[0148] When the resource parameters include GPU utilization and queue depth, the inefficient scheduling and resource waste can be significantly reduced by weight adjustment, without assigning inference models with high GPU utilization requirements to resource nodes with low GPU utilization, and without assigning inference models with low latency requirements to resource nodes with long queue length.

[0149] The deployment cost of large-scale inference models is high, and the cost of failure is large, so the weights with high memory fragmentation rate and task failure rate can be selected to perceive potential risk resource nodes in advance to avoid these risk resource nodes, thereby improving the overall stability of the system.

[0150] As shown in Figure 3 , a structural block diagram of a scheduling device of a resource node corresponding to an inference model is shown, which can be applied to a computer device, and includes:

[0151] The acquisition module 310 is configured to acquire model information of the inference model, the model information at least including a model type and a model size;

[0152] The determination module 320 is configured to determine a target weight combination matched with the model information, wherein the target weight combination is a weight combination set according to the sensitivity of the inference model to different resource parameters;

[0153] The calculation module 330 is configured to acquire each resource parameter in each resource node, and calculate a score of the resource node according to the resource parameters and the target weight combination;

[0154] The scheduling module 340 is configured to select a target resource node according to the score, and schedule the target resource node to perform an inference task of the inference model.

[0155] In an optional embodiment, the calculation of the score of the resource node according to the resource parameters and the target weight combination includes:

[0156] The score of the resource node is calculated by using a score formula .

[0157] Wherein, C i represents the i th resource node, j represents the j th resource parameter, n represents the total number of resource parameters, f j (C i ) represents the normalized score of the j th resource parameter in the i th resource node, T represents the model type, and w represents the weight of the j th resource parameter under the model type T.

[0158] In an optional embodiment, the determination module 320 is further configured to:

[0159] The target weight combination matching the model information is searched in the preset plurality of weight combinations, wherein different model information corresponds to different weight combinations.

[0160] In an optional embodiment, the resource parameters include at least two of GPU utilization, GPU memory fragmentation rate, queue length, inference delay, throughput, and task failure rate.

[0161] In an optional embodiment, when the model type is an LLM,

[0162] The size of the LLM is negatively correlated with the weight corresponding to the GPU utilization;

[0163] The size of the LLM is positively correlated with the weight corresponding to the GPU memory fragmentation rate;

[0164] The size of the LLM is negatively correlated with the weight corresponding to the queue length;

[0165] The size of the LLM is negatively correlated with the weight corresponding to the throughput;

[0166] The size of the LLM is positively correlated with the weight corresponding to the task failure rate.

[0167] In an optional embodiment, when the model type is a Diffusion model,

[0168] The size of the Diffusion model is positively correlated with the weight corresponding to the GPU utilization;

[0169] The size of the Diffusion model is positively correlated with the weight corresponding to the GPU memory fragmentation rate;

[0170] The size of the Diffusion model is negatively correlated with the weight corresponding to the queue length;

[0171] The size of the Diffusion model is negatively correlated with the weight corresponding to the throughput;

[0172] The size of the Diffusion model is positively correlated with the weight corresponding to the task failure rate.

[0173] In an optional embodiment, when the model type is an RAG model,

[0174] The size of the RAG model is positively correlated with the weight corresponding to the GPU utilization;

[0175] The size of the RAG model is negatively correlated with the weight corresponding to the queue length;

[0176] The size of the RAG model is positively correlated with the weight corresponding to the inference delay.

[0177] The size of the RAG model is positively correlated with the weight corresponding to the task failure rate.

[0178] In summary, the scheduling device of the resource node corresponding to the inference model provided in the embodiments of the present application is different in sensitivity to resource parameters for different types of inference models, and even for the same type but different sizes of inference models. Therefore, the target weight combination can be dynamically selected according to the model type and model size of the inference model, and the score of each resource node can be calculated according to each resource parameter and the target weight combination. Finally, the target resource node is selected according to the score, and the target resource node is scheduled to perform the inference task. In this way, the scheduling logic can be applied to inference models of different types and different sizes, and the intelligence and accuracy of scheduling can be improved.

[0179] When the resource parameters include GPU utilization and queue depth, the inference model with high GPU utilization requirement can no longer be allocated to the resource node with low GPU utilization, and the inference model with low delay requirement can no longer be allocated to the resource node with long queue length through weight adjustment, which can significantly reduce inefficient scheduling and resource waste.

[0180] The deployment cost of large-scale inference models is high, and the cost of failure is large. Therefore, the weights of the memory fragmentation rate and the task failure rate can be selected to perceive potential risk resource nodes in advance to avoid these risk resource nodes, thereby improving the overall stability of the system.

[0181] One embodiment of the present application provides a computer readable storage medium, the storage medium stores at least one instruction, the at least one instruction is loaded and executed by a processor to implement the scheduling method of the resource node corresponding to the inference model as described above.

[0182] One embodiment of the present application provides a computer device, the computer device includes the scheduling device of the resource node corresponding to any inference model.

[0183] It should be noted that the scheduling apparatus of the resource node corresponding to the inference model provided in the above embodiments is only exemplified by the above division of the functional modules when scheduling the resource node corresponding to the inference model. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the scheduling apparatus of the resource node corresponding to the inference model is divided into different functional modules to complete all or part of the functions described above. In addition, the scheduling apparatus of the resource node corresponding to the inference model provided in the above embodiments and the method embodiment of scheduling the resource node corresponding to the inference model belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0184] A person of ordinary skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0185] The above description does not limit the embodiments of the present application. Any adjustment, equivalent replacement, improvement, etc. made within the spirit and principles of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A method for scheduling resource nodes corresponding to an inference model, characterized in that, The method comprises: obtaining model information of an inference model, the model information at least comprising a model type and a model size; determining a target weight combination matched with the model information, wherein the target weight combination is a weight combination set according to the sensitivity of the inference model to different resource parameters; obtaining various resource parameters in each resource node, and calculating a score of the resource node according to the various resource parameters and the target weight combination; selecting a target resource node according to the score, and scheduling the target resource node to perform an inference task of the inference model. 2.The method of claim 1, wherein, The calculation of the score of the resource node according to the various resource parameters and the target weight combination comprises: Using a scoring formula calculating a score for the resource node; wherein C i represents the i-th resource node, j represents the j-th resource parameter, n represents the total number of resource parameters, f j (C i ) represents the normalized score of the j-th resource parameter on the i-th resource node, T represents the model type, represents the weight of the j-th resource parameter under the model type T. 3.The method of claim 1, wherein, The determination of the target weight combination matched with the model information comprises: searching for the target weight combination matched with the model information in a plurality of preset weight combinations, wherein different model information corresponds to different weight combinations.

4. The method of claim 1 to 3, wherein, The resource parameters comprise at least two of a GPU utilization rate, a video memory fragmentation rate, a queue length, an inference delay, a throughput, and a task failure rate.

5. The method of claim 4, wherein the resource node corresponding to the inference model is scheduled based on the inference model. When the model type is a large language model (LLM), the size of the LLM is negatively correlated with the weight corresponding to the GPU utilization rate; the size of the LLM is positively correlated with the weight corresponding to the video memory fragmentation rate; the size of the LLM is negatively correlated with the weight corresponding to the queue length; the size of the LLM is negatively correlated with the weight corresponding to the throughput; the size of the LLM is positively correlated with the weight corresponding to the task failure rate.

6. The method of claim 4, wherein the resource node corresponding to the inference model is scheduled based on a resource node corresponding to a previous inference model. When the model type is a Diffusion model, the size of the Diffusion model is positively correlated with the weight corresponding to the GPU utilization rate; the size of the Diffusion model is positively correlated with the weight corresponding to the video memory fragmentation rate; the size of the Diffusion model is negatively correlated with the weight corresponding to the queue length; the size of the Diffusion model is negatively correlated with the weight corresponding to the throughput; the size of the Diffusion model is positively correlated with the weight corresponding to the task failure rate.

7. The method of claim 4, wherein the resource node corresponding to the inference model is scheduled based on a result of the inference model. When the model type is a RAG model, the size of the RAG model is positively correlated with the weight corresponding to the GPU utilization rate; the size of the RAG model is negatively correlated with the weight corresponding to the queue length; the size of the RAG model is positively correlated with the weight corresponding to the inference delay; the size of the RAG model is positively correlated with the weight corresponding to the task failure rate.

8. A scheduling device for resource nodes corresponding to a reasoning model, characterized in that, The device comprises: an obtaining module configured to obtain model information of an inference model, the model information at least comprising a model type and a model size; A determination module is configured to determine a target weight combination matched with the model information, wherein the target weight combination is a weight combination set according to the sensitivity of the inference model to different resource parameters; A calculation module is configured to obtain each resource parameter in each resource node, and calculate a score of the resource node according to the resource parameters and the target weight combination; A scheduling module is configured to select a target resource node according to the score, and schedule the target resource node to perform an inference task of the inference model.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the scheduling method of the resource node corresponding to the inference model according to any one of claims 1 to 7.

10. A computer device, comprising: The computer device comprises the scheduling apparatus of the resource node corresponding to the inference model according to claim 8.

Images