Resource allocation method and electronic equipment

Abstract

The invention discloses a resource allocation method and electronic equipment, target resources required by each large model during task processing are determined, different large models are used for processing different types of tasks, and the target resources at least comprise computing power resources; predicting the number and the type of tasks to be processed in the (t + n) th time period based on the historical record, t time period being the current time period, t being greater than or equal to 0, and n being greater than 0; determining a resource allocation strategy of the (t + n) th time period based on the number and the type of the tasks to be processed in the (t + n) th time period according to target resources required by each large model during task processing; initial target resources of different large models in the (t + n) th time period are allocated based on a resource allocation strategy, pre-stored resources are allocated, and the pre-stored resources are part of resources reserved from the total target resources and are used for guaranteeing the processing efficiency of tasks processed in the (t + n) th time period.

Description

Technical Field

[0001] The present application relates to the field of model processing technology, and in particular to a resource allocation method and electronic equipment. Background Art

[0002] With the rapid development of big models, it is becoming more and more common to deploy big models on enterprise systems to solve business problems.

[0003] Due to different business scenarios, users may need to use multiple large models at the same time to complete inference tasks under different business directions. The system resources are fixed, while the resource requirements of the large model's inference operations are elastic. If a certain amount of resources are fixedly allocated to each large model, when the resource requirements of the large model change, the resources allocated to a certain large model may be insufficient to support its inference tasks, while the resources allocated to another large model may not be effectively utilized. Summary of the invention

[0004] In view of this, the present application provides a resource allocation method and an electronic device, and the specific scheme thereof is as follows:

[0005] A resource allocation method, comprising:

[0006] Determine the target resources required by each large model when processing tasks, where different large models are used to process different types of tasks, and the target resources include at least computing resources;

[0007] Predict the number and type of tasks to be processed in the t+n period based on historical records, where the t period is the current period, t is greater than or equal to 0, and n is greater than 0;

[0008] Based on the number and type of tasks to be processed in the t+nth period, and according to the target resources required by each large model when processing tasks, determine the resource allocation strategy for the t+nth period;

[0009] Based on the resource allocation strategy, the initial target resources of different large models in the t+n period are allocated, and the pre-stored resources are allocated. The pre-stored resources are part of the resources reserved from the total target resources to ensure the processing efficiency of the tasks processed in the t+n period.

[0010] Furthermore, it also includes:

[0011] In the t+nth time period, if it is determined that the first large model satisfies the target condition in executing the target task, at least part of the pre-stored resources are allocated to the first large model, so that the first large model executes the target task using the initial target resources and at least part of the pre-stored resources;

[0012] The target conditions include: the efficiency of the first large model in executing the target task is lower than a first threshold, and / or the number of tasks executed by the first large model is greater than a second threshold.

[0013] Furthermore, it also includes:

[0014] In the t+nth time period, if it is determined that the first large model performs the target task and satisfies the target condition, the target task is transferred to the second large model so that the second large model performs the target task;

[0015] The target condition includes at least one of the following: the efficiency of the first large model in executing the target task is lower than a first threshold, and / or the number of tasks executed by the first large model is greater than a second threshold.

[0016] Further, transferring the target task to the second largest model includes:

[0017] Determine a second large model whose compatibility value with the target task reaches a third threshold based on a preset large model and task matching relationship table, wherein the large model and task matching relationship table can characterize the compatibility values ​​of different large models for different tasks;

[0018] The target task is transferred to the second large model.

[0019] Furthermore, the determining of the second large model whose compatibility value with the target task reaches a third threshold based on the preset matching relationship table of large models and tasks includes:

[0020] Determine a large model whose compatibility value with the target task reaches a third threshold based on a preset large model and task matching relationship table;

[0021] If it is determined that the number of tasks currently executed by the large model whose compatibility value with the target task reaches the third threshold does not reach the second threshold, the large model whose compatibility value with the target task reaches the third threshold is determined as the second large model.

[0022] Furthermore, based on the number and type of tasks to be processed in the t+nth period, and according to the target resources required by each large model when processing tasks, determining the resource allocation strategy for the t+nth period includes:

[0023] Determine the priority queue for each task type;

[0024] Determine the priority order corresponding to the task type to be processed in the t+nth time period based on the priority queue of each task type;

[0025] Determine the resource allocation strategy corresponding to the tasks to be processed in the t+nth period according to the priority order, the number of tasks to be processed in the t+nth period, and the target resources required by each large model when processing the tasks;

[0026] The task is a task in a cloud computing process, and the computing power resources in the target resources allocated by the resource allocation strategy are computing power resources of a graphics processor.

[0027] Furthermore, allocating the initial target resources of different large models in the t+n period based on the resource allocation strategy, and allocating the pre-stored resources, includes:

[0028] If it is determined that the time difference between the current moment and the initial moment of the t+nth period is within a specific range, preloading the large model corresponding to the task to be processed in the t+nth period;

[0029] Allocate initial computing resources and initial video memory resources of different large models in the t+n period based on the resource allocation strategy, and allocate pre-stored resources;

[0030] The data corresponding to the task to be processed in the t+nth time period is transferred to the initial video memory resource corresponding to the corresponding large model.

[0031] Furthermore, it also includes:

[0032] Determine the difference between the target resources required in the resource allocation strategy for the t+nth time period and the target resources at the current moment;

[0033] The target resource is released for the t+nth time period based on the difference.

[0034] Furthermore, the prediction of the number and type of tasks to be processed in the t+n period based on historical records includes:

[0035] Determine the number and type of tasks in different time periods based on the historical records;

[0036] Determine the priority queue for each task type;

[0037] Configuring a confidence value for each task type based on the priority queue;

[0038] The number of tasks and the type of tasks to be processed in the t+n period are predicted based on the number of tasks and the type of tasks in the different time periods and the confidence value of each task type.

[0039] An electronic device, comprising:

[0040] A processor is used to determine the target resources required by each large model when processing tasks, where different large models are used to process different types of tasks, and the target resources include at least computing resources; predict the number of tasks to be processed and the type of tasks in the t+n period based on historical records, where the t period is the current period, t is greater than or equal to 0, and n is greater than 0; determine the resource allocation strategy for the t+n period based on the number of tasks to be processed and the type of tasks in the t+n period and the target resources required by each large model when processing tasks; allocate initial target resources for different large models in the t+n period based on the resource allocation strategy, and allocate pre-stored resources, where the pre-stored resources are part of the resources reserved from the total target resources to ensure the processing efficiency of the tasks processed in the t+n period;

[0041] The memory is used to store the program required by the processor to execute the above processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flowchart of a resource allocation method disclosed in an embodiment of the present application;

[0044] Figure 2 A flowchart of another resource allocation method disclosed in an embodiment of the present application;

[0045] Figure 3 A flowchart of another resource allocation method disclosed in an embodiment of the present application;

[0046] Figure 4 A flowchart of another resource allocation method disclosed in an embodiment of the present application;

[0047] Figure 5 A flowchart of another resource allocation method disclosed in an embodiment of the present application;

[0048] Figure 6 A flowchart of another resource allocation method disclosed in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of the task volume of application A, application B and application C in a recording cycle disclosed in an embodiment of the present application;

[0050] Figure 8A schematic diagram of a resource allocation method disclosed in an embodiment of the present application;

[0051] Fig. 9 A schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0053] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0054] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0055] This application discloses a resource allocation method, and its flow chart is as follows: Figure 1 As shown, including:

[0056] Step S11: determine the target resources required by each large model when processing tasks. Different large models are used to process different types of tasks. The target resources include at least computing resources.

[0057] Step S12: predicting the number and type of tasks to be processed in the t+n period based on historical records, where the t period is the current period, t is greater than or equal to 0, and n is greater than 0;

[0058] Step S13: Based on the number and type of tasks to be processed in the t+nth period, and according to the target resources required by each large model when processing tasks, determine the resource allocation strategy for the t+nth period;

[0059] Step S14: allocate initial target resources of different large models in the t+n period based on the resource allocation strategy, and allocate pre-stored resources, which are part of the resources reserved from the total target resources to ensure the processing efficiency of the tasks processed in the t+n period.

[0060] With the rapid development of big models, it is becoming more and more common to deploy big models on enterprise systems to solve business problems.

[0061] Due to different business scenarios, users may need to use multiple large models at the same time to complete inference tasks under different business directions. The system resources are fixed, while the resource requirements of the large model's inference operations are elastic. If a certain amount of resources are fixedly allocated to each large model, when the resource requirements of the large model change, the resources allocated to a certain large model may be insufficient to support its inference tasks, while the resources allocated to another large model may not be effectively utilized.

[0062] Based on this, in this solution, it is necessary to determine the target resources required by each large model when performing task processing, and to predict the number of tasks to be processed and the type of tasks in the t+n period, and to allocate resources based on the predicted number of tasks to be processed and the type of tasks in the t+n period and the target resources required by each large model when performing task processing, so as to ensure that each large model in the t+n period has the target resources required for the corresponding task processing when performing the corresponding task processing, so as to avoid the problem of too many resources allocated to it causing resource waste, and also avoid the situation where too few resources are allocated to it resulting in the inability to complete the task; and, when allocating resources, reserved resources are also allocated for different large models to ensure the processing efficiency and accuracy of task inference in the process of large models executing tasks, and to ensure the stability of the system.

[0063] Various large models are used to process different types of tasks, such as large language models for processing natural language text, large visual models for image analysis and processing, and large industry models for specific industries or fields.

[0064] Different large models require different target resources when executing their corresponding tasks. Therefore, when multiple large models are configured in a system, the target resources required by each large model when performing task processing can be determined in advance.

[0065] The target resources may at least include: computing power resources, which may be provided by devices such as a central processing unit (CPU), a graphics processing unit (GPU), memory, and storage; in addition, the target resources may also include: video memory resources, etc.

[0066] Among them, the computing power resources required for different large models are related to the prompt word template of the task type, and the number of input and output words. The complexity, structure and length of the prompt word template directly affect the number of processed words, and therefore also affect the computing load; the number of input words affects the amount of calculation required for encoding and the content required to store intermediate states; the number of output words affects the decoding process, which is usually the most computationally intensive part.

[0067] Specifically, for the length The word unit and parameter quantity are The number of floating point operations per second (FLOPs) required for the encoding process of a large model can be calculated as , by predicting the length of the output word To calculate the FLOPs required for the decoding process, it can be expressed as , the total FLOPs required is , as shown below:

[0068]

[0069]

[0070]

[0071] When calculating the memory resources required for inference of a large model, it is usually necessary to consider the model structure, input data size, batch size, and data type. The memory resources required to store activations during forward and backward propagation are proportional to the number of parameters and the layer size of the large model, which can be calculated as ; Each parameter in a large model usually requires float / 8 bytes of storage space, such as float32 requires 4 bytes of storage space. The video memory resources required by the parameter can be determined as ; In addition, temporary variables, intermediate calculations, and buffer space during the calculation process also require certain video memory resources, which can be expressed as The total video memory resources required can be expressed as ,but:

[0072]

[0073]

[0074]

[0075] If it is necessary to allocate the resources of the system in the t+n period, the tasks that need to be processed in the system in the t+n period can be predicted, that is, how many tasks need to be processed in the t+n period and the type of each task that needs to be processed can be predicted. Since different types of tasks require large models of different levels and sizes to process, when allocating the target resources of the system in a certain period to different large models, it is necessary to determine the type of each task to be processed, so as to determine the corresponding large model based on the type of task and allocate target resources to it.

[0076] The tth time period is the current time period, the t+nth time period is a time period after the current time period, t is a number greater than or equal to 0, and n is a number greater than 0.

[0077] The number and type of tasks to be processed in the t+n period can be predicted based on historical records. The number and type of tasks processed in the period corresponding to the t+n period in different recording cycles in the historical records can be determined, and based on this, the number and type of tasks to be processed in the t+n period in the current cycle can be predicted.

[0078] For example: a recording cycle is 1 day, and the t+nth period is the period from 13:00 to 14:00 on the day. Then, the number of tasks and the type of tasks processed during the period from 13:00 to 14:00 on each day should be determined based on the records of the period from 13:00 to 14:00 on the previous day or each of the previous 5 days in the historical records, and then the number of tasks and the type of tasks to be processed during the period from 13:00 to 14:00 on the day should be predicted based on this. If the number of tasks processed during the period from 13:00 to 14:00 on each of the previous 5 days is m, and the task type is type 1, then it can be predicted that the number of tasks to be processed during the period from 13:00 to 14:00 on the day is m, and the task type is type 1.

[0079] After determining the number and type of tasks to be processed in the t+n period, the resource allocation strategy for the t+n period is determined based on the target resources required by each large model when processing tasks, that is, the amount of target resources allocated to each large model.

[0080] After determining the task type of the task to be processed in the t+n period, determine the large model for processing the task type based on the task type of each task, and then determine the resources required for the large model to process the tasks of the task type in the t+n period based on the number of tasks corresponding to each task type and the target resources required for the large model to process the tasks of the task type.

[0081] For example, it is determined that the tasks to be processed in the t+n period include: task 1, task 2 and task 3, among which, the number of task 1 in the t+n period is m1, the number of task 2 is m2, and the number of task 3 is m3. The large model for processing tasks of the type of task 1 is large model 1, the large model for processing tasks of the type of task 2 is large model 2, and the large model for processing tasks of the type of task 3 is large model 3. Among them, the target resource required by large model 1 when processing tasks is k1, the target resource required by large model 2 when processing tasks is k2, and the target resource required by large model 3 when processing tasks is k3. Then the predicted t+n period is The number of tasks that the large model 1 needs to perform in the n time period is m1, and the resources required for each task are k1, so the target resources required for the large model 1 in the t+n time period are predicted to be k1×m1; the number of tasks that the large model 2 needs to perform in the t+n time period is predicted to be m2, and the resources required for each task are k2, so the target resources required for the large model 2 in the t+n time period are predicted to be k2×m2; the number of tasks that the large model 3 needs to perform in the t+n time period is predicted to be m3, and the resources required for each task are k3, so the target resources required for the large model 3 in the t+n time period are predicted to be k3×m3, thereby determining the resource allocation strategy.

[0082] Afterwards, the target resources of the system in the t+n period are allocated based on the resource allocation strategy. While allocating the corresponding initial target resources to each large model, pre-stored resources must also be allocated. Pre-stored resources are part of the resources reserved from the total target resources. During the resource allocation process, they will not be allocated to any large model, but will be reserved to ensure the processing efficiency of the tasks processed in the t+n period, to avoid a sudden increase in tasks in the t+n period, or a sudden increase in the amount of data. Reserved resources can be used for execution, to avoid the inability to execute the corresponding tasks due to insufficient resources, thereby ensuring the smooth execution of each task in the t+n period.

[0083] The resource allocation method disclosed in this embodiment determines the target resources required by each large model when performing task processing, and different large models are used to process different types of tasks, and the target resources include at least computing resources; predicts the number of tasks to be processed and the type of tasks in the t+n period based on historical records, where the t period is the current period, t is greater than or equal to 0, and n is greater than 0; based on the number of tasks to be processed and the type of tasks in the t+n period, and according to the target resources required by each large model when performing task processing, determines the resource allocation strategy for the t+n period; allocates initial target resources for different large models in the t+n period based on the resource allocation strategy, and allocates pre-stored resources, which are part of the resources reserved from the total target resources to ensure the processing efficiency of the tasks processed in the t+n period. This scheme predicts the number and type of tasks to be processed in the t+n period, and determines the resource allocation strategy based on the number and type of tasks to be processed in the t+n period and the target resources required by each large model when processing the tasks. Based on this, the initial target resources of different large models in the t+n period and the allocated pre-stored resources are allocated to ensure that each task to be processed in the t+n period can be executed by the corresponding large model according to its required target resources, so as to ensure that the large model has sufficient resources to execute the corresponding tasks and avoid resource waste and insufficient resources.

[0084] This embodiment discloses a resource allocation method, and its flow chart is as follows: Figure 2 As shown, including:

[0085] Step S21: determine the target resources required by each large model when processing tasks. Different large models are used to process different types of tasks. The target resources include at least computing resources.

[0086] Step S22, predicting the number and type of tasks to be processed in the t+n period based on historical records, where the t period is the current period, t is greater than or equal to 0, and n is greater than 0;

[0087] Step S23, based on the number and type of tasks to be processed in the t+nth period, and according to the target resources required by each large model when processing tasks, determine the resource allocation strategy for the t+nth period;

[0088] Step S24: allocating initial target resources of different large models in the t+n period based on the resource allocation strategy, and allocating pre-stored resources, where the pre-stored resources are part of the resources reserved from the total target resources to ensure the processing efficiency of the tasks processed in the t+n period;

[0089] Step S25: In the t+nth time period, if it is determined that the first large model satisfies the target conditions for executing the target task, at least part of the pre-stored resources will be allocated to the first large model so that the first large model uses the initial target resources and at least part of the pre-stored resources to execute the target task. The target conditions include: the efficiency of the first large model in executing the target task is lower than a first threshold, and / or the number of tasks executed by the first large model is greater than a second threshold.

[0090] The number and type of tasks to be processed in the t+n period are predicted based on historical records, and based on the predicted number and type of tasks to be processed in the t+n period, initial target resources and pre-stored resources are allocated to different large models in the t+n period according to the target resources required by each large model when processing tasks.

[0091] In the t+nth time period, the initial target resources allocated to each large model are used to perform the corresponding tasks, such as: large model 1 uses the initial target resource k1 allocated to it to perform tasks 1 and 3, and large model 2 uses the initial target resource k2 allocated to it to perform task 2.

[0092] In the t+nth time period, if it is determined that the efficiency of the first large model in executing its corresponding target task is low and is lower than the first threshold, such as lower than 50%, it is considered that the current resources of the first large model are insufficient. At this time, at least part of the pre-stored resources can be allocated to the first large model to improve the efficiency of the first large model in executing the target task and reach the first threshold, so as to avoid the low efficiency of the first large model in executing the task;

[0093] Alternatively, in the t+nth time period, if it is determined that the number of tasks executed by the first large model is large and greater than the second threshold, such as greater than 150, it is considered that the first large model currently executes many tasks and there may be a shortage of resources. Therefore, at least part of the pre-stored resources can be allocated to the first large model to ensure the efficiency of the first large model in executing tasks and avoid its low efficiency.

[0094] Among them, the pre-stored resources can be divided into several parts at the same time, and the divided parts can be allocated to different large models respectively to make up for the problem of insufficient resources of the large model; or the pre-stored resources can be completely allocated to a large model, and a large model can use all the pre-stored resources to ensure the task execution efficiency of the large model.

[0095] Alternatively, the pre-stored resources are allocated to a large model in the first time period of the t+n period. After the first time period ends, the large model no longer needs to replenish the pre-stored resources. At this time, the pre-stored resources are recovered and allocated to other large models in other time periods of the t+n period so that other large models can perform tasks, thereby avoiding insufficient resources for other large models.

[0096] For example, in the first time period of the t+nth time period, the task execution efficiency of the first large model is lower than the first threshold value. At this time, the first part of the pre-stored resources can be allocated to the first large model, and the first large model uses the initial target resources allocated to it and the first part of the pre-stored resources to perform the corresponding tasks. After the first time period ends, the task execution of the first large model is completed, or the efficiency of the first large model in executing tasks is greater than the first threshold value. At this time, the pre-stored resources can be recycled to the first part of the pre-stored resources allocated to the first large model, that is, the first large model can only use the initial target resources allocated to it to perform subsequent tasks, and no longer use the first part of the pre-stored resources to perform tasks; in the second time period after the first time period in the t+nth time period, if it is determined that the number of tasks executed by the second large model is greater than the second threshold value, at this time, in order to avoid the situation where the second large model is short of resources, the second part can be divided from the pre-stored resources and allocated to the second large model, so that the second large model can use the initial target resources allocated to it and the second part of the pre-stored resources to perform tasks. Among them, the second part of the pre-stored resources is generated after the first part is recycled to the pre-stored resources. Therefore, there is no obvious boundary between the first part and the second part, that is, at least part of the first part of the pre-stored resources allocated to the first large model in the first time period may be allocated to the second large model as at least part of the second part in the second time period.

[0097] The resource allocation method disclosed in this embodiment predicts the number and type of tasks to be processed in the t+n period, and allocates corresponding initial target resources and pre-stored resources to different large models in the t+n period, so that when the efficiency of the first large model in executing the target task is low or the number of tasks it executes is too large in the t+n period, at least part of the pre-stored resources will be allocated to the first large model to ensure the efficiency of the first large model in executing tasks and avoid the situation where the task cannot be completed in time due to insufficient resources.

[0098] This embodiment discloses a resource allocation method, and its flow chart is as follows: Figure 3 As shown, including:

[0099] Step S31: determine the target resources required by each large model when processing tasks. Different large models are used to process different types of tasks. The target resources include at least computing resources.

[0100] Step S32: predict the number and type of tasks to be processed in the t+n period based on historical records, where the t period is the current period, t is greater than or equal to 0, and n is greater than 0;

[0101] Step S33: Based on the number and type of tasks to be processed in the t+nth period, and according to the target resources required by each large model when processing tasks, determine the resource allocation strategy for the t+nth period;

[0102] Step S34: allocating initial target resources of different large models in the t+n period based on the resource allocation strategy, and allocating pre-stored resources, where the pre-stored resources are part of the resources reserved from the total target resources to ensure the processing efficiency of the tasks processed in the t+n period;

[0103] Step S35: In the t+nth time period, if it is determined that the first large model satisfies the target condition for executing the target task, the target task is transferred to the second large model so that the second large model executes the target task. The target condition includes at least one of the following: the efficiency of the first large model in executing the target task is lower than a first threshold, and / or the number of tasks executed by the first large model is greater than a second threshold.

[0104] The number and type of tasks to be processed in the t+n period are predicted based on historical records, and based on the predicted number and type of tasks to be processed in the t+n period, initial target resources and pre-stored resources are allocated to different large models in the t+n period according to the target resources required by each large model when processing tasks.

[0105] In the t+nth time period, the initial target resources allocated to each large model are used to perform the corresponding tasks, such as: large model 1 uses the initial target resource k1 allocated to it to perform tasks 1 and 3, and large model 2 uses the initial target resource k2 allocated to it to perform task 2.

[0106] In the t+nth period, if it is determined that the efficiency of the first large model in executing its corresponding target task is low and lower than the first threshold, such as lower than 50%, it is considered that the current resources of the first large model are insufficient. At this time, the target task executed by the first large model can be directly transferred to the second large model for execution, that is, the first large model no longer executes the target task, but continues to execute other tasks, so as to avoid the situation where the target task is executed with low efficiency;

[0107] Alternatively, in the t+nth time period, if it is determined that the number of tasks executed by the first largest model is large and greater than the second threshold, such as greater than 150, it is considered that the first largest model currently executes many tasks and there may be insufficient resources. Therefore, the target task is transferred to the second largest model, and the second largest model executes the target task. The first largest model continues to execute other tasks except the target task to reduce the number of tasks executed by the first largest model and avoid its low execution efficiency.

[0108] Among them, the target task is transferred to the second largest model, and the target task is executed by the second largest model. Then, the compatibility value between the second largest model and the target task must meet certain conditions, namely: based on the pre-set matching relationship table between the large model and the task, the second largest model whose compatibility value with the target task reaches the third threshold is determined, and the matching relationship table between the large model and the task can characterize the compatibility values ​​of different large models for different tasks; the target task is transferred to the second largest model.

[0109] The inference effects of different tasks on different large models are different. Therefore, in order to ensure the inference quality during task execution, a matching relationship table between large models and tasks can be set in advance to characterize the compatibility values ​​of different large models and different tasks, and the large models for executing different tasks can be selected based on the compatible values.

[0110] As shown in Table 1, this is a table that illustrates the adaptation relationship between the task and the large model through compatible numerical representations, where the compatible numerical representation is the compatibility score. The higher the score, the higher the adaptability of the large model to the task, that is, the better the effect when the large model is used to infer the task. The lower the score, the lower the adaptability of the large model to the task, that is, the poorer the effect when the large model is used to infer the task. The "×" in the table indicates that the application is incompatible with the large model, that is, the large model cannot be used to infer the task.

[0111] Table 1

[0112]

[0113] In Table 1, the tasks of Application 1 can be executed by large model A or large model C, but large model A is preferred based on the compatibility score. If the execution efficiency of large model A is lower than the first threshold during the execution of the task, or the number of tasks executed by large model A is greater than the second threshold, then one of the tasks can be transferred to other large models for execution, such as transferring the tasks of Application 1 to large model C, and having large model C execute the tasks of Application 1. Large model B is incompatible with Application 1 and cannot execute the tasks of Application 1.

[0114] Furthermore, when transferring the target task to the second large model, it can be specifically as follows: based on a pre-set matching relationship table between large models and tasks, a large model whose compatibility value with the target task reaches a third threshold is determined; if it is determined that the number of tasks currently executed by the large model whose compatibility value with the target task reaches the third threshold does not reach the second threshold, the large model whose compatibility value with the target task reaches the third threshold is determined as the second large model.

[0115] That is, when it is determined based on the matching relationship table between large models and tasks that the compatibility value between a large model and the target task reaches the third threshold, such as: the compatibility score is greater than 6, the number of tasks currently executed by the large model is further determined. If the number of tasks executed by it reaches the second threshold, the target task will not be transferred to the large model, and other large models will continue to be selected. If the number of tasks executed by the large model does not reach the second threshold, the large model will be directly determined as the second large model, and the target task will be transferred to the second large model.

[0116] Alternatively, when it is determined based on the matching relationship table between large models and tasks that the compatibility value between a large model and the target task reaches a third threshold, it is further determined whether the efficiency of the large model in executing the task is lower than the first threshold. If it is determined that the efficiency of the large model in executing the task is lower than the first threshold, it indicates that the resources required for the large model to execute the task are insufficient, and the target task cannot be transferred to the large model. Other large models continue to be selected. If the efficiency of the large model in executing the task is not lower than the first threshold, the large model is directly determined as the second large model, and the target task is transferred to the second large model.

[0117] Furthermore, in the resource allocation method disclosed in this embodiment, it may also be:

[0118] In the t+nth time period, if it is determined that the efficiency of the first large model in executing the target task is lower than the first threshold, and / or the number of tasks executed by the first large model is greater than the second threshold, firstly, at least part of the pre-stored resources are allocated to the first large model, so that the first large model uses the initial target resources allocated to it and at least part of the pre-stored resources to execute the target task; thereafter, the efficiency of the first large model in executing the task is continued to be monitored, and if it is determined that the efficiency of the first large model in executing the task is no longer lower than the first threshold, the current state can be maintained to continue to execute the target task through the first large model; if it is determined that the efficiency of the first large model in executing the task is still lower than the first threshold, the target task can be transferred to the second large model, so that the second large model can be used to execute the target task.

[0119] Among them, the selection of the second large model can be: based on a preset matching relationship table between large models and tasks, determine the second large model whose compatibility value with the target task reaches a third threshold, or, based on a preset matching relationship table between large models and tasks, determine the large model whose compatibility value with the target task reaches the third threshold. If the number of tasks currently executed by the large model whose compatibility value with the target task reaches the third threshold does not reach the second threshold, the large model is determined as the second large model.

[0120] That is, in order to improve the efficiency of large models in executing tasks, when there is a shortage of resources, the reserved resources are first used to supplement the resources of the large model to execute the corresponding tasks. This can be achieved through the request processing gateway settings; if the reserved resources are not sufficient to support the execution of the large model, the target task can be migrated to other large models. At this time, the inference quality of the target task will be reduced, but the task execution efficiency of the large model can be guaranteed.

[0121] The resource allocation method disclosed in this embodiment predicts the number and type of tasks to be processed in the t+n period, and allocates corresponding initial target resources and pre-stored resources to different large models in the t+n period, so that in the t+n period, when the efficiency of the first large model in executing the target task is low or the number of tasks it executes is too large, the target task is transferred to the second large model so that the second large model executes the target task, thereby reducing the number of tasks executed by the first large model, improving the efficiency of the first large model in executing tasks, and avoiding the situation where tasks cannot be completed in time due to insufficient resources of the large models.

[0122] This embodiment discloses a resource allocation method, and its flow chart is as follows: Figure 4 As shown, including:

[0123] Step S41: determine the target resources required by each large model when processing tasks. Different large models are used to process different types of tasks. The target resources include at least computing resources.

[0124] Step S42: predict the number and type of tasks to be processed in the t+n period based on historical records, where the t period is the current period, t is greater than or equal to 0, and n is greater than 0;

[0125] Step S43, determining the priority queue of each task type;

[0126] Step S44, determining the priority order corresponding to the task type to be processed in the t+nth time period based on the priority queue of each task type;

[0127] Step S45, determining the resource allocation strategy corresponding to the tasks to be processed in the t+nth period according to the priority order, the number of tasks to be processed in the t+nth period, and the target resources required by each large model when processing the tasks;

[0128] Step S46, allocating initial target resources for different large models in the t+n period and allocating pre-stored resources based on the resource allocation strategy, where the pre-stored resources are part of the resources reserved from the total target resources to ensure the processing efficiency of the tasks processed in the t+n period, wherein the tasks are tasks in the cloud computing process, and the computing power resources in the target resources allocated by the resource allocation strategy are the computing power resources of the graphics processor.

[0129] Determine the target resources required for each large model when processing tasks, predict the number and type of tasks to be processed in the t+n period based on historical records, determine the resource allocation strategy corresponding to the tasks to be processed in the t+n period based on the number and type of tasks to be processed in the t+n period and the target resources required for each large model when processing tasks, and allocate the initial target resources and pre-stored resources of different large models in the t+n period based on the resource allocation strategy to ensure that each task to be processed in the t+n period can be processed in a timely and effective manner.

[0130] When determining the resource allocation strategy, the resource allocation strategy for the t+n period is determined based on the predicted priority order of tasks to be executed in the t+n period, the number and type of tasks to be processed in the t+n period, and the target resources required by each large model when processing tasks.

[0131] Among them, the inference tasks applied in the production environment can be divided into online tasks and near-line tasks according to the degree of delay sensitivity. Online tasks are more sensitive to delay, and online tasks with high sensitivity to delay are identified as high-delay-sensitive tasks, which are usually inference tasks that respond to user requests in real time. High-delay-sensitive tasks are the highest priority tasks; near-line tasks are usually batch processing tasks, which have low requirements for the delay of a single inference, but have requirements for the completion time of a batch of data in hours to minutes, and near-line tasks with low delay sensitivity are regarded as low-priority tasks.

[0132] The request processing gateway can be used as the entry point for all inference requests, and the inference requests can be routed to appropriate nodes and containers. When multiple tasks arrive at the entry point at the same time, the execution order can be determined based on the priority order of the multiple tasks. High-priority tasks will get the target resources first, while low-priority tasks need to wait until the high-priority tasks are processed before they can be executed. For example: if the priority of task A is greater than that of task B and task C, task A will be executed first; for example: if the priority of application 1 is greater than that of application 2 and application 3, the priority of the tasks in application 1 is greater than the priority of the tasks in application 2 and the priority of the tasks in application 3, and the tasks in application 1 will be executed first.

[0133] The priority queue of each task is determined according to the delay sensitivity of each task. The priority queue can be predetermined. Then, after predicting the type of task to be processed in the t+nth period, the priority order corresponding to the type of task to be processed in the t+nth period is determined according to the priority queue.

[0134] Furthermore, after determining the priority order corresponding to the task types to be processed in the t+n period, the priority of the task can be dynamically adjusted according to parameters such as the waiting time of the task, the task size or the system load. For example, if the waiting time of task 1 exceeds a certain threshold, the priority of task 1 can be increased; for another example, when there is a shortage of resources, such as: the task execution efficiency of large model A is low and below a certain threshold, the tasks with lower priorities can be suspended, and only the tasks with higher priorities can be executed to avoid the problem of resource preemption and ensure the execution efficiency of high-priority tasks.

[0135] After determining the priority order corresponding to the task types to be processed in the t+n period, the resource allocation strategy for each large model in the t+n period can be determined based on the priority order, the number of tasks to be processed, and the target resources required for each large model to execute the task.

[0136] After determining the priority order corresponding to the task types to be processed in the t+n period, determine the large model that executes the tasks of each task type, then sort the tasks that need to be executed by each large model according to the priority order corresponding to the task type, and determine the number of tasks of each task type in the sorting of each large model, so as to determine the target resources required for each task type in the sorting of each large model under its corresponding number of tasks, and allocate target resources to the large model according to the target resources required for each task type in the sorting of each large model under its corresponding number of tasks.

[0137] For example: in the t+nth period, the tasks that need to be processed by the large model 1 include: type a tasks, type b tasks and type c tasks. According to the priority order corresponding to the task types to be processed in the t+nth period, the priority of the above three tasks can be determined from high to low as: type b tasks, type a tasks, type c tasks, and the number of type b tasks is 5, the number of type a tasks is 10, and the number of type c tasks is 1. The unit target resource required for the large model 1 to execute 1 type b task is k, then the unit target resource required for the large model 1 to execute 5 type b tasks is 5k, the unit target resource required for the large model 1 to execute 1 type a task is m, then the unit target resource required for the large model 1 to execute 10 type a tasks is 10m, and the unit target resource required for the large model 1 to execute 1 type c task is n. Then, it can be determined that the target resources required by large model 1 during the time period of executing type b tasks are 5k, the target resources required during the time period of executing type a tasks are 10m, and the target resources required during the time period of executing type c tasks are n. The t+n period is divided into 3 time periods. The first time period is used to execute type b tasks. At this time, the target resources allocated to large model 1 are 5k. The second time period is used to execute type a tasks. At this time, the target resources allocated to large model 1 are 10m. The third time period is used to execute type c tasks. At this time, the target resources allocated to large model 1 are n. The length of each time period can be predicted based on historical records, where k, m, and n are all positive numbers.

[0138] Alternatively, after determining that the tasks that need to be executed by each large model are sorted according to the priority order corresponding to the task type, for any large model, after determining the target resources required for the large model to execute all the tasks of the highest priority task type according to the number of tasks of the highest priority task type among the tasks that need to be executed by the large model and the unit target resources required for the large model to execute this type of tasks, the target resources are directly allocated according to the target resources required for all the tasks of the highest priority task type, that is, as long as the tasks corresponding to the highest priority task type have sufficient target resources during execution.

[0139] For tasks with lower priority, if all tasks of a certain task type with a lower priority than the highest priority are executed, such as: task type 1 is the highest priority task type to be executed by a large model, and task type 2 is a task type with a lower priority than task type 1, the total target resources required when executing task type 1 are m, and the total target resources required when executing task type 2 are n. If m is greater than n, that is, the target resources required to execute the lower priority task type are less than the target resources required to execute all tasks of the highest priority task type, then mn, that is, the remaining target resources in the target resources allocated to it when the large model executes task type 2, can be used when other large models execute tasks. When there is a shortage of resources, they are allocated to other large models to avoid low efficiency in executing tasks by other large models; if m is less than n, that is, the target resources required to execute lower priority task types are more than the target resources required to execute all tasks of the highest priority task type, at this time, nm, that is, the difference between the target resources required for the large model to execute task type 2 and all target resources allocated to the large model, needs to be allocated nm of resources to the large model to ensure the efficiency of the large model in executing task type 2. At this time, pre-stored resources can be used to provide the large model with target resources to execute task type 2, or the remaining resources of the large model with remaining target resources can be used to provide the large model with target resources to execute task type 2.

[0140] The resource allocation method disclosed in this embodiment, when determining the resource allocation strategy for the tasks to be processed in the t+nth time period, determines the target resources required by each large model when performing task processing, and predicts the number of tasks and task types to be processed in the t+nth time period, and determines the priority order corresponding to the task types to be processed in the t+nth time period, so as to determine the resource allocation strategy corresponding to the tasks to be processed in the t+nth time period based on the priority order, the number of tasks to be processed in the t+nth time period, and the target resources required by each large model when performing task processing, so as to ensure that the target resources allocated to the large models in the t+nth time period according to the resource allocation strategy can support the task execution of the large models and ensure a certain execution efficiency.

[0141] This embodiment discloses a resource allocation method, and its flow chart is as follows: Figure 5 As shown, including:

[0142] Step S51: determine the target resources required by each large model when processing tasks. Different large models are used to process different types of tasks. The target resources include at least computing resources.

[0143] Step S52: predict the number and type of tasks to be processed in the t+n period based on historical records, where the t period is the current period, t is greater than or equal to 0, and n is greater than 0;

[0144] Step S53: Based on the number and type of tasks to be processed in the t+nth period, and according to the target resources required by each large model when processing tasks, determine the resource allocation strategy for the t+nth period;

[0145] Step S54: if it is determined that the time difference between the current time and the initial time of the t+nth period is within a specific range, preload the large model corresponding to the task to be processed in the t+nth period;

[0146] Step S55: allocating initial computing resources and initial video memory resources of different large models in the t+n period based on the resource allocation strategy, and allocating pre-stored resources;

[0147] Step S56: Transmit the data corresponding to the task to be processed in the t+nth time period to the initial video memory resource corresponding to the corresponding large model.

[0148] Determine the target resources required for each large model when performing task processing, predict the number of tasks and task types to be processed in the t+n period based on historical records, and based on the number of tasks and task types to be processed in the t+n period, determine the resource allocation strategy for the t+n period according to the target resources required by each large model when performing task processing, and allocate initial target resources for different large models in the t+n period and allocate pre-stored resources based on the resource allocation strategy to ensure the processing efficiency of the tasks processed in the t+n period.

[0149] After determining the resource allocation strategy for the t+nth period, resource allocation is performed for the t+nth period only when a specific period is reached to ensure that the major models in the t+nth period can execute the inference tasks in a timely manner and will not affect the task execution of the major models in the periods before the t+nth period.

[0150] Determine whether the time difference between the current moment and the initial moment in the t+nth time period is within a specific range, that is, determine whether the time difference between the current moment and the t+nth time period is within a specific range. If it is within the specific range, it indicates that the current moment is close to the t+nth time period, and the resource allocation of the t+nth time period can be executed; if it is not within the specific range, it indicates that the current moment is far from the t+nth time period, and it will take some time to reach the t+nth time period. Therefore, there is no need to execute the resource allocation of the t+nth time period.

[0151] If it is determined that the time difference between the current moment and the t+nth period is within a specific range, the resource allocation for the t+nth period begins. The large model corresponding to the task to be processed in the t+nth period can be preloaded first to ensure that when the t+nth period is reached, the large model is in a loaded state.

[0152] In addition, when it is determined that the time difference between the current moment and the t+nth period is within a specific range, it is also necessary to allocate the initial computing power resources and initial video memory resources of different large models in the t+nth period based on the resource allocation strategy, and allocate pre-stored resources, that is, allocate the initial computing power resources and initial video memory resources required for each large model in the t+nth period, so that after reaching the t+nth period, the large model can directly use the initial target resources allocated to it to perform the corresponding task, ensuring that the large model has sufficient resources to perform the corresponding task.

[0153] In addition, the data corresponding to the task to be processed in the t+n period needs to be transferred to the initial video memory resources corresponding to the corresponding large model, that is, after allocating the corresponding initial computing power resources and initial video memory resources to each large model, the data required for each large model to execute the task also needs to be stored in advance, such as: it is predicted that in the t+n period, task A is executed by large model 1, the initial video memory resources allocated to large model 1 are video memory resources 1, and the data required for task A is data a. Before reaching the t+n period, data a is transferred to video memory resource 1 in advance, so that when the t+n period is reached, the preloaded large model 1 can use the data a in video memory resource 1 to directly execute task A, without the need to load large model 1 after reaching the t+n period, allocate initial target resources to large model 1, and transfer data a to video memory resource 1, thereby improving the efficiency of the large model in processing data in the t+n period, and only when the time difference between the current moment and the initial moment of the t+n period is within a specific range, the above preprocessing process is performed to avoid the problem of insufficient resources when the large model executes tasks in the period before the t+n period.

[0154] Furthermore, the resource allocation method disclosed in this embodiment may also include:

[0155] The difference between the target resource required in the resource allocation policy for the t+nth period and the target resource at the current moment is determined, and the target resource is released for the t+nth period based on the difference.

[0156] That is, when executing resource allocation for the t+nth time period, we can first determine the number of target resources allocated to each model at the current moment, and then predict the number of target resources required by each model in the t+nth time period. Based on the difference between the two, we adjust the resources allocated to each model to ensure that the target resources allocated to each model in the t+nth time period can meet the predetermined resource allocation strategy, avoiding the problem of insufficient resources or waste of resources.

[0157] More specifically, when performing resource allocation for the t+nth time period, the number of target resources allocated to each model at the current moment is determined, wherein the current moment is a moment in a time period before the t+nth time period, and the time period at which the moment is located is the time period closest to the t+nth time period. Between the current moment and the t+nth time period, the number of target resources allocated to each model will not change. Therefore, it is only necessary to ensure that the target resources are released based on the difference between the number of target resources allocated to each model at the current moment and the predicted number of target resources allocated to each model in the t+nth time period. There is no need to recycle all resources and then release them again, thereby ensuring the efficiency of resource allocation.

[0158] The resource allocation method disclosed in this embodiment determines the target resources required by each large model when performing task processing, and the number of tasks and task types to be processed in the t+nth time period predicted based on historical records. Then, based on the number of tasks to be processed and the task type in the t+nth time period, the resource allocation strategy for the t+nth time period is determined according to the target resources required by each large model when performing task processing. When it is determined that the time difference between the current moment and the initial moment of the t+nth time period is within a specific range, resource allocation is performed for the large model applied in the t+nth time period to ensure that resource allocation is completed in advance before reaching the t+nth time period. When reaching the t+nth time period, the large model can be directly used to execute the corresponding resources, thereby ensuring the efficiency of the large model in executing tasks and improving the task processing speed.

[0159] This embodiment discloses a resource allocation method, and its flow chart is as follows: Figure 6 As shown, including:

[0160] Step S61: determine the target resources required by each large model when processing tasks. Different large models are used to process different types of tasks. The target resources include at least computing resources.

[0161] Step S62: Determine the number and type of tasks in different time periods based on historical records;

[0162] Step S63, determining the priority queue of each task type;

[0163] Step S64: configuring a confidence value for each task type based on the priority queue;

[0164] Step S65: predicting the number of tasks and task types to be processed in the t+nth time period based on the number of tasks and task types in different time periods and the confidence values ​​of each task type;

[0165] Step S66: Based on the number and type of tasks to be processed in the t+nth period, and according to the target resources required by each large model when processing tasks, determine the resource allocation strategy for the t+nth period;

[0166] Step S67, allocating initial target resources of different large models in the t+n period based on the resource allocation strategy, and allocating pre-stored resources, where the pre-stored resources are part of the resources reserved from the total target resources to ensure the processing efficiency of the tasks processed in the t+n period.

[0167] Determine the target resources required for each large model when processing tasks, predict the number of tasks and task types to be processed in the t+n period based on historical records, determine the resource allocation strategy for the t+n period based on the number of tasks and task types in the t+n period and the target resources required by each large model when processing tasks, allocate initial target resources for different large models in the t+n period based on the resource allocation strategy, and allocate pre-stored resources to ensure the processing efficiency of tasks processed in the t+n period.

[0168] Among them, the number and type of tasks to be processed in the t+n period are predicted based on historical records. The number and type of tasks in different periods can be determined based on historical records first, and the inferred task volume in different periods within a recording cycle can be estimated through the arrival value of historically monitored tasks of different applications.

[0169] The number of tasks for different types of applications will show different curves and resource requirements over time, such as Figure 7 The figure shows the task volume of application A, application B and application C in a recording cycle. Taking a recording cycle of one day as an example, the number of tasks of application A with the highest priority will increase sharply at a certain time period of the day, while the number of tasks of application B with the second highest priority is relatively stable in the day without obvious fluctuations. Application C with the lowest priority has two fluctuations in the day. Figure 7 In the example, the horizontal axis time is divided into different time periods, and the minimum computing power resources can be allocated and the pre-stored resources can be determined directly according to the predicted value in each time period.

[0170] If an application has the highest priority, then the priority of tasks in the application with the highest priority is higher than the priorities of tasks in other applications. Of course, different tasks in the same application may also have different priorities.

[0171] Since different large models process tasks with different priorities, different confidence values ​​can be configured for large models that perform different tasks when making predictions. The larger model that performs the higher priority task has a higher corresponding confidence value, and the larger model that performs the lower priority task has a lower corresponding confidence value. For example, a set of confidence values ​​is set to [30%, 50%, 70%, 90%], where the confidence value of the large model corresponding to the highest priority task is 90%, the confidence value of the large model corresponding to the second highest priority task is 70%, and the confidence value of the large model corresponding to the lowest priority task is 30%, so as to ensure that the prediction of the higher priority task is more accurate and that sufficient target resources can be allocated to the corresponding large model to ensure the processing efficiency of high priority tasks.

[0172] After configuring corresponding confidence values ​​for each model, the number of tasks and task types in the t+nth time period can be predicted based on the set confidence value and the number of tasks and task types in different time periods determined based on historical records. Alternatively, the number of tasks and task types at the next moment, i.e., the t+nth time period, can be predicted based on the set confidence value and the number of tasks and task types in the time period before the t+nth time period.

[0173] Furthermore, after predicting the number and type of tasks in the t+nth period, the compatibility between the large model and the target resources needs to be considered when determining the resource allocation strategy for the t+nth period.

[0174] The target resource is a resource in a GPU resource pool. For the GPU resource pool, at least the following three resources are available: multi-card joint inference, single GPU inference of different models, and inference using vGPU.

[0175] Among them, multi-GPU joint inference is a technology that uses multiple GPUs for parallel processing to accelerate the inference process of large models. Due to the different scales of parameters and other parameters of different large models, some very large models cannot be loaded into a single GPU. It is necessary to deploy large models on multiple GPUs and use corresponding frameworks to manage synchronization and communication strategies;

[0176] Inference on a single GPU of different models, including mixed models of GPUs from different manufacturers, taking into account the specific characteristics of each GPU, such as computing power, memory bandwidth, etc., and assigning tasks based on the GPU's strengths, such as memory-intensive tasks are assigned to GPUs with larger memory, and more computationally intensive tasks are assigned to GPUs with higher FLOPs (floating point operations per second);

[0177] vGPU (virtual GPU) inference is a technology that allows multiple virtual machines to share a single physical GPU. vGPU can divide the physical GPU into multiple virtual slices, each of which is assigned to a virtual machine, enabling it to independently access and utilize GPU resources. Alternatively, vGPU can use time slicing to allow multiple virtual machines to share a single GPU by dividing processing time into discrete slices. Each slice sequentially allocates part of the GPU's computing and memory resources to different tasks.

[0178] The matching between large models and target resources must not only consider computing power, storage bandwidth, and memory capacity, but also factors such as deployment methods and compatibility.

[0179] For example, large model 1 can apply multi-card joint inference and vGPU inference, but cannot use the resources of a single GPU inference of different models to perform tasks. Large model 2 can apply multi-card joint inference, vGPU inference, and use part of the resources of a single GPU inference of different models to perform tasks.

[0180] According to the predicted number of tasks to be processed in the t+n period and the type of tasks, target resources are allocated to different large models, including the initial target resources and pre-stored resources allocated to each large model. When allocating resources, it is necessary to consider the computing power and video memory required by each large model to perform tasks, as well as the compatibility between the large model and the target resources. For high-priority tasks, the corresponding target resources can be allocated according to the upper limit of the confidence value. For tasks other than high-priority tasks, a value within the confidence value can be selected to allocate target resources. Since the prediction may be inaccurate, it is also necessary to reserve certain resources to deal with emergencies, that is, reserve resources.

[0181] During the task processing process, tasks, large models and target resources are combined according to the target resources, resource allocation strategies and compatibility between large models and target resources required by each large model when performing task processing, and the best hybrid strategy and computing power management are achieved through combined optimization.

[0182] The goal of combinatorial optimization is to improve the target resources' ability to support the elastic demands of large model tasks and to improve inference efficiency. The target resources' ability to support the elastic demands of large model tasks can be represented by the number of tasks assigned to the target resources corresponding to the large model, and the inference efficiency can be represented by the input word length of the task, the average output word length of the task, and the average word generation time of the large model.

[0183] Specifically, the optimization goal can be achieved through the following constraints: the tasks of each application must be assigned to the corresponding large model; the video memory required by the large model assigned to the GPU does not exceed the maximum video memory of the GPU; the computing power resources required by the large model assigned to the GPU does not exceed the maximum computing power of the GPU; the compatibility of the GPU and the large model; the tasks assigned to the model do not exceed its processing capacity; the inference time caused by the queuing of tasks does not exceed the maximum threshold; the predicted number of tasks in the t+n period does not exceed the processing capacity of the initial target resources and initial reserved resources corresponding to the large model; the sum of the pre-stored resources allocated to different applications does not exceed the total amount of pre-stored resources; the GPUs allocated to different models do not exceed the number of compatible GPU / vGPU resources; a large model Specific models of GPUs may be required for joint inference. A large model can improve resource utilization and system processing power through vGPU virtualization. Different large models run in different GPU environments, and the inference time of high-priority tasks does not exceed the requirements of SLAs (service level agreements). According to predictions, under a certain type of task request scale, the corresponding number of GPUs / vGPUs and large model instances meets the resource requirements under a certain confidence prediction, thereby obtaining different total cache capacities. Since different large models may not be loaded on any cache GPU, different types of GPU caches are required to load the corresponding large models. Based on predictions, it is necessary to determine whether the large model processing capacity for specific tasks in the t+n period can be expanded through pre-stored resources.

[0184] The optimization goals are: minimize the loading time of large models corresponding to different tasks, minimize the inference time of large models corresponding to high-priority tasks, minimize the inference time of large models corresponding to different tasks that does not meet the SLAs (Service Level Agreement) level, minimize the number of large model loading and unloading times in the t+n period; minimize the time interval between releasing pre-stored resources and reloading them to different large models, etc.

[0185] The resource allocation method disclosed in this embodiment can monitor the indicators of target resources through tools such as NVIDIA or Zabbix. The indicators that are regularly checked may include: utilization, memory usage, temperature, power consumption, etc. Task scheduling is adjusted according to the currently set utilization threshold, and tasks are dynamically allocated based on real-time data.

[0186] Furthermore, the schematic diagram of the resource allocation method disclosed in this embodiment can be as follows: Figure 8As shown, based on the historical records of each application, the number and type of tasks to be processed in the t+n period are predicted, and based on this, the initial target resources of different large models in the t+n period are allocated, and the pre-stored resources are allocated; then, the request processing gateway obtains the inference request input by each application, obtains the tasks to be processed, determines the priority order of the tasks to be processed, and then combines and optimizes the job scheduling and resource allocation, and then each large model uses the target resources to perform the corresponding tasks; and the target resources can also be monitored regularly to adjust the task scheduling.

[0187] The resource allocation method disclosed in this embodiment, when predicting the number of tasks and task types to be processed in the t+n period based on historical records, configures a confidence value for each task type based on the priority queue of each task type, and adds the confidence value parameter during the prediction to ensure the accuracy of the predicted number of tasks and task types to be processed in the t+n period, and further includes effective utilization of resources in the resource allocation process.

[0188] This embodiment discloses an electronic device, and its structural diagram is as follows: Fig. 9 As shown, including:

[0189] Processor 91 and memory 92.

[0190] The processor 91 is used to determine the target resources required by each large model when processing tasks, and different large models are used to process different types of tasks, and the target resources include at least computing resources; based on historical records, the number of tasks to be processed and the type of tasks in the t+n period are predicted, where the t period is the current period, t is greater than or equal to 0, and n is greater than 0; based on the number of tasks to be processed and the type of tasks in the t+n period, according to the target resources required by each large model when processing tasks, the resource allocation strategy for the t+n period is determined; based on the resource allocation strategy, the initial target resources of different large models in the t+n period are allocated, and pre-stored resources are allocated, and the pre-stored resources are part of the resources reserved from the total target resources to ensure the processing efficiency of the tasks processed in the t+n period;

[0191] The memory 92 is used to store the programs required by the processor to execute the above processing process.

[0192] The electronic device disclosed in this embodiment is implemented based on the resource allocation method disclosed in the above embodiments, which will not be described in detail here.

[0193] The electronic device disclosed in the present embodiment determines the target resources required by each large model when performing task processing, and different large models are used to process different types of tasks, and the target resources include at least computing resources; predicts the number of tasks to be processed and the type of tasks in the t+n period based on historical records, wherein the t period is the current period, t is greater than or equal to 0, and n is greater than 0; determines the resource allocation strategy for the t+n period based on the number of tasks to be processed and the type of tasks in the t+n period and the target resources required by each large model when performing task processing; allocates initial target resources for different large models in the t+n period based on the resource allocation strategy, and allocates pre-stored resources, wherein the pre-stored resources are part of the resources reserved from the total target resources, and are used to ensure the processing efficiency of the tasks processed in the t+n period. This scheme predicts the number and type of tasks to be processed in the t+n period, and determines the resource allocation strategy based on the number and type of tasks to be processed in the t+n period and the target resources required by each large model when processing the tasks. Based on this, the initial target resources of different large models in the t+n period and the allocated pre-stored resources are allocated to ensure that each task to be processed in the t+n period can be executed by the corresponding large model according to its required target resources, so as to ensure that the large model has sufficient resources to execute the corresponding tasks and avoid resource waste and insufficient resources.

[0194] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0195] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0196] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0197] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A resource allocation method, comprising: Determine the target resources required by each large model when processing tasks, where different large models are used to process different types of tasks, and the target resources include at least computing resources; Predict the number and type of tasks to be processed in the t+n period based on historical records, where the t period is the current period, t is greater than or equal to 0, and n is greater than 0; Based on the number and type of tasks to be processed in the t+nth period, and according to the target resources required by each large model when processing tasks, determine the resource allocation strategy for the t+nth period; Based on the resource allocation strategy, the initial target resources of different large models in the t+n period are allocated, and the pre-stored resources are allocated. The pre-stored resources are part of the resources reserved from the total target resources to ensure the processing efficiency of the tasks processed in the t+n period.

2. The method according to claim 1, further comprising: In the t+nth time period, if it is determined that the first large model satisfies the target condition in executing the target task, at least part of the pre-stored resources are allocated to the first large model, so that the first large model executes the target task using the initial target resources and at least part of the pre-stored resources; The target conditions include: the efficiency of the first large model in executing the target task is lower than a first threshold, and / or the number of tasks executed by the first large model is greater than a second threshold.

3. The method according to claim 1, further comprising: In the t+nth time period, if it is determined that the first large model performs the target task and satisfies the target condition, the target task is transferred to the second large model so that the second large model performs the target task; The target condition includes at least one of the following: the efficiency of the first large model in executing the target task is lower than a first threshold, and / or the number of tasks executed by the first large model is greater than a second threshold.

4. The method according to claim 3, wherein transferring the target task to the second largest model comprises: Determine a second large model whose compatibility value with the target task reaches a third threshold based on a preset large model and task matching relationship table, wherein the large model and task matching relationship table can characterize the compatibility values ​​of different large models for different tasks; The target task is transferred to the second large model.

5. The method according to claim 4, wherein determining the second large model whose compatibility value with the target task reaches a third threshold based on a preset matching relationship table between the large model and the task comprises: Determine a large model whose compatibility value with the target task reaches a third threshold based on a preset large model and task matching relationship table; If it is determined that the number of tasks currently executed by the large model whose compatibility value with the target task reaches the third threshold does not reach the second threshold, the large model whose compatibility value with the target task reaches the third threshold is determined as the second large model.

6. The method according to claim 1, wherein the resource allocation strategy for the t+nth period is determined based on the number and type of tasks to be processed in the t+nth period and according to the target resources required by each large model when processing tasks, comprising: Determine the priority queue for each task type; Determine the priority order corresponding to the task type to be processed in the t+nth time period based on the priority queue of each task type; Determine the resource allocation strategy corresponding to the tasks to be processed in the t+nth period according to the priority order, the number of tasks to be processed in the t+nth period, and the target resources required by each large model when processing the tasks; The task is a task in a cloud computing process, and the computing power resources in the target resources allocated by the resource allocation strategy are computing power resources of a graphics processor.

7. The method according to claim 1, allocating the initial target resources of different large models in the t+n period based on the resource allocation strategy, and allocating the pre-stored resources, comprises: If it is determined that the time difference between the current moment and the initial moment of the t+nth period is within a specific range, preloading the large model corresponding to the task to be processed in the t+nth period; Allocate initial computing resources and initial video memory resources of different large models in the t+n period based on the resource allocation strategy, and allocate pre-stored resources; The data corresponding to the task to be processed in the t+nth time period is transferred to the initial video memory resource corresponding to the corresponding large model.

8. The method according to claim 7, further comprising: Determine the difference between the target resources required in the resource allocation strategy for the t+nth time period and the target resources at the current moment; The target resource is released for the t+nth time period based on the difference.

9. The method according to claim 1, wherein the predicting the number and type of tasks to be processed in the t+n period based on historical records comprises: Determine the number and type of tasks in different time periods based on the historical records; Determine the priority queue for each task type; Configuring a confidence value for each task type based on the priority queue; The number of tasks and the type of tasks to be processed in the t+n period are predicted based on the number of tasks and the type of tasks in the different time periods and the confidence value of each task type.

10. An electronic device comprising: A processor is used to determine the target resources required by each large model when processing tasks, where different large models are used to process different types of tasks, and the target resources include at least computing resources; predict the number of tasks to be processed and the type of tasks in the t+n period based on historical records, where the t period is the current period, t is greater than or equal to 0, and n is greater than 0; determine the resource allocation strategy for the t+n period based on the number of tasks to be processed and the type of tasks in the t+n period and the target resources required by each large model when processing tasks; allocate initial target resources for different large models in the t+n period based on the resource allocation strategy, and allocate pre-stored resources, where the pre-stored resources are part of the resources reserved from the total target resources to ensure the processing efficiency of the tasks processed in the t+n period; The memory is used to store the program required by the processor to execute the above processing process.

Images