AIGC model resource scheduling system and method based on task sharing

Abstract

The invention discloses an AIGC model resource scheduling system and method based on task sharing. The AIGC model resource scheduling system based on task sharing comprises an information acquisition module, a cluster resource management module and a model scheduling module. The information acquisition module is used for acquiring model resource demand information; the cluster resource management module is used for acquiring server resource information of equipment in a cluster; the model scheduling module is used for generating a scheduling equipment sequence queue based on the model resource demand information and the server resource information, and matching and deploying a plurality of AIGC models to corresponding computing cards; the objective is to use up computing power and video memory resources of equipment as much as possible. Therefore, the AIGC model resource scheduling system based on task sharing can reasonably allocate the models with different resource demands to the same computing card, and realizes efficient utilization of computing power and video memory resources while ensuring that the tasks meet the service level target.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and particularly to an AIGC model resource scheduling system and method based on task sharing. Background Art

[0002] The development of Artificial Intelligence Generated Content (AIGC) technology has been rapid. Especially since ChatGPT (Chat Generative Pre-trained Transformer) was introduced at the end of 2022, innovative applications of various AIGC models have emerged in an endless stream. The wide application of these models in multiple industries not only promotes the popularization of artificial intelligence technology but also injects unprecedented innovative impetus into the industry, becoming the mainstream trend of current technological development.

[0003] The successful deployment, stable operation, and efficient inference of AIGC models highly depend on powerful computing device support, including hardware facilities such as Graphics Processing Unit (GPU) and Neural-network Processing Unit (NPU). In addition, enterprises providing services need to deploy the inference models involved in numerous user AIGC requests to the server cluster through model scheduling strategies to provide users with diverse services.

[0004] These user requests often cover multiple industry fields, including education, medical, finance, entertainment, etc. The requirements for the Service Level Objective (SLO) vary greatly in each industry, with different technical requirements and service standards in terms of response speed and reliability. Therefore, service providers need to precisely optimize scheduling strategies and resource allocation plans, not only to ensure the efficient utilization of computing power and video memory resources during model inference but also to ensure that all services can meet the corresponding SLO standards, so as to provide users with high-quality and stable service experiences.

[0005] With the further iteration of AIGC models and the continuous improvement of hardware support, its application potential and industry influence will continue to expand, bringing more possibilities for the digital and intelligent transformation of various fields.

[0006] Currently, there are two common solutions for AIGC model cluster scheduling: (1) Exclusive resource mode: Allocate a single model to a computing card in an exclusive resource manner; (2) Hybrid deployment mode: Schedule multiple models to a single card for hybrid deployment; however, both of these solutions have drawbacks.

[0007] Specifically: (1) In the exclusive mode, each model fully occupies the computing power and video memory resources of a specific computing card. However, this scheduling method often causes the model to be unable to fully utilize these two resources, leading to the problem of resource fragmentation and increased costs.

[0008] (2) In the hybrid deployment mode, the system can allocate multiple models to the same computing card to provide inference services. These models cover various tasks from compute-intensive to storage-intensive. However, existing scheduling strategies often concentrate tasks of the same type on a single computing card, leading to two problems: one is violating the service level objective (SLO); for example, when two compute-intensive tasks are scheduled to the same computing card, the excessive contention for computing power between tasks will significantly increase the inference latency, thereby resulting in the violation of SLO and reducing the service quality. The other is that resources are not fully utilized. For example, when two storage-intensive tasks are scheduled to the same device, the computing power often cannot be effectively used, causing expensive computing power to be idle and increasing costs.

[0009] Current AIGC models can be classified into the following two categories according to the degree of scheduling flexibility: one category is models with relatively fixed computing and video memory resource requirements, such as Residual Neural Network (ResNet), Bidirectional Encoder Representation from Transformers (Bert), etc.; another category of models can flexibly adjust resource requirements through certain inference optimization techniques. For example, in the Generative Pretrained Transformer model, due to the need to compute a large amount of intermediate data (such as Key data and Value data), the demand for computing power is relatively high (compute-intensive). To optimize performance, existing frameworks introduce the Key-Value Cache (KV Cache) technology. By the method of "replacing computing with storage", part of the Key-Value (KV) data is stored in the video memory, thus avoiding the computation of this part of the data and reducing the demand for computing power. However, this method significantly increases the demand for video memory resources (storage-intensive).

[0010] Therefore, there is an urgent need for a new AIGC model resource scheduling scheme based on task sharing. Summary of the Invention

[0011] One advantage of this application is to provide an AIGC model resource scheduling system and method based on task sharing. Among them, the AIGC model resource scheduling system and method based on task sharing can reasonably allocate models with different resource requirements to the same computing card, and while ensuring that tasks meet the service level objective (SLO), achieve the efficient utilization of computing power and video memory resources.

[0012] Another advantage of the present application is to provide an AIGC model resource scheduling system and method based on task sharing. Among them, the AIGC model resource scheduling system and method based on task sharing can be compatible with and support relatively fixed models and models that flexibly adjust resource requirements through inference optimization technology.

[0013] According to one aspect of the present application, an AIGC model resource scheduling system based on task sharing is provided, which includes: An information collection module for collecting model resource requirement information, where the model requirement information includes computing power requirements and video memory requirements; A cluster resource management module for obtaining server resource information of devices in the cluster, where the server resource information includes server computing power information and server video memory information; and A model scheduling module for generating a scheduling device sequence queue based on the model resource requirement information and the server resource information, and matching and deploying multiple AIGC models to corresponding computing cards; Among them, the objective function in the process of matching and deploying multiple AIGC models to corresponding computing cards is: ; where represents the number of tasks waiting to be scheduled in the scheduling device sequence queue; represents the number of key-value buffer policies for task under the condition of meeting the user service level target, and each key-value buffer policy corresponds to a cache ratio; is greater than or equal to 1. For models that do not support key-value buffer policies, is 1; for models that support key-value buffer policies, is greater than 1; represents setting the key-value cache policy for task to ; represents task 's computing power requirement under the key-value cache policy ; represents task 's video memory requirement under the key-value cache policy ; represents the remaining computing power of the target device; represents the remaining video memory of the target device; The constraint conditions include: ; ; 。

[0014] In an embodiment of the AIGC model resource scheduling system based on task sharing according to the present application, the model scheduling module is further configured to preprocess the user request before generating the scheduling device sequence queue. During the preprocessing of the user request, the AIGC models that cannot meet the user service level target are excluded.

[0015] In an embodiment of the AIGC model resource scheduling system based on task sharing according to the present application, during the process of generating the scheduling device sequence queue, the model scheduling module sequentially assigns the tasks corresponding to the user request to the computing cards with existing running tasks.

[0016] In an embodiment of the AIGC model resource scheduling system based on task sharing according to the present application, during the process of sequentially assigning the tasks corresponding to the user request to the computing cards with existing running tasks, first, the remaining computing power of the computing cards is sorted in ascending order; when the remaining computing power of the computing cards is equal, further sorting is performed in ascending order according to the video memory resource amount.

[0017] In an embodiment of the AIGC model resource scheduling system based on task sharing according to the present application, the information collection module is configured to pre-run the AIGC model and perform offline inference on the AIGC model to obtain the model resource requirement information.

[0018] In an embodiment of the AIGC model resource scheduling system based on task sharing according to the present application, the information collection module is further configured to collect the acquisition model resource requirement information under different cache data ratios.

[0019] In an embodiment of the AIGC model resource scheduling system based on task sharing according to the present application, the cluster resource management module is further configured to offline obtain the server resource information of the devices in the cluster.

[0020] In an embodiment of the AIGC model resource scheduling system based on task sharing according to the present application, the cluster resource management module is further configured to maintain the task information and computing power information of each computing card in real time.

[0021] According to another aspect of the present application, the present application proposes an AIGC model resource scheduling method based on task sharing, which includes the steps of: Collect model resource requirement information, where the model requirement information includes computing power requirement and video memory requirement; Obtain the server resource information of the devices in the cluster, where the server resource information includes server computing power information and server video memory information; and Generate a scheduling device sequence queue based on the model resource requirement information and the server resource information, and match and deploy multiple AIGC models to corresponding computing cards; Among them, the objective function in the process of matching and deploying multiple AIGC models to corresponding computing cards is: ; where represents the number of tasks waiting to be scheduled in the scheduling device sequence queue; represents the number of key-value buffering policies for task under the condition of meeting the user service level target, and each key-value buffering policy corresponds to a caching ratio; is greater than or equal to 1. For models that do not support key-value buffering policies, is 1; for models that support key-value buffering policies, is greater than 1; represents that for task , its key-value caching policy is set to ; represents task under the key-value caching policy the computing power requirement; represents task under the key-value caching policy the video memory requirement; represents the remaining computing power of the target device; represents the remaining video memory of the target device; The constraint conditions include: ; ; .

[0022] Through the understanding of the following description and the drawings, the further objectives and advantages of the present application will be fully reflected.

[0023] These and other objectives, features and advantages of the present application are fully reflected through the following detailed description, drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] By describing the embodiments of the present application in more detail in conjunction with the drawings, the above and other objectives, features and advantages of the present application will become more obvious. The drawings are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation to the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1 Illustrates a schematic structural diagram of an AIGC model resource scheduling system based on task sharing according to an embodiment of the present application.

[0026] Figure 2 The figure shows a schematic block diagram of a cluster to which the AIGC model resource scheduling system based on task sharing according to an embodiment of the present application is applied.

[0027] Figure 3 The figure shows a schematic flow chart of task execution of the AIGC model resource scheduling system based on task sharing according to an embodiment of the present application.

[0028] Figure 4 The figure shows a schematic flow chart of model scheduling of the AIGC model resource scheduling system based on task sharing according to an embodiment of the present application.

[0029] Figure 5 The figure shows a schematic flow chart of the AIGC model resource scheduling method based on task sharing according to an embodiment of the present application. Detailed implementation manners

[0030] Next, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments of the present application. It should be understood that the present application is not limited by the exemplary embodiments described herein.

[0031] It can be understood that the term "a" should be understood as "at least one" or "one or more". That is, in one embodiment, the number of an element can be one, while in other embodiments, the number of the element can be multiple. The term "a" cannot be understood as a limitation on the number. "Multiple" means greater than or equal to two.

[0032] Although ordinal numbers such as "first", "second", etc. will be used to describe various components, those components are not limited herein. The term is only used to distinguish one component from another. For example, the first component can be called the second component, and similarly, the second component can also be called the first component without departing from the teachings of the concept of the present application. The term "and / or" used herein includes any and all combinations of one or more of the associated listed items.

[0033] The terms used herein are only for the purpose of describing various embodiments and are not intended to be limiting. As used herein, the singular form also includes the plural form unless the context clearly indicates otherwise. Additionally, it will be understood that the terms "comprising" and / or "having" when used in this specification specify the presence of the stated features, numbers, operations, components, elements, or combinations thereof, without excluding the presence or addition of one or more other features, numbers, operations, components, elements, or combinations thereof.

[0034] As Figures 1 to 4 shown, the AIGC model resource scheduling system based on task sharing according to an embodiment of the present application is illustrated. As Figure 1As shown in the figure, the task-sharing-based AIGC model resource scheduling system includes an information collection module 10, a model scheduling module 30, and a cluster resource management module 20.

[0035] The information collection module 10 is communicatively connected to multiple AIGC models, and is used to pre-run the AIGC models in the task-sharing-based AIGC model resource scheduling system during the initialization phase of the task-sharing-based AIGC model resource scheduling system, perform offline inference on the AIGC models, and collect model resource requirement information; among them, the model resource requirement information includes computing power requirements and video memory requirements. The computing power requirement refers to the minimum computing power requirement of the AIGC model; the video memory requirement refers to the minimum video memory requirement of the AIGC model. The unit of computing power requirement is floating-point operations per second (FLOPS); the unit of video memory requirement is gigabyte (GB); the unit of service level target is millisecond (ms).

[0036] Specifically, the information collection module 10 performs offline inference on common AIGC models in the business through the method of offline pre-running, so as to capture the AIGC model resource requirement information of the AIGC model under the condition of meeting the preset service level objective (SLO), and obtain a model resource requirement table (as shown in Table 1), including computing power requirements and video memory requirements. The preset service level objective can be designed according to requirements. In an example of this application, the preset service level objective is 170 ms.

[0037]

[0038] For AIGC models with flexible adjustable resource requirements, the task-sharing-based AIGC model resource scheduling system further collects the model resource requirements under different cache data ratios, as shown in Table 2, including the requirements of the AIGC model for computing power and video memory. The cache data ratio can adopt the KV cache data ratio. The cache data ratio in the table can be represented by α, and the unit is %; the unit of inference time is millisecond (ms).

[0039] The cluster resource management module 20 is communicatively connected to multiple servers in the cluster, and is used to obtain server resource information from each server in the cluster; among them, the server resource information includes server computing power information and server video memory information. In an example of this application, the cluster includes n servers; each server has 8 computing powers. The 8 computing powers are respectively represented by G1, G2, G3, G4, G5, G6, G7, and G8.

[0040] Specifically, the server computing power information pre-acquired by the cluster resource management module 20 includes the number of servers in the cluster, the type of computing cards, and the number of computing cards on each server, as shown in Table 3; where Table 3 is the server resource table.

[0041]

[0042] In addition, the cluster resource management module 20 is also used to maintain the task information and computing power information on each computing card in real time. For example, the number of tasks, the remaining computing power, and the remaining video memory resource amount, as shown in Table 4, where Table 4 is the computing card resource table. When the task execution ends and the resources are released, the cluster resource management module 20 will statistically calculate the latest device resource situation in real time and update the resource table.

[0043]

[0044] During the process of the information collection module 10 performing offline inference on the AIGC model, the user submits a model inference request to the AIGC model resource scheduling system based on task sharing; correspondingly, the AIGC model resource scheduling system based on task sharing receives the model inference request submitted by the user. The model inference request includes the model category and the samples required for inference, such as pictures, inference prompts, and inference prompt questions, etc.

[0045] After the model inference request enters the AIGC model resource scheduling system based on task sharing, the model scheduling module 30 selects appropriate tasks for combination based on the AIGC model resource requirement information collected by the information collection module 10, and combines the server resource information provided by the cluster resource management module 20 to match and deploy multiple AIGC models to suitable computing cards, as Figure 3 shown. When task scheduling is required, it is necessary to make full use of the computing power and video memory resources on each computing card as much as possible to improve the resource utilization efficiency while meeting the SLO of the user's task.

[0046] Specifically, the model scheduling module 30 is the core module of the AIGC model resource scheduling system based on task sharing, and is used for request preprocessing, resource perception, model combination, and model deployment.

[0047] As Figure 4 shown, the model scheduling module 30 first adds the request submitted by the user to the queue. In the request preprocessing stage, the model scheduling module 30 eliminates the AIGC models that cannot meet the SLO and terminates the service for this request.

[0048] In the resource awareness stage, the model scheduling module 30 sequentially assigns the tasks corresponding to the user requests to the computing cards with existing running tasks to maximize the utilization of fragmented resources. Specifically, the model scheduling module 30 first arranges the computing cards in ascending order according to the remaining computing power; if the remaining computing power of the computing cards is equal, it is further sorted in ascending order according to the video memory resource amount to generate a "scheduling device sequence queue".

[0049] After obtaining the scheduling device sequence queue, select the first device from the queue, obtain the remaining resource information of the device, and then try to deploy multiple tasks in combination to the device. Such a combination strategy faces two major challenges: (1) how to select appropriate models to maximize the utilization of computing power and video memory resources; (2) for models that support KV Cache, how to adjust the KV buffer ratio of the model to adaptively adjust the computing power and video memory requirements.

[0050] Based on this, the present application designs a task combination algorithm based on linear programming. Table 5 is a variable table, showing the description of the variables. The variable data is obtained by offline collection of the information collection module 10 and the cluster resource management module 20.

[0051]

[0052] When the goal is to try to exhaust the computing power and video memory resources of the device to improve the utilization rate of computing power and video memory, the objective function can be defined as: ; Further, this objective function can be converted to: ; where, is greater than or equal to 1.

[0053] The constraint conditions are as follows: Constraint condition 1: Each requested model can only select one buffer ratio for configuration; expressed by the relational expression as: .

[0054] Constraint condition 2: The total computing power requirement of the combined models does not exceed the remaining computing power of the device; expressed by the relational expression as: .

[0055] Constraint condition 3: The total video memory requirement of the combined models does not exceed the remaining video memory of the device; expressed by the relational expression as: .

[0056] Constraint condition 4: The decision variable is a binary variable; expressed by the relational expression as: .

[0057] The model scheduling module 30 deploys the model to the device for inference tasks according to the task combination strategy; subsequently, it updates the numerical value of the scheduling device sequence queue N and sequentially calls the next device to be scheduled from the scheduling device sequence queue, and repeats the combination strategy based on the objective function and the above constraints until all tasks are processed.

[0058] Through the above objective function and constraints, the AIGC model resource scheduling system based on task sharing of the present application can, while ensuring the SLO, make full use of the computing power and video memory resources in the cluster devices, improve resource utilization rate, and save costs; and can be compatible with and support relatively fixed models and models that flexibly adjust resource requirements through inference optimization technology, and selectively configure the KV buffer strategy.

[0059] According to the working mechanism of the above AIGC model resource scheduling system based on task sharing, as Figure 5 shown, the present application proposes an AIGC model resource scheduling method based on task sharing, which includes the steps of: S110, collecting model resource requirement information, where the model requirement information includes computing power requirement and video memory requirement; S120, obtaining the server resource information of the devices in the cluster, where the server resource information includes server computing power information and server video memory information; S130, based on the model resource requirement information and the server resource information, generating a scheduling device sequence queue, and matching and deploying multiple AIGC models to the corresponding computing cards; where the objective function in the process of matching and deploying multiple AIGC models to the corresponding computing cards is: ; where represents the number of tasks waiting to be scheduled in the scheduling device sequence queue; represents the number of key-value buffer strategies for task under the condition of meeting the user service level target, and each key-value buffer strategy corresponds to a cache ratio; is greater than or equal to 1. For models that do not support the key-value buffer strategy, is 1; for models that support the key-value buffer strategy, is greater than 1; represents that for task , its key-value cache strategy is set to ; represents task under the key-value cache strategy The computing power requirement; represents task under the key-value cache strategy The video memory requirement; Indicates the remaining computing power of the target device; Indicates the remaining video memory of the target device; The constraints include: ; ; .

[0060] In summary, the AIGC model resource scheduling system and method based on task sharing are elucidated. The AIGC model resource scheduling system and method based on task sharing can achieve efficient utilization of computing power and video memory resources while ensuring the SLO, saving costs; and can be compatible with and support relatively fixed models and models that flexibly adjust resource requirements through inference optimization technology.

[0061] The above description of the present application and its implementation manners is not restrictive. What is shown in the drawings is only one of the implementation manners of the present application, and the actual structure is not limited thereto. Generally speaking, if those of ordinary skill in the art are inspired by it and without departing from the creative purpose of the present application, structures and embodiments similar to this technical solution are designed without creative efforts, they should all fall within the protection scope of the present application.

Claims

1. An AIGC model resource scheduling system based on task sharing, characterized in that: include: An information collection module, used to collect model resource demand information, wherein the model demand information includes computing power demand and video memory demand; A cluster resource management module, used to obtain server resource information of devices in the cluster, wherein the server resource information includes server computing power information and server video memory information; and A model scheduling module, used to generate a scheduling device sequence queue based on the model resource demand information and the server resource information, and match and deploy multiple AIGC models to corresponding computing cards; Among them, the objective function in the process of matching and deploying multiple AIGC models to the corresponding computing cards is: ;in, Indicates the number of tasks waiting to be scheduled in the sequential queue of the scheduling device; Indicates that under the premise of meeting the user service level target, The number of key-value buffering strategies, each of which corresponds to a cache ratio; Greater than or equal to 1, for models that do not support the key-value buffering strategy, is 1; for models that support the key-value buffering strategy, Greater than 1; Indicates that for the task , set its key-value cache strategy to ; Indicates the task In the key-value cache strategy The computing power requirements under Indicates the task In the key-value cache strategy The video memory requirements are as follows: Indicates the remaining computing power of the target device; Indicates the remaining video memory of the target device; the constraints include: ; ; .

2. The AIGC model resource scheduling system based on task sharing according to claim 1, wherein: The model scheduling module is also used to pre-process user requests before generating a scheduling device sequential queue. During the pre-processing of user requests, AIGC models that cannot meet the user service level target are eliminated.

3. The AIGC model resource scheduling system based on task sharing according to claim 2, wherein: In the process of generating the sequential queue of the scheduling device, the model scheduling module sequentially allocates the tasks corresponding to the user requests to the computing cards that have already run the tasks.

4. The AIGC model resource scheduling system based on task sharing according to claim 3, wherein: When the model scheduling module allocates tasks corresponding to user requests to computing cards that are already running tasks, it first arranges them in ascending order based on the remaining computing power of the computing cards; when the remaining computing power of the computing cards is equal, they are further sorted in ascending order based on the amount of video memory resources.

5. The AIGC model resource scheduling system based on task sharing according to claim 1, wherein: The information collection module is used to pre-run the AIGC model and perform offline reasoning on the AIGC model to obtain the model resource requirement information.

6. The AIGC model resource scheduling system based on task sharing according to claim 1, wherein: The information collection module is also used to collect collection model resource demand information under different cache data ratios.

7. The AIGC model resource scheduling system based on task sharing according to claim 1, wherein: The cluster resource management module is also used to obtain server resource information of devices in the cluster offline.

8. The AIGC model resource scheduling system based on task sharing according to claim 1, wherein: The cluster resource management module is also used to maintain the task information and computing power information of each computing card in real time.

9. A resource scheduling method for an AIGC model based on task sharing, characterized in that: Includes steps: Collecting model resource requirement information, wherein the model requirement information includes computing power requirement and video memory requirement; Obtaining server resource information of devices in the cluster, wherein the server resource information includes server computing power information and server video memory information; and Based on the model resource requirement information and the server resource information, a scheduling device sequence queue is generated, and multiple AIGC models are matched and deployed to corresponding computing cards; Among them, the objective function in the process of matching and deploying multiple AIGC models to the corresponding computing cards is: ;in, Indicates the number of tasks waiting to be scheduled in the sequential queue of the scheduling device; Indicates that under the premise of meeting the user service level target, The number of key-value buffering strategies, each of which corresponds to a cache ratio; Greater than or equal to 1, for models that do not support the key-value buffering strategy, is 1; for models that support the key-value buffering strategy, Greater than 1; Indicates that for the task , set its key-value cache strategy to ; Indicates the task In the key-value cache strategy The computing power requirements under Indicates the task In the key-value cache strategy The video memory requirements are as follows: Indicates the remaining computing power of the target device; Indicates the remaining video memory of the target device; the constraints include: ; ; .

Images