Large model application-oriented task distribution method and system

Through the combination of message queue and optimal matching algorithm, the problems of unreasonable task distribution, performance bottlenecks and low availability in large model systems are solved, efficient task scheduling and multi-model management are realized, and the throughput and availability of large models are improved. .

CN118939395APending Publication Date: 2024-11-12BEIJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410970947.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing locally deployed large-model systems have low throughput, inability to support multiple users, lack of multi-model management and scheduling capabilities, and insufficient hardware monitoring.

Method used

The task is distributed to the request distributor using the user-configured grouping strategy, and the task execution components are scheduled through the optimal matching algorithm to achieve efficient distribution and management of tasks.

Benefits of technology

It improves task scheduling efficiency and availability, makes full use of large-model inference optimization methods, supports large-model management and efficient task scheduling of multiple nodes, and supports expansion or reduction of large-model task types.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939395A_ABST
    Figure CN118939395A_ABST
Patent Text Reader

Abstract

The invention provides a large model application-oriented task distribution method and system, and relates to the technical field of resource scheduling, and the method comprises the following steps: distributing different types of tasks to a request distributor by using a message queue; grouping the tasks of the same type by using a grouping strategy configured by a user to obtain different task groups; and scheduling different task groups to an execution component for execution based on an optimal matching algorithm. According to the method, multiple large model tasks are scheduled, the most important batch processing method in large model reasoning optimization means can be fully utilized, the large models deployed at multiple nodes can be effectively managed, the large model tasks can be distributed according to different request types, the large models can be efficiently scheduled to execute specific tasks, and the task scheduling efficiency is improved. And expansion or reduction of large model task types is also supported.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of resource scheduling, and in particular to a task distribution method and system for large model applications. Background Art

[0002] With the development of artificial intelligence technology, the use scenarios of big models are becoming more and more extensive. In addition to the basic question-and-answer function itself, more and more applications are developing rich functions based on the capabilities of big models for people to use, such as AI Agent. In addition to using the API interfaces provided by big model companies, more and more developers tend to deploy locally based on open source big models, such as ChatGLM and Meta Llama. For the local deployment of big models, users often need to pay attention to the use, management and expansion of big models. Existing local deployment frameworks, such as Hugging Face's TextGeneration Inference framework, often only focus on the use and deployment of a single big model. However, in actual situations, the low throughput of a single locally deployed big model makes it impossible to support the use of multiple users. Therefore, a system that supports the use, scheduling and management of multiple locally deployed big models is the cornerstone of supporting applications based on locally deployed big models.

[0003] For the management and use of multiple large models, the system solves the following problems. First, the system needs to be able to manage and use different types of large models to perform tasks. Secondly, the cost of executing a task with a large model is relatively high, and the system needs to be able to use the batch processing method in large model optimization to optimize the efficiency of large model task execution. At the same time, large model reasoning frameworks such as vLLM and TextGeneration Inference also provide optimization for the execution of a single large model task, and the system needs to have the ability to support the use of the above frameworks. Finally, the operation of large models requires a large amount of GPU video memory. The system needs to be able to monitor the running status of the large model and the hardware on which it runs, and be able to quickly locate problems after they occur.

[0004] Based on the above-mentioned problems, it is necessary to develop a task distribution method and system for large model applications. Summary of the invention

[0005] In order to overcome the deficiencies of the prior art, the object of the present invention is to provide a task distribution method and system for large model applications.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A task distribution method for large model applications, comprising:

[0008] Use message queues to distribute different types of tasks to request dispatchers;

[0009] Use the grouping strategy configured by the user to group the tasks of the same type to obtain different task groups;

[0010] Different task groups are scheduled to task execution components for execution based on the optimal matching algorithm.

[0011] Preferably, a message queue is used to distribute different types of tasks to the request dispatcher, including:

[0012] Use HTTP gateway to set a routing key and a request ID for receiving user requests; each routing key identifies a type of task; each request ID identifies a user request;

[0013] Use a topic exchange for message queues, binding each routing key to a queue;

[0014] Use each request dispatcher to accept and consume tasks from the corresponding queue; the number of request dispatchers is the same as the number of queues.

[0015] Preferably, tasks of the same type are grouped using a grouping strategy configured by a user to obtain different task groups, including:

[0016] Based on the time-priority grouping strategy, the unexecuted tasks are grouped according to the time sequence of the unexecuted tasks to obtain a task group; or

[0017] Based on the limited length grouping strategy, the length from 0 to the maximum length of the allowed task is divided into intervals group; where MAX_INPUT_LENGTH represents the maximum length and delta represents the interval; each time a task arrives, it is added to a group of corresponding length according to the length of the task, and the tasks in the same group are sorted in the order of arrival time to obtain a task group.

[0018] Preferably, after grouping the tasks of the same type using the grouping strategy configured by the user to obtain different task groups, the method further includes:

[0019] Set the key value of the task group to the arrival time of the first task in the task group and sort them from small to large according to the timestamp value. Select the group with the smallest key value as the scheduling group by the request distributor.

[0020] Preferably, the request distributor manages several task execution components; each task execution component encapsulates a large model program execution task; the number of batch processing rows is determined based on the GPU video memory allocated by the administrator for each large model program; the request distributor is also used to select the task execution component for the scheduling group using the optimal matching algorithm based on the number of scheduling group rows and the number of batch processing rows of different task execution components.

[0021] Preferably, an optimal matching algorithm is used to select a task execution component for the scheduling group, including:

[0022] Calculate the scheduling key value for each task execution component; the calculation formula of the scheduling key value is: SchedulingKey = BatchSize - SchedulingGroupSize; wherein SchedulingKey represents the scheduling key value, BatchSize represents the number of batch processing rows, and SchedulingGroupSize represents the number of scheduling group rows;

[0023] Select the task execution component with a scheduling key value greater than or equal to 0 and the smallest scheduling key value to execute the tasks in the scheduling group, and set the scheduling group to empty; if all scheduling key values ​​are less than 0, select the task execution component with the largest number of tasks in the first batch of processing rows of the scheduling group, remove the tasks that have been sent to the task execution component from the scheduling group, and reset the key value of the scheduling group to the minimum value of the timestamp of the tasks in the current group.

[0024] A task distribution system for large model applications, characterized in that it is applied to the above-mentioned task distribution method for large model applications, and the task distribution system for large model applications comprises:

[0025] A task scheduler, which is used to accept task requests and distribute and schedule tasks to task execution components;

[0026] Multiple task execution components, each task execution component includes a large model program encapsulated by a task execution component framework, and is used to execute one or a group of requests assigned by a request distributor and return a processing result.

[0027] Preferably, the task scheduler comprises:

[0028] HTTP gateway, used to receive and manage user requests;

[0029] Message queue, used to cache and distribute tasks and receive responses;

[0030] A request distributor for scheduling task groups and task execution components using an optimal matching algorithm;

[0031] Request sending buffer, used to group tasks based on grouping strategy;

[0032] The scheduling policy manager is used to configure system information for managing grouping policies.

[0033] Preferably, the HTTP gateway comprises:

[0034] The interface unit is used to provide an HTTP interface, receive the dialogue request sent by the user side, assign a unique ID to the question raised by the user to the big model, and send it to the message queue;

[0035] A question storage unit is used to obtain answers from the answer queue of the message queue, determine the question to which the answer belongs according to the question ID in the answer message, determine whether the message has been answered according to the sequence ID in the answer message, and store the questions that have been answered in the database;

[0036] The answer determination unit is used to provide a responsive HTTP interface, allowing the user to responsively return the latest answer according to the sequence ID when the question has not been answered until the answer is completed, or the user obtains the answer to the question at one time when the question is answered.

[0037] Preferably, the request distributor comprises:

[0038] The management unit is used to manage multiple task execution components of the same type, maintain the status of each task execution component, and perform heartbeat detection and error recovery;

[0039] A transceiver unit, used for receiving a problem message from a message queue and sending it to a request-send buffer;

[0040] The scheduling unit is used to schedule task groups and task execution components using an optimal matching algorithm.

[0041] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0042] The present invention provides a task distribution method and system for large model applications, the method comprising: using a message queue to distribute different types of tasks to a request distributor; using a user-configured grouping strategy to group tasks of the same type to obtain different task groups; and scheduling different task groups to execution components for execution based on an optimal matching algorithm. The present invention solves the technical problems of unreasonable scheduling process, scheduler performance bottleneck, insufficient scalability and low availability in the prior art, and improves task scheduling efficiency and availability. Using the method provided by the present invention to schedule multiple large model tasks can make full use of the most important batch processing method in large model reasoning optimization means, can effectively manage large models deployed on multiple nodes, can distribute large model tasks according to different request types, can efficiently schedule large models to perform specific tasks, and also supports the expansion or reduction of large model task types. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A schematic diagram of a method flow chart provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a technical route provided for an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the structure of a task distribution system provided by an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of a task scheduling structure provided by an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of a scheduling strategy provided by an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of the execution component structure provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Figure 1 A schematic diagram of a method flow chart provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides a task distribution method for large model applications, comprising:

[0053] Step 100: Use the message queue to distribute different types of tasks to the request distributor;

[0054] Step 200: grouping tasks of the same type using a grouping strategy configured by a user to obtain different task groups;

[0055] Step 300: Scheduling different task groups to task execution components for execution based on an optimal matching algorithm.

[0056] Optionally, the method of this embodiment is implemented by a task scheduler, such as Figure 2 As shown, including:

[0057] The system pre-processes user requests and assigns a unique ID to the request

[0058] Use RabbitMQ message queue to distribute different types of tasks to request dispatchers;

[0059] Use user-configured grouping strategies to group tasks of the same type;

[0060] Different task groups are scheduled to execution units (task execution components) for execution based on the optimal matching algorithm.

[0061] Furthermore, RabbitMQ message queue is used to distribute different types of tasks to the request distributor, which is characterized by including:

[0062] The HTTP gateway sets a routing key and a request ID for the received user request. Each routing key identifies a type of task, and each request ID identifies a user request.

[0063] Use RabbitMQ's Topic Exchange and bind each routing key to a queue;

[0064] The number of request dispatchers is the same as the number of queues, and each request dispatcher can consume tasks from the corresponding queue.

[0065] Furthermore, tasks of the same type are grouped using a grouping strategy configured by a user, which is characterized by including:

[0066] Based on the time-priority grouping strategy, the unexecuted tasks are grouped according to the order of their arrival time;

[0067] Based on the limited length grouping strategy, the length from 0 to the maximum length of the allowed task is divided into (MAX_INPUT_LENGTH) according to the interval (delta). Group. Every time a task arrives, it is added to a group of corresponding length according to its length, and the tasks in the same group are sorted in the order of arrival time;

[0068] For the above task group, no matter which grouping method is used, the key value of the group is set to the arrival time of the first task in the group and sorted from small to large according to the timestamp value. The group with the smallest key value is selected as the scheduling group by the request distributor.

[0069] Furthermore, the method of this embodiment is implemented by a request distributor, including:

[0070] The request distributor manages several task execution components. Each task execution component encapsulates a large model program execution task and determines the number of batch processing rows based on the GPU video memory allocated by the administrator for each large model program.

[0071] The request distributor uses the best matching algorithm to select a task execution component for the scheduling group based on the number of scheduling group rows (SchedulingGroupSize) and the number of batch rows (BatchSize) of different task execution components. That is, for each task execution component, SchedulingKey = BatchSize – SchedulingGroupSize is calculated. The task execution component with SchedulingKey ≥ 0 and the smallest SchedulingKey is selected to execute the tasks in the scheduling group, and the scheduling group is set to empty. If all SchedulingKeys are less than 0, the task execution component with the largest SchedulingKey is selected to execute the first BatchSize tasks of the scheduling group, and the tasks that have been sent to the task execution component for execution are removed from the scheduling group, and the key value of the scheduling group is reset to the minimum value of the timestamp of the tasks in the current group.

[0072] like Figure 3 As shown, a large model task distribution system of the above method in this embodiment includes:

[0073] A task scheduler, configured to accept task requests and distribute and schedule tasks to execution components;

[0074] Multiple execution components, each of which includes a large model program encapsulated by an execution component framework, and is configured to execute one or a group of requests assigned by a request distributor and return a processing result.

[0075] The task scheduler includes:

[0076] HTTP gateway, configured to receive and manage user requests;

[0077] Message queue, configured to cache and distribute tasks and receive responses;

[0078] A request distributor configured to schedule task groups and task execution components using an optimal matching algorithm;

[0079] The request sending buffer is configured to group tasks based on the grouping strategy;

[0080] A scheduling policy manager configured to manage grouping policies and configure system information;

[0081] Specifically, the HTTP gateway is used to:

[0082] Provides HTTP interface to receive dialogue requests sent by users, assigns a unique ID to the questions asked by users to the big model, and sends it to the message queue;

[0083] Get the answer from the answer queue of the message queue, determine the question to which the answer belongs based on the question ID in the answer message, determine whether the message has been answered based on the sequence ID in the answer message, and store the questions that have been answered in the database;

[0084] Provide a responsive HTTP interface, allowing users to responsively return the latest answer based on the sequence ID when the question has not been answered until the answer is completed, or the user can get the answer to the question at once when the question is answered;

[0085] Furthermore, the message queue is specifically used for:

[0086] Provide multiple problem queues, which support messages in character and binary formats. Use different keys to route problem messages to the problem queues. Each problem queue has only one request distributor as a consumer.

[0087] Provide a reply queue, which supports character and binary messages, receives the reply message sent by the execution component and sends it to the request manager;

[0088] Furthermore, the request distributor is specifically used to:

[0089] Manages multiple execution components of the same type, maintains the status of each execution component, and performs heartbeat detection and error recovery;

[0090] Receive the problem message from the message queue and send it to the request send buffer;

[0091] Use the optimal matching algorithm to schedule task groups and task execution units.

[0092] Furthermore, the request send buffer is specifically used for:

[0093] Cache problem messages and group tasks according to the grouping strategy configured by the scheduling strategy manager;

[0094] Furthermore, the scheduling policy manager is specifically used to:

[0095] Provide administrators with manual management of grouping strategies for request sending buffers corresponding to different request distributors;

[0096] Provide administrators with system configurations such as the maximum length of dialog input, the maximum length of output, and the maximum batch length supported by each execution component;

[0097] Specifically, the multiple execution components of this embodiment are specifically used for:

[0098] like Figures 4 to 6 As shown, this embodiment introduces the execution component framework implemented by this system to implement interfaces including large model task processing, heartbeat detection, and error recovery; it provides performance monitoring functions to monitor the running status of large models, occupied GPU resources, idle GPU resources, etc.

[0099] Specifically, this embodiment provides a task distribution system for large model applications, including a task scheduler 200 and a task execution component, wherein the task scheduler 200 includes an HTTP gateway, a message queue 100, a request distributor 300, a request sending buffer, and a scheduling policy management;

[0100] The HTTP gateway receives the user's task execution request, sets a unique ID for each request, selects a routing key according to the request type provided in the request, and then sends the request to the message queue. The HTTP gateway also receives the "response" message in the "response" queue in the message queue, and determines the request corresponding to the response according to the request ID of the "response" message. If this request does not exist, discard this "response" message. At the same time, the responses are sorted according to the sequence number of the "response" in the "response" queue. The response with a sequence number of 0 indicates that this response is the first response of the large model, and the response with a sequence number of 1 indicates that this response is the second response of the large model, and so on. When the sequence number is -1, it means that the response is over. When the response sequence number is -1, check whether the corresponding sequence number conforms to the form of 0, 1, 2, ..., n-1, n, -1. If it conforms, the above response results are combined in order as the final response to the user's request. If it does not conform, it is necessary to wait for the response of the missing sequence number. The HTTP gateway also provides a responsive interface. When the user obtains the result of the request, if the corresponding serial number conforms to the aforementioned 0,1,2,…,n-1,n,-1 format, the combined result is returned to the user. If it does not conform to this format, the response is based on the following rules: If a response with serial number 0 exists, this response is returned; then, if a response with serial number 1 exists, this response is returned; and so on, until a response with serial number m does not exist, then wait for this response until the response is obtained from the "response" queue and then returned to the user. Finally, if the serial number is -1, the response is returned and the user request ends.

[0101] The message queue 100 can distribute and cache requests and cache responses. The message queue 100 includes a request topic switch, several request queues, a response topic switch and a response queue. The request topic switch is bound to all request queues through different routing keys, and each request queue is bound to the request topic switch through a unique routing key. The response topic switch and the response queue are bound through a routing key. The HTTP request module sends the pre-processed request to the request distributor 300 via the request topic switch and the request queue corresponding to the routing key. The execution component sends the response of the large model to the HTTP gateway via the response topic switch and the corresponding queue.

[0102] The request distributor 300 is capable of managing and scheduling execution components. The request distributor 300 maintains a list of execution component addresses and uses the gRPC protocol to connect to the execution components. The request distributor 300 maintains the status of the execution components through timed heartbeat detection and the return results of the execution tasks. For execution components in an error state, the fault recovery interface of the execution component is called to restore the status of the execution component. The request distributor 300 can obtain the current execution index of the execution component through the index interface of the execution component. Through the index, the administrator can observe the specific operation status of the execution component. The request distributor 300 also has the function of scheduling execution components. The request distributor 300 obtains the highest priority task group from the request sending buffer, and uses the optimal matching algorithm to select the optimal execution component to execute the task.

[0103] The optimal matching algorithm calculates the scheduling key value SchedulingKey = BatchSize – SchedulingGroupSize for each task execution component based on the number of scheduling group rows (SchedulingGroupSize) and the number of batch processing rows (BatchSize) of different task execution components. The task execution component with SchedulingKey ≥ 0 and the smallest SchedulingKey is selected to execute the tasks in the scheduling group, and the scheduling group is set to empty. If all SchedulingKeys are less than 0, the task execution component with the largest SchedulingKey is selected to execute the first BatchSize tasks of the scheduling group, and the tasks that have been sent to the task execution component for execution are removed from the scheduling group, and the key value of the scheduling group is reset to the minimum value of the timestamps of the tasks in the current group.

[0104] The request sending buffer can cache tasks and use grouping strategies to group tasks. The grouping strategies include time-based grouping strategies and length-based grouping strategies. The time-based grouping strategy is to group the unexecuted tasks according to the order of their arrival time; the length-based grouping strategy is to divide the task length from 0 to the maximum allowed task length (MAX_INPUT_LENGTH) into groups according to the interval (delta). Each time a task arrives, it is added to a group of corresponding length according to its length, and the tasks in the same group are sorted in the order of arrival time; for the above task group, no matter which grouping method is used, the key value of each group is set to the arrival time of the first task in the group and sorted from small to large according to the timestamp value, and the group with the smallest key group is selected as the scheduling group by the request distributor 300.

[0105] The scheduling strategy management can manage the grouping strategy of the request sending buffer, and configure the system configuration such as the execution component address and the maximum batch processing length managed and scheduled by the request distributor 300 of the system.

[0106] The execution component is used to manage and execute specific large models. The execution component implements the large model task execution interface, heartbeat detection interface, fault recovery interface and indicator acquisition interface by introducing the execution component library of this system. By implementing the above interfaces, the system has the ability to manage and execute specific large models.

[0107] In the above scheme, tasks are grouped by request sending buffer, and the most important batch processing method in the large model reasoning optimization method is realized by means of the optimal matching algorithm of task group scheduling used by the request distributor 300. For a group of tasks sent by the request distributor 300 to the execution unit, the execution unit can use the batch processing interface of the large model to process a group of tasks at the same time. This greatly improves the efficiency of large model processing.

[0108] In the above scheme, the segmented return of the response is realized by separating the request and the return, so that the user can see part of the result when the task is executed, which improves the user experience. The request distributor 300 sends the task to the specific execution unit. At this time, the request distributor 300 does not need to pay attention to the result of the execution unit, but only pays attention to the status of the execution unit. The execution unit is responsible for sending the result of the task to the message queue 100. In this way, the segmented return of the response can be supported.

[0109] In the above scheme, the message queue 100 is used to distribute the request and multiple request distributors 300 are used, each request distributor 300 manages a large model task. This method simplifies the task distribution and management process, so that a type of large model task can be distributed only by routing keys, so that the system supports the simultaneous management and scheduling of multiple types of large model tasks. At the same time, this management method also simplifies the expansion and deletion operations of large model tasks.

[0110] Furthermore, if the administrator wishes to add a type of large model task, the administrator only needs to add a request distributor 300 and configure the corresponding execution components for the request distributor 300, and then add a request queue and set the routing key and bind it to the request topic switch.

[0111] Furthermore, if the administrator wishes to remove a class of large model tasks, the administrator only needs to cancel the binding of the routing key and the request topic switch, and remove the corresponding request queue, request distributor 300 and execution component.

[0112] This embodiment also provides a computer-readable storage medium, including instructions, which, when executed on a computer, enable the computer to execute the task distribution method provided by the embodiment of the present invention.

[0113] This embodiment also provides a computer program product including instructions. When the computer program product is run on a computer, the computer is enabled to execute the task distribution method provided by the embodiment of the present invention.

[0114] Those skilled in the art can understand that all or part of the steps of the above embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a computer, all or part of the steps of the above embodiments can be executed. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0115] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] The beneficial effects of the present invention are as follows:

[0117] The present invention can be used to schedule multiple large model tasks, making full use of the most important batch processing method in large model reasoning optimization, effectively managing large models deployed on multiple nodes, distributing large model tasks according to different request types, efficiently scheduling large models to perform specific tasks, and supporting the expansion or reduction of large model task types. Based on the above beneficial effects, the present invention greatly simplifies the localized management and use of large models.

[0118] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0119] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A task distribution method for large model applications, characterized in that: include: Use message queues to distribute different types of tasks to request dispatchers; Use the grouping strategy configured by the user to group the tasks of the same type to obtain different task groups; Different task groups are scheduled to task execution components for execution based on the optimal matching algorithm.

2. The task distribution method for large model applications according to claim 1 is characterized in that: Use message queues to dispatch different types of tasks to request dispatchers, including: Use HTTP gateway to set a routing key and a request ID for receiving user requests; each routing key identifies a type of task; each request ID identifies a user request; Use a topic exchange for message queues, binding each routing key to a queue; Use each request dispatcher to accept and consume tasks from the corresponding queue; the number of request dispatchers is the same as the number of queues.

3. The task distribution method for large model applications according to claim 1 is characterized in that: Use the grouping strategy configured by the user to group tasks of the same type to obtain different task groups, including: Based on the time-priority grouping strategy, the unexecuted tasks are grouped according to the time sequence of the unexecuted tasks to obtain a task group; or Based on the limited length grouping strategy, the length from 0 to the maximum length of the allowed task is divided into intervals group; where MAX_INPUT_LENGTH represents the maximum length and delta represents the interval; each time a task arrives, it is added to a group of corresponding length according to the length of the task, and the tasks in the same group are sorted in the order of arrival time to obtain a task group.

4. The task distribution method for large model applications according to claim 3 is characterized in that: After the tasks of the same type are grouped using the grouping strategy configured by the user to obtain different task groups, the following is also included: Set the key value of the task group to the arrival time of the first task in the task group and sort them from small to large according to the timestamp value. Select the group with the smallest key value as the scheduling group by the request distributor.

5. The task distribution method for large model applications according to claim 1 is characterized in that: The request distributor manages several task execution components; each task execution component encapsulates a large model program execution task; the number of batch processing rows is determined based on the GPU video memory allocated by the administrator for each large model program; the request distributor is also used to select the task execution component for the scheduling group using the optimal matching algorithm based on the number of scheduling group rows and the number of batch processing rows of different task execution components.

6. The task distribution method for large model applications according to claim 5 is characterized in that: Use the best matching algorithm to select task execution components for the scheduling group, including: Calculate the scheduling key value for each task execution component; the calculation formula of the scheduling key value is: SchedulingKey = BatchSize - SchedulingGroupSize; wherein SchedulingKey represents the scheduling key value, BatchSize represents the number of batch processing rows, and SchedulingGroupSize represents the number of scheduling group rows; Select the task execution component with a scheduling key value greater than or equal to 0 and the smallest scheduling key value to execute the tasks in the scheduling group, and set the scheduling group to empty; if all scheduling key values ​​are less than 0, select the task execution component with the largest number of tasks in the first batch of processing rows of the scheduling group, remove the tasks that have been sent to the task execution component from the scheduling group, and reset the key value of the scheduling group to the minimum value of the timestamp of the tasks in the current group.

7. A task distribution system for large model applications, characterized in that: The task distribution method for large model applications according to any one of claims 1 to 6, wherein the task distribution system for large model applications comprises: A task scheduler, which is used to accept task requests and distribute and schedule tasks to task execution components; Multiple task execution components, each task execution component includes a large model program encapsulated by a task execution component framework, and is used to execute one or a group of requests assigned by a request distributor and return a processing result.

8. The task distribution system for large model applications according to claim 7, characterized in that: The task scheduler comprises: HTTP gateway, used to receive and manage user requests; Message queue, used to cache and distribute tasks and receive responses; A request distributor for scheduling task groups and task execution components using an optimal matching algorithm; Request sending buffer, used to group tasks based on grouping strategy; The scheduling policy manager is used to configure system information for managing grouping policies.

9. The task distribution system for large model applications according to claim 8, characterized in that: The HTTP gateway includes: The interface unit is used to provide an HTTP interface, receive the dialogue request sent by the user side, assign a unique ID to the question raised by the user to the big model, and send it to the message queue; A question storage unit is used to obtain answers from the answer queue of the message queue, determine the question to which the answer belongs according to the question ID in the answer message, determine whether the message has been answered according to the sequence ID in the answer message, and store the questions that have been answered in the database; The answer determination unit is used to provide a responsive HTTP interface, allowing the user to responsively return the latest answer according to the sequence ID when the question has not been answered until the answer is completed, or the user obtains the answer to the question at one time when the question is answered.

10. The task distribution system for large model applications according to claim 8, characterized in that: The request distributor comprises: The management unit is used to manage multiple task execution components of the same type, maintain the status of each task execution component, and perform heartbeat detection and error recovery; A transceiver unit, used for receiving a problem message from a message queue and sending it to a request-send buffer; The scheduling unit is used to schedule task groups and task execution components using an optimal matching algorithm.

Citation Information

Cited By

  • Request scheduling method, server and storage medium

    CN122293744A

  • Task scheduling method in commodity operation service system, and electronic device

    WO2026208542A1