Scheduling method, scheduling device, computing cluster and system

Abstract

The invention discloses a scheduling method, a scheduling device, a computing cluster and a system, relates to the technical field of communication, and is used for reducing the time delay of LLM reasoning, meeting the time delay requirement of a single text unit and improving the throughput and computing power utilization rate of equipment. The method is used for scheduling a first computing cluster instance and a second computing cluster instance, and comprises the steps that a first request of a target application is sent to the first computing cluster instance, and the first request is used for requesting to calculate first input information; receiving first indication information from the first computing cluster instance, wherein the first indication information is used for indicating that the first text unit is completed; and in response to the first indication information, sending a second request to the second computing cluster instance, and sending a third request of the target application to the first computing cluster instance, the second request being used for indicating to compute the first text unit, and the third request being used for requesting to compute the second input information.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a scheduling method, a scheduling device, a computing cluster and a system. Background Art

[0002] Large language model (LLM) refers to a type of language model based on neural networks with large-scale parameters (usually billions or more). LLM can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and is an important path to artificial intelligence. At present, LLM mainly includes the chat generative pre-trained transformer (GPT) series, the LLAMA (large language modelmeta AI) series, and the generalized linear model (GLM) series.

[0003] The basic architecture of the above models all use a decoder-only transformer architecture. The LLM of this type of architecture uses an auto-aggressive generative mode when inferring and predicting, that is, the generation of each text unit (token) requires a forward reasoning process of the LLM model, so a sentence including M tokens requires M forward reasoning processes. The above models are usually deployed on a heterogeneous accelerated computing cluster with multiple processing units, and are executed using the parallel computing capabilities of the heterogeneous accelerated computing cluster.

[0004] However, when using this heterogeneous accelerated computing cluster to perform calculations, how to reduce the latency of LLM reasoning and improve the throughput and computing power utilization of the device is still a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a scheduling method, a scheduling device, a computing cluster and a system for reducing the latency of LLM reasoning, meeting the latency requirements of a single text unit, and improving the throughput and computing power utilization of the device.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, a scheduling method is provided for scheduling a computing cluster, which may be a heterogeneous accelerated computing cluster, and which includes a first computing cluster (for example, a pre-filling computing cluster) and a second computing cluster (for example, a decoding computing cluster), the first computing cluster includes M first computing cluster instances, the second computing cluster includes N second computing cluster instances, and M and N are positive integers. The method includes: sending a first request for a target application (for example, an LLM application) to the first computing cluster instance, the first request including first input information (for example, text information), the first request being used to request calculation of the first input information, such as pre-filling calculation; receiving first indication information from the first computing cluster instance, the first indication information being used to indicate that a first text unit has been completed, the first text unit being obtained by the first computing cluster instance calculating the first input information; in response to the first indication information, sending a second request to the second computing cluster instance, and sending a third request for the target application to the first computing cluster instance, the second request being used to indicate calculation of the first text unit (for example, decoding calculation), the third request including the second input information, and the third request being used to request calculation of the second input information. Optionally, the M first computing cluster instances and the N second computing cluster instances communicate via an interconnection device.

[0008] In the above technical solution, a first computing cluster including M first computing cluster instances and a second computing cluster including N second computing cluster instances are deployed separately, and then the first computing cluster instances and the second computing cluster instances are scheduled to perform different calculations according to the request of the target application. After the first computing cluster instance is processed, the first computing cluster instance can continue to be scheduled to calculate the third request without waiting for the execution of the second computing cluster instance to be completed. Therefore, when this method is applied to LLM reasoning, the latency of LLM reasoning can be reduced, the latency requirements of a single text unit can be met, and the throughput and computing power utilization of the device can be improved.

[0009] In a possible implementation of the first aspect, the first text unit is stored in a local cache of a first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance, and the first identifier is used by the second computing cluster instance to obtain the first text unit from the first computing cluster instance. In the above possible implementation, the local cache of the first computing cluster instance can be used to store the text units generated by the computing cluster instance during the computing process, thereby reducing the storage delay of the text units.

[0010] In a possible implementation of the first aspect, the method includes: receiving second indication information from a second computing cluster instance, the second indication information being used to indicate that the first text unit has been acquired; sending first notification information to the first computing cluster instance according to the second indication information, the first notification information being used to instruct the first computing cluster instance to release a first storage space storing the first text unit in the local cache. In the above possible implementation, after acquiring the first text, the second computing cluster instance sends the second indication information to a scheduling device in the general processing device, and the scheduling device notifies the first computing cluster instance to release the first storage space storing the first text unit according to the second indication information, thereby improving the utilization rate of the storage space in the first computing cluster instance.

[0011] In a possible implementation of the first aspect, the first computing cluster instance and the second computing cluster instance also communicate with a storage device (for example, communicate through an interconnection device), the first text unit is stored in the storage device, and the second request includes storage indication information, and the storage indication information is used for the second computing cluster instance to obtain the first text unit from the first computing cluster instance. In the above possible implementation, using the storage device to store the text units generated by the computing cluster instance during the computing process can adaptively meet the storage requirements brought about by the growth of business traffic and longer input information.

[0012] In a possible implementation of the first aspect, the method includes: receiving second indication information from a second computing cluster instance, the second indication information being used to indicate that the first text unit has been acquired; sending a second notification information to the storage device according to the second indication information, the second notification information being used to instruct the storage device to release a second storage space storing the first text unit. In the above possible implementation, after acquiring the first text, the second computing cluster instance sends a second indication information to a scheduling device in the general processing device, and the scheduling device notifies the storage device to release the second storage space storing the first text unit according to the second indication information, thereby improving the utilization rate of the storage space in the storage device. In addition, using the storage device to store the text units generated by the computing cluster instance during the computing process can adaptively meet the storage requirements brought about by the growth of business traffic and longer input information.

[0013] In a possible implementation of the first aspect, the number of the M first computing cluster instances and the number of the N second computing cluster instances are determined based on data characteristics of the target application, and the data characteristics include the length of input information, the length of output information, and the number of iterations of decoding calculations. In the above possible implementation, the number of first computing cluster instances and the number of second computing cluster instances are determined based on the data characteristics of the target application, so as to improve the adaptability of deploying the first computing cluster and the second computing cluster, and at the same time ensure that the first computing cluster and the second computing cluster meet load balancing.

[0014] In a possible implementation of the first aspect, the method further includes: initializing the M first computing cluster instances; initializing the N second computing cluster instances. Optionally, initializing interconnection devices between the M first computing cluster instances and the N second computing cluster instances.

[0015] In a possible implementation of the first aspect, when the first preset condition is met, the method further includes: if the computing cluster also includes X expanded first computing cluster instances, the X first computing cluster instances are initialized, and the interconnection device between the X first computing cluster instances and the N second computing cluster instances can be further initialized, where X is a positive integer; or, X second computing cluster instances among the N second computing cluster instances are configured as X first computing cluster instances, and the interconnection device between the X first computing cluster instances and the remaining NX second computing cluster instances can be initialized. In the above possible implementation, according to the change of the data characteristics of the target application, the number of first computing cluster instances can be dynamically increased, thereby improving the responsiveness of the business request of the target application and enhancing the cost and performance ratio. In addition, in the deployment device, by changing the role of some second computing cluster instances to the first computing cluster instances, the computing cluster instances whose roles have been changed can also share the model weights in the pre-filled calculation and decoding calculation process, thereby avoiding the transfer of model weights.

[0016] Optionally, the first preset condition includes: the local cache occupancy of the M first computing cluster instances is greater than a preset occupancy, or the computing load of the M first computing cluster instances is greater than a first load threshold.

[0017] In a possible implementation of the first aspect, when the second preset condition is met, the method further includes: if the computing cluster also includes Y expanded second computing cluster instances, the Y second computing cluster instances are initialized, and the interconnection device between the Y second computing cluster instances and the M first computing cluster instances can be further initialized; or, Y first computing cluster instances among the M first computing cluster instances are configured as Y second computing cluster instances, and the interconnection device between the Y second computing cluster instances and the remaining MY first computing cluster instances can be initialized. In the above possible implementation, according to the change of data characteristics of the target application, the number of second computing cluster instances can be dynamically increased, thereby improving the responsiveness of the business request of the target application and enhancing the cost and performance ratio. In addition, in the deployment device, by changing the role of some first computing cluster instances to second computing cluster instances, the computing cluster instances whose roles have been changed can also share the model weights in the pre-filled calculation and decoding calculation process, thereby avoiding the transfer of model weights.

[0018] Optionally, the second preset condition includes: the local cache occupancy of the N second computing cluster instances is greater than a preset occupancy, or the computing load of the N second computing cluster instances is greater than a first load threshold.

[0019] In a possible implementation of the first aspect, the target application includes a large language model (LLM) application, the first computing cluster is a computing cluster for performing pre-filling calculations, and the second computing cluster is a computing cluster for performing decoding calculations. In the above possible implementations, when the scheduling method is applied to LLM reasoning, the latency of LLM reasoning can be reduced, the latency requirements of a single text unit can be met, and the throughput and computing power utilization of the device can be improved.

[0020] In a second aspect, a scheduling method is provided for scheduling a computing cluster, the computing cluster comprising a first computing cluster and a second computing cluster, the first computing cluster comprising M first computing cluster instances, the second computing cluster comprising N second computing cluster instances, M and N being positive integers, the method comprising: the first computing cluster instance receiving a first request of a target application sent by a general processing device, the first request comprising first input information, the first request being used to request calculation of the first input information; the first computing cluster instance calculating the first input information to obtain a first text unit according to the first request; the first computing cluster instance sending a first indication information to the general processing device, the first indication information being used to indicate that the first text unit has been completed; in response to the first indication information, the first computing cluster instance receiving a third request of the target application sent by the general processing device, the third request comprising second input information, the third request being used to request calculation of the second input information; the second computing cluster instance receiving a second request sent by the general processing device, the second request being used to indicate calculation of the first text unit. Optionally, the M first computing cluster instances and the N second computing cluster instances communicate via an interconnection device.

[0021] In a possible implementation of the second aspect, the first text unit is stored in a local cache of a first computing cluster instance, the second request includes a first identifier of the first computing cluster instance, and the method further includes: the second computing cluster instance obtains the first text unit from the first computing cluster instance through the interconnection device according to the first identifier of the first computing cluster instance.

[0022] In a possible implementation of the second aspect, the method also includes: the second computing cluster instance sends a second indication message to the general processing device, the second indication message is used to indicate that the first text unit has been acquired; the first computing cluster instance receives a first notification message sent by the general processing device, the first notification message is sent by the general processing device according to the second indication message; the first computing cluster instance releases the first storage space storing the first text unit in the local cache according to the first notification message.

[0023] In a possible implementation of the second aspect, the first computing cluster instance and the second computing cluster instance also communicate with a storage device (for example, through an interconnection device), and the second request includes storage indication information. The method also includes: the first computing cluster instance stores the first text unit in the storage device; and the second computing cluster instance obtains the first text unit from the storage device according to the storage indication information.

[0024] In a possible implementation of the second aspect, the method also includes: the second computing cluster instance sends a second indication message to the general processing device, the second indication message is used to indicate that the first text unit has been acquired; wherein the second indication message is used by the general processing device to send a second notification message to the storage device, and the second notification message is used by the storage device to release a first storage space storing the first text unit.

[0025] In a possible implementation manner of the second aspect, the method further includes: the first computing cluster instance performs an initialization operation; and the second computing cluster instance performs an initialization operation.

[0026] In a possible implementation manner of the second aspect, the method further includes: the first computing cluster instance receives first configuration information sent by the general processing device, where the first configuration information is used to configure the first computing cluster instance as a second computing cluster instance.

[0027] In a possible implementation manner of the second aspect, the method further includes: the second computing cluster instance receives second configuration information sent by the general processing device, where the second configuration information is used to configure the first computing cluster instance as the second computing cluster instance.

[0028] In a possible implementation manner of the second aspect, the target application includes a large language model (LLM) application, the first computing cluster is a computing cluster for performing pre-filling calculations, and the second computing cluster is a computing cluster for performing decoding calculations.

[0029] In a third aspect, a scheduling method is provided for scheduling a computing cluster, the computing cluster comprising a first computing cluster and a second computing cluster, the first computing cluster comprising M first computing cluster instances, the second computing cluster comprising N second computing cluster instances, M and N being positive integers, the method comprising: a general processing device sending a first request for a target application to the first computing cluster instance, the first request comprising first input information, the first request being used to request calculation of the first input information; the first computing cluster instance receiving the first request, and calculating the first input information to obtain a first text unit according to the first request; the first computing cluster instance sending a first indication information to the general processing device, the first indication information being used to Indicate that the first text unit has been completed; the general processing device receives first indication information from the first computing cluster instance, the first indication information is used to indicate that the first text unit has been completed; the general processing device responds to the first indication information, sends a second request to the second computing cluster instance, and sends a third request of the target application to the first computing cluster instance, the second request is used to indicate that the first text unit is to be calculated, the third request includes second input information, and the third request is used to request that the second input information be calculated; the first computing cluster instance receives the third request, and calculates the second input information according to the third request; the second computing cluster instance receives the second request and calculates the first text unit. Optionally, the M first computing cluster instances and the N second computing cluster instances communicate via an interconnection device.

[0030] In a fourth aspect, a scheduling device is provided for scheduling a computing cluster, the computing cluster comprising a first computing cluster and a second computing cluster, the first computing cluster comprising M first computing cluster instances, the second computing cluster comprising N second computing cluster instances, M and N being positive integers, the device comprising: a sending unit, for sending a first request of a target application to the first computing cluster instance, the first request comprising first input information, the first request being used to request calculation of the first input information; a receiving unit, for receiving first indication information from the first computing cluster instance, the first indication information being used to indicate that a first text unit has been completed, the first text unit being obtained by the first computing cluster instance calculating the first input information; the sending unit is further used to send a second request to the second computing cluster instance in response to the first indication information, and to send a third request of the target application to the first computing cluster instance, the second request being used to indicate calculation of the first text unit, the third request comprising second input information, the third request being used to request calculation of the second input information. Optionally, the M first computing cluster instances and the N second computing cluster instances communicate via an interconnection device.

[0031] In a possible implementation of the fourth aspect, the first text unit is stored in a local cache of the first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance, which is used by the second computing cluster instance to obtain the first text unit from the first computing cluster instance.

[0032] In a possible implementation of the fourth aspect, the receiving unit is further used to receive second indication information from the second computing cluster instance, and the second indication information is used to indicate that the first text unit has been acquired; the sending unit is further used to send first notification information to the first computing cluster instance according to the second indication information, and the first notification information is used to instruct the first computing cluster instance to release the first storage space storing the first text unit in the local cache.

[0033] In a possible implementation of the fourth aspect, the first computing cluster instance and the second computing cluster instance also communicate with a storage device, the first text unit is stored in the storage device, and the second request includes storage indication information, which is used for the second computing cluster instance to obtain the first text unit from the first computing cluster instance.

[0034] In a possible implementation of the fourth aspect, the device includes: the receiving unit is also used to receive second indication information from the second computing cluster instance, the second indication information is used to indicate that the first text unit has been acquired; the sending unit is also used to send second notification information to the storage device according to the second indication information, and the second notification information is used to instruct the storage device to release a second storage space storing the first text unit.

[0035] In a possible implementation of the fourth aspect, the number of the M first computing cluster instances and the number of the N second computing cluster instances are determined based on data characteristics of the target application, where the data characteristics include the length of input information, the length of output information, and the number of iterations of decoding calculations.

[0036] In a possible implementation of the fourth aspect, the device further includes: a processing unit, configured to initialize the M first computing cluster instances; the processing unit is further configured to initialize the N second computing cluster instances. Optionally, the processing unit is further configured to initialize an interconnection device between the M first computing cluster instances and the N second computing cluster instances.

[0037] In a possible implementation of the fourth aspect, the device also includes a processing unit, and when the first preset condition is met, the processing unit is further used to: if the computing cluster also includes X expanded first computing cluster instances, initialize the X first computing cluster instances, and further initialize the interconnection device between the X first computing cluster instances and the N second computing cluster instances, where X is a positive integer; or, configure X second computing cluster instances among the N second computing cluster instances as X first computing cluster instances, and further initialize the interconnection device between the X first computing cluster instances and the remaining NX second computing cluster instances.

[0038] In a possible implementation manner of the fourth aspect, the first preset condition includes: the local cache occupancy of the M first computing cluster instances is greater than a preset occupancy, or the computing load of the M first computing cluster instances is greater than a first load threshold.

[0039] In a possible implementation of the fourth aspect, the device also includes a processing unit, and when the second preset condition is met, the processing unit is also used to: if the computing cluster also includes Y expanded second computing cluster instances, initialize the Y second computing cluster instances, and further initialize the interconnection device between the Y second computing cluster instances and the M first computing cluster instances; or, configure Y first computing cluster instances among the M first computing cluster instances as Y second computing cluster instances, and further initialize the interconnection device between the Y second computing cluster instances and the remaining MY first computing cluster instances.

[0040] In a possible implementation manner of the fourth aspect, the second preset condition includes: the local cache occupancy of the N second computing cluster instances is greater than a preset occupancy, or the computing load of the N second computing cluster instances is greater than a first load threshold.

[0041] In a possible implementation manner of the fourth aspect, the target application includes a large language model (LLM) application, the first computing cluster is a computing cluster for performing pre-filling calculations, and the second computing cluster is a computing cluster for performing decoding calculations.

[0042] In a fifth aspect, a computing cluster is provided, the computing cluster comprising a first computing cluster and a second computing cluster, the first computing cluster comprising M first computing cluster instances, the second computing cluster comprising N second computing cluster instances, M and N being positive integers; the first computing cluster instance is used to receive a first request of a target application sent by a general processing device, the first request comprising first input information, the first request being used to request calculation of the first input information; the first computing cluster instance is also used to calculate the first input information to obtain a first text unit according to the first request; the first computing cluster instance is also used to send a first indication information to the general processing device, the first indication information being used to indicate that the first text unit has been completed; the first computing cluster instance is also used to receive a third request of the target application sent by the general processing device in response to the first indication information, the third request comprising second input information, the third request being used to request calculation of the second input information; the second computing cluster instance is used to receive a second request sent by the general processing device, the second request being used to indicate calculation of the first text unit. Optionally, the M first computing cluster instances and the N second computing cluster instances communicate via an interconnection device.

[0043] In a possible implementation of the fifth aspect, the first text unit is stored in a local cache of a first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance; the second computing cluster instance is further used to obtain the first text unit from the first computing cluster instance according to the first identifier of the first computing cluster instance.

[0044] In a possible implementation of the fifth aspect, the second computing cluster instance is further used to send a second indication message to the general processing device, and the second indication message is used to indicate that the first text unit has been acquired; the first computing cluster instance is further used to receive a first notification message sent by the general processing device, and the first notification message is sent by the general processing device according to the second indication message; the first computing cluster instance is further used to release a first storage space storing the first text unit in the local cache according to the first notification message.

[0045] In a possible implementation of the fifth aspect, the first computing cluster instance and the second computing cluster instance also communicate with a storage device, and the second request includes storage indication information; the first computing cluster instance is further used to store the first text unit in the storage device; and the second computing cluster instance is further used to obtain the first text unit from the storage device according to the storage indication information.

[0046] In a possible implementation of the fifth aspect, the second computing cluster instance is also used to send a second indication message to the general processing device, and the second indication message is used to indicate that the first text unit has been acquired; wherein the second indication message is used by the general processing device to send a second notification message to the storage device, and the second notification message is used by the storage device to release a first storage space for storing the first text unit.

[0047] In a possible implementation manner of the fifth aspect, the first computing cluster instance is further used to perform an initialization operation; and the second computing cluster instance is further used to perform an initialization operation.

[0048] In a possible implementation manner of the fifth aspect, the first computing cluster instance is further used to receive first configuration information sent by the general processing device, where the first configuration information is used to configure the first computing cluster instance as the second computing cluster instance.

[0049] In a possible implementation manner of the fifth aspect, the second computing cluster instance is further used to receive second configuration information sent by the general processing device, where the second configuration information is used to configure the first computing cluster instance as the second computing cluster instance.

[0050] In a possible implementation manner of the fifth aspect, the target application includes a large language model (LLM) application, the first computing cluster is a computing cluster for performing pre-filling calculations, and the second computing cluster is a computing cluster for performing decoding calculations.

[0051] In a sixth aspect, a scheduling device is provided, which includes a processor and a memory, wherein instructions are stored in the memory, and when the processor runs the instructions, the device executes the scheduling method provided by the first aspect or any possible implementation of the first aspect.

[0052] In a seventh aspect, a computing cluster is provided, which includes a processor and a memory, wherein instructions are stored in the memory, and when the processor executes the instructions, the computing cluster executes the scheduling method provided by the second aspect or any possible implementation of the second aspect.

[0053] In an eighth aspect, a data processing system is provided, which includes a general processing device and a computing cluster, the general processing device is used to execute the scheduling method provided by the first aspect or any possible implementation of the first aspect, and the computing cluster is used to execute the scheduling method provided by the second aspect or any possible implementation of the second aspect.

[0054] In a ninth aspect, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed by a device, the device executes the scheduling method provided by the first aspect or any possible implementation of the first aspect.

[0055] In a tenth aspect, a computer-readable storage medium is provided, which includes computer instructions. When the computer instructions are executed by a device, the device executes the scheduling method provided by the second aspect or any possible implementation of the second aspect.

[0056] In an eleventh aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the scheduling method provided by the first aspect or any possible implementation of the first aspect.

[0057] In a twelfth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the scheduling method provided by the second aspect or any possible implementation of the second aspect.

[0058] It can be understood that the beneficial effects that can be achieved by the scheduling method of the second aspect provided above, as well as any scheduling device, computing cluster, data processing system, computer-readable storage medium and computer program product can correspond to the beneficial effects in the scheduling method provided by the first aspect or any possible implementation of the first aspect above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of M-times forward reasoning provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of LLM reasoning provided in an embodiment of the present application;

[0061] Figure 3 A schematic diagram of a batch level processing provided in an embodiment of the present application;

[0062] Figure 4 A schematic diagram of an iterative level processing provided in an embodiment of the present application;

[0063] Figure 5 A schematic diagram of another iterative level processing provided in an embodiment of the present application;

[0064] Figure 6 A schematic diagram of a preemption strategy for enabling iteration levels provided in an embodiment of the present application;

[0065] Figure 7 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application;

[0066] Figure 8 A flowchart of a scheduling method provided in an embodiment of the present application;

[0067] Fig. 9 A schematic diagram of a scheduling method provided in an embodiment of the present application;

[0068] Fig.10 A flowchart of another scheduling method provided in an embodiment of the present application;

[0069] Fig.11 A schematic diagram of a deployment computing cluster provided in an embodiment of the present application;

[0070] Fig.12 A schematic diagram of the structure of a general processing device provided in an embodiment of the present application;

[0071] Fig.13 A schematic diagram of the structure of another general processing device provided in an embodiment of the present application;

[0072] Fig.14 A schematic diagram of the structure of a computing cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] The following will discuss the making and use of various embodiments in detail. However, it should be understood that many applicable inventive concepts provided by this application can be implemented in a variety of specific environments. The specific embodiments discussed are only illustrative of specific ways to implement and use this description and this technology, and do not limit the scope of this application.

[0074] Unless defined otherwise, all technical and scientific terms used herein have the same meanings as commonly understood by one of ordinary skill in the art.

[0075] Various circuits or other components may be described or referred to as being "configured to" perform one or more tasks. In this case, "configured to" is used to imply structure by indicating that the circuit / component includes structure (e.g., circuitry) that performs the one or more tasks during operation. Thus, even when the specified circuit / component is not currently operational (e.g., not turned on), the circuit / component may be referred to as being configured to perform the task. Circuits / components used with the phrase "configured to" include hardware, such as circuits that perform an operation, etc.

[0076] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the present application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, a, b and c; where a, b and c can be single or multiple.

[0077] The embodiments of the present application use words such as "first" and "second" to distinguish objects with similar names, functions or effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order. The term "coupled" is used to indicate electrical connection, including direct connection through wires or connection terminals or indirect connection through other devices. Therefore, "coupled" should be regarded as a broad electronic communication connection.

[0078] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0079] Before introducing the embodiments of the present application, the relevant application scenarios involved in the present application are first introduced and explained.

[0080] Large language model (LLM) refers to a type of language model based on neural network with large-scale parameters (usually billions or more). LLM can handle a variety of natural language tasks, such as text classification, question and answer, and dialogue, and is an important path to artificial intelligence. At present, LLM mainly includes generative pre-trained transformer (chat generative pre-trained transformer, GPT) series, large language model meta AI (largelanguage model meta AI, LLAMA) series and generalized linear model (generalized linear model, GLM) series. Exemplarily, the GPT system may include a chat generative pre-trained transformer chatGPT, which is a natural language processing tool driven by artificial intelligence technology. It can communicate by understanding and learning human language, and can also interact according to the context of the chat, and can even be used to complete emails, video scripts, copywriting, translation, code, papers and other writing tasks.

[0081] The basic architecture of the above models all adopts a decoder-only transformer architecture. When making inferences and predictions, the LLM of this type of architecture adopts an auto-aggressive generative mode, that is, the generation of each text unit (token) needs to go through a forward reasoning process of the LLM model. For a sentence including M tokens, it needs to go through M forward reasoning processes of the LLM model. This forward reasoning can also be called forward iteration. For example, Figure 1 As shown, the LLM model may include M forward reasoning processes, and the processing architecture corresponding to each forward reasoning process may include a pre-filling layer (embedding layer) and N transformer layers (transformer layer). Figure 1 The input text unit represents the token that needs to be input when each forward reasoning process is executed, the output text unit represents the token output after each forward reasoning process is completed, EOS represents the end mark (that is, it is used to indicate the end of M forward reasoning processes), and the N transformer layers are represented as TL 1 to TL N.

[0082] In practical applications, the number of parameters of LLM is usually large, and the order of magnitude is generally billions, tens of billions, and hundreds of billions. Taking GPT3 used by chatGPT as an example, it includes a total of 96 transformer layers and contains 175B (billion) parameters. Therefore, the above model has high requirements for human-machine responsiveness. For example, the above chatGPT, as a typical application using interactive AI technology, requires the reasoning of the LLM model to meet low latency, and it is inferred from the human reading speed that at least 50ms / token must be met.

[0083] The above model is usually deployed on a heterogeneous accelerated computing cluster with multiple processing units, and is executed using the parallel computing capabilities of the heterogeneous accelerated computing cluster to maximize the data concurrency density. The heterogeneous accelerated computing cluster can also be called a high-performance heterogeneous acceleration device. For example, the multiple processing units may include, but are not limited to: a central processing unit (CPU), a tensor processing unit (TPU), a digital signal processing unit (DSP), a neural network processing unit (NPU), a graphic processing unit (GPU), an image signal processor (ISP), a data processing unit (DPU), a microcontroller or microprocessor, an application-specific integrated circuit, a field programmable gate array or other programmable logic devices, etc.

[0084] In one possible implementation, a fast transformer (FT) is provided. The FT proposes to use KV cache technology in the transformer reasoning process to reduce the amount of calculation in the decryption stage, greatly reducing the redundant calculation of the LLM model in the generative stage, and becoming an essential technology for the LLM reasoning system. The following is an introduction to the reasoning process of the LLM reasoning system using KV cache technology. Figure 2As shown, the LLM reasoning process using KV cache technology includes a prefill stage and a decode stage. The prefill stage is mainly used to complete the processing of the user's input text information (prompt) to generate the first text unit (first token), corresponding to the first text unit completion time (first token complete time), calculated according to the maximum text length (context length) supported by the LLM model, and the maximum length ranges from 1K to 4K. The decoding stage is mainly used to use KV cache technology to generate subsequent multiple text units through token-by-token until LLM generates the end marker EOS (end of sequence), corresponding to the decode complete time (decode complete time); wherein, a response can be output for each generated text unit.

[0085] In the LLM application of real-time human-computer interaction, the completion time of the first text unit determines the first response delay (job first token completion time, FTCT) of the LLM request triggered by the LLM application. The LLM reasoning delay (LLMinference job time) is equal to the sum of N times the decoding time and the first response delay FTCT.

[0086] By analyzing the computing characteristics of different LLMs, it is found that the computing characteristics and distributed parallel characteristics of the pre-filling stage and the decoding stage are different. Among them, the characteristics of the pre-filling stage are: high computing power requirements, and linear growth with the length of the user's input text information. Businesses that have requirements for the first text unit delay index usually use a single batch to perform LLM reasoning. The characteristics of the decoding stage are: after adopting KV caching technology, the computational complexity of a single text unit is low, and the main bottleneck is the amount of parameters to be moved, so it is usually necessary to use multiple batches to improve utilization.

[0087] The total execution time of a single LLM inference depends on the length of the input text information and the output text information, but the length of the output text information cannot be known in advance. When executing in multiple batches, the batch-level scheduling method needs to wait until the longest output text information in the entire batch has completed its iterations before the entire batch can be exited. For example, Figure 3As shown, in the LLM reasoning process of multiple requests (for example, req1 to req4), if the input text length of req1 is three-sixths of 1024 tokens (represented as 1024T in the figure), the input text length of req2 is four-sixths of 1024 tokens, the input text length of req3 is five-sixths of 1024 tokens, and the input text length of req4 is two-sixths of 1024 tokens, the output text length of req1 is five-sixths of 1024 tokens, the output text length of req2 is three-sixths of 1024 tokens, the input text length of req3 is two-sixths of 1024 tokens, and the input text length of req4 is 1024 tokens, then the output text of any request from req1 to req4 needs to wait until the output texts of req1 to req4 are completed before it can be output. For example, the final output of req2 with the shortest output text length needs to wait until the output text of req4 with the longest output length is completed before it can be output. Therefore, computing power is idle and wasted during this period, causing the cost of the heterogeneous acceleration computing cluster for LLM reasoning to increase.

[0088] To address the problems existing in the above batch-level scheduling method, an iteration-level dynamic scheduling solution can be adopted to support early exit of each task (or task corresponding to the request) in the batch, monitor the output status of the output text unit of each request, and once a request ends early (that is, the end marker EOS appears), reschedule a new task to be inserted into the exit position for calculation, so as to improve the utilization rate of batch-level calculations of heterogeneous accelerated computing clusters in the decoding stage.

[0089] For example, Figure 4 As shown, assuming that there are different numbers of text units corresponding to the first task S1 to the fourth task S4, then: when the end mark EOS appears in the text unit T6 of the first task S1, the sixth task S6 can be inserted at the position of the text unit T7 for processing; when the end mark EOS appears in the text unit T8 of the second task S2; when the end mark EOS appears in the text unit T4 of the third task S3, the fifth task S5 can be inserted at the position of the text unit T5 for processing; when the end mark EOS appears in the text unit T7 of the fourth task S4, the seventh task S7 can be inserted at the position of the text unit T8 for processing. In the figure, T1 to T8 represent different text units T6.

[0090] From the analysis of the completion process of the above-mentioned LLM reasoning, it can be known that the entire LLM reasoning process includes a pre-filling stage and a decoding stage of multiple rounds of iterations. Therefore, the premise for implementing the above-mentioned dynamic scheduling scheme at the iteration level is that each scheduled task needs to complete the calculation of the pre-filling stage first, that is, to perform the pre-filling calculation first through the idle computing power. However, the amount of calculation in the pre-filling stage is large and the corresponding calculation delay is usually more than 500ms, while the amount of calculation in the decoding stage is small and the delay of a single text unit is required to be less than 50ms. If the pre-filling of a new task and the decoding of the original task are scheduled to be executed simultaneously on a heterogeneous accelerated computing cluster, the calculation delay of the pre-filling of the new task will far exceed the calculation delay of a single text unit required in the decoding calculation of 50ms. For example, if Figure 5 As shown, if the end mark EOS appears at the position of the text unit Ti+1 in the third task S3, it is necessary to pre-fill the newly inserted fifth task S5 with calculations to generate the first text unit T1 of the fifth task S5. At this time, the calculation of other text units of the original task (for example, Ti+2, Ti+3, etc.) needs to wait until the first text unit T1 of the fifth task S5 is generated.

[0091] In addition, an iteration-level preemption policy can be enabled at the granularity of each task's text unit, that is, whenever a task generates a text unit, it can be determined whether to continue executing the task or use other tasks in the queue to preempt the task's computing resources. For example, Figure 6 As shown in (a), after the text units T1 and T2 of the first task S1 are generated, the second task S2 preempts and generates the text units T1 and T2 of the second task S2, and then the third task S3 can preempt and generate the text units T1 and T2 of the third task S3; or, Figure 6 As shown in (b), after the text unit T1 of the first task S1 is generated, the second task S2 preempts and generates the text unit T1 of the second task S2, and then the third task S3 preempts and generates the text unit T1 of the third task S3, the first task S1 preempts and generates the text unit T2 of the first task S1 again, and then the second task S2 preempts and generates the text unit T2 of the second task S2, and then the third task S3 preempts and generates the text unit T2 of the third task S3; or, Figure 6 As shown in (c), after the text unit T1 of the second task S2 is generated, the third task S3 preempts and generates the text unit T1 of the third task S3, then the second task S2 preempts and generates the text unit T2 of the second task S2, then the third task S3 preempts and generates the text unit T2 of the third task S3 again, and then the first task S1 preempts and generates the text units T1 and T2 of the first task S1; or, Figure 6As shown in (d), after the text units T1 and T2 of the second task S2 are generated, the third task S3 preempts and generates the text units T1 and T2 of the third task S3, and then the first task S1 preempts and generates the text units T1 and T2 of the first task S1. However, this method cannot meet the latency requirements for the first text unit and a single text unit in real-time AI interaction scenarios.

[0092] Based on this, an embodiment of the present application provides a data processing system and a scheduling method. The data processing system can meet the existing real-time LL application experience through distributed deployment, batch dynamic scheduling technology, and distributed transmission and storage key technologies of event and data separation, while improving the computing power utilization and system throughput of the LLM heterogeneous computing cluster. Accordingly, the method can be used to reduce the latency of LLM reasoning, meet the latency requirements of a single text unit, and improve the throughput and computing power utilization of the device. Optionally, the data processing system can be a chip or an electronic device. The electronic device may include, but is not limited to: a mobile phone, a tablet computer, a computer, a laptop computer, an ultra-mobile personal computer (ultra-mobile personal computer, umPC), a wearable device, a vehicle (for example, a car, a bicycle, an electric car, an airplane, a ship, a train, a high-speed rail, etc.), a vehicle-mounted device, a virtual reality (virtual reality, VR) device, an augmented reality (augmented reality, AR) device or an intelligent robot, etc.

[0093] Figure 7 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application. The data processing system may include a general processing device 10, a computing cluster 20 and an interconnection device 30, wherein the general processing device 10 may be used to schedule the computing cluster 20 to perform computing, and different computing clusters in the computing cluster 20 may communicate with each other through the interconnection device 30.

[0094] Optionally, the general processing device 10 may be a host, and the computing cluster 20 may be a heterogeneous accelerated computing cluster. The computing cluster 20 may include a variety of processing units, which may include but are not limited to: a central processing unit CPU, a tensor processing unit TPU, a digital signal processing unit DSP, a neural network processing unit NPU, a graphics processing unit GPU, an image signal processor ISP, a data processing unit DPU, a microcontroller or microprocessor, an application-specific integrated circuit, a field programmable gate array or other programmable logic devices, etc.

[0095] In a possible embodiment, the general processing device 10 may include a scheduling device 11, and the computing cluster 20 may include a first computing cluster 21 and a second computing cluster 22 communicating through the interconnection device 30. The scheduling device 11 may be used to schedule the first computing cluster 21 and the second computing cluster 22 to perform different calculations. Exemplarily, the first computing cluster 21 may be used to perform pre-filling calculations in LLM reasoning, and the second computing cluster 22 may be used to perform decoding calculations in LLM reasoning, so that the first computing cluster 21 may also be called a pre-filling computing cluster, and the second computing cluster 22 may also be called a decoding computing cluster. Exemplarily, when the data processing system is used for LLM reasoning, the scheduling device 11 may be called an LLM scheduling device.

[0096] Optionally, the scheduling device 11 can also be used to provide control, time, and data transmission between the first computing cluster 21 and the second computing cluster 22, provide iteration level scheduling control and early exit control of the decoding computing cluster, and at the same time meet the latency of a single request in the pre-filling stage, as well as the latency and multi-batch computing power utilization requirements of the decoding stage.

[0097] In actual applications, the first computing cluster 21 and the second computing cluster 22 can be two different heterogeneous computing clusters, each of which can include multiple processing units. The types and ratios of the processing units included in different heterogeneous computing clusters, as well as the processed batch size, can be set according to actual needs, and the embodiments of the present application do not make specific limitations on this.

[0098] In another possible embodiment, the general processing device 10 may further include a deployment device 12, which may be used to deploy the first computing cluster 21 and the second computing cluster 22, such as being used to initialize the first computing cluster 21 and the second computing cluster 22. Exemplarily, the first computing cluster 21 may include M first computing cluster instances, and the second computing cluster 22 may include N second computing cluster instances, and the deployment device 12 may be used to initialize the M first computing cluster instances and the N second computing cluster instances. Exemplarily, when the data processing system is used for LLM reasoning, the deployment device 12 may be referred to as an LLM deployment device.

[0099] Optionally, the number of computing cluster instances included in the first computing cluster 21 and the number of computing cluster instances included in the second computing cluster 22 can also be dynamically increased or decreased. In one possible example, the computing cluster 20 may include an expanded X number of first computing cluster instances or Y number of second computing cluster instances, and the deployment device 12 may also be used to configure the X number of first computing cluster instances or Y number of second computing cluster instances. In another possible example, the deployment device 12 may also be used to change the roles of the first computing cluster instance and the second computing cluster instance to dynamically adjust the number of different computing instances.

[0100] In another possible embodiment, the data processing system may further include a deployment evaluation device 40, which may be used to evaluate and determine the number of first computing cluster instances included in the first computing cluster 21 and the number of second computing cluster instances included in the second computing cluster 22. Exemplarily, taking LLM reasoning as an example, the deployment evaluation device 40 may be an LLM deployment evaluation device, which may evaluate and determine the number of first computing cluster instances and the number of second computing cluster instances according to the historical data characteristics of the LLM application, that is, determine the number of pre-filled computing cluster instances and the number of decoding computing cluster instances, so as to improve the adaptability of the deployment. The data characteristics may include, but are not limited to: the length of the input information, the length of the output information, and the number of iterations of the decoding calculation, etc. Further, the data processing system may further include a resource management and allocation system, and the deployment evaluation device 40 may apply for corresponding computing resources from the resource management and allocation system for deploying the number of first computing cluster instances and the second computing cluster instances.

[0101] In some other possible embodiments, the data processing system may further include a storage device 50, which may be a scalable distributed storage device, for example, the storage device 50 may be a KV cache storage device. The first computing cluster 21 and the second computing cluster 22 may also communicate with the storage device 50 through the interconnection device 30, for example, the first computing cluster 21 and the second computing cluster 22 may also write data to the storage device 50 or read data from the storage device 50 through the interconnection device 30.

[0102] Optionally, when the data processing system also includes a storage device 50, the deployment device 12 can also be used to configure the storage device 50, for example, when initializing the M first computing cluster instances and the N second computing cluster instances, the storage device 50 is initialized at the same time.

[0103] It is understandable that the above Figure 7The structure of the data processing system shown is exemplary, and the structure described in the diagram does not constitute a limitation on the structure of the data processing system. In actual applications, the data processing system may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0104] Figure 8 A scheduling method is provided for an embodiment of the present application. The scheduling method is used to schedule a computing cluster, wherein the computing cluster includes a first computing cluster and a second computing cluster, the first computing cluster includes M first computing cluster instances, the second computing cluster includes N second computing cluster instances, the M first computing cluster instances and the N second computing cluster instances communicate through an interconnection device, M and N are positive integers, and the method includes the following steps.

[0105] S201: The general processing device sends a first request of a target application to a first computing cluster instance, where the first request includes first input information and is used to request computing the first input information.

[0106] The first computing cluster instance may be any one of the M first computing cluster instances, the first computing cluster instance may be in an idle state, or there may be available storage space in the cache of the first computing cluster instance. The first request may be sent by a target application to the general processing device based on a user operation, the target application may be an LLM application, and the first input information in the first request may be text information input by a user through the target application.

[0107] In a possible embodiment, when the scheduling device in the general processing device receives the first request of the target application, the scheduling device may send a first request to a first computing cluster instance among the M first computing cluster instances when it is determined that there is available storage space in the local cache of the first computing cluster instance, so as to schedule the first computing cluster instance to perform a first calculation on the first input information through the first request, such as the first computing cluster instance performing a pre-fill calculation on the first input information.

[0108] S202: When the first computing cluster instance receives the first request, the first computing cluster instance calculates the first input information according to the first request to obtain a first text unit.

[0109] Specifically, when the first computing cluster instance receives the first request, the first computing cluster instance may perform a pre-fill calculation on the first input information to obtain a first text unit, where the first text unit may refer to the first text unit (first token) or the first text unit.

[0110] Optionally, after the first computing cluster instance obtains the first text unit, the first computing cluster instance may store the first text unit in a local cache. For example, the first computing cluster instance may store the first text unit in a first storage space of the local cache in the form of a KV cache.

[0111] Or, optionally, after the first computing cluster instance obtains the first text unit, if the local cache of the first computing cluster instance is full or the remaining available space cannot store the first text unit, the first computing cluster instance can store the first text unit in an externally extended storage device through an interconnection device, for example, the first computing cluster instance can store the first text unit in the second storage space of the storage device in the form of a KV cache. The first computing cluster instance can also generate storage indication information corresponding to the first text unit, for example, the storage indication information can be a KV-Cache-ID.

[0112] S203: The first computing cluster instance sends first indication information to the general processing device, where the first indication information is used to indicate that the first text unit has been completed.

[0113] Specifically, after the first computing cluster instance obtains the first text unit, the first computing cluster instance can send a first indication message to the scheduling device in the general processing device to indicate through the first indication message that the first text unit has been completed, or to indicate through the first indication message that the pre-filled calculation corresponding to the first request has been completed.

[0114] S204: In response to the general processing device receiving the first indication information, the general processing device sends a second request to the second computing cluster instance, and sends a third request for the target application to the first computing cluster instance.

[0115] The second request is used to instruct to calculate the first text unit, for example, the first request is used to instruct the second computing device to perform decoding calculation on the first text unit. The second computing cluster instance can be a second computing cluster instance in an idle state among the N second computing cluster instances.

[0116] In addition, the third request includes second input information, which may also be text information input by the user through the target application. The third request is used to schedule the first computing cluster instance to calculate the second input information, such as scheduling the first computing cluster instance to perform pre-fill calculation on the second input information.

[0117] In a possible embodiment, when the scheduling device in the general processing device receives the first indication information, in response to the first indication information, the scheduling device can send a second request to the second computing cluster instance to schedule the second computing cluster instance to perform decoding calculation on the first text unit; at the same time, the scheduling device can also send a third request of the target application to the first computing cluster instance to schedule the first computing cluster instance to perform pre-fill calculation on the second input information. Accordingly, when the first computing cluster instance receives the third request, the first computing cluster instance can perform pre-fill calculation on the second input information.

[0118] Optionally, when the scheduling device in the general processing device receives the first indication information, the scheduling device may also send a first response to the target application to indicate to the target application through the first response that the first text unit has been completed. The first response may also be referred to as a first token response. In one example, the first response may include the first text unit, and the target application may display the first text unit when receiving the first text unit.

[0119] S205: When the second computing cluster instance receives the second request, the second computing cluster instance calculates the first text unit according to the second request.

[0120] Optionally, when receiving the second request, the second computing cluster instance can add the computing task corresponding to the second request to the task queue, and execute it in sequence according to the order of arrangement of the tasks in the task queue. For example, before the second request, the task queue also includes other tasks job1, job2, job3 and job4, and the second computing cluster instance can calculate the kth token of job1, job2, job3 and job4; if after the kth token calculation is completed, the end mark EOS appears in job3, then the second computing cluster instance can take out the second request from the task queue and obtain the first text unit; after that, the second computing cluster instance can decode and calculate the second request, as well as job1, job2 and job4, that is, job1, job2 and job4 perform the k+1th token calculation, and perform the second token calculation (also referred to as the second text unit) on the second request. In addition, the second computing cluster instance can also perform the calculation of multiple output text units such as the third token and the fourth token on the second request in a similar manner until the last output text unit is calculated, and the embodiments of the present application are not repeated here.

[0121] In a possible embodiment, when the first text unit is stored in the local cache of the first computing cluster instance, the second request may include the first identifier of the first computing cluster instance, and the first identifier is used by the second computing cluster instance to obtain the first text unit from the first computing cluster instance. In this way, when the second computing cluster instance receives the second request, the second computing cluster instance can obtain the first text unit from the first computing cluster instance according to the first identifier in the second request, and perform decoding calculation on the obtained first text unit to obtain the second text unit.

[0122] Optionally, after acquiring the first text unit from the first computing cluster instance, the second computing cluster instance may also send a second indication message to the scheduling device in the general processing device, and the second indication message is used to indicate that the first text unit has been acquired. When the scheduling device in the general processing device receives the second indication message, it may send a first notification message to the first computing cluster instance according to the second indication message, and the first notification message is used to instruct the first computing cluster instance to release the first storage space in the local cache storing the first text unit. In this way, when the first computing cluster instance receives the first notification message, it may release the first storage space in the local cache storing the first text unit.

[0123] In an embodiment of the present application, after obtaining the first text, the second computing cluster instance sends a second indication message to the scheduling device in the general processing device. The scheduling device notifies the first computing cluster instance to release the first storage space storing the first text unit according to the second indication message, thereby improving the utilization rate of the storage space in the first computing cluster instance.

[0124] In another possible embodiment, when the first text unit is stored in an externally extended storage device, the second request may include storage indication information (e.g., KV-Cache-ID), and the storage indication information is used by the second computing cluster instance to obtain the first text unit from the first computing cluster instance. In this way, when the second computing cluster instance receives the second request, the second computing cluster instance may obtain the first text unit from the storage device according to the storage indication information in the second request, and perform decoding calculation on the obtained first text unit to obtain the second text unit.

[0125] Optionally, after acquiring the first text unit from the storage device, the second computing cluster instance may also send a second indication message to the scheduling device in the general processing device, the second indication message being used to indicate that the first text unit has been acquired. Upon receiving the second indication message, the scheduling device in the general processing device may send a second notification message to the storage device according to the second indication message, the second notification message being used to instruct the storage device to release the second storage space storing the first text unit. In this way, upon receiving the second notification message, the storage device may release the second storage space storing the first text unit.

[0126] In the embodiment of the present application, after acquiring the first text, the second computing cluster instance sends a second indication message to the scheduling device in the general processing device, and the scheduling device notifies the storage device to release the second storage space storing the first text unit according to the second indication message, thereby improving the utilization rate of the storage space in the storage device. In addition, by using the storage device to store the text units generated by the computing cluster instance during the computing process, the storage requirements brought about by the growth of business traffic and longer input information can be adaptively met.

[0127] For ease of understanding, the following uses the first computing cluster instance as a pre-filling computing cluster instance and the second computing cluster instance as a decoding computing cluster instance. Fig. 9 and Fig.10 The process of scheduling pre-filled computing cluster instances and decoding computing cluster instances in the embodiments of the present application is illustrated by way of example. Fig. 9 The first computing cluster instance stores the first text unit in a local cache, Fig.10 The first computing cluster instance stores the first text unit in an external storage device.

[0128] like Fig. 9As shown, the scheduling method includes: S11. The scheduling device receives a first request job-j, and the first request includes first input information; S12. When the scheduling device determines that there is available storage space in the pre-filled computing cluster instance i, the scheduling device sends a first request to the pre-filled computing cluster instance i, and the computing task corresponding to the first request can be represented as job-j; S13. The pre-filled computing cluster instance i performs pre-filling calculation on the first input information, and stores the calculated first text unit in the local cache; S14. The pre-filled computing cluster instance i sends a second request job-j-t1 to the scheduling device; S15. The scheduling device returns the received second request job-j-t1 to the business request, and completes the first token response of job-j-t1; S16. When the scheduling device determines that the decoding computing cluster instance k is idle, it sends information {job-j-t1, ID-i} to the decoding computing cluster instance k, where ID-i is the identifier of the pre-filled computing cluster instance i; S17. When the decoding computing cluster instance k receives the information, it adds it to any In the task queue, the task queue also includes other tasks job1, job2, job3 and job4. The decoding computing cluster instance k calculates the kth token of job1, job2, job3 and job4. If the end mark EOS appears in job3 after the kth token calculation is completed, the decoding computing cluster instance k takes out the information from the task queue and obtains the first text unit from the pre-filled computing cluster instance i; S18. The decoding computing cluster instance k sends a second indication message {job-j, ID-i} to the scheduling device, which is used to indicate that the first text information has been obtained, and decodes and calculates the first text unit, as well as decodes and calculates job1, job2 and job4, that is, job1, job2 and job4 perform the k+1th token calculation, and perform the second token calculation on the second request; S19. When receiving the second indication message, the scheduling device sends a notification to the pre-filled computing cluster instance i to notify the pre-filled computing cluster instance i to release the storage space for storing the first text unit.

[0129] like Fig.10As shown, the scheduling method includes: S21. The scheduling device receives a first request job-j, and the first request includes first input information; S22. When the scheduling device determines that there is available storage space in the storage device and the pre-filled computing cluster instance i is idle, the scheduling device sends a first request to the pre-filled computing cluster instance i, and the computing task corresponding to the first request can be represented as job-j; S23. The pre-filled computing cluster instance i performs pre-filling calculations on the first input information, stores the calculated first text unit in the storage device through the interconnection device, and generates storage indication information KV-Cache-ID; S24. The pre-filled computing cluster instance i sends a second request job-j-t1 to the scheduling device; S25. The scheduling device returns the received second request job-j-t1 to the business request, and completes the first token response of job-j-t1; S26. When the scheduling device determines that the decoding computing cluster instance k is idle, it sends information {job-j-t1, KV-Cache-ID} to the decoding computing cluster instance k; S27. The decoding computing cluster instance k receives When receiving the information, add it to the task queue, which also includes other tasks job1, job2, job3 and job4. The decoding computing cluster instance k calculates the kth token of job1, job2, job3 and job4. If the end mark EOS appears in job3 after the kth token calculation is completed, the decoding computing cluster instance k takes out the information from the task queue and obtains the first text unit from the storage device through the interconnection device according to the KV-Cache-ID; S28. The decoding computing cluster instance k sends the second indication information {job-j, ID-i} to the scheduling device, which is used to indicate that the first text information has been obtained, and decodes and calculates the first text unit, and decodes and calculates job1, job2 and job4, that is, job1, job2 and job4 perform the k+1th token calculation, and perform the second token calculation for the second request; S29. When receiving the second indication information, the scheduling device sends a notification to the storage device to notify the storage device to release the storage space for storing the first text unit.

[0130] Above Fig. 9 and Fig.10In the example, the scheduling device includes a request monitor, a job scheduler and a response streaming, the pre-filled computing cluster instance i includes a pre-filled executor and a cache, and the decoding computing cluster instance k includes a decoder executor and a cache. The above example does not limit the structure of the scheduling device, the pre-filled computing cluster instance and the decoding computing cluster instance.

[0131] Furthermore, the number of the M first computing cluster instances and the number of the N second computing cluster instances may be determined according to data characteristics of the target application, and the data characteristics may include the length of input information, the length of output information, and the number of iterations of decoding calculations.

[0132] The target application may be an LLM application, and the length of input information, the length of output information, and the number of iterations of decoding calculations may be different for different services of the LL application. For example, the characteristics of service 1 are: the length of input information is long, usually ranging from 4K to 8K, the length of output information is short, and the number of iterations of decoding calculations is small; the characteristics of service 2 are: the length of input information is short, the length of output information is long, and the number of iterations of decoding calculations is large.

[0133] In a possible embodiment, Fig.11 As shown, if the target application is an LLM application, the deployment evaluation device in the above data processing system can evaluate and determine the number of the M first computing cluster instances and the number of the N second computing cluster instances in the first computing cluster according to the historical data characteristics of the LLM application, for example, according to the length of the input information, the length of the output information and the number of iterations of the decoding calculation corresponding to the target application. In this way, the number of first computing cluster instances that need to be deployed in the first computing cluster and the number of second computing cluster instances that need to be deployed in the second computing cluster can be adaptively determined under the data characteristics of the different businesses of the LLM application to meet the load balancing. Fig. 9 In the description, the first computing cluster instance is a pre-filling computing cluster instance, the second computing cluster instance is a decoding computing cluster instance, and the number of pre-filling computing cluster instances determined by evaluation is 3 and the number of decoding computing cluster instances is 2.

[0134] Further, after evaluating and determining the number of the M first computing cluster instances and the number of the N second computing cluster instances, the deployment device in the data processing system may also deploy the M first computing cluster instances and the N second computing cluster instances.

[0135] In a possible embodiment, the deployment device may initialize the M first computing cluster instances and the N second computing cluster instances; further, the deployment device may also initialize the interconnection device between the M first computing cluster instances and the N second computing cluster instances. Fig.11 As shown, assuming that the M first computing cluster instances are 3 pre-filled computing cluster instances (represented as instance 1 to instance 3), and the N second computing cluster instances are 2 pre-filled computing cluster instances, the deployment device can initialize the 3 pre-filled computing cluster instances, the 2 pre-filled computing cluster instances, and the interconnection device between the above two computing cluster instances.

[0136] In an embodiment of the present application, the deployment evaluation device determines the number of first computing cluster instances that need to be deployed in the first computing cluster and the number of second computing cluster instances that need to be deployed in the second computing cluster based on the data characteristics of the target application. The deployment device performs deployment based on the evaluation results, which can improve the adaptability of deploying the first computing cluster and the second computing cluster, while ensuring that the first computing cluster and the second computing cluster meet load balancing.

[0137] Optionally, the number of first computing cluster instances included in the first computing cluster and the number of second computing cluster instances included in the second computing cluster may also be dynamically increased or decreased.

[0138] In a first possible embodiment, when a first preset condition is met, the deployment device can dynamically increase the number of first computing cluster instances. Exemplarily, the first preset condition may include: the local cache occupancy of the M first computing cluster instances is greater than the first preset occupancy, or the computing load of the M first computing cluster instances is greater than the first load threshold. Among them, the first preset occupancy and the first load threshold can be set in advance, and can be specifically set by those skilled in the art according to actual conditions, and the embodiments of the present application do not impose specific restrictions on this.

[0139] In the initial deployment phase, the deployment device deploys M first computing cluster instances and N second computing cluster instances according to the evaluation results of the evaluation device. After running for a period of time, the business data characteristics of the target application change. For example, the length of the input information increases several times, causing the local cache of the M first computing cluster instances to be full or the computing load to reach the upper limit. At this time, the number of first computing cluster instances can be dynamically increased.

[0140] In one example, if the computing cluster further includes X expanded first computing cluster instances, the deployment device may initialize the X first computing cluster instances; further, the deployment device may also initialize the interconnection device between the X first computing cluster instances and the N second computing cluster instances, where X is a positive integer. That is, the number of first computing cluster instances may be dynamically increased by adding X first computing cluster instances.

[0141] In another example, if a large number of tasks of the N second computing cluster instances have been completed and there is idle computing power, the deployment device can also configure X second computing cluster instances among the N second computing cluster instances as X first computing cluster instances; further, the deployment device can also initialize the interconnection device between the X first computing cluster instances and the remaining NX second computing cluster instances. That is, the number of first computing cluster instances can be dynamically increased by changing the roles of some second computing cluster instances to first computing cluster instances.

[0142] In a second possible embodiment, when the second preset condition is met, the deployment device can dynamically increase the number of second computing cluster instances. Exemplarily, the second preset condition may include: the local cache occupancy of the N second computing cluster instances is greater than the second preset occupancy, or the computing load of the N second computing cluster instances is greater than the second load threshold. Among them, the second preset occupancy and the second load threshold can be set in advance, and can be specifically set by those skilled in the art according to actual conditions, and the embodiments of the present application do not impose specific restrictions on this.

[0143] In the initial deployment phase, the deployment device deploys M first computing cluster instances and N second computing cluster instances according to the evaluation results of the evaluation device. After running for a period of time, the business data characteristics of the target application have changed, for example, the length of the output information has increased several times, and the number of iterations has doubled, resulting in the local cache of the N second computing cluster instances being full or the computing load reaching the upper limit. At this time, the number of second computing cluster instances can be dynamically increased.

[0144] In one example, if the computing cluster further includes Y expanded second computing cluster instances, the deployment device may initialize the Y second computing cluster instances; further, the deployment device may also initialize the interconnection device between the Y second computing cluster instances and the M first computing cluster instances, where Y is a positive integer. That is, the number of second computing cluster instances may be dynamically increased by adding Y second computing cluster instances.

[0145] In another example, if a large number of tasks have been completed for the M first computing cluster instances and there is idle computing power, the deployment device can also configure Y first computing cluster instances among the M first computing cluster instances as Y second computing cluster instances; further, the deployment device can also initialize the interconnection device between the Y second computing cluster instances and the remaining MY first computing cluster instances. That is, the number of second computing cluster instances can be dynamically increased by changing the roles of some first computing cluster instances to second computing cluster instances.

[0146] In the embodiment of the present application, the deployment device can dynamically add or reduce the number of first computing cluster instances and the number of second computing cluster instances according to the changes in the data characteristics of the target application, thereby improving the responsiveness of the business request of the target application and enhancing the cost-performance ratio. In addition, the deployment device changes the role of some first computing cluster instances to second computing cluster instances, and the computing cluster instances whose roles have been changed can also share the model weights in the pre-filled calculation and decoding calculation process, thereby avoiding the transfer of model weights.

[0147] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the interaction between the general processing device and the computing cluster. It is understandable that the general processing device and the computing cluster include hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0148] The embodiment of the present application can divide the functional modules of the general processing device and the computing cluster point according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above functional modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation. The following is an example of dividing each functional module according to the corresponding function:

[0149] In the case of an integrated unit, Fig.12A possible structural schematic diagram of a general processing device involved in the above embodiment is shown. The device includes: a sending unit 301 and a receiving unit 302. In a possible embodiment, the sending unit 301 can be used to support the device to execute one or more of S201, S204, and steps of sending the first notification information and the second communication information in the above method embodiment; the receiving unit 302 can be used to support the device to receive the first indication information in S203 of the above method embodiment, or support the device to execute the step of receiving the second indication information sent by the second computing cluster instance in the above method embodiment. Further, the device also includes a processing unit 303; wherein the processing unit 303 can be used to support the device to execute the steps of initializing the first computing cluster instance, the second computing cluster instance, the interconnection device and the storage device in the above method embodiment, and the steps of changing the roles of the first computing cluster instance and the second computing cluster instance, and / or other technical processes described in this document.

[0150] Based on the hardware implementation, the processing unit 303 may be a processor, the sending unit 301 may be a transmitter, the receiving unit 302 may be a receiver, the receiver and the transmitter may be integrated into a transceiver, and the transceiver may also be called a communication interface.

[0151] Fig.13 A possible structural diagram of a general processing device involved in an embodiment of the present application. The device includes: a memory 311 and a processor 312. Among them, the memory 311 is used to store the program code and data of the device, and the processor 312 is used to control the actions of the general processing device in the above-mentioned method embodiment. For example, the processor 312 is used to support the device in executing the steps of initializing the first computing cluster instance, the second computing cluster instance, the interconnection device and the storage device in the above-mentioned method embodiment, and the steps of changing the roles of the first computing cluster instance and the second computing cluster instance, and / or other technical processes described in this document. Optionally, the device may also include a communication interface 313, and the communication interface 313 is used to support the device in executing the steps of communicating with the computing cluster and the storage device in the above-mentioned method embodiment.

[0152] Among them, the processor 312 can be a central processing unit, a general processor, a digital signal processor, an application-specific integrated circuit, a processing chip, a field programmable gate array or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various logic boxes, modules and circuits described in conjunction with the disclosure of the embodiments of the present application. The processor 312 can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and the like. The communication interface 313 can be a transceiver, a transceiver circuit or a transceiver interface, and the like. The memory 311 can be a volatile memory or a non-volatile memory, and the like.

[0153] For example, the communication interface 313, the processor 312, and the memory 311 are interconnected via a bus 314; the bus 314 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 314 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0154] Optionally, the memory 311 may be included in the processor 312 .

[0155] It can be understood that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0156] Fig.14 A structural diagram of a computing cluster provided in an embodiment of the present application, the computing cluster includes a first computing cluster and a second computing cluster, the first computing cluster includes M first computing cluster instances, the second computing cluster includes N second computing cluster instances, the M first computing cluster instances and the N second computing cluster instances communicate through an interconnection device, and M and N are positive integers.

[0157] In a possible embodiment, a first computing cluster instance is used to: receive a first request for a target application sent by a general processing device, the first request including first input information, and the first request is used to request calculation of the first input information; according to the first request, calculate the first input information to obtain a first text unit; send first indication information to the general processing device, and the first indication information is used to indicate that the first text unit has been completed; in response to the first indication information, receive a third request for the target application sent by the general processing device, the third request including second input information, and the third request is used to request calculation of the second input information; and a second computing cluster instance is used to: receive a second request sent by the general processing device, and the second request is used to indicate calculation of the first text unit.

[0158] Optionally, the first text unit is stored in a local cache of the first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance. In one example, the second computing cluster instance is further configured to: obtain the first text unit from the first computing cluster instance through the interconnection device according to the first identifier of the first computing cluster instance.

[0159] Furthermore, the second computing cluster instance is also used to: send a second indication message to the general processing device, the second indication message is used to indicate that the first text unit has been acquired; the first computing cluster instance is also used to: receive a first notification message sent by the general processing device, the first notification message is sent by the general processing device based on the second indication message; the first computing cluster instance is also used to: release the first storage space storing the first text unit in the local cache based on the first notification message.

[0160] Alternatively, optionally, the first computing cluster instance and the second computing cluster instance further communicate with the storage device via an interconnection device, and the second request includes storage indication information. In one example, the first computing cluster instance is further configured to: store the first text unit in the storage device via the interconnection device; and the second computing cluster instance is further configured to: obtain the first text unit from the storage device via the interconnection device according to the storage indication information.

[0161] Furthermore, the second computing cluster instance is also used to: send a second indication message to the general processing device, the second indication message is used to indicate that the first text unit has been acquired; wherein the second indication message is used by the general processing device to send a second notification message to the storage device, and the second notification message is used by the storage device to release a first storage space for storing the first text unit.

[0162] In another possible embodiment, the first computing cluster instance is further used to perform an initialization operation, and the second computing cluster instance is further used to perform an initialization operation.

[0163] Furthermore, the first computing cluster instance is also used to: receive first configuration information sent by the general processing device, the first configuration information is used to configure the first computing cluster instance as the second computing cluster instance. Similarly, the second computing cluster instance is also used to: receive second configuration information sent by the general processing device, the second configuration information is used to configure the first computing cluster instance as the second computing cluster instance.

[0164] Optionally, the target application includes a large language model (LLM) application, the first computing cluster is a computing cluster for performing pre-filling calculations, and the second computing cluster is a computing cluster for performing decoding calculations.

[0165] In another aspect of the present application, a computing cluster is provided, which includes a processor and a memory, wherein instructions are stored in the memory. When the processor executes the instructions, the computing cluster executes the steps of the first computing cluster instance and the second computing cluster instance included in the computing cluster in the method embodiment provided above.

[0166] In another aspect of the present application, a data processing system is provided, which includes a general processing device and a computing cluster; wherein the general processing device can be any one of the general processing devices provided above, and is used for the steps of the general processing device group in the above method embodiment; the computing cluster can be any one of the computing clusters provided above, and is used for the steps of the first computing cluster instance and the second computing cluster instance included in the computing cluster in the above method embodiment.

[0167] In the method or device provided in the embodiment of the present application, it can be implemented in whole or in part by software, hardware or a combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, twisted pair) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any medium that a computer can access or a data storage device such as a server or a data center that includes one or more media integrations. The medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., an optical disk), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0168] In another aspect of the present application, a computer-readable storage medium is provided, which includes computer instructions. When the computer instructions are executed by a device, the device executes the steps of the general processing device in the scheduling method provided by the above method embodiment.

[0169] In another aspect of the present application, a computer-readable storage medium is also provided, which includes computer instructions. When the computer instructions are executed by a device, the device executes the steps of the first computing cluster instance and / or the second computing cluster instance in the scheduling method provided by the above method embodiment.

[0170] In another aspect of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the steps of the general processing device in the scheduling method provided by the above method embodiment.

[0171] In another aspect of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the steps of the first computing cluster instance and / or the second computing cluster instance in the scheduling method provided in the above method embodiment.

[0172] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A scheduling method, characterized in that: For scheduling a computing cluster, the computing cluster includes a first computing cluster and a second computing cluster, the first computing cluster includes M first computing cluster instances, the second computing cluster includes N second computing cluster instances, M and N are positive integers, the method includes: Sending a first request of a target application to the first computing cluster instance, where the first request includes first input information, and the first request is used to request computing the first input information; Receiving first indication information from the first computing cluster instance, where the first indication information is used to indicate that a first text unit has been completed, and the first text unit is obtained by the first computing cluster instance calculating the first input information; In response to the first indication information, a second request is sent to the second computing cluster instance, and a third request for the target application is sent to the first computing cluster instance, wherein the second request is used to indicate calculation of the first text unit, and the third request includes second input information, and the third request is used to request calculation of the second input information.

2. The method according to claim 1, characterized in that: The first text unit is stored in a local cache of the first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance, where the first identifier is used by the second computing cluster instance to obtain the first text unit from the first computing cluster instance.

3. The method according to claim 2, characterized in that The method further comprises: receiving second indication information from the second computing cluster instance, where the second indication information is used to indicate that the first text unit has been acquired; First notification information is sent to the first computing cluster instance according to the second indication information, where the first notification information is used to instruct the first computing cluster instance to release a first storage space in the local cache storing the first text unit.

4. The method according to claim 1, characterized in that: The first computing cluster instance and the second computing cluster instance also communicate with a storage device, the first text unit is stored in the storage device, and the second request includes storage indication information, and the storage indication information is used by the second computing cluster instance to obtain the first text unit from the first computing cluster instance.

5. The method according to claim 4, characterized in that The method further comprises: receiving second indication information from the second computing cluster instance, where the second indication information is used to indicate that the first text unit has been acquired; Second notification information is sent to the storage device according to the second indication information, where the second notification information is used to instruct the storage device to release a second storage space storing the first text unit.

6. The method according to any one of claims 1 to 5, characterized in that: The number of the M first computing cluster instances and the number of the N second computing cluster instances are determined according to data characteristics of the target application, where the data characteristics include the length of input information, the length of output information, and the number of iterations of decoding calculations.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Initializing the M first computing cluster instances; Initializing the N second computing cluster instances; Initialize the interconnection device between the M first computing cluster instances and the N second computing cluster instances.

8. The method according to any one of claims 1 to 7, characterized in that: When the first preset condition is met, the method further includes: If the computing cluster further includes X expanded first computing cluster instances, initializing the X first computing cluster instances, and initializing interconnection devices between the X first computing cluster instances and the N second computing cluster instances, where X is a positive integer; or, X second computing cluster instances among the N second computing cluster instances are configured as X first computing cluster instances, and interconnection devices between the X first computing cluster instances and the remaining NX second computing cluster instances are initialized.

9. The method according to claim 8, characterized in that The first preset condition includes: the local cache occupancy of the M first computing cluster instances is greater than a preset occupancy, or the computing load of the M first computing cluster instances is greater than a first load threshold.

10. The method according to any one of claims 1 to 9, characterized in that: When the second preset condition is met, the method further includes: If the computing cluster further includes Y expanded second computing cluster instances, initializing the Y second computing cluster instances, and initializing interconnection devices between the Y second computing cluster instances and the M first computing cluster instances; or, Y first computing cluster instances among the M first computing cluster instances are configured as Y second computing cluster instances, and interconnection devices between the Y second computing cluster instances and the remaining MY first computing cluster instances are initialized.

11. The method according to claim 10, characterized in that The second preset condition includes: the local cache occupancy of the N second computing cluster instances is greater than a preset occupancy, or the computing load of the N second computing cluster instances is greater than a first load threshold.

12. The method according to any one of claims 1 to 11, characterized in that: The target application includes a large language model (LLM) application, the first computing cluster is a computing cluster for performing pre-filling calculations, and the second computing cluster is a computing cluster for performing decoding calculations.

13. A scheduling method for scheduling a computing cluster, characterized in that: The computing cluster includes a first computing cluster and a second computing cluster, the first computing cluster includes M first computing cluster instances, the second computing cluster includes N second computing cluster instances, M and N are positive integers, and the method includes: The first computing cluster instance receives a first request of a target application sent by a general processing device, wherein the first request includes first input information, and the first request is used to request computing the first input information; The first computing cluster instance calculates the first input information according to the first request to obtain a first text unit; The first computing cluster instance sends first indication information to the general processing device, where the first indication information is used to indicate that the first text unit has been completed; In response to the first indication information, the first computing cluster instance receives a third request of the target application sent by the general processing device, the third request including second input information, and the third request is used to request computing the second input information; The second computing cluster instance receives a second request sent by the general processing device, where the second request is used to instruct to calculate the first text unit.

14. The method according to claim 13, characterized in that The first text unit is stored in a local cache of the first computing cluster instance, the second request includes a first identifier of the first computing cluster instance, and the method further includes: The second computing cluster instance obtains the first text unit from the first computing cluster instance according to the first identifier of the first computing cluster instance.

15. The method according to claim 14, characterized in that The method further comprises: The second computing cluster instance sends second indication information to the general processing device, where the second indication information is used to indicate that the first text unit has been acquired; The first computing cluster instance receives first notification information sent by the general processing device, where the first notification information is sent by the general processing device according to the second indication information; The first computing cluster instance releases a first storage space storing the first text unit in the local cache according to the first notification information.

16. The method according to claim 13, characterized in that The first computing cluster instance and the second computing cluster instance further communicate with a storage device, the second request includes storage indication information, and the method further includes: The first computing cluster instance stores the first text unit in the storage device; The second computing cluster instance obtains the first text unit from the storage device according to the storage indication information.

17. The method according to claim 16, characterized in that The method further comprises: The second computing cluster instance sends second indication information to the general processing device, where the second indication information is used to indicate that the first text unit has been acquired; The second indication information is used by the general processing device to send second notification information to the storage device, and the second notification information is used by the storage device to release a first storage space storing the first text unit.

18. The method according to any one of claims 13 to 17, characterized in that: The method further comprises: The first computing cluster instance performs an initialization operation; The second computing cluster instance performs an initialization operation.

19. The method according to any one of claims 13 to 18, characterized in that: The method further comprises: The first computing cluster instance receives first configuration information sent by the general processing device, where the first configuration information is used to configure the first computing cluster instance as a second computing cluster instance.

20. The method according to any one of claims 13 to 19, characterized in that: The method further comprises: The second computing cluster instance receives second configuration information sent by the general processing device, where the second configuration information is used to configure the first computing cluster instance as a second computing cluster instance.

21. The method according to any one of claims 13 to 20, characterized in that: The target application includes a large language model (LLM) application, the first computing cluster is a computing cluster for performing pre-filling calculations, and the second computing cluster is a computing cluster for performing decoding calculations.

22. A scheduling method, characterized in that: For scheduling a computing cluster, the computing cluster includes a first computing cluster and a second computing cluster, the first computing cluster includes M first computing cluster instances, the second computing cluster includes N second computing cluster instances, M and N are positive integers, the method includes: The general processing device sends a first request of a target application to the first computing cluster instance, where the first request includes first input information, and the first request is used to request computing the first input information; The first computing cluster instance receives the first request, and calculates the first input information according to the first request to obtain a first text unit; The first computing cluster instance sends first indication information to the general processing device, where the first indication information is used to indicate that the first text unit has been completed; The general processing device receives first indication information from the first computing cluster instance, where the first indication information is used to indicate that the first text unit has been completed; The general processing device sends a second request to the second computing cluster instance in response to the first indication information, and sends a third request of the target application to the first computing cluster instance, wherein the second request is used to instruct to calculate the first text unit, the third request includes second input information, and the third request is used to request to calculate the second input information; The first computing cluster instance receives the third request, and calculates the second input information according to the third request; The second computing cluster instance receives the second request and calculates the first text unit.

23. A scheduling device, characterized in that: For scheduling a computing cluster, the computing cluster includes a first computing cluster and a second computing cluster, the first computing cluster includes M first computing cluster instances, the second computing cluster includes N second computing cluster instances, M and N are positive integers, the device includes: A sending unit, configured to send a first request of a target application to the first computing cluster instance, wherein the first request includes first input information, and the first request is used to request computing the first input information; A receiving unit, configured to receive first indication information from the first computing cluster instance, wherein the first indication information is used to indicate that a first text unit has been completed, and the first text unit is obtained by the first computing cluster instance calculating the first input information; The sending unit is also used to send a second request to the second computing cluster instance in response to the first indication information, and to send a third request for the target application to the first computing cluster instance, wherein the second request is used to indicate calculation of the first text unit, and the third request includes second input information, and the third request is used to request calculation of the second input information.

24. The device according to claim 23, characterized in that The first text unit is stored in a local cache of the first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance, where the first identifier is used by the second computing cluster instance to obtain the first text unit from the first computing cluster instance.

25. The device according to claim 24, characterized in that The receiving unit is further used to receive second indication information from the second computing cluster instance, where the second indication information is used to indicate that the first text unit has been acquired; The sending unit is further used to send first notification information to the first computing cluster instance according to the second indication information, where the first notification information is used to instruct the first computing cluster instance to release a first storage space in the local cache that stores the first text unit.

26. The device according to claim 23, characterized in that The first computing cluster instance and the second computing cluster instance also communicate with a storage device, the first text unit is stored in the storage device, and the second request includes storage indication information, and the storage indication information is used by the second computing cluster instance to obtain the first text unit from the first computing cluster instance.

27. The device according to claim 26, characterized in that The device comprises: The receiving unit is further used to receive second indication information from the second computing cluster instance, where the second indication information is used to indicate that the first text unit has been acquired; The sending unit is further used to send second notification information to the storage device according to the second indication information, where the second notification information is used to instruct the storage device to release a second storage space storing the first text unit.

28. The device according to any one of claims 23 to 27, characterized in that The number of the M first computing cluster instances and the number of the N second computing cluster instances are determined according to data characteristics of the target application, where the data characteristics include the length of input information, the length of output information, and the number of iterations of decoding calculations.

29. The device according to any one of claims 23 to 28, characterized in that The device also includes: A processing unit, configured to initialize the M first computing cluster instances; The processing unit is further used to initialize the N second computing cluster instances; The processing unit is further used to initialize the interconnection device between the M first computing cluster instances and the N second computing cluster instances.

30. The device according to any one of claims 23 to 29, characterized in that The device further comprises a processing unit, and when the first preset condition is met, the processing unit is further configured to: If the computing cluster further includes X expanded first computing cluster instances, initializing the X first computing cluster instances, and initializing interconnection devices between the X first computing cluster instances and the N second computing cluster instances, where X is a positive integer; or, X second computing cluster instances among the N second computing cluster instances are configured as X first computing cluster instances, and interconnection devices between the X first computing cluster instances and the remaining NX second computing cluster instances are initialized.

31. The device according to claim 30, characterized in that The first preset condition includes: the local cache occupancy of the M first computing cluster instances is greater than a preset occupancy, or the computing load of the M first computing cluster instances is greater than a first load threshold.

32. The device according to any one of claims 23 to 31, characterized in that The device further includes a processing unit, and when the second preset condition is met, the processing unit is further configured to: If the computing cluster further includes Y expanded second computing cluster instances, initializing the Y second computing cluster instances, and initializing interconnection devices between the Y second computing cluster instances and the M first computing cluster instances; or, Y first computing cluster instances among the M first computing cluster instances are configured as Y second computing cluster instances, and interconnection devices between the Y second computing cluster instances and the remaining MY first computing cluster instances are initialized.

33. The device according to claim 32, characterized in that The second preset condition includes: the local cache occupancy of the N second computing cluster instances is greater than a preset occupancy, or the computing load of the N second computing cluster instances is greater than a first load threshold.

34. The device according to any one of claims 23 to 33, characterized in that The target application includes a large language model (LLM) application, the first computing cluster is a computing cluster for performing pre-filling calculations, and the second computing cluster is a computing cluster for performing decoding calculations.

35. A computing cluster, characterized in that: The computing cluster includes a first computing cluster and a second computing cluster, the first computing cluster includes M first computing cluster instances, the second computing cluster includes N second computing cluster instances, and M and N are positive integers; The first computing cluster instance is used to receive a first request of a target application sent by a general processing device, wherein the first request includes first input information, and the first request is used to request computing the first input information; The first computing cluster instance is further configured to calculate the first input information according to the first request to obtain a first text unit; The first computing cluster instance is further used to send first indication information to the general processing device, where the first indication information is used to indicate that the first text unit has been completed; The first computing cluster instance is further configured to receive, in response to the first indication information, a third request of the target application sent by the general processing device, wherein the third request includes second input information and is used to request computing the second input information; The second computing cluster instance is used to receive a second request sent by the general processing device, where the second request is used to instruct to calculate the first text unit.

36. The computing cluster according to claim 35, characterized in that: The first text unit is stored in a local cache of the first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance; The second computing cluster instance is further configured to obtain the first text unit from the first computing cluster instance according to the first identifier of the first computing cluster instance.

37. The computing cluster according to claim 36, characterized in that: The second computing cluster instance is further used to send second indication information to the general processing device, where the second indication information is used to indicate that the first text unit has been obtained; The first computing cluster instance is further configured to receive first notification information sent by the general processing device, where the first notification information is sent by the general processing device according to the second indication information; The first computing cluster instance is further configured to release a first storage space in the local cache storing the first text unit according to the first notification information.

38. The computing cluster according to claim 35, characterized in that: The first computing cluster instance and the second computing cluster instance also communicate with a storage device, and the second request includes storage indication information; The first computing cluster instance is further used to store the first text unit in the storage device; The second computing cluster instance is further configured to obtain the first text unit from the storage device according to the storage indication information.

39. The computing cluster according to claim 38, characterized in that: The second computing cluster instance is further used to send second indication information to the general processing device, where the second indication information is used to indicate that the first text unit has been obtained; The second indication information is used by the general processing device to send second notification information to the storage device, and the second notification information is used by the storage device to release a first storage space storing the first text unit.

40. The computing cluster according to any one of claims 35 to 39, characterized in that: The first computing cluster instance is also used to perform an initialization operation; The second computing cluster instance is also used to perform initialization operations.

41. The computing cluster according to any one of claims 35 to 40, characterized in that: The first computing cluster instance is further used to receive first configuration information sent by the general processing device, where the first configuration information is used to configure the first computing cluster instance as a second computing cluster instance.

42. The computing cluster according to any one of claims 35 to 41, characterized in that: The second computing cluster instance is further used to receive second configuration information sent by the general processing device, where the second configuration information is used to configure the first computing cluster instance as a second computing cluster instance.

43. The computing cluster according to any one of claims 35 to 42, characterized in that: The target application includes a large language model (LLM) application, the first computing cluster is a computing cluster for performing pre-filling calculations, and the second computing cluster is a computing cluster for performing decoding calculations.

44. A scheduling device, characterized in that: The scheduling device includes a processor and a memory, wherein instructions are stored in the memory. When the processor executes the instructions, the device executes the scheduling method according to any one of claims 1 to 12.

45. A computing cluster, characterized in that: The computing cluster includes a processor and a memory, wherein instructions are stored in the memory. When the processor executes the instructions, the computing cluster executes the scheduling method according to any one of claims 13 to 21.

46. ​​A data processing system, characterized in that: The data processing system includes a general processing device and a computing cluster, the general processing device is used to execute the scheduling method as described in any one of claims 1-12, and the computing cluster is used to execute the scheduling method as described in any one of claims 13-21.

47. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed by a device, the device executes the scheduling method according to any one of claims 1 to 21.

Images