A scheduling method, a scheduling device, a computing cluster and a system

Abstract

The application discloses a scheduling method, a scheduling device, a computing cluster and a system, relates to the technical field of communication, and is used for reducing the time delay of LLM inference, meeting the time delay requirement of a single text unit, and improving the throughput and computing power utilization rate of equipment. The method is used for scheduling a first computing cluster instance and a second computing cluster instance, and comprises the following steps: a first request of a target application is sent to the first computing cluster instance, the first request is used for requesting to perform computation on first input information; first indication information from the first computing cluster instance is received, the first indication information is used for indicating that a first text unit has been completed; in response to the first indication information, a second request is sent to the second computing cluster instance, the second request is used for indicating to perform computation on the first text unit, and a third request of the target application is sent to the first computing cluster instance, the third request is used for requesting to perform computation on second input information.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a scheduling method, scheduling device, computing cluster and system. Background Technology

[0002] Large language models (LLMs) refer to a class of neural network-based language models with a massive number of parameters (typically billions or more). LLMs can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence. Currently, LLMs mainly include the chat generative pre-trained transformer (GPT) series, the LLAMA (large language model meta AI) series, and the generalized linear model (GLM) series.

[0003] The underlying architecture of the aforementioned models all employs a decoder-only transformer architecture. During inference and prediction, this type of LLM uses an auto-aggressive generative mode, meaning that the generation of each text unit (token) requires one forward inference process from the LLM model. Therefore, a sentence containing M tokens requires M forward inference processes. These models are typically deployed on heterogeneous accelerated computing clusters with multiple processing units, leveraging the parallel computing capabilities of these clusters for execution.

[0004] However, when using this heterogeneous accelerated computing cluster to perform computations, how to reduce the latency of LLM inference and improve the throughput and computing power utilization of the device remains a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a scheduling method, scheduling device, computing cluster, and system for reducing the latency of LLM inference, meeting the latency requirements of a single text unit, and improving the throughput and computing power utilization of the device.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, a scheduling method is provided for scheduling a computing cluster, which can be a heterogeneous accelerated computing cluster, including a first computing cluster (e.g., a pre-filled computing cluster) and a second computing cluster (e.g., a decoding computing cluster). The first computing cluster includes M instances of the first computing cluster, and the second computing cluster includes N instances of the second computing cluster, where M and N are positive integers. The method includes: sending a first request for a target application (e.g., an LLM application) to a first computing cluster instance, the first request including first input information (e.g., text information), the first request being used to request computation on the first input information, such as pre-filled computation; receiving a first indication from a first computing cluster instance, the first indication indicating that a first text unit has been completed, the first text unit being obtained by the first computing cluster instance from the first input information; and responding to the first indication, sending a second request to a second computing cluster instance and a third request for the target application to a first computing cluster instance, the second request being used to indicate computation on the first text unit (e.g., decoding computation), the third request including second input information, the third request being used to request computation on the second input information. Optionally, the M first computing cluster instances and the N second computing cluster instances communicate via an interconnect device.

[0008] In the above technical solution, by deploying a first computing cluster comprising M first computing cluster instances and a second computing cluster comprising N second computing cluster instances separately, and then scheduling the first computing cluster instances and the second computing cluster instances to perform different calculations according to the request of the target application, and after the first computing cluster instance has finished processing, the first computing cluster instance can continue to be scheduled to calculate the third request without waiting for the second computing cluster instance to finish executing. Therefore, when this method is applied to LLM inference, the latency of LLM inference can be reduced, the latency requirement of a single text unit can be met, and the throughput and computing power utilization of the device can be improved.

[0009] In one possible implementation of the first aspect, the first text unit is stored in the local cache of the first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance, which is used by the second computing cluster instance to retrieve the first text unit from the first computing cluster instance. In the above possible implementation, the local cache of the first computing cluster instance can be used to store the text units generated by the computing cluster instance during computation, thereby reducing the storage latency of the text units.

[0010] In one possible implementation of the first aspect, the method includes: receiving second indication information from a second computing cluster instance, the second indication information indicating that the first text unit has been acquired; and sending first notification information to the first computing cluster instance according to the second indication information, the first notification information instructing the first computing cluster instance to release the first storage space storing the first text unit in its local cache. In this possible implementation, after acquiring the first text, the second computing cluster instance sends the second indication information to a scheduling device in the general-purpose processing device, and the scheduling device notifies the first computing cluster instance to release the first storage space storing the first text unit according to the second indication information, which can improve the utilization rate of storage space in the first computing cluster instance.

[0011] In one possible implementation of the first aspect, the first computing cluster instance and the second computing cluster instance also communicate with a storage device (e.g., via an interconnect device), where the first text unit is stored. The second request includes storage indication information for the second computing cluster instance to retrieve the first text unit from the first computing cluster instance. In the above possible implementations, using this storage device to store text units generated by the computing cluster instances during computation can adaptively meet the storage requirements arising from increased business traffic and longer input information.

[0012] In one possible implementation of the first aspect, the method includes: receiving second indication information from a second computing cluster instance, the second indication information indicating that the first text unit has been acquired; and sending second notification information to the storage device according to the second indication information, the second notification information indicating that the storage device releases the second storage space storing the first text unit. In this possible implementation, after acquiring the first text, the second computing cluster instance sends the second indication information to a scheduling device in the general-purpose processing device. The scheduling device then notifies the storage device to release the second storage space storing the first text unit according to the second indication information, which can improve the utilization rate of the storage space in the storage device. Furthermore, using this storage device to store text units generated by the computing cluster instance during computation can adaptively meet the storage requirements brought about by increased business traffic and longer input information.

[0013] In one possible implementation of the first aspect, the number of the M first computing cluster instances and the number of the N second computing cluster instances are determined based on the data characteristics of the target application, including the length of the input information, the length of the output information, and the number of iterations for decoding computation. In the above possible implementation, determining the number of first and second computing cluster instances based on the data characteristics of the target application improves the adaptability of deploying the first and second computing clusters while ensuring load balancing between them.

[0014] In one possible implementation of the first aspect, the method further includes: initializing the M first computing cluster instances; and initializing the N second computing cluster instances. Optionally, the interconnection device between the M first computing cluster instances and the N second computing cluster instances is initialized.

[0015] In one possible implementation of the first aspect, when the first preset condition is met, the method further includes: if the computing cluster also includes X extended first computing cluster instances, initializing the X first computing cluster instances, and further initializing the interconnection device between the X first computing cluster instances and the N second computing cluster instances, where X is a positive integer; or, configuring X of the N second computing cluster instances as X first computing cluster instances, and further initializing the interconnection device between the X first computing cluster instances and the remaining N X second computing cluster instances. In the above possible implementations, the number of first computing cluster instances can be dynamically increased according to changes in the data characteristics of the target application, thereby improving the responsiveness of the target application's business requests and enhancing the cost-performance ratio. Furthermore, in the deployment device, by changing the roles of some second computing cluster instances to first computing cluster instances, the computing cluster instances whose roles have changed can share model weights during pre-filling and decoding calculations, thereby avoiding the transfer of model weights.

[0016] Optionally, the first preset condition includes: the local cache usage of the M first computing cluster instances is greater than the preset usage, or the computing load of the M first computing cluster instances is greater than the first load threshold.

[0017] In one possible implementation of the first aspect, when the second preset condition is met, the method further includes: if the computing cluster also includes Y extended second computing cluster instances, initializing the Y second computing cluster instances, and further initializing the interconnection device between the Y second computing cluster instances and the M first computing cluster instances; or, configuring Y of the M first computing cluster instances as Y second computing cluster instances, and further initializing the interconnection device between the Y second computing cluster instances and the remaining MY first computing cluster instances. In the above possible implementations, the number of second computing cluster instances can be dynamically increased according to changes in the data characteristics of the target application, thereby improving the responsiveness of the target application's business requests and enhancing the cost-performance ratio. Furthermore, in the deployment device, by changing the roles of some first computing cluster instances to second computing cluster instances, the computing cluster instances whose roles have changed can share model weights during pre-filling and decoding calculations, thereby avoiding the transfer of model weights.

[0018] Optionally, the second preset condition includes: the local cache usage of the N second computing cluster instances is greater than the preset usage, or the computing load of the N second computing cluster instances is greater than the first load threshold.

[0019] In one possible implementation of the first aspect, the target application includes a Large Language Model (LLM) application, where a first computing cluster is used to perform pre-filling computations, and a second computing cluster is used to perform decoding computations. In the above possible implementations, when this scheduling method is applied to LLM inference, the latency of LLM inference can be reduced, the latency requirements of individual text units can be met, and the throughput and computing power utilization of the device can be improved.

[0020] Secondly, a scheduling method is provided for scheduling computing clusters, the computing clusters including a first computing cluster and a second computing cluster, the first computing cluster including M first computing cluster instances, and the second computing cluster including N second computing cluster instances, where M and N are positive integers. The method includes: a first computing cluster instance receiving a first request from a general-purpose processing device for a target application, the first request including first input information, the first request being used to request computation on the first input information; the first computing cluster instance performing computation on the first input information to obtain a first text unit according to the first request; the first computing cluster instance sending a first indication message to the general-purpose processing device, the first indication message being used to indicate that the first text unit has been completed; in response to the first indication message, the first computing cluster instance receiving a third request from the general-purpose processing device for the target application, the third request including second input information, the third request being used to request computation on the second input information; and the second computing cluster instance receiving a second request from the general-purpose processing device, the second request being used to indicate computation on the first text unit. Optionally, the M first computing cluster instances and the N second computing cluster instances communicate via an interconnection device.

[0021] In one possible implementation of the second aspect, the first text unit is stored in a local cache of the first computing cluster instance, the second request includes a first identifier of the first computing cluster instance, and the method further includes: the second computing cluster instance obtaining the first text unit from the first computing cluster instance through the interconnect device based on the first identifier of the first computing cluster instance.

[0022] In one possible implementation of the second aspect, the method further includes: a second computing cluster instance sending a second indication message to the general processing device, the second indication message indicating that the first text unit has been acquired; the first computing cluster instance receiving a first notification message sent by the general processing device, the first notification message being sent by the general processing device according to the second indication message; and the first computing cluster instance releasing the first storage space storing the first text unit in the local cache according to the first notification message.

[0023] In one possible implementation of the second aspect, the first computing cluster instance and the second computing cluster instance also communicate with a storage device (e.g., via an interconnect device), the second request includes storage indication information, and the method further includes: the first computing cluster instance storing the first text unit in the storage device; and the second computing cluster instance retrieving the first text unit from the storage device according to the storage indication information.

[0024] In one possible implementation of the second aspect, the method further includes: a second computing cluster instance sending a second indication message to the general-purpose processing device, the second indication message indicating that the first text unit has been acquired; wherein the second indication message is used by the general-purpose processing device to send a second notification message to the storage device, the second notification message being used by the storage device to release the first storage space storing the first text unit.

[0025] In one possible implementation of the second aspect, the method further includes: a first computing cluster instance performing an initialization operation; and a second computing cluster instance performing an initialization operation.

[0026] In one possible implementation of the second aspect, the method further includes: a first computing cluster instance receiving first configuration information sent by the general processing device, the first configuration information being used to configure the first computing cluster instance as a second computing cluster instance.

[0027] In one possible implementation of the second aspect, the method further includes: a second computing cluster instance receiving second configuration information sent by the general processing device, the second configuration information being used to configure the first computing cluster instance as the second computing cluster instance.

[0028] In one possible implementation of the second aspect, the target application includes a large language model LLM application, a first computing cluster for performing pre-filling computations, and a second computing cluster for performing decoding computations.

[0029] Thirdly, a scheduling method is provided for scheduling a computing cluster, the computing cluster including a first computing cluster and a second computing cluster, the first computing cluster including M instances of the first computing cluster, and the second computing cluster including N instances of the second computing cluster, where M and N are positive integers. The method includes: a general-purpose processing device sending a first request of a target application to a first computing cluster instance, the first request including first input information, the first request being used to request computation on the first input information; the first computing cluster instance receiving the first request, and calculating the first input information according to the first request to obtain a first text unit; the first computing cluster instance sending first indication information to the general-purpose processing device, the first indication information being used to... The system indicates that the first text unit has been completed. The general-purpose processing device receives a first indication from a first computing cluster instance, indicating that the first text unit has been completed. In response to the first indication, the general-purpose processing device sends a second request to a second computing cluster instance and a third request for the target application to the first computing cluster instance. The second request indicates that computation be performed on the first text unit, and the third request includes second input information, requesting computation on the second input information. The first computing cluster instance receives the third request and performs computation on the second input information accordingly. The second computing cluster instance receives the second request and performs computation on the first text unit. Optionally, the M first computing cluster instances and the N second computing cluster instances communicate via an interconnection device.

[0030] Fourthly, a scheduling device is provided for scheduling computing clusters, the computing clusters including a first computing cluster and a second computing cluster. The first computing cluster includes M instances of the first computing cluster, and the second computing cluster includes N instances of the second computing cluster, where M and N are positive integers. The device includes: a sending unit, configured to send a first request for a target application to the first computing cluster instances, the first request including first input information, the first request being used to request computation on the first input information; a receiving unit, configured to receive first indication information from the first computing cluster instances, the first indication information being used to indicate that a first text unit has been completed, the first text unit being obtained by the first computing cluster instances from the first input information; the sending unit is further configured to, in response to the first indication information, send a second request to the second computing cluster instances and a third request for the target application to the first computing cluster instances, the second request being used to indicate computation on the first text unit, the third request including second input information, the third request being used to request computation on the second input information. Optionally, the M instances of the first computing cluster and the N instances of the second computing cluster communicate via an interconnection device.

[0031] In one possible implementation of the fourth aspect, the first text unit is stored in the local cache of the first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance, which is used by the second computing cluster instance to retrieve the first text unit from the first computing cluster instance.

[0032] In one possible implementation of the fourth aspect, the receiving unit is further configured to receive second indication information from the second computing cluster instance, the second indication information being used to indicate that the first text unit has been acquired; the sending unit is further configured to send first notification information to the first computing cluster instance according to the second indication information, the first notification information being used to instruct the first computing cluster instance to release the first storage space in the local cache that stores the first text unit.

[0033] In one possible implementation of the fourth aspect, the first computing cluster instance and the second computing cluster instance also communicate with a storage device in which the first text unit is stored, and the second request includes storage indication information for the second computing cluster instance to retrieve the first text unit from the first computing cluster instance.

[0034] In one possible implementation of the fourth aspect, the apparatus includes: the receiving unit, further configured to receive second indication information from a second computing cluster instance, the second indication information indicating that the first text unit has been acquired; and the sending unit, further configured to send second notification information to the storage device according to the second indication information, the second notification information indicating that the storage device releases the second storage space storing the first text unit.

[0035] In one possible implementation of the fourth aspect, the number of the M first computing cluster instances and the number of the N second computing cluster instances are determined based on the data characteristics of the target application, including the length of the input information, the length of the output information, and the number of iterations for decoding computation.

[0036] In one possible implementation of the fourth aspect, the apparatus further includes: a processing unit for initializing the M first computing cluster instances; the processing unit is also configured to initialize the N second computing cluster instances. Optionally, the processing unit is further configured to initialize the interconnection device between the M first computing cluster instances and the N second computing cluster instances.

[0037] In one possible implementation of the fourth aspect, the device further includes a processing unit, which, when the first preset condition is met, is further configured to: if the computing cluster further includes an extended X first computing cluster instances, initialize the X first computing cluster instances, and further initialize the interconnection device between the X first computing cluster instances and the N second computing cluster instances, where X is a positive integer; or, configure X of the N second computing cluster instances as X first computing cluster instances, and further initialize the interconnection device between the X first computing cluster instances and the remaining N X second computing cluster instances.

[0038] In one possible implementation of the fourth aspect, the first preset condition includes: the local cache usage of the M first computing cluster instances is greater than a preset usage, or the computing load of the M first computing cluster instances is greater than a first load threshold.

[0039] In one possible implementation of the fourth aspect, the device further includes a processing unit, which, when the second preset condition is met, is further configured to: if the computing cluster further includes Y extended second computing cluster instances, initialize the Y second computing cluster instances, and further initialize the interconnection device between the Y second computing cluster instances and the M first computing cluster instances; or, configure Y of the M first computing cluster instances as Y second computing cluster instances, and further initialize the interconnection device between the Y second computing cluster instances and the remaining MY first computing cluster instances.

[0040] In one possible implementation of the fourth aspect, the second preset condition includes: the local cache usage of the N second computing cluster instances is greater than the preset usage, or the computing load of the N second computing cluster instances is greater than the first load threshold.

[0041] In one possible implementation of the fourth aspect, the target application includes a large language model LLM application, with a first computing cluster for performing pre-filling computations and a second computing cluster for performing decoding computations.

[0042] Fifthly, a computing cluster is provided, comprising a first computing cluster and a second computing cluster. The first computing cluster includes M instances of the first computing cluster, and the second computing cluster includes N instances of the second computing cluster, where M and N are positive integers. The first computing cluster instances are configured to receive a first request from a general-purpose processing device for a target application. The first request includes first input information and is used to request computation on the first input information. The first computing cluster instances are further configured to perform computation on the first input information to obtain a first text unit according to the first request. The first computing cluster instances are further configured to send a first indication to the general-purpose processing device, indicating that the first text unit has been completed. The first computing cluster instances are further configured to receive a third request from the general-purpose processing device for the target application, in response to the first indication, the third request including second input information and being used to request computation on the second input information. The second computing cluster instances are configured to receive a second request from the general-purpose processing device, the second request indicating that computation on the first text unit is performed. Optionally, the M instances of the first computing cluster and the N instances of the second computing cluster communicate via an interconnection device.

[0043] In one possible implementation of the fifth aspect, the first text unit is stored in the local cache of the first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance; the second computing cluster instance is further configured to retrieve the first text unit from the first computing cluster instance based on the first identifier of the first computing cluster instance.

[0044] In one possible implementation of the fifth aspect, the second computing cluster instance is further configured to send a second indication message to the general-purpose processing device, the second indication message indicating that the first text unit has been acquired; the first computing cluster instance is further configured to receive a first notification message sent by the general-purpose processing device, the first notification message being sent by the general-purpose processing device according to the second indication message; the first computing cluster instance is further configured to release the first storage space storing the first text unit in the local cache according to the first notification message.

[0045] In one possible implementation of the fifth aspect, the first computing cluster instance and the second computing cluster instance also communicate with the storage device, and the second request includes storage indication information; the first computing cluster instance is further configured to store the first text unit in the storage device; and the second computing cluster instance is further configured to retrieve the first text unit from the storage device according to the storage indication information.

[0046] In one possible implementation of the fifth aspect, the second computing cluster instance is further configured to send a second indication message to the general-purpose processing device, the second indication message indicating that the first text unit has been acquired; wherein the second indication message is used by the general-purpose processing device to send a second notification message to the storage device, the second notification message being used by the storage device to release the first storage space storing the first text unit.

[0047] In one possible implementation of the fifth aspect, the first computing cluster instance is also used to perform initialization operations; the second computing cluster instance is also used to perform initialization operations.

[0048] In one possible implementation of the fifth aspect, the first computing cluster instance is further configured to receive first configuration information sent by the general-purpose processing device, the first configuration information being configured to configure the first computing cluster instance as a second computing cluster instance.

[0049] In one possible implementation of the fifth aspect, the second computing cluster instance is further configured to receive second configuration information sent by the general-purpose processing device, the second configuration information being configured to configure the first computing cluster instance as the second computing cluster instance.

[0050] In one possible implementation of the fifth aspect, the target application includes a large language model LLM application, with a first computing cluster for performing pre-filling computation and a second computing cluster for performing decoding computation.

[0051] In a sixth aspect, a scheduling apparatus is provided, the scheduling apparatus including a processor and a memory storing instructions that, when executed by the processor, cause the apparatus to perform the scheduling method provided by the first aspect or any possible implementation thereof.

[0052] In a seventh aspect, a computing cluster is provided, the computing cluster including a processor and a memory, the memory storing instructions that, when the processor executes the instructions, cause the computing cluster to perform the scheduling method provided by the second aspect or any possible implementation thereof.

[0053] Eighthly, a data processing system is provided, the data processing system including a general-purpose processing unit and a computing cluster, the general-purpose processing unit being used to execute the scheduling method provided by the first aspect or any possible implementation of the first aspect, and the computing cluster being used to execute the scheduling method provided by the second aspect or any possible implementation of the second aspect.

[0054] Ninth aspect, a computer-readable storage medium is provided that stores instructions which, when executed by a device, cause the device to perform the scheduling method provided by the first aspect or any possible implementation thereof.

[0055] In a tenth aspect, a computer-readable storage medium is provided, the computer-readable storage medium including computer instructions that, when executed by a device, cause the device to perform the scheduling method provided by the second aspect or any possible implementation thereof.

[0056] In the eleventh aspect, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the scheduling method provided by the first aspect or any possible implementation thereof.

[0057] In a twelfth aspect, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the scheduling method provided by the second aspect or any possible implementation thereof.

[0058] It is understood that the beneficial effects achieved by the scheduling method of the second aspect provided above, as well as any scheduling device, computing cluster, data processing system, computer-readable storage medium and computer program product, can be referred to in accordance with the beneficial effects of the scheduling method provided in the first aspect or any possible implementation of the first aspect above, and will not be repeated here. Attached Figure Description

[0059] Figure 1 A schematic diagram illustrating an M-fold forward inference provided in an embodiment of this application;

[0060] Figure 2 A schematic diagram of an LLM inference provided in an embodiment of this application;

[0061] Figure 3 A schematic diagram illustrating a batch-level processing method provided in an embodiment of this application;

[0062] Figure 4 A schematic diagram of an iterative level processing provided in an embodiment of this application;

[0063] Figure 5 A schematic diagram illustrating another iteration-level processing provided in this application embodiment;

[0064] Figure 6 A schematic diagram illustrating an iteration-level preemption strategy provided in an embodiment of this application;

[0065] Figure 7 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0066] Figure 8 A flowchart illustrating a scheduling method provided in an embodiment of this application;

[0067] Figure 9 A schematic diagram illustrating a scheduling method provided in an embodiment of this application;

[0068] Figure 10 A flowchart illustrating another scheduling method provided in an embodiment of this application;

[0069] Figure 11 A schematic diagram illustrating the deployment of a computing cluster, provided as an embodiment of this application;

[0070] Figure 12 This is a schematic diagram of the structure of a general processing device provided in an embodiment of this application;

[0071] Figure 13 This is a schematic diagram of another general processing device provided in the embodiments of this application;

[0072] Figure 14 This is a schematic diagram of the structure of a computing cluster provided in an embodiment of this application. Detailed Implementation

[0073] The following sections will discuss the fabrication and use of various embodiments in detail. However, it should be understood that many applicable inventive concepts provided in this application can be implemented in a variety of specific environments. The specific embodiments discussed are merely illustrative of specific ways of implementing and using this description and technology, and do not limit the scope of this application.

[0074] Unless otherwise defined, all technical terms used herein have the same meaning as commonly known to one of ordinary skill in the art.

[0075] Each circuit or other component may be described or referred to as "for" performing one or more tasks. In this context, "for" is used to imply a structure by indicating that the circuit / component includes a structure (e.g., a circuit system) that performs one or more tasks during operation. Therefore, even when the specified circuit / component is currently inoperable (e.g., not turned on), it can still be referred to as "for performing that task." Circuits / components used with the term "for" include hardware, such as circuits that perform operations.

[0076] The technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, a, b, and c; where a, b, and c can be single or multiple.

[0077] The embodiments of this application use terms such as "first" and "second" to distinguish objects with similar names, functions, or effects. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or order of execution. The term "coupling" is used to indicate an electrical connection, including direct connection via wires or terminals or indirect connection via other devices. Therefore, "coupling" should be considered as a broad type of electronic communication connection.

[0078] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0079] Before introducing the embodiments of this application, the relevant application scenarios involved in this application will be described first.

[0080] Large language models (LLMs) refer to a class of neural network-based language models with a massive number of parameters (typically billions or more). LLMs can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence. Currently, LLMs mainly include the chat generative pre-trained transformer (GPT) series, the large language model meta AI (LLAMA) series, and the generalized linear model (GLM) series. For example, this GPT system could include the chat generative pre-trained transformer chatGPT. chatGPT is a natural language processing tool driven by artificial intelligence technology, capable of engaging in dialogue by understanding and learning human language. It can also interact based on the context of the chat and can even be used to complete tasks such as writing emails, video scripts, copywriting, translation, code, and academic papers.

[0081] The basic architecture of the above models all adopts a transformer architecture with only a decoder. During inference and prediction, this type of LLM architecture uses an auto-aggressive generative mode, meaning that the generation of each text unit (token) requires one forward inference process from the LLM model. For a sentence containing M tokens, it requires M forward inference processes from the LLM model. This forward inference can also be called forward iteration. For example, as shown... Figure 1 As shown, the LLM model can include M forward inference processes, and the processing architecture corresponding to each forward inference process can include an embedding layer and N transformer layers. Figure 1 The input text unit represents the token required to be input when each forward inference process is executed, the output text unit represents the token output after each forward inference process is completed, EOS represents the end marker (i.e., used to indicate the end of M forward inference processes), and the N transformer layers are represented as TL 1 to TL N.

[0082] In practical applications, LLMs typically have a large number of parameters, often on the order of billions, tens of billions, or even hundreds of billions. For example, GPT3, used in chatGPT, includes 96 transformer layers and contains 175 billion parameters. Therefore, these models require high responsiveness from both humans and machines. For instance, chatGPT, as a typical application of interactive AI technology, requires low latency inference from the LLM model, needing at least 50ms / token based on human reading speed.

[0083] The aforementioned models are typically deployed on heterogeneous accelerated computing clusters with multiple processing units, utilizing the parallel computing capabilities of these clusters to maximize data concurrency density. These heterogeneous accelerated computing clusters can also be referred to as high-performance heterogeneous acceleration devices. For example, these multiple processing units may include, but are not limited to: central processing unit (CPU), tensor processing unit (TPU), digital signal processor (DSP), neural network processing unit (NPU), graphics processing unit (GPU), image signal processor (ISP), data processing unit (DPU), microcontrollers or microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices.

[0084] One possible implementation provides a Faster Transformer (FT), which proposes using a Key-Value (KV) cache during transformer inference to reduce the computational load in the decryption stage. This significantly reduces redundant computations in the generative phase of the LLM model, making it an essential technique for LLM inference systems. The inference process of an LLM inference system employing KV caching is described below. Figure 2As shown, the LLM inference process using KV caching technology includes a prefill phase and a decoding phase. The prefill phase primarily processes the user's input text information (prompt) to generate the first text unit (first token). The first token complete time is calculated based on the maximum text length supported by the LLM model, which ranges from 1K to 4K. The decoding phase uses KV caching technology to generate subsequent text units token-by-token until the LLM generates the end-of-sequence (EOS) marker. The decoding complete time corresponds to this phase. Each generated text unit can output a response.

[0085] In real-time interactive LLM applications, the completion time of the first text unit determines the job first token complete time (FTCT) of the LLM request triggered by the LLM application. The LLM inference time (LLMinference job time) is equal to N times the decoding time and the first response time (FTCT).

[0086] By analyzing the computational characteristics of different LLMs, it was found that the computational features and distributed parallelism characteristics of the pre-filling and decoding stages differ. The pre-filling stage is characterized by high computational demands, which increase linearly with the length of the user's input text. Businesses with strict latency requirements for the first text unit typically use a single-batch approach for LLM inference. The decoding stage, on the other hand, features low computational cost per text unit after employing key-value caching technology. The main bottleneck lies in the amount of parameters to be moved, thus requiring multiple batches to improve utilization.

[0087] The total execution time of a single LLM inference run depends on the lengths of the input and output text information, but the length of the output text information cannot be known in advance. In multi-batch execution, batch-level scheduling requires waiting for the longest output text information within the entire batch to complete its iteration count before the entire batch can exit. For example, as shown... Figure 3As shown, in the LLM inference process of multiple requests (e.g., req1 to req4), if the length of the input text of req1 is three-sixths of 1024 tokens (represented as 1024T in the figure), the length of the input text of req2 is four-sixths of 1024 tokens, the length of the input text of req3 is five-sixths of 1024 tokens, and the length of the input text of req4 is two-sixths of 1024 tokens, and the length of the output text of req1 is five-sixths of 1024 tokens, the length of the output text of req2 is three-sixths of 1024 tokens, the length of the input text of req3 is two-sixths of 1024 tokens, and the length of the input text of req4 is 1024 tokens, then the output text of any request from req1 to req4 must wait until the output text of all requests from req1 to req4 is completed before it can be output. For example, the final output of req2, which has the shortest output text length, must wait until the output text of req4, which has the longest output text length, is completed before it can be output. Therefore, computing power is wasted during this period, which increases the cost of heterogeneous accelerated computing clusters for LLM inference.

[0088] To address the issues with the batch-level scheduling methods described above, an iteration-level dynamic scheduling scheme can be adopted. This scheme supports early exit for each task (or task corresponding to a request) within a batch, monitors the output status of the output text unit for each request, and reschedules a new task to be inserted at the exit position for computation once a request ends early (i.e., an end marker EOS appears). This improves the utilization rate of batch-level computation in the decoding phase of the heterogeneous accelerated computing cluster.

[0089] For example, such as Figure 4 As shown, assuming that tasks S1 to S4 correspond to different numbers of text units, then: In task S1, when the end marker EOS appears in text unit T6, task S6 can be inserted at position T7 for processing; in task S2, the end marker EOS appears in text unit T8; in task S3, when the end marker EOS appears in text unit T4, task S5 can be inserted at position T5 for processing; and in task S4, when the end marker EOS appears in text unit T7, task S7 can be inserted at position T8 for processing. In the diagram, T1 to T8 represent different text units T6.

[0090] Analysis of the LLM inference process reveals that it comprises a pre-filling phase and a multi-round iterative decoding phase. Therefore, the prerequisite for implementing this iterative-level dynamic scheduling scheme is that each scheduled task must first complete the pre-filling phase computation, i.e., utilize idle computing power for pre-filling computation. However, the pre-filling phase involves significant computational load and typically has a latency exceeding 500ms, while the decoding phase requires a smaller computational load and a latency of less than 50ms for a single text unit. If a new task's pre-filling and an existing task's decoding are simultaneously scheduled and executed on a heterogeneous accelerated computing cluster, the computational latency of the new task's pre-filling will far exceed the 50ms latency required for a single text unit in the decoding computation. For example, as shown... Figure 5 As shown, if the third task S3 has an end marker EOS at the position of text unit Ti+1, then the newly inserted fifth task S5 needs to be pre-filled to generate the first text unit T1 of the fifth task S5. At this time, the calculation of other text units of the original task (such as Ti+2, Ti+3, etc.) needs to wait until the first text unit T1 of the fifth task S5 is generated before the calculation is performed.

[0091] Furthermore, an iterative preemption strategy can be enabled at the granularity of each task's text unit. This means that whenever a task generates a text unit, it can be determined whether to continue executing the task or use other tasks in the queue to preempt its computational resources. For example, such as... Figure 6 As shown in (a), after the text units T1 and T2 of the first task S1 are generated, the second task S2 preempts and generates the text units T1 and T2 of the second task S2, and then the third task S3 can preempt and generate the text units T1 and T2 of the third task S3; or, as shown in (a), after the text units T1 and T2 of the third task S3 are generated; or, as shown in (a), after the text units T1 and T2 of the third task S3 are generated; or, as shown in (a), after the text units T1 and T2 of the third task S3 are generated; or, as shown in (a), after the text units T1 and T2 of the first task S1 ... second task S2 are generated; or, as shown in (a), after the text units T1 and T2 of the first task S1 Figure 6 As shown in (b), after the text unit T1 of the first task S1 is generated, the second task S2 preempts and generates the text unit T1 of the second task S2, then the third task S3 preempts and generates the text unit T1 of the third task S3, the first task S1 preempts and generates the text unit T2 of the first task S1 again, then the second task S2 preempts and generates the text unit T2 of the second task S2, and then the third task S3 preempts and generates the text unit T2 of the third task S3; or, as shown in (b), after the text unit T1 of the first task S1 is generated, the second task S2 preempts and generates the text unit T2 of the second task S2, then the third task S3 preempts and generates the text unit T2 of the third task S3; or, as shown in (b), after the text unit T1 of the first task S1 is generated, the second task S2 preempts and generates the text unit T2 of the second task S2, then the third task S3 preempts and generates the text unit T2 of the third task S3; or, as shown in (b), after the text unit T1 of the first task S1 is generated, the second task S2 preempts and generates the text unit T1 ...1 of the third task S3; or, as shown in (b), after the text unit T1 of the first task S1 is generated, the second task S2 preempts and generates the text Figure 6 As shown in (c), after the text unit T1 of the second task S2 is generated, the third task S3 preempts and generates the text unit T1 of the third task S3. Then, the second task S2 preempts and generates the text unit T2 of the second task S2. Next, the third task S3 preempts and generates the text unit T2 of the third task S3 again. Then, the first task S1 preempts and generates the text units T1 and T2 of the first task S1; or, as shown in (c), after the text unit T1 of the second task S2 is generated, the third task S3 preempts and generates the text unit T1 and T2 of the first task S1. Figure 6As shown in (d), after text units T1 and T2 of the second task S2 are generated, the third task S3 preempts and generates text units T1 and T2 of the third task S3, and then the first task S1 preempts and generates text units T1 and T2 of the first task S1. However, this method cannot meet the latency requirements for the first text unit and individual text units in real-time AI interaction scenarios.

[0092] Based on this, embodiments of this application provide a data processing system and scheduling method. This data processing system, through distributed deployment, batch dynamic scheduling technology, and key technologies for distributed transmission and storage that separate events and data, can meet the experience requirements of existing real-time LLM applications while improving the computing power utilization and system throughput of heterogeneous LLM computing clusters. Correspondingly, this method can be used to reduce the latency of LLM inference, meet the latency requirements of individual text units, and improve the throughput and computing power utilization of devices. Optionally, the data processing system can be a chip or an electronic device. This electronic device can include, but is not limited to: mobile phones, tablets, computers, laptops, ultra-mobile personal computers (umPCs), wearable devices, vehicles (e.g., cars, bicycles, electric vehicles, airplanes, ships, trains, high-speed trains, etc.), in-vehicle equipment, virtual reality (VR) devices, augmented reality (AR) devices, or intelligent robots, etc.

[0093] Figure 7 This is a schematic diagram of a data processing system provided in an embodiment of this application. The data processing system may include a general-purpose processing device 10, a computing cluster 20, and an interconnection device 30. The general-purpose processing device 10 can be used to schedule the computing cluster 20 to perform calculations, and different computing clusters in the computing cluster 20 can communicate with each other through the interconnection device 30.

[0094] Optionally, the general-purpose processing device 10 can be a host, and the computing cluster 20 can be a heterogeneous accelerated computing cluster. The computing cluster 20 may include a variety of processing units, which may include, but are not limited to: a central processing unit (CPU), a tensor processing unit (TPU), a digital signal processing unit (DSP), a neural network processing unit (NPU), a graphics processing unit (GPU), an image signal processor (ISP), a data processing unit (DPU), a microcontroller or microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices.

[0095] In one possible embodiment, the general-purpose processing device 10 may include a scheduling device 11, and the computing cluster 20 may include a first computing cluster 21 and a second computing cluster 22 communicating via the interconnect device 30. The scheduling device 11 can be used to schedule the first computing cluster 21 and the second computing cluster 22 to perform different computations. For example, the first computing cluster 21 can be used to perform pre-filling computations in LLM inference, and the second computing cluster 22 can be used to perform decoding computations in LLM inference; thus, the first computing cluster 21 can also be referred to as the pre-filling computing cluster, and the second computing cluster 22 can also be referred to as the decoding computing cluster. For example, when the data processing system is used for LLM inference, the scheduling device 11 can be referred to as an LLM scheduling device.

[0096] Optionally, the scheduling device 11 can also be used to provide control, time, and data transmission between the first computing cluster 21 and the second computing cluster 22, provide iterative-level scheduling control and early exit control for the decoding computing cluster, and simultaneously meet the requirements of latency for a single request in the pre-filling stage, as well as latency and multi-batch computing power utilization in the decoding stage.

[0097] In practical applications, the first computing cluster 21 and the second computing cluster 22 can be two different heterogeneous computing clusters. Each heterogeneous computing cluster can include a variety of processing units. The types and proportions of processing units included in different heterogeneous computing clusters, as well as the batch size, can be set according to actual needs. This application embodiment does not make specific limitations on this.

[0098] In another possible embodiment, the general-purpose processing device 10 may further include a deployment device 12, which can be used to deploy the first computing cluster 21 and the second computing cluster 22, for example, to initialize the first computing cluster 21 and the second computing cluster 22. For example, the first computing cluster 21 may include M first computing cluster instances, and the second computing cluster 22 may include N second computing cluster instances, and the deployment device 12 can be used to initialize the M first computing cluster instances and the N second computing cluster instances. For example, when the data processing system is used for LLM inference, the deployment device 12 may be referred to as an LLM deployment device.

[0099] Optionally, the number of computing cluster instances included in the first computing cluster 21 and the number of computing cluster instances included in the second computing cluster 22 can be dynamically increased or decreased. In one possible example, the computing cluster 20 may include an expanded X first computing cluster instances or Y second computing cluster instances, and the deployment device 12 can also be used to configure the X first computing cluster instances or Y second computing cluster instances. In another possible example, the deployment device 12 can also be used to change the roles of the first computing cluster instances and the second computing cluster instances to dynamically adjust the number of different computing instances.

[0100] In another possible embodiment, the data processing system may further include a deployment evaluation device 40, which can be used to evaluate and determine the number of first computing cluster instances included in the first computing cluster 21 and the number of second computing cluster instances included in the second computing cluster 22. For example, taking LLM inference as an example, the deployment evaluation device 40 can be an LLM deployment evaluation device. This device can evaluate and determine the number of first and second computing cluster instances based on historical data characteristics of the LLM application, i.e., determine the number of pre-filled computing cluster instances and the number of decoding computing cluster instances, to improve the adaptability of the deployment. These data characteristics may include, but are not limited to, the length of the input information, the length of the output information, and the number of iterations of the decoding computation. Furthermore, the data processing system may also include a resource management and allocation system, from which the deployment evaluation device 40 can request corresponding computing resources for deploying the number of first and second computing cluster instances.

[0101] In some other possible embodiments, the data processing system may further include a storage device 50, which may be a scalable distributed storage device, such as a key-value cache storage device. The first computing cluster 21 and the second computing cluster 22 may also communicate with the storage device 50 via the interconnect device 30. For example, the first computing cluster 21 and the second computing cluster 22 may write data to or read data from the storage device 50 via the interconnect device 30.

[0102] Optionally, when the data processing system also includes a storage device 50, the deployment device 12 can also be used to configure the storage device 50, for example, when initializing the M first computing cluster instances and the N second computing cluster instances, the storage device 50 can be initialized at the same time.

[0103] It is understandable that the above Figure 7The structure of the data processing system shown is exemplary. The structure described in the figure does not constitute a limitation on the structure of the data processing system. In practical applications, the data processing system may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0104] Figure 8 A scheduling method is provided in this application embodiment for scheduling a computing cluster, which includes a first computing cluster and a second computing cluster. The first computing cluster includes M instances of the first computing cluster, and the second computing cluster includes N instances of the second computing cluster. The M instances of the first computing cluster and the N instances of the second computing cluster communicate through an interconnection device, where M and N are positive integers. The method includes the following steps.

[0105] S201: The general-purpose processing device sends a first request for the target application to the first computing cluster instance. The first request includes first input information and is used to request computation on the first input information.

[0106] The first computing cluster instance can be any one of the M first computing cluster instances. This first computing cluster instance can be in an idle state, or its cache may contain available storage space. The first request can be sent by the target application to the general-purpose processing device based on user operations. The target application can be an LLM application, and the first input information in the first request can be text information entered by the user through the target application.

[0107] In one possible embodiment, when the scheduling device in the general processing device receives a first request from the target application, the scheduling device may send a first request to the first computing cluster instance if it determines that there is available storage space in the local cache of one of the M first computing cluster instances, so as to schedule the first computing cluster instance to perform a first calculation on the first input information through the first request, such as the first computing cluster instance performing a pre-fill calculation on the first input information.

[0108] S202: When the first computing cluster instance receives the first request, the first computing cluster instance calculates the first input information according to the first request to obtain the first text unit.

[0109] Specifically, when the first computing cluster instance receives the first request, it can perform pre-filling calculations on the first input information to obtain the first text unit, which may refer to the first text unit (first token) or the first text unit.

[0110] Optionally, after the first computing cluster instance obtains the first text unit, the first computing cluster instance can store the first text unit in a local cache. For example, the first computing cluster instance can store the first text unit in the first storage space of the local cache in the form of a KV cache.

[0111] Alternatively, after the first computing cluster instance obtains the first text unit, if its local cache is full or the remaining available space is insufficient to store the first text unit, the first computing cluster instance can store the first text unit in an externally extended storage device via an interconnect device. For example, the first computing cluster instance can store the first text unit in the second storage space of the storage device in the form of a KV cache. The first computing cluster instance can also generate storage indication information corresponding to the first text unit, such as a KV-Cache-ID.

[0112] S203: The first computing cluster instance sends a first instruction message to the general processing device, the first instruction message being used to indicate that the first text unit has been completed.

[0113] Specifically, after the first computing cluster instance obtains the first text unit, the first computing cluster instance can send a first indication message to the scheduling device in the general processing device to indicate that the first text unit has been completed, or to indicate that the pre-filling calculation corresponding to the first request has been completed.

[0114] S204: In response to the general-purpose processing device receiving the first instruction information, the general-purpose processing device sends a second request to the second computing cluster instance and a third request for the target application to the first computing cluster instance.

[0115] The second request is used to instruct the first text unit to be computed; for example, the first request is used to instruct the second computing device to perform decoding computation on the first text unit. The second computing cluster instance can be one of the N second computing cluster instances that is in an idle state.

[0116] Additionally, the third request includes the second input information, which can also be text information entered by the user through the target application. The third request is used to schedule the first computing cluster instance to perform computation on the second input information, such as scheduling the first computing cluster instance to perform pre-filling computation on the second input information.

[0117] In one possible embodiment, when the scheduling device in the general-purpose processing device receives the first indication information, in response to the first indication information, the scheduling device may send a second request to the second computing cluster instance to schedule the second computing cluster instance to perform decoding calculation on the first text unit; simultaneously, the scheduling device may also send a third request for the target application to the first computing cluster instance to schedule the first computing cluster instance to perform pre-filling calculation on the second input information. Accordingly, upon receiving the third request, the first computing cluster instance may perform pre-filling calculation on the second input information.

[0118] Optionally, when the scheduling device in the general processing device receives the first indication information, the scheduling device may also send a first response to the target application to indicate that the first text unit has been completed. This first response may also be called a first token response. In one example, the first response may include the first text unit, which the target application may display upon receiving it.

[0119] S205: When the second computing cluster instance receives the second request, the second computing cluster instance performs calculations on the first text unit according to the second request.

[0120] Optionally, upon receiving the second request, the second computing cluster instance can add the computing task corresponding to the second request to a task queue and execute them sequentially according to the order of the tasks in the task queue. For example, before the second request, the task queue may include other tasks job1, job2, job3, and job4. The second computing cluster instance can calculate the k-th token of job1, job2, job3, and job4. If job3 has an end marker EOS after the k-th token calculation is completed, the second computing cluster instance can retrieve the second request from the task queue and obtain the first text unit. Afterward, the second computing cluster instance can perform decoding calculations on the second request, as well as job1, job2, and job4, that is, calculate the (k+1)-th token of job1, job2, and job4, and calculate the second token (also called the second text unit) of the second request. In addition, the second computing cluster instance can also calculate multiple output text units such as the third token, the fourth token, etc., of the second request in a similar manner until the last output text unit is calculated. This embodiment of the application will not be elaborated further here.

[0121] In one possible embodiment, when the first text unit is stored in the local cache of the first computing cluster instance, the second request may include a first identifier of the first computing cluster instance, which is used by the second computing cluster instance to retrieve the first text unit from the first computing cluster instance. Thus, when the second computing cluster instance receives the second request, it can retrieve the first text unit from the first computing cluster instance based on the first identifier in the second request, and perform decoding calculations on the retrieved first text unit to obtain the second text unit.

[0122] Optionally, after obtaining the first text unit from the first computing cluster instance, the second computing cluster instance may also send a second indication message to the scheduling device in the general-purpose processing device. The second indication message indicates that the first text unit has been successfully obtained. Upon receiving the second indication message, the scheduling device in the general-purpose processing device may send a first notification message to the first computing cluster instance, instructing the first computing cluster instance to release the first storage space in its local cache containing the first text unit. Thus, upon receiving the first notification message, the first computing cluster instance can release the first storage space in its local cache containing the first text unit.

[0123] In this embodiment of the application, after the second computing cluster instance obtains the first text, it sends a second instruction to the scheduling device in the general processing device. The scheduling device notifies the first computing cluster instance to release the first storage space storing the first text unit according to the second instruction, which can improve the utilization rate of the storage space in the first computing cluster instance.

[0124] In another possible embodiment, when the first text unit is stored in an externally extended storage device, the second request may include storage indication information (e.g., KV-Cache-ID), which is used by the second computing cluster instance to retrieve the first text unit from the first computing cluster instance. Thus, when the second computing cluster instance receives the second request, it can retrieve the first text unit from the storage device according to the storage indication information in the second request, and perform decoding calculations on the retrieved first text unit to obtain the second text unit.

[0125] Optionally, after retrieving the first text unit from the storage device, the second computing cluster instance may also send a second indication message to the scheduling device in the general-purpose processing unit. The second indication message indicates that the retrieval of the first text unit has been completed. Upon receiving the second indication message, the scheduling device in the general-purpose processing unit may send a second notification message to the storage device, instructing the storage device to release the second storage space storing the first text unit. Thus, upon receiving the second notification message, the storage device can release the second storage space storing the first text unit.

[0126] In this embodiment, after acquiring the first text, the second computing cluster instance sends a second instruction to the scheduling device in the general processing device. The scheduling device, based on the second instruction, notifies the storage device to release the second storage space storing the first text unit, thereby improving the utilization rate of the storage space in the storage device. Furthermore, using this storage device to store the text units generated by the computing cluster instance during computation can adaptively meet the storage requirements brought about by increasing business traffic and longer input information.

[0127] For ease of understanding, the first computing cluster instance will be used as the pre-filled computing cluster instance, and the second computing cluster instance will be used as the decoding computing cluster instance. The explanations will be provided below. Figure 9 and Figure 10 The process of scheduling pre-filled computing cluster instances and decoding computing cluster instances in the embodiments of this application is illustrated by example. Figure 9 The first computing cluster instance stores the first text unit in its local cache. Figure 10 The first computing cluster instance stores the first text unit in an external storage device.

[0128] like Figure 9As shown, the scheduling method includes: S11. The scheduling device receives a first request job-j, the first request including first input information; S12. When the scheduling device determines that there is available storage space in the pre-filled computing cluster instance i, it sends the first request to the pre-filled computing cluster instance i, the computing task corresponding to the first request can be represented as job-j; S13. The pre-filled computing cluster instance i performs pre-filled calculation on the first input information and stores the calculated first text unit in a local cache; S14. The pre-filled computing cluster instance i sends a second request job-j-t1 to the scheduling device; S15. The scheduling device returns the received second request job-j-t1 to the business request and completes the first token response of job-j-t1; S16. When the scheduling device determines that the decoding computing cluster instance k is idle, it sends the information {job-j-t1, ID-i} to the decoding computing cluster instance k, where ID-i is the identifier of the pre-filled computing cluster instance i; S17. When the decoding computing cluster instance k receives the information, it adds it to the task... In the task queue, which also includes other tasks job1, job2, job3, and job4, the decoding computing cluster instance k calculates the k-th token for job1, job2, job3, and job4. If job3 has an end marker EOS after the k-th token calculation is completed, the decoding computing cluster instance k retrieves the information from the task queue and obtains the first text unit from the pre-filled computing cluster instance i. S18. The decoding computing cluster instance k sends a second indication message {job-j, ID-i} to the scheduling device to indicate that the first text information has been obtained and to perform decoding calculation on the first text unit, as well as decoding calculation on job1, job2, and job4, that is, to perform the (k+1)-th token calculation on job1, job2, and job4, and to perform the second token calculation on the second request. S19. When the scheduling device receives the second indication message, it sends a notification to the pre-filled computing cluster instance i to notify the pre-filled computing cluster instance i to release the storage space storing the first text unit.

[0129] like Figure 10As shown, the scheduling method includes: S21. The scheduling device receives a first request job-j, the first request including first input information; S22. When the scheduling device determines that there is available storage space in the storage device and the pre-filled computing cluster instance i is idle, the scheduling device sends a first request to the pre-filled computing cluster instance i, the computing task corresponding to the first request can be represented as job-j; S23. The pre-filled computing cluster instance i performs pre-filled calculation on the first input information, stores the calculated first text unit in the storage device through the interconnection device, and generates storage indication information KV-Cache-ID; S24. The pre-filled computing cluster instance i sends a second request job-j-t1 to the scheduling device; S25. The scheduling device returns the received second request job-j-t1 to the business request and completes the first token response of job-j-t1; S26. When the scheduling device determines that the decoding computing cluster instance k is idle, it sends information {job-j-t1, KV-Cache-ID} to the decoding computing cluster instance k; S27. When the decoding computing cluster instance k receives... When this information is received, it is added to the task queue, which also includes other tasks job1, job2, job3, and job4. The decoding computing cluster instance k calculates the k-th token for job1, job2, job3, and job4. If job3 has an end marker EOS after the k-th token calculation is completed, the decoding computing cluster instance k retrieves the information from the task queue and obtains the first text unit from the storage device according to KV-Cache-ID through the interconnect device; S28. The decoding computing cluster instance k sends a second indication message {job-j, ID-i} to the scheduling device to indicate that the first text information has been obtained and to perform decoding calculation on the first text unit, as well as decoding calculation on job1, job2, and job4, that is, job1, job2, and job4 perform the (k+1)-th token calculation, and the second request performs the second token calculation; S29. When the scheduling device receives the second indication message, it sends a notification to the storage device to notify the storage device to release the storage space storing the first text unit.

[0130] The above Figure 9 and Figure 10The above example illustrates the scheduling device, which includes a request monitor, a job scheduler, and a response stream; the prefilled computing cluster instance i, which includes a prefill executor and a cache; and the decoding computing cluster instance k, which includes a decode executor and a cache. This example does not limit the structure of the scheduling device, the prefilled computing cluster instance, and the decoding computing cluster instance.

[0131] Furthermore, the number of the M first computing cluster instances and the number of the N second computing cluster instances can be determined based on the data characteristics of the target application, which may include the length of the input information, the length of the output information, and the number of iterations of the decoding computation.

[0132] The target application can be an LLM application. Different services within an LLM application may have different input lengths, output lengths, and decoding iterations. For example, service 1 is characterized by long input lengths, typically ranging from 4K to 8K, short output lengths, and fewer decoding iterations; service 2 is characterized by short input lengths, long output lengths, and more decoding iterations.

[0133] In one possible embodiment, such as Figure 11 As shown, if the target application is an LLM application, the deployment evaluation device in the aforementioned data processing system can evaluate and determine the number of M first computing cluster instances and the number of N second computing cluster instances in the first computing cluster based on the historical data characteristics of the LLM application, such as the length of the input information, the length of the output information, and the number of decoding iterations corresponding to the target application. In this way, under the diverse data characteristics of LLM applications, the number of first computing cluster instances to be deployed in the first computing cluster and the number of second computing cluster instances to be deployed in the second computing cluster can be adaptively determined to meet load balancing requirements. Figure 9 The following example illustrates the process, using the first computing cluster instance as the pre-filled computing cluster instance and the second computing cluster instance as the decoding computing cluster instance, with an estimated number of 3 pre-filled computing cluster instances and 2 decoding computing cluster instances.

[0134] Furthermore, after assessing and determining the number of the M first computing cluster instances and the N second computing cluster instances, the deployment device in the data processing system can also deploy the M first computing cluster instances and the N second computing cluster instances.

[0135] In one possible embodiment, the deployment device can initialize the M first computing cluster instances and the N second computing cluster instances; further, the deployment device can also initialize the interconnection device between the M first computing cluster instances and the N second computing cluster instances. For example, as shown... Figure 11 As shown, assuming that the M first computing cluster instances are 3 pre-filled computing cluster instances (represented as instance 1 to instance 3), and the N second computing cluster instances are 2 pre-filled computing cluster instances, the deployment device can initialize the 3 pre-filled computing cluster instances, the 2 pre-filled computing cluster instances, and the interconnection device between the two types of computing cluster instances.

[0136] In this embodiment, the deployment evaluation device determines the number of first computing cluster instances that need to be deployed in the first computing cluster and the number of second computing cluster instances that need to be deployed in the second computing cluster based on the data characteristics of the target application. The deployment device performs deployment based on the evaluation results, which can improve the adaptability of the deployment of the first computing cluster and the second computing cluster, while ensuring that the first computing cluster and the second computing cluster meet the load balance.

[0137] Optionally, the number of first computing cluster instances included in the first computing cluster and the number of second computing cluster instances included in the second computing cluster can be dynamically increased or decreased.

[0138] In a first possible embodiment, when a first preset condition is met, the deployment device can dynamically increase the number of first computing cluster instances. For example, the first preset condition may include: the local cache usage of the M first computing cluster instances is greater than a first preset usage limit, or the computing load of the M first computing cluster instances is greater than a first load threshold. The first preset usage limit and the first load threshold can be preset and can be set by those skilled in the art according to actual conditions; this application embodiment does not impose specific limitations on them.

[0139] During the initial deployment phase, the deployment device deploys M first computing cluster instances and N second computing cluster instances based on the evaluation results from the evaluation device. After running for a period of time, the business data characteristics of the target application change; for example, the length of the input information increases several times, causing the local cache of the M first computing cluster instances to be full or the computing load to reach its limit. At this point, the number of first computing cluster instances can be dynamically increased.

[0140] In one example, if the computing cluster also includes X additional first computing cluster instances, the deployment device can initialize these X first computing cluster instances. Furthermore, the deployment device can also initialize the interconnection devices between these X first computing cluster instances and the N second computing cluster instances, where X is a positive integer. That is, the number of first computing cluster instances can be dynamically increased by adding X more first computing cluster instances.

[0141] In another example, if a large number of tasks have been completed in the N second computing cluster instances, resulting in idle computing power, the deployment device can configure X of the N second computing cluster instances as X first computing cluster instances. Furthermore, the deployment device can initialize the interconnection devices between the X first computing cluster instances and the remaining N X second computing cluster instances. That is, the number of first computing cluster instances can be dynamically increased by changing the role of some second computing cluster instances to first computing cluster instances.

[0142] In a second possible embodiment, when a second preset condition is met, the deployment device can dynamically increase the number of second computing cluster instances. For example, the second preset condition may include: the local cache usage of the N second computing cluster instances is greater than a second preset usage limit, or the computing load of the N second computing cluster instances is greater than a second load threshold. The second preset usage limit and the second load threshold can be preset and can be specifically set by those skilled in the art according to actual conditions; this application embodiment does not impose specific limitations on this.

[0143] During the initial deployment phase, the deployment device deploys M first computing cluster instances and N second computing cluster instances based on the evaluation results from the evaluation device. After running for a period of time, the business data characteristics of the target application change; for example, the length of the output information increases several times, and the number of iterations doubles, causing the local cache of the N second computing cluster instances to be full or the computing load to reach its limit. At this point, the number of second computing cluster instances can be dynamically increased.

[0144] In one example, if the computing cluster also includes Y additional second computing cluster instances, the deployment device can initialize these Y second computing cluster instances. Furthermore, the deployment device can also initialize the interconnection devices between these Y second computing cluster instances and the M first computing cluster instances, where Y is a positive integer. That is, the number of second computing cluster instances can be dynamically increased by adding Y new second computing cluster instances.

[0145] In another example, if a large number of tasks have been completed in the M first computing cluster instances, resulting in idle computing power, the deployment device can configure Y of the M first computing cluster instances as Y second computing cluster instances. Furthermore, the deployment device can initialize the interconnection devices between the Y second computing cluster instances and the remaining MY first computing cluster instances. That is, the number of second computing cluster instances can be dynamically increased by changing the role of some first computing cluster instances to second computing cluster instances.

[0146] In this embodiment, the deployment device can dynamically increase or decrease the number of first and second computing cluster instances based on changes in the data characteristics of the target application, thereby improving the responsiveness of the target application's business requests and enhancing its cost-performance ratio. Furthermore, by changing the roles of some first computing cluster instances to second computing cluster instances, the computing cluster instances whose roles have changed can share model weights during the pre-filling and decoding computation processes, thus avoiding the relocation of model weights.

[0147] The foregoing primarily describes the solutions provided in the embodiments of this application from the perspective of the interaction between general-purpose processing devices and computing clusters. It is understood that, in order to achieve the aforementioned functions, general-purpose processing devices and computing clusters include corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0148] This application embodiment can divide the general processing device and computing cluster points into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The above functional modules can be implemented in hardware or software. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. The following description uses the division of each functional module according to its corresponding function as an example:

[0149] When using integrated units, Figure 12A possible structural diagram of the general processing apparatus involved in the above embodiments is shown. The apparatus includes a sending unit 301 and a receiving unit 302. In one possible embodiment, the sending unit 301 can be used to support the apparatus in executing one or more steps of S201, S204, and sending first notification information and second communication information in the above method embodiments; the receiving unit 302 can be used to support the apparatus in receiving first indication information in S203 of the above method embodiments, or to support the apparatus in executing the step of receiving second indication information sent by a second computing cluster instance in the above method embodiments. Further, the apparatus also includes a processing unit 303; wherein the processing unit 303 can be used to support the apparatus in executing the steps of initializing the first computing cluster instance, the second computing cluster instance, the interconnect device, and the storage device in the above method embodiments, as well as the steps of changing the roles of the first computing cluster instance and the second computing cluster instance, and / or other technical processes described herein.

[0150] Based on hardware implementation, the above-mentioned processing unit 303 can be a processor, the sending unit 301 can be a transmitter, and the receiving unit 302 can be a receiver. The receiver and the transmitter can be integrated into a transceiver, which can also be called a communication interface.

[0151] Figure 13 This is a schematic diagram of a possible structure of a general-purpose processing device according to an embodiment of this application. The device includes a memory 311 and a processor 312. The memory 311 stores the device's program code and data, and the processor 312 controls the operation of the general-purpose processing device in the above method embodiments. For example, the processor 312 supports the device in performing the steps of initializing the first computing cluster instance, the second computing cluster instance, the interconnect device, and the storage device in the above method embodiments, as well as the steps of changing the roles of the first computing cluster instance and the second computing cluster instance, and / or other technical processes described herein. Optionally, the device may also include a communication interface 313, which supports the device in performing the steps of communicating with the computing cluster and storage device in the above method embodiments.

[0152] The processor 312 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a processing chip, a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various logic blocks, modules, and circuits described in connection with the embodiments of this application. The processor 312 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The communication interface 313 can be a transceiver, transceiver circuitry, or transceiver interface, etc. The memory 311 can be volatile memory or non-volatile memory, etc.

[0153] For example, communication interface 313, processor 312, and memory 311 are interconnected via bus 314; bus 314 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. Bus 314 can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the figure, but this does not mean that there is only one bus or one type of bus.

[0154] Optionally, the memory 311 may be included in the processor 312.

[0155] It is understood that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0156] Figure 14 This is a schematic diagram of a computing cluster structure provided in an embodiment of this application. The computing cluster includes a first computing cluster and a second computing cluster. The first computing cluster includes M instances of the first computing cluster, and the second computing cluster includes N instances of the second computing cluster. The M instances of the first computing cluster and the N instances of the second computing cluster communicate through an interconnection device, where M and N are positive integers.

[0157] In one possible embodiment, a first computing cluster instance is configured to: receive a first request from a general-purpose processing device for a target application, the first request including first input information, the first request being used to request computation on the first input information; calculate the first input information to obtain a first text unit according to the first request; send a first indication to the general-purpose processing device, the first indication being used to indicate that the first text unit has been completed; and, in response to the first indication, receive a third request from the general-purpose processing device for the target application, the third request including second input information, the third request being used to request computation on the second input information; a second computing cluster instance is configured to: receive a second request from the general-purpose processing device, the second request being used to indicate computation on the first text unit.

[0158] Optionally, the first text unit is stored in a local cache of the first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance. In one example, the second computing cluster instance is further configured to: retrieve the first text unit from the first computing cluster instance via the interconnect device based on the first identifier of the first computing cluster instance.

[0159] Furthermore, the second computing cluster instance is also configured to: send a second indication message to the general-purpose processing device, the second indication message being used to indicate that the first text unit has been acquired; the first computing cluster instance is also configured to: receive a first notification message sent by the general-purpose processing device, the first notification message being sent by the general-purpose processing device according to the second indication message; the first computing cluster instance is also configured to: release the first storage space storing the first text unit in the local cache according to the first notification message.

[0160] Alternatively, the first computing cluster instance and the second computing cluster instance may further communicate with the storage device via an interconnect device, and the second request may include storage indication information. In one example, the first computing cluster instance may further be configured to: store the first text unit in the storage device via the interconnect device; and the second computing cluster instance may further be configured to: retrieve the first text unit from the storage device via the interconnect device according to the storage indication information.

[0161] Furthermore, the second computing cluster instance is also used to: send a second indication message to the general-purpose processing device, the second indication message being used to indicate that the first text unit has been acquired; wherein, the second indication message is used by the general-purpose processing device to send a second notification message to the storage device, the second notification message being used by the storage device to release the first storage space storing the first text unit.

[0162] In another possible embodiment, the first computing cluster instance is also used to perform initialization operations, and the second computing cluster instance is also used to perform initialization operations.

[0163] Furthermore, the first computing cluster instance is also configured to: receive first configuration information sent by the general-purpose processing device, the first configuration information being used to configure the first computing cluster instance as a second computing cluster instance. Similarly, the second computing cluster instance is also configured to: receive second configuration information sent by the general-purpose processing device, the second configuration information being used to configure the first computing cluster instance as a second computing cluster instance.

[0164] Optionally, the target application includes a large language model LLM application, where the first computing cluster is a computing cluster for performing pre-filling computations and the second computing cluster is a computing cluster for performing decoding computations.

[0165] In another aspect of this application, a computing cluster is also provided, the computing cluster including a processor and a memory, the memory storing instructions that, when the processor executes the instructions, cause the computing cluster to perform the steps of the first computing cluster instance and the second computing cluster instance included in the method embodiments provided above.

[0166] In another aspect of this application, a data processing system is also provided, which includes a general-purpose processing device and a computing cluster; wherein the general-purpose processing device can be any of the general-purpose processing devices provided above, and is used for the steps of the general-purpose processing device group in the above method embodiments; the computing cluster can be any of the computing clusters provided above, and is used for the steps of the first computing cluster instance and the second computing cluster instance included in the computing cluster in the above method embodiments.

[0167] The methods or apparatus provided in this application can be implemented entirely or partially by software, hardware, or a combination thereof. When implemented using software, they can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, twisted pair) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more media. The medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., optical disk), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0168] In another aspect of this application, a computer-readable storage medium is provided, which includes computer instructions that, when executed by a device, cause the device to perform the steps of the general processing apparatus in the scheduling method provided in the above-described method embodiments.

[0169] In another aspect of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including computer instructions, which, when executed by a device, cause the device to perform the steps of the scheduling method provided in the above method embodiments and the first computing cluster instance and / or the second computing cluster instance.

[0170] In another aspect of this application, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the steps of the general processing apparatus in the scheduling method provided in the above-described method embodiments.

[0171] In another aspect of this application, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the steps of the scheduling method provided in the above-described method embodiments for the first computing cluster instance and / or the second computing cluster instance.

[0172] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A scheduling method, characterized in that, For scheduling computing clusters, the computing clusters include a first computing cluster and a second computing cluster, the first computing cluster is for performing pre-filling computation, and the second computing cluster is for performing decoding computation, the first computing cluster includes M instances of the first computing cluster, and the second computing cluster includes N instances of the second computing cluster, where M and N are positive integers, the method includes: Send a first request for the target application to the first computing cluster instance. The first request includes first input information and is used to request computation on the first input information. Receive a first indication message from the first computing cluster instance, the first indication message being used to indicate that a first text unit has been completed, the first text unit being calculated by the first computing cluster instance from the first input information; In response to the first indication information, a second request is sent to the second computing cluster instance, and a third request for the target application is sent to the first computing cluster instance. The second request is used to instruct computation on the first text unit, and the third request includes second input information and is used to request computation on the second input information. The first text unit is stored in the local cache of the first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance, which is used by the second computing cluster instance to retrieve the first text unit from the first computing cluster instance; or The first computing cluster instance and the second computing cluster instance also communicate with a storage device, in which the first text unit is stored, and the second request includes storage indication information for the second computing cluster instance to retrieve the first text unit from the storage device.

2. The method according to claim 1, characterized in that, The method further includes: Receive a second indication message from the second computing cluster instance, the second indication message being used to indicate that the first text unit has been acquired; According to the second instruction information, a first notification information is sent to the first computing cluster instance, the first notification information being used to instruct the first computing cluster instance to release the first storage space in the local cache that stores the first text unit.

3. The method according to claim 1, characterized in that, The method further includes: Receive a second indication message from the second computing cluster instance, the second indication message being used to indicate that the first text unit has been acquired; A second notification message is sent to the storage device according to the second instruction message, the second notification message being used to instruct the storage device to release the second storage space storing the first text unit.

4. The method according to any one of claims 1-3, characterized in that, The number of the M first computing cluster instances and the number of the N second computing cluster instances are determined based on the data characteristics of the target application, which include the length of the input information, the length of the output information, and the number of iterations for decoding computation.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: Initialize the M first computing cluster instances; Initialize the N second computing cluster instances; The interconnection device between the M first computing cluster instances and the N second computing cluster instances is initialized.

6. The method according to any one of claims 1-3, characterized in that, When the first preset condition is met, the method further includes: If the computing cluster further includes X extended first computing cluster instances, the X first computing cluster instances are initialized, and the interconnection device between the X first computing cluster instances and the N second computing cluster instances is initialized, where X is a positive integer; or, X of the N second computing cluster instances are configured as X first computing cluster instances, and the interconnection device between the X first computing cluster instances and the remaining NX second computing cluster instances is initialized.

7. The method according to claim 6, characterized in that, The first preset condition includes: the local cache usage of the M first computing cluster instances is greater than a preset usage, or the computing load of the M first computing cluster instances is greater than a first load threshold.

8. The method according to any one of claims 1-3 and 7, characterized in that, When the second preset condition is met, the method further includes: If the computing cluster further includes Y extended second computing cluster instances, the Y second computing cluster instances are initialized, and the interconnection device between the Y second computing cluster instances and the M first computing cluster instances is initialized; or, Configure Y of the M first computing cluster instances as Y second computing cluster instances, and initialize the interconnection device between the Y second computing cluster instances and the remaining MY first computing cluster instances.

9. The method according to claim 8, characterized in that, The second preset condition includes: the local cache usage of the N second computing cluster instances is greater than the preset usage, or the computing load of the N second computing cluster instances is greater than the first load threshold.

10. The method according to any one of claims 1-3, 7, and 9, characterized in that, The target applications include Large Language Model (LLM) applications.

11. A scheduling method for scheduling a computing cluster, characterized in that, The computing cluster includes a first computing cluster and a second computing cluster. The first computing cluster is used to perform pre-filling computation, and the second computing cluster is used to perform decoding computation. The first computing cluster includes M instances of the first computing cluster, and the second computing cluster includes N instances of the second computing cluster, where M and N are positive integers. The method includes: The first computing cluster instance receives a first request from a target application sent by a general processing device. The first request includes first input information and is used to request computation on the first input information. The first computing cluster instance calculates the first text unit based on the first request and the first input information; The first computing cluster instance sends a first indication message to the general processing device, the first indication message being used to indicate that the first text unit has been completed; In response to the first indication information, the first computing cluster instance receives a third request from the target application sent by the general processing device, the third request including second input information, the third request being used to request computation on the second input information; The second computing cluster instance receives a second request sent by the general processing device, the second request being used to instruct computation to be performed on the first text unit; The first text unit is stored in the local cache of the first computing cluster instance, the second request includes a first identifier of the first computing cluster instance, and the method further includes: The second computing cluster instance obtains the first text unit from the first computing cluster instance based on the first identifier of the first computing cluster instance; or The first computing cluster instance and the second computing cluster instance also communicate with a storage device, the second request includes storage indication information, and the method further includes: The first computing cluster instance stores the first text unit in the storage device; The second computing cluster instance retrieves the first text unit from the storage device according to the storage instruction information.

12. The method according to claim 11, characterized in that, The method further includes: The second computing cluster instance sends a second indication message to the general processing device, the second indication message being used to indicate that the first text unit has been acquired; The first computing cluster instance receives a first notification message sent by the general processing device, wherein the first notification message is sent by the general processing device according to the second instruction message; The first computing cluster instance releases the first storage space in the local cache that stores the first text unit according to the first notification information.

13. The method according to claim 11, characterized in that, The method further includes: The second computing cluster instance sends a second indication message to the general processing device, the second indication message being used to indicate that the first text unit has been acquired; Wherein, the second instruction information is used by the general processing device to send a second notification information to the storage device, and the second notification information is used by the storage device to release the first storage space storing the first text unit.

14. The method according to any one of claims 11-13, characterized in that, The method further includes: The first computing cluster instance performs initialization operations; The second computing cluster instance performs initialization operations.

15. The method according to any one of claims 11-13, characterized in that, The method further includes: The first computing cluster instance receives first configuration information sent by the general processing device, the first configuration information being used to configure the first computing cluster instance as a second computing cluster instance.

16. The method according to any one of claims 11-13, characterized in that, The method further includes: The second computing cluster instance receives second configuration information sent by the general processing device, the second configuration information being used to configure the first computing cluster instance as the second computing cluster instance.

17. The method according to any one of claims 11-13, characterized in that, The target applications include Large Language Model (LLM) applications.

18. A scheduling method, characterized in that, For scheduling computing clusters, the computing clusters include a first computing cluster and a second computing cluster, the first computing cluster is for performing pre-filling computation, and the second computing cluster is for performing decoding computation, the first computing cluster includes M instances of the first computing cluster, and the second computing cluster includes N instances of the second computing cluster, where M and N are positive integers, the method includes: The general-purpose processing device sends a first request for the target application to the first computing cluster instance. The first request includes first input information and is used to request computation on the first input information. The first computing cluster instance receives the first request and calculates the first text unit based on the first input information according to the first request; The first computing cluster instance sends a first indication message to the general processing device, the first indication message being used to indicate that the first text unit has been completed; The general processing device receives first indication information from the first computing cluster instance, the first indication information being used to indicate that the first text unit has been completed; In response to the first indication information, the general processing device sends a second request to the second computing cluster instance and a third request for the target application to the first computing cluster instance. The second request is used to instruct the first text unit to be computed, and the third request includes second input information and is used to request the second input information to be computed. The first computing cluster instance receives the third request and performs calculations on the second input information based on the third request; The second computing cluster instance receives the second request and performs calculations on the first text unit; The first text unit is stored in the local cache of the first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance, which is used by the second computing cluster instance to retrieve the first text unit from the first computing cluster instance; or The first computing cluster instance and the second computing cluster instance also communicate with a storage device, in which the first text unit is stored, and the second request includes storage indication information for the second computing cluster instance to retrieve the first text unit from the storage device.

19. A scheduling device, characterized in that, For scheduling computing clusters, the computing clusters include a first computing cluster and a second computing cluster, the first computing cluster being a computing cluster for performing pre-filling computation, and the second computing cluster being a computing cluster for performing decoding computation, the first computing cluster including M instances of the first computing cluster, and the second computing cluster including N instances of the second computing cluster, where M and N are positive integers, the device includes: The sending unit is configured to send a first request of the target application to the first computing cluster instance, the first request including first input information, and the first request is used to request computation on the first input information; The receiving unit is configured to receive first indication information from the first computing cluster instance, the first indication information being used to indicate that the first text unit has been completed, the first text unit being calculated by the first computing cluster instance from the first input information; The sending unit is further configured to, in response to the first indication information, send a second request to the second computing cluster instance and a third request to the first computing cluster instance for the target application, wherein the second request is configured to instruct the first text unit to be computed, and the third request includes second input information and is configured to request the second input information to be computed. The first text unit is stored in the local cache of the first computing cluster instance, and the second request includes a first identifier of the first computing cluster instance, which is used by the second computing cluster instance to retrieve the first text unit from the first computing cluster instance; or The first computing cluster instance and the second computing cluster instance also communicate with a storage device, in which the first text unit is stored, and the second request includes storage indication information for the second computing cluster instance to retrieve the first text unit from the storage device.

20. The apparatus according to claim 19, characterized in that, The receiving unit is further configured to receive second indication information from the second computing cluster instance, the second indication information being used to indicate that the first text unit has been acquired; The sending unit is further configured to send a first notification message to the first computing cluster instance according to the second instruction information, wherein the first notification message is configured to instruct the first computing cluster instance to release the first storage space in the local cache that stores the first text unit.

21. The apparatus according to claim 19, characterized in that, The device includes: The receiving unit is further configured to receive second indication information from the second computing cluster instance, the second indication information being used to indicate that the first text unit has been acquired; The sending unit is further configured to send a second notification message to the storage device according to the second instruction information, the second notification message being used to instruct the storage device to release the second storage space storing the first text unit.

22. The apparatus according to any one of claims 19-21, characterized in that, The number of the M first computing cluster instances and the number of the N second computing cluster instances are determined based on the data characteristics of the target application, which include the length of the input information, the length of the output information, and the number of iterations for decoding computation.

23. The apparatus according to any one of claims 19-21, characterized in that, The device further includes: A processing unit is used to initialize the M first computing cluster instances; The processing unit is also used to initialize the N second computing cluster instances; The processing unit is also used to initialize the interconnection device between the M first computing cluster instances and the N second computing cluster instances.

24. The apparatus according to any one of claims 19-21, characterized in that, The device further includes a processing unit, which, when a first preset condition is met, is further configured to: If the computing cluster further includes X extended first computing cluster instances, the X first computing cluster instances are initialized, and the interconnection device between the X first computing cluster instances and the N second computing cluster instances is initialized, where X is a positive integer; or, X of the N second computing cluster instances are configured as X first computing cluster instances, and the interconnection device between the X first computing cluster instances and the remaining NX second computing cluster instances is initialized.

25. The apparatus according to claim 24, characterized in that, The first preset condition includes: the local cache usage of the M first computing cluster instances is greater than a preset usage, or the computing load of the M first computing cluster instances is greater than a first load threshold.

26. The apparatus according to any one of claims 19-21, 25, characterized in that, The device further includes a processing unit, which, when a second preset condition is met, is further configured to: If the computing cluster further includes Y extended second computing cluster instances, the Y second computing cluster instances are initialized, and the interconnection device between the Y second computing cluster instances and the M first computing cluster instances is initialized; or, Configure Y of the M first computing cluster instances as Y second computing cluster instances, and initialize the interconnection device between the Y second computing cluster instances and the remaining MY first computing cluster instances.

27. The apparatus according to claim 26, characterized in that, The second preset condition includes: the local cache usage of the N second computing cluster instances is greater than the preset usage, or the computing load of the N second computing cluster instances is greater than the first load threshold.

28. The apparatus according to any one of claims 19-21, 25, and 27, characterized in that, The target applications include Large Language Model (LLM) applications.

29. A computing cluster, characterized in that, The computing cluster includes a first computing cluster and a second computing cluster. The first computing cluster is used to perform pre-filling calculations, and the second computing cluster is used to perform decoding calculations. The first computing cluster includes M instances of the first computing cluster, and the second computing cluster includes N instances of the second computing cluster, where M and N are positive integers. The first computing cluster instance is configured to receive a first request from a target application sent by a general-purpose processing device. The first request includes first input information and is configured to request computation on the first input information. The first computing cluster instance is further configured to calculate the first text unit based on the first request using the first input information; The first computing cluster instance is further configured to send a first indication message to the general processing device, the first indication message being used to indicate that the first text unit has been completed; The first computing cluster instance is further configured to, in response to the first indication information, receive a third request from the target application sent by the general processing device, the third request including second input information, the third request being used to request computation on the second input information; The second computing cluster instance is configured to receive a second request sent by the general processing device, the second request being used to instruct computation to be performed on the first text unit; The first text unit is stored in the local cache of the first computing cluster instance, and the second request includes the first identifier of the first computing cluster instance; The second computing cluster instance is further configured to obtain the first text unit from the first computing cluster instance based on the first identifier of the first computing cluster instance; or The first computing cluster instance and the second computing cluster instance also communicate with the storage device, and the second request includes storage indication information; The first computing cluster instance is further configured to store the first text unit in the storage device; The second computing cluster instance is further configured to retrieve the first text unit from the storage device according to the storage indication information.

30. The computing cluster according to claim 29, characterized in that, The second computing cluster instance is further configured to send a second indication message to the general processing device, the second indication message being used to indicate that the first text unit has been acquired; The first computing cluster instance is further configured to receive a first notification message sent by the general processing device, wherein the first notification message is sent by the general processing device according to the second instruction message; The first computing cluster instance is further configured to release the first storage space in the local cache that stores the first text unit, based on the first notification information.

31. The computing cluster according to claim 29, characterized in that, The second computing cluster instance is further configured to send a second indication message to the general processing device, the second indication message being used to indicate that the first text unit has been acquired; Wherein, the second instruction information is used by the general processing device to send a second notification information to the storage device, and the second notification information is used by the storage device to release the first storage space storing the first text unit.

32. The computing cluster according to any one of claims 29-31, characterized in that, The first computing cluster instance is also used to perform initialization operations; The second computing cluster instance is also used to perform initialization operations.

33. The computing cluster according to any one of claims 29-31, characterized in that, The first computing cluster instance is further configured to receive first configuration information sent by the general processing device, the first configuration information being configured as a second computing cluster instance.

34. The computing cluster according to any one of claims 29-31, characterized in that, The second computing cluster instance is further configured to receive second configuration information sent by the general processing device, the second configuration information being configured to configure the first computing cluster instance as the second computing cluster instance.

35. The computing cluster according to any one of claims 29-31, characterized in that, The target applications include Large Language Model (LLM) applications.

36. A scheduling device, characterized in that, The scheduling device includes a processor and a memory, the memory storing instructions that, when executed by the processor, cause the device to perform the scheduling method as described in any one of claims 1-10.

37. A computing cluster, characterized in that, The computing cluster includes a processor and a memory, the memory storing instructions that, when executed by the processor, cause the computing cluster to perform the scheduling method as described in any one of claims 11-17.

38. A data processing system, characterized in that, The data processing system includes a general-purpose processing unit and a computing cluster. The general-purpose processing unit is used to execute the scheduling method as described in any one of claims 1-10, and the computing cluster is used to execute the scheduling method as described in any one of claims 11-17.

39. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by the device, cause the device to perform the scheduling method as described in any one of claims 1-17.

40. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, the computer performs the scheduling method as described in any one of claims 1-17.

Images