Computing system, computing power allocation method, electronic device and readable storage medium

Abstract

The present invention discloses a computing system, a computing power allocation method, an electronic device and a readable storage medium, which relate to the field of high-performance computing technology, including: a resource scheduler in the computing system pools multiple computing devices to obtain a first computing resource pool for providing computing power for the pre-filling stage, and a third computing resource pool for providing computing power for the decoding stage. According to the computing power indicators of the first computing resource pool and / or the third computing resource pool, the computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool can be adjusted to achieve dynamic adjustment of the computing power of the computing resource pools in different stages. Therefore, it can solve the technical problem that the related technology requires multiple readings and writings to complete the processing task of input information once, resulting in large equipment idling and computing resource waste, thereby achieving the technical effect of reducing the frequency of equipment idling and reducing computing resource waste.

Description

Technical Field

[0001] The present application relates to the field of high-performance computing technology, and in particular to a computing system, a computing power allocation method, an electronic device, and a readable storage medium. Background Art

[0002] During the inference process of an AI model (such as a large language model (LLM)), the prefill stage and the decoding stage are two separate stages (referred to as "PD separation").

[0003] When the scale of input information is large, the AI ​​model provided by the relevant technology spends a lot of time reading and writing input information in the Prefill stage. After waiting for all the input information to be read and written, the accelerator card in the Decode stage starts calculating. It may even take multiple reading and writing steps to complete the input information processing task, resulting in device idling and a large waste of computing resources. Summary of the Invention

[0004] The present application provides a computing system, a computing power allocation method, an electronic device and a readable storage medium to at least solve the problem in the related art that multiple readings and writings are required to complete a processing task of input information, resulting in device idling and a large waste of computing resources.

[0005] The present application provides a computing system, comprising:

[0006] Multiple computing devices are used to provide computing power for the target AI model;

[0007] A resource scheduler, which is used to pool multiple computing devices to obtain a first computing resource pool, a second computing resource pool, and a third computing resource pool; the multiple computing devices in the first computing resource pool are used to provide computing power for the pre-filling phase of the target AI model; and the multiple computing devices in the third computing resource pool are used to provide computing power for the decoding phase of the target AI model;

[0008] The resource scheduler is also used to adjust the computing devices allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool according to the computing power indicators of the first computing resource pool and the third computing resource pool during the process of the target AI model processing input information, so as to adjust the computing power of the first computing resource pool and the third computing resource pool.

[0009] This application also provides a computing power allocation method, which is applied to a resource scheduler of a computing system, comprising:

[0010] Pooling multiple computing devices of the computing system to obtain a first computing resource pool, a second computing resource pool, and a third computing resource pool; the multiple computing devices are used to provide computing power for the target AI model; the multiple computing devices in the first computing resource pool are used to provide computing power for the pre-population phase of the target AI model; and the multiple computing devices in the third computing resource pool are used to provide computing power for the decoding phase of the target AI model;

[0011] During the process of the target AI model processing input information, the computing devices allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool are adjusted according to the computing power indicators of the first computing resource pool and the third computing resource pool to adjust the computing power of the first computing resource pool and the third computing resource pool.

[0012] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned computing power allocation methods when executing the computer program.

[0013] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned computing power allocation methods are implemented.

[0014] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned computing power allocation methods when executed by a processor.

[0015] Through the present application, since the resource scheduler pools the computing devices of the target AI model, a first computing resource pool is obtained to provide computing power for the pre-filling stage of the target AI model, a third computing resource pool is used to provide computing power for the decoding stage of the target AI model, and a second computing resource pool is used to allocate its own computing devices to the first computing resource pool and the third computing resource pool. According to the computing power indicators of the first computing resource pool and / or the third computing resource pool, the computing devices allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool are adjusted to dynamically adjust the computing power of the first computing resource pool and the third computing resource pool. The first computing resource pool and the third computing resource pool can be calculated in parallel. Therefore, it can solve the technical problem that the related technology requires multiple readings and writings to complete the processing task of an input information, resulting in device idling and a large waste of computing resources, thereby achieving the technical effect of reducing the frequency of device idling and reducing the waste of computing resources.

[0016] In addition, the computing power of the first computing resource pool is higher than that of the second computing resource pool; the computing power of the second computing resource pool is higher than that of the third computing resource pool, which is also set based on the computing power requirements in the pre-filling stage and the decoding stage. On this basis, in the process of processing input information, according to the computing power indicators of the first computing resource pool and / or the third computing resource pool, the computing devices allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool are adjusted, so as to achieve the technical effect of adaptively dynamic computing resources for the first computing resource pool and the third computing resource pool according to actual computing power requirements at a deeper level, reducing the waste of computing resources while improving the efficiency of the target AI model in processing input information. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A schematic diagram of various computing resource pools obtained by pooling the computing resources of the target AI model provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of allocating computing devices of a scaling computing resource pool to a pre-filled computing resource pool and a decoding computing resource pool according to an embodiment of the present application;

[0020] Figure 3 This is one of the structural diagrams of a computing system provided in an embodiment of the present application;

[0021] Figure 4 This is a second structural diagram of a computing system provided in an embodiment of the present application;

[0022] Figure 5 A flowchart of a resource allocation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0025] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0026] In conjunction with the specific application environment architecture or specific hardware architecture that the computing system relies on, the specific application environment architecture or the specific hardware architecture is described here.

[0027] An embodiment of the present application provides a computing system, which includes:

[0028] Multiple computing devices are used to provide computing power for the target AI model;

[0029] A resource scheduler, which is used to pool multiple computing devices to obtain a first computing resource pool, a second computing resource pool, and a third computing resource pool; the multiple computing devices in the first computing resource pool are used to provide computing power for the pre-filling phase of the target AI model; and the multiple computing devices in the third computing resource pool are used to provide computing power for the decoding phase of the target AI model;

[0030] The resource scheduler is also used to adjust the computing devices allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool according to the computing power indicators of the first computing resource pool and the third computing resource pool during the process of the target AI model processing input information, so as to adjust the computing power of the first computing resource pool and the third computing resource pool.

[0031] The computing device in the embodiments of this application refers to a hardware device used to process and compute AI tasks. The computing device is responsible for performing computational operations in model training, inference (including pre-population, encoding, decoding, and other stages), and other related tasks. Generally speaking, the computing device is used to provide computing power for the target AI model.

[0032] The type of computing device may be one or more of a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), and a field-programmable gate array (FPGA).

[0033] Computational power, also known as computing capacity, is usually used to measure the processing power of a computing device or computing resource pool, and involves how many computing tasks or operations the system can perform per unit time.

[0034] The computing system in the embodiment of the present application also includes a resource scheduler, which refers to a hardware device capable of allocating or scheduling computing resources. In actual applications, the resource scheduler can be a CXL (Compute ExpressLink) resource scheduler, which can be deployed on a server.

[0035] The target AI model in the embodiment of the present application can be an AI model with serialized serial output. The target AI model can perform at least one of the generation tasks such as image generation, speech generation, text generation, content analysis, and text translation based on the user's input information, and there is no restriction on this.

[0036] For example, for a target AI model that generates text, the target AI model may be a Large Language Model (LLM).

[0037] The target AI model of the embodiment of the present application includes a prefill stage and a decoding stage.

[0038] The pre-population phase is the initialization phase of the model after receiving input information during a generation task. During this phase, the model generates input feature data and initial KV cache data based on the given input information (such as text, questions, or prompts).

[0039] The feature data includes a feature vector for each text unit in the input information.

[0040] The parallelism of the pre-filling stage can fully utilize the computing power of the GPU and is computationally intensive.

[0041] The initial KV cache data includes the KV vector of each text unit. Related technologies obtain the query Q (Query) vector and KV vector of the text unit by linearly projecting the text representation vector of the text unit. The KV vector consists of a K (Key) vector and a V (Value) vector.

[0042] The decoding phase begins the process of gradually generating the actual output content based on the feature data obtained in the pre-population phase and the initial KV cache data. For text generation tasks, the decoding process typically generates the next word or sentence based on the previous word or sentence, until a complete answer, paragraph, or text is generated.

[0043] During the decoding phase, memory access only calculates one token (a text unit, such as a character, word, or sentence). The computing power required is not that high, and it is mainly limited by memory bandwidth, making it memory-intensive.

[0044] In order to achieve dynamic allocation of computing resources without affecting the calculations in the pre-filling stage and the decoding stage, the resource scheduler in the embodiment of the present application pools the computing resources of the target AI model to obtain a first computing resource pool, a second computing resource pool, and a third computing resource pool. The three computing resource pools are all resource collections composed of multiple computing devices.

[0045] Among them, the first computing resource pool is also called the "pre-filling computing resource pool". The first computing resource pool provides computing power for the pre-filling stage of the target AI model and is responsible for the conversion calculation of the model's input information. Since the model is multi-user, it is easy for multiple users to initiate tasks. Therefore, there is a lot of input information. Therefore, the first computing resource pool usually uses acceleration card devices with strong computing power (computing ability) (such as chips with more computing units), and the number can be controlled at a smaller scale.

[0046] The second computing resource pool is also called the "scalable computing resource pool". The load is dynamically allocated. The computing devices of the second computing resource pool can be allocated to the first computing resource pool or the third computing resource pool. That is, when the computing resources or computing power of the first computing resource pool or the third computing resource pool are insufficient, some computing devices are divided to temporarily supplement the computing resources or computing power. The second computing resource pool can usually select computing devices with moderate computing power.

[0047] The third computing resource pool, also known as the "decoding computing resource pool," is responsible for decoding the model output. This stage often involves multiple batches of parallel decoding, so the amount of cached data read will increase exponentially. Therefore, the third computing resource pool can use accelerator card devices with weaker computing power but greater access bandwidth (such as devices with HBM high-bandwidth memory).

[0048] In general, based on the computing power requirements in the pre-filling stage and the decoding stage, the computing power of the first computing resource pool obtained after pooling is higher than that of the second computing resource pool; and the computing power of the second computing resource pool is higher than that of the third computing resource pool.

[0049] The computing power of a computing device refers to the ability of a single computing device to perform computing tasks per unit time. It can be expressed in floating-point operations per second (FLOPS). FLOPS measures how many floating-point operations a device can perform per second. Of course, other indicators can also be used for measurement, such as processor frequency (Clock Speed), and there is no restriction on this.

[0050] The computing power of a computing resource pool refers to the total computing power of all computing devices contained in the computing resource pool, which can be measured by the total FLOPS of all computing devices.

[0051] The above embodiments have illustrated that the target AI model can process user input information, which can be voice input, text input, image input, etc., without limitation.

[0052] In the embodiment of the present application, the resource scheduler monitors indicators of each computing resource pool while it is processing data. These indicators include, but are not limited to, computing power indicators, temperature data, and memory indicators. Specifically, the computing power indicator may be computing power utilization, which is the ratio of used computing power to total computing power. Specifically, the memory indicator may be memory utilization, which is the ratio of used memory to total memory.

[0053] It is worth noting that the above embodiments have explained that the computing resources of the second computing resource pool are either allocated to the first computing resource pool or to the third computing resource pool. Therefore, monitoring indicators such as computing power utilization of the first computing resource pool or the third computing resource pool take into account the computing devices allocated from the second computing resource pool to the first computing resource pool or the third computing resource pool.

[0054] It is understandable that since the first computing resource pool and the third computing resource pool are usually distributed on different devices, their computing power requirements and computing power utilization are also different, and the computing power requirements and computing power utilization are also constantly changing. For example, in the early stage of the pre-filling stage, there is more input information. In this case, there is a higher computing power requirement and computing power utilization, and the decoding stage has not yet begun. In this case, all computing devices in the second computing resource pool can be allocated to the first computing resource pool.

[0055] Next, when the first computing resource pool processes the input information in the pre-filling stage, it continuously generates feature data and KV cache data of the input data. As the amount of data decreases, the computing power demand decreases and the computing power utilization rate also continues to decrease. After the feature data and KV cache data of the first input information are generated, the decoding stage begins, and the third computing resource pool needs to process the feature data and KV cache data. In this case, the computing power demand and computing power utilization rate of the third computing resource pool are gradually increasing, and the second computing resource pool may be required to allocate computing power equipment to it to provide the required computing power.

[0056] Based on the above, it can be found that the target scheduler of the embodiment of the present application dynamically adjusts the computing power allocated from the second computing resource pool to the first computing resource pool or the third computing resource pool according to the computing power indicators of the first computing resource pool and / or the third computing resource pool, thereby meeting the computing power required for the pre-filling stage and the decoding stage of the standard AI model.

[0057] See also Figure 1 , an embodiment of the present application provides a schematic diagram of each computing resource pool obtained after pooling the computing resources of the target AI model. Each computing resource pool includes a pre-filled computing resource pool, a scaling computing resource pool, and a decoding computing resource pool. The computing power of the pre-filled computing resource pool is higher than the computing power of the scaling computing resource pool, the computing power of the scaling computing resource pool, and the computing power of the decoding computing resource pool.

[0058] Each computing resource pool includes several computing devices. For example, the pre-filled computing resource pool includes 12 pre-filled computing devices, the telescopic computing resource pool includes 18 telescopic computing devices, and the decoding computing resource pool includes 30 decoding computing devices. The computing power of the pre-filled computing device is higher than that of the telescopic computing device, and the computing power of the telescopic computing device is higher than that of the decoding computing resource pool.

[0059] See also Figure 2 , the embodiment of the present application provides a schematic diagram of allocating computing devices of the scaling computing resource pool to the pre-filled computing resource pool and the decoding computing resource pool, continuing Figure 1 In an embodiment, the pre-filled computing resource pool includes 12 pre-filled computing devices of its own and 7 telescopic computing devices allocated from the telescopic computing resource pool, and the decoding computing resource pool includes 30 decoding computing devices of its own and 11 telescopic computing devices allocated from the telescopic computing resource pool.

[0060] Through the computing system provided by the embodiments of the present application, since the resource scheduler in the computing system pools the computing resources of the target AI model, a first computing resource pool is obtained for providing computing power for the pre-filling stage of the target AI model, a third computing resource pool is used to provide computing power for the decoding stage of the target AI model, and a second computing resource pool is used to allocate its own resources to the first computing resource pool and the third computing resource pool. According to the computing power indicators of the first computing resource pool and / or the third computing resource pool, the computing resources allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool are adjusted to achieve dynamic adjustment of the computing power of the first computing resource pool and the third computing resource pool. The first computing resource pool and the third computing resource pool can be calculated in parallel (for example, at the same time, the first computing resource pool processes the 5th to 6th input information, and the third computing resource pool processes the feature data and KV cache data of the 3rd to 4th input information). Therefore, it can solve the technical problem of multiple reads and writes required to complete the processing task of one input information in the related technology, resulting in device idling and large waste of computing resources, thereby achieving the technical effect of reducing the frequency of device idling and reducing the waste of computing resources.

[0061] In some embodiments, the computing power of the first computing resource pool is higher than the computing power of the second computing resource pool; the computing power of the second computing resource pool is higher than the computing power of the third computing resource pool;

[0062] The computing power of the computing devices in the first computing resource pool is higher than that of the computing devices in the second computing resource pool; the computing power of the computing devices in the second computing resource pool is higher than that of the computing devices in the third computing resource pool.

[0063] Since the computing power of the first computing resource pool in the pre-filling stage is greater than that of the third computing resource pool in the decoding stage, and the second computing resource pool is responsible for dynamic allocation, that is, when the computing resources of the first computing resource pool or the third computing resource pool are insufficient, part of the computing resources are allocated for temporary support. Therefore, the computing power of the first computing resource pool obtained after pooling is higher than that of the second computing resource pool; the computing power of the second computing resource pool is higher than that of the third computing resource pool. These meet the computing power requirements of the pre-filling stage and the decoding stage.

[0064] On this basis, in the process of processing input information, the computing resources allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool are adjusted according to the computing power indicators of the first computing resource pool and / or the third computing resource pool, so as to achieve a deeper technical effect of adaptively and dynamically providing computing resources for the first computing resource pool and the third computing resource pool according to actual computing power requirements, reduce the waste of computing resources, and improve the efficiency of the target AI model in processing input information.

[0065] In some embodiments, the resource scheduler includes a scheduling manager; the scheduling manager is further configured to allocate each computing device in the second computing resource pool to the first computing resource pool during an initialization phase of the pre-population phase.

[0066] Considering the promotion of long text and multimodal applications, the pre-population stage will receive a large amount of input information in the early stages of inference. However, the decoding stage has not yet obtained processable feature data and KV cache data, and does not require additional computing resources. Therefore, during the initialization phase, all computing devices in the second computing resource pool (scaling computing resource pool) are initialized as pre-population computing resources. This provides sufficient computing resources for the pre-population stage and avoids wasting computing resources in the scaling computing resource pool.

[0067] In some embodiments, the computing power indicator is computing power utilization; the resource scheduler includes a scheduling manager and a topology manager;

[0068] The scheduling manager is configured to determine, when the computing power utilization of the first computing resource pool is less than the first utilization threshold but the computing power utilization of the third computing resource pool is greater than the second utilization threshold, or when the computing power utilization of the third computing resource pool is less than the second utilization threshold, a change amount of computing devices allocated from the second computing resource pool to the first computing resource pool based on the actual total computing power, the original theoretical total computing power, and the computing power loss of the first computing resource pool;

[0069] The topology manager is further configured to adjust computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool according to the change amount.

[0070] After all computing devices in the second computing resource pool are initialized and allocated to the first computing resource pool, the resource scheduler starts dynamic scheduling, that is, adjusting the computing resources allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool according to the computing power indicators of the first computing resource pool and / or the third computing resource pool. The dynamic scheduling process can be executed by the scheduling manager in the resource scheduler.

[0071] As described in the previous embodiment, the computing power indicator can be the computing power utilization rate. The computing power utilization rate is the ratio of the used computing power to the total computing power. The higher the computing power utilization rate, the less computing power resources are wasted; conversely, the lower the computing power utilization rate, the more computing power resources are wasted.

[0072] The computing power requirements and computing power utilization rates of the first and third computing resource pools are constantly changing. Although the computing power requirement of the first computing resource pool is generally higher than that of the third computing resource pool, when the computing power utilization rate of the first computing resource pool is low (for example, the computing power utilization rate of the first computing resource pool is less than a first utilization threshold, which is a preset threshold, such as 30%), there is a large waste of computing resources in the first computing resource pool. The third computing resource pool may require more computing resources due to the increase in acquired feature data and KV cache data. In other words, the computing power requirement of the third computing resource pool is relatively high (for example, the computing power utilization rate of the third computing resource pool is greater than a second utilization threshold, which is also a preset threshold, such as 35%). In this case, a change in the computing resources allocated by the second computing resource pool to the first and third computing resource pools is triggered, namely, a reduction in the number of computing resources allocated to the first computing resource pool and an increase in the number of computing resources allocated to the third computing resource pool.

[0073] In addition, as the third computing resource pool processes feature data and KV cache data, its computing power demand is also continuously decreasing, and the computing power utilization is also continuously decreasing (for example, the computing power utilization of the third computing resource pool is less than the second utilization threshold). In this case, in order to ensure that the first computing resource pool has sufficient computing power to process subsequent input information, the computing resources allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool will also be changed, that is: the number of computing resources allocated to the third computing resource pool is reduced, and the number of computing resources allocated to the first computing resource pool is increased.

[0074] Specifically, the scheduling manager may determine the change amount of computing devices allocated from the second computing resource pool to the first computing resource pool based on the actual total computing power, the original theoretical total computing power, and the computing power loss of the first computing resource pool.

[0075] The actual total computing power of the first computing resource pool includes the sum of the actual computing power of each computing device originally in the first computing resource pool and the actual computing power of each computing device allocated from the second computing resource pool to the first computing resource pool.

[0076] The original theoretical total computing power of the first computing resource pool is the sum of the peak computing powers of the original computing devices in the first computing resource pool.

[0077] Computing power loss refers to the loss of computing power due to communication or other interference. It can be determined based on computing power utilization and is generally set to 50% of the peak computing power of the computing devices in the scalable computing resource pool.

[0078] The computing power difference between the actual total computing power and the theoretical total computing power of the first computing resource pool can be determined; the computing power difference is adjusted according to the computing power loss to obtain the computing power adjustment amount of the first computing resource pool; and the change amount of the computing equipment allocated from the second computing resource pool to the first computing resource pool is determined based on the ratio of the computing power adjustment amount of the first computing resource pool to the peak computing power of the second computing resource pool.

[0079] The change amount may be an increase or a decrease. The topology manager may adjust the computing resources allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool according to the change amount of the computing devices allocated to the first computing resource pool.

[0080] Specifically, when the computing power utilization of the first computing resource pool is less than the first utilization threshold but the computing power utilization of the third computing resource pool is greater than the second utilization threshold, the change amount is a reduction. Therefore, it is necessary to reduce the computing devices allocated to the first computing resource pool and increase the computing devices allocated to the third computing resource pool, thereby reducing the idling of computing devices and the waste of computing resources.

[0081] When the computing power utilization of the third computing resource pool is less than the second utilization threshold, the change amount is an increase. Therefore, it is necessary to increase the computing devices allocated to the first computing resource pool and reduce the computing devices allocated to the third computing resource pool, thereby reducing the idleness of computing devices and the waste of computing resources.

[0082] In some embodiments, the scheduling manager is further configured to determine a computing power difference between an actual total computing power of the first computing resource pool and an original theoretical total computing power of the first computing resource pool; and adjust the computing power difference according to the computing power loss to obtain a computing power adjustment amount for the first computing resource pool.

[0083] The scheduling manager is also used to determine the change amount of computing devices allocated from the second computing resource pool to the first computing resource pool based on the ratio of the computing power adjustment amount of the first computing resource pool to the peak computing power of the second computing resource pool.

[0084] Assume that any one of the computing devices in the first computing resource pool and the computing devices allocated from the second computing resource pool to the first computing resource pool uses i p Characterize that the actual computing power of any computing device is f p , then the actual total computing power of the first computing resource pool can be represented as .

[0085] The total number of original computing devices in the first computing resource pool of the present application embodiment is N p , assuming that each computing device in the first computing resource pool has the same peak computing power, the peak computing power is F p, then the original theoretical total computing power of the first computing resource pool can be represented as .

[0086] Then, the computing power difference can be represented as .

[0087] Then, the scheduling manager can adjust the computing power difference according to the computing power loss to obtain the computing power adjustment amount of the first computing resource pool. Specifically, the sum (algebraic sum, weighted sum, mean square sum, etc.) between the computing power loss and the computing power difference can be used as the computing power adjustment amount of the first computing resource pool.

[0088] The scheduling manager can then determine the ratio of the computing power adjustment amount of the first computing resource pool to the peak computing power of the second computing resource pool, and round the ratio up or down to obtain the change in the computing device allocated from the second computing resource pool to the first computing resource pool.

[0089] See the following formula (1):

[0090] (1),

[0091] Wherein, n represents the change in the amount of computing devices allocated from the second computing resource pool to the first computing resource pool, " represents rounding up, C represents computing power loss, F S Represents the peak computing power of the second computing resource pool, i p represents any one of the computing devices in the first computing resource pool and the computing devices allocated from the second computing resource pool to the first computing resource pool, f p The actual computing power of any one of the computing devices in the first computing resource pool and the computing devices allocated from the second computing resource pool to the first computing resource pool is f p , N p Represents the total number of original computing devices in the first computing resource pool, F p Characterizes the peak computing power of the original computing device.

[0092] In the embodiment of the present application, the computing power adjustment amount of the first computing resource pool is obtained based on the actual total computing power of the first computing resource pool and the original theoretical total computing power and computing power loss of the first computing resource pool. Then, based on the ratio of the computing power adjustment amount of the first computing resource pool to the peak computing power of the second computing resource pool, the change amount of the computing equipment allocated from the second computing resource pool to the first computing resource pool is determined. This can realize dynamic adjustment of the computing equipment allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool according to the computing power requirements of the first computing resource pool and the third computing resource pool, thereby reducing the waste of computing resources and improving computing efficiency.

[0093] In some embodiments, the resource scheduler includes a topology manager; the topology manager adjusts the computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool by creating or deleting a first high-speed connection between the first computing resource and the computing devices allocated from the second computing resource pool to the first computing resource, and creating or deleting a second high-speed connection between the third computing resource and the computing devices allocated from the second computing resource pool to the third computing resource.

[0094] The topology manager (specifically, the CXL topology manager) in the embodiment of the present application does not itself process large amounts of feature data and cached data. The topology manager serves as a hardware controller for managing cluster topology, and is used to analyze and process the topological relationships after dynamic scheduling of each resource pool. After the resource pool is dynamically scheduled, the high-speed connection relationship between each node is updated in real time.

[0095] The high-speed connection may specifically be a CXL connection, which is based on the CXL protocol. CXL is a high-speed serial protocol that allows for fast and reliable data transmission between different components within a computer system.

[0096] The high-speed connection between the first computing resource and the computing device allocated to the first computing resource from the second computing resource pool is the first high-speed connection, and the high-speed connection between the third computing resource and the computing device allocated to the third computing resource from the second computing resource pool is the second high-speed connection. Efficient allocation of computing resources can be achieved by creating and deleting the first high-speed connection or the second high-speed connection.

[0097] In some embodiments, the topology manager is specifically used to determine, from the second computing resource pool, each first candidate computing device that has a first high-speed connection with the first computing resource pool when the computing power utilization of the first computing resource pool is less than the first utilization threshold but the computing power utilization of the third computing resource pool is greater than the second utilization threshold; select the first target computing device of the change amount from the first candidate computing devices; disconnect the first high-speed connection between the first computing resource pool and the first target computing device of the change amount; and create a second high-speed connection between the first target computing device of the change amount and the third computing resource pool.

[0098] The above embodiment has explained that when the computing power utilization of the first computing resource pool is less than the first utilization threshold but the computing power utilization of the third computing resource pool is greater than the second utilization threshold, the change amount of the computing devices allocated to the first computing resource pool is a decrease, that is, the computing devices of the change amount are recovered from the computing devices allocated from the second computing resource pool to the first computing resource pool, and the recovered computing devices are allocated to the third computing resource pool. When the computing power utilization of the third computing resource pool is less than the second utilization threshold, the computing devices of the change amount are recovered from the computing devices allocated from the second computing resource pool to the third computing resource pool, and the recovered computing devices are allocated to the first computing resource pool. This can achieve dynamic allocation of computing resources in the second computing resource pool and reduce the consumption of computing resources.

[0099] In an embodiment of the present application, a computing device of a corresponding change amount is recovered from the computing devices allocated from the second computing resource pool to the first computing resource pool, that is, the topology manager determines from the second computing resource pool each first candidate computing device that has a first high-speed connection with the first computing resource pool; selects a first target computing device of a corresponding change amount from the first candidate computing devices; and disconnects the first high-speed connection between the first computing resource pool and the first target computing device of the corresponding change amount.

[0100] On the contrary, the reclaimed computing devices are allocated to the third computing resource pool, that is, the topology manager creates a second high-speed connection between the changed computing devices and the third computing resource pool.

[0101] In some embodiments, the topology manager is specifically used to determine, from the second computing resource pool, each second candidate computing device that has a second high-speed connection with the third computing resource pool when the computing power utilization of the third computing resource pool is less than the second utilization threshold; select the second target computing device of the corresponding change amount from the second candidate computing devices; disconnect the second high-speed connection between the third computing resource pool and the second target computing device of the change amount; and create a first high-speed connection between the second target computing device of the change amount and the first computing resource pool.

[0102] Similarly, the topology manager recovers the computing devices of the changed amount from the computing devices allocated from the second computing resource pool to the third computing resource pool, that is, determines each second candidate computing device that has a second high-speed connection with the third computing resource pool from the second computing resource pool; selects the second target computing device of the changed amount from the second candidate computing devices; and disconnects the second high-speed connection between the third computing resource pool and the second target computing device of the changed amount.

[0103] In addition, the topology manager needs to allocate the recovered computing devices to the first computing resource pool, that is, to create a second high-speed connection between the computing devices of the corresponding change amount and the first computing resource pool.

[0104] The embodiments of the present application can dynamically increase or decrease the computing resources of the computing resource pool by disconnecting or creating high-speed connections between computing devices in the computing resource pool, realize adaptive adjustment of computing resources, and have high flexibility.

[0105] In some embodiments, the computing power indicator is computing power utilization; the scheduling manager is also used to maintain the computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool when the computing power utilization of the first computing resource pool is greater than the first utilization threshold, or the computing power utilization of the third computing resource pool is greater than the second utilization threshold.

[0106] It is worth noting that when the computing power utilization of the first computing resource pool is greater than the first utilization threshold, it means that the computing amount of the first computing resource pool is large. In this case, the computing resources allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool are maintained.

[0107] In addition, when the computing power utilization of the third computing resource pool is greater than the second utilization threshold, it means that the computing amount of the third computing resource pool is large. In this case, the computing resources allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool are maintained, thereby reducing the waste of computing resources and ensuring the stability of the system.

[0108] In some embodiments, the computing system further includes a memory extension resource pool; the memory extension resource pool is obtained by pooling and expanding the memory resources of the target AI model based on the resource manager;

[0109] The computing devices in the first computing resource pool, the second computing resource pool, and the third computing resource pool share a memory extension resource pool.

[0110] In an embodiment of the present application, the resource scheduler expands and pools the memory resources of the target AI model based on the target protocol to obtain a memory expansion resource pool. The target protocol may be the CXL protocol. The memory expansion resource pool is responsible for subsequent unified access to cached data such as KV cache. The memory expansion resource pool may be composed of CXL type 3 devices.

[0111] The embodiments of the present application expand and pool memory resources to meet the greater data processing requirements of the target AI model, especially when processing large-scale data, avoiding performance bottlenecks caused by insufficient memory.

[0112] In addition, the present embodiment also provides a shared memory expansion resource pool for each computing resource in the first, second, and third computing resource pools. Memory resources can be flexibly allocated to different computing resource pools to meet different task requirements, enabling more efficient use of memory resources and avoiding excess or insufficient memory in a particular computing resource pool. Dynamically adjusting memory allocation helps avoid resource waste and improves overall system memory utilization.

[0113] In some embodiments, the input information includes a plurality of text units; the resource scheduler further includes a data exchange module;

[0114] The first computing resource pool is used to process the input information in the pre-filling stage of each input information, obtain and output the feature data and initial KV cache data of each input information to the data exchange module;

[0115] The data exchange module is used to store the feature data of each input information and the initial KV cache data into the memory extension resource pool;

[0116] The third computing resource pool is used to obtain and process feature data and the previous round of KV cache data from the memory extension resource pool for at least one round during the decoding stage of each input information, so as to update the KV cache data in the memory extension resource pool as the target KV cache data; the target KV cache data is used to generate the prediction results of the target AI model; the first round of KV cache data is the initial KV cache data.

[0117] Feature data includes the feature vector of each text unit; initial KV cache data includes the KV vector of each text unit;

[0118] The third computing resource pool is used to process the text vector of a text unit corresponding to each round of the decoding stage and the KV cache data of the previous stage read from the memory extension resource pool, and obtain and output the incremental KV vector of the corresponding round to the data exchange module;

[0119] The data exchange module is used to store the incremental KV vector of the corresponding round into the memory extension resource pool to update the KV cache data in the memory extension resource pool to obtain the KV cache data of the corresponding round; the KV cache data of the last round is the target KV cache data.

[0120] See also Figure 3 , an embodiment of the present application provides one of the structural diagrams of a computing system, including a first computing resource pool, a second computing resource pool, a third computing resource pool, a memory extension resource pool and a resource scheduler.

[0121] The first computing resource pool includes a plurality of pre-filled computing devices, for example, 12, and the computing capacity of the first computing resource pool is relatively large.

[0122] The second computing resource pool includes multiple scalable computing devices, for example, 18, and the computing capacity of the second computing resource pool is moderate.

[0123] The third computing resource pool includes a plurality of decoding computing devices, for example, 30. The computing capability of the third computing resource pool is relatively low, but the access bandwidth is relatively high.

[0124] The first computing resource pool, the second computing resource pool, and the third computing resource pool can exchange data with the memory extension resource pool through the resource scheduler.

[0125] continue Figure 3 , Figure 3 The steps are numbered (1) to (5) in the figure. The interactive process includes the following steps:

[0126] (1): In the pre-filling phase, the computing devices of the first computing resource pool (including the scaling computing devices allocated from the second computing resource pool to the first computing resource pool) process the input information to obtain the feature data of the input information and the initial KV cache data.

[0127] (2): The resource scheduler stores the feature data and initial KV cache data in the memory extension resource pool through the CXL.mem protocol.

[0128] (3): The resource scheduler will also determine the computing devices to be allocated from the second computing resource pool to the third computing resource pool based on the computing power indicators of the first computing resource pool and / or the computing power indicators of the third computing resource pool. The computing devices of the third computing resource pool and the pre-populated ones can obtain feature data and KV cache data through the CXL.cache protocol.

[0129] (4): The computing devices allocated from the second computing resource pool to the third computing resource pool obtain feature data and KV cache data through the CXL.cache protocol.

[0130] (5): In each round of processing in the decoding stage, the resource scheduler receives the text vector of a text unit corresponding to the sub-round and the KV cache data of the previous stage read from the memory extension resource pool from the computing device of the third computing resource pool, obtains the incremental KV vector of the corresponding round, and stores the incremental KV vector of the corresponding round to the memory extension resource pool to update the KV cache data in the memory extension resource pool to obtain the KV cache data of the corresponding round; the KV cache data of the first round is the initial KV cache data

[0131] Repeat (4) and (5) above until the decoding phase ends (the decoding calculation process outputs the end symbol).

[0132] The KV cache data obtained in the last round may be used as the target KV cache data. The target KV cache data may be output, and after the target KV cache data is output, the target KV cache data is deleted from the memory extension resource pool.

[0133] In the embodiment of the present application, when the first computing resource processes input information, it can obtain temporarily added computing resources from the second computing resource. When the third computing resource pool processes feature data and KV cache data, it can also obtain temporarily added computing resources from the second computing resource pool, so that there are sufficient computing resources in both the pre-filling stage and the decoding stage.

[0134] In addition, in the embodiment of the present application, the first computing resource pool, the second computing resource pool, and the third computing resource pool share the feature data and KV cache data in the memory extension resource pool. The resource pools communicate with each other through the CXL protocol, which has lower latency and improves communication efficiency. The memory extension resource pool can store more data and support more concurrent computing tasks.

[0135] In some embodiments, the computing device includes device memory;

[0136] During the decoding phase, if the computing devices allocated from the second computing resource pool to the third computing resource pool remain unchanged, for any round of processing in the decoding phase, the feature vectors of the text units of the corresponding round are obtained by the computing devices of the third computing resource pool from their own device memory; the feature data in the device memory is obtained and cached from the memory extension resource pool;

[0137] When the computing device allocated from the second computing resource pool to the third computing resource pool changes, for any round of processing in the decoding stage, the feature vector of the text unit of the corresponding round is read by the computing device of the third computing resource pool from the memory extension resource pool.

[0138] In the embodiment of the present application, for any input information, after the incremental KV vector of any text unit in the input information is decoded by the third computing resource pool, the feature data is still retained in the device memory of the third computing resource pool. During this period, if some computing devices in the second computing resource pool are scheduled to the first computing resource pool, the feature data is also stored in the memory extension resource pool, and the feature vector of the next text unit is subsequently obtained from the memory extension resource pool; on the contrary, when the computing device allocated from the second computing resource pool to the third computing resource pool remains unchanged, for any round of processing in the decoding stage, the feature vector of the text unit of the corresponding round is obtained by the computing device of the third computing resource pool from its own device memory, thereby reducing the number of accesses to the memory extension resource pool and improving data processing efficiency.

[0139] In some embodiments, the computing device has an accelerator card memory; the accelerator card memory is used to store at least part of the weight parameters, bias parameters, hyperparameters and system operating parameters of the target AI model.

[0140] In this embodiment of the application, the accelerator card memory of each computing device in the computing resource pool is used to store the model's weight parameters, bias parameters, hyperparameters, and system operating parameters. This design is designed to place more model parameters in the accelerator card memory, reduce the cluster size for the same model, and thus reduce the synchronization communication overhead of large-scale clusters.

[0141] In some embodiments, the resource scheduler is further configured to: obtain temperature data and memory occupancy rate of each of the first computing resource pool, the second computing resource pool, and the third computing resource pool during the process of the target AI model processing input information;

[0142] When it is determined that the temperature data of any computing resource pool is greater than a temperature threshold and / or the memory usage is greater than a memory usage threshold, the operating system where the computing resource pool is located is frequency-reduced.

[0143] In addition to monitoring computing power utilization, the resource scheduler in the embodiment of the present application also monitors temperature data and memory occupancy. The temperature data can be obtained from a temperature sensor, and the memory occupancy refers to the proportion of the computing resource pool that occupies the system memory at a certain moment.

[0144] Temperature data and memory utilization will affect the overall operational stability of the computing cluster. Therefore, these two indicators are used for independent stability control scheduling. When temperature data and memory utilization exceed the set thresholds (when the temperature data of the computing resource pool is greater than the temperature threshold and / or the memory utilization is greater than the memory utilization threshold), the entire system will be frequency-reduced. The frequency reduction operation can specifically reduce the operating frequency of the processor (CPU frequency, GPU frequency, etc.), thereby reducing heat generation and improving system stability.

[0145] In some embodiments, the data exchange module is further used to perform integrity check on feature data, initial KV cache data and incremental KV data, and after passing the check, store the feature data, initial KV cache data and incremental KV data in the memory extension resource pool.

[0146] Under the guidance of the CXL topology manager, the data exchange module (specifically, a CXL exchange module) in the embodiment of the present application processes the forwarding and integrity verification (which can be performed according to the encoding format) of feature data and cache data (initial KV cache data and incremental KV data). After the verification passes, the feature data, initial KV cache data, and incremental KV data are stored in the memory extension resource pool. In addition, the data exchange module can also transfer data in the memory extension resource pool to the designated computing resource pool, thereby ensuring the accuracy and stability of data transmission.

[0147] See also Figure 4 , an embodiment of the present application provides a second structural diagram of a computing system, which includes a resource scheduler, a first computing resource pool, a second computing resource pool, and a third computing resource pool.

[0148] The resource scheduler is a CXL scheduler, which includes a CXL topology manager, a CXL switch module, and a CXL scheduler manager. The CXL switch module can obtain monitoring metrics generated by any computing device in any computing resource pool. Each computing device may include multiple accelerator cards, whose memory is used to store at least some of the target AI model's weight parameters, bias parameters, hyperparameters, and system operating parameters. The multiple monitoring metrics include feature data, temperature data (collected by the accelerator card's temperature sensor), computing power metrics, memory utilization, and cache data (KV cache data).

[0149] These monitoring indicators are transmitted to the CXL scheduler through the CXL protocol. The CXL scheduler can store these monitoring indicators on the memory expansion card of the memory expansion resource pool.

[0150] The CXL topology manager is responsible for creating or deleting high-speed connections between resource pools.

[0151] The CXL scheduler is configured to adjust computing resources allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool.

[0152] The details of each part can be found in the above embodiments and will not be described again here.

[0153] The CXL scheduler in the embodiment of the present application is an additional component that can be flexibly deployed to reduce the difficulty of cluster deployment.

[0154] In some embodiments, the resource scheduler is further configured to:

[0155] Obtain the computing power of the corresponding computing devices for each computing resource of the target AI model;

[0156] Pooling computing devices whose computing power is within a first computing power range to obtain a first computing resource pool;

[0157] Pooling computing devices whose computing power is within a second computing power range into a second computing resource pool;

[0158] Pooling computing devices with computing power within a third computing power range into a third computing resource pool;

[0159] The minimum value of the first computing power range is greater than the maximum value of the second computing power range; the minimum value of the second computing power range is greater than the maximum value of the third computing power range.

[0160] When the resource scheduler of the embodiment of the present application pools the computing resources of the target AI model, it can pool based on the computing power of each computing device. Specifically, each computing device can be classified based on the computing power of the computing device, for example, into computing devices with computing power within a first computing power range, computing devices with computing power within a second computing power range, and computing devices with computing power within a third computing power range.

[0161] Therefore, computing devices with computing power within the first computing power range can be pooled into a first computing resource pool; computing devices with computing power within the second computing power range can be pooled into a second computing resource pool; and computing devices with computing power within the third computing power range can be pooled into a third computing resource pool.

[0162] The embodiment of the present application pools each computing device into a first computing resource pool, a second computing resource pool, and a third computing resource pool with different computing powers based on the computing power of the computing device. The first computing resource pool, the second computing resource pool, and the third computing resource pool can process different model tasks, which helps to improve computing efficiency, optimize resource utilization, enhance system elasticity and scalability, while simplifying management, reducing costs, and supporting diverse computing tasks.

[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0164] See also Figure 5 , an embodiment of the present application further provides a computing power allocation method, which is applied to a resource scheduler of a computing system, and the method includes steps 510 to 520:

[0165] Step 510: Pool multiple computing devices of the computing system to obtain a first computing resource pool, a second computing resource pool, and a third computing resource pool; the multiple computing devices are used to provide computing power for the target AI model; the multiple computing devices in the first computing resource pool are used to provide computing power for the pre-population phase of the target AI model; and the multiple computing devices in the third computing resource pool are used to provide computing power for the decoding phase of the target AI model.

[0166] Step 520: During the process of the target AI model processing input information, the computing devices allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool are adjusted according to the computing power indicators of the first computing resource pool and the third computing resource pool to adjust the computing power of the first computing resource pool and the third computing resource pool.

[0167] In some embodiments, the computing power of the first computing resource pool is higher than the computing power of the second computing resource pool; the computing power of the second computing resource pool is higher than the computing power of the third computing resource pool;

[0168] The computing power of the computing devices in the first computing resource pool is higher than that of the computing devices in the second computing resource pool; the computing power of the computing devices in the second computing resource pool is higher than that of the computing devices in the third computing resource pool.

[0169] In some embodiments, the computing power indicator is computing power utilization; the resource scheduler includes a scheduling manager and a topology manager; and adjusting computing devices allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool according to the computing power indicators of the first computing resource pool and the third computing resource pool includes:

[0170] determining, by the scheduling manager, a change amount of computing devices allocated from the second computing resource pool to the first computing resource pool based on the actual total computing power, the original theoretical total computing power, and the computing power loss of the first computing resource pool when the computing power utilization of the first computing resource pool is less than the first utilization threshold but the computing power utilization of the third computing resource pool is greater than the second utilization threshold, or when the computing power utilization of the third computing resource pool is less than the second utilization threshold;

[0171] The topology manager adjusts the computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool according to the change amount.

[0172] In some embodiments, determining a change in the amount of computing devices allocated from the second computing resource pool to the first computing resource pool based on the actual total computing power, the original theoretical total computing power, and the computing power loss of the first computing resource pool includes:

[0173] Determining, by the scheduling manager, a computing power difference between an actual total computing power of the first computing resource pool and an original theoretical total computing power of the first computing resource pool;

[0174] Adjust the computing power difference according to the computing power loss to obtain the computing power adjustment amount of the first computing resource pool;

[0175] The scheduling manager determines the change amount of computing devices allocated from the second computing resource pool to the first computing resource pool based on the ratio of the computing power adjustment amount of the first computing resource pool to the peak computing power of the second computing resource pool.

[0176] In some embodiments, the resource scheduler includes a topology manager; the method further includes: adjusting the computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool by creating or deleting a first high-speed connection between the first computing resource and the computing device allocated from the second computing resource pool to the first computing resource, and creating or deleting a second high-speed connection between the third computing resource and the computing device allocated from the second computing resource pool to the third computing resource according to the topology manager.

[0177] In some embodiments, when the computing power utilization of the first computing resource pool is less than a first utilization threshold but the computing power utilization of the third computing resource pool is greater than a second utilization threshold, adjusting computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool includes:

[0178] Determine, by the topology manager, from the second computing resource pool, respective first candidate computing devices that have a first high-speed connection to the first computing resource pool;

[0179] Selecting a first target computing device for the change amount from the first candidate computing devices;

[0180] disconnecting a first high-speed connection between the first computing resource pool and the first target computing device of the changed amount;

[0181] A second high-speed connection is created between the first target computing device of the varying amount and the third computing resource pool.

[0182] In some embodiments, when the computing power utilization of the third computing resource pool is less than the second utilization threshold, adjusting computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool includes:

[0183] Determine, by the topology manager, from the second computing resource pool, respective second candidate computing devices that have a second high-speed connection to the third computing resource pool;

[0184] Selecting a second target computing device for the change amount from the second candidate computing devices;

[0185] disconnecting a second high-speed connection between the third computing resource pool and the second target computing device of the changed amount;

[0186] A first high-speed connection is created between the second target computing device of the variable amount and the first computing resource pool.

[0187] In some embodiments, the computing power indicator is computing power utilization; and during the process of the target AI model processing input information, the method further includes:

[0188] When the computing power utilization of the first computing resource pool is greater than the first utilization threshold, or the computing power utilization of the third computing resource pool is greater than the second utilization threshold, the scheduling manager maintains the computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool.

[0189] In some embodiments, before adjusting the computing devices allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool based on the computing power indicators of the first computing resource pool and the third computing resource pool, the method further includes:

[0190] During the initialization phase of the pre-population phase, each computing device in the second computing resource pool is allocated to the first computing resource pool through the scheduling manager.

[0191] In some embodiments, the computing system further includes a memory extension resource pool; the memory extension resource pool is obtained by pooling and expanding the memory resources of the target AI model based on the resource manager;

[0192] The computing devices in the first computing resource pool, the second computing resource pool, and the third computing resource pool share a memory extension resource pool.

[0193] In some embodiments, the input information includes a plurality of text units; the resource scheduler further includes a data exchange module;

[0194] The first computing resource pool is used to process the input information in the pre-filling stage of each input information, obtain and output to the data exchange module the feature data and initial KV cache data of each input information; the method further includes:

[0195] The feature data of each input information and the initial KV cache data are stored in the memory extension resource pool through the data exchange module;

[0196] The third computing resource pool is used to obtain and process feature data and the previous round of KV cache data from the memory extension resource pool for at least one round during the decoding stage of each input information, so as to update the KV cache data in the memory extension resource pool as the target KV cache data; the target KV cache data is used to generate the prediction results of the target AI model; the first round of KV cache data is the initial KV cache data.

[0197] In some embodiments, the feature data includes a feature vector for each text unit; the initial KV cache data includes a KV vector for each text unit;

[0198] The third computing resource pool is used to process the text vector of a text unit corresponding to each round of the decoding stage and the KV cache data of the previous stage read from the memory extension resource pool, and obtain and output the incremental KV vector of the corresponding round to the data exchange module;

[0199] The method also includes

[0200] The incremental KV vector of the corresponding round is stored in the memory extension resource pool through the data exchange module to update the KV cache data in the memory extension resource pool to obtain the KV cache data of the corresponding round; the KV cache data of the last round is the target KV cache data.

[0201] In some embodiments, the computing device includes device memory;

[0202] During the decoding phase, if the computing devices allocated from the second computing resource pool to the third computing resource pool remain unchanged, for any round of processing in the decoding phase, the feature vectors of the text units of the corresponding round are obtained by the computing devices of the third computing resource pool from their own device memory; the feature data in the device memory is obtained and cached from the memory extension resource pool;

[0203] When the computing device allocated from the second computing resource pool to the third computing resource pool changes, for any round of processing in the decoding stage, the feature vector of the text unit of the corresponding round is read by the computing device of the third computing resource pool from the memory extension resource pool.

[0204] In some embodiments, the method further comprises:

[0205] The data exchange module performs integrity verification on the feature data, initial KV cache data, and incremental KV data. After the verification passes, the feature data, initial KV cache data, and incremental KV data are stored in the memory extension resource pool.

[0206] In some embodiments, the computing device has an accelerator card memory; the accelerator card memory is used to store at least part of the weight parameters, bias parameters, hyperparameters and system operating parameters of the target AI model.

[0207] In some embodiments, the method further comprises:

[0208] While the target AI model is processing input information, temperature data and memory occupancy rate of each of the first computing resource pool, the second computing resource pool, and the third computing resource pool are obtained;

[0209] When it is determined that the temperature data of any computing resource pool is greater than a temperature threshold and / or the memory usage is greater than a memory usage threshold, the operating system where the computing resource pool is located is frequency-reduced.

[0210] In some embodiments, multiple computing devices of a computing system are pooled, including

[0211] Obtain the computing power of the corresponding computing devices for each computing resource of the target AI model;

[0212] Pooling computing devices whose computing power is within a first computing power range to obtain a first computing resource pool;

[0213] Pooling computing devices whose computing power is within a second computing power range into a second computing resource pool;

[0214] Pooling computing devices with computing power within a third computing power range into a third computing resource pool;

[0215] The minimum value of the first computing power range is greater than the maximum value of the second computing power range; the minimum value of the second computing power range is greater than the maximum value of the third computing power range.

[0216] For the description of the features in the embodiment corresponding to the computing power allocation method, please refer to the relevant description of the embodiment corresponding to the computing system, and will not be repeated here.

[0217] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above resource allocation method embodiments.

[0218] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned resource allocation method embodiments when running.

[0219] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0220] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned resource allocation method embodiments are implemented.

[0221] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned resource allocation method embodiments are implemented.

[0222] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0223] The above is a detailed introduction to a computing system, computing power allocation method, electronic device, computer-readable storage medium and computer program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only applicable to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A computing system, characterized in that: include: Multiple computing devices, each of which is used to provide computing power for the target AI model; a resource scheduler, the resource scheduler being configured to pool the plurality of computing devices to obtain a first computing resource pool, a second computing resource pool, and a third computing resource pool; the plurality of computing devices in the first computing resource pool being configured to provide computing power for the pre-population phase of the target AI model; The multiple computing devices in the third computing resource pool are used to provide computing power for the decoding phase of the target AI model; The resource scheduler is further configured to adjust, during the process of the target AI model processing input information, computing devices allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool based on computing power indicators of the first computing resource pool and the third computing resource pool, so as to adjust the computing power of the first computing resource pool and the third computing resource pool; the computing power indicator is computing power utilization; the resource scheduler includes a scheduling manager; The scheduling manager is configured to adjust the computing power difference between the actual total computing power of the first computing resource pool and the original theoretical total computing power of the first computing resource pool according to computing power loss to obtain a computing power adjustment amount of the first computing resource pool when the computing power utilization of the first computing resource pool is less than a first utilization threshold but the computing power utilization of the third computing resource pool is greater than a second utilization threshold, or when the computing power utilization of the third computing resource pool is less than the second utilization threshold; The scheduling manager is further used to determine a change in the amount of computing equipment allocated from the second computing resource pool to the first computing resource pool based on a ratio of the computing power adjustment amount of the first computing resource pool to the peak computing power of the second computing resource pool.

2. The computing system according to claim 1, wherein: The computing power of the first computing resource pool is higher than that of the second computing resource pool; the computing power of the second computing resource pool is higher than that of the third computing resource pool; The computing power of the computing devices in the first computing resource pool is higher than the computing power of the computing devices in the second computing resource pool; The computing power of the computing devices in the second computing resource pool is higher than the computing power of the computing devices in the third computing resource pool.

3. The computing system according to claim 2, wherein: The resource scheduler includes a topology manager; The topology manager is further configured to adjust computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool according to the change amount.

4. The computing system according to claim 3, wherein: The resource scheduler includes a topology manager; the topology manager adjusts the computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool by creating or deleting a first high-speed connection between the first computing resource and the computing device allocated from the second computing resource pool to the first computing resource, and creating or deleting a second high-speed connection between the third computing resource and the computing device allocated from the second computing resource pool to the third computing resource.

5. The computing system according to claim 4, wherein: The topology manager is specifically configured to determine, from the second computing resource pool, respective first candidate computing devices that have a first high-speed connection to the first computing resource pool when the computing power utilization of the first computing resource pool is less than a first utilization threshold but the computing power utilization of the third computing resource pool is greater than a second utilization threshold; Selecting a first target computing device for the change amount from the first candidate computing devices; disconnecting a first high-speed connection between the first computing resource pool and a first target computing device of the change amount; A second high-speed connection is created between the first target computing device of the variable and the third computing resource pool.

6. The computing system according to claim 4, wherein: The topology manager is specifically configured to determine, from the second computing resource pool, each second candidate computing device that has a second high-speed connection to the third computing resource pool when the computing power utilization of the third computing resource pool is less than a second utilization threshold; Selecting a second target computing device for the change amount from the second candidate computing devices; disconnecting a second high-speed connection between the third computing resource pool and a second target computing device of the change amount; A first high-speed connection is created between a second target computing device of the variable and the first computing resource pool.

7. The computing system according to claim 1, wherein: The computing power indicator is computing power utilization; the scheduling manager is also used to maintain the computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool when the computing power utilization of the first computing resource pool is greater than a first utilization threshold, or the computing power utilization of the third computing resource pool is greater than a second utilization threshold.

8. The computing system according to any one of claims 4 to 7, wherein: The scheduling manager is further configured to allocate each computing device in the second computing resource pool to the first computing resource pool during an initialization phase of the pre-filling phase.

9. The computing system according to claim 1, wherein: The computing system further includes a memory extension resource pool; the memory extension resource pool is obtained by pooling and expanding the memory resources of the target AI model based on the resource scheduler; The computing devices in the first computing resource pool, the second computing resource pool, and the third computing resource pool share the memory extension resource pool.

10. The computing system according to claim 9, wherein: The input information includes a plurality of text units; the resource scheduler further includes a data exchange module; The first computing resource pool is used to process the input information in the pre-filling stage of each input information, obtain and output to the data exchange module the feature data and initial KV cache data of each input information; The data exchange module is used to store the feature data of each input information and the initial KV cache data into the memory extension resource pool; The third computing resource pool is used to obtain and process the feature data and the KV cache data of the previous round from the memory extension resource pool for at least one round during the decoding stage of each input information, so as to update the KV cache data in the memory extension resource pool to the target KV cache data; the target KV cache data is used to generate the prediction result of the target AI model; the KV cache data of the first round is the initial KV cache data.

11. The computing system according to claim 10, wherein: The feature data includes a feature vector of each text unit; the initial KV cache data includes a KV vector of each text unit; The third computing resource pool is used to process, for each round of processing in the decoding stage, a text vector of a text unit corresponding to the corresponding round and the KV cache data of the previous stage read from the memory extension resource pool, to obtain and output an incremental KV vector of the corresponding round to the data exchange module; The data exchange module is used to store the incremental KV vector of the corresponding round into the memory extension resource pool to update the KV cache data in the memory extension resource pool to obtain the KV cache data of the corresponding round; the KV cache data of the last round is the target KV cache data.

12. The computing system according to claim 11, wherein: The computing device includes a device memory; In the decoding stage, when the computing devices allocated from the second computing resource pool to the third computing resource pool remain unchanged, for any round of processing in the decoding stage, the feature vectors of the text units of the corresponding round are obtained by the computing devices of the third computing resource pool from their own device memory; the feature data in the device memory is obtained and cached from the memory extension resource pool; When the computing device allocated from the second computing resource pool to the third computing resource pool changes, for any round of processing in the decoding stage, the feature vector of the text unit of the corresponding round is read by the computing device of the third computing resource pool from the memory extension resource pool.

13. The computing system according to claim 11, wherein: The data exchange module is further configured to perform integrity verification on the feature data, the initial KV cache data, and the incremental KV vector, and store the feature data, the initial KV cache data, and the incremental KV data in the memory extension resource pool after the verification passes.

14. The computing system according to claim 1, wherein: The computing device has an accelerator card memory; the accelerator card memory is used to store at least part of the weight parameters, bias parameters, hyperparameters and system operation parameters of the target AI model.

15. The computing system according to claim 1, wherein: The resource scheduler is further configured to: obtain temperature data and memory occupancy rate of each of the first computing resource pool, the second computing resource pool, and the third computing resource pool during the process of the target AI model processing input information; When it is determined that the temperature data of any computing resource pool is greater than a temperature threshold and / or the memory occupancy is greater than a memory occupancy threshold, the operating system where the computing resource pool is located is frequency-reduced.

16. The computing system according to claim 1, wherein: The resource scheduler is further configured to: Obtain the computing power of the computing devices corresponding to each computing resource of the target AI model; Pooling computing devices whose computing power is within a first computing power range to obtain a first computing resource pool; Pooling computing devices whose computing power is within a second computing power range into a second computing resource pool; Pooling computing devices with computing power within a third computing power range into a third computing resource pool; The minimum value of the first computing power range is greater than the maximum value of the second computing power range; the minimum value of the second computing power range is greater than the maximum value of the third computing power range.

17. A computing power allocation method, characterized in that: Resource schedulers for computing systems, including: Pooling multiple computing devices of the computing system to obtain a first computing resource pool, a second computing resource pool, and a third computing resource pool; the multiple computing devices are used to provide computing power for the target AI model; the multiple computing devices in the first computing resource pool are used to provide computing power for the pre-filling phase of the target AI model; and the multiple computing devices in the third computing resource pool are used to provide computing power for the decoding phase of the target AI model; During the process of the target AI model processing input information, the computing devices allocated from the second computing resource pool to the first computing resource pool and the third computing resource pool are adjusted according to the computing power indicators of the first computing resource pool and the third computing resource pool, so as to adjust the computing power of the first computing resource pool and the third computing resource pool; the computing power indicator is computing power utilization; Adjusting computing devices allocated from the second computing resource pool to the first computing resource pool and to the third computing resource pool according to computing power indicators of the first computing resource pool and the third computing resource pool, including: When the computing power utilization of the first computing resource pool is less than the first utilization threshold but the computing power utilization of the third computing resource pool is greater than the second utilization threshold, or when the computing power utilization of the third computing resource pool is less than the second utilization threshold, adjusting the computing power difference between the actual total computing power of the first computing resource pool and the original theoretical total computing power of the first computing resource pool according to the computing power loss to obtain a computing power adjustment amount of the first computing resource pool; The change amount of computing devices allocated from the second computing resource pool to the first computing resource pool is determined according to the ratio of the computing power adjustment amount of the first computing resource pool to the peak computing power of the second computing resource pool.

18. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the computing power allocation method as claimed in claim 17 when executing the computer program.

19. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the computing power allocation method according to claim 17 are implemented.

Images