A large model inference method and device

Abstract

Embodiments of the present application disclose a large model inference method and device, which are used to improve the inference efficiency of the large model. The method of the embodiments of the present application comprises: obtaining an inference task of a large model, the inference task comprising an inference task of a pre-padding stage and an inference task of a decoding stage. A first instance executes the inference task of the pre-padding stage, generates key-value cache data, the key-value cache data being used to execute the inference task of the decoding stage, the key-value cache data comprising first key-value cache data and second key-value cache data, the first key-value cache data being retained in the first instance, and the second key-value cache data being sent to a second instance. The first instance executes the inference task of the decoding stage based on the first key-value cache data, and the second instance executes the inference task of the decoding stage based on the second key-value cache data.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a reasoning method and apparatus for large models. Background Technology

[0002] As large-scale models become increasingly widely used, optimizing their inference processes becomes crucial. The inference process of large-scale models refers to performing prediction or generation tasks on a pre-trained model, typically including model loading, input processing, forward propagation, post-processing, and result generation. Due to the complexity of large-scale models, the inference process requires efficient computational resources and optimization techniques.

[0003] To further improve the inference efficiency of large models, current large models often employ a prefill and decode disaggregation architecture to perform inference tasks. The PD disaggregation architecture separates the prefill and decode stages of a large model. The prefill stage is computationally intensive, used to encode input data and generate key-value cache data, while the decode stage is memory-intensive, used to load the key-value cache data for token computation. By introducing PD disaggregation, large models become more flexible in handling different types of input, allowing for inference optimization and adjustments based on the characteristics of the two inference stages, thus improving the inference performance of large models.

[0004] However, in the current large-model PD separation architecture in inference configuration, the number of instances in the pre-filling stage and the decoding stage needs to be pre-configured. During inference, the number of instances in the decoding stage is limited by the pre-configured number of instances. In some long-sequence processing scenarios, it is impossible to expand the GPU memory space of the instances, thus leading to extended inference time. At the same time, in the current PD separation architecture inference scheme, after the instances in the pre-filling stage execute the inference task of the pre-filling stage to generate key-value cache data, they need to send all key-value cache data to the instances in the decoding stage for the decoding stage calculation. The communication load between instances is high, and the high communication load further increases the inference latency. Summary of the Invention

[0005] This application provides a method for inference of large models, which reduces inference latency and improves inference efficiency. This application also provides an inference apparatus, computing device, computing device cluster, computer-readable storage medium, and computer program product corresponding to the method for inference of large models.

[0006] In a first aspect, embodiments of this application provide a method for inference of a large model. This method can be executed by an inference system of the large model, or by components of the inference system of the large model, such as the processor, chip, or chip system of the inference system of the large model, or by a logic module or software capable of implementing all or part of the functions of the inference system of the large model. The method provided in the first aspect includes: the inference system of the large model acquiring an inference task of the large model, the inference task including a pre-filling stage inference task and a decoding stage inference task. A first instance in the inference system of the large model executes the pre-filling stage inference task, generating key-value cache data. The key-value cache data is used to execute the decoding stage inference task. The key-value cache data includes first key-value cache data and second key-value cache data. The first instance retains the first key-value cache data in the first instance and sends the second key-value cache data to a second instance. The first instance is the pre-filling instance, and the second instance is the decoding instance. The first instance executes the decoding stage inference task based on the first key-value cache data, and the second instance executes the decoding stage inference task based on the second key-value cache data. That is, the first instance and the second instance can execute the decoding stage inference task in parallel.

[0007] In this embodiment, after the first instance generates key-value cache data by performing the inference task in the pre-filling stage, it can retain a portion of the key-value cache data and continue to perform the inference task in the decoding stage in parallel with the second instance. That is, the video memory space of the first instance in this application can also be used to perform the inference task in the decoding stage. Compared with the prior art, which pre-configures a fixed number of pre-filling instances and decoding instances, and has the pre-filling instances and decoding instances perform the inference tasks in the pre-filling stage and decoding stage respectively, the inference method in this embodiment can utilize the bandwidth capability of the first instance to share the bandwidth pressure of the second instance, thereby improving the video memory utilization of different instances, reducing the inference latency of large models, and improving the inference efficiency of large models.

[0008] In one possible implementation, the large model's inference system configures inference tasks based on an inference configuration strategy. This strategy configures a first instance and a second instance to execute inference tasks, including configuring the first instance to execute inference tasks during the pre-filling phase and configuring the first and second instances to execute inference tasks during the decoding phase in parallel. The inference configuration strategy can also allocate the first instance's GPU memory space for executing inference tasks during the decoding phase. Furthermore, the strategy can configure the number of instances of the first instance executing the pre-filling phase and the second instance executing the decoding phase.

[0009] In this embodiment, the inference system of the large model can be flexibly configured to perform a first instance of inference tasks in the pre-filling stage and a second instance of inference tasks in the decoding stage. At the same time, the video memory space of the first instance can be configured to perform inference tasks in the decoding stage. That is, in this embodiment, the bandwidth capacity of the first instance can be used to share the bandwidth pressure of the second instance, thereby improving the video memory utilization of different instances, reducing the inference latency of the large model, and improving the inference efficiency of the large model.

[0010] In one possible implementation, the inference configuration strategy is used to configure the first key-value cache data in the video memory space of the first instance. The inference configuration strategy includes a video memory borrowing ratio, which is used to determine the portion of video memory space in the first instance that retains the first key-value cache data. After the large model's inference system performs the inference task of the pre-filling phase based on the first instance, it saves the first key-value cache data in the video memory space of the first instance based on the borrowed video memory ratio. The borrowed video memory ratio is either a fixed ratio or an adaptively adjusted ratio. That is, the video memory borrowing method includes a fixed borrowed video memory ratio and adaptively adjusted borrowed video memory. A fixed borrowed video memory ratio means that the first instance is configured to retain a fixed proportion of video memory space for performing the inference task of the decoding phase. Adaptively adjusted borrowed video memory space means that the first instance can adaptively determine the proportion of video memory space to retain.

[0011] In this embodiment, the inference system of the large model can set the borrowing of video memory space for the first instance based on the inference configuration strategy. It can be configured to borrow a fixed proportion of video memory space for the first instance to perform inference tasks in the decoding stage, or it can be configured to adaptively determine the proportion of video memory space borrowed by the first instance, thereby improving the flexibility of the inference tasks in the process of borrowing video memory space for the first instance.

[0012] In one possible implementation, the inference configuration strategy includes an instance scaling method. The inference system of a large model scales up or down the first or second instance based on this method. Instance scaling refers to increasing or decreasing the number of instances. The instance scaling method includes real-time adjustment or delayed adjustment. Real-time adjustment means that the instance is expanded or reduced immediately after the instance scaling conditions are met, while delayed adjustment means that the instance is expanded or reduced according to a delayed rule after the instance scaling conditions are met.

[0013] In this embodiment, the large model inference system can scale the first instance or the second instance based on different instance scaling methods in the inference configuration strategy, thereby flexibly adjusting the first instance performing the inference task in the pre-filling stage and the second instance performing the inference task in the decoding stage, thereby improving the bandwidth utilization efficiency and computing power utilization efficiency of different instances, and further improving the inference efficiency of the inference task.

[0014] In one possible implementation, the inference configuration strategy further includes instance scaling conditions. The inference system of the large model scales up or down the first instance or the second instance based on the instance scaling conditions. The instance scaling conditions include adjustment according to the first-word delay (TTFT) or adjustment according to the memory space threshold. Adjustment according to the first-word delay (TTFT) refers to expanding or shrinking instances based on the first-word delay threshold corresponding to the first-word delay, while adjustment according to the memory space threshold refers to expanding or shrinking instances based on the memory space threshold corresponding to the remaining memory space.

[0015] In this embodiment, the large model inference system can scale the first instance or the second instance based on the instance scaling conditions in the inference configuration strategy, thereby flexibly adjusting the first instance performing the inference task in the pre-filling stage and the second instance performing the inference task in the decoding stage, thereby improving the bandwidth utilization efficiency and computing power utilization efficiency of different instances, and further improving the inference efficiency of the inference task.

[0016] In one possible implementation, if the first-word latency of the inference task is greater than a first-word latency threshold, the first instance expands to execute the inference task; or, if the remaining video memory space of the first instance is less than a video memory space threshold, the first instance expands to execute the inference task. Similarly, if the first-word latency of the inference task is less than or equal to the first-word latency threshold, the first instance can reduce to execute the inference task; or, if the remaining video memory space of the first instance is greater than or equal to a video memory space threshold, the first instance reduces to execute the inference task.

[0017] Different instances in the embodiments of this application can be expanded or reduced according to the first word delay or remaining video memory space for performing the inference task, thereby improving the feasibility of expanding or reducing instances.

[0018] In one possible implementation, when the instance scaling method is delayed adjustment, the first character delay threshold includes a first first character delay threshold and a second first character delay threshold, and the video memory space threshold includes a first video memory space threshold and a second video memory space threshold. When the first character delay of the inference task is greater than the first first character delay threshold and less than the second first character delay threshold, the first instance delays the execution of the inference task by expanding the instance. Alternatively, when the remaining video memory space of the first instance is greater than the first video memory space threshold and less than the second video memory space threshold, the first instance delays the execution of the inference task by expanding the instance.

[0019] In this embodiment of the application, when the instance scaling method is delayed adjustment, the instance scaling is delayed when the first instance scaling condition is met, and the instance scaling is performed immediately when the second instance scaling condition is met, thereby improving the feasibility of instance scaling under the delayed adjustment instance scaling method.

[0020] In one possible implementation, during the inference configuration process of the large model's inference system for inference tasks based on an inference configuration strategy, the memory space management method for the second instance is configured based on the inference configuration strategy. This memory space management method instructs the second instance on an instance selection strategy for instance expansion. The memory space management method includes a polling strategy and length prediction assistance. Specifically, the polling strategy refers to the inference system polling multiple second instances to query their remaining memory space. The length prediction assistance refers to the second instance predicting the memory space requirements for executing the inference task based on a length prediction model and managing the second instance's memory space accordingly.

[0021] In this embodiment, the inference system of the large model configures the memory space management method of the second instance based on the inference configuration strategy. The inference system of the large model can manage the memory space of the second instance based on the polling strategy and length prediction assistance, thereby improving the utilization efficiency of the memory space of the second instance and further improving the inference efficiency of the large model.

[0022] In one possible implementation, when the memory space management method of the second instance is length prediction-assisted, the second instance determines the second instance to perform the inference task based on the prediction result of the length prediction model. The length prediction model is used to predict the memory space required for the inference task in the decoding stage.

[0023] In this embodiment, the second instance determines the second instance to perform the inference task based on the prediction results of the length prediction model, thereby improving the accuracy of selecting the second instance to perform the inference task and further improving the execution efficiency of the inference task.

[0024] In one possible implementation, during the parallel execution of the decoding phase reasoning task by the large model's reasoning system based on a first instance and a second instance, the large model's reasoning system transmits query terms between the first instance and the second instance, and the query terms are used to execute the decoding phase reasoning task.

[0025] In this embodiment, the first instance retains a portion of the key-value cache data during the inference task of the first instance in the decoding stage, and transmits query terms between the first instance and the second instance during the inference task of the decoding stage. This reduces the amount of communication between the first instance and the second instance in transmitting key-value cache data, and further improves the inference efficiency of the large model.

[0026] In one possible implementation, the inference configuration strategy is also used to configure the basic settings for large model inference, wherein the basic settings include the configuration of information such as model parameters. For example, the inference configuration strategy is also used to indicate the address for configuring the uploaded model parameters, the maximum inference length supported by large model inference, and the inference accuracy.

[0027] In this embodiment, the large model inference system can configure the address for uploading model parameters, the maximum inference length supported by large model inference, and the inference accuracy based on the inference configuration strategy, thereby improving the feasibility of large model inference configuration.

[0028] Secondly, embodiments of this application provide an inference configuration apparatus for a large model. The apparatus includes an acquisition unit and a processing unit. The acquisition unit acquires inference tasks for the large model, including inference tasks in a pre-filling stage and inference tasks in a decoding stage. The processing unit executes the inference tasks in the pre-filling stage, generating key-value cache data. This key-value cache data is used to execute the inference tasks in the decoding stage. The key-value cache data includes first key-value cache data and second key-value cache data. The first key-value cache data is stored in a first instance, and the second key-value cache data is sent to a second instance. The processing unit further executes the inference tasks in the decoding stage based on the first key-value cache data, and the second instance executes the inference tasks in the decoding stage based on the second key-value cache data.

[0029] In one possible implementation, the processing unit is further configured to configure the inference task based on an inference configuration strategy, wherein the inference configuration strategy is used to configure the first instance and the second instance to perform the inference task.

[0030] In one possible implementation, the inference configuration strategy is used to configure the first key-value cache data in the video memory space of the first instance. The inference configuration strategy includes a borrowed video memory space ratio, which is used to determine the portion of video memory space in the first instance that retains the first key-value cache data. The processing unit is also used to save the first key-value cache data in the video memory space of the first instance based on the borrowed video memory space ratio. The borrowed video memory space ratio is a fixed ratio, or it is an adaptively adjusted ratio.

[0031] In one possible implementation, the inference configuration strategy includes an instance scaling method, and the processing unit is further configured to scale the first instance or the second instance based on the instance scaling method, which includes real-time adjustment or delayed adjustment.

[0032] In one possible implementation, the inference configuration strategy includes instance scaling conditions, and the processing unit is further configured to scale the first instance or the second instance based on the instance scaling conditions. The instance scaling conditions include adjustment according to the first word delay TTFT or adjustment according to the video memory space threshold.

[0033] In one possible implementation, the processing unit is further configured to enable the first instance to expand the instance to perform the inference task when the first word latency of the inference task is greater than the first word latency threshold, or the processing unit is further configured to enable the first instance to expand the instance to perform the inference task when the remaining video memory space of the first instance is less than the video memory space threshold.

[0034] In one possible implementation, when the instance scaling method is a delay adjustment, the first character delay threshold includes a first first character delay threshold and a second first character delay threshold, and the video memory space threshold includes a first video memory space threshold and a second video memory space threshold. The processing unit is further configured to delay the execution of the inference task by expanding the first instance when the first character delay of the inference task is greater than the first first character delay threshold and less than the second first character delay threshold. Alternatively, the processing unit is further configured to delay the execution of the inference task by expanding the first instance when the remaining video memory space of the first instance is greater than the first video memory space threshold and less than the second video memory space threshold.

[0035] In one possible implementation, the processing unit is specifically used to configure the memory space management method of the second instance based on the inference configuration strategy. The memory space management method is used to instruct the second instance to perform an instance selection strategy for instance expansion. The memory space management method includes a polling strategy and length prediction assistance.

[0036] In one possible implementation, the processing unit is further configured to determine the second instance to perform the inference task based on the prediction result of the length prediction model when the memory space management method of the second instance is length prediction-assisted. The length prediction model is used to predict the memory space required for the inference task in the decoding stage.

[0037] In one possible implementation, the processing unit is specifically used to transmit query terms between the first instance and the second instance, the query terms being used to perform inference tasks in the decoding phase. In a third aspect, embodiments of this application provide a computing device including a processor coupled to a memory. The processor stores instructions that, when executed by the processor, cause the computing device to perform the method described in the first aspect or any possible implementation thereof.

[0038] Fourthly, embodiments of this application provide a computing device cluster, which includes one or more computing devices. Each computing device includes a processor coupled to a memory. The processor is used to store instructions, which, when executed by the processor, cause the computing device cluster to perform the method described in the first aspect or any possible implementation thereof.

[0039] Fifthly, embodiments of this application provide a computer-readable storage medium having instructions stored thereon, which, when executed, cause a computer to perform the method described in the first aspect or any possible implementation thereof.

[0040] Sixthly, embodiments of this application provide a computer program product including instructions that, when executed, cause a computer to implement the method described in the first aspect or any possible implementation thereof.

[0041] It is understood that the beneficial effects achievable by any of the large-scale model inference devices, computing devices, computing device clusters, computer-readable media, or computer program products provided above can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0042] Figure 1 A schematic diagram of the system architecture of a large-scale model inference system provided for embodiments of this application;

[0043] Figure 2 A flowchart illustrating a large-scale model inference method provided in an embodiment of this application;

[0044] Figure 3 A schematic diagram of an inference deployment interface for a large model provided in an embodiment of this application;

[0045] Figure 4 A schematic diagram of another large model inference deployment interface provided in an embodiment of this application;

[0046] Figure 5 A schematic diagram of another large model inference deployment interface provided in an embodiment of this application;

[0047] Figure 6 A schematic diagram of another large model inference deployment interface provided in an embodiment of this application;

[0048] Figure 7 A schematic diagram illustrating a large model performing an inference task, provided as an embodiment of this application;

[0049] Figure 8 A schematic diagram of a large-scale inference device provided in an embodiment of this application;

[0050] Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0051] Figure 10 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0052] Figure 11 This is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0053] This application provides a method and apparatus for reasoning large models, which can reduce the reasoning latency of large models and improve the reasoning efficiency of large models.

[0054] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0055] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0056] First, some of the terms used in the embodiments of this application are introduced to facilitate understanding of the technical solutions by those skilled in the art.

[0057] Large language models (LLMs) are natural language processing models built using deep learning techniques. They have a large number of parameters and complex structures, and are capable of processing and generating natural language text.

[0058] Inference of large language models (LLMs) refers to the process of predicting or generating new text on a pre-trained large language model. This process can be divided into two stages: pre-filling and decoding.

[0059] The prefill phase calculates all user input and generates corresponding key-value (KV) cached data, outputting the first token. Prefilling is a computationally intensive process, involving a large amount of computation, but it is a one-time calculation process that performs the calculation for all words in the user input at once.

[0060] The decoding phase involves progressively generating the output. First, the model initializes the decoding based on a pre-filled cache and the input state. Then, the model generates each word or token sequentially, using the previously generated results as input for the next step. During decoding, the model calculates the probability distribution at each step and selects the most likely word or token. This process continues until a termination condition is met. The decoding phase is a memory-intensive process.

[0061] Prefill and decode (PD) separation refers to instantiating and deploying the prefill and decode stages of model inference on different machine resources and performing inference simultaneously. It combines the computationally intensive characteristics of the prefill stage and the memory-intensive characteristics of the decode stage. By adjusting the ratio of PD nodes, the batch size of the decode nodes can be increased to fully utilize the computing power of the computing devices and thus improve the overall throughput of the computing device cluster.

[0062] Sequence parallelism involves segmenting a very long text sequence into multiple shorter subsequences and processing these subsequences in parallel across multiple computing devices. Sequence parallelism better utilizes the computing power of hardware (such as GPUs), reduces the computational burden on individual computing devices, and improves overall training or inference performance. Sequence parallelism makes models more scalable when processing long sequence data, allowing computational power to be scaled as hardware resources increase.

[0063] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0064] Please see Figure 1 , Figure 1 This application provides a schematic diagram of the system architecture for a large-model inference system. Figure 1 In the example shown, the large model inference system 10 includes a borrowed component 101, an adaptive scaling module 102, and a space management module 103. The large model inference system 10 is applied to large model inference service scenarios, and can configure the large model inference service when deploying it. The specific functions of each part of the large model inference system 10 are described below.

[0065] The borrowed component 101 is used to perform inference tasks by borrowing instances during large model inference. The borrowed component 101 can be a component in different instances, including instances in the pre-filling stage and instances in the decoding stage. Instances in the pre-filling stage can also be called P instances or P nodes, and instances in the decoding stage can also be called D instances or D nodes. That is, the borrowed component 101 also includes borrowed components of P instances and borrowed components of D instances.

[0066] In the process of implementing the borrowed instance to perform inference tasks, the P instance can borrow video memory space from the D instance to perform inference tasks during the decoding phase. Specifically, after the P instance obtains key-value cache data by performing the inference task in the pre-filling phase, it retains a portion of the key-value cache data corresponding to user requests in the P instance. Thus, during the decoding phase, the P instance can perform the inference task in the decoding phase based on the query transmitted by the D instance and the retained key-value cache data.

[0067] During the inference process of the large model, since the inference task in the pre-filling stage is a computationally intensive task and the inference task in the decoding stage is a memory-intensive task, the borrowed component 101 utilizes part of the bandwidth capacity of the P instance during the inference process of the large model, thereby alleviating the bandwidth requirements of the D instance in the decoding stage, thus realizing the borrowing of the P instance. In addition, the P instance retains key-value cache data and executes the inference task in the decoding stage in parallel with the D instance, which also realizes the sequential parallelism between the P instance and the D instance in the PD separation architecture.

[0068] The adaptive scaling module 102 is used to expand or shrink the inference instances of a large model according to the actual load. Inference instances include P instances and D instances. Expanding inference instances can be achieved by adding computing devices to perform inference tasks, while shrinking inference instances can be achieved by reducing the number of computing devices performing inference tasks. For example, the adaptive scaling module 102 achieves optimal efficiency for the entire inference cluster by launching new instances or releasing existing instances in real time based on the load.

[0069] The adaptive scaling module 102 can also expand or shrink the inference instances of large models according to the instance scaling conditions. For example, the adaptive scaling module 102 can determine whether it is necessary to expand the memory space of P instances or D instances to process the current long sequence, or to shrink the memory space of P instances or D instances to save resources, based on information such as the sequence length of the current inference request, the current resource usage of the inference system, and the inference first word delay TTFT.

[0070] The adaptive scaling module 102 can also choose to expand or shrink the inference instance of the large model in real time, or delay the expansion or shrinkage of the inference instance of the large model. For example, if the first word latency (TTFT) of the current inference request is greater than 500ms, the adaptive scaling module 102 will expand the inference instance of the large model in real time; if the first word latency (TTFT) of the current inference request is greater than 300ms and less than 500ms, the adaptive scaling module 102 will delay the expansion of the inference instance of the large model.

[0071] The space management module 103 manages the video memory space of the inference instance, including calculating the amount of video memory space that needs to be expanded or reduced, and selecting the instance to perform the inference task. The video memory space is used to store key-value cache (KV cache) data. In this embodiment, the video memory space of the inference instance is also called KV space, and the space management module 103 can also be called the KV space management module. The space management module 103 can be a module in different instances, including the space management module for instance P and the space management module for instance D.

[0072] For example, after instance P obtains key-value cache data during the pre-filling phase inference task, space management module 103 can calculate and determine the amount of video memory required for the key-value cache data retained by instance P, as well as the amount of video memory occupied by the key-value cache data transferred to instance D. As another example, during the decoding phase inference task, space management module 103 can calculate and determine the instance D performing the inference task based on the video memory space already used by different instances D.

[0073] The space management module 103 includes a length prediction model, which predicts the video memory space required to perform inference tasks. This predicted video memory space can also be referred to as the KV budget. Based on the video memory space requirements predicted by the length prediction model, the space management module 103 determines the inference instance to perform the inference task. When the space management module 103 determines that the video memory space is no longer sufficient to handle more requests, the space management module 103 and the adaptive scaling module 102 work together to expand the instance.

[0074] It should be noted that the trained large model can be deployed on nodes in a cloud environment or on nodes in an edge environment. That is, the borrowed component 101, the adaptive scaling module 102, and the space management module 103 in the large model's inference system 10 can be modules from the aforementioned nodes. Nodes in the cloud environment can be virtual machine instances, container instances, computing devices, physical servers, etc. In this embodiment, computing devices or clusters of computing devices can also be used to refer to the various modules in the large model's inference system 10.

[0075] based on Figure 1 The present application also provides a reasoning method for large models, in addition to the large model reasoning system 10 shown. The reasoning method for large models provided in this application will be described below with reference to embodiments.

[0076] Please see Figure 2 , Figure 2 This is a flowchart illustrating a large-scale model inference method provided in an embodiment of this application. Figure 2 In the example shown, the method includes the following steps:

[0077] 201. Obtain the inference tasks for the large model, which include inference tasks in the pre-filling stage and inference tasks in the decoding stage.

[0078] The large model inference method provided in this application embodiment is applied to the large model inference service scenario. In the large model inference service scenario, after the user trains the large model and completes the deployment, the user can perform inference tasks based on the large model. The inference tasks performed by the large model inference system 10 include tasks such as dialogue chat, knowledge question answering, code continuation, creative tasks, and document summarization.

[0079] The large model inference system 10 acquires the inference task of the large model. In this embodiment, the inference task of the large model includes a pre-filling stage inference task and a decoding stage inference task. The pre-filling stage inference task includes processing the user's input text and caching intermediate calculation results, which are key-value cache data. The decoding stage inference task includes generating an output sequence step by step based on the intermediate calculation results of the pre-filling stage.

[0080] The prefill and decode stages mentioned above refer to two phases in the autoregressive inference process of a large model. The large model needs to go through these two stages to perform inference tasks. The prefill stage involves the large model calculating the input content and outputting the first token. The prefill stage is a single computation, generating key-value cache data during this process. The decode stage, on the other hand, uses the key-value cache generated in the prefill stage and the current output token to iteratively calculate the next token, continuing until the maximum sequence length or the end-of-sequence character is reached. Therefore, prefilling and decoding are two inference processes with different computational characteristics. Prefilling is a computationally intensive inference process, while decoding requires frequent access to the key-value cache in memory, making it a memory-intensive inference process.

[0081] Since pre-filling and decoding are two inference processes with different computational characteristics, traditional large-model inference processes employ a separate architecture for pre-filling and decoding to execute inference tasks. That is, the inference task in the pre-filling stage is executed on the pre-filling instance, and the inference task in the decoding stage is executed on the decoding instance. However, in the large-model inference method provided in this application embodiment, the computing device configures the large model to utilize external memory space to execute the inference task in the decoding stage when performing inference tasks, through an inference configuration strategy. The execution process of the inference task in this application embodiment will be described in detail below with reference to subsequent steps.

[0082] 202. The first instance performs the inference task of the pre-filling stage and generates key-value cache data. The key-value cache data is used to perform the inference task of the decoding stage. The key-value cache data includes first key-value cache data and second key-value cache data. The first instance keeps the first key-value cache data in the first instance and sends the second key-value cache data to the second instance.

[0083] In the large-scale model inference system 10, the first instance performs the inference task of the pre-filling phase, generating key-value cache data. This key-value cache data is used to perform the inference task of the decoding phase. The key-value cache data includes first key-value cache data and second key-value cache data. The first instance retains the first key-value cache data within itself and sends the second key-value cache data to the second instance. The first key-value cache data is used to perform the inference task of the decoding phase within the first instance, and the second key-value cache data is used to perform the inference task of the decoding phase within the second instance. Here, the first instance is the pre-filling instance, used to perform the inference task of the pre-filling phase, and can also be called the P instance; the second instance is the decoding instance, used to perform the inference task of the decoding phase, and can also be called the D instance.

[0084] It is understood that the first instance and the second instance in the embodiments of this application can be virtual machine instances, container instances, processors, computing devices, physical servers, etc., and there is no specific limitation.

[0085] Before a large model performs an inference task, the computing device needs to obtain an inference configuration strategy and configure the inference task based on the inference configuration strategy. The inference configuration strategy is used to configure the first instance and the second instance to perform the inference task. The inference configuration strategy can instruct the memory space allocation rules when different instances perform large model inference tasks. Different instances include the first instance and the second instance.

[0086] The inference configuration strategy in this embodiment can be used to configure the first instance to perform the inference task in the pre-filling stage, and to configure the first instance and the second instance to perform the inference task in the decoding stage in parallel. The inference configuration strategy can also configure the video memory space of the first instance for performing the inference task in the decoding stage, i.e., the aforementioned video memory allocation rule. The inference configuration strategy can also configure the number of the first instance performing the inference task in the pre-filling stage and the number of the second instance performing the inference task in the decoding stage.

[0087] In one possible implementation, the large model inference system 10 can also configure the basic settings for large model inference based on an inference configuration strategy. The basic settings include the configuration of information such as model parameters. For example, the large model inference system 10 can configure the address for uploading model parameters, the maximum inference length supported by large model inference, and the inference accuracy based on the inference configuration strategy.

[0088] Please see Figure 3 , Figure 3 This is a schematic diagram of an inference deployment interface for a large model provided in an embodiment of this application. Figure 3 In the example shown, when deploying an inference service for a large model, users can configure the basic settings of the large model based on the inference configuration strategy. For example, users can upload model parameters of the large model through the configuration interface. The uploaded model parameters can come from local storage or from an online storage system, such as Object Storage Service (OBS).

[0089] exist Figure 3 In the example shown, for instance, the user can configure the maximum supported length for inference of the large model to be 4096 through the configuration interface. The maximum supported length refers to the maximum length of input text that the large model can process when performing inference. The user can also configure whether the large model uses FP16 precision, where FP16 stands for half-precision floating-point number.

[0090] In one possible implementation, the large-model inference system 10 can configure the deployment architecture of the large model and the parallel strategy for executing inference tasks based on an inference configuration strategy. The deployment architecture of the large model can be, for example, a hybrid PD architecture and a PD separate architecture. In the hybrid PD architecture, the pre-filling stage and the decoding stage are deployed together on the same computing device or in the same cluster; that is, the processing of the pre-filling stage and the decoding stage may be performed by the same model or the same computing resource. The PD separate architecture processes the pre-filling stage and the decoding stage separately, allowing the two inference stages to select different hardware.

[0091] Parallelism strategies include tensor parallelism (TP), pipeline parallelism (PP), and sequence parallelism (SP). Tensor parallelism, in particular, accelerates computation and improves efficiency by distributing model parameters and computational tasks across multiple computing devices. Tensor parallelism can break down a large tensor (such as the weight matrix of a neural network) into multiple smaller sub-tensors and distribute these sub-tensors across different computing devices (such as multiple GPUs).

[0092] Pipeline parallelism refers to distributing different layers or stages of a neural network model to different computing devices, allowing data to be processed sequentially through each device like on a pipeline. This enables different devices to process different parts of the computational tasks at different times, thereby improving overall computational efficiency and model training speed.

[0093] Sequence parallelism reduces computation time by dividing long sequences into smaller segments and processing these segments in parallel. This approach makes models more scalable when processing long sequences of data, allowing for easy scaling of computing power as hardware resources increase, and fully utilizing the computing resources of multiple devices to accelerate the model's training or inference process.

[0094] Please see Figure 4 , Figure 4 This is a schematic diagram of another large-scale model inference deployment interface provided in an embodiment of this application. Figure 4 In the example shown, users can configure the deployment architecture of large model inference through the large model inference configuration interface. The deployment architecture includes a PD hybrid architecture and a PD separate architecture. For example, users can configure the deployment architecture of large model inference as a PD separate architecture through the large model inference configuration interface. The PD separate architecture can reduce resource conflicts and latency fluctuations between the pre-filling stage and the decoding stage by separating the pre-filling stage and the decoding stage, thereby improving the stability of inference.

[0095] exist Figure 4 In the example shown, users can also configure parallel strategies for large model inference through the large model inference configuration interface. Parallel strategies include tensor parallelism (TP), pipelined parallelism (PP), and sequence parallelism (SP). For example, users can select sequence parallelism (SP) as the parallel strategy for large model inference in the configuration interface.

[0096] The inference configuration strategy in this application embodiment is mainly used to configure the memory space allocation rules for the first instance and the second instance to perform large model inference tasks. That is, the inference configuration strategy in this application embodiment can configure the first instance to borrow external memory space to perform inference tasks in the decoding stage, so that the first instance and the second instance can perform inference tasks in the decoding stage in parallel.

[0097] It should be noted that, during the process of determining the inference of the large model based on the inference configuration strategy, the large model inference system 10 allocates the GPU memory space of the first instance for executing the inference task in the decoding phase, which is a high-level configuration under a specific configuration that serves the large model inference. After completing the basic configuration and selecting the deployment architecture as a PD separation architecture and the parallel strategy as a sequential parallel strategy, the large model inference system 10 can further perform advanced configurations such as GPU memory space allocation and instance scaling under the sequential parallel scheme. These advanced configurations are described in detail below with examples:

[0098] Please see Figure 5 , Figure 5 This is a schematic diagram of another large-scale model inference deployment interface provided in an embodiment of this application. Figure 5 In the example shown, the user completes Figure 3 and Figure 4After configuring the example shown, you can configure the details of the sequence parallel scheme in the inference deployment interface of the large model based on the inference configuration strategy, including the sequence parallel mode under the PD separation architecture, the granularity of sequence parallelism, the instance scaling strategy, and the management method of instance memory space.

[0099] exist Figure 5 In the example shown, the sequence parallelism method under the PD separation architecture includes hybrid KV borrowing, where the P instance can borrow memory space from the D instance to store key-value cache data. The granularity of sequence parallelism refers to the degree of parallelism of the large model during sequence parallel inference. This configuration includes fixed granularity and dynamic adaptive adjustment. Fixed granularity means that the inference task is executed with a fixed degree of parallelism, while dynamic adaptive adjustment means that the parallelism can be adaptively adjusted according to load conditions, etc. For example, a user can configure the sequence parallelism method under the PD separation architecture as "Method 1: Hybrid KV Borrowing" in the large model's inference deployment interface, and set the granularity (degree) of sequence parallelism to dynamic adaptive adjustment, meaning the inference system can adaptively adjust the degree of sequence parallelism according to the load conditions.

[0100] The inference configuration strategy in this application includes an instance scaling strategy. This strategy determines the instance scaling method and conditions for scaling. The instance scaling method includes real-time adjustment and delay adjustment, and the instance scaling conditions include adjustment based on the first-word delay (TTFT) or adjustment based on a memory space threshold. The following examples illustrate this in detail:

[0101] In one possible implementation, the inference configuration strategy includes an instance scaling method. The computing device scales up or down a first instance or a second instance based on this method. The instance scaling method includes real-time adjustment or delayed adjustment. Real-time adjustment means immediately expanding or shrinking instances after the instance scaling conditions are met, while delayed adjustment means expanding or shrinking instances according to a delayed rule after the instance scaling conditions are met. Expanding instances means increasing the number of instances executing inference tasks, and shrinking instances means decreasing the number of instances executing inference tasks.

[0102] Please continue reading. Figure 5 ,exist Figure 5 In the example shown, users can configure instance scaling strategies in the configuration interface for large model inference deployment. Instance scaling strategies include instance scaling methods, which include real-time adjustment and delayed adjustment. For example, users can select "delayed adjustment" as the instance scaling method in the "Instance Scaling Strategy" configuration bar of the configuration interface.

[0103] In one possible implementation, the inference configuration strategy further includes instance scaling conditions. The computing device scales the first instance or the second instance based on the instance scaling conditions. The instance scaling conditions include adjustment according to the first-word delay (TTFT) or adjustment according to a memory space threshold. Adjustment according to the first-word delay (TTFT) refers to expanding or shrinking the instance based on the first-word delay threshold corresponding to the first-word delay, while adjustment according to the memory space threshold refers to expanding or shrinking the instance based on the memory space threshold corresponding to the remaining memory space.

[0104] Please continue reading. Figure 5 ,exist Figure 5 In the example shown, users can also configure instance scaling conditions in the instance scaling strategy in the configuration interface for large model inference deployment. Instance scaling conditions include adjustment based on first word latency (TTFT) and adjustment based on memory space threshold. Users can set the first word latency threshold and the remaining memory space threshold as instance scaling conditions in the "Instance Scaling Strategy" configuration column of the configuration interface. For example, the user sets the first word latency threshold to 500ms, or the user sets the remaining memory space threshold to 5%.

[0105] In one possible implementation, when the first-word latency of the large model executing the inference task is greater than a first-word latency threshold, the large model inference system 10 can expand the number of first instances executing the inference task; or, when the remaining video memory space of the first instance is less than a video memory space threshold, the large model inference system 10 can expand the number of first instances executing the inference task. When the first-word latency of the large model executing the user inference task is less than a first-word latency reduction threshold, the large model inference system 10 can maintain or reduce the number of first instances executing the inference task; or, when the remaining video memory space of the first instance is greater than a video memory space reduction threshold, the large model inference system 10 can reduce the number of first instances executing the inference task.

[0106] Please continue reading. Figure 5 ,exist Figure 5 In the example shown, users can also set instance scaling conditions. For example, they can set the first-word latency threshold to 500ms, or set the remaining video memory space threshold to 5%. That is, when the first-word latency threshold of the second instance is greater than 500ms, or when the remaining video memory space of the second instance is less than 5%, the second instance will be scaled up.

[0107] In one possible implementation, when the instance scaling method is delayed adjustment, instance scaling is performed delayedly when the instance meets the first instance scaling condition, and immediately when the instance meets the second instance scaling condition. In this case, the first-character latency threshold includes a first first-character latency threshold and a second first-character latency threshold, and the video memory space reduction threshold includes a first video memory space threshold and a second video memory space threshold. The first instance scaling condition is that the first-character latency of the inference task is greater than the first first-character latency threshold and less than the second first-character latency threshold; the second instance scaling condition is that the first-character latency is greater than the second first-character latency threshold. Alternatively, the first instance scaling condition is that the remaining video memory space of the first instance is less than the first video memory space threshold but greater than the second video memory space threshold; the second instance scaling condition is that the remaining video memory space of the first instance is less than the second video memory space threshold.

[0108] For example, if the delay of the first character of the reasoning task is greater than the first delay threshold but less than the second delay threshold, the first instance will delay the execution of the extended instance of the reasoning task. If the delay of the first character of the reasoning task is greater than the second delay threshold, the first instance will immediately extend the instance of the reasoning task, that is, the delay time of the extended instance is 0.

[0109] Alternatively, if the remaining video memory space of the first instance is greater than the first video memory space threshold and less than the second video memory space threshold, then the first instance delays the expansion of the instance to execute the inference task. If the remaining video memory space of the first instance is less than the first video memory space threshold, then the first instance immediately expands the instance to execute the inference task.

[0110] Please continue reading. Figure 5 ,exist Figure 5 In the example shown, when the instance scaling method configured for different instances is delayed adjustment, and the first-character delay threshold includes a first first-character delay threshold and a second first-character delay threshold, for example, the first first-character delay threshold is 300ms and the second first-character delay threshold is 500ms, then when the first instance performs an inference task and its first-character delay threshold is greater than 300ms but less than 500ms, the first instance will delay its instance scaling. When the first instance performs an inference task and its first-character delay threshold is greater than 500ms, the first instance will scale immediately.

[0111] In one possible implementation, the inference configuration strategy in this application includes a borrowed video memory space ratio. The borrowed video memory space ratio is used to determine the portion of video memory space in the first instance that retains the first key-value cache data. After the computing device performs the inference task of the pre-filling stage based on the first instance, it saves the first key-value cache data in the video memory space of the first instance based on the borrowed video memory space ratio. The borrowed video memory space ratio is a fixed ratio, or the borrowed video memory space ratio is an adaptively adjusted ratio.

[0112] In this application embodiment, the methods for borrowing video memory space include a fixed proportion of borrowed video memory space and adaptive adjustment of borrowed video memory space. A fixed proportion of borrowed video memory space means that the first instance is set to allow a fixed proportion of video memory space to be reserved for performing inference tasks in the decoding stage. Adaptive adjustment of borrowed video memory space means that the first instance can adaptively decide the proportion of video memory space to be reserved.

[0113] For example, a fixed percentage of external memory can be allocated to the P instance in a large model inference system 10, which reserves 20% of its memory for inference tasks during the decoding phase, based on an inference configuration strategy. Another method is adaptive adjustment of external memory allocation, where the large model inference system 10 adaptively adjusts the amount of memory allocated to the P instance for inference tasks during the decoding phase based on the available memory available during inference.

[0114] Please continue reading. Figure 5 ,exist Figure 5 In the example shown, users can configure the memory management method in the configuration interface for large model inference deployment. For example, users can select the memory management method for P instances in the "KV Space Management" configuration bar. The memory management methods for P instances include fixed memory ratio and adaptive adjustment of memory. Users can choose to set a fixed memory ratio, such as 20%, or choose to dynamically and adaptively adjust the memory ratio. There are no specific limitations.

[0115] It should be noted that in the embodiments of this application, the fixed external memory space ratio and the adaptive adjustment of external memory space can be selected simultaneously. That is, within the configured fixed external memory space ratio, the proportion of memory space used for the inference task in the decoding stage in the first instance can be adaptively adjusted.

[0116] In one possible implementation, during the process of configuring the inference task based on the inference configuration strategy, the computing device configures the memory space management method of the second instance based on the inference configuration strategy. The memory space management method is used to instruct the second instance to perform an instance selection strategy for instance expansion. The memory space management method includes a polling strategy and length prediction assistance.

[0117] The polling strategy refers to a memory management method in which the inference system polls multiple second instances to query their remaining memory space. Based on this polling strategy, the inference system can determine the remaining memory space of multiple second instances and, based on this remaining memory space, select the second instance to execute the decoding phase inference task.

[0118] Length prediction assistance refers to the second instance predicting the memory space requirements for performing the inference task based on a length prediction model, and managing the memory space of the second instance according to these requirements, including determining the second instance to perform the inference task in the decoding stage. When the memory space management method of the second instance is length prediction assistance, the second instance determines the second instance to perform the inference task based on the prediction result of the length prediction model, wherein the length prediction model is used to predict the memory space required for the inference task in the decoding stage.

[0119] Please continue reading. Figure 5 ,exist Figure 5 In the example shown, users can configure the memory management method in the configuration interface for large model inference deployment. For example, users can select the memory management method for instance D in the "Memory Management" configuration bar. The memory management methods for instance D include polling strategy and length prediction assistance. If the user selects "Length Prediction Assistance" as the memory management method for instance D, instance D can then predict the memory requirements for performing inference tasks based on the length prediction model and manage the memory space of the second instance according to the memory requirements.

[0120] After determining the inference configuration of the large model based on the inference configuration strategy, the large model inference system 10 deploys the configured large model and executes the user's inference task based on the deployed large model.

[0121] Please see Figure 6 , Figure 6 This is a schematic diagram of a deployment interface for a configured large model provided in an embodiment of this application. Figure 6 In the example shown, after the user completes the inference configuration on the deployment interface of the large model, the configured large model can be deployed to the cloud environment or the edge environment. The inference deployment interface of the large model can display the deployment progress of the large model. When the deployment progress reaches 100%, the large model is successfully deployed.

[0122] exist Figure 6 In the example shown, the deployed large model can become a large model application or a part of an application. Users can invoke the large model deployed in edge or cloud environments via application programming interface (API) calls. For example, users can access the large model application online via a web page or a client app.

[0123] 203. The first instance performs the inference task of the decoding stage based on the first key-value cache data, and the second instance performs the inference task of the decoding stage based on the second key-value cache data.

[0124] After the first instance generates key-value cache data by performing the inference task in the pre-filling phase, it sends the second key-value cache data to the second instance for performing the inference task in the decoding phase, while retaining a portion of the first key-value cache data in the first instance. Since the first instance's GPU memory stores the first key-value cache data, the first instance performs the inference task in the decoding phase based on the first key-value cache data, and the second instance performs the inference task in the decoding phase based on the second key-value cache data. That is, the first and second instances can execute the inference task in the decoding phase in parallel, thus achieving sequential parallelism between the first and second instances.

[0125] In one possible implementation, the large model's inference system 10 transmits query terms between the first instance and the second instance during the parallel execution of the decoding phase inference task based on the first key-value cache data in the first instance and the second instance. The query terms are used to execute the decoding phase inference task.

[0126] Specifically, since the first instance retains a portion of the key-value cache data after the pre-filling phase computation is completed, the first instance can perform the inference task of the decoding phase in parallel with the second instance based on the retained first key-value cache data. When the first instance and the second instance perform the self-attention computation in the decoding phase, only the query term needs to be transmitted between the first instance and the second instance, without transmitting the key-value cache data.

[0127] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating the execution of an inference task by a large model in an embodiment of this application. Figure 7 In the example shown, the inference task of the large model includes a pre-filling phase inference task and a decoding phase inference task, wherein the pre-filling phase inference task is executed in one or more P instances, and the decoding phase inference task is executed in one or more D instances.

[0128] exist Figure 7 In the example shown, the P instance configured according to the inference configuration strategy provided in this application embodiment can borrow video memory space to perform inference tasks in the decoding phase. Specifically, when the P instance performs the inference task in the pre-filling phase, after generating the KV cache, the P instance sends a portion of the KV cache to the D instance, while retaining a portion of the KV cache in the P instance. The D instance performs the inference task in the decoding phase based on the received KV cache, and the P instance can also perform the inference task in the decoding phase in parallel with the D instance based on the retained KV cache.

[0129] For example, a computing device can store a portion of the key-value cache for one or more large model inference requests in a P instance, allowing it to execute inference tasks in parallel with the D instance sequence. During decoding, the D instance transmits the query to the P instance, which performs self-attention computation. After computation, the P instance synchronizes back to the D instance. Since the query size is on the kilobyte scale, the communication load between the D and P instances is smaller compared to transmitting the key-value cache. Furthermore, utilizing the bandwidth of the P instance reduces the bandwidth requirements of the D instance, increases the batch size of the inference tasks executed by the D instance, and improves memory utilization.

[0130] As can be seen from the above embodiments, in the large model inference method provided in this application embodiment, the large model inference system can configure the video memory space of the first instance to perform the inference task in the decoding stage. That is, it can use the bandwidth capability of the first instance to share the bandwidth pressure of the second instance, thereby using a scheme with lower communication cost to achieve sequence parallelism under the PD separation architecture, thereby improving the computing power utilization of different instances, reducing the inference latency of the large model, and improving the inference efficiency of the large model.

[0131] Based on the above method embodiments, this application also provides a large model inference device, and the large model inference device provided by this application embodiment is described in detail below.

[0132] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a large-scale inference device provided in an embodiment of this application. Figure 8 In the example shown, the large model inference device 800 is used to implement the various steps performed by the large model inference system in the above embodiments. The large model inference device 800 includes an acquisition unit 801 and a processing unit 802.

[0133] The acquisition unit 801 is used to acquire the inference tasks of the large model, including inference tasks in the pre-filling stage and inference tasks in the decoding stage. The processing unit 802 is used to execute the inference tasks in the pre-filling stage, generating key-value cache data. The key-value cache data is used to execute the inference tasks in the decoding stage. The key-value cache data includes first key-value cache data and second key-value cache data. The first key-value cache data is stored in the first instance, and the second key-value cache data is sent to the second instance. The processing unit 802 is also used to execute the inference tasks in the decoding stage based on the first key-value cache data, and the second instance executes the inference tasks in the decoding stage based on the second key-value cache data.

[0134] In one possible implementation, the processing unit 802 is further configured to configure the inference task based on an inference configuration strategy, wherein the inference configuration strategy is used to configure the first instance and the second instance to perform the inference task.

[0135] In one possible implementation, the inference configuration strategy is used to configure the first key-value cache data in the video memory space of the first instance. The inference configuration strategy includes a borrowed video memory space ratio, which is used to determine the portion of video memory space in the first instance that retains the first key-value cache data. The processing unit 802 is also used to save the first key-value cache data in the video memory space of the first instance based on the borrowed video memory space ratio. The borrowed video memory space ratio is a fixed ratio, or it is an adaptively adjusted ratio.

[0136] In one possible implementation, the inference configuration strategy includes an instance scaling method, and the processing unit 802 is further configured to scale the first instance or the second instance based on the instance scaling method, which includes real-time adjustment or delayed adjustment.

[0137] In one possible implementation, the inference configuration strategy includes instance scaling conditions, and the processing unit 802 is further configured to scale the first instance or the second instance based on the instance scaling conditions. The instance scaling conditions include adjustment according to the first word delay TTFT or adjustment according to the video memory space threshold.

[0138] In one possible implementation, the processing unit 802 is further configured to enable the first instance to expand the instance to perform the inference task when the first word latency of the inference task is greater than the first word latency threshold, or the processing unit 802 is further configured to enable the first instance to expand the instance to perform the inference task when the remaining video memory space of the first instance is less than the video memory space threshold.

[0139] In one possible implementation, when the instance scaling method is a delay adjustment, the first character delay threshold includes a first first character delay threshold and a second first character delay threshold, and the video memory space threshold includes a first video memory space threshold and a second video memory space threshold. The processing unit 802 is further configured to delay the execution of the inference task by expanding the first instance when the first character delay of the inference task is greater than the first first character delay threshold and less than the second first character delay threshold. Alternatively, the processing unit 802 is further configured to delay the execution of the inference task by expanding the first instance when the remaining video memory space of the first instance is greater than the first video memory space threshold and less than the second video memory space threshold.

[0140] In one possible implementation, the processing unit 802 is specifically used to configure the memory space management method of the second instance based on the inference configuration strategy. The memory space management method is used to instruct the second instance to perform an instance selection strategy for instance expansion. The memory space management method includes a polling strategy and length prediction assistance.

[0141] In one possible implementation, the processing unit 802 is further configured to determine the second instance to perform the inference task based on the prediction result of the length prediction model when the memory space management method of the second instance is length prediction-assisted. The length prediction model is used to predict the memory space required for the inference task in the decoding stage.

[0142] In one possible implementation, the processing unit 802 is specifically used to transmit query terms between the first instance and the second instance, the query terms being used to perform inference tasks during the decoding phase.

[0143] It is understandable that the acquisition unit 801 and processing unit 802 in the inference device 800 of the large model can serve as functional modules. Figure 1 The various modules in the inference system 10 of the medium-to-large model are mapped to each other, thereby realizing the functions of each module in the inference system 10 of the large model.

[0144] It should be understood that the division of units in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, all units in the device can be implemented entirely through software calls from processing elements; all units can be implemented entirely in hardware; or some units can be implemented through software calls from processing elements, and others in hardware. For example, each unit can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as a program in memory, called and executed by a processing element of the device. Moreover, these units can be fully or partially integrated together, or implemented independently. The processing element mentioned here can also be called a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above units can be implemented through integrated logic circuits in the processor element or through software calls from processing elements.

[0145] It is worth noting that, for the sake of simplicity, the above method embodiments are described as a series of actions. However, those skilled in the art should know that this application is not limited to the order of the described actions. Furthermore, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by this application.

[0146] Other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Furthermore, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0147] Please see Figure 9, Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 9 As shown, the computing device 900 includes a processor 901, a memory 902, a communication interface 903, and a bus 904. The processor 901, memory 902, and communication interface 903 are coupled via the bus (not shown in the figure). The memory 902 stores instructions. When the instructions in the memory 902 are executed, the computing device 900 executes the method performed by the inference system of the large model in the above method embodiment.

[0148] The computing device 900 may be one or more integrated circuits configured to implement the methods described above, such as: one or more application-specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), or a combination of at least two of these forms of integrated circuits. Furthermore, when the units in the device can be implemented in the form of a processing element scheduler, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Alternatively, these units may be integrated together and implemented as a system-on-a-chip (SOC).

[0149] Processor 901 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0150] The memory 902 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0151] The memory 902 stores executable program code, and the processor 901 executes the executable program code to implement the functions of the aforementioned units or modules, thereby realizing the reasoning method of the aforementioned large model. That is, the memory 902 stores instructions for executing the reasoning method of the aforementioned large model.

[0152] The communication interface 903 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

[0153] In addition to the data bus, the 904 bus can also include a power bus, a control bus, and a status signal bus. The bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The bus can be categorized into address bus, data bus, and control bus.

[0154] Please see Figure 10 , Figure 10 This is a schematic diagram of a computing device cluster provided in an embodiment of this application. Figure 10 As shown, the computing device cluster 1000 includes at least one computing device 900.

[0155] like Figure 10 As shown, the computing device cluster 1000 includes at least one computing device 900. The memory 902 of one or more computing devices 900 in the computing device cluster 1000 may store the same instructions for executing the inference method of the large model described above.

[0156] In some possible implementations, the memory 902 of one or more computing devices 900 in the computing device cluster 1000 may also store partial instructions for executing the inference method of the aforementioned large model. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing the inference method of the aforementioned large model.

[0157] It should be noted that the memories 902 in the different computing devices 900 within the computing device cluster 1000 can store different instructions, each used to execute a portion of the functions of the inference device of the aforementioned large model. That is, the instructions stored in the memories 902 of the different computing devices 900 can implement the functions of one or more modules in the processing unit and the acquisition unit.

[0158] In some possible implementations, one or more computing devices 900 in the computing device cluster 1000 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0159] Please see Figure 11 , Figure 11This is a schematic diagram illustrating the network connection of computer devices in a computer cluster, as provided in an embodiment of this application. Figure 11 As shown, the computing device cluster 1100 includes computing devices 900A and 900B, which are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device.

[0160] In one possible implementation, the memory in computing device 900A stores instructions for performing the fetch unit function. Meanwhile, the memory in computing device 900B stores instructions for performing the processing unit function.

[0161] It should be understood that Figure 11 The functions of computing device 900A shown can also be performed by multiple computing devices. Similarly, the functions of computing device 900B can also be performed by multiple computing devices.

[0162] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device executes the method performed by the inference system of the large model in the above method embodiment.

[0163] In another embodiment of this application, a computer program product is also provided, which includes computer-executable instructions stored in a computer-readable storage medium. When the processor of the device executes the computer-executable instructions, the device performs the method executed by the inference system of the large model in the above method embodiments.

[0164] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0166] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0167] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0168] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A reasoning method for large models, characterized in that, include: Obtain the inference task of the large model, the inference task including the inference task in the pre-filling stage and the inference task in the decoding stage; The first instance executes the inference task of the pre-filling stage and generates key-value cache data. The key-value cache data is used to execute the inference task of the decoding stage. The key-value cache data includes first key-value cache data and second key-value cache data. The first key-value cache data is kept in the first instance, and the second key-value cache data is sent to the second instance. The first instance performs the inference task of the decoding stage based on the first key-value cache data, and the second instance performs the inference task of the decoding stage based on the second key-value cache data.

2. The method according to claim 1, characterized in that, The method further includes: The inference task is configured based on an inference configuration strategy, which is used to configure the first instance and the second instance to execute the inference task.

3. The method according to claim 2, characterized in that, The inference configuration strategy is used to configure the first key-value cache data in the video memory space of the first instance. The inference configuration strategy includes a proportion of external video memory, which is used to determine the portion of video memory space in the first instance where the first key-value cache data is retained. After performing the inference task of the pre-filling phase based on the first instance, the method includes: The first key-value cache data is stored in the video memory space of the first instance based on the proportion of external video memory space borrowed. The proportion of external video memory space borrowed is a fixed proportion, or the proportion of external video memory space borrowed is an adaptively adjusted proportion.

4. The method according to any one of claims 1 to 3, characterized in that, The inference configuration strategy includes instance scaling methods, and the method further includes: The instance scaling is performed on the first instance or the second instance based on the instance scaling method, which includes real-time adjustment or delayed adjustment.

5. The method according to any one of claims 1 to 4, characterized in that, The inference configuration strategy includes instance scaling conditions, and the method further includes: The first instance or the second instance is scaled up or down based on instance scaling conditions, which include adjusting according to the first word delay TTFT or adjusting according to the memory space threshold.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: If the first character delay of the inference task is greater than the first character delay threshold, then the first instance expands the instance to execute the inference task; Alternatively, if the remaining video memory space of the first instance is less than the video memory space threshold, the first instance expands to execute the inference task.

7. The method according to claim 6, characterized in that, When the instance scaling method is delay adjustment, the first character delay threshold includes a first first character delay threshold and a second first character delay threshold, and the video memory space threshold includes a first video memory space threshold and a second video memory space threshold. The method further includes: If the first character delay of the inference task is greater than the first first character delay threshold and less than the second first character delay threshold, then the first instance delays the execution of the inference task by the extended instance. Alternatively, if the remaining video memory space of the first instance is greater than the first video memory space threshold and less than the second video memory space threshold, then the first instance delays the execution of the inference task by expanding the instance.

8. The method according to any one of claims 2 to 7, characterized in that, The reasoning configuration for the reasoning task based on the reasoning configuration strategy includes: Based on the inference configuration strategy, the memory space management method of the second instance is configured. The memory space management method is used to instruct the second instance to perform an instance selection strategy for instance expansion. The memory space management method includes a polling strategy and length prediction assistance.

9. The method according to claim 8, characterized in that, The method further includes: When the memory space management method of the second instance is length prediction-assisted, the second instance determines the second instance to perform the inference task based on the prediction result of the length prediction model. The length prediction model is used to predict the memory space required for the inference task in the decoding stage.

10. The method according to any one of claims 1 to 9, characterized in that, The inference task of the decoding phase, which is performed in parallel based on the first instance and the second instance, includes: Query terms are transmitted between the first instance and the second instance, and the query terms are used to perform the reasoning task in the decoding phase.

11. A large-scale model reasoning device, characterized in that, include: An acquisition unit is used to acquire the inference task of a large model, the inference task including the inference task in the pre-filling stage and the inference task in the decoding stage. The processing unit is used to execute the inference task of the pre-filling stage, generate key-value cache data, the key-value cache data is used to execute the inference task of the decoding stage, the key-value cache data includes first key-value cache data and second key-value cache data, the first key-value cache data is kept in the first instance, and the second key-value cache data is sent to the second instance; The processing unit is further configured to perform the inference task of the decoding stage based on the first key-value cache data, and the second instance performs the inference task of the decoding stage based on the second key-value cache data.

12. The apparatus according to claim 11, characterized in that, The processing unit is also used for: The inference task is configured based on an inference configuration strategy, which is used to configure the first instance and the second instance to execute the inference task.

13. The apparatus according to claim 12, characterized in that, The inference configuration strategy is used to configure the first key-value cache data in the video memory space of the first instance, wherein the inference configuration strategy includes a proportion of external video memory space borrowed, the proportion of external video memory space borrowed is used to determine the portion of video memory space in the first instance that retains the first key-value cache data, and the processing unit is further used to: The first key-value cache data is stored in the video memory space of the first instance based on the proportion of external video memory space borrowed. The proportion of external video memory space borrowed is a fixed proportion, or the proportion of external video memory space borrowed is an adaptively adjusted proportion.

14. The apparatus according to any one of claims 11 to 13, characterized in that, The inference configuration strategy includes an instance scaling method, and the processing unit is further configured to: The instance scaling is performed on the first instance or the second instance based on the instance scaling method, which includes real-time adjustment or delayed adjustment.

15. The apparatus according to any one of claims 11 to 14, characterized in that, The inference configuration strategy includes instance scaling conditions, and the processing unit is further configured to: The first instance or the second instance is scaled up or down based on instance scaling conditions, which include adjusting according to the first word delay TTFT or adjusting according to the memory space threshold.

16. The apparatus according to any one of claims 11 to 15, characterized in that, The processing unit is also used for: If the first character delay of the inference task is greater than the first character delay threshold, then the first instance is expanded to execute the inference task. Alternatively, if the remaining video memory space of the first instance is less than the video memory space threshold, then the first instance is expanded to perform the inference task.

17. The apparatus according to claim 16, characterized in that, When the instance scaling method is delay adjustment, the first character delay threshold includes a first first character delay threshold and a second first character delay threshold, and the video memory space threshold includes a first video memory space threshold and a second video memory space threshold. The processing unit is further configured to: If the first character delay of the inference task is greater than the first first character delay threshold and less than the second first character delay threshold, then the first instance will delay the execution of the inference task by the expanded instance. Alternatively, if the remaining video memory space of the first instance is greater than the first video memory space threshold and less than the second video memory space threshold, then the first instance is delayed in performing the inference task by expanding the instance.

18. The apparatus according to any one of claims 12 to 17, characterized in that, The processing unit is specifically used for: Based on the inference configuration strategy, the memory space management method of the second instance is configured. The memory space management method is used to instruct the second instance to perform an instance selection strategy for instance expansion. The memory space management method includes a polling strategy and length prediction assistance.

19. The apparatus according to claim 18, characterized in that, The processing unit is also used for: When the memory space management method of the second instance is length prediction-assisted, the second instance determines the second instance to perform the inference task based on the prediction result of the length prediction model. The length prediction model is used to predict the memory space required for the inference task in the decoding stage.

20. The apparatus according to any one of claims 11 to 19, characterized in that, The processing unit is specifically used for: Query terms are transmitted between the first instance and the second instance, and the query terms are used to perform the reasoning task in the decoding phase.

21. A computing device, characterized in that, The device includes a processor coupled to a memory, the processor storing instructions which, when executed by the processor, cause the computing device to perform the method of any one of claims 1 to 10.

22. A computing device cluster, characterized in that, The system includes at least one computing device, the computing device including a processor coupled to a memory, the processor being used to store instructions that, when executed by the processor, cause the cluster of computing devices to perform the method of any one of claims 1 to 10.

23. A computer-readable storage medium having instructions stored thereon, characterized in that, When the instructions are executed, they cause the computer to perform the method of any one of claims 1 to 10.

24. A computer program product, the computer program product comprising instructions, characterized in that, When the instructions are executed, they cause the computer to perform the method of any one of claims 1 to 10.

Images