Method, device, equipment and storage medium for deploying inference service of large language model

Abstract

The present application relates to a method, apparatus, device and storage medium for deploying inference services for a large language model, and relates to the field of artificial intelligence technology. The method comprises: obtaining the number of parameters of the large language model, the respective video memory capacity of a plurality of devices to be allocated, the number of computing cores and network information, and determining a device combination of a plurality of devices to be allocated, each device combination consisting of a first device for the pre-filling stage and a second device for the decoding stage; for each device combination, determining the text unit processing capability based on the number of parameters, the respective video memory capacity of a plurality of devices to be allocated and the space occupied by a single input text unit; determining the inference delay under the current device combination based on the text unit processing capability, the respective number of computing cores of a plurality of devices to be allocated, and network information; determining a target device combination whose inference delay meets the preset delay condition among the plurality of device combinations. The use of this method can improve the inference efficiency of large language models.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for deploying inference services for a large language model. Background Art

[0002] With the rapid development of artificial intelligence (AI), large language models (LLMs), a major breakthrough in natural language processing (NLP), are gradually transforming the application landscape in areas such as human-computer interaction, content generation, and knowledge reasoning. Trained on massive amounts of text data, large language models possess superior language understanding capabilities during the reasoning phase and can be flexibly applied in a variety of fields, including creative writing, text summarization, conversational question-answering, and code generation.

[0003] However, as large language models continue to scale, deploying large language model inference services faces increasing challenges in terms of computational complexity and resource consumption. In particular, in real-world applications, device allocation significantly impacts the overall performance of large language model inference services. Therefore, a method for deploying inference services that improves the efficiency of large language model inference is urgently needed. Summary of the Invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for deploying inference services for a large language model to address the above technical issues, which can improve the inference efficiency of large language models.

[0005] In a first aspect, the present application provides a method for deploying an inference service for a large language model, including:

[0006] Obtaining the number of parameters of the large language model, the memory capacity, the number of computing cores, and network information of each of the multiple devices to be allocated, and determining multiple device combinations of the multiple devices to be allocated, each device combination consisting of a first device for the pre-population phase and a second device for the decoding phase;

[0007] For each device combination, determine the text unit processing capability of the current device combination based on the number of parameters, the video memory capacity of the first device in the current device combination, the video memory capacity of the second device in the current device combination, and the space occupied by the intermediate hidden layer result corresponding to a single input text unit;

[0008] Determine the inference latency for the current device combination based on the text unit processing capability for the current device combination, the number of computing cores for the first device for the current device combination, the number of computing cores for the second device for the current device combination, and network information for each of the multiple devices to be assigned.

[0009] A target device combination whose inference delay meets a preset delay condition is determined among multiple device combinations, the first device under the target device combination is used in a pre-filling stage of inference of a large language model, and the second device under the target device combination is used in a decoding stage of inference of the large language model.

[0010] In one embodiment, determining the inference latency for the current device combination based on the text unit processing capability for the current device combination, the number of computing cores of the first device for the current device combination, the number of computing cores of the second device for the current device combination, and network information of each of the plurality of devices to be assigned includes:

[0011] Determining a prefill delay for the current device combination based on a text unit processing capability for the current device combination and a number of computing cores of the first device for the current device combination;

[0012] Determining the key-value cache delay for the current device combination based on the text unit processing capability for the current device combination, the capacity occupied by the key-value cache, the number of computing cores of the first device for the current device combination, and network information of each of the multiple devices to be assigned;

[0013] Determine the decoding delay for the current device combination based on the number of computing cores, data load, memory bandwidth, and decoding calculation time corresponding to a single input text unit of the second device for the current device combination;

[0014] The sum of the pre-fill latency, key-value cache latency, and decoding latency is divided by the text unit processing capability to obtain the inference latency for the current device combination.

[0015] In one embodiment, determining the pre-fill delay under the current device combination based on the text unit processing capability under the current device combination and the number of computing cores of the first device under the current device combination includes:

[0016] Get the pre-filled computing time corresponding to each computing core in the first device in the current device combination;

[0017] The prefill delay under the current device combination is obtained by multiplying the quotient of the text unit processing capability under the current device combination and the number of computing cores of the first device under the current device combination by the prefill calculation time corresponding to each computing core.

[0018] In one embodiment, there are multiple first devices in the current device combination; determining the key-value cache delay in the current device combination based on the text unit processing capability of the current device combination, the capacity occupied by the key-value cache, the number of computing cores of the first device in the current device combination, and network information of each of the multiple devices to be assigned includes:

[0019] Determining a data load of each of the plurality of first devices in the current device combination based on the number of computing cores of each of the plurality of first devices in the current device combination and the text unit processing capability of the current device combination;

[0020] For each first device in the current device combination, obtain the sum of the bandwidths of all data transmission links sent from the first device; determine the key-value cache transmission duration corresponding to the first device based on the data load, the sum of the bandwidth, and the capacity occupied by the key-value cache of the first device;

[0021] The maximum value of the key-value cache transmission durations corresponding to the multiple first devices in the current device combination is used as the key-value cache delay in the current device combination.

[0022] In one embodiment, there are multiple second devices in the current device combination; determining the decoding delay in the current device combination based on the number of computing cores, data load, video memory bandwidth, and decoding calculation time corresponding to a single input text unit of the second devices in the current device combination includes:

[0023] For each second device in the current device combination, obtain the predicted number of output text units, and multiply the number of output text units by the decoding calculation duration as the output text calculation duration corresponding to the second device;

[0024] Determining the data read and write time corresponding to the second device based on the number of output text units, the number of computing cores of the second device, the data load, and the video memory bandwidth; and taking the sum of the data read and write time and the output text calculation time as the decoding delay corresponding to the second device;

[0025] The maximum value of the decoding delays corresponding to the plurality of second devices in the current device combination is used as the decoding delay in the current device combination.

[0026] In one embodiment, determining the text unit processing capability of the current device combination based on the number of parameters, the video memory capacity of the first device in the current device combination, the video memory capacity of the second device in the current device combination, and the space occupied by the intermediate hidden layer result corresponding to a single input text unit includes:

[0027] Obtaining a first difference between the video memory capacity and the number of parameters of a first device in the current device combination, and a second difference between the video memory capacity and the number of parameters of a second device in the current device combination;

[0028] The minimum value of the quotient between the first difference and the placeholder space and the quotient between the second difference and the placeholder space is used as the text unit processing capability of the current device combination.

[0029] In a second aspect, the present application also provides an inference service deployment device for a large language model, including:

[0030] an acquisition module, configured to obtain the number of parameters of the large language model, the memory capacity, the number of computing cores, and network information of each of the multiple devices to be allocated, and determine a device combination of the multiple devices to be allocated, each device combination consisting of a first device for the pre-filling phase and a second device for the decoding phase;

[0031] A first determination module is configured to determine, for each device combination, a text unit processing capability of the current device combination based on the number of parameters, the video memory capacity of the first device in the current device combination, the video memory capacity of the second device in the current device combination, and the space occupied by the intermediate hidden layer result corresponding to the single input text unit;

[0032] A second determination module is configured to determine an inference delay under the current device combination based on a text unit processing capability under the current device combination, a number of computing cores of the first device under the current device combination, a number of computing cores of the second device under the current device combination, and network information of each of the plurality of devices to be allocated;

[0033] The third determination module is used to determine a target device combination whose inference delay meets the preset delay condition among multiple device combinations, the first device under the target device combination is used for the pre-filling stage of inference of the large language model, and the second device under the target device combination is used for the decoding stage of inference of the large language model.

[0034] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Obtaining the number of parameters of the large language model, the memory capacity, the number of computing cores, and network information of each of the multiple devices to be allocated, and determining multiple device combinations of the multiple devices to be allocated, each device combination consisting of a first device for the pre-population phase and a second device for the decoding phase;

[0036] For each device combination, determine the text unit processing capability of the current device combination based on the number of parameters, the video memory capacity of the first device in the current device combination, the video memory capacity of the second device in the current device combination, and the space occupied by the intermediate hidden layer result corresponding to a single input text unit;

[0037] Determine the inference latency for the current device combination based on the text unit processing capability for the current device combination, the number of computing cores for the first device for the current device combination, the number of computing cores for the second device for the current device combination, and network information for each of the multiple devices to be assigned.

[0038] A target device combination whose inference delay meets a preset delay condition is determined among multiple device combinations, the first device under the target device combination is used in a pre-filling stage of inference of a large language model, and the second device under the target device combination is used in a decoding stage of inference of the large language model.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0040] Obtaining the number of parameters of the large language model, the memory capacity, the number of computing cores, and network information of each of the multiple devices to be allocated, and determining multiple device combinations of the multiple devices to be allocated, each device combination consisting of a first device for the pre-population phase and a second device for the decoding phase;

[0041] For each device combination, determine the text unit processing capability of the current device combination based on the number of parameters, the video memory capacity of the first device in the current device combination, the video memory capacity of the second device in the current device combination, and the space occupied by the intermediate hidden layer result corresponding to a single input text unit;

[0042] Determine the inference latency for the current device combination based on the text unit processing capability for the current device combination, the number of computing cores for the first device for the current device combination, the number of computing cores for the second device for the current device combination, and network information for each of the multiple devices to be assigned.

[0043] A target device combination whose inference delay meets a preset delay condition is determined among multiple device combinations, the first device under the target device combination is used in a pre-filling stage of inference of a large language model, and the second device under the target device combination is used in a decoding stage of inference of the large language model.

[0044] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0045] Obtaining the number of parameters of the large language model, the memory capacity, the number of computing cores, and network information of each of the multiple devices to be allocated, and determining multiple device combinations of the multiple devices to be allocated, each device combination consisting of a first device for the pre-population phase and a second device for the decoding phase;

[0046] For each device combination, determine the text unit processing capability of the current device combination based on the number of parameters, the video memory capacity of the first device in the current device combination, the video memory capacity of the second device in the current device combination, and the space occupied by the intermediate hidden layer result corresponding to a single input text unit;

[0047] Determine the inference latency for the current device combination based on the text unit processing capability for the current device combination, the number of computing cores for the first device for the current device combination, the number of computing cores for the second device for the current device combination, and network information for each of the multiple devices to be assigned.

[0048] A target device combination whose inference delay meets a preset delay condition is determined among multiple device combinations, the first device under the target device combination is used in a pre-filling stage of inference of a large language model, and the second device under the target device combination is used in a decoding stage of inference of the large language model.

[0049] The above-mentioned large language model inference service deployment method, apparatus, computer device, computer-readable storage medium and computer program product obtain the number of parameters of the large language model, the video memory capacity, number of computing cores and network information of each of the multiple devices to be allocated, and determine multiple device combinations for the pre-filling stage and the decoding stage. For each device combination, first, based on the number of parameters of the large language model, the video memory capacity of each device and the space occupied by the intermediate hidden layer result corresponding to a single input text unit, the text unit processing capability of the current device combination is determined. The text unit processing capability is used to represent the number of text units that the current device combination can process. Subsequently, based on the text unit processing capability of the current device combination, the number of computing cores of each device to be allocated and the network information, the inference delay of the current device combination is determined. Since the determination of the inference delay integrates the video memory capacity, number of computing cores, network information and number of parameters of the multiple devices to be allocated, the inference delay has a high accuracy. Finally, a target device combination whose inference delay meets the preset delay condition is determined, and the first device under the target device combination is allocated to the pre-filling stage of the large language model inference, and the second device under the target device combination is allocated to the decoding stage of the large language model inference, which is conducive to improving the inference efficiency of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 This is a diagram of an application environment for a method for deploying an inference service for a large language model in one embodiment;

[0052] Figure 2 A flowchart of a method for deploying an inference service for a large language model in one embodiment;

[0053] Figure 3 A schematic diagram of data communication between multiple devices to be allocated in one embodiment;

[0054] Figure 4 Schematic diagram of the overall process of a method for deploying an inference service for a large language model in one embodiment;

[0055] Figure 5 A structural block diagram of an apparatus for deploying an inference service for a large language model in one embodiment;

[0056] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] The inference service deployment method of the large language model provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. The data storage system can store data that server 104 needs to process. The data storage system can be integrated on server 104, or placed on a cloud or other network server. This embodiment uses the method applied to terminal 102 as an example. It is understandable that the method can also be applied to server 104, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. Terminal 102 obtains the number of parameters of the large language model, the video memory capacity, the number of computing cores and network information of each of the multiple devices to be allocated, and determines a device combination of multiple devices to be allocated, each device combination consisting of a first device for the pre-filling stage and a second device for the decoding stage; for each device combination, based on the number of parameters, the video memory capacity of the first device under the current device combination, the video memory capacity of the second device under the current device combination and the space occupied by the intermediate hidden layer result corresponding to a single input text unit, the text unit processing capability under the current device combination is determined; based on the text unit processing capability under the current device combination, the number of computing cores of the first device under the current device combination, the number of computing cores of the second device under the current device combination, and the network information of each of the multiple devices to be allocated, the inference delay under the current device combination is determined; a target device combination whose inference delay meets the preset delay condition is determined among the multiple device combinations, the first device under the target device combination is used for the pre-filling stage of inference of the large language model, and the second device under the target device combination is used for the decoding stage of inference of the large language model. Terminal 102 may include, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car devices, and projectors. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Head-mounted devices may include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, and the like. Server 104 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0059] In an exemplary embodiment, Figure 2 As shown in the figure, a method for deploying inference services for a large language model is provided. Figure 1 The terminal 102 in the example is used as an example to illustrate the process, including the following steps 202 to 208. Among them:

[0060] Step 202 , obtain the number of parameters of the large language model, the video memory capacity, the number of computing cores, and network information of each of the multiple devices to be allocated, and determine a device combination of the multiple devices to be allocated, each device combination consisting of a first device for the pre-filling stage and a second device for the decoding stage.

[0061] A large language model refers to a deep learning model trained using large amounts of text data. It has the ability to generate and understand natural language text. The number of parameters refers to the number of parameters in a large language model.

[0062] Devices to be allocated refer to devices required for large language model reasoning, including high-performance GPUs, large-capacity memories, high-speed storage devices, and network devices supporting high bandwidth, etc. The reasoning of large language models can be carried out in a heterogeneous cluster, using different types of devices to be allocated to form a cluster system, thereby providing multiple types of computing resources for large language model reasoning to improve model reasoning efficiency and performance. Different types of devices to be allocated in a heterogeneous cluster refer to devices of the same or different models provided by the same or different manufacturers. Exemplarily, the first device and the second device can adopt devices of the same or different models provided by different manufacturers, or devices of the same or different models provided by the same manufacturer, for example, the first device is a chip of model A, and the second device is a chip of model B. Similarly, multiple first devices and multiple second devices can be provided by the same or different manufacturers, or have devices of the same or different models.

[0063] Graphics memory capacity refers to the memory capacity of the device to be assigned for storing graphics processing data. Number of compute cores refers to the number of compute cores in the device to be assigned. Network information refers to network-related information about the device to be assigned, such as network bandwidth and speed.

[0064] Device combination refers to the method of arranging and combining the devices to be assigned into multiple clusters. Since the large language model inference process mainly includes the prefill stage and the decoding stage, the multiple devices to be assigned can be split into two clusters, including the cluster for the prefill stage. , clustering for the decoding phase The first device refers to the cluster The second device in the cluster is the device to be allocated. Devices to be allocated in .

[0065] By splitting and reorganizing multiple devices to be allocated, multiple device combinations can be determined. For example, there are N devices to be allocated, which are respectively ,......, ,equipment With equipment The communication bandwidth between .Will ,......, Dividing the device clusters into two groups can obtain multiple device combinations, so that the prefill and decode operations of the large language model can be performed on these two groups of device clusters respectively, so as to achieve better inference results. For example, the number of device combinations can be ,in, Indicates the number of combinations of x devices selected from N devices to be allocated.

[0066] Step 204: For each device combination, determine the text unit processing capability of the current device combination based on the number of parameters, the video memory capacity of the first device in the current device combination, the video memory capacity of the second device in the current device combination, and the space occupied by the intermediate hidden layer result corresponding to a single input text unit.

[0067] An input text unit refers to a unit of text input to a large language model, such as a token. When a large language model performs inference based on input text, it often needs to convert the input text into input text units, such as words, vocabularies, or characters.

[0068] Intermediate hidden layer results refer to the intermediate results of the large language model's processing of each input text unit, and can be represented as vectors. Each input text unit is calculated by each hidden layer in the large language model, resulting in an intermediate result. Multiple hidden layer calculations can yield multiple intermediate hidden layer results. The space occupied by these intermediate hidden layer results after processing an input text unit by the large language model is defined as the total space occupied by the multiple intermediate hidden layer results.

[0069] The text unit processing capability is used to characterize the maximum number of input text units that the current device combination can process. In some embodiments, the processing capabilities of the same device to be assigned for different input text units are similar, while the processing capabilities of different devices to be assigned for the same input text unit are different. Therefore, the text unit processing capability can be determined for each device combination. The text unit processing capability under each device combination can be determined based on the number of parameters of the large language model, the memory capacity of each device to be assigned, and the space occupied by the intermediate hidden layer result corresponding to a single input text unit.

[0070] Step 206, based on the text unit processing capability under the current device combination, the number of computing cores of the first device under the current device combination, the number of computing cores of the second device under the current device combination, and the network information of each of the multiple devices to be allocated, determine the inference delay under the current device combination.

[0071] Inference latency refers to the time spent inferring large language models. Based on the processing stages, it can be divided into pre-population latency, key-value cache latency, and decoding latency. The inference latency for the current device combination can be determined using a preset calculation formula based on the text unit processing capabilities of the current device combination, the number of computing cores on the first device in the current device combination, the number of computing cores on the second device in the current device combination, and the network information of the multiple devices to be assigned.

[0072] Step 208: Determine a target device combination whose inference delay meets the preset delay condition among multiple device combinations, the first device under the target device combination is used for the pre-filling stage of inference of the large language model, and the second device under the target device combination is used for the decoding stage of inference of the large language model.

[0073] Among them, since the device storage capacity, computing capacity and communication bandwidth between devices under each device combination are different, the inference delay under each device combination may be different. Figure 3 The figure shows a schematic diagram of data communication between multiple devices to be allocated. Devices 1, 2, and 3 are first devices used in the pre-filling phase, while devices 4, 5, and 6 are second devices used in the decoding phase. Multiple first devices can communicate with each other, multiple second devices can communicate with each other, and first and second devices can communicate with each other across phases.

[0074] The preset delay condition is a condition for screening target device combinations. For example, the device combination with the smallest inference delay can be used as the target device combination, and the device combination with an inference delay less than the preset delay can also be used as the target device combination. The preset delay condition can be determined according to demand. The terminal determines the target device combination that meets the preset delay condition from multiple device combinations, adopts the device allocation plan corresponding to the target device combination, and uses the first device under the target device combination in the pre-inference stage of inference of the large language model, and uses the second device under the target device combination in the decoding stage of inference of the large language model. This helps to reduce the time consumption of inference of the large language model and improve inference efficiency.

[0075] In the above-mentioned large language model inference service deployment method, the number of parameters of the large language model, the video memory capacity, number of computing cores and network information of each of the multiple devices to be allocated are obtained, and multiple device combinations for the pre-filling stage and the decoding stage are determined; for each device combination, first, based on the number of parameters of the large language model, the video memory capacity of each device and the space occupied by the intermediate hidden layer result corresponding to a single input text unit, the text unit processing capability of the current device combination is determined, and the text unit processing capability is used to represent the number of text units that the current device combination can process; then, based on the text unit processing capability of the current device combination, the number of computing cores of each device to be allocated and the network information, the inference delay of the current device combination is determined. Since the determination of the inference delay integrates the video memory capacity, number of computing cores, network information and number of parameters of the multiple devices to be allocated, the inference delay has a high accuracy; finally, the target device combination whose inference delay meets the preset delay condition is determined, and the first device under the target device combination is allocated to the pre-filling stage of the large language model inference, and the second device under the target device combination is allocated to the decoding stage of the large language model inference, which is conducive to improving the inference efficiency of the large language model.

[0076] In an exemplary embodiment, based on the text unit processing capability under the current device combination, the number of computing cores of the first device under the current device combination, the number of computing cores of the second device under the current device combination, and the network information of each of multiple devices to be assigned, the inference delay under the current device combination is determined, including: determining the pre-filling delay under the current device combination based on the text unit processing capability under the current device combination and the number of computing cores of the first device under the current device combination; determining the key-value cache delay under the current device combination based on the text unit processing capability under the current device combination, the capacity occupied by the key-value cache, the number of computing cores of the first device under the current device combination, and the network information of each of multiple devices to be assigned; determining the decoding delay under the current device combination based on the number of computing cores of the second device under the current device combination, data load, video memory bandwidth, and the decoding calculation time corresponding to a single input text unit; and dividing the sum of the pre-filling delay, the key-value cache delay, and the decoding delay by the text unit processing capability to obtain the inference delay under the current device combination.

[0077] Pre-population latency refers to the time spent in the pre-population phase. Because each device in each device combination has different computing capabilities and text unit processing capabilities, the pre-population latency can be calculated using a corresponding formula based on the text unit processing capabilities and the number of computing cores.

[0078] The key-value results calculated by each network layer of a large language model are often stored in a key-value cache (KVCache). The key-value cache can be used to trade space for time, avoid repeated calculations, and improve inference efficiency. The key-value cache latency refers to the time required for KVCache to transmit data between different devices. Due to the different computing capabilities, text unit processing capabilities, and network bandwidths between different devices to be assigned, the time required for KVCache to transmit data between different devices under different device combinations is different. Based on the text unit processing capabilities of the current device combination, the capacity occupied by the key-value cache, the number of computing cores of the first device under the current device combination, and the network information of each of the multiple devices to be assigned, the corresponding calculation formula can be used to calculate the key-value cache latency under the current device combination.

[0079] Decoding computation time refers to the computation time required to complete a decoding operation on the second device for a single input text unit, that is, the computation time of the decoding phase. If multiple network layers are involved in the decoding phase, the decoding computation time is the sum of the computation times of all the network layers in the decoding phase.

[0080] Decoding latency refers to the duration of the decoding phase. Because different devices may vary in the number of compute cores, data load, memory bandwidth, and the decoding computation time for a single input text unit, the decoding duration may vary across different device combinations. The decoding latency for the current device combination can be calculated using the corresponding formula based on the number of compute cores, data load, memory bandwidth, and decoding computation time for a single input text unit of the second device in the current device combination.

[0081] The terminal sums the pre-population latency, key-value cache latency, and decoding latency to obtain the total latency for the current device combination when processing an input text unit that matches the text unit processing capability. Dividing this sum by the text unit processing capability yields the inference latency for the current device combination. The inference latency indicates the inference latency for processing a single input text unit for the current device combination.

[0082] In this embodiment, by respectively calculating the pre-fill delay, key-value cache delay and decoding delay under the current device combination, the inference delay of the current device unit is calculated based on the delay of each stage, and the determined inference delay has high accuracy.

[0083] In an exemplary embodiment, based on the text unit processing capability under the current device combination and the number of computing cores of the first device under the current device combination, the pre-filling delay under the current device combination is determined, including: obtaining the pre-filling calculation time corresponding to each computing core in the first device under the current device combination; multiplying the quotient between the text unit processing capability under the current device combination and the number of computing cores of the first device under the current device combination by the pre-filling calculation time corresponding to each computing core to obtain the pre-filling delay under the current device combination.

[0084] The pre-fill calculation duration refers to the duration required for each computing core in the first device to perform calculations on a single input text unit during the pre-fill phase. In some embodiments, the pre-fill phase includes multiple network layers, and the pre-fill calculation duration is the sum of the durations required by the multiple network layers.

[0085] In some embodiments, the calculation formula of the pre-fill delay can be expressed as follows:

[0086]

[0087] in, Indicates the number of computing cores of the first device D in the current device combination, Indicates the pre-filled computing time corresponding to each computing core in the first device in the current device combination. Indicates the text unit processing capability of the current device combination. Indicates the pre-fill delay for the current device combination.

[0088] In this embodiment, the pre-filling calculation time corresponding to each computing core in the first device under the current device combination, the number of computing cores and the pre-filling calculation time are used, and the pre-filling delay calculation formula is used to calculate the pre-filling delay under the current device combination, which is conducive to improving the accuracy of the pre-filling delay.

[0089] In an exemplary embodiment, there are multiple first devices under the current device combination; based on the text unit processing capability under the current device combination, the capacity occupied by the key-value cache, the number of computing cores of the first device under the current device combination, and the network information of each of the multiple devices to be allocated, the key-value cache delay under the current device combination is determined, including: based on the number of computing cores of each of the multiple first devices under the current device combination and the text unit processing capability under the current device combination, the data load of each of the multiple first devices under the current device combination is determined; for each first device under the current device combination, the sum of the bandwidths of all data transmission links sent from the first device is obtained; according to the data load, the sum of the bandwidths, and the capacity occupied by the key-value cache of the first device, the key-value cache transmission duration corresponding to the first device is determined; and the maximum value of the key-value cache transmission duration corresponding to each of the multiple first devices under the current device combination is used as the key-value cache delay under the current device combination.

[0090] The data load refers to the number of input text units allocated to a single first device. The data load of each first device is distributed according to the number of computing cores. The data load of each first device can be calculated based on the number of computing cores and the text unit processing capability.

[0091] For example, the input text units are distributed to different devices according to the proportional relationship between the number of computing cores. The terminal divides the number of computing cores of each first device in the current device combination by the sum of the number of computing cores of multiple first devices in the current device combination, and multiplies the obtained quotient by the text unit processing capacity of the current device combination to obtain the data load of each of the multiple first devices.

[0092] refer to Figure 3 KvCache is transmitted between devices, and the two devices transmitting KvCache form a data transmission link. Taking the first device as the device that outputs KvCache as an example, the terminal obtains the bandwidth of all data transmission links sent from the first device, and sums the bandwidth of all data transmission links sent from the first device to obtain the total bandwidth.

[0093] The key-value cache capacity refers to the space occupied by the key-value cache in video memory. The key-value cache transmission duration refers to the time it takes for the KvCache sent from the first device to be transmitted over the data transmission link. The terminal multiplies the quotient of the first device's data load and the total bandwidth by the key-value cache capacity to obtain the key-value cache transmission duration corresponding to the first device.

[0094] The maximum value of the key-value cache transmission durations of the multiple first devices in the current device combination is used as the key-value cache delay in the current device combination. The key-value cache delay indicates the maximum key-value cache transmission duration in the current device combination.

[0095] In some embodiments, the calculation formula for the key-value cache delay may be:

[0096]

[0097] in, Indicates any first device in the current device combination, Representation device The number of computing cores, Indicates slave device The sum of the bandwidth of all data transmission links sent, Indicates the capacity of the key-value cache. Indicates the key-value cache latency under the current device combination.

[0098] Among them, due to Input text units are distributed to different devices in proportion to the number of computing cores, so Representation device data load.

[0099] In this embodiment, the data load of each of the multiple first devices under the current device combination is calculated based on the number of computing cores of each of the multiple first devices under the current device combination and the text unit processing capability of the current device combination. The key-value cache transmission duration is determined based on the sum of the bandwidths of all data transmission links emitted from each first device under the current device combination, the data load of the first device, and the capacity occupied by the key-value cache. The maximum value of the key-value cache transmission durations among the multiple first devices is used as the key-value cache delay under the current device combination, which is conducive to determining an accurate key-value cache delay.

[0100] In an exemplary embodiment, there are multiple second devices in the current device combination; based on the number of computing cores, data load, memory bandwidth and decoding calculation time corresponding to a single input text unit of the second device in the current device combination, the decoding delay in the current device combination is determined, including: for each second device in the current device combination, obtaining the predicted number of output text units, and taking the product of the number of output text units and the decoding calculation time as the output text calculation time corresponding to the second device; based on the number of output text units, the number of computing cores, data load and memory bandwidth of the second device, determining the data reading and writing time corresponding to the second device; taking the sum of the data reading and writing time and the output text calculation time as the decoding delay corresponding to the second device; and taking the maximum value of the decoding delays corresponding to the multiple second devices in the current device combination as the decoding delay in the current device combination.

[0101] The output text unit refers to the text unit output by the large language model. In some embodiments, the terminal may use an output text length predictor to make predictions. The output text length predictor may be a pre-trained prediction model that predicts the number of output text units based on an input prompt.

[0102] Output text computation time refers to the computation time required during the decoding phase when the large language model outputs text units that match the number of output text units. The terminal calculates the output text computation time by multiplying the number of output text units by the decoding computation time.

[0103] Data read / write time refers to the time it takes for a second device to read or write data transferred to the current device. This time is related to the number of output text units, the number of computing cores, the data load, and the video memory bandwidth.

[0104] The terminal uses the maximum value of the decoding delays of the plurality of second devices in the current device combination as the decoding delay in the current device combination. The decoding delay indicates the maximum decoding operation time in the current device combination.

[0105] In some embodiments, the calculation formula for decoding delay can be expressed as follows:

[0106]

[0107] in, Indicates the memory bandwidth, which can be calculated as "memory frequency Memory width / 8", Indicates the decoding calculation time corresponding to a single input text unit. represents the number of predicted output text units, Representation device The data load can be obtained by accumulating the transmission load of the pre-filled devices connected to it. Indicates decoding delay.

[0108] In this embodiment, for each second device in the current device combination, the output text calculation time and the data reading and writing time are determined respectively. The sum of the two is the decoding delay corresponding to each second device. The maximum decoding delay is used as the decoding delay in the current device combination, which is conducive to determining the accurate decoding delay.

[0109] In an exemplary embodiment, the text unit processing capability under the current device combination is determined based on the number of parameters, the video memory capacity of the first device under the current device combination, the video memory capacity of the second device under the current device combination, and the placeholder space of the intermediate hidden layer result corresponding to a single input text unit, including: obtaining a first difference between the video memory capacity of the first device under the current device combination and the number of parameters, and a second difference between the video memory capacity of the second device under the current device combination and the number of parameters; and taking the minimum value of the quotient between the first difference and the placeholder space and the quotient between the second difference and the placeholder space as the text unit processing capability under the current device combination.

[0110] The first difference is the difference between the video memory capacity of the first device in the current device combination and the number of parameters, indicating the space available in the video memory of the first device for processing input text units. The quotient between the first difference and the occupied space is the text unit processing capability of the first device in the current device combination, indicating the number of input text units that the first device can process in the current device combination.

[0111] The second difference, the difference between the video memory capacity of the second device in the current device combination and the number of parameters, represents the space available in the video memory of the second device for processing input text units. The quotient between the second difference and the occupied space represents the text unit processing capability of the second device in the current device combination, indicating the number of input text units that the second device can process in the current device combination.

[0112] The terminal uses the minimum value of the text unit processing capabilities of the first device and the second device in the current device combination as the text unit processing capability of the current device combination. The text unit processing capability of the current device combination indicates the minimum number of input text units that can be processed by the current device combination.

[0113] In some embodiments, the calculation formula for the text unit processing capability under the current device combination can be expressed as follows:

[0114]

[0115] in, Indicates the video memory capacity of device D. represents the number of parameters of the large language model, Represents the placeholder for the intermediate hidden layer result corresponding to a single input text unit, Indicates the text unit processing capability of the current device combination.

[0116] In this embodiment, the text unit processing capability under the current device combination is determined by calculating the minimum value of the text unit processing capability of the first device and the text unit processing capability of the second device under the current device combination. Based on the text unit processing capabilities of multiple device combinations, it is helpful to determine a suitable target device combination for service deployment of large language model inference.

[0117] To illustrate the deployment method and effect of the inference service for a large language model in this solution, the following is a detailed example:

[0118] Given the storage information, computing power, and communication bandwidth of each device to be allocated in the heterogeneous cluster, allocate respective execution devices for the prefill phase and the decode phase. For example, there are N devices to be allocated, which are denoted as ,......, ,equipment With equipment The communication bandwidth between . Equipment combination refers to the ,......, Divide into two groups of device clusters, so that the prefill and decode operations of the large language model are performed on these two groups of device clusters respectively, so as to achieve better inference effect. Therefore, the number of device combinations can be ,in, Indicates the number of combinations of x devices selected from N devices to be allocated.

[0119] like Figure 4 The figure shows a schematic diagram of the overall process of the inference service deployment method of a large language model in one embodiment. The terminal obtains the number of parameters of the large language model, the video memory capacity of each of the multiple devices to be allocated, the number of computing cores, and network information, and determines a device combination of multiple devices to be allocated, each device combination consisting of a first device for the pre-filling stage and a second device for the decoding stage. Traverse multiple device combinations, and for the current device combination traversed, when the video memory capacity of the first device is greater than the number of parameters, and the video memory capacity of the second device is greater than the number of parameters, calculate the inference delay corresponding to the current device combination; when the inference delay is less than the preset delay, use the current device combination as the target device combination, and continue to traverse multiple device combinations until the traversal of multiple device combinations is completed, and output the target device combination.

[0120] In this inference process, the latency of computing a batch of data is considered, that is, the entire batch of data is computed after the prefill phase is completed before the next phase of computation is performed. Pipeline parallelism and tensor parallelism are ignored, and only data parallelism is considered. Given a device combination, let the cluster in the prefill phase be , the cluster in the decoding stage is , the set of all data transmission links between two clusters is The main performance considerations include the following:

[0121] 1. Text unit processing capabilities:

[0122]

[0123] in, Indicates the video memory capacity of device D. represents the number of parameters of the large language model, Represents the placeholder for the intermediate hidden layer result corresponding to a single input text unit, Indicates the text unit processing capability of the current device combination.

[0124] 2. Prefill delay:

[0125]

[0126] in, Indicates the number of computing cores of the first device D in the current device combination, Indicates the pre-filled computing time corresponding to each computing core in the first device in the current device combination. Indicates the text unit processing capability of the current device combination. Indicates the pre-fill delay for the current device combination.

[0127] 3. Key-value cache delay:

[0128]

[0129] in, Indicates any first device in the current device combination, Representation device The number of computing cores, Indicates slave device The sum of the bandwidth of all data transmission links sent, Indicates the capacity of the key-value cache. Indicates the key-value cache latency under the current device combination.

[0130] Among them, due to Input text units are distributed to different devices in proportion to the number of computing cores, so Representation device data load.

[0131] 4. Decoding delay:

[0132]

[0133] in, Indicates the memory bandwidth, which can be calculated as "memory frequency Memory width / 8", Indicates the decoding calculation time corresponding to a single input text unit. represents the number of predicted output text units, Representation device The data load can be obtained by accumulating the transmission load of the pre-filled devices connected to it. Indicates decoding delay.

[0134] Determine the target device combination based on various performance indicators:

[0135]

[0136] Where L represents the inference delay.

[0137] A target device combination, determined from among the multiple device combinations, whose inference latency meets a preset latency condition, for example, a device combination whose inference latency is less than the preset latency, is selected as the target device combination. The first device in the target device combination is used in the pre-population phase of inference for the large language model, and the second device in the target device combination is used in the decoding phase of inference for the large language model.

[0138] The above-mentioned inference service deployment method for a large language model obtains the number of parameters of the large language model, the video memory capacity, the number of computing cores, and network information of each of the multiple devices to be allocated, and determines multiple device combinations for the pre-filling stage and the decoding stage. For each device combination, first, based on the number of parameters of the large language model, the video memory capacity of each device, and the space occupied by the intermediate hidden layer results corresponding to a single input text unit, the text unit processing capability of the current device combination is determined. The text unit processing capability is used to represent the number of text units that the current device combination can process. Subsequently, based on the text unit processing capability of the current device combination, the number of computing cores of each device to be allocated, and the network information, the inference delay of the current device combination is determined. Since the determination of the inference delay integrates the video memory capacity, the number of computing cores, the network information of the multiple devices to be allocated, and the number of parameters of the large language model, the inference delay has a high accuracy. Finally, a target device combination whose inference delay meets the preset delay condition is determined, and the first device under the target device combination is allocated to the pre-filling stage of the large language model inference, and the second device under the target device combination is allocated to the decoding stage of the large language model inference, which is conducive to improving the inference efficiency of the large language model.

[0139] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0140] Based on the same inventive concept, embodiments of the present application also provide a large language model inference service deployment device for implementing the aforementioned large language model inference service deployment method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the one or more embodiments of the large language model inference service deployment device provided below can be found in the limitations of the large language model inference service deployment method described above and will not be repeated here.

[0141] In an exemplary embodiment, Figure 5As shown, a large language model reasoning service deployment device 500 is provided, including: an acquisition module 520, a first determination module 540, a second determination module 560 and a third determination module 580, wherein:

[0142] an acquisition module 520 for acquiring the number of parameters of the large language model, the memory capacity, the number of computing cores, and network information of each of the multiple devices to be allocated, and determining a plurality of device combinations to be allocated, each device combination comprising a first device for the pre-population phase and a second device for the decoding phase;

[0143] A first determination module 540 is configured to determine, for each device combination, the text unit processing capability of the current device combination based on the number of parameters, the video memory capacity of the first device in the current device combination, the video memory capacity of the second device in the current device combination, and the space occupied by the intermediate hidden layer result corresponding to a single input text unit;

[0144] A second determination module 560 is configured to determine an inference delay for the current device combination based on a text unit processing capability for the current device combination, a number of computing cores of the first device for the current device combination, a number of computing cores of the second device for the current device combination, and network information of each of the plurality of devices to be assigned;

[0145] The third determination module 580 is used to determine a target device combination whose inference delay meets the preset delay condition among multiple device combinations, the first device under the target device combination is used for the pre-filling stage of inference of the large language model, and the second device under the target device combination is used for the decoding stage of inference of the large language model.

[0146] The above-mentioned inference service deployment device for the large language model obtains the number of parameters of the large language model, the video memory capacity, the number of computing cores and network information of each of the multiple devices to be allocated, and determines multiple device combinations for the pre-filling stage and the decoding stage; for each device combination, first, based on the number of parameters of the large language model, the video memory capacity of each device and the space occupied by the intermediate hidden layer result corresponding to a single input text unit, the text unit processing capability of the current device combination is determined, and the text unit processing capability is used to characterize the number of text units that the current device combination can process; then, based on the text unit processing capability of the current device combination, the number of computing cores of each device to be allocated and the network information, the inference delay of the current device combination is determined. Since the determination of the inference delay integrates the video memory capacity, the number of computing cores, the network information of the multiple devices to be allocated and the number of parameters of the large language model, the inference delay has a high accuracy; finally, the target device combination whose inference delay meets the preset delay condition is determined, and the first device under the target device combination is allocated to the pre-filling stage of the large language model inference, and the second device under the target device combination is allocated to the decoding stage of the large language model inference, which is conducive to improving the inference efficiency of the large language model.

[0147] In one embodiment, the inference delay under the current device combination is determined based on the text unit processing capability under the current device combination, the number of computing cores of the first device under the current device combination, the number of computing cores of the second device under the current device combination, and the network information of each of the multiple devices to be allocated. The second determination module 560 is also used to: determine the pre-filling delay under the current device combination based on the text unit processing capability under the current device combination and the number of computing cores of the first device under the current device combination; determine the key-value cache delay under the current device combination based on the text unit processing capability under the current device combination, the capacity occupied by the key-value cache, the number of computing cores of the first device under the current device combination, and the network information of each of the multiple devices to be allocated; determine the decoding delay under the current device combination based on the number of computing cores of the second device under the current device combination, data load, video memory bandwidth, and the decoding calculation time corresponding to a single input text unit; and divide the sum of the pre-filling delay, key-value cache delay, and decoding delay by the text unit processing capability to obtain the inference delay under the current device combination.

[0148] In one embodiment, the pre-filling delay under the current device combination is determined based on the text unit processing capability under the current device combination and the number of computing cores of the first device under the current device combination. The second determination module 560 is also used to: obtain the pre-filling calculation time corresponding to each computing core in the first device under the current device combination; multiply the quotient between the text unit processing capability under the current device combination and the number of computing cores of the first device under the current device combination by the pre-filling calculation time corresponding to each computing core to obtain the pre-filling delay under the current device combination.

[0149] In one embodiment, there are multiple first devices under the current device combination; based on the text unit processing capability under the current device combination, the capacity occupied by the key-value cache, the number of computing cores of the first device under the current device combination, and the network information of each of the multiple devices to be allocated, the key-value cache delay under the current device combination is determined, and the second determination module 560 is also used to: determine the data load of each of the multiple first devices under the current device combination based on the number of computing cores of each of the multiple first devices under the current device combination and the text unit processing capability under the current device combination; for each first device under the current device combination, obtain the sum of the bandwidths of all data transmission links sent from the first device; determine the key-value cache transmission duration corresponding to the first device based on the data load, the sum of the bandwidths, and the capacity occupied by the key-value cache of the first device; and take the maximum value of the key-value cache transmission duration corresponding to each of the multiple first devices under the current device combination as the key-value cache delay under the current device combination.

[0150] In one embodiment, there are multiple second devices in the current device combination; based on the number of computing cores, data load, memory bandwidth and decoding calculation time corresponding to a single input text unit of the second device in the current device combination, the decoding delay in the current device combination is determined, and the second determination module 560 is also used to: obtain the predicted number of output text units for each second device in the current device combination, and use the product of the number of output text units and the decoding calculation time as the output text calculation time corresponding to the second device; determine the data reading and writing time corresponding to the second device based on the number of output text units, the number of computing cores, data load and memory bandwidth of the second device; use the sum of the data reading and writing time and the output text calculation time as the decoding delay corresponding to the second device; and use the maximum value of the decoding delays corresponding to the multiple second devices in the current device combination as the decoding delay in the current device combination.

[0151] In one embodiment, the text unit processing capability under the current device combination is determined based on the number of parameters, the video memory capacity of the first device under the current device combination, the video memory capacity of the second device under the current device combination, and the placeholder space of the intermediate hidden layer result corresponding to a single input text unit. The first determination module 540 is also used to: obtain a first difference between the video memory capacity of the first device under the current device combination and the number of parameters, and a second difference between the video memory capacity of the second device under the current device combination and the number of parameters; and take the minimum value of the quotient between the first difference and the placeholder space, and the minimum value of the quotient between the second difference and the placeholder space as the text unit processing capability under the current device combination.

[0152] Each module in the aforementioned large language model inference service deployment device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0153] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, mobile cellular networks, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements a method for deploying an inference service for a large language model. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0154] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0155] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0156] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0157] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0158] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0159] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0160] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0161] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for deploying an inference service for a large language model, characterized in that: The method comprises: Obtaining the number of parameters of the large language model, the memory capacity, the number of computing cores, and network information of each of the multiple devices to be allocated, and determining multiple device combinations of the multiple devices to be allocated, each device combination consisting of a first device for the pre-population phase and a second device for the decoding phase; For each device combination, determine the text unit processing capability of the current device combination based on the number of parameters, the video memory capacity of the first device in the current device combination, the video memory capacity of the second device in the current device combination, and the space occupied by the intermediate hidden layer result corresponding to a single input text unit; the text unit processing capability is used to represent the maximum number of input text units that the current device combination can process; Determining an inference delay for the current device combination based on a text unit processing capability for the current device combination, a number of computing cores for a first device for the current device combination, a number of computing cores for a second device for the current device combination, and network information for each of a plurality of devices to be assigned; the inference delay indicating an inference delay for processing a single input text unit for the current device combination; Determining a target device combination whose inference delay meets a preset delay condition from among multiple device combinations, wherein a first device in the target device combination is used for a pre-filling phase of inference of the large language model, and a second device in the target device combination is used for a decoding phase of inference of the large language model; The determining the inference delay under the current device combination based on the text unit processing capability under the current device combination, the number of computing cores of the first device under the current device combination, the number of computing cores of the second device under the current device combination, and network information of each of the multiple devices to be allocated includes: Determining a prefill delay for the current device combination based on a text unit processing capability for the current device combination and a number of computing cores of the first device for the current device combination; Determining a key-value cache delay for the current device combination based on a text unit processing capability for the current device combination, a capacity occupied by a key-value cache, a number of computing cores of a first device for the current device combination, and network information of each of a plurality of devices to be assigned; Determining a decoding delay for the current device combination based on the number of computing cores, data load, video memory bandwidth, and decoding calculation duration corresponding to a single input text unit of the second device for the current device combination; The inference delay under the current device combination is obtained by dividing the sum of the pre-filling delay, the key-value cache delay, and the decoding delay by the text unit processing capability.

2. The method according to claim 1, characterized in that The determining the pre-fill delay under the current device combination based on the text unit processing capability under the current device combination and the number of computing cores of the first device under the current device combination includes: Obtaining the pre-filled computing time corresponding to each computing core in the first device under the current device combination; The quotient between the text unit processing capability under the current device combination and the number of computing cores of the first device under the current device combination is multiplied by the pre-filling calculation time corresponding to each computing core to obtain the pre-filling delay under the current device combination.

3. The method according to claim 1, characterized in that There are multiple first devices in the current device combination; determining the key-value cache delay in the current device combination based on the text unit processing capability of the current device combination, the capacity occupied by the key-value cache, the number of computing cores of the first device in the current device combination, and the network information of each of the multiple devices to be allocated includes: determining a data load of each of the plurality of first devices in the current device combination based on the number of computing cores of each of the plurality of first devices in the current device combination and a text unit processing capability of the current device combination; For each first device in the current device combination, obtain the sum of bandwidths of all data transmission links sent from the first device; determine the key-value cache transmission duration corresponding to the first device based on the data load of the first device, the sum of bandwidths, and the capacity occupied by the key-value cache; The maximum value of the key-value cache transmission durations corresponding to the multiple first devices in the current device combination is used as the key-value cache delay in the current device combination.

4. The method according to claim 1, wherein There are multiple second devices in the current device combination; and determining the decoding delay in the current device combination based on the number of computing cores, data load, video memory bandwidth, and decoding calculation time corresponding to a single input text unit of the second devices in the current device combination includes: For each second device in the current device combination, obtaining a predicted number of output text units, and multiplying the number of output text units by the decoding calculation duration as the output text calculation duration corresponding to the second device; Determining data read and write time corresponding to the second device based on the number of output text units, the number of computing cores of the second device, data load, and video memory bandwidth; and using the sum of the data read and write time and the output text calculation time as the decoding delay corresponding to the second device; The maximum value of the decoding delays corresponding to the plurality of second devices in the current device combination is used as the decoding delay in the current device combination.

5. The method according to claim 1, characterized in that The determining, based on the number of parameters, the video memory capacity of the first device in the current device combination, the video memory capacity of the second device in the current device combination, and the occupied space of the intermediate hidden layer result corresponding to the single input text unit, of the text unit processing capability of the current device combination includes: Obtaining a first difference between the video memory capacity of a first device in a current device combination and the parameter quantity, and a second difference between the video memory capacity of a second device in the current device combination and the parameter quantity; The minimum value of the quotient between the first difference and the placeholder space and the quotient between the second difference and the placeholder space is used as the text unit processing capability of the current device combination.

6. A large language model reasoning service deployment device, characterized in that: The device comprises: an acquisition module, configured to obtain the number of parameters of the large language model, the memory capacity, the number of computing cores, and network information of each of the multiple devices to be allocated, and determine a device combination of the multiple devices to be allocated, each device combination consisting of a first device for the pre-filling phase and a second device for the decoding phase; a first determination module configured to determine, for each device combination, a text unit processing capability of the current device combination based on the number of parameters, a video memory capacity of the first device in the current device combination, a video memory capacity of the second device in the current device combination, and a space occupied by an intermediate hidden layer result corresponding to a single input text unit; the text unit processing capability being used to represent a maximum number of input text units that the current device combination can process; a second determining module, configured to determine an inference delay under the current device combination based on a text unit processing capability under the current device combination, a number of computing cores of a first device under the current device combination, a number of computing cores of a second device under the current device combination, and network information of each of a plurality of devices to be assigned; the inference delay indicating an inference delay for processing a single input text unit under the current device combination; a third determining module, configured to determine, from a plurality of device combinations, a target device combination whose inference delay satisfies a preset delay condition, wherein a first device in the target device combination is used in a pre-filling phase of inference performed by the large language model, and a second device in the target device combination is used in a decoding phase of inference performed by the large language model; The second determination module is also used to determine the pre-filling delay under the current device combination based on the text unit processing capability under the current device combination and the number of computing cores of the first device under the current device combination; determine the key-value cache delay under the current device combination based on the text unit processing capability under the current device combination, the capacity occupied by the key-value cache, the number of computing cores of the first device under the current device combination, and the network information of multiple devices to be allocated; determine the decoding delay under the current device combination based on the number of computing cores, data load, video memory bandwidth, and decoding calculation time corresponding to a single input text unit of the second device under the current device combination; and obtain the inference delay under the current device combination by dividing the sum of the pre-filling delay, the key-value cache delay, and the decoding delay by the text unit processing capability.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images