Inference service method, processing device, equipment, storage medium and program product

Abstract

The invention provides a reasoning service method, a processing device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence. The large model reasoning service method comprises the following steps: monitoring first computing power resources of a plurality of decoding nodes and monitoring performance indexes of migration links between pre-filling nodes and the decoding nodes; in response to the large model reasoning task, distributing the reasoning task to a plurality of pre-filling nodes, and generating corresponding key value cache KV Cache data; performing fragmentation processing on the KV Cache data based on the first computing power resource and / or the performance index to obtain KV Cache fragments; and migrating the KV Cache fragments to the plurality of decoding nodes, so that the plurality of decoding nodes decode the KV Cache fragments based on a multi-head parallel attention balancing mechanism and / or an assembly line parallel balancing mechanism to obtain a reasoning result. According to the technical scheme, the migration efficiency and migration stability of KV Cache data can be improved, and the reasoning efficiency of a large model reasoning task can be improved in combination with a balance mechanism.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a large model reasoning service method, a large model reasoning service device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Large Language Model (LLM) has demonstrated powerful performance in many fields such as natural language processing. However, with the rapid growth of model parameters, the computing resources and memory required for reasoning calculations are also rising sharply. The distributed reasoning architecture effectively expands computing resources by distributing the model's computing tasks to multiple computing devices (such as multiple GPU servers). However, in actual applications, there is an imbalance in resource and load distribution, which affects the reasoning efficiency and stability of the model.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide a large model reasoning service method, a large model reasoning service device, an electronic device, a computer-readable storage medium and a computer program product, which at least to some extent overcome the problem of low overall energy utilization of the Internet of Things system in the related art.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a large model reasoning service method is provided, including: the large model reasoning service includes multiple pre-filling nodes and multiple decoding nodes, monitors the first computing power resources of the multiple decoding nodes and monitors the performance indicators of the migration link between the pre-filling nodes and the decoding nodes; in response to a large model reasoning task, allocates the reasoning task to multiple pre-filling nodes, so that the multiple pre-filling nodes perform pre-filling operations on the reasoning task in parallel to generate corresponding key-value cache KV Cache data; based on the first computing power resources and / or the performance indicators, the KV Cache data is sliced ​​to obtain KV Cache slices; the KV Cache slices are migrated to the multiple decoding nodes, so that the multiple decoding nodes perform decoding operations on the KV Cache slices based on a multi-head parallel attention balancing mechanism and / or a pipeline parallel balancing mechanism to obtain reasoning results.

[0007] In one embodiment of the present disclosure, the multiple decoding nodes perform decoding operations on the KV Cache shards based on a multi-head parallel attention balancing mechanism and / or a pipeline parallel balancing mechanism to obtain inference results, including: allocating pipeline branches to the multiple decoding nodes based on the decoding operations, the pipeline branches including multiple processing layers, dividing the KV Cache shards based on the number of the processing layers to obtain first sub-shards, wherein the pipeline branches in different decoding nodes are executed concurrently; in each of the processing layers, dividing the first sub-shards based on the number of attention heads in the multi-head parallel attention balancing mechanism to obtain second sub-shards, so as to allocate the second shards to different attention heads for parallel calculation to obtain calculation results; the calculation results obtained by the attention heads are directly input into the next layer of the processing layer for pipeline processing, until the last layer of the processing layer outputs the inference results.

[0008] In one embodiment of the present disclosure, the decoding node includes multiple processing layers along the processing order, and also includes: allocating the multiple processing layers to different decoding nodes, and performing load balancing distribution of the task amount of each processing layer based on the first computing power resources of the decoding node.

[0009] In one embodiment of the present disclosure, allocating the second shards to different attention heads for parallel computing also includes: allocating the computing of different attention heads to different decoding nodes based on the first computing power resources.

[0010] In one embodiment of the present disclosure, the KVCache data is sharded based on the first computing power resource and / or the performance indicator to obtain KV Cache shards, including: normalizing the first computing power resource and the performance indicator respectively to obtain computing power resource parameters and performance indicator parameters; determining a unit shard quantity for sharding the KV Cache data based on the smaller value of the computing power resource parameter and the performance indicator parameter; and sharding the KV Cache data based on the unit shard quantity to obtain the KV Cache shards.

[0011] In one embodiment of the present disclosure, the KVCache data is sharded based on the first computing power resource and / or the performance indicator to obtain KV Cache shards, including: determining the computing power of each of the decoding nodes based on the first computing power resource, and sharding the KV Cache data based on the computing power to obtain the KV Cache shards that match the computing power; or determining the migration capability of each of the migration links based on the performance indicator, and sharding the KV Cache data based on the migration capability to obtain the KV Cache shards that match the migration capability.

[0012] In one embodiment of the present disclosure, migrating the KV Cache shards to the multiple decoding nodes includes: determining priorities of the multiple migration links based on the performance indicators, so as to migrate the KV Cache shards to the multiple decoding nodes based on the priorities.

[0013] In one embodiment of the present disclosure, the priorities of the plurality of migration links are determined based on the performance indicators, including: the performance indicators include network bandwidth, network delay and packet loss rate, and the network quality of different migration links from the plurality of pre-filled nodes to the plurality of decoding nodes is calculated based on the network bandwidth, the network delay and the packet loss rate; and the corresponding priorities are configured based on the network quality.

[0014] In one embodiment of the present disclosure, the network quality of different migration links between the plurality of pre-filled nodes and the plurality of decoding nodes is calculated based on the network bandwidth, the network delay and the packet loss rate, including: based on the quality calculation formula Calculate the network quality, where Bandwidth prefill,decoding is the network bandwidth, Latency prefill,decoding is the network delay, Loss prefill,decoding is the packet loss rate.

[0015] In one embodiment of the present disclosure, migrating the KV Cache shards to the multiple decoding nodes further includes: determining a first KV Cache shard based on the priority of the decoding operation, so as to preferentially migrate the first KV Cache shard to the decoding node.

[0016] In one embodiment of the present disclosure, migrating the KV Cache shards to the multiple decoding nodes also includes: determining a second KV Cache shard based on the degree of dependence of the multi-head parallel attention balancing mechanism on the KV Cache shard, so as to preferentially migrate the second KV Cache shard to the decoding node.

[0017] In one embodiment of the present disclosure, the KV Cache data is sharded based on the unit shard amount to obtain the KV Cache shards, and also includes: sharding the KV Cache data based on the unit shard amount to obtain initial shards; determining the link bandwidth of the migration link based on the performance indicator; constructing an optimization model with the utilization rate of the link bandwidth and the migration time of the KV Cache shards as optimization targets; and optimizing the initial shards based on the optimization model to obtain the KV Cache shards.

[0018] In one embodiment of the present disclosure, migrating the KV Cache shards to the multiple decoding nodes also includes: determining the computing power of each of the decoding nodes based on the first computing power resources; determining the amount of data of the KV Cache shards allocated to the corresponding decoding nodes based on the computing power, so as to migrate the KV Cache shards to the corresponding decoding nodes.

[0019] In one embodiment of the present disclosure, monitoring the performance indicators of the migration link between the pre-filling node and the decoding node includes: scanning the alternative links between the multiple pre-filling nodes and the multiple decoding nodes; selecting the migration link for migrating the KVCache shard from the alternative links based on the network quality of the alternative links and the prediction results of the network trend, so as to monitor the performance indicators of the migration link.

[0020] In one embodiment of the present disclosure, in response to a large model reasoning task, the reasoning task is allocated to the multiple pre-filled nodes, including: monitoring a second computing power resource of the multiple pre-filled nodes; in response to the large model reasoning task, the reasoning task is allocated to the multiple pre-filled nodes based on the monitoring result of the second computing power resource.

[0021] According to another aspect of the present disclosure, a large model reasoning service device is provided, including: a monitoring module, used to monitor the first computing power resources of the multiple decoding nodes and monitor the performance indicators of the migration link between the pre-filling node and the decoding node; an allocation module, used to respond to the large model reasoning task, allocate the reasoning task to the multiple pre-filling nodes, so that the multiple pre-filling nodes perform pre-filling operations on the reasoning task in parallel to generate corresponding key-value cache KV Cache data; a processing module, used to perform sharding processing on the KV Cache data based on the first computing power resources and / or the performance indicators to obtain KV Cache shards; a migration module, used to migrate the KV Cache shards to the multiple decoding nodes, so that the multiple decoding nodes perform decoding operations on the KV Cache shards based on a multi-head parallel attention balancing mechanism and / or a pipeline parallel balancing mechanism to obtain inference results.

[0022] According to another aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; the processor is configured to execute the above-mentioned large model reasoning service method by executing the executable instructions.

[0023] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the large model reasoning service method described above is implemented.

[0024] According to another aspect of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the computer program is executed by a processor, the large model reasoning service method described above is implemented.

[0025] The large model inference service solution provided by the embodiment of the present disclosure monitors the first computing power resources of multiple decoding nodes and the performance indicators of the migration link between the pre-filling node and the decoding node in real time. When there is a large model inference task, the task is first assigned to multiple pre-filling nodes. The pre-filling nodes will perform the pre-filling operation in parallel to generate corresponding KV Cache data. Based on the monitored first computing power resources and / or the performance indicators of the migration link, the generated KV Cache data is dynamically adjusted to have a shard size, which can optimize the transmission efficiency and computing resource utilization. Through the multi-path transmission protocol, the KV Cache shards are migrated to multiple decoding nodes, which can reduce the transmission bottleneck. At the decoding node, the multi-head parallel attention balancing mechanism and / or the pipeline parallel balancing mechanism are used to perform decoding operations on the shards, and finally generate the inference results. By comprehensively considering the computing power resources and link performance, the KV Cache data is reasonably sharded, which can improve the migration efficiency and migration stability of the KV Cache data. With the help of the multi-head parallel attention balancing mechanism and the pipeline parallel balancing mechanism, the decoding processing is performed at the decoding node, which can improve the inference efficiency of the large model inference task.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0028] Figure 1 A schematic diagram of a large model reasoning service system according to an embodiment of the present disclosure is shown;

[0029] Figure 2 A schematic flow chart showing a large model reasoning service method according to an embodiment of the present disclosure;

[0030] Figure 3 A schematic flow chart showing another large model reasoning service method in an embodiment of the present disclosure;

[0031] Figure 4 A schematic flow chart showing another large model reasoning service method in an embodiment of the present disclosure is shown;

[0032] Figure 5 A schematic flow chart showing another large model reasoning service method in an embodiment of the present disclosure;

[0033] Figure 6 A schematic diagram of a large model reasoning service device in an embodiment of the present disclosure is shown;

[0034] Figure 7 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0036] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0037] As the number of parameters in large language models increases rapidly, the demand for inference computing and memory also increases dramatically. Traditional single-device inference methods are difficult to meet real-time requirements. Distributed inference architecture is an important method to improve inference efficiency, but in actual applications, it faces the problem of resource imbalance, including: uneven heterogeneous computing power, uneven network conditions, and uneven load.

[0038] Unbalanced heterogeneous computing power refers to the large differences in computing power between different nodes, making it difficult to evenly distribute computing tasks. Unbalanced network conditions refer to the significant differences in network bandwidth and latency between devices. Directly allocating inference tasks will result in transmission bottlenecks. Load imbalance refers to the inability to achieve uniform load on inference tasks due to differences in computing power and network between devices, resulting in a decrease in the overall efficiency of the system.

[0039] Based on the above defects, the present invention designs an inference architecture that can perceive heterogeneous computing power and network status, and dynamically adjust the allocation of inference tasks to improve the efficiency and stability of inference.

[0040] To facilitate understanding, several terms involved in this application are first explained below.

[0041] Prefill: The process of generating the first output token based on the input sequence in a large language model generation task.

[0042] Decoding: The process by which a large language model gradually generates each token in a generation task, and each new token generated depends on the previously generated token.

[0043] KV Cache: intermediate cache data generated during model inference, including Key (K) and Value (V) vectors, used to store the context information of the previous token.

[0044] Separation architecture: The separation architecture for large language model reasoning specifically refers to the separation of prefill and decoding, as well as the management strategy of KV Cache under this setting.

[0045] Shard transmission: The process of dividing a large-capacity KV Cache into multiple small pieces for parallel transmission in a heterogeneous network environment.

[0046] Load balancing: Dynamically adjust shard transmission and computing task allocation based on GPU computing power and network bandwidth to balance system load and improve overall performance.

[0047] Below, each step of the large model reasoning service method in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.

[0048] like Figure 1 As shown, according to an embodiment of the present disclosure, a large model inference service system includes: a controller, a scheduler, multiple pre-filled instances and multiple decoding instances. The pre-filled instances will perform preliminary processing on the input text, calculate and store key-value pair information related to the input according to the architecture and parameters of the large language model LLM, and for the large language model LLM of the Transformer architecture, the decoding instance will use a multi-head parallel attention balancing mechanism and / or a pipeline parallel balancing mechanism to perform decoding operations on KVCache shards.

[0049] In some embodiments, in a parallel runtime environment, the pre-filling instance can utilize a multi-threaded or multi-process mechanism to simultaneously process data pre-filling for multiple input tasks. For the decoding instance, the parallel runtime can implement parallel computing of a multi-head parallel attention mechanism. In models that include multi-head attention, such as the Transformer architecture, each attention head of the decoding instance can be calculated simultaneously in a parallel runtime environment.

[0050] In some embodiments, in the migration configuration of KV Cache shards, different task amounts can be dynamically allocated between the replicas in the pre-filling and decoding stages according to the GPU performance and bandwidth characteristics to alleviate load imbalance. For example, smaller cache migration tasks can be allocated to nodes with lower bandwidth or lower GPU performance to reduce the overall migration time.

[0051] In addition, in the migration configuration of KV Cache shards, different shards can be transmitted in parallel between different links and different GPUs, which can fully utilize network bandwidth resources and reduce congestion on a single path.

[0052] In some embodiments, the large model inference service method disclosed herein may be executed by the controller and / or scheduler described above.

[0053] Figure 2 A flow chart of a large model reasoning service method in an embodiment of the present disclosure is shown.

[0054] like Figure 2 As shown, according to a large model reasoning service method of an embodiment of the present disclosure, the large model reasoning service includes multiple pre-filled nodes and multiple decoding nodes, and the method includes:

[0055] Step S202 , monitoring first computing resources of multiple decoding nodes and monitoring performance indicators of migration links between pre-filling nodes and decoding nodes.

[0056] Among them, the first computing power resource usually refers to the computing power of the decoding node (such as CPU, GPU, etc.), including processing speed, number of cores, parallel processing capability, etc. The performance indicators of the migration link include bandwidth (the rate at which the link transmits data), latency (the time it takes for data to be transmitted from the pre-filled node to the decoding node), packet loss rate (the proportion of data packets lost during transmission), etc.

[0057] In some embodiments, the large model inference service is modeled based on a separation architecture, including a prefill stage and a decoding stage. The Prefill stage includes M copies, and the generated KV Cache data needs to be migrated to the decoding node. The Decoding stage has N copies, and each decoding copy needs to access the KV Cache generated by the Prefill node.

[0058] Step S204, in response to the large model reasoning task, the reasoning task is allocated to multiple pre-filling nodes, so that the multiple pre-filling nodes perform pre-filling operations on the reasoning task in parallel to generate corresponding key-value cache KV Cache data.

[0059] In some embodiments, the data size of the KV Cache generated by the Prefill copy m (m=1, 2, ..., M) is B m .

[0060] Step S206: Slice the KV Cache data based on the first computing resource and / or performance indicator to obtain KV Cache slices.

[0061] Among them, sharding processing refers to dividing the generated KV Cache data into multiple smaller parts, namely KV Cache shards, according to certain rules and bases (such as computing resources, link performance indicators, etc.).

[0062] Step S208, migrating the KV Cache slices to multiple decoding nodes, so that the multiple decoding nodes perform decoding operations on the KV Cache slices based on a multi-head parallel attention balancing mechanism and / or a pipeline parallel balancing mechanism to obtain inference results.

[0063] Among them, each attention head can focus on the input information from different angles and perform calculations. Based on the multi-head parallel attention balancing mechanism, balanced parallel calculations can be achieved among multiple attention heads through reasonable configuration and scheduling.

[0064] The pipeline parallel balancing mechanism divides the decoding operation into multiple stages in a certain order to form a pipeline-like workflow, and achieves load balancing between the stages to ensure that data flows through each stage in sequence for processing.

[0065] In some embodiments, the decoding replica n (n=1, 2, ..., N) needs to receive the responding KV Cache to start the decoding process.

[0066] In some embodiments, in an environment where GPUs and network bandwidths are heterogeneous, migration time is modeled. The migration time is affected by two main factors, namely, the performance indicators of the first computing power resource and the migration link. The first computing power resource includes but is not limited to GPU bandwidth and performance differences. Each GPU may have different computing power and memory bandwidth. The performance indicators of the migration link include but are not limited to network bandwidth differences. The network bandwidth between nodes of different network types may be inconsistent. For example, in a distributed system, the bandwidth differences of NVLink, InfiniBand, Ethernet, etc. are large.

[0067] Assume that the size of the KV Cache generated by the mth replica in the Prefill phase is B m , then the network bandwidth from Prefill node m to decoding node n is Bandwidth m,n .

[0068] The KV Cache migration time from node m to node n is shown in formula (1):

[0069]

[0070] Under the conditions of multiple replicas and heterogeneous network bandwidth, in order to ensure that the decoding phase can start smoothly, the decoding replica must wait for all the KV Cache data it needs to be migrated. Therefore, the total migration time depends on the slowest migration link. The total migration time T migrate, total,Assume that the decoding node needs to wait for all M Prefill replicas to complete the migration, and there are N decoding nodes performing decoding, so the total migration time is the maximum value among all migration links, as shown in Equation (2).

[0071]

[0072] That is, among all the M×N migration links, the total migration time depends on the slowest path, or the migration bottleneck path. By sharding the KV Cache data to obtain KV Cache shards, each KV Cache shard is transmitted using a migration link alone. The total migration time depends on the slowest path of the KV Cache shards.

[0073] In some embodiments, if the KV Cache data volume is large, B m Divide into P pieces of KV Cache for fragment transmission, each piece is of size Segment transmission time T migrate,shard As shown in formula (3):

[0074]

[0075] The total migration time of KV Cache data is shown in formula (4):

[0076]

[0077] Shard transmission refers to allocating KV Cache shards to different migration links, which can effectively reduce the latency of a single migration link, especially on low-bandwidth paths, and can significantly improve migration efficiency.

[0078] In some embodiments, by combining the above optimizations, the inference efficiency can be modeled as the optimized migration time Compared with non-optimized migration time The ratio of is taken as the comprehensive reasoning efficiency, as shown in formula (5).

[0079]

[0080] The unoptimized migration time is shown in formula (2), and the optimized migration time is As shown in formula (4), it is the total migration time after optimization measures such as shard transmission, priority transmission and load balancing. Through the above modeling, we can more clearly understand the impact of KV Cache migration under heterogeneous GPU and bandwidth conditions, and significantly improve the reasoning efficiency through measures such as shard transmission, task scheduling optimization and priority transmission strategy.

[0081] In this embodiment, the first computing resources of multiple decoding nodes and the performance indicators of the migration link between the pre-filling node and the decoding node are monitored in real time. When there is a large model reasoning task, the task is first assigned to multiple pre-filling nodes. The pre-filling nodes will perform pre-filling operations in parallel to generate corresponding KV Cache data. Based on the monitored first computing resources and / or the performance indicators of the migration link, the generated KV Cache data is dynamically adjusted in size, which can optimize the transmission efficiency and computing resource utilization. Through the multi-path transmission protocol, the KV Cache fragments are migrated to multiple decoding nodes, which can reduce the transmission bottleneck. At the decoding node, the multi-head parallel attention balancing mechanism and / or the pipeline parallel balancing mechanism are used to perform decoding operations on the fragments, and finally generate the reasoning results. By comprehensively considering the computing resources and link performance, the KV Cache data is reasonably fragmented, which can improve the migration efficiency and migration stability of the KV Cache data. With the help of the multi-head parallel attention balancing mechanism and the pipeline parallel balancing mechanism, decoding processing is performed at the decoding node, which can improve the reasoning efficiency of the large model reasoning task.

[0082] like Figure 3 As shown, in one embodiment of the present disclosure, multiple decoding nodes perform decoding operations on KV Cache fragments based on a multi-head parallel attention balancing mechanism and / or a pipeline parallel balancing mechanism to obtain inference results, including:

[0083] Step S302, allocating pipeline branches to multiple decoding nodes based on decoding operations, the pipeline branches including multiple processing layers, dividing the KV Cache shard based on the number of processing layers to obtain a first sub-shard, wherein the pipeline branches in different decoding nodes are executed concurrently.

[0084] In the decoding stage, a sharded pipeline serial strategy is introduced to divide the transmission and calculation into multiple steps, so that the decoding node can start calculation immediately after receiving some shards.

[0085] In some embodiments, each shard It can be divided into s first sub-slices, which are transmitted and calculated in sequence according to the pipeline order. Assuming that the pipeline has L layers, each layer processes a first sub-slice, the pipeline processing time of the kth slice is shown in formula (6):

[0086]

[0087] Based on the pipeline processing method, whenever the first sub-slice arrives at the decoding node, it immediately enters the next layer of the pipeline for calculation, thereby reducing waiting time.

[0088] In one embodiment of the present disclosure, the decoding node includes multiple processing layers along the processing order, and also includes: allocating the multiple processing layers to different decoding nodes, and performing load balancing distribution of the task volume of each processing layer based on the first computing power resources of the decoding node.

[0089] In some embodiments, in pipeline parallelism, the decoding task is decomposed into multiple pipeline stages, and each stage is processed in parallel on different GPU nodes. Assume that the decoding stage is divided into L pipeline steps, and the task amount of each step is Load balancing is performed according to the computing power of each node, as shown in formula (7).

[0090]

[0091] in, It is the total time of the decoding task. Nodes with stronger computing power will take on more computing steps, so that the decoding time of each node tends to be equal, reducing waiting time and improving pipeline efficiency.

[0092] In this embodiment, each pipeline branch includes multiple processing layers, and the KVCache shard is divided based on the number of processing layers to obtain the first sub-shard. For example, assuming there are 5 processing layers, the received KV Cache shard is divided into 5 parts according to a certain rule (which can be an even distribution, or based on the data demand weight of each layer, etc.), each of which is a first sub-shard, and the pipeline branches in different decoding nodes are executed concurrently, indicating that multiple decoding nodes can start their own pipeline processing processes at the same time, effectively utilizing the parallel processing capabilities of the hardware.

[0093] Step S304: In each processing layer, the first sub-slice is divided based on the number of attention heads in the multi-head parallel attention balancing mechanism to obtain a second sub-slice, so as to allocate the second sub-slice to different attention heads for parallel calculation to obtain a calculation result.

[0094] For the first subshard The decoding stage usually uses a multi-head attention mechanism to calculate the attention distribution of each token. Assuming that the model has h attention heads, each slice can be divided into h small slices, and each small slice is calculated in parallel on different attention heads. The multi-head attention calculation formula is shown in formula (8).

[0095]

[0096] Among them, i=1, 2,..., h.

[0097] In one embodiment of the present disclosure, allocating the second shards to different attention heads for parallel computing also includes: allocating the computing of different attention heads to different decoding nodes based on the first computing power resources.

[0098] By allocating different shards to the attention heads of different GPU nodes, the computing efficiency is improved. The total time of multi-head parallel computing is shown in formula (9).

[0099]

[0100] in, represents the computation time of the i-th attention head.

[0101] In the multi-head attention mechanism, different attention heads can be calculated in parallel on different GPU nodes. Assuming that the model has h attention heads, and each head can be calculated in parallel on an independent GPU, in order to achieve load balancing, the calculation tasks of the attention heads can be allocated according to the proportion of the GPU's computing power. For each attention head i , the computational load L assigned to node j ij It can be defined as shown in formula (10).

[0102]

[0103] in is the total amount of computational tasks per head, C j is the computing power of node j. In this way, the computing tasks of each attention head are distributed to different nodes in proportion to maintain load balancing.

[0104] Step S306, the calculation result obtained by the attention head calculation is directly input into the next processing layer for pipeline processing until the last processing layer outputs the inference result.

[0105] In some embodiments, the shard processing task is as shown in formula (11).

[0106]

[0107] The total parallel decoding time is the maximum value of all allocated slice decoding times as shown in formula (12).

[0108]

[0109] Among them, T decode,total Based on T multi-head and T pipeline Sure, is the total time of shard transmission, which depends on the network bandwidth and the slowest transmission time of the path, T multi-head It is the multi-head parallel computing time, used for multi-head attention computing on the shards.

[0110] Accordingly, combining the sharding transmission and parallel computing strategies, the total reasoning time T total As shown in formula (13).

[0111]

[0112] In this embodiment, the decoding nodes are arranged in multiple processing layers in a processing order, and the KV Cache slices are divided according to the number of processing layers to obtain the first sub-slice. Then, the second sub-slice is further divided in each processing layer according to the number of attention heads in the multi-head parallel attention balancing mechanism. After being assigned to different attention heads for parallel calculation, the results are directly input into the next layer for pipeline processing until the last layer outputs the inference results. This design makes full use of the advantages of parallel computing and pipeline processing, effectively improves the decoding efficiency, reduces the data waiting and transmission time between each link, and ensures that the entire large model inference process can be carried out more smoothly and efficiently, thereby quickly and accurately outputting high-quality inference results.

[0113] like Figure 4 As shown, in one embodiment of the present disclosure, KVCache data is sliced ​​based on the first computing resource and / or performance indicator to obtain KV Cache slices, including:

[0114] Step 402, normalize the first computing power resource and the performance indicator respectively to obtain computing power resource parameters and performance indicator parameters.

[0115] Step 404: Determine the unit sharding amount for sharding the KV Cache data based on the smaller value of the computing resource parameter and the performance indicator parameter.

[0116] In some embodiments, the number of shards P and the size of each shard B shard Depending on the network bandwidth and GPU load, smaller shards can usually better adapt to low-bandwidth links, while larger shards can reduce transmission overhead in high-bandwidth links. Due to the heterogeneity of GPUs and network bandwidth, the shard size of KV Cache can be based on the computing power of the target GPU node. i and network bandwidth i,j Dynamic adjustment. Assume that the computing power of node i is C i , the network bandwidth between nodes j and i is Bandwidth i,j , then for each piece Its transmission and processing time should satisfy formula (14):

[0117]

[0118] Each shard The size is determined based on the smaller value between the GPU computing power and the network bandwidth to ensure that the bandwidth and computing resources can be efficiently matched.

[0119] Step 406: Slice the KV Cache data based on the unit slicing amount to obtain KV Cache slicing.

[0120] In some embodiments, assuming that the size of the KV Cache generated in the pre-filling phase is B, B is divided into P small pieces Each small piece after segmentation can be transmitted independently, thus maximizing the utilization of network resources.

[0121] In some embodiments, different slices can be transmitted in parallel between different links and different GPUs, which can make full use of network bandwidth resources and reduce congestion on a single path. Each slice is transmitted to the decoding node through a different path, making full use of the bandwidth of all network links. It can be modeled as shown in formula (15):

[0122]

[0123] The total migration time is the maximum migration time among all paths as shown in formula (16).

[0124]

[0125] In one embodiment of the present disclosure, sharding the KV Cache data based on the first computing resource and / or the performance indicator to obtain the KV Cache shards includes:

[0126] The computing power of each decoding node is determined based on the first computing power resource, so as to perform sharding processing on the KV Cache data based on the computing power to obtain KV Cache shards that match the computing power.

[0127] In some embodiments, under heterogeneous GPU conditions, the decoding stage can be based on the computing power C of each GPU. j Dynamically adjust task allocation. For GPUs with stronger computing power, more or larger shards of tasks are allocated. For GPUs with lower computing power, fewer or smaller shards of tasks are allocated.

[0128] The migration capability of each migration link is determined based on the performance index, so that KV Cache data is segmented based on the migration capability to obtain KV Cache segments that match the migration capability.

[0129] In this embodiment, by determining the computing power of each decoding node based on the first computing power resource, and then sharding the KV Cache data according to the computing power, the obtained KV Cache shards can be adapted to the computing power of the decoding nodes, thereby preventing problems such as inefficiency caused by the amount of data exceeding the node processing capacity. The migration capacity of each migration link is clarified based on performance indicators, and the KV Cache data is sharded accordingly, which can match the shards with the link migration capacity and ensure efficient and stable transmission of data between the pre-filled nodes and the decoding nodes.

[0130] In one embodiment of the present disclosure, migrating KV Cache shards to multiple decoding nodes includes:

[0131] The priorities of multiple migration links are determined based on performance indicators, so as to migrate KVCache fragments to multiple decoding nodes based on the priorities.

[0132] In some embodiments, if the KV Cache data volume is large, B m Divide into P pieces for fragment transmission, each piece is .Slice transmission time T migrate,shard As shown in formula (17).

[0133]

[0134] The total migration time obtained is shown in formula (18).

[0135]

[0136] In some embodiments, different task amounts can be dynamically allocated between the copies in the pre-filling and decoding stages according to the GPU performance and bandwidth characteristics to alleviate load imbalance. Smaller cache migration tasks can be allocated to nodes with lower bandwidth or lower GPU performance to reduce the overall migration time. Let the GPU computing power of node m be C m , then the task load L assigned to it m It can be defined as shown in formula (19).

[0137]

[0138] Among them, L total is the total amount of reasoning tasks.

[0139] In this embodiment, sharded transmission can effectively reduce the delay of a single migration link, especially on a low-bandwidth path. Such allocation helps to balance computing power and reduce the bottleneck of migration load, thereby significantly improving migration efficiency.

[0140] In one embodiment of the present disclosure, determining priorities of multiple migration links based on performance indicators includes:

[0141] The performance indicators include network bandwidth, network delay and packet loss rate, and the network quality of different migration links from multiple pre-filled nodes to multiple decoding nodes is calculated based on the network bandwidth, network delay and packet loss rate.

[0142] The network status monitoring submodule detects the network status between the Prefill node and the Decoding node in real time, including the following main network parameters:

[0143] For network bandwidth, measure the bandwidth between the Prefill node and the Decoding node. prefill,decoding ,The transmission rate can be calculated as the network bandwidth by sending and receiving data packets, as shown in formula (20).

[0144]

[0145] For network latency, measure the round-trip latency of packets between the Prefill node and the Decoding node. prefill,decoding , use the ping packet to implement the round-trip delay calculation, as shown in formula (21).

[0146]

[0147] For the packet loss rate, the packet loss rate between the Prefill node and the Decoding node is calculated by transmitting the data packet multiple times, as shown in formula (22).

[0148]

[0149] In some embodiments, the network quality weight Q is calculated based on bandwidth, delay and packet loss rate. prefill,decoding , used to select the optimal transmission path and load balancing strategy, the network quality weight can be expressed as shown in formula (23).

[0150]

[0151] Configure the corresponding priority based on network quality.

[0152] Among them, the weight value Q prefill,decoding The higher the value, the better the network quality. It can be used to transmit important KVCache data first. In a multi-path transmission environment, the path with the highest weight is selected for transmission, and the transmission task is dynamically adjusted based on the weight.

[0153] In this embodiment, the network quality of different migration links is calculated by comprehensively considering network bandwidth, network delay and packet loss rate, and the priority is reasonably configured according to the network quality, so as to comprehensively and accurately evaluate the pros and cons of each migration link, so that during the data transmission process, KV Cache data can be efficiently transferred according to the priority order of the links. Giving priority to high-quality links can significantly reduce transmission time, reduce the risk of data loss, and ensure the stability and efficiency of data transmission, thereby optimizing the data interaction between pre-filled nodes and decoding nodes in large model reasoning tasks, effectively improving the execution efficiency and reliability of the entire reasoning task, and ensuring that the reasoning process proceeds smoothly and accurate reasoning results can be obtained in a timely manner.

[0154] In one embodiment of the present disclosure, the network quality of different migration links from multiple pre-filled nodes to multiple decoding nodes is calculated based on network bandwidth, network delay and packet loss rate, including:

[0155] The network quality is calculated based on the quality calculation formula (24).

[0156]

[0157] Bandwidth prefill,decoding is the network bandwidth, Latency prefill,decoding is network delay, Loss prefill,decoding is the packet loss rate.

[0158] In one embodiment of the present disclosure, migrating KV Cache shards to multiple decoding nodes further includes: determining a first KV Cache shard based on a priority of a decoding operation, so as to preferentially migrate the first KV Cache shard to the decoding node.

[0159] In some embodiments, if some decoding nodes need a specific KV Cache earlier, i.e., the first KV Cache fragment, these caches can be transmitted first, thereby reducing the decoding waiting time. For example, important cache fragments can be transmitted first as shown in formula (25).

[0160]

[0161] Among them, w m , is the cache importance priority. By prioritizing the transmission of key fragments, it can ensure that the decoding task is supported by the cache on demand, reducing the overall delay.

[0162] In one embodiment of the present disclosure, migrating KV Cache shards to multiple decoding nodes also includes: determining a second KV Cache shard based on the degree of dependence of the multi-head parallel attention balancing mechanism on the KV Cache shard, so as to preferentially migrate the second KV Cache shard to the decoding node.

[0163] In this embodiment, in the process of migrating KV Cache shards to multiple decoding nodes, the second KV Cache shard is determined based on the degree of dependence of the multi-head parallel attention balancing mechanism on the KV Cache shard, and it is migrated to the decoding node first. This can grasp the difference in the importance of different shards for decoding calculations, and prioritize the shards that play a key role in the multi-head parallel attention balancing mechanism to arrive at the decoding node first, so that the decoding node can use key data to start calculations more quickly, reduce the time consumption caused by waiting for important shards, and further improve the efficiency of the entire decoding process.

[0164] In one embodiment of the present disclosure, sharding the KV Cache data based on the unit sharding amount to obtain KV Cache shards further includes:

[0165] The KV Cache data is sharded based on the unit shard quantity to obtain the initial shard; the link bandwidth of the migration link is determined based on the performance index; an optimization model is constructed with the utilization of the link bandwidth and the migration time of the KV Cache shard as optimization targets; the initial shard is optimized based on the optimization model to obtain the KV Cache shard.

[0166] In some embodiments, the size and transmission path of each shard are adjusted to ensure balanced utilization of network bandwidth. Assuming that the total size of the KV Cache generated in the pre-filling phase is B KV , and divided into P small pieces, each piece size B shard According to the network bandwidth of the target node i,j The optimization goal of shard transmission load balancing is to maximize bandwidth utilization and minimize transmission time. For shard k, the migration time from Prefill node i to decoding node j is The model is constructed as shown in formula (26).

[0167]

[0168] in, By optimizing each The size of the transmission time is balanced, that is, to keep all paths as close as possible. Similarly, the optimization method is shown in formula (27).

[0169]

[0170] In this embodiment, the transmission task of each path is allocated according to the bandwidth ratio to ensure that the high-bandwidth path undertakes a larger transmission task. The KV Cache data is fragmented based on the unit fragmentation amount to obtain the initial fragmentation. The link bandwidth of the migration link is determined based on the performance index. Then, an optimization model is constructed with the improvement of link bandwidth utilization and the shortening of KV Cache fragmentation migration time as the optimization goals. The initial fragmentation is optimized through the model, thereby realizing real-time adjustment of KV Cache data fragmentation.

[0171] In one embodiment of the present disclosure, migrating KV Cache shards to multiple decoding nodes also includes:

[0172] The computing power of each decoding node is determined based on the first computing power resource; and the amount of data of the KV Cache shard allocated to the corresponding decoding node is determined based on the computing power to migrate the KV Cache shard to the corresponding decoding node.

[0173] In some embodiments, during the decoding phase, the computing tasks of each shard are assigned to different GPU nodes. Since the computing capabilities (such as FLOPS or memory bandwidth) of different GPUs are different, load balancing can be performed based on the computing capabilities of the GPUs. j Dynamically allocate the amount of tasks so that the computing time of each node is similar. Assume that there are N decoding nodes and the computing power of each node is C j , then the load distribution ratio of each node is L j As shown in formula (28):

[0174]

[0175] The total amount of shard data allocated to node j is shown in formula (29):

[0176]

[0177] In this embodiment, nodes with stronger computing power will be assigned more sharding tasks, while nodes with weaker computing power will be assigned fewer tasks, so that the KV Cache shards received by each decoding node can be perfectly adapted to its own computing power, thereby achieving computing load balancing.

[0178] In one embodiment of the present disclosure, monitoring the performance index of the migration link between the pre-filling node and the decoding node includes:

[0179] Scan candidate links between multiple pre-filling nodes and multiple decoding nodes; select a migration link for migrating KV Cache fragments from the candidate links based on network quality of the candidate links and prediction results of network trends, so as to monitor performance indicators of the migration links.

[0180] In some embodiments, the computing power detection module is a key component in the large model reasoning separation architecture, responsible for real-time detection and monitoring of the computing power of each computing node to achieve load balancing of reasoning tasks. The module can provide a reference for the task allocation strategy based on the GPU load of the node, thereby optimizing resource utilization and preventing computing bottlenecks.

[0181] Among them, the computing power detection module includes a hardware resource monitoring sub-module and a performance evaluation sub-module. The hardware resource monitoring sub-module is used to detect the GPU utilization and video memory occupancy of each computing node; the performance evaluation sub-module is used to evaluate the computing power of each device and dynamically calculate the computing power indicators based on the utilization and performance.

[0182] In some embodiments, hardware resource monitoring includes GPU detection and memory detection. GPU detection is used to collect GPU model, video memory size, computing power (such as CUDA core number), utilization rate, etc., and calculate the GPU computing power index Cap of the node in real time. GPU (t), as shown in formula (30).

[0183] Cap GPU (t)=CUDA cores × frequency × utilization rate(t) (30)

[0184] Memory detection is used to collect the memory usage of the node, including total memory, available memory, and utilization.

[0185] In this embodiment, the real-time detected computing power data is transmitted to the task allocation module through the communication interface to support load-balanced task allocation decisions. The task allocation module can dynamically adjust the task load according to the current computing power status, and the computing power detection module provides real-time feedback on the adjustment effect.

[0186] In some embodiments, after starting the computing power detection module, the resources of each computing node are scanned to perform preliminary GPU and video memory detection. The computing power detection module collects the resource utilization of each node at regular intervals (such as 1 second) and updates the computing power indicators of each node. When a significant change in the node load is detected, the computing power detection module notifies the task allocation module through the interface to assist in dynamic task adjustment.

[0187] In this embodiment, the computing power detection module can monitor the changes in computing power in real time, provide a basis for load balancing in a timely manner, be compatible with multiple device types and computing resources, realize comprehensive monitoring of heterogeneous computing power, and dynamically allocate tasks according to computing power and utilization rate, which helps to improve the efficiency and stability of distributed reasoning. Through the computing power detection module, the utilization rate of the computing power of each node in the distributed reasoning architecture can be significantly improved, and precise support can be provided for the task allocation of large model reasoning, thereby realizing the overall load balancing and performance optimization of the system.

[0188] In some embodiments, a network status awareness module is designed for KV Cache migration. The network status awareness module plays a key role in the KV Cache migration between the Prefill node and the Decoding node. It is responsible for monitoring and evaluating the network quality to ensure load balancing under different network paths. By collecting network bandwidth, latency, packet loss rate and other indicators in real time, the module can dynamically adjust the KV Cache's shard transmission path and task allocation strategy to optimize transmission efficiency and reduce latency.

[0189] The network status perception module needs to realize real-time detection of network status, network quality assessment and weight calculation, dynamic feedback and load balancing support, historical data recording and trend prediction.

[0190] Among them, the real-time network status detection continuously detects the bandwidth, delay and packet loss rate between the Prefill node and the Decoding node to obtain the accurate network status;

[0191] Network quality evaluation and weight calculation The network quality weight is calculated based on various indicators of the network status to determine the transmission path priority of KVCache;

[0192] Dynamic feedback and load balancing support means that when the network status changes, the transmission path and task allocation strategy are dynamically updated to optimize the migration efficiency of KV Cache;

[0193] Historical data recording and trend prediction refers to storing historical data of network status, analyzing network trends, and providing reference for future transmission load prediction.

[0194] The network status perception module includes a network status monitoring submodule, a network quality assessment submodule, a data storage and trend analysis submodule, and a task allocation and transmission optimization submodule.

[0195] Among them, the network status monitoring submodule is used to detect the network bandwidth, delay and packet loss rate between the Prefill node and the Decoding node in real time; the network quality assessment submodule is used to calculate the network quality weight in order to determine the best path priority in multi-path transmission; the data storage and trend analysis submodule is used to record network historical data, analyze trends and predict future network quality; the task allocation and transmission optimization submodule is used to dynamically adjust the transmission path of KV Cache according to the real-time network quality.

[0196] like Figure 5 As shown, in some embodiments, the network status perception process includes:

[0197] Step S502, start the network status perception module to scan all possible paths between the Prefill node and the Decoding node.

[0198] Step S504: The network status perception module collects bandwidth, delay, and packet loss rate data between the Prefill node and the Decoding node at regular intervals.

[0199] Step S506, calculate the network quality weight of each path based on the real-time data, and select a high-quality path to transmit the KVCache.

[0200] In some embodiments, the network status perception module is also used to periodically store network status data to support future network quality prediction and transmission strategy optimization.

[0201] Historical data storage is used to save bandwidth, latency, packet loss rate and other data for long-term network quality assessment and trend analysis;

[0202] Predict the future network status based on historical data. Assume that the moving average method is used to predict the network quality weight Q prefill,decoding , as shown in formula (31).

[0203]

[0204] Where N is the size of the sliding window.

[0205] Step S508, when a significant change in the network status is detected, the transmission path is notified to be adjusted and the transmission task of the KVCache is reallocated.

[0206] Among them, the network status perception module adjusts the transmission path of KV Cache according to the real-time network status to ensure transmission efficiency and stability under network fluctuations. When the bandwidth, delay or packet loss rate of a transmission path changes significantly, the module dynamically adjusts the transmission path and gives priority to the path with high quality weight; for the fragmented KV Cache, important fragmented data is preferentially allocated to the path with better network quality for transmission to reduce the transmission delay of key data; when a change in network status is detected, the task allocation module is notified in real time to support fast path optimization and task reallocation.

[0207] Step S510, regularly storing network data in a database for trend analysis and future status prediction.

[0208] In this embodiment, the network status perception module can capture the changes in network status between the Prefill and Decoding nodes in real time, provide a basis for the load balancing of the KV Cache, and dynamically adjust the network transmission path to ensure that the KV Cache data can be transmitted efficiently and stably under network fluctuations. Through trend analysis and prediction of historical data, potential network bottlenecks can be identified in advance to provide a basis for future transmission strategies. The network status perception module for KV Cache migration monitors and evaluates the network status between the Prefill node and the Decoding node in real time, dynamically adjusts the transmission path of the KV Cache in a multi-path transmission environment, effectively realizes network load balancing, and can maintain efficient data transmission under network quality fluctuations, providing a strong guarantee for load balancing and performance optimization of the large model reasoning separation architecture.

[0209] In one embodiment of the present disclosure, in response to a large model reasoning task, the reasoning task is distributed to multiple pre-filled nodes, including: monitoring the second computing power resources of the multiple pre-filled nodes; in response to the large model reasoning task, the reasoning task is distributed to the multiple pre-filled nodes based on the monitoring results of the second computing power resources.

[0210] In this embodiment, by real-time monitoring of the second computing power resources of multiple pre-filled nodes, when facing large model reasoning tasks, tasks are reasonably allocated to each pre-filled node based on the monitoring results, so as to fully consider the actual computing power of each pre-filled node, ensure that the reasoning tasks can be accurately and evenly distributed to different nodes, and prevent some nodes from running slowly due to computing power overload while the computing power of other nodes is idle, so that the computing power resources of each pre-filled node can be effectively utilized, thereby significantly improving the overall efficiency of the pre-filling stage.

[0211] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0212] Refer to the following Figure 6 The large model reasoning service device 600 according to an embodiment of the present disclosure is described. Figure 6 The large model reasoning service device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0213] The large model reasoning service device 600 is presented in the form of a hardware module or a software module. The components of the large model reasoning service device 600 may include but are not limited to: a monitoring module 602 for monitoring the first computing power resources of the multiple decoding nodes and monitoring the performance indicators of the migration link between the pre-filling node and the decoding node; an allocation module 604 for responding to the large model reasoning task, allocating the reasoning task to multiple pre-filling nodes, so that the multiple pre-filling nodes perform the pre-filling operation on the reasoning task in parallel, and generate the corresponding key-value cache KV Cache data; a processing module 606 for slicing the KV Cache data based on the first computing power resource and / or performance indicator to obtain KV Cache slicing; a migration module 608 for migrating the KV Cache slicing to multiple decoding nodes, so that the multiple decoding nodes perform decoding operations on the KV Cache slicing based on the multi-head parallel attention balancing mechanism and / or the pipeline parallel balancing mechanism to obtain the reasoning result.

[0214] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710).

[0215] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 710 can perform the following steps: Figure 2 The scheme described in .

[0216] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0217] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0218] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0219] The electronic device 700 may also communicate with one or more external devices 770 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0220] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or an electronic device, etc.) to execute the method according to the implementation of the present disclosure.

[0221] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to enable the electronic device to perform the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0222] According to the program product for implementing the above method according to the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0223] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0224] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0225] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0226] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the IoT terminal, as an independent software package, partially on the user computing device and partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0227] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0228] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0229] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or an electronic device, etc.) to execute the method according to the implementation of the present disclosure.

[0230] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A large model reasoning service method, characterized in that: include: The large model inference service includes a plurality of pre-filled nodes and a plurality of decoding nodes, monitoring a first computing resource of the plurality of decoding nodes and monitoring a performance indicator of a migration link between the pre-filled nodes and the decoding nodes; In response to a large model reasoning task, the reasoning task is assigned to a plurality of pre-filling nodes, so that the plurality of pre-filling nodes perform a pre-filling operation on the reasoning task in parallel to generate corresponding key-value cache KV Cache data; Slice the KV Cache data based on the first computing resource and / or the performance indicator to obtain KV Cache slices; The KV Cache slices are migrated to the multiple decoding nodes, so that the multiple decoding nodes perform decoding operations on the KV Cache slices based on a multi-head parallel attention balancing mechanism and / or a pipeline parallel balancing mechanism to obtain inference results.

2. The large model reasoning service method according to claim 1, characterized in that: The multiple decoding nodes perform decoding operations on the KV Cache shards based on a multi-head parallel attention balancing mechanism and / or a pipeline parallel balancing mechanism to obtain inference results, including: Allocating pipeline branches to the multiple decoding nodes based on the decoding operation, the pipeline branches including multiple processing layers, dividing the KV Cache shard based on the number of the processing layers to obtain a first sub-shard, wherein the pipeline branches in different decoding nodes are executed concurrently; In each of the processing layers, the first sub-slice is divided based on the number of attention heads in the multi-head parallel attention balancing mechanism to obtain second sub-slices, so as to allocate the second sub-slices to different attention heads for parallel calculation to obtain calculation results; The calculation result obtained by the attention head is directly input into the next processing layer for pipeline processing until the last processing layer outputs the inference result.

3. The large model reasoning service method according to claim 2, characterized in that: The decoding node includes a plurality of processing layers in a processing order, and further includes: The multiple processing layers are allocated to different decoding nodes, and the load balancing distribution of the task amount of each processing layer is performed based on the first computing power resources of the decoding nodes.

4. The large model reasoning service method according to claim 2, characterized in that: Allocating the second shards to different attention heads for parallel computing, further comprising: Based on the first computing power resource, the calculation of different attention heads is allocated to different decoding nodes.

5. The large model reasoning service method according to claim 1, characterized in that: Slicing the KV Cache data based on the first computing resource and / or the performance indicator to obtain KV Cache slices includes: Normalizing the first computing power resource and the performance indicator respectively to obtain a computing power resource parameter and a performance indicator parameter; Determine, based on the smaller value of the computing resource parameter and the performance indicator parameter, a unit sharding amount for sharding the KV Cache data; The KV Cache data is sliced ​​based on the unit slice quantity to obtain the KV Cache slice.

6. The large model reasoning service method according to claim 1, characterized in that: Slicing the KV Cache data based on the first computing resource and / or the performance indicator to obtain KV Cache slices includes: Determine the computing power of each of the decoding nodes based on the first computing power resource, so as to perform sharding processing on the KV Cache data based on the computing power to obtain the KV Cache shards matching the computing power; or The migration capability of each migration link is determined based on the performance indicator, so as to perform sharding processing on the KVCache data based on the migration capability to obtain the KV Cache shards matching the migration capability.

7. The large model reasoning service method according to claim 1, characterized in that: Migrating the KV Cache fragments to the multiple decoding nodes includes: The priorities of the plurality of migration links are determined based on the performance indicator, so as to migrate the KV Cache fragments to the plurality of decoding nodes based on the priorities.

8. The large model reasoning service method according to claim 7, characterized in that: Determining priorities of the plurality of migration links based on the performance indicators includes: The performance indicators include network bandwidth, network delay and packet loss rate, and the network quality of different migration links from the multiple pre-filling nodes to the multiple decoding nodes is calculated based on the network bandwidth, the network delay and the packet loss rate; The corresponding priority is configured based on the network quality.

9. The large model reasoning service method according to claim 8, characterized in that: Calculating the network quality of different migration links from the plurality of pre-filled nodes to the plurality of decoding nodes based on the network bandwidth, the network delay, and the packet loss rate includes: Based on the mass calculation formula Calculating the network quality, Bandwidth prefill,decoding is the network bandwidth, Latency prefill,decoding is the network delay, Loss prefill,decoding is the packet loss rate.

10. The large model reasoning service method according to claim 1, characterized in that: Migrating the KV Cache fragments to the multiple decoding nodes also includes: A first KV Cache slice is determined based on the priority of the decoding operation, so as to preferentially migrate the first KV Cache slice to the decoding node.

11. The large model reasoning service method according to claim 2, characterized in that: Migrating the KV Cache fragments to the multiple decoding nodes also includes: A second KVCache segment is determined based on the degree of dependence of the multi-head parallel attention balancing mechanism on the KV Cache segment, so as to preferentially migrate the second KV Cache segment to the decoding node.

12. The large model reasoning service method according to claim 5, characterized in that: Slicing the KV Cache data based on the unit slicing amount to obtain the KV Cache slicing further includes: Slice the KV Cache data based on the unit slicing amount to obtain initial slicing; determining a link bandwidth of the migration link based on the performance indicator; An optimization model is constructed with the utilization rate of the link bandwidth and the migration time of the KV Cache shard as optimization targets; The initial shard is optimized based on the optimization model to obtain the KV Cache shard.

13. The large model reasoning service method according to claim 1, characterized in that: Migrating the KV Cache fragments to the multiple decoding nodes also includes: Determining the computing power of each of the decoding nodes based on the first computing power resource; The amount of data of the KV Cache slice allocated to the corresponding decoding node is determined based on the computing capacity, so as to migrate the KV Cache slice to the corresponding decoding node.

14. The large model reasoning service method according to any one of claims 1 to 13, characterized in that: Monitoring a performance indicator of a migration link between the pre-filling node and the decoding node includes: scanning candidate links between the plurality of pre-filled nodes and the plurality of decoding nodes; The migration link for migrating the KV Cache fragment is selected from the candidate links based on the network quality of the candidate links and the prediction result of the network trend, so as to monitor the performance index of the migration link.

15. The large model reasoning service method according to any one of claims 1 to 13, characterized in that: In response to a large model inference task, allocating the inference task to the plurality of pre-populated nodes comprises: Monitoring a second computing resource of the plurality of pre-populated nodes; In response to the large model reasoning task, the reasoning task is allocated to the multiple pre-filled nodes based on the monitoring result of the second computing power resource.

16. A large model reasoning service device, characterized in that: Applicable to the task allocation module, including: A monitoring module, configured to monitor the first computing resources of the plurality of decoding nodes and monitor the performance indicators of the migration links between the pre-filling nodes and the decoding nodes; An allocation module, configured to allocate the reasoning task to a plurality of pre-filling nodes in response to a large model reasoning task, so that the plurality of pre-filling nodes perform pre-filling operations on the reasoning task in parallel to generate corresponding key-value cache KV Cache data; A processing module, configured to perform sharding processing on the KV Cache data based on the first computing resource and / or the performance indicator to obtain KV Cache shards; A migration module is used to migrate the KV Cache slices to the multiple decoding nodes, so that the multiple decoding nodes can perform decoding operations on the KV Cache slices based on a multi-head parallel attention balancing mechanism and / or a pipeline parallel balancing mechanism to obtain inference results.

17. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the large model reasoning service method described in any one of claims 1 to 15 by executing the executable instructions.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large model reasoning service method described in any one of claims 1 to 15 is implemented.

19. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large model reasoning service method described in any one of claims 1 to 15 is implemented.

Images