Distributed reasoning method and electronic device

Abstract

The application relates to the computer technical field, in particular to a distributed reasoning method and an electronic device, the method comprises the following steps: performing parallel processing on an input sequence of a current reasoning request to obtain an initial key-value cache and an initial hidden state, then obtaining a first hidden state and expert distribution metadata, and performing matrix multiplication on the first hidden state based on the expert distribution metadata to generate a second hidden state; updating the initial key-value cache based on the second hidden state to obtain an updated key-value cache, taking the updated key-value cache as the initial key-value cache, taking the second hidden state as the initial hidden state, and re-executing the above steps until a preset reasoning end condition is reached to obtain a final reasoning result. Therefore, the problem of resource mismatch and low utilization caused by the isomorphism deployment of the attention module and the feedforward network module in the related art is solved, the hardware utilization is improved, and the reasoning cost is reduced.

Claims

1. A method of distributed reasoning, the method comprising: Includes the following steps: In response to the current inference request, the input sequence of the current inference request is processed in parallel to obtain the initial key-value cache and the initial hidden state; The first hidden state and expert distribution metadata are obtained based on the initial key-value cache and the initial hidden state. Then, based on the expert distribution metadata, matrix multiplication is performed on the first hidden state to generate the second hidden state. The updated key-value cache is obtained by updating the initial key-value cache based on the second hidden state, and the updated key-value cache is used as the initial key-value cache. The second hidden state is used as the initial hidden state. The step of obtaining the first hidden state and the expert distribution metadata based on the initial key-value cache and the initial hidden state is re-executed until the preset inference termination condition is reached, and the final inference result is obtained. The step of obtaining the first hidden state and the expert distribution metadata based on the initial key-value cache and the initial hidden state includes: loading the initial key-value cache, performing multi-head attention computation on the initial hidden state to generate intermediate attention output; performing normalization and residual connection processing on the intermediate attention output in sequence to obtain the first hidden state, and generating the expert distribution metadata based on the first hidden state; wherein, the expert distribution metadata includes activated expert indexes and corresponding gating weights; The step of performing matrix multiplication on the first hidden state to generate a second hidden state based on the expert distribution metadata includes: determining the target minimum batch size of the hybrid expert unit; and, in response to the cumulative number of tokens corresponding to the first hidden state reaching the target minimum batch size, scheduling the hybrid expert unit to perform matrix multiplication on the first hidden state to generate the second hidden state based on the expert distribution metadata. The hybrid expert unit includes a model hidden layer and an expert hidden layer. Determining the target minimum batch size of the hybrid expert unit includes: obtaining the current computing power-bandwidth ratio, the current dimension of the model hidden layer, and the current dimension of the expert hidden layer; determining the sparsity of the hybrid expert unit based on a preset sparsity constraint; and determining the target minimum batch size based on the current computing power-bandwidth ratio, the current dimension of the model hidden layer, the current dimension of the expert hidden layer, and the sparsity of the hybrid expert unit.

2. The method of claim 1, wherein, Before determining the sparsity of the hybrid expert unit based on preset sparsity constraints, the method further includes: Obtain the current network bandwidth, maximum allowable latency in a single stage, current batch size, and total number of transformation layers in the model; The preset sparsity constraints are determined based on the current network bandwidth, the maximum allowable latency in a single stage, the current computing power-bandwidth ratio, the current dimension of the model hidden layer, the current dimension of the expert hidden layer, the current batch size, and the total number of transformation layers of the model.

3. The method according to claim 1, characterized in that, After using the second hidden state as the initial hidden state, and re-executing the step of obtaining the first hidden state and the expert distribution metadata based on the initial key-value cache and the initial hidden state until a preset inference termination condition is met and the final inference result is obtained, the method further includes: Release current hardware and / or network resources, output a word sequence, and generate the time, throughput, and inference cost of a single word output.

4. The method according to claim 3, characterized in that, The single-word inference cost includes attention computation cost and feedforward network computation cost, and before generating the single-word inference cost, it also includes: Obtain the attention computation cost, attention memory access cost, hardware unit floating-point operation cost, hardware unit memory access cost, feedforward network computation cost, and hardware utilization rate of the single word; The attention computation cost is obtained based on the attention computation cost of a single word, the attention memory access cost of a single word, the hardware unit floating-point operation cost, and the hardware unit memory access cost. The computational cost of the feedforward network is obtained based on the computational cost of the feedforward network for a single word, the cost of the hardware unit floating-point operation, and the hardware utilization rate.

5. The method according to claim 1, characterized in that, The preset reasoning termination condition is: The currently generated word is the preset terminator; And / or, the length of the generated word sequence has reached a preset length threshold; And / or, the cumulative delay of the current inference step exceeds the maximum allowed delay duration threshold for a single stage.

6. The method according to claim 1, characterized in that, Also includes: Monitor the total processing time of the current round; In response to the total processing time exceeding the preset processing time, the current batch size is reduced based on a preset reduction strategy.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, the processor executing the computer program to implement the distributed reasoning method as described in any one of claims 1-6.

Images