Model inference method, system, apparatus, and computer-readable storage medium

Abstract

This application discloses a model inference method, system, apparatus, and storage medium, belonging to the field of deep learning technology. The method is used in a model inference system, which includes a scheduling device and a processor cluster. The method includes: upon receiving a model inference request, the scheduling device divides the task sequence corresponding to the model inference request based on sequence parallelism, obtaining multiple task sub-sequences; the scheduling device allocates the multiple task sub-sequences to various processor groups in the processor cluster, where the number of processors in each processor group is the same as the sequence parallelism; each processor in each processor group is configured with model sub-weights, which are obtained by partitioning the model weights based on tensor parallelism; the processor cluster generates the model inference result corresponding to the task sequence based on the sequence output results generated by each processor executing the task sub-sequences. The method provided in this application can improve the memory access efficiency of distributed inference systems.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to model inference methods, systems, devices and computer-readable storage media. Background Technology

[0002] With the continuous development and popularization of deep learning technology, distributed inference for large models has gradually become a hot topic of common concern in academia and industry. Distributed inference includes various parallelism methods, such as data parallelism, sequence parallelism, tensor parallelism, and pipelined parallelism.

[0003] In related technologies, when implementing distributed inference through tensor parallelism, key-value weights need to be segmented along the attention head dimension according to the tensor parallelism. This means that the tensor parallelism is strongly dependent on the number of attention heads in the model. However, for models using grouped query attention (GQA), since GQA reduces the number of attention heads and the cache space occupied by key-value tensors by reducing the number of attention heads, the number of attention heads may be less than the number of processors. When the number of attention heads is less than the number of processors, at least two processors may be configured with the same attention head to execute the task sequence and cache the same key-value tensors, leading to duplicate caching of key-value tensors and thus affecting the memory access efficiency of the model's inference system. Summary of the Invention

[0004] This application provides a model reasoning method, system, apparatus, and computer-readable storage medium to improve the memory access efficiency of distributed reasoning systems. The technical solution is as follows.

[0005] Firstly, a model inference method is provided for a model inference system, which includes a scheduling device and a processor cluster. The method includes: upon receiving a model inference request, the scheduling device divides the task sequence corresponding to the request into multiple task sub-sequences based on sequence parallelism; the scheduling device assigns the multiple task sub-sequences to various processor groups in the processor cluster, where the number of processors in each processor group is the same as the sequence parallelism, and each processor in a processor group corresponds one-to-one with a task sub-sequence; each processor in a processor group is configured with model sub-weights, which are obtained by partitioning the model weights based on tensor parallelism, and different processor groups correspond to different model attention heads; the processor cluster generates a model inference result corresponding to the task sequence based on the sequence output results generated by each processor executing the task sub-sequences.

[0006] In the scheme shown in this application, tensor parallelism and sequence parallelism are combined. Model weights are divided based on tensor parallelism, and task sequences are divided based on sequence parallelism. This allows different processors in the processor cluster to use their respective attention heads to process corresponding task subsequences. That is, processors with the same attention head process different task subsequences, while processors processing the same task subsequences correspond to different attention heads. This enables different processors to cache different key-value tensors during model inference, avoiding the problem of duplicate key-value tensor caching during model inference and improving the memory access efficiency of the distributed inference system.

[0007] In one possible implementation, the processor cluster generates model inference results corresponding to the task sequence based on the sequence output results generated by each processor executing the task sub-sequence, including: for the i-th processor in the processor group, the i-th processor pre-fills the i-th task sub-sequence based on the model sub-weights to obtain the key-value tensor and sequence feature representation corresponding to the i-th task sub-sequence, where i is a positive integer greater than 1; the i-th processor decodes the sequence feature representation based on the model sub-weights and the key-value tensor to obtain the intermediate decoding result corresponding to the i-th task sub-sequence; the i-th processor performs linear transformation and feedforward neural network processing on the intermediate decoding result based on device communication between processors to obtain the sequence output result corresponding to the task sequence; and the processor cluster generates model inference results corresponding to the task sequence based on the sequence output results output by each processor.

[0008] In the scheme shown in this application, the hybrid parallelism of tensor parallelism and sequence parallelism is applied to the pre-filling stage and the decoding stage of model inference. This can realize distributed caching of key-value tensors in the pre-filling stage and improve the memory access efficiency of key-value tensors in the decoding stage.

[0009] In one possible implementation, the i-th processor pre-fills the i-th task subsequence based on model sub-weights to obtain the key-value tensor and sequence feature representation corresponding to the i-th task subsequence. This includes: the i-th processor performing key-value calculation on the i-th task subsequence based on model sub-weights and a circular attention mechanism to obtain the key-value tensor corresponding to the i-th task subsequence; the i-th processor aggregating the key-value tensors transmitted by other processors based on intra-group communication within the processor group to obtain the intra-group key-value tensor of the task sequence on the i-th processor, wherein the intra-group key-value tensor includes key-value tensors of all sequence dimensions and key-value tensors of the attention head dimension corresponding to the i-th processor; and the i-th processor performing linear transformation and feedforward neural network processing on the intra-group key-value tensor based on global communication within the processor cluster to obtain the sequence feature representation corresponding to the i-th task subsequence.

[0010] In the scheme shown in this application, by constructing intra-group communication of the processor group and global communication within the processor cluster during the model pre-filling stage, it is possible to pre-fill the task subsequences based on different tensor parallelisms before and after intra-group communication, thereby improving the model memory access efficiency and reducing peak activation during the pre-filling stage.

[0011] In one possible implementation, the model subweights include subquery weights, subkey weights, and subvalue weights, which are partitioned based on the first tensor parallelism.

[0012] The aforementioned i-th processor performs key-value calculation on the i-th task subsequence based on model sub-weights and a circular attention mechanism to obtain the key-value tensor corresponding to the i-th task subsequence. This includes: the i-th processor performing key-value calculation on the i-th task subsequence based on the i-th sub-query weight, the i-th sub-key weight, and the i-th sub-value weight to obtain the i-th key-value tensor corresponding to the i-th task subsequence; the i-th processor receiving the i-1-th sub-key weight and the i-1-th sub-value weight transmitted by the (i-1)-th processor based on the circular attention mechanism, and sending the i-th sub-key weight and the i-th sub-value weight to the (i+1)-th processor; the i-th processor performing key-value calculation on the i-th task subsequence based on the received i-1-th sub-key weight and the i-1-th sub-value weight to obtain the i-1-th key-value tensor corresponding to the i-th task subsequence; and the i-th processor updating the i-th key-value tensor based on the i-1-th key-value tensor to obtain the key-value tensor corresponding to the i-th task subsequence.

[0013] In the scheme shown in this application, during the key-value calculation of the i-th task subsequence, the key-value weights are transferred between different processors according to the circular attention mechanism, which can reduce the communication cost between processors and enable each processor in the processor group to learn different task subsequences, thereby improving the key-value calculation efficiency.

[0014] In one possible implementation, the i-th processor aggregates key-value tensors transmitted by other processors based on intra-processor group communication to obtain an intra-group key-value tensor of the task sequence on the i-th processor. This includes: the i-th processor sending a portion of the key-value tensor corresponding to the i-th task sub-sequence to other processors in the processor group based on fully connected communication established within the processor group, wherein the portion of the key-value tensor corresponding to the i-th task sub-sequence is a key-value tensor of an attention head dimension other than the attention head dimension corresponding to the i-th processor; the i-th processor receiving a portion of the key-value tensor corresponding to other task sub-sequences sent by other processors in the processor group, wherein the portion of the key-value tensor corresponding to other task sub-sequences is a key-value tensor of the attention head dimension corresponding to the i-th processor; and the i-th processor aggregating the key-value tensors of each task sub-sequence in the attention head dimension corresponding to the i-th processor to obtain an intra-group key-value tensor of the task sequence on the i-th processor.

[0015] In the scheme shown in this application, by building fully connected communication within the processor group to realize the transfer of key-value tensors between different processors, each processor in the processor group can have key-value tensors of all sequence dimensions as well as key-value tensors of its own corresponding attention head dimension, which can realize full tensor parallelism in subsequent processing and improve the parallel efficiency of subsequent processing.

[0016] In one possible implementation, the model subweights include sublinear transformation weights and subfeedforward network weights, which are partitioned based on the second tensor parallelism.

[0017] The i-th processor, based on global communication within the processor cluster, performs linear transformation and feedforward neural network processing on the intra-group key-value tensors to obtain the sequence feature representation corresponding to the i-th task sub-sequence. This includes: the i-th processor performing linear transformation on the intra-group key-value tensors based on sub-linear transformation weights to obtain the linearly transformed intra-group key-value tensors; the processor cluster aggregating the linearly transformed intra-group key-value tensors on each processor based on the fully reduced communication established within the processor cluster, and distributing the aggregated global key-value tensors to each processor; the i-th processor performing feedforward neural network processing on the global key-value tensors based on sub-feedforward network weights to obtain a partial sequence feature representation corresponding to the task sequence, which is the sequence feature representation of the attention head dimension corresponding to the i-th processor; the processor cluster aggregating the partial sequence feature representations corresponding to the task sequence on each processor based on the fully reduced communication established within the processor cluster, and distributing the aggregated sequence feature representations corresponding to the task sequence to each processor; and the i-th processor performing sequence partitioning on the sequence feature representations corresponding to the task sequence to obtain the sequence feature representation corresponding to the i-th task sub-sequence.

[0018] In the scheme presented in this application, by dividing the sub-linear transformation weights and sub-feedforward network weights according to the parallelism of the second tensor, different processors can process their respective intra-group key-value tensors according to their respective sub-linear transformation weights and sub-feedforward network weights, thereby improving the parallel efficiency of the model. Furthermore, by establishing full reduction communication within the processor cluster and aggregating the key-value tensor or sequence feature representations on different processors, the communication overhead between processors can be reduced.

[0019] In one possible implementation, the model subweights include subquery weights, subkey weights, and subvalue weights, which are partitioned based on the first tensor parallelism.

[0020] The i-th processor decodes the sequence feature representation based on the model sub-weights and the key-value tensor to obtain the intermediate decoding result corresponding to the i-th task sub-sequence, including: the i-th processor performs key-value calculation on the sequence feature representation based on the i-th sub-query weight, the i-th sub-key weight, and the i-th sub-value weight to obtain the intermediate decoding result corresponding to the i-th task sub-sequence.

[0021] In the scheme shown in this application, during the model decoding stage, query weights, key weights, and value weights are divided according to the parallelism of the first tensor, thereby enabling different processors to decode the sequence feature representation in parallel, which can improve decoding efficiency.

[0022] In one possible implementation, the i-th processor performs linear transformation and feedforward neural network processing on the intermediate decoding results based on inter-processor device communication to obtain the sequence output result corresponding to the task sequence. This includes: the i-th processor aggregating the intermediate decoding results transmitted by other processors based on intra-group communication of the processor group to obtain the intra-group intermediate decoding result of the task sequence on the i-th processor. The intra-group intermediate decoding result includes the intermediate decoding results of all sequence dimensions and the intermediate decoding result of the attention head dimension corresponding to the i-th processor; the i-th processor performs linear transformation and feedforward neural network processing on the intra-group intermediate decoding results based on global communication within the processor cluster to obtain the sequence output result corresponding to the task sequence.

[0023] In the scheme shown in this application, during the model decoding stage, by constructing intra-group communication of the processor group and global communication within the processor cluster, decoding based on different tensor parallelism can be achieved before and after intra-group communication, thereby improving the model memory access efficiency and reducing peak activation during the decoding stage.

[0024] In one possible implementation, the i-th processor, based on intra-group communication within the processor group, aggregates intermediate decoding results transmitted by other processors to obtain intra-group intermediate decoding results of the task sequence on the i-th processor. This includes: the i-th processor sending partial intermediate decoding results corresponding to the i-th task sub-sequence to other processors within the processor group based on fully connected communication established within the processor group; the partial intermediate decoding results corresponding to the i-th task sub-sequence being intermediate decoding results for attention head dimensions other than the attention head dimension corresponding to the i-th processor; the i-th processor receiving partial intermediate decoding results corresponding to other task sub-sequences sent by other processors within the processor group; the partial intermediate decoding results corresponding to other task sub-sequences being intermediate decoding results for the attention head dimension corresponding to the i-th processor; and the i-th processor aggregating the intermediate decoding results of each task sub-sequence on the attention head dimension corresponding to the i-th processor to obtain intra-group intermediate decoding results of the task sequence on the i-th processor.

[0025] In the scheme shown in this application, by building fully connected communication within the processor group, the intermediate decoding results can be transferred between different processors. This allows each processor in the processor group to have intermediate decoding results for all sequence dimensions as well as intermediate decoding results for its corresponding attention head dimension. This enables full tensor parallelism in subsequent processing and improves the parallel efficiency of subsequent processing.

[0026] In one possible implementation, the model subweights include sublinear transformation weights and subfeedforward network weights, which are partitioned based on the second tensor parallelism.

[0027] The i-th processor, based on global communication within the processor cluster, performs linear transformation and feedforward neural network processing on the intermediate decoding results within the group to obtain the sequence output results corresponding to the task sequence. This includes: the i-th processor performing linear transformation processing on the intermediate decoding results within the group based on sub-linear transformation weights to obtain the intermediate decoding results within the group after linear transformation; the processor cluster aggregating the intermediate decoding results within the group after linear transformation on each processor based on the full reduction communication established within the processor cluster, and distributing the aggregated global intermediate decoding results to each processor; the i-th processor performing feedforward neural network processing on the global intermediate decoding results based on sub-feedforward network weights to obtain the partial sequence output results corresponding to the task sequence, where the partial sequence output results are the sequence output results of the attention head dimension corresponding to the i-th processor; and the processor cluster aggregating the partial sequence output results corresponding to the task sequence on each processor based on the full reduction communication established within the processor cluster, and distributing the aggregated sequence output results corresponding to the task sequence to each processor.

[0028] In the scheme presented in this application, by dividing the sub-linear transform weights and sub-feedforward network weights according to the parallelism of the second tensor, different processors can process their respective intra-group intermediate decoding results according to their respective sub-linear transform weights and sub-feedforward network weights, thereby improving the model parallel efficiency. Furthermore, by establishing full reduction communication within the processor cluster and aggregating the intra-group intermediate decoding results or sequence output results on different processors, the communication overhead between processors can be reduced.

[0029] In one possible implementation, the method further includes: the scheduling device determines the processor with the smallest key-value cache size as the target processor in the processor group based on the key-value cache size of each processor in the processor group; after obtaining the sequence output result corresponding to the task sequence, the target processor caches the sequence output result in the key-value cache.

[0030] In the scheme shown in this application, by comparing the key-value cache size of each processor, the sequence output result is cached in the processor with the smallest current key-value cache size, which can achieve balanced storage among different processors and ensure load balance among processors.

[0031] Secondly, a model inference system is provided, which includes a scheduling device and a processor cluster;

[0032] The scheduling device is used to divide the task sequence corresponding to the model inference request based on the sequence parallelism when a model inference request is received, and obtain multiple task sub-sequences.

[0033] The scheduling device is used to allocate multiple task subsequences to various processor groups in the processor cluster. The number of processors in a processor group is the same as the sequence parallelism, and there is a one-to-one correspondence between processors and task subsequences within a processor group. Each processor in a processor group is configured with model subweights, which are obtained by partitioning the model weights based on tensor parallelism. Different processor groups correspond to different model attention heads.

[0034] The processor cluster is used to generate model inference results corresponding to the task sequence based on the sequence output results generated by each processor executing the task sub-sequence.

[0035] In one possible design, for the i-th processor within the processor cluster, the i-th processor is used to pre-fill the i-th task subsequence based on the model sub-weights to obtain the key-value tensor and sequence feature representation corresponding to the i-th task subsequence, where i is a positive integer greater than 1; the i-th processor is used to decode the sequence feature representation based on the model sub-weights and key-value tensor to obtain the intermediate decoding result corresponding to the i-th task subsequence; the i-th processor is used to perform linear transformation and feedforward neural network processing on the intermediate decoding result based on device communication between processors to obtain the sequence output result corresponding to the task sequence; the processor cluster is used to generate the model inference result corresponding to the task sequence based on the sequence output results output by each processor.

[0036] In one possible design, the i-th processor is used to: perform key-value calculation on the i-th task subsequence based on model sub-weights and a ring attention mechanism to obtain the key-value tensor corresponding to the i-th task subsequence; aggregate the key-value tensors transmitted by other processors based on intra-group communication of the processor group to obtain the intra-group key-value tensor of the task sequence on the i-th processor, wherein the intra-group key-value tensor includes the key-value tensors of all sequence dimensions and the key-value tensor of the attention head dimension corresponding to the i-th processor; and perform linear transformation and feedforward neural network processing on the intra-group key-value tensor based on global communication within the processor cluster to obtain the sequence feature representation corresponding to the i-th task subsequence.

[0037] Optionally, the model sub-weights include subquery weights, subkey weights, and subvalue weights, which are partitioned based on the parallelism of the first tensor.

[0038] In one possible design, the i-th processor is configured to: perform key-value calculation on the i-th task subsequence based on the i-th subquery weight, the i-th subkey weight, and the i-th subvalue weight, to obtain the i-th key-value tensor corresponding to the i-th task subsequence; receive the i-1-th subkey weight and the i-1-th subvalue weight transmitted by the (i-1)-th processor based on a circular attention mechanism, and send the i-th subkey weight and the i-th subvalue weight to the (i+1)-th processor; perform key-value calculation on the i-th task subsequence based on the received i-1-th subkey weight and the i-1-th subvalue weight, to obtain the i-1-th key-value tensor corresponding to the i-th task subsequence; and update the i-th key-value tensor based on the i-1-th key-value tensor to obtain the key-value tensor corresponding to the i-th task subsequence.

[0039] In one possible design, the i-th processor is configured to: send a partial key-value tensor corresponding to the i-th task subsequence to other processors in the processor group based on the fully connected communication established within the processor group, wherein the partial key-value tensor corresponding to the i-th task subsequence is a key-value tensor of attention head dimensions other than the attention head dimension corresponding to the i-th processor; receive partial key-value tensors corresponding to other task subsequences sent by other processors in the processor group, wherein the partial key-value tensors corresponding to other task subsequences are key-value tensors of attention head dimensions corresponding to the i-th processor; and aggregate the key-value tensors of each task subsequence in the attention head dimension corresponding to the i-th processor to obtain the intra-group key-value tensor of the task sequence on the i-th processor.

[0040] Optionally, the model sub-weights include sub-linear transformation weights and sub-feedforward network weights, which are partitioned based on the second tensor parallelism.

[0041] In one possible design, the i-th processor performs a linear transformation on the in-group key-value tensor based on the sub-linear transformation weights to obtain the linearly transformed in-group key-value tensor; the processor cluster aggregates the linearly transformed in-group key-value tensors on each processor based on the full reduction communication established within the processor cluster, and distributes the aggregated global key-value tensor to each processor; the i-th processor performs a feedforward neural network process on the global key-value tensor based on the sub-feedforward network weights to obtain a partial sequence feature representation corresponding to the task sequence, which is the sequence feature representation of the attention head dimension corresponding to the i-th processor; the processor cluster aggregates the partial sequence feature representations corresponding to the task sequence on each processor based on the full reduction communication established within the processor cluster, and distributes the aggregated sequence feature representations corresponding to the task sequence to each processor; the i-th processor performs sequence partitioning on the sequence feature representations corresponding to the task sequence to obtain the sequence feature representations corresponding to the i-th task subsequence.

[0042] Optionally, the model sub-weights include subquery weights, subkey weights, and subvalue weights, which are partitioned based on the parallelism of the first tensor.

[0043] In one possible design, the i-th processor is used to: perform key-value calculation on the sequence feature representation based on the i-th subquery weight, the i-th subkey weight, and the i-th subvalue weight, to obtain the intermediate decoding result corresponding to the i-th task subsequence.

[0044] In one possible design, the i-th processor is used to: aggregate intermediate decoding results transmitted by other processors based on intra-processor group communication to obtain intra-group intermediate decoding results of the task sequence on the i-th processor, the intra-group intermediate decoding results including intermediate decoding results of all sequence dimensions and intermediate decoding results of the attention head dimension corresponding to the i-th processor; and perform linear transformation and feedforward neural network processing on the intra-group intermediate decoding results based on global communication within the processor cluster to obtain the sequence output results corresponding to the task sequence.

[0045] In one possible design, the i-th processor is configured to: send partial intermediate decoding results corresponding to the i-th task subsequence to other processors in the processor group based on the fully connected communication established within the processor group, wherein the partial intermediate decoding results corresponding to the i-th task subsequence are intermediate decoding results of attention head dimensions other than the attention head dimension corresponding to the i-th processor; receive partial intermediate decoding results corresponding to other task subsequences sent by other processors in the processor group, wherein the partial intermediate decoding results corresponding to other task subsequences are intermediate decoding results of the attention head dimension corresponding to the i-th processor; and aggregate the intermediate decoding results of each task subsequence in the attention head dimension corresponding to the i-th processor to obtain the intra-group intermediate decoding result of the task sequence on the i-th processor.

[0046] Optionally, the model sub-weights include sub-linear transformation weights and sub-feedforward network weights, which are partitioned based on the second tensor parallelism.

[0047] In one possible design, the i-th processor is used to perform linear transformation processing on the intermediate decoding results within the group based on the sub-linear transformation weights, to obtain the intermediate decoding results within the group after linear transformation; the processor cluster is used to aggregate the intermediate decoding results within the group after linear transformation on each processor based on the full reduction communication established within the processor cluster, and distribute the aggregated global intermediate decoding results to each processor; the i-th processor is used to perform feedforward neural network processing on the global intermediate decoding results based on the sub-feedforward network weights, to obtain the partial sequence output results corresponding to the task sequence, and the partial sequence output results are the sequence output results of the attention head dimension corresponding to the i-th processor; the processor cluster is used to aggregate the partial sequence output results corresponding to the task sequence on each processor based on the full reduction communication established within the processor cluster, and distribute the aggregated sequence output results corresponding to the task sequence to each processor.

[0048] In one possible design, a scheduling device is used to determine the processor with the smallest key-value cache size as the target processor in the processor group based on the key-value cache size of each processor in the processor group; the target processor is used to: cache the sequence output result corresponding to the task sequence into the key-value cache after obtaining the sequence output result.

[0049] Thirdly, a model inference apparatus is provided, the apparatus comprising:

[0050] The sequence partitioning module is used to divide the task sequence corresponding to the model inference request based on the sequence parallelism when a model inference request is received, and obtain multiple task sub-sequences.

[0051] The sequence allocation module is used to allocate multiple task subsequences to various processor groups in the processor cluster. The number of processors in a processor group is the same as the sequence parallelism, and there is a one-to-one correspondence between processors and task subsequences within a processor group. Each processor in a processor group is configured with model subweights, which are obtained by partitioning the model weights based on tensor parallelism. Different processor groups correspond to different model attention heads.

[0052] Fourthly, a computer-readable storage medium is provided that stores a program or instructions, wherein when the program or instructions are run on a computer, the methods described in the above aspects are executed.

[0053] Fifthly, a computer program (product) is provided, comprising: computer program code, which, when executed by a computer, causes the computer to perform the methods described in the above aspects.

[0054] In a sixth aspect, a chip is provided, including a processor for retrieving and executing instructions stored in memory, causing a computer equipped with the chip to perform the methods described in the foregoing aspects.

[0055] In a seventh aspect, another chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, the output interface, the processor, and the memory are connected via an internal connection path, the processor is used to execute code in the memory, and when the code is executed, a computer with the chip installed performs the methods in the above aspects.

[0056] It should be understood that the beneficial effects of the technical solutions and corresponding possible implementations of the second to seventh aspects of this application can be found in the above description of the technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0057] Figure 1 This is a schematic diagram illustrating the use of tensor parallelism to partition model weights, as provided in related technologies.

[0058] Figure 2 This is a schematic diagram illustrating the inference process executed using tensor parallelism technology, as provided in related technologies.

[0059] Figure 3 This is a schematic diagram illustrating the use of sequence parallelism to segment the input sequence, as provided in related technologies.

[0060] Figure 4 This is a schematic diagram illustrating the inference process executed using sequential parallelism technology, as provided in related technologies.

[0061] Figure 5 A schematic diagram illustrating the implementation environment of a model reasoning method provided in this application embodiment;

[0062] Figure 6 A flowchart of a model reasoning method provided in an embodiment of this application;

[0063] Figure 7 A schematic diagram illustrating a distributed model inference parallel strategy provided in an embodiment of this application;

[0064] Figure 8 A flowchart of a model reasoning method provided in an embodiment of this application;

[0065] Figure 9A flowchart of the pre-filling stage in the model inference process provided in this application embodiment;

[0066] Figure 10 This application provides a schematic diagram of the structure of the pre-filling stage in a model inference process.

[0067] Figure 11 A flowchart of the decoding stage in a model inference process is provided for an embodiment of this application;

[0068] Figure 12 This application provides a schematic diagram of the structure of the decoding stage in a model inference process.

[0069] Figure 13 This is a schematic diagram of the structure of a model inference system provided in an embodiment of this application;

[0070] Figure 14 A flowchart of a scheduling model for a scheduling device in a model inference system provided in this application embodiment;

[0071] Figure 15 This is a schematic diagram of the structure of a model inference device provided in an embodiment of this application;

[0072] Figure 16 A schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0073] Figure 17 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0074] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0075] First, a brief introduction to the names involved in the embodiments of this application:

[0076] Processor: A device used to perform various computing tasks. Optionally, the processor may be a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), etc., but this application embodiment does not limit it.

[0077] Tensor parallelism (TP) is a technique that partitions tensors (multidimensional arrays or matrices) in a deep learning model according to specific dimensions and distributes them across different processors for parallel computation. Each processor is responsible for only a portion of the computational task. (Illustrative example follows.) Figure 1 As shown, there are 4 NPUs, namely NPU1, NPU2, NPU3 and NPU4. The model weights can be divided into 4 parts and assigned to the corresponding 4 NPUs by tensor parallelism.

[0078] Indicative Figure 2 This diagram illustrates a method for executing an inference process using tensor parallelism, as provided in related technologies. For example... Figure 2 As shown, taking the task sequence input to the model as having dimensions (b, s, h) and the model weights as having dimensions (h, h) as an example, when using tensor parallelism, the entire task sequence is distributed to each processor, but the scheduling device only loads a portion of the model weights (h, h / TP) to each processor. Thus, each processor processes the entire task sequence based on a portion of the model weights. After attention calculation and distributed caching of key-value (KV) tensors, the model output is obtained after processing by a feedforward neural network (FFN).

[0079] Sequence parallelism (SP) is a technique that divides the input sequence of a deep learning model according to a specific dimension and distributes it across different processors for parallel computation. Each processor is responsible for only a portion of the computational task. (Illustrative example follows.) Figure 3 As shown, there are 4 NPUs, namely NPU1, NPU2, NPU3 and NPU4. The input sequence can be divided into 4 parts by using sequence parallelism and assigned to the corresponding 4 NPUs.

[0080] Indicative Figure 4 This diagram illustrates a method for executing an inference process using sequential parallelism, as provided in related technologies. For example... Figure 4As shown, taking the task sequence input to the model as having dimensions (b, s, h) and the model weights as having dimensions (h, h) as an example, when using sequence parallelism, the task sequence is divided and distributed to each processor. However, the scheduling device loads the full set of model weights (h, h) to each processor. Thus, each processor processes the task subsequence (b, s / SP, h) based on the full set of model weights. After attention calculation and distributed caching of key-value (KV) tensors, the model output is obtained after processing by a feedforward neural network (FFN).

[0081] With the continuous development and popularization of deep learning technology, distributed inference for large models has gradually become a hot topic of common concern in academia and industry. Distributed inference includes various parallelism methods, such as data parallelism, sequence parallelism, tensor parallelism, and pipelined parallelism.

[0082] In related technologies, when implementing distributed inference using tensor parallelism, key-value weights need to be segmented along the attention head dimension based on the tensor parallelism degree. This means the tensor parallelism degree is strongly dependent on the number of attention heads in the model. However, for models employing grouped query attention techniques, since this technique reduces the number of attention heads by decreasing the number of attention heads, thus reducing the cache space occupied by key-value weights and key-value tensors, the number of attention heads may be less than the number of processors. When the number of attention heads is less than the number of processors, at least two processors may use the same attention head to process the task sequence and cache the same key-value tensors, leading to duplicate key-value tensor caching and consequently impacting the memory access efficiency of the model's inference system.

[0083] In this embodiment, to reduce key-value cache duplication, sequence parallelism is added on top of tensor parallelism. By combining tensor parallelism and sequence parallelism, model weights are divided based on tensor parallelism, and task sequences are divided based on sequence parallelism. This allows different processors in the processor cluster to use their respective attention heads to process corresponding task subsequences. That is, processors with the same attention head process different task subsequences, while processors processing the same task subsequences correspond to different attention heads. This enables different processors to cache different key-value tensors during model inference, avoiding the problem of duplicate key-value tensor caching during model inference and improving the memory access efficiency of the distributed inference system.

[0084] The model reasoning method proposed in this application will be described below through specific embodiments.

[0085] Please refer to Figure 5This diagram illustrates an implementation environment for a model inference method provided in this application. The implementation environment includes a terminal 501 and a model inference system 502. The terminal 501 and the model inference system 502 communicate via a communication network. Optionally, the communication network can be a wired network or a wireless network, and can be at least one of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0086] Terminal 501 is an electronic device with an application having inference query functionality installed. This inference query functionality can be a native application of terminal 501 or a third-party application; terminal 501 can be a smartphone, tablet, laptop, desktop computer, smart TV, wearable device, or in-vehicle terminal, etc. Figure 5 The example of a desktop computer, terminal 501, is used for illustration only, but this is not a limitation.

[0087] The model inference system 502 includes a scheduling device and a processor cluster. Optionally, the scheduling device and the processor cluster can be deployed on the same server, on different servers, or on a cloud computing platform or virtualization center.

[0088] Optionally, in the model inference system 502, the scheduling device can divide the model weights according to the tensor parallelism to obtain multiple model sub-weights, the number of which is consistent with the tensor parallelism. It can also group the processors based on the sequence parallelism to obtain multiple processor groups, the number of processors in each group being consistent with the sequence parallelism. Furthermore, by configuring the model sub-weights on the processors of each processor group, the scheduling device can ensure that different processor groups correspond to different model attention heads.

[0089] In some embodiments, data interaction exists between the model inference system 502 and the terminal 501. Upon receiving a model inference request from the terminal 501, the scheduling device in the model inference system 502 divides the task sequence corresponding to the model inference request based on the sequence parallelism, obtaining multiple task sub-sequences. These sub-sequences are then assigned to various processor groups, ensuring a one-to-one correspondence between processors within a processor group and the task sub-sequences. Specifically, processors with the same attention head process different task sub-sequences, while processors processing the same task sub-sequences correspond to different attention heads. The processor cluster then generates the model inference result corresponding to the task sequence based on the sequence output results generated by each processor executing the task sub-sequences. Finally, the model inference system 502 returns the model inference result to the terminal 501.

[0090] Please refer to Figure 6 The diagram illustrates a flowchart of a model inference method provided in an embodiment of this application. This method can be applied to... Figure 5The implementation environment shown. For example... Figure 6 As shown, the inference method of this model includes the following steps 610 to 630.

[0091] Step 610: Upon receiving a model inference request, the scheduling device divides the task sequence corresponding to the model inference request based on the sequence parallelism, resulting in multiple task sub-sequences.

[0092] Optionally, a model inference request refers to a request initiated by the client to the model inference system after model configuration to obtain the model's inference results. Optionally, the task sequence included in the model inference request refers to the data input to the model to facilitate inference. Optionally, the task sequence can be text data, numerical data, image data, etc., and this embodiment of the application does not limit this.

[0093] In one possible implementation, upon receiving a model inference request, in order to achieve sequence parallelism, the scheduling device can divide the task sequence corresponding to the model inference request according to the sequence parallelism degree, thereby obtaining a sequence parallelism degree of task subsequences. Optionally, if the sequence parallelism degree SP = n, then the scheduling device needs to divide the task sequence into n task subsequences.

[0094] Sequence parallelism characterizes the degree of parallel processing where a task sequence is divided across multiple processors. Combined with... Figure 3 As shown, the task sequence is divided into 4 task subsequences, which are distributed to 4 NPUs respectively, i.e., the sequence parallelism SP = 4.

[0095] Indicative, such as Figure 7 As shown, taking a task sequence with dimensions (b, s, h) as an example, the scheduling device divides the task sequence according to the sequence parallelism to obtain two task sub-sequences with dimensions (b, s / 2, h).

[0096] Step 620: The scheduling device assigns multiple task subsequences to various processor groups in the processor cluster. The number of processors in a processor group is the same as the parallelism of the sequence. Each processor in a processor group corresponds one-to-one with a task subsequence. Each processor in a processor group is configured with a model subweight. The model subweight is obtained by dividing the model weights based on the tensor parallelism. Different processor groups correspond to different model attention heads.

[0097] Optionally, after dividing the task into multiple task subsequences, the scheduling device can distribute the task subsequences to each processor. In one possible implementation, the scheduling device can allocate the multiple task subsequences to each processor group on a processor group basis.

[0098] In one possible design, the model inference system includes a processor cluster comprising multiple processors. To ensure that processors processing the same task sequence correspond to different attention heads, a scheduling device can partition the model weights based on tensor parallelism to obtain multiple model sub-weights, the number of which equals the tensor parallelism. This allows the model sub-weights to be allocated across the processors, resulting in different processor groups corresponding to different model attention heads and different model sub-weights.

[0099] Tensor parallelism characterizes the degree of parallel processing where model weights (tensors) are distributed across multiple processors. Combined with... Figure 1 As shown, the model weights are divided into 4 model sub-weights, which are respectively configured on 4 NPUs, i.e., tensor parallelism TP = 4.

[0100] Optionally, the model can be a large language model (LLM), a graph neural network model, or other types of large-scale models; this embodiment of the application does not limit this. Optionally, the model weights can include query weights W. Q Key weight W K Value weight W V Linear transformation weight W O and feedforward network weights W FFN Among them, query weights, key weights, and value weights are used for multi-head attention to learn sequence features, linear transformation weights are used to perform linear transformation processing on the output of multi-head attention, and feedforward network weights are used to extract sequence features through nonlinear transformation.

[0101] In one possible design, the tensor parallelism is less than or equal to the number of processors. For example, if the number of processors is 4 and the tensor parallelism is 2, the sequence parallelism can be set to 2. In this case, the model weights are divided into 2 model sub-weights. Each processor group contains 2 processors, and each processor group corresponds to one model sub-weight. That is, the processors in each processor group are configured with the same model sub-weights.

[0102] For models employing grouped query attention, the number of query attention heads can be greater than the number of key-value attention heads. This means that grouped query attention requires grouping query attention heads, with each group corresponding to one key-value attention head. For example, in a processor, every two query attention heads might correspond to one key-value attention head. By using grouped query attention, it's possible to reduce computational load and caching requirements while maintaining model performance.

[0103] In this embodiment, when the number of key-value attention heads is greater than the tensor parallelism and equal to the number of processors, one processor can correspond to one key-value attention head. When the number of key-value attention heads is equal to the tensor parallelism but less than the number of processors, one processor group can correspond to one key-value attention head. That is, processors within a processor group correspond to the same key-value attention head but process different task subsequences. In other words, different processors generate different key-value tensors by processing task subsequences through key-value attention heads, which avoids duplicate caching of key-value tensors.

[0104] Optionally, the number of processors in a processor group is equal to the sequence parallelism, and the number of task subsequences is equal to the sequence parallelism, so that there is a one-to-one correspondence between processors in the processor group and task subsequences.

[0105] Optionally, different processor groups correspond to their own model sub-weights, and each processor group corresponds to a complete task sequence. Therefore, for a certain task sub-sequence, it can be processed by processors in different processor groups based on different model sub-weights.

[0106] Indicative, such as Figure 7 As shown, taking a processor cluster with 4 processors, tensor parallelism TP=2, and sequence parallelism SP=2 as an example, the scheduling module can divide the model weights into two model sub-weights, and group the two processors into a processor group. The first processor group includes processor a1 and processor a2, corresponding to attention heads head0 and head1. The second processor group includes processor b1 and processor b2, corresponding to attention heads head2 and head3. The weight dimension of the model sub-weights in each processor group is h / 2. The scheduling device distributes the two task sub-sequences to one processor group. Thus, processor a1 in the first processor group is responsible for processing the task sub-sequence "this is", and processor a2 is responsible for processing the task sub-sequence "seq parallel". Similarly, processor b1 in the second processor group is responsible for processing the task sub-sequence "this is", and processor b2 is responsible for processing the task sub-sequence "seq parallel".

[0107] Step 630: The processor cluster generates model inference results corresponding to the task sequence based on the sequence output results generated by each processor executing the task sub-sequence.

[0108] In one possible implementation, after determining the model sub-weights corresponding to each processor and receiving the corresponding task sub-sequence, the processor cluster can execute the task sub-sequence in parallel through the processors of each processor group according to the model sub-weights configured on each processor, thereby obtaining the model inference result corresponding to the task sequence based on the sequence output results generated by each processor executing the task sub-sequence.

[0109] exist Figure 6 The method shown combines tensor parallelism with sequence parallelism, divides model weights based on tensor parallelism, and divides task sequences based on sequence parallelism. This allows different processors in the processor cluster to use their respective attention heads to process corresponding task subsequences. That is, processors with the same attention head process different task subsequences, while processors processing the same task subsequences correspond to different attention heads. This enables different processors to cache different key-value tensors during model inference, avoiding the problem of duplicate key-value tensor caching during model inference and improving the memory access efficiency of the distributed inference system.

[0110] In some embodiments, the model inference process can be divided into a prefill phase and a decoding phase. The prefill phase refers to the computational process of generating the first output token after the model receives the task sequence. The prefill phase is responsible for processing the input task sequence and generating the first output token, preparing for the subsequent decoding phase. The decoding phase refers to the process of converting the internal feature representation generated in the prefill phase into an output sequence. The decoding phase uses an autoregressive approach, meaning that the generation of each output token depends on the previously generated token sequence.

[0111] Optionally, in the pre-filling stage, each processor processes the task sub-sequence in parallel to obtain the sequence feature representation corresponding to the task sub-sequence, and caches the calculated key-value tensor. In the decoding stage, the processor can decode the sequence feature representation generated in the pre-filling stage based on the key-value tensor, and then generate the sequence output result based on the decoding result. The model inference process in the pre-filling and decoding stages will be described below through specific embodiments.

[0112] Please refer to Figure 8 The diagram illustrates a flowchart of a model inference method provided in an embodiment of this application. This method can be applied to... Figure 5 The implementation environment shown.

[0113] Step 631: For the i-th processor in the processor group, the i-th processor performs pre-filling processing on the i-th task subsequence based on the model sub-weights to obtain the key-value tensor and sequence feature representation corresponding to the i-th task subsequence, where i is a positive integer and greater than 1.

[0114] Optionally, if the tensor parallelism is m, then the model weights are divided into m model sub-weights, the processor cluster includes m processor groups, and each processor group corresponds to its own model sub-weights.

[0115] Optionally, if the sequence parallelism is n, the task sequence is divided into n task subsequences. Each processor group includes n processors, and each processor corresponds to a different task subsequence. For the i-th processor in the processor group, the i-th processor is responsible for processing the i-th task subsequence.

[0116] In one possible implementation, after the scheduling device configures the model sub-weights on the i-th processor and receives the i-th task sub-sequence distributed by the scheduling device, the i-th processor can pre-fill the i-th task sub-sequence according to the model sub-weights, thereby obtaining the key-value tensor and sequence feature representation corresponding to the i-th task sub-sequence.

[0117] In one possible design, the pre-filling stage comprises three sub-stages: attention learning, linear transformation, and feedforward neural network processing. In the attention learning stage, the processor performs self-attention mechanism calculations, using QKV weights to compute key values ​​for the task subsequences, obtaining the corresponding key-value tensors. In the linear transformation stage, the processor performs a linear transformation on the output of the self-attention layer, obtaining the linearly transformed key-value tensors. In the feedforward neural network processing stage, the processor performs non-linear activation processing on the key-value tensors to output the sequence feature representations corresponding to the task subsequences.

[0118] Optionally, the model includes multiple network layers, each containing a self-attention mechanism and a feedforward neural network. During the pre-filling stage, the processor can pre-fill the task subsequences through multiple network layers, while caching the key-value tensors generated by each network layer, and finally outputting the sequence feature representations corresponding to the task subsequences.

[0119] Step 632: The i-th processor decodes the sequence feature representation based on the model sub-weights and key-value tensors to obtain the intermediate decoding result corresponding to the i-th task sub-sequence.

[0120] In one possible implementation, after obtaining the key-value tensor and sequence feature representation corresponding to the i-th task subsequence, the i-th processor can retrieve the key-value tensor from the key-value cache and decode the sequence feature representation corresponding to the i-th task subsequence according to the model subweights and the key-value tensor, thereby obtaining the intermediate decoding result corresponding to the i-th task subsequence.

[0121] Optionally, during the self-attention learning process in the decoding stage, the processor performs the calculation of the self-attention mechanism, calculates the key value of the sequence feature representation through QKV weights, and thus obtains the intermediate decoding result corresponding to the i-th task subsequence.

[0122] The QKV weights in the decoding stage and the QKV weights in the pre-filling stage are not the same; both are determined based on the model training process.

[0123] Step 633: The i-th processor performs linear transformation and feedforward neural network processing on the intermediate decoding result based on device communication between processors to obtain the sequence output result corresponding to the task sequence.

[0124] Optionally, after obtaining the intermediate decoding result corresponding to the i-th task subsequence, the i-th processor can obtain the subsequence output result corresponding to the i-th task subsequence by performing a linear transformation and feedforward neural network processing on the intermediate decoding result. However, this subsequence output result only corresponds to the i-th task subsequence and is obtained only based on the model sub-weights calculated on the i-th processor. Therefore, in order to obtain the sequence output result corresponding to the complete task sequence during the decoding stage, it is also necessary to establish device communication between processors.

[0125] In one possible implementation, after each processor obtains the intermediate decoding results of its corresponding task sub-sequence, it can acquire the intermediate decoding results of other task sub-sequences on different processors through intra-processor communication. The i-th processor then performs a linear transformation on the intermediate decoding results based on the intermediate decoding results of the complete task sequence under a single model sub-weight. Furthermore, to generate the sequence output result based on the global model weight, global communication between the processor clusters can be used to acquire the linearly transformed intermediate decoding results under different model sub-weights on other processors. Thus, through feedforward neural network processing and global communication, the sequence output result corresponding to the task sequence can be obtained.

[0126] Step 634: The processor cluster generates model inference results corresponding to the task sequence based on the output results of each processor's sequence.

[0127] Optionally, the decoding stage adopts an autoregressive approach, that is, the processor outputs only one sequence output result in each round. After multiple rounds of output, the processor cluster can generate the model inference result corresponding to the task sequence based on the multiple rounds of sequence output results output by each processor.

[0128] For example, when the model inference result is a text statement, the processor outputs only one token per round. After obtaining multiple rounds of token output, the processor cluster can convert the token output in each round into text by looking up a table, thereby obtaining the model inference result.

[0129] exist Figure 8 The method shown applies a hybrid parallel approach of tensor parallelism and sequence parallelism to the pre-filling and decoding stages of model inference. This enables distributed storage of key-value tensors during the pre-filling stage, avoiding duplicate key-value tensor caching, while improving the memory access efficiency of key-value tensors during the decoding stage, thereby improving the parallel processing efficiency of the model.

[0130] The pre-filling stage in the model inference process will be explained below through specific examples.

[0131] Please refer to Figure 9 This document illustrates a flowchart of the pre-filling stage in the model inference process provided in an embodiment of this application. This method can be applied to... Figure 5 The implementation environment shown.

[0132] Step 901: The i-th processor performs key-value calculation on the i-th task subsequence based on the model sub-weights and the ring attention mechanism to obtain the key-value tensor corresponding to the i-th task subsequence.

[0133] Optionally, by distributing multiple task subsequences to each processor within the processor group, so that each processor corresponds to a task subsequence, each processor within the processor group can learn its corresponding task subsequence using a fast attention mechanism (FA) during the self-attention learning process in the pre-filling stage.

[0134] Among them, the fast attention mechanism is a technique used to accelerate self-attention computation and reduce memory usage. The fast attention mechanism divides the task sequence of the model input, distributing the computation process across different processors, thereby accelerating the computation process and reducing memory access overhead.

[0135] In traditional models, self-attention mechanisms require calculating the attention weights of each element in the task sequence with respect to all other elements. However, when the task sequence is divided into parallel sequences and different processors process the corresponding task subsequences, a circular attention mechanism needs to be added between the processors in the processor group to achieve global attention. This mechanism passes key weights and value weights between different processors so that each processor can calculate the relationship between its corresponding task subsequence and the global key weights and value weights.

[0136] In one possible implementation, after the i-th processor performs key-value calculation on the i-th task subsequence according to the model sub-weights and obtains the key-value tensor corresponding to the i-th task subsequence, it also needs to receive the model sub-weights passed by other processors according to the circular attention mechanism, and perform key-value calculation on the i-th task subsequence based on the received model sub-weights to obtain the new key-value tensor corresponding to the i-th task subsequence. The original key-value tensor is then updated with the new key-value tensor to obtain the updated key-value tensor corresponding to the i-th task subsequence.

[0137] Optionally, during the self-attention calculation process, the model sub-weights include subquery weights, subkey weights, and subvalue weights. Optionally, the subquery weights, subkey weights, and subvalue weights are partitioned based on the parallelism of the first tensor, so that each processor within the processor group corresponds to its own subquery weight, subkey weight, and subvalue weight. Where the parallelism of the first tensor equals the number of processor groups, each processor within the processor group corresponds to the same subquery weight, subkey weight, and subvalue weight.

[0138] For example, with a processor count of 4, sequence parallelism SP = 2, and first tensor parallelism TP = 2, the query weight W... Q Key weight W K Value weight W V Each part is divided into two parts and assigned to different processor groups. Each processor group contains two processors, and the two processors correspond to the same subquery weight, subkey weight, and subvalue weight.

[0139] Optionally, ring attention is an attention mechanism for ultra-long sequence information that overcomes the memory bottleneck of a single device by distributing self-attention computation across multiple processors. In ring attention, the task sequence is switched into multiple task subsequences along the sequence dimension. Each processor exchanges subkey weights and subvalue weights through a ring network structure. After receiving new subkey weights and subvalue weights, each processor can continue to the next key-value calculation. That is, the i-th processor needs to calculate the partial attention score between the i-th task subsequence and the received subkey weights and subvalue weights.

[0140] Here, the i-th processor corresponds to the i-th subquery weight, the i-th subkey weight, and the i-th subvalue weight. In one possible implementation, after receiving the i-th task subsequence, the i-th processor first performs key-value calculation on the i-th task subsequence according to its own configured model subweights. That is, based on the i-th subquery weight, the i-th subkey weight, and the i-th subvalue weight, it performs key-value calculation on the i-th task subsequence to obtain the i-th key-value tensor corresponding to the i-th task subsequence. Furthermore, in order to calculate the relationship between the i-th task subsequence and other subkey and subvalue weights, the processors within the same processor group need to pass the subkey and subvalue weights to the next processor and receive the subkey and subvalue weights sent by the previous processor based on a circular attention mechanism.

[0141] In one possible design, based on a circular attention mechanism, the i-th processor receives the i-1th subkey weight and i-1th subvalue weight transmitted by the i-1th processor, and sends the i-th subkey weight and i-th subvalue weight to the i+1th processor. Based on the received i-1th subkey weight and i-1th subvalue weight, the i-th processor performs key-value calculation on the i-th task subsequence to obtain the i-1th key-value tensor corresponding to the i-th task subsequence, and updates the i-th key-value tensor based on the i-1th key-value tensor to obtain the key-value tensor corresponding to the i-th task subsequence.

[0142] Optionally, updating the i-th key-value tensor through the (i-1)-th key-value tensor can be achieved by the i-th processor aggregating the i-th key-value tensor and the (i-1)-th key-value tensor to obtain the key-value tensor corresponding to the i-th task subsequence.

[0143] Specifically, the subkey weights and subvalue weights transmitted between processors are all updated subkey weights and subvalue weights after their own key-value calculations. Optionally, the i-th subkey weight and i-th subvalue weight sent by the i-th processor to the i+1-th device are the i-th subkey weight and i-th subvalue weight after key-value calculations.

[0144] Furthermore, by cyclically passing subkey weights and subvalue weights among the processors in the processor group, the i-th processor can calculate the partial attention score between the i-th task subsequence and each subkey weight and subvalue weight. After each key-value calculation, the key-value tensor is iteratively updated, thus finally obtaining the key-value tensor corresponding to the i-th task subsequence.

[0145] For example, with a sequence parallelism of 3, each processor in the processor group first performs key-value calculations on its corresponding task subsequence based on QKV weights to obtain a key-value tensor and update the KV weights. Then, the first processor sends the calculated and updated first subkey weights and first subvalue weights to the second processor. Simultaneously, the first processor receives the third subkey weights and third subvalue weights from the third processor. Thus, the first processor, based on its own first subquery weights, the received third subkey weights, and the third subvalue weights, performs key-value calculations on the first task subsequence and updates the key-value tensor and KV weights. After one round of KV weight transmission, the first processor continues to send the updated first subkey weights and first subvalue weights to the second processor and executes a new round of key-value calculations until the cyclic attention calculation is completed, resulting in the final key-value tensor corresponding to the first task subsequence.

[0146] Indicative, such as Figure 10 As shown, taking the task sequence input to the model as having dimensions (b, s, h), sequence parallelism SP = 2, and first tensor parallelism TP = 2 as an example, for processor Rank0 and processor Rank2 within the processor group, processor Rank0 and processor Rank2 correspond to attention heads head0 and head1, respectively. Furthermore, using GQA technology, two query attention heads are grouped together to correspond to one key-value attention head. In this case, the dimension of the first sub-query weight in processor Rank0 is (b, s / 2, h / 2), and the dimension of the first sub-key-value weight is (b, s / 2, h / 4).

[0147] For processor Rank0, processor Rank0 first calculates the key value of the first task subsequence "this is" using the FA mechanism based on the first subquery weight Q, the first subkey weight K, and the first subvalue weight V, resulting in a key value tensor with dimensions (b, s / 2, h / 2). Then, through a circular attention mechanism, processor Rank0 passes the first subkey weight and the first subvalue weight to processor Rank2, and processor Rank2 passes the second subkey weight and the second subvalue weight to processor Rank0. Thus, processor Rank0 calculates the key value of the task subsequence again and updates the previous key value tensor based on the calculated key value tensor, obtaining the key value tensor corresponding to the first task subsequence. At this time, the dimension of the key value tensor Y is (b, s / 2, h / 2).

[0148] Step 902: Based on the intra-processor group communication, the i-th processor aggregates the key-value tensors transmitted by other processors to obtain the intra-group key-value tensor of the task sequence on the i-th processor. The intra-group key-value tensor includes the key-value tensors of all sequence dimensions and the key-value tensor of the attention head dimension corresponding to the i-th processor.

[0149] In one possible design, to further optimize model parallelism, different tensor parallelisms can be set in the pre-filling stage, i.e., the first tensor parallelism is used for parallel computation in the self-attention stage, and the second tensor parallelism is used for parallel computation in the subsequent linear transformation and feedforward neural network stages.

[0150] Optionally, in order to ensure the connection between the first tensor parallelism and the second tensor parallelism, after obtaining the key-value tensors output by each processor in the processor group, the key-value tensors output by each processor can be transposed through intra-group communication of the processor group, so that each processor has key-value tensors of all sequence dimensions and key-value tensors of its corresponding attention head dimension.

[0151] In one possible implementation, based on intra-group communication of the processor group, the i-th processor receives key-value tensors transmitted by other processors and sends key-value tensors to other processors. By combining its own output key-value tensors and the key-value tensors transmitted by other processors, the intra-group key-value tensors of the task sequence on the i-th processor can be obtained through aggregation processing.

[0152] The intra-group key-value tensor includes key-value tensors of all sequence dimensions as well as the key-value tensor of the attention head dimension corresponding to the i-th processor. That is, the intra-group key-value tensor is complete in the sequence dimension, but only contains the key-value tensor of the attention head dimension corresponding to the i-th processor.

[0153] Optionally, intra-group communication within a processor group can employ all-to-all communication. All-to-all communication is a global communication operation in parallel or distributed systems in which every processor participates, sending and receiving messages to and from all other processors.

[0154] In one possible implementation, based on the fully connected communication established within the processor group, the i-th processor can send a partial key-value tensor corresponding to the i-th task subsequence to other processors within the processor group. The partial key-value tensor corresponding to the i-th task subsequence is a key-value tensor of other attention head dimensions besides the attention head dimension corresponding to the i-th processor. The i-th processor also receives partial key-value tensors corresponding to other task subsequences sent by other processors within the processor group. The partial key-value tensors corresponding to other task subsequences are key-value tensors of the attention head dimension corresponding to the i-th processor. Thus, the i-th processor aggregates the key-value tensors of each task subsequence in the attention head dimension corresponding to the i-th processor to obtain the intra-group key-value tensor of the task sequence on the i-th processor.

[0155] Indicative, such as Figure 10As shown, based on the full-connection communication of the processor group, processor Rank0 passes the key-value tensor of the sequence dimension corresponding to head1 to processor Rank2, and processor Rank2 passes the key-value tensor of the sequence dimension corresponding to head0 to processor Rank0. Thus, processor Rank0 obtains the key-value tensors of all sequence dimensions under head0, and processor Rank2 obtains the key-value tensors of all sequence dimensions under head1. At this time, the dimension of the intra-group key-value tensor Y on each processor is (b,s,h / 4).

[0156] Step 903: The i-th processor performs linear transformation and feedforward neural network processing on the key-value tensor within the group based on global communication within the processor cluster to obtain the sequence feature representation corresponding to the i-th task subsequence.

[0157] Optionally, after obtaining the in-group key-value tensor, the i-th processor can perform linear transformation and feedforward neural network processing on the in-group key-value tensor according to the configured model sub-weights. Since the in-group key-value tensor corresponding to the i-th processor only has the key-value tensor of the attention head dimension corresponding to the i-th processor, during the linear transformation process, the model sub-weights configured on the i-th processor can only process the key-value tensor of the attention head dimension corresponding to the i-th processor. Therefore, in order to ensure that the model parameters on all processors are consistent and to achieve data synchronization between processors, global communication can be built within the processor cluster. Thus, during the linear transformation process and the feedforward neural network processing, based on the global communication between the processor clusters, the i-th processor outputs the sequence feature representation corresponding to the i-th task sub-sequence.

[0158] In the linear transformation and feedforward neural network processing, the model sub-weights include sub-linear transformation weights and sub-feedforward network weights. These sub-linear transformation weights and sub-feedforward network weights are partitioned based on the second tensor parallelism, thus each processor in the processor cluster corresponds to its own sub-linear transformation weights and sub-feedforward network weights. When the second tensor parallelism equals the number of processors, each processor corresponds to different sub-query weights, sub-key weights, and sub-value weights, achieving full tensor parallelism in both linear transformation and feedforward neural network processing.

[0159] In one possible design, the parallelism of the second tensor differs from that of the first tensor. To improve model parallelism, the parallelism of the second tensor can be greater than that of the first. For example, with 4 processors and a sequence parallelism SP = 2, the parallelism TP of the first tensor can be 2, and the parallelism TP of the second tensor can be 4. That is, in the linear transformation processing and feedforward neural network processing, the linear transformation weight W... O and feedforward network weights W FFNIt is divided into four parts and distributed to different processors, each processor being configured with 1 / 4 of the linear transformation weight W. O and feedforward network weights W FFN .

[0160] Optionally, global communication within the processor cluster can employ all-reduce communication. All-reduce communication is a type of aggregate communication operation in distributed computing, involving data exchange between multiple processors. In all-reduce communication, each processor sends its own data to all other processors and receives data from all other processors, thus each processor obtains a reduced result. Optionally, the reduction operation can be summation, finding the maximum value, finding the minimum value, etc.

[0161] In one possible implementation, the i-th processor first performs a linear transformation on the in-group key-value tensor according to the sub-linear transformation weights, obtaining a linearly transformed in-group key-value tensor. At this point, the in-group key-value tensor is complete in the sequence dimension, but only contains the key-value tensor corresponding to the attention head dimension of the i-th processor. Furthermore, to achieve data synchronization between processors, based on the fully reduced communication established within the processor cluster, the processor cluster aggregates the linearly transformed in-group key-value tensors from each processor and distributes the aggregated global key-value tensor to each processor. At this point, the global key-value tensor is complete in both the sequence dimension and the attention head dimension.

[0162] Furthermore, the i-th processor performs feedforward neural network processing on the global key-value tensor based on the sub-feedforward network weights to obtain a partial sequence feature representation corresponding to the task sequence. Since the global key-value tensor is processed by feedforward neural network only through the sub-feedforward network weights configured on the i-th processor, the partial sequence feature representation is only the sequence feature representation of the attention head dimension corresponding to the i-th processor. Therefore, in order to achieve data synchronization between processors, based on the full reduction communication established within the processor cluster, the processor cluster also needs to aggregate the partial sequence feature representations corresponding to the task sequences on each processor and distribute the aggregated sequence feature representations corresponding to the task sequences to each processor. Thus, after receiving the sequence feature representation corresponding to the complete task sequence, the i-th processor can obtain the sequence feature representation corresponding to the i-th task subsequence by performing sequence partitioning on the sequence feature representation.

[0163] Indicative, such as Figure 10As shown, processor Rank0 first performs a linear transformation on the in-group key-value tensor based on the sub-linear transformation weights, obtaining a linearly transformed in-group key-value tensor with dimensions (b, s, h / 4). Then, to achieve data synchronization between processors, based on the fully reduced communication established within the processor cluster, the processor cluster aggregates the linearly transformed in-group key-value tensors from processors Rank0-3 and distributes the aggregated global key-value tensor to each processor. At this point, the global key-value tensor O has dimensions (b, s, h). Further, processor Rank0 performs FFN processing on the global key-value tensor based on the sub-feedforward network weights, obtaining a partial sequence feature representation corresponding to the task sequence. The processor cluster, based on the fully reduced communication established within the processor cluster, again aggregates the partial sequence feature representation corresponding to the task sequence and distributes the sequence feature representation corresponding to the task sequence. At this point, the dimension of the sequence feature representation is (b, s, h). After receiving the sequence feature representation corresponding to the complete task sequence, processor Rank0 can obtain the sequence feature representation corresponding to the task subsequence by performing sequence segmentation (slice) on the sequence feature representation. The dimension of the sequence feature representation corresponding to the task subsequence is (b, s / 2, h).

[0164] exist Figure 9 The method described includes three processes in the pre-filling stage: attention calculation, linear transformation, and feedforward neural network processing. By combining tensor parallelism and sequence parallelism for distributed model inference, for a single processor in the processor cluster, the fast attention mechanism can be combined with the ring attention mechanism during the attention calculation process, reducing peak activation of the entire network and decreasing the cache size of key-value tensors. Furthermore, by setting the first tensor parallelism during the attention calculation process and the second tensor parallelism during the linear transformation and feedforward neural network processing, custom configuration of the parallelism can be achieved, effectively improving the efficiency of model parallel processing and reducing inference system latency.

[0165] The decoding stage in the model inference process will be explained below through specific examples.

[0166] Please refer to Figure 11 This document illustrates a flowchart of the decoding stage in the model inference process provided in an embodiment of this application. This method can be applied to... Figure 5 The implementation environment shown.

[0167] Step 1101: The i-th processor decodes the sequence feature representation based on the model sub-weights and key-value tensors to obtain the intermediate decoding result corresponding to the i-th task sub-sequence.

[0168] After obtaining the sequence feature representation corresponding to the i-th task subsequence through the pre-filling stage, in the decoding stage, the i-th processor can obtain the key-value tensor cached in the pre-filling stage from the key-value cache, and decode the sequence feature representation according to the model sub-weights and the key-value tensor, thereby obtaining the intermediate decoding result corresponding to the i-th task subsequence.

[0169] In one possible design, multiple task subsequences are distributed to each processor within the processor group, so that each processor corresponds one-to-one with a task subsequence. Thus, for each processor within the processor group, a fast decoding mechanism (flash decoding, FD) can be used to process the sequence feature representation of its corresponding task subsequence during the self-attention learning process in the decoding phase.

[0170] Among them, the fast decoding mechanism is a mechanism based on fast attention, which can improve the inference speed of the model during the decoding stage. The fast decoding mechanism accelerates the computation process and optimizes processor utilization by decoding the sequence feature representations of task subsequences in parallel on multiple processors.

[0171] Optionally, during the self-attention calculation process, the model sub-weights include subquery weights, subkey weights, and subvalue weights. These weights are partitioned based on the parallelism of the first tensor, thus assigning each processor within a processor group its own subquery weight, subkey weight, and subvalue weight. Specifically, when the parallelism of the first tensor equals the number of processor groups, each processor within a processor group receives the same subquery weight, subkey weight, and subvalue weight.

[0172] For example, with a processor count of 4, sequence parallelism SP = 2, and first tensor parallelism TP = 2, the query weight W... Q Key weight W K Value weight W V Each part is divided into two parts and assigned to different processor groups. Each processor group contains two processors, and the two processors correspond to the same subquery weight, subkey weight, and subvalue weight.

[0173] Here, the i-th processor corresponds to the i-th subquery weight, the i-th subkey weight, and the i-th subvalue weight. In one possible implementation, after obtaining the sequence feature representation corresponding to the i-th task subsequence, the i-th processor can perform key value calculation on the sequence feature representation based on the i-th subquery weight, the i-th subkey weight, and the i-th subvalue weight to obtain the intermediate decoding result corresponding to the i-th task subsequence.

[0174] Indicative, such as Figure 12As shown, processor Rank0 decodes the sequence feature representation using the FD mechanism based on the first subquery weight Q, the first subkey weight K, the first subvalue weight V, and the key-value tensor, to obtain the intermediate decoding results (m1, l1, o1) corresponding to the task subsequence. The intermediate decoding result m1 has a dimension of (b, 1, h / 2), the intermediate decoding result k1 has a dimension of (b, 1, 2), and the intermediate decoding result o1 has a dimension of (b, 1, 2).

[0175] Step 1102: The i-th processor aggregates the intermediate decoding results transmitted by other processors based on the intra-group communication of the processor group to obtain the intra-group intermediate decoding result of the task sequence on the i-th processor. The intra-group intermediate decoding result includes the intermediate decoding results of all sequence dimensions and the intermediate decoding result of the attention head dimension corresponding to the i-th processor.

[0176] Optionally, to further improve the parallelism of the model, different tensor parallelisms can be set in the decoding stage. That is, the first tensor parallelism is used for parallel computation in the self-attention stage, and the second tensor parallelism is used for parallel computation in the subsequent linear transformation and feedforward neural network stages.

[0177] To ensure the connection between the first tensor parallelism and the second tensor parallelism, after obtaining the intermediate decoding results output by each processor in the processor group, the intermediate decoding results output by each processor can be transposed through intra-group communication of the processor group, so that each processor has intermediate decoding results of all sequence dimensions as well as intermediate decoding results of its corresponding attention head dimension.

[0178] In one possible implementation, based on intra-group communication of the processor group, the i-th processor receives intermediate decoding results transmitted by other processors and sends intermediate decoding results to other processors. By combining its own output intermediate decoding results and the received intermediate decoding results transmitted by other processors, the intra-group intermediate decoding results of the task sequence on the i-th processor can be obtained through aggregation processing.

[0179] The intermediate decoding results within a group include the intermediate decoding results of all sequence dimensions as well as the intermediate decoding results of the attention head dimension corresponding to the i-th processor. That is, the intermediate decoding results within a group are complete in the sequence dimension, but only contain the intermediate decoding results of the attention dimension corresponding to the i-th processor in the attention head dimension.

[0180] Optionally, intra-group communication within a processor group can employ all-to-all communication. All-to-all communication is a global communication operation in parallel or distributed systems in which every processor participates, sending and receiving messages to and from all other processors.

[0181] In one possible implementation, based on the fully connected communication established within the processor group, the i-th processor sends a partial intermediate decoding result corresponding to the i-th task subsequence to other processors within the processor group. The partial intermediate decoding result corresponding to the i-th task subsequence is the intermediate decoding result of other attention head dimensions besides the attention head dimension corresponding to the i-th processor. The i-th processor also receives partial intermediate decoding results corresponding to other task subsequences sent by other processors within the processor group. The partial intermediate decoding results corresponding to other task subsequences are the intermediate decoding results of the attention head dimension corresponding to the i-th processor. Thus, the i-th processor aggregates the intermediate decoding results of each task subsequence in the attention head dimension corresponding to the i-th processor to obtain the intra-group intermediate decoding result of the task sequence on the i-th processor.

[0182] Indicative, such as Figure 12 As shown, based on the full-connection communication of the processor group, processor Rank0 transmits the intermediate decoding result of the sequence dimension corresponding to head1 to processor Rank2, and processor Rank2 transmits the intermediate decoding result of the sequence dimension corresponding to head0 to processor Rank0. Thus, processor Rank0 obtains the intermediate decoding result of all sequence dimensions under head0, and processor Rank2 obtains the intermediate decoding result of all sequence dimensions under head1. At this time, the dimensions of the intermediate decoding result within the group on each processor are (b,2,h / 4), (b,2,1), and (b,2,1).

[0183] Step 1103: The i-th processor performs linear transformation and feedforward neural network processing on the intermediate decoding results within the group based on global communication within the processor cluster to obtain the sequence output results corresponding to the task sequence.

[0184] Optionally, after obtaining the intermediate decoding results within the group, the i-th processor can perform linear transformation and feedforward neural network processing on the intermediate decoding results within the group according to the configured model sub-weights. Since the intermediate decoding results within the group corresponding to the i-th processor only have the intermediate decoding results of the attention head dimension corresponding to the i-th processor, during the linear transformation process, the model sub-weights configured on the i-th processor can only process the intermediate decoding results of the attention head dimension corresponding to the i-th processor. Therefore, in order to ensure that the model parameters on all processors are consistent and to achieve data synchronization between processors, global communication can be built between the processor clusters. Thus, during the linear transformation process and the feedforward neural network process, based on the global communication between the processor clusters, the i-th processor can output the sequence output results corresponding to the task sequence.

[0185] In the linear transformation and feedforward neural network processing, the model sub-weights include sub-linear transformation weights and sub-feedforward network weights. These sub-linear transformation weights and feedforward network weights are partitioned based on the second tensor parallelism, so that each processor in the processor cluster corresponds to its own sub-linear transformation weights and feedforward network weights. Specifically, when the second tensor parallelism equals the number of processors, each processor corresponds to different sub-query weights, sub-key weights, and sub-value weights, thus achieving full tensor parallelism in both linear transformation and feedforward neural network processing.

[0186] In one possible design, the parallelism of the second tensor differs from that of the first tensor; the second tensor parallelism can be greater than the first. For example, with 4 processors and a sequence parallelism SP = 2, the first tensor parallelism TP can be 2, and the second tensor parallelism TP can be 4. That is, in the linear transformation processing and feedforward neural network processing, the linear transformation weight W... O and feedforward network weights W FFN It is divided into four parts and distributed to different processors, each processor being configured with 1 / 4 of the linear transformation weight W. O and feedforward network weights W FFN .

[0187] In one possible design, global communication between the processor cluster can employ all-reduce communication. Optionally, all-reduce communication is a collective communication operation in distributed computing involving data exchange between multiple processors. In all-reduce communication, each processor sends its own data to all other processors and receives data from all other processors, thus each processor obtains a reduced result. Optionally, the reduction operation can be summation, finding the maximum value, finding the minimum value, etc.

[0188] In one possible implementation, the i-th processor first performs a linear transformation on the intra-group intermediate decoding results according to the sub-linear transformation weights, obtaining the linearly transformed intra-group intermediate decoding results. At this point, the intra-group intermediate decoding results are complete in the sequence dimension, but only contain the intermediate decoding results corresponding to the attention head dimension of the i-th processor. Furthermore, to achieve data synchronization between processors, based on the full reduction communication established within the processor cluster, the processor cluster aggregates the linearly transformed intra-group intermediate decoding results from each processor and distributes the aggregated global intermediate decoding results to each processor. At this point, the global intermediate decoding results are complete in both the sequence dimension and the attention head dimension.

[0189] Furthermore, the i-th processor performs feedforward neural network processing on the global intermediate decoding results based on the sub-feedforward network weights to obtain the partial sequence output results corresponding to the task sequence. Since the global intermediate decoding results are processed by feedforward neural network only through the sub-feedforward network weights configured on the i-th processor, the partial sequence output results are only the sequence output results of the attention head dimension corresponding to the i-th processor. Therefore, in order to achieve data synchronization between processors, based on the full reduction communication established within the processor cluster, the processor cluster also needs to aggregate the partial sequence output results corresponding to the task sequence on each processor and distribute the aggregated sequence output results corresponding to the task sequence to each processor, so that each processor obtains the sequence output results corresponding to the task sequence.

[0190] Indicative, such as Figure 12 As shown, processor Rank0 first performs a linear transformation on the intra-group intermediate decoding results according to the sub-linear transformation weights, obtaining the intra-group intermediate decoding results after the linear transformation. At this time, the dimension of the intra-group intermediate decoding result Y is (b, 1, h / 4). Then, in order to achieve data synchronization between processors, based on the full reduction communication established within the processor cluster, the processor cluster aggregates the intra-group intermediate decoding results of processors Rank0-3 after the linear transformation, and distributes the aggregated global intermediate decoding results to each processor. At this time, the dimension of the global intermediate decoding result O is (b, 1, h). Further, processor Rank0 performs FFN processing on the global intermediate decoding results based on the sub-feedforward network weights to obtain the partial sequence output results corresponding to the task sequence. The processor cluster, based on the full reduction communication established within the processor cluster, again aggregates the partial sequence output results corresponding to the task sequence and distributes the sequence output results corresponding to the task sequence. At this time, the dimension of the sequence output result O is (b, 1, h).

[0191] Optionally, since the decoding stage adopts an autoregressive approach, the sequence output result of each round of the processor needs to depend on the sequence output result of the previous round. Therefore, after obtaining the sequence output result of the current round, it is also necessary to cache the sequence output result of the current round.

[0192] In one possible design, in order to achieve load balancing among processors, the scheduling device can also determine the processor with the smallest key-value cache size as the target processor in the processor group based on the key-value cache size of each processor in the processor group. Thus, after obtaining the sequence output result corresponding to the task sequence, the target processor caches the sequence output result in the key-value cache, and other processors in the processor group do not need to cache the sequence output result.

[0193] In one possible implementation, to achieve dynamic adjustment of load balancing, the scheduling device can determine the target processor once in each output round. That is, based on the key-value cache size of each processor in the processor group in the current output round, the scheduling device determines the processor with the smallest key-value cache size as the target processor in the current output round of the processor group. Thus, after obtaining the sequence output result corresponding to the task sequence of the current output round, the target processor caches the sequence output result in the key-value cache.

[0194] Indicative, such as Figure 12 As shown, after obtaining the sequence output result corresponding to the task sequence of the current inference round, the scheduling device determines the processors Rank0 and Rank1 as target processors, and the sequence output result is cached by processors Rank0 and Rank1. Therefore, the dimension of the key-value tensor of processors Rank0 and Rank1 becomes (b, (s / 2)+1, h / 4).

[0195] exist Figure 11 The method described includes three processes in the decoding phase: attention calculation, linear transformation, and feedforward neural network processing. By combining tensor parallelism and sequence parallelism for distributed model inference, intermediate decoding results on each processor can be updated via intra-processor communication after attention calculation, reducing peak activation of the entire network. Furthermore, by setting a first degree of tensor parallelism during attention calculation and a second degree of tensor parallelism during linear transformation and feedforward neural network processing, custom configuration of parallelism can be achieved, effectively improving model parallel processing efficiency and reducing inference system latency.

[0196] In addition, for the sequence output results generated in each round, the scheduling device determines the target processor to cache the sequence output results based on the current key value cache amount of each processor, thereby ensuring load balance among the processors in the processor group.

[0197] The above describes the model reasoning method provided in the embodiments of this application. Corresponding to the above method, the embodiments of this application also provide a model reasoning system.

[0198] Please refer to Figure 13 This illustrates a schematic diagram of the structure of a model inference system provided in an embodiment of this application. Figure 13 As shown, the model inference system 1300 includes a scheduling device 1301 and a processor cluster 1302.

[0199] The scheduling device 1301 is used to divide the task sequence corresponding to the model inference request based on the sequence parallelism when a model inference request is received, and obtain multiple task sub-sequences.

[0200] The scheduling device 1301 is used to allocate multiple task subsequences to various processor groups in the processor cluster. The number of processors in a processor group is the same as the sequence parallelism. There is a one-to-one correspondence between the processors in the processor group and the task subsequence. Each processor in the processor group is configured with model subweights. The model subweights are obtained by dividing the model weights based on the tensor parallelism. Different processor groups correspond to different model attention heads.

[0201] Processor cluster 1302 is used to generate model inference results corresponding to the task sequence based on the sequence output results generated by each processor executing the task sub-sequence.

[0202] In one possible design, for the i-th processor within the processor cluster, the i-th processor is used to pre-fill the i-th task subsequence based on the model sub-weights to obtain the key-value tensor and sequence feature representation corresponding to the i-th task subsequence, where i is a positive integer greater than 1; the i-th processor is used to decode the sequence feature representation based on the model sub-weights and key-value tensor to obtain the intermediate decoding result corresponding to the i-th task subsequence; the i-th processor is used to perform linear transformation and feedforward neural network processing on the intermediate decoding result based on device communication between processors to obtain the sequence output result corresponding to the task sequence; the processor cluster is used to generate the model inference result corresponding to the task sequence based on the sequence output results output by each processor.

[0203] In one possible design, the i-th processor is used to: perform key-value calculation on the i-th task subsequence based on model sub-weights and a ring attention mechanism to obtain the key-value tensor corresponding to the i-th task subsequence; aggregate the key-value tensors transmitted by other processors based on intra-group communication of the processor group to obtain the intra-group key-value tensor of the task sequence on the i-th processor, wherein the intra-group key-value tensor includes the key-value tensors of all sequence dimensions and the key-value tensor of the attention head dimension corresponding to the i-th processor; and perform linear transformation and feedforward neural network processing on the intra-group key-value tensor based on global communication within the processor cluster to obtain the sequence feature representation corresponding to the i-th task subsequence.

[0204] Optionally, the model sub-weights include subquery weights, subkey weights, and subvalue weights, which are partitioned based on the parallelism of the first tensor.

[0205] In one possible design, the i-th processor is configured to: perform key-value calculation on the i-th task subsequence based on the i-th subquery weight, the i-th subkey weight, and the i-th subvalue weight, to obtain the i-th key-value tensor corresponding to the i-th task subsequence; receive the i-1-th subkey weight and the i-1-th subvalue weight transmitted by the (i-1)-th processor based on a circular attention mechanism, and send the i-th subkey weight and the i-th subvalue weight to the (i+1)-th processor; perform key-value calculation on the i-th task subsequence based on the received i-1-th subkey weight and the i-1-th subvalue weight, to obtain the i-1-th key-value tensor corresponding to the i-th task subsequence; and update the i-th key-value tensor based on the i-1-th key-value tensor to obtain the key-value tensor corresponding to the i-th task subsequence.

[0206] In one possible design, the i-th processor is configured to: send a partial key-value tensor corresponding to the i-th task subsequence to other processors in the processor group based on the fully connected communication established within the processor group, wherein the partial key-value tensor corresponding to the i-th task subsequence is a key-value tensor of attention head dimensions other than the attention head dimension corresponding to the i-th processor; receive partial key-value tensors corresponding to other task subsequences sent by other processors in the processor group, wherein the partial key-value tensors corresponding to other task subsequences are key-value tensors of attention head dimensions corresponding to the i-th processor; and aggregate the key-value tensors of each task subsequence in the attention head dimension corresponding to the i-th processor to obtain the intra-group key-value tensor of the task sequence on the i-th processor.

[0207] Optionally, the model sub-weights include sub-linear transformation weights and sub-feedforward network weights, which are partitioned based on the second tensor parallelism.

[0208] In one possible design, the i-th processor performs a linear transformation on the in-group key-value tensor based on the sub-linear transformation weights to obtain the linearly transformed in-group key-value tensor; the processor cluster aggregates the linearly transformed in-group key-value tensors on each processor based on the full reduction communication established within the processor cluster, and distributes the aggregated global key-value tensor to each processor; the i-th processor performs a feedforward neural network process on the global key-value tensor based on the sub-feedforward network weights to obtain a partial sequence feature representation corresponding to the task sequence, which is the sequence feature representation of the attention head dimension corresponding to the i-th processor; the processor cluster aggregates the partial sequence feature representations corresponding to the task sequence on each processor based on the full reduction communication established within the processor cluster, and distributes the aggregated sequence feature representations corresponding to the task sequence to each processor; the i-th processor performs sequence partitioning on the sequence feature representations corresponding to the task sequence to obtain the sequence feature representations corresponding to the i-th task subsequence.

[0209] Optionally, the model sub-weights include subquery weights, subkey weights, and subvalue weights, which are partitioned based on the parallelism of the first tensor.

[0210] In one possible design, the i-th processor is used to: perform key-value calculation on the sequence feature representation based on the i-th subquery weight, the i-th subkey weight, and the i-th subvalue weight, to obtain the intermediate decoding result corresponding to the i-th task subsequence.

[0211] In one possible design, the i-th processor is used to: aggregate intermediate decoding results transmitted by other processors based on intra-processor group communication to obtain intra-group intermediate decoding results of the task sequence on the i-th processor, the intra-group intermediate decoding results including intermediate decoding results of all sequence dimensions and intermediate decoding results of the attention head dimension corresponding to the i-th processor; and perform linear transformation and feedforward neural network processing on the intra-group intermediate decoding results based on global communication within the processor cluster to obtain the sequence output results corresponding to the task sequence.

[0212] In one possible design, the i-th processor is configured to: send partial intermediate decoding results corresponding to the i-th task subsequence to other processors in the processor group based on the fully connected communication established within the processor group, wherein the partial intermediate decoding results corresponding to the i-th task subsequence are intermediate decoding results of attention head dimensions other than the attention head dimension corresponding to the i-th processor; receive partial intermediate decoding results corresponding to other task subsequences sent by other processors in the processor group, wherein the partial intermediate decoding results corresponding to other task subsequences are intermediate decoding results of the attention head dimension corresponding to the i-th processor; and aggregate the intermediate decoding results of each task subsequence in the attention head dimension corresponding to the i-th processor to obtain the intra-group intermediate decoding result of the task sequence on the i-th processor.

[0213] Optionally, the model sub-weights include sub-linear transformation weights and sub-feedforward network weights, which are partitioned based on the second tensor parallelism.

[0214] In one possible design, the i-th processor is used to perform linear transformation processing on the intermediate decoding results within the group based on the sub-linear transformation weights, to obtain the intermediate decoding results within the group after linear transformation; the processor cluster is used to aggregate the intermediate decoding results within the group after linear transformation on each processor based on the full reduction communication established within the processor cluster, and distribute the aggregated global intermediate decoding results to each processor; the i-th processor is used to perform feedforward neural network processing on the global intermediate decoding results based on the sub-feedforward network weights, to obtain the partial sequence output results corresponding to the task sequence, and the partial sequence output results are the sequence output results of the attention head dimension corresponding to the i-th processor; the processor cluster is used to aggregate the partial sequence output results corresponding to the task sequence on each processor based on the full reduction communication established within the processor cluster, and distribute the aggregated sequence output results corresponding to the task sequence to each processor.

[0215] In one possible design, a scheduling device is used to determine the processor with the smallest key-value cache size as the target processor in the processor group based on the key-value cache size of each processor in the processor group; the target processor is used to: cache the sequence output result corresponding to the task sequence into the key-value cache after obtaining the sequence output result.

[0216] Please refer to Figure 14 The diagram illustrates a flowchart of a scheduling model for a scheduling device in a model inference system, as provided in an embodiment of this application.

[0217] like Figure 14 As shown, upon receiving user configuration operations for model parallel parameters and a user-submitted model inference request, the scheduling device configures and initializes the underlying model based on the parallel parameters and the inference request. Furthermore, the scheduling device distributes and loads model weights according to the parallel parameters (tensor parallelism). After obtaining the model sub-weights, the scheduling device also needs to divide the task sequence contained in the inference request according to the parallel parameters (sequence parallelism). Then, based on the model sub-weights and the task sub-sequences, the scheduling device builds the corresponding inference model on each processor, thereby executing the corresponding task sub-sequences on each processor.

[0218] During the execution of task subsequences by the processor, for key-value tensors generated in the pre-filling stage, the processor can cache them in the key-value cache (KVcache) resource pool within the model inference system, and obtain the required key-value tensors by accessing the KVcache resource pool during the decoding stage.

[0219] In addition, users can configure corresponding model quantization and compression in the model inference system according to their own needs, thereby accelerating the inference speed of the model.

[0220] The above describes the model inference method and system provided in the embodiments of this application. Corresponding to the above method and system, the embodiments of this application also provide a model inference device. This device is applied to the scheduling device in the model inference system. The device is used to... Figure 15 Each module shown performs the above... Figure 6 The model inference method executed by the scheduling equipment. For example... Figure 15 As shown, the model inference device provided in this application embodiment includes the following modules.

[0221] The sequence partitioning module 1501 is used to partition the task sequence corresponding to the model inference request based on the sequence parallelism when a model inference request is received, and obtain multiple task sub-sequences.

[0222] The sequence allocation module 1502 is used to allocate multiple task subsequences to various processor groups in the processor cluster. The number of processors in a processor group is the same as the sequence parallelism, and there is a one-to-one correspondence between processors and task subsequences within a processor group. Each processor in a processor group is configured with model subweights, which are obtained by partitioning the model weights based on tensor parallelism. Different processor groups correspond to different model attention heads.

[0223] It should be understood that the above Figure 15 The beneficial effects that the provided device possesses in performing its function are... Figure 6 The provided model inference methods possess the same beneficial effects, which will not be elaborated upon here. Additionally, Figure 15 The provided device, in implementing its functions, is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.

[0224] See Figure 16 , Figure 16 A schematic diagram of the structure of an exemplary computer device 1600 of this application is shown. The computer device 1600 includes at least one processor 1601, a memory 1603, and at least one network interface 1604.

[0225] Processor 1601 may be, for example, a general-purpose central processing unit (CPU), a digital signal processor (DSP), a network processor (NP), a GPU, a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits or application-specific integrated circuits (ASICs), programmable logic devices (PLDs), other general-purpose processors or other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components, or any combination thereof for implementing the scheme of this application. A PLD may be, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor. It is worth noting that the processor may be a processor supporting an advanced reduced instruction set machine (RISC) machine (ARM) architecture. It can implement or execute the various logic blocks, modules, and circuits described in conjunction with the disclosure of this application. A processor can also be a combination of components that perform computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and so on.

[0226] Optionally, the computer device 1600 also includes a bus 1602. The bus 1602 is used to transmit information between the various components of the computer device 1600. The bus 1602 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 1602 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 16 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0227] The memory 1603 may be, for example, volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache.

[0228] By way of example, but not limitation, many forms of ROM and RAM are available. For example, ROM is a compact disc read-only memory (CD-ROM). RAM includes, but is not limited to, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0229] The memory 1603 can also be other types of storage devices capable of storing static information and instructions. Alternatively, it can be other types of dynamic storage devices capable of storing information and instructions. It can also be other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. The memory 1603 may exist independently, for example, and be connected to the processor 1601 via bus 1602. The memory 1603 may also be integrated with the processor 1601.

[0230] Network interface 1604 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), or wireless local area network (WLAN). Network interface 1604 may include wired network interfaces and wireless network interfaces. Specifically, network interface 1604 can be an Ethernet interface, such as Fast Ethernet (FE), Gigabit Ethernet (GE), Asynchronous Transfer Mode (ATM), WLAN, cellular network, or combinations thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In some embodiments of this application, network interface 1604 can be used for computer device 1600 to communicate with other devices.

[0231] In specific implementations, as some embodiments, the processor 1601 may include one or more CPUs, such as Figure 16 The CPU0 and CPU1 shown are examples of processors. Each of these processors can be a single-core processor or a multi-core processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0232] In specific implementations, as some methods, the computer device 1600 may include multiple processors, such as... Figure 16 The processors 1601 and 1605 are shown. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0233] In some embodiments, memory 1603 is used to store program instructions 1610 for executing the scheme of this application, and processor 1601 can execute the program instructions 1610 stored in memory 1603. That is, computer device 1600 can implement the method provided in the method embodiment through processor 1601 and program instructions 1610 in memory 1603, i.e. Figure 6 , 8 The methods executed in steps 9 and 11. Program instructions 1610 may include one or more software modules. Optionally, the processor 1601 itself may also store program instructions for executing the scheme of this application.

[0234] In specific implementation, the computer device 1600 of this application can correspond to a first network element device for executing the above method. The processor 1601 in the computer device 1600 reads instructions from the memory 1603, causing... Figure 16 The computer device 1600 shown is capable of performing all or part of the steps in the method embodiments.

[0235] Computer device 1600 can also correspond to the above. Figure 15 The device shown, Figure 15 Each functional module in the illustrated device is implemented using software from computer device 1600. In other words, Figure 15 The device shown includes functional modules generated by the processor 1601 of the computer device 1600 after reading the program instructions 1610 stored in the memory 1603.

[0236] in, Figure 6 , 8 Each step of the method shown in 9 and 11 is completed by the integrated logic circuitry of the hardware in the processor of the computer device 1600 or by instructions in the form of software. The steps of the method embodiments disclosed in this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. Since this storage medium is located in memory, the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method embodiments. To avoid repetition, these steps will not be described in detail here.

[0237] See Figure 17 , Figure 17 A schematic diagram of the structure of an exemplary computer device 1700 according to this application is shown. The computer device 1700 includes a main control board 1710 and an interface board 1730. Figure 17 The computer device 1700 shown is used to perform the above. Figure 6 , 8 The operations involved in the model reasoning methods shown in 9 and 11. The computer device 1700 is, for example, a switch, router, controller, etc. Computer device 1700 can be an example of a computer device.

[0238] The main control board 1710, also known as the main processing unit (MPU) or route processor card, is used to control and manage the various components in the computer device 1700, including routing calculation, device management, device maintenance, and protocol processing functions. The main control board 1710 includes a central processing unit 1711 and a memory 1712.

[0239] Interface board 1730 is also known as a line processing unit (LPU), linecard, or service board. Interface board 1730 provides various service interfaces and implements packet forwarding. Service interfaces include, but are not limited to, Ethernet interfaces, POS (Packet over SONET / SDH) interfaces, etc., with Ethernet interfaces including, for example, flexible Ethernet clients (FlexE Clients). Interface board 1730 includes: a central processing unit 1731, a network processor 1732, a forwarding table entry memory 1734, and a physical interface card (PIC) 1733.

[0240] The central processing unit 1731 on the interface board 1730 is used to control and manage the interface board 1730 and communicate with the central processing unit 1711 on the main control board 1710.

[0241] Network processor 1732 is used to implement packet forwarding processing. Network processor 1732 can be in the form of a forwarding chip. Specifically, network processor 1732 forwards received packets based on the forwarding table stored in forwarding table entry memory 1734. If the destination address of the packet is the address of computer device 1700, the packet is sent to the CPU (such as central processing unit 1711) for processing; if the destination address of the packet is not the address of computer device 1700, the next hop and outgoing interface corresponding to the destination address are looked up in the forwarding table according to the destination address, and the packet is forwarded to the outgoing interface corresponding to the destination address. Uplink packet processing includes: packet ingress interface processing, forwarding table lookup; downlink packet processing includes forwarding table lookup, etc.

[0242] The physical interface card 1733 is used to implement physical layer interfacing functions. Raw traffic enters the interface board 1730 through this card, and processed packets are sent out from the physical interface card 1733. The physical interface card 1733, also known as a daughter card, can be installed on the interface board 1730. It is responsible for converting photoelectric signals into packets, performing validity checks on the packets, and forwarding them to the network processor 1732 for processing. In some implementations, the central processing unit can also perform the functions of the network processor 1732, such as implementing software forwarding based on a general-purpose CPU, thus eliminating the need for a network processor 1732 within the physical interface card 1733.

[0243] Optionally, the computer device 1700 includes multiple interface boards. For example, the computer device 1700 also includes an interface board 1740, which includes a central processing unit 1741, a network processor 1742, a forwarding table entry memory 1744, and a physical interface card 1743.

[0244] Optionally, the computer device 1700 also includes a switching fabric board 1720. The switching fabric board 1720 can also be referred to as a switch fabric unit (SFU). In cases where the computer device has multiple interface boards 1730, the switching fabric board 1720 is used to complete data exchange between the interface boards. For example, interface boards 1730 and 1740 can communicate via the switching fabric board 1720.

[0245] The main control board 1710 and the interface board 1730 are coupled. For example, the main control board 1710, interface board 1730, interface board 1740, and switching network board 1720 communicate with each other via a system bus connected to the system backplane. In one possible implementation, an inter-process communication (IPC) channel is established between the main control board 1710 and the interface board 1730, and the main control board 1710 and the interface board 1730 communicate with each other through the IPC channel.

[0246] Logically, computer device 1700 includes a control plane and a forwarding plane. The control plane includes a main control board 1710 and a central processing unit 1731, while the forwarding plane includes various components that perform forwarding, such as a forwarding table entry memory 1734, a physical interface card 1733, and a network processor 1732. The control plane performs functions such as router operation, generating forwarding tables, processing signaling and protocol messages, and configuring and maintaining the device's status. The control plane distributes the generated forwarding tables to the forwarding plane. In the forwarding plane, the network processor 1732 looks up and forwards messages received by the physical interface card 1733 based on the forwarding tables distributed by the control plane. The forwarding tables distributed by the control plane can be stored in the forwarding table entry memory 1734. In some embodiments, the control plane and the forwarding plane can be completely separated and not on the same device.

[0247] It's worth noting that there may be one or more main control boards, including a primary and a backup main control board. There may also be one or more interface boards; the more powerful the computer equipment's data processing capabilities, the more interface boards it can provide. Each interface board may also have one or more physical interface cards. There may be no switching network board, or one or more; multiple boards can share the load and provide redundancy. In a centralized forwarding architecture, the computer equipment may not need a switching network board, as the interface boards handle the entire system's business data processing. In a distributed forwarding architecture, the computer equipment can have at least one switching network board, which enables data exchange between multiple interface boards, providing high-capacity data exchange and processing capabilities. Therefore, the data access and processing capabilities of a distributed architecture computer equipment are greater than those of a centralized architecture equipment. Alternatively, the computer device can also be a single board, without a switching network board. The functions of the interface board and the main control board are integrated on this one board. In this case, the central processing unit (CPU) on the interface board and the CPU on the main control board can be combined into a single CPU to perform the combined functions. This type of device has lower data exchange and processing capabilities (e.g., low-end switches or routers). The specific architecture adopted depends on the specific network deployment scenario, and no restrictions are imposed here.

[0248] In an exemplary embodiment, a computer program (product) is provided, comprising: computer program code, which, when executed by a computer, causes the computer to perform... Figure 6 , 8 The methods in 9 and 11.

[0249] In an exemplary embodiment, a computer-readable storage medium is provided that stores a program or instructions, which, when executed on a computer, cause the computer to perform the aforementioned actions. Figure 6 , 8 The methods in 9 and 11.

[0250] In an exemplary embodiment, a chip is provided, including a processor for recalling and executing instructions stored in memory, causing a computer with the chip installed to perform... Figure 6 , 8 The methods in 9 and 11.

[0251] In an exemplary embodiment, another chip is provided, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected via internal interconnection paths. The processor is used to execute code in the memory. When the code is executed, a computer with the chip installed performs... Figure 6 , 8 The methods in 9 and 11.

[0252] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk, SSD), etc.

[0253] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another.

[0254] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0255] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple second devices means two or more second devices. The terms "system" and "network" are often used interchangeably herein.

[0256] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0257] It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship.

[0258] It should also be understood that the terms “if” and “if” can be interpreted as meaning “when” or “upon”, or “in response to determination” or “in response to detection”. Similarly, depending on the context, the phrases “if determination…” or “if detection [the stated condition or event]” can be interpreted as meaning “when determination…”, or “in response to determination…”, or “when detection [the stated condition or event]” or “in response to detection [the stated condition or event]”.

[0259] The above description is merely an embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A model reasoning method, characterized in that, The method is used in a model inference system, the system including a scheduling device and a processor cluster, and the method includes: Upon receiving a model inference request, the scheduling device divides the task sequence corresponding to the model inference request based on the sequence parallelism to obtain multiple task sub-sequences; The scheduling device assigns the multiple task subsequences to each processor group in the processor cluster. The number of processors in each processor group is the same as the parallelism of the sequence. Each processor in the processor group corresponds one-to-one with the task subsequence. Each processor in the processor group is configured with model subweights. The model subweights are obtained by partitioning the model weights based on the tensor parallelism. Different processor groups correspond to different model attention heads. The processor cluster generates model inference results corresponding to the task sequence based on the sequence output results generated by each processor executing the task sub-sequence.

2. The method according to claim 1, characterized in that, The processor cluster generates model inference results corresponding to the task sequence based on the sequence output results generated by each processor executing the task sub-sequence, including: For the i-th processor in the processor group, the i-th processor performs pre-filling processing on the i-th task sub-sequence based on the model sub-weights to obtain the key-value tensor and sequence feature representation corresponding to the i-th task sub-sequence, where i is a positive integer and greater than 1; The i-th processor decodes the sequence feature representation based on the model sub-weights and the key-value tensor to obtain the intermediate decoding result corresponding to the i-th task sub-sequence; The i-th processor performs linear transformation and feedforward neural network processing on the intermediate decoding result based on device communication between the processors to obtain the sequence output result corresponding to the task sequence; The processor cluster generates the model inference result corresponding to the task sequence based on the sequence output results of each processor.

3. The method according to claim 2, characterized in that, The i-th processor pre-fills the i-th task sub-sequence based on the model sub-weights to obtain the key-value tensor and sequence feature representation corresponding to the i-th task sub-sequence, including: The i-th processor performs key-value calculation on the i-th task sub-sequence based on the model sub-weights and the ring attention mechanism to obtain the key-value tensor corresponding to the i-th task sub-sequence; Based on the intra-group communication of the processor group, the i-th processor aggregates the key-value tensors transmitted by other processors to obtain the intra-group key-value tensor of the task sequence on the i-th processor. The intra-group key-value tensor includes key-value tensors of all sequence dimensions and key-value tensors of the attention head dimension corresponding to the i-th processor. The i-th processor performs linear transformation and feedforward neural network processing on the key-value tensor within the group based on global communication within the processor cluster to obtain the sequence feature representation corresponding to the i-th task subsequence.

4. The method according to claim 3, characterized in that, The model sub-weights include subquery weights, subkey weights, and subvalue weights, which are divided based on the first tensor parallelism. The i-th processor performs key-value calculation on the i-th task sub-sequence based on the model sub-weights and the circular attention mechanism to obtain the key-value tensor corresponding to the i-th task sub-sequence, including: The i-th processor performs key-value calculation on the i-th task subsequence based on the i-th sub-query weight, the i-th sub-key weight, and the i-th sub-value weight to obtain the i-th key-value tensor corresponding to the i-th task subsequence. Based on the circular attention mechanism, the i-th processor receives the i-1th subkey weight and the i-1th subvalue weight transmitted by the i-1th processor, and sends the i-th subkey weight and the i-th subvalue weight to the i+1th processor. The i-th processor performs key value calculation on the i-th task subsequence based on the received i-1-th subkey weight and i-1-th subvalue weight, and obtains the i-1-th key value tensor corresponding to the i-th task subsequence; The i-th processor updates the i-th key-value tensor based on the (i-1)-th key-value tensor to obtain the key-value tensor corresponding to the i-th task subsequence.

5. The method according to claim 3, characterized in that, The i-th processor, based on intra-group communication within the processor group, aggregates the key-value tensors transmitted by other processors to obtain the intra-group key-value tensor of the task sequence on the i-th processor, including: Based on the fully connected communication established within the processor group, the i-th processor sends the partial key-value tensor corresponding to the i-th task subsequence to other processors within the processor group. The partial key-value tensor corresponding to the i-th task subsequence is the key-value tensor of other attention head dimensions besides the attention head dimension corresponding to the i-th processor. The i-th processor receives a partial key-value tensor corresponding to other task subsequences sent by other processors in the processor group, and the partial key-value tensor corresponding to other task subsequences is the key-value tensor of the attention head dimension corresponding to the i-th processor. The i-th processor aggregates the key-value tensors of each task subsequence in the corresponding attention head dimension of the i-th processor to obtain the in-group key-value tensor of the task sequence on the i-th processor.

6. The method according to claim 3, characterized in that, The model sub-weights include sub-linear transformation weights and sub-feedforward network weights, which are partitioned based on the second tensor parallelism. The i-th processor, based on global communication within the processor cluster, performs linear transformation and feedforward neural network processing on the key-value tensor within the group to obtain the sequence feature representation corresponding to the i-th task sub-sequence, including: The i-th processor performs a linear transformation on the in-group key-value tensor based on the sub-linear transformation weights to obtain the in-group key-value tensor after linear transformation. The processor cluster aggregates the intra-group key-value tensors after linear transformation on each processor based on the full reduction communication established within the processor cluster, and distributes the aggregated global key-value tensor to each processor. The i-th processor performs feedforward neural network processing on the global key-value tensor based on the weights of the sub-feedforward network to obtain a partial sequence feature representation corresponding to the task sequence. The partial sequence feature representation is the sequence feature representation of the attention head dimension corresponding to the i-th processor. The processor cluster, based on the full reduction communication established within the processor cluster, aggregates the partial sequence feature representations corresponding to the task sequences on each processor, and distributes the aggregated sequence feature representations corresponding to the task sequences to each processor. The i-th processor performs sequence partitioning on the sequence feature representation corresponding to the task sequence to obtain the sequence feature representation corresponding to the i-th task sub-sequence.

7. The method according to any one of claims 2 to 6, characterized in that, The model sub-weights include subquery weights, subkey weights, and subvalue weights, which are divided based on the first tensor parallelism. The i-th processor decodes the sequence feature representation based on the model sub-weights and the key-value tensor to obtain the intermediate decoding result corresponding to the i-th task sub-sequence, including: The i-th processor performs key value calculation on the sequence feature representation based on the i-th sub-query weight, the i-th sub-key weight, and the i-th sub-value weight to obtain the intermediate decoding result corresponding to the i-th task sub-sequence.

8. The method according to any one of claims 2 to 7, characterized in that, The i-th processor, based on device communication between the processors, performs linear transformation and feedforward neural network processing on the intermediate decoding result to obtain the sequence output result corresponding to the task sequence, including: The i-th processor, based on the intra-group communication of the processor group, aggregates the intermediate decoding results transmitted by other processors to obtain the intra-group intermediate decoding result of the task sequence on the i-th processor. The intra-group intermediate decoding result includes the intermediate decoding results of all sequence dimensions and the intermediate decoding result of the attention head dimension corresponding to the i-th processor. The i-th processor performs linear transformation and feedforward neural network processing on the intermediate decoding results within the group based on global communication within the processor cluster, to obtain the sequence output result corresponding to the task sequence.

9. The method according to claim 8, characterized in that, The i-th processor, based on intra-group communication within the processor group, aggregates the intermediate decoding results transmitted by other processors to obtain the intra-group intermediate decoding result of the task sequence on the i-th processor, including: The i-th processor sends a portion of the intermediate decoding result corresponding to the i-th task subsequence to other processors in the processor group based on the fully connected communication established within the processor group. The portion of the intermediate decoding result corresponding to the i-th task subsequence is the intermediate decoding result of other attention head dimensions besides the attention head dimension corresponding to the i-th processor. The i-th processor receives partial intermediate decoding results corresponding to other task subsequences sent by other processors in the processor group. The partial intermediate decoding results corresponding to other task subsequences are the intermediate decoding results of the attention head dimension corresponding to the i-th processor. The i-th processor aggregates the intermediate decoding results of each task subsequence in the corresponding attention head dimension of the i-th processor to obtain the intra-group intermediate decoding result of the task sequence on the i-th processor.

10. The method according to claim 8, characterized in that, The model sub-weights include sub-linear transformation weights and sub-feedforward network weights, which are partitioned based on the second tensor parallelism. The i-th processor, based on global communication within the processor cluster, performs linear transformation and feedforward neural network processing on the intermediate decoding results within the group to obtain the sequence output result corresponding to the task sequence, including: The i-th processor performs a linear transformation on the intermediate decoding results within the group based on the sub-linear transformation weights to obtain the intermediate decoding results within the group after linear transformation. The processor cluster, based on the full reduction communication established within the processor cluster, aggregates the intra-group intermediate decoding results after linear transformation on each processor, and distributes the aggregated global intermediate decoding results to each processor. The i-th processor performs feedforward neural network processing on the global intermediate decoding result based on the sub-feedforward network weights to obtain a partial sequence output result corresponding to the task sequence. The partial sequence output result is the sequence output result of the attention head dimension corresponding to the i-th processor. The processor cluster, based on the full reduction communication established within the processor cluster, aggregates the partial sequence output results corresponding to the task sequences on each processor, and distributes the aggregated sequence output results corresponding to the task sequences to each processor.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: The scheduling device determines the processor with the smallest key-value cache size as the target processor in the processor group based on the key-value cache size of each processor in the processor group. After obtaining the sequence output result corresponding to the task sequence, the target processor caches the sequence output result in the key-value cache.

12. A model reasoning system, characterized in that, The system includes scheduling equipment and a processor cluster; The scheduling device is used to, upon receiving a model inference request, divide the task sequence corresponding to the model inference request based on the sequence parallelism to obtain multiple task sub-sequences; The scheduling device is used to allocate the multiple task sub-sequences to each processor group in the processor cluster. The number of processors in each processor group is the same as the parallelism of the sequence. Each processor in the processor group corresponds one-to-one with the task sub-sequence. Each processor in the processor group is configured with model sub-weights. The model sub-weights are obtained by partitioning the model weights based on the tensor parallelism. Different processor groups correspond to different model attention heads. The processor cluster is used to generate model inference results corresponding to the task sequence based on the sequence output results generated by each processor executing the task sub-sequence.

13. A model reasoning device, characterized in that, The device includes: The sequence partitioning module is used to partition the task sequence corresponding to the model inference request based on the sequence parallelism when a model inference request is received, thereby obtaining multiple task sub-sequences; The sequence allocation module is used to allocate the multiple task sub-sequences to each processor group in the processor cluster. The number of processors in each processor group is the same as the parallelism of the sequence. Each processor in the processor group corresponds one-to-one with the task sub-sequence. Each processor in the processor group is configured with model sub-weights. The model sub-weights are obtained by partitioning the model weights based on the tensor parallelism. Different processor groups correspond to different model attention heads.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement the model inference method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes a computer program / instruction that is executed by a processor to enable a computer to implement the model reasoning method according to any one of claims 1 to 11.

Images