Optimization method and device of mixed expert model, computer equipment, readable storage medium and program product

Abstract

The application relates to an optimization method and device of a hybrid expert model, computer equipment, a readable storage medium and a program product. The method comprises the following steps: a preset tensor parallel strategy is adopted in an attention module to perform calculation on a current input tensor on multiple artificial intelligence chips; a preset sequence parallel strategy is adopted by a full reduction operator to perform full reduction communication on the multiple artificial intelligence chips; during the communication process, the attention local calculation result is cut based on an input mark dimension, and the reduction calculation is performed in parallel by each artificial intelligence chip; a residual connection normalization module performs residual connection and normalization processing on the full reduction local calculation result output by each artificial intelligence chip on the artificial intelligence chip; and a hybrid expert module performs hybrid expert reasoning calculation on the normalized local result output by each artificial intelligence chip to obtain a final hybrid expert reasoning result. The method can improve the model reasoning efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence chips, in particular to a mixed expert model optimization method and device, computer equipment, readable storage medium and program product. BACKGROUND

[0002] Large language models are widely used in text translation, search, text creation and other fields, and it is of great significance to improve their reasoning efficiency. With the development of mixed expert models (Mixture of Experts, MoE), the demand for industrial-level optimization landing is increasingly urgent.

[0003] For parallel reasoning of large language models, the current traditional technical solution is to use a hybrid scheme of tensor parallelism (Tensor Parallelism, TP) and sequence parallelism (Sequence Parallelism, SP). Taking TP4SP4 as an example, the attention module (Attention module) and MLP (Multi-Layer Perceptron) module in the large language model perform TP4 parallelism, and the residual connection operator (Add) and normalization operator (RMSNorm) use the independent characteristics between input tokens (which can also be represented as Token) to perform SP4 parallelism. The input sequence is divided into 4 cards for independent calculation to ensure the correctness of reasoning; since the MLP module needs to perform TP4 parallelism, an AllGather operation needs to be introduced before the MLP to combine the calculation results of each card to meet the input requirements.

[0004] In the above scheme, the introduction of the AllGather operation will generate a large amount of inter-card communication overhead, completely offsetting the performance improvement brought by the SP4 parallelism of the Add / RMSNorm operator, and reducing the model reasoning efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a mixed expert model optimization method, device, computer equipment, readable storage medium and program product capable of improving the model reasoning efficiency in view of the above technical problems.

[0006] In a first aspect, the present application provides a mixed expert model optimization method, the mixed expert model comprising an attention module, an all-reduction operator, a residual connection normalization module and a mixed expert module, comprising:

[0007] In the calculation process of the mixed expert model, the attention module adopts a preset tensor parallelism strategy to calculate the current input tensor on multiple artificial intelligence chips to obtain attention local calculation results output by each artificial intelligence chip;

[0008] The full reduction operator adopts a preset sequence parallel strategy, performs full reduction communication on the plurality of artificial intelligence chips, and in the communication process, the attention local calculation result is divided based on the input label dimension, the reduction calculation is performed in parallel by each artificial intelligence chip, and the full reduction local calculation result output by each artificial intelligence chip is obtained.

[0009] The residual connection normalization module respectively performs residual connection and normalization processing on the full reduction local calculation result output by each artificial intelligence chip on each artificial intelligence chip to obtain the normalized local result output by the artificial intelligence chip.

[0010] The hybrid expert module performs hybrid expert reasoning calculation on the normalized local result output by each artificial intelligence chip to obtain the final hybrid expert reasoning result.

[0011] In one of the embodiments, the current input tensor is obtained by splicing different input sequences in different batches, and the data shape of the current input tensor is [B*S, H], wherein B represents the number of batches, S represents the sequence length, and H represents the feature dimension size.

[0012] In one of the embodiments, the data shape of the attention local calculation result is [B*S, H], and the data shape of the full reduction local calculation result is [B*S / N, H], wherein N represents the number of artificial intelligence chips.

[0013] In one of the embodiments, the full reduction communication is performed on the plurality of artificial intelligence chips, and in the communication process, the attention local calculation result is divided based on the input label dimension, the reduction calculation is performed in parallel by each artificial intelligence chip, and the full reduction local calculation result output by each artificial intelligence chip is obtained, which comprises:

[0014] In any artificial intelligence chip, the attention local calculation result output by each artificial intelligence chip is uniformly divided based on the input label dimension to obtain N input label fragment data, the input label fragment data corresponding to the artificial intelligence chip is retained, and the remaining input label fragment data is distributed to the corresponding artificial intelligence chip, and the data shape of the input label fragment data is [B*S / N, H];

[0015] Each artificial intelligence chip performs reduction calculation on the locally retained and received input label fragment data in parallel, and outputs the corresponding full reduction local calculation result, and the data shape of the full reduction local calculation result is [B*S / N, H].

[0016] In one of the embodiments, the retaining the input label shard data corresponding to the artificial intelligence chip and distributing the remaining input label shard data to the corresponding artificial intelligence chip comprises:

[0017] For the i-th artificial intelligence chip, the i-th input label shard data is retained, and the j-th input label shard data is distributed to the j-th artificial intelligence chip, i is a positive integer less than or equal to N, j is a positive integer less than or equal to N, and i and j are not equal.

[0018] In one of the embodiments, the hybrid expert module comprises a routing operator, a distribution operator, a routing-expert operator and an aggregation operator, and the hybrid expert module performs hybrid expert reasoning calculation on the normalized local results output by each of the artificial intelligence chips to obtain a final hybrid expert reasoning result, comprising:

[0019] The routing operator performs routing calculation on the normalized local result held by itself on each of the artificial intelligence chips to obtain target expert index data output by each of the artificial intelligence chips;

[0020] The distribution operator distributes the input label feature data in the normalized local result to the corresponding routing-expert operator across chips based on the target expert index data output by each of the artificial intelligence chips;

[0021] The routing-expert operator deployed on each of the artificial intelligence chips performs feature transformation calculation on the input label feature data distributed to itself to output corresponding expert local calculation results;

[0022] The aggregation operator deployed on each of the artificial intelligence chips performs weighted accumulation on different expert local calculation results corresponding to the same input label feature data on the artificial intelligence chip where the aggregation operator is located to obtain an aggregation intermediate result of the input label feature data, and synchronizes the aggregation intermediate result to the original artificial intelligence chip to which the input label feature data belongs, and each of the artificial intelligence chips integrates the aggregation intermediate results corresponding to the input label feature data responsible by itself to obtain a hybrid expert reasoning result.

[0023] In a second aspect, the application further provides an optimization device of a hybrid expert model, the hybrid expert model comprising an attention module, a full-reduction operator, a residual connection normalization module and a hybrid expert module, comprising:

[0024] The attention calculation module is configured to, in the calculation process of the hybrid expert model, the preset tensor parallel strategy is adopted in the attention module to perform calculation on a current input tensor on multiple artificial intelligence chips to obtain attention local calculation results output by each of the artificial intelligence chips;

[0025] The specification calculation module is configured to perform full-reduction communication on the plurality of artificial intelligence chips using a preset sequence parallel strategy, and to split the attention local calculation result based on an input label dimension during the communication, and to perform reduction calculation in parallel by each of the artificial intelligence chips to obtain a full-reduction local calculation result output by each of the artificial intelligence chips.

[0026] The normalization calculation module is configured to perform residual connection and normalization processing on the full-reduction local calculation result output by each of the artificial intelligence chips respectively on each of the artificial intelligence chips to obtain a normalized local result output by the artificial intelligence chip.

[0027] The inference module is configured to perform mixed expert inference calculation on the normalized local result output by each of the artificial intelligence chips by the mixed expert module to obtain a final mixed expert inference result.

[0028] In one of the embodiments, the current input tensor is obtained by splicing different input sequences in different batches, and the data shape of the current input tensor is [B*S, H], where B represents the number of batches, S represents the sequence length, and H represents the feature dimension size.

[0029] In one of the embodiments, the data shape of the attention local calculation result is [B*S, H], and the data shape of the full-reduction local calculation result is [B*S / N, H], where N represents the number of artificial intelligence chips.

[0030] In one of the embodiments, the full-reduction communication is performed on the plurality of artificial intelligence chips using a preset sequence parallel strategy, and the attention local calculation result is split based on an input label dimension during the communication, and reduction calculation is performed in parallel by each of the artificial intelligence chips to obtain a full-reduction local calculation result output by each of the artificial intelligence chips, including:

[0031] In each of the artificial intelligence chips, the attention local calculation result output by each of the artificial intelligence chips is uniformly split based on an input label dimension to obtain N input label slice data, the input label slice data corresponding to the artificial intelligence chip is retained, and the remaining input label slice data is distributed to the corresponding artificial intelligence chip, and the data shape of the input label slice data is [B*S / N, H];

[0032] Each of the artificial intelligence chips performs reduction calculation on the locally retained and received input label slice data in parallel, and outputs a corresponding full-reduction local calculation result respectively, and the data shape of the full-reduction local calculation result is [B*S / N, H].

[0033] In one of the embodiments, the reserving the input label shard data corresponding to the artificial intelligence chip and distributing the remaining input label shard data to the corresponding artificial intelligence chip comprises:

[0034] For the i-th artificial intelligence chip, the i-th input label shard data is reserved, and the j-th input label shard data is distributed to the j-th artificial intelligence chip, i is a positive integer less than or equal to N, j is a positive integer less than or equal to N, and i and j are not equal.

[0035] In one of the embodiments, the hybrid expert module comprises a routing operator, a distribution operator, a routing-expert operator and an aggregation operator, and the hybrid expert module performs hybrid expert reasoning calculation on the normalized local results output by each of the artificial intelligence chips to obtain a final hybrid expert reasoning result, comprising:

[0036] The routing operator performs routing calculation on the normalized local result held by itself on each of the artificial intelligence chips to obtain target expert index data output by each of the artificial intelligence chips;

[0037] The distribution operator distributes the input label feature data in the normalized local result to the corresponding routing-expert operator across chips based on the target expert index data output by each of the artificial intelligence chips;

[0038] The routing-expert operator deployed on each of the artificial intelligence chips performs feature transformation calculation on the input label feature data distributed to itself to output corresponding expert local calculation results;

[0039] The aggregation operator deployed on each of the artificial intelligence chips performs weighted accumulation on different expert local calculation results corresponding to the same input label feature data on the artificial intelligence chip where the aggregation operator is located to obtain an aggregation intermediate result of the input label feature data, and synchronizes the aggregation intermediate result to the original artificial intelligence chip to which the input label feature data belongs, and each of the artificial intelligence chips integrates the aggregation intermediate results corresponding to the input label feature data responsible by itself to obtain a hybrid expert reasoning result.

[0040] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the optimization method of the hybrid expert model of any one of the above aspects when executing the computer program.

[0041] In a fourth aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the optimization method of the hybrid expert model of any one of the above aspects.

[0042] In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the optimization method of the hybrid expert model of any one of the above.

[0043] The optimization method of the hybrid expert model, the device, the computer device, the readable storage medium and the program product, the hybrid expert model comprises an attention module, a full reduction operator, a residual connection normalization module and a hybrid expert module. In the calculation process of the hybrid expert model, the preset tensor parallel strategy is adopted in the attention module to calculate the current input tensor on multiple artificial intelligence chips to obtain the attention local calculation result output by each artificial intelligence chip. The full reduction operator adopts the preset sequence parallel strategy to perform full reduction communication on multiple artificial intelligence chips. In the communication process, the attention local calculation result is divided based on the input mark dimension, and the reduction calculation is performed in parallel by each artificial intelligence chip to obtain the full reduction local calculation result output by each artificial intelligence chip. The residual connection normalization module performs residual connection and normalization processing on the full reduction local calculation result output by each artificial intelligence chip on each artificial intelligence chip to obtain the normalized local result output by the artificial intelligence chip. The hybrid expert module performs hybrid expert reasoning calculation on the normalized local result output by each artificial intelligence chip to obtain the final hybrid expert reasoning result. The optimization method of the hybrid expert model, the device, the computer device, the readable storage medium and the program product provided in the present application embodiment reduce the communication cost between the artificial intelligence chips after the attention module by dividing the attention local calculation result into N parts based on the input mark dimension, and each artificial intelligence chip only holds 1 / N of the input mark data, that is, the full reduction communication only performs reduction aggregation on the 1 / N input mark data, so the communication amount is reduced to 1 / N of the original, which greatly reduces the communication cost between the artificial intelligence chips after the attention module. Moreover, the hybrid expert module in the present application embodiment can directly perform reasoning based on the input mark fragment data, so that the full reduction local calculation result output by the full reduction operator can directly obtain the normalized local result adapted to the hybrid expert module after residual connection normalization processing, without setting a global collection operator after the residual connection normalization module to perform data merging, that is, compared with the traditional technical solution, the global collection operator is eliminated in the present application embodiment, so that the model reasoning efficiency can be greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other related drawings without creative labor based on these drawings.

[0045] Figure 1A flowchart of the parallel inference of the existing dense model in one embodiment;

[0046] Figure 2 A flowchart of the optimization method of the hybrid expert model in one embodiment;

[0047] Figure 3 A structure diagram of the GPGPU in one embodiment;

[0048] Figure 4 A flowchart of step 202 in one embodiment;

[0049] Figure 5 A flowchart of step 208 in one embodiment;

[0050] Figure 6 A structure diagram of the MLP in one embodiment;

[0051] Figure 7 A parallel inference diagram of the optimized hybrid expert model in one embodiment;

[0052] Figure 8 A structure block diagram of the optimization device of the hybrid expert model in one embodiment;

[0053] Figure 9 An internal structure diagram of the computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0055] It should be noted that the terms "first", "second", and the like used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application refers to two or more. The term "and / or" used in the present application refers to one of the options, or any combination of a plurality of options.

[0056] Referring to Figure 1 It is shown that the parallel inference scheme of the existing dense model is shown. Taking the parallel inference scheme of the dense model using TP4+SP4 as an example, that is, the dense model uses 4-way tensor parallelism (TP4) + 4-way sequence parallelism (SP4) parallel inference configuration, the parallel strategy and logic of each module in the dense model are as follows:

[0057] The attention module and the MLP module perform TP4 parallelism; the residual connection normalization module (a residual connection operator and a normalization operator or a fusion operator of the two) performs SP4 parallelism. The SP4 parallelism logic of the residual connection normalization module is as follows: because the residual connection operator and the normalization operator are independent calculations (without cross-input token interaction) between input tokens, the implementation manner of SP4 parallelism is to split the sequence dimension to four chips, and each chip independently calculates the input token feature data allocated thereto. This manner not only improves the efficiency by using multi-chip parallelism, but also ensures the correctness of the inference result.

[0058] Due to the memory limitation, the weights of the MLP module need to be split and deployed on different artificial intelligence chips, and therefore the MLP module needs to adopt TP4 parallelism; and the TP4 parallelism requires that the input data is complete in the shape of [B, S, Hidden_Size], where B represents the batch number, S represents the sequence length, and Hidden_Size represents the size of the feature data dimension. Therefore, an AllGather operation needs to be introduced before the MLP module to combine the input token split data of each artificial intelligence chip into complete data, and then the complete data is input into the MLP module for calculation.

[0059] When the hybrid expert model replaces the MLP module of the dense model with the MoE module, the problems of the above parallelism scheme will appear: the sequence parallelism (SP4) can bring performance improvement to the Add / RMSNorm operator, but the AllGather operation introduced in the dense model to adapt to the TP4 of the MLP module will offset this part of performance gain; finally, the parallelism scheme cannot realize effective parallel performance optimization in the MoE model inference scene.

[0060] The embodiment of the application provides an optimization method of a hybrid expert model, which eliminates the AllGather operation after the attention module, thereby eliminating the performance influence of the AllGather operation on the SP4 parallel operation of the Add / RMSNorm operator, and further improving the model inference efficiency.

[0061] As Figure 2As shown, a hybrid expert model optimization method is provided and applied to an artificial intelligence chip. In this embodiment, the artificial intelligence chip can be any one of GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), NPU (Neural Network Processing Unit), DPU (Deep Learning Processing Unit), APU (Accelerated Processing Unit), and GPGPU (General-Purpose Graphics Processing Unit). This embodiment does not specifically limit the specific type of chip, and the following description uses GPGPU as an example.

[0062] Reference Figure 3 The diagram shows a schematic of a GPGPU. A GPGPU is actually an array of Streaming Processor Clusters (SPCs), including, for example,... Figure 3 The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a graphics processing unit (GPU), one streaming processor cluster processes one computational task, or multiple streaming processor clusters process one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0063] like Figure 3 As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 3 The system is structured as Computation Unit 1, Computation Unit 2, ..., Computation Unit N, where N is a positive integer. Each Computation Unit (CU) performs arithmetic and logical operations other than matrix calculations such as matrix multiplication and convolution, including operations like accumulation, reduction, and standard addition, subtraction, multiplication, and division. A Computation Unit contains multiple cores (also called computational kernels), each including an Arithmetic Logic Unit (ALU), a floating-point unit, etc., which are used to execute specific computational tasks. Furthermore, the Computation Unit also includes registers (e.g., ...). Figure 3 The register file and shared cache in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared cache in a computing unit is used to share data between the cores of that computing unit.

[0064] In parallel computing, a computing task is generally executed by multiple threads. These threads are divided into multiple thread blocks before being executed in a general-purpose graphics processor (or parallel computing processor), and then the multiple thread blocks are distributed to each computing unit via a thread block distribution module (not shown in Figure 3 ). All threads in a thread block must be assigned to the same computing unit for execution. Meanwhile, a thread block is split into minimum execution thread bundles (or simply referred to as thread bundles, warp), each of which contains a fixed number (or less than the fixed number) of threads, for example, 32 threads. Multiple thread blocks can be executed in the same computing unit or in different computing units.

[0065] In each computing unit, a thread bundle scheduling / distribution module (not shown in Figure 3 ) schedules and assigns thread bundles so that multiple computing cores of the computing unit run the thread bundles. According to the number of computing cores in the computing unit, multiple thread bundles in a thread block can be executed simultaneously or time-divisionally. Multiple threads in each thread bundle execute the same instruction. Memory execution instructions are transmitted to a shared cache in the computing unit or further transmitted to an intermediate-level cache or a global cache or a global memory for read / write operations, etc.

[0066] As shown in Figure 3 , the stream processor cluster 1 further includes a tensor operation unit for performing tensor computation, for example, the tensor computation can include matrix multiplication, convolution operation, etc.

[0067] Referring to Figure 2 , the embodiments of the present application provide an optimization method of a hybrid expert model, wherein the hybrid expert model includes an attention module, a full reduction operator, a residual connection normalization module and a hybrid expert module, and the method can include the following steps:

[0068] Step 202, in the calculation process of the hybrid expert model, the preset tensor parallel strategy is adopted in the attention module to calculate the current input tensor on multiple artificial intelligence chips to obtain the attention local calculation results output by each artificial intelligence chip.

[0069] The optimization method of the mixed expert model provided in the embodiments of the present application can be applied to the field of artificial intelligence such as natural language processing and image recognition to improve the efficiency and accuracy of the model in processing complex tasks. In the following embodiments, text classification in natural language processing tasks will be taken as an example for illustration. It is assumed that the input of the model is a conversation text initiated by a user, which is cut into multiple input tokens (which can also be represented as Token) after preprocessing, forms a batched input sequence, and obtains a current input tensor after embedding layer and position encoding processing, and the data shape of the current input tensor is [B*S, H] (where B represents the number of batches, S represents the input token sequence length in a single batch, and H represents the feature dimension size).

[0070] In the embodiments of the present application, the attention module is used to capture the semantic association between input tokens in the input sequence, and its calculation process involves a large number of matrix multiplication and softmax operation, which has a high demand for hardware memory and computing power. In order to solve the single-chip memory bottleneck and improve the calculation efficiency, the attention module can adopt the preset tensor parallel strategy. Exemplarily, the following embodiments of the present application take a 4-way tensor parallel strategy as an example (i.e., the calculation task is distributed and deployed on 4 artificial intelligence chips).

[0071] Exemplarily, first, weight division is performed, that is, the weights of the attention module (including query matrix weight, key matrix weight, and value matrix weight) are uniformly divided according to the feature dimension (i.e., H dimension), and each artificial intelligence chip only holds 1 / 4 of the complete weight; each artificial intelligence chip receives a complete current input tensor and independently performs local attention calculation based on the held weight fragment to obtain an attention local calculation result. Since the weight deployed on a single artificial intelligence chip is only a weight fragment in the feature dimension, the attention local calculation result output by each artificial intelligence chip is still in a fragmented state in the feature dimension, and the attention calculation result can be restored after splicing the attention local calculation results of all artificial intelligence chips.

[0072] It should be noted that the specific number of preset tensor parallel in the embodiments of the present application is not limited, and can be dynamically adjusted according to the hardware cluster size (such as TP8 when 8 artificial intelligence chips are used) or the model complexity, as long as the weight division and distributed parallel calculation can be realized, which can be applied to the embodiments of the present application.

[0073] In step 204, the all-reduce operator adopts the preset sequence parallel strategy to perform all-reduce communication on multiple artificial intelligence chips, and the attention local calculation result is divided based on the input token dimension during the communication, and the reduction calculation is performed in parallel by each artificial intelligence chip to obtain the all-reduce local calculation result output by each artificial intelligence chip.

[0074] In the embodiments of the present application, the target of the all-reduce operator (also referred to as an AllReduce operator) is to simultaneously realize feature dimension aggregation and input token dimension segmentation. The operator adopts a preset sequence parallel strategy matching the number of tensor parallel paths. For example, the preset tensor parallel strategy adopts 4-way parallelism, and the preset sequence parallel strategy also adopts 4-way parallelism, that is, the all-reduce communication and calculation are cooperatively executed on 4 artificial intelligence chips.

[0075] For example, in the communication initialization phase, the attention local calculation results output by each artificial intelligence chip can be uniformly and non-overlappingly segmented based on the input token (also referred to as Token) dimension (i.e., the dimension corresponding to the input sequence length B*S), and the total input token quantity is divided into 4 equal parts, and each chip only retains 1 / 4 of the input token segment data. Then, each artificial intelligence chip uses a distributed communication protocol such as ring communication and tree aggregation to perform global reduction aggregation on the feature dimension of the input token segment data it holds, that is, each artificial intelligence chip sends its input token segment data to other chips, and at the same time receives the input token segment data sent by other chips, and through element-by-element addition, mean calculation and other reduction operations, the 4 input token segment data are aggregated into complete input token feature data. After the reduction calculation is completed, each chip only retains the input token feature data corresponding to the input token segment it is responsible for, and this data is the all-reduce local calculation result.

[0076] In step 206, the residual connection normalization module performs residual connection and normalization processing on the all-reduce local calculation results output by each artificial intelligence chip on each artificial intelligence chip, to obtain the normalized local results output by the artificial intelligence chip.

[0077] In the embodiments of the present application, the residual connection normalization module is composed of a residual connection operation (corresponding to an Add operator) and a normalization operation (corresponding to an RMSNorm operator) in series, which functions to alleviate the gradient vanishing problem in deep network inference, and optimize the feature value distribution through normalization to improve the model calculation stability. Since the residual connection and normalization operations are both input token (also referred to as Token) level independent calculations, that is, there is no need for cross-input token interaction, the residual connection normalization module can directly adapt to the all-reduce local calculation results (i.e., input token feature data corresponding to input token segments) output by the all-reduce operator, and independently perform processing on each artificial intelligence chip without additional cross-card communication.

[0078] Exemplarily, the full-reduction local calculation result of the current artificial intelligence chip can be subjected to element-by-element addition operation with the residual input data transmitted by the previous layer network to the chip, to fuse the features of the current layer and the historical layer features; then, the RMSNorm (root mean square normalization) algorithm is adopted to perform normalization calculation in the feature dimension in units of a single input token, to eliminate the difference in the feature value magnitude between different input tokens, and then the mixed expert module can be directly input for processing.

[0079] In step 208, the mixed expert module performs mixed expert reasoning calculation on the normalized local results output by each artificial intelligence chip to obtain a final mixed expert reasoning result.

[0080] In the embodiment of the application, the mixed expert module can directly adapt to the input token slicing characteristics of the normalized local results, and perform reasoning calculation in parallel on multiple artificial intelligence chips. Exemplarily, each artificial intelligence chip only performs mixed expert reasoning on the normalized local results held by itself, including routing decision (determining the expert network to be activated for each input token), input token distribution (sending the input token to the corresponding expert chip), expert calculation (the activated expert network transforms the input token feature), result aggregation (weighted accumulation of multiple expert outputs for the same input token), and the like.

[0081] Since the expert networks are distributedly deployed on different artificial intelligence chips, the input token distribution and result aggregation process can be realized through low-overhead cross-card communication, but only the input token data to be processed by the current expert and the aggregated local results are transmitted, which can avoid full data transfer. After all the artificial intelligence chips complete local reasoning and cross-card collaboration, the aggregated results corresponding to the input token slices respectively responsible by the artificial intelligence chips are spliced in the order of the input token dimension, and finally the complete mixed expert reasoning result is obtained. Taking the intelligent dialogue scenario as an example, the reasoning result can be directly input to the output layer (such as linear projection layer + softmax layer) of the model to generate a reply text corresponding to the user dialogue, and complete the intelligent dialogue generation task.

[0082] The optimization method of the mixed expert model comprises an attention module, a full reduction operator, a residual connection normalization module and a mixed expert module. In the calculation process of the mixed expert model, the preset tensor parallel strategy is adopted in the attention module to calculate the current input tensor on multiple artificial intelligence chips to obtain the attention local calculation results output by each artificial intelligence chip. The full reduction operator adopts the preset sequence parallel strategy to perform full reduction communication on the multiple artificial intelligence chips. In the communication process, the attention local calculation results are divided based on the input mark dimension, and the reduction calculation is performed in parallel by each artificial intelligence chip to obtain the full reduction local calculation results output by each artificial intelligence chip. The residual connection normalization module performs residual connection and normalization processing on the full reduction local calculation results output by each artificial intelligence chip on each artificial intelligence chip to obtain the normalized local results output by the artificial intelligence chip. The mixed expert module performs mixed expert reasoning calculation on the normalized local results output by each artificial intelligence chip to obtain the final mixed expert reasoning result. By using the optimization method of the mixed expert model provided in the application, the full reduction operator divides the attention local calculation results into N parts based on the input mark dimension, each artificial intelligence chip only holds 1 / N of the input mark data, that is, the full reduction communication only performs reduction aggregation on the 1 / N input mark data, so the communication amount is reduced to 1 / N of the original amount, greatly reducing the communication overhead between the artificial intelligence chips after the attention module. Moreover, since the mixed expert module in the application example can directly perform reasoning based on the input mark fragment data, the full reduction local calculation results output by the full reduction operator can directly obtain the normalized local results suitable for the mixed expert module after the residual connection normalization processing, without the need to set a global collection operator after the residual connection normalization module to perform data merging, that is, compared with the traditional technical solution, the global collection operator is eliminated in the application example, so the model reasoning efficiency can be greatly improved.

[0083] In an exemplary embodiment, the current input tensor is obtained by splicing different input sequences in different batches, and the data shape of the current input tensor is [B*S, H], wherein B represents the number of batches, S represents the sequence length, and H represents the feature dimension size.

[0084] In the embodiments of the present application, the current input tensor is obtained by splicing different input sequences in different batches, thereby adapting to the input requirements of distributed parallel computing and avoiding the waste of computing power and communication redundancy caused by separate processing of multiple batches of data. Taking the data shape of the current input tensor as [B*S, H] for example, B represents the number of batches, i.e., the number of groups of input sequences processed in parallel at a time; S represents the length of the input sequence in a single batch, i.e., the number of input tokens included in each group of input sequences; therefore, B*S represents the total number of input tokens after splicing the input sequences of all batches, i.e., the length of the first dimension of the current input tensor; and H represents the feature dimension size, i.e., the dimension of the feature vector corresponding to each input token. For example, when H=1024, it means that each input token is encoded into a 1024-dimensional feature vector.

[0085] For example, after the input text is preprocessed (including word segmentation, embedding encoding, length padding, etc.), it is arranged in the batch dimension in turn, and then the dimension splicing operation is performed to integrate multiple batches of [1, S, H] shaped sequences into a single tensor [B*S, H]. For example, after splicing 3 groups of sequences with a length of 256, an input token sequence with a total length of 768 is generated, and the corresponding tensor shape is [768, 1024]. This design enables multiple batches of data to be processed through a single calculation, and the tensor shape is highly compatible with the parallel computing logic of subsequent modules, without the need for additional dimension reconstruction operations.

[0086] In an exemplary embodiment, the data shape of the attention local calculation result is [B*S, H], and the full-reduction local calculation result has a data shape of [B*S / N, H], where N represents the number of artificial intelligence chips.

[0087] In the embodiments of the present application, the data shape of the attention local calculation result is [B*S, H], which is exactly the same as the shape of the current input tensor. The attention module adopts a preset tensor parallel strategy (the number of parallel paths matches the number of artificial intelligence chips N), i.e., the weights of the attention module are divided into N chips according to the feature dimension, and each chip holds 1 / N of the complete weights, but the input data received is the complete current input tensor [B*S, H]. Each chip independently performs local attention calculation based on the weight fragment it holds (including QKV (query, key and value) matrix multiplication, attention score calculation, context vector aggregation, etc.), and since the calculation process only splits the weights but not the input tokens or feature dimensions of the input data, the output attention local calculation result still maintains the complete shape of [B*S, H].

[0088] The data shape of the full reduction local computation result is [B*S / N, H], where N represents the number of AI chips, i.e. the hardware scale of the distributed computing cluster; B*S / N represents the number of input labels that each chip is responsible for processing after being partitioned by the input label dimension, i.e. the length of the first dimension of the current input tensor; H is still the feature dimension size, ensuring the integrity of the feature dimension of each input label.

[0089] For example, the full reduction operator employs a pre-defined sequence parallel strategy, uniformly and non-overlappingly partitioning the attention local computation results output by each chip based on the input label dimension. That is, the total number of input labels (B*S) is divided into N equal parts, and each chip retains only the input label fragment data corresponding to 1 / N input labels. At the same time, feature dimension aggregation is performed on the partitioned input label fragment data through distributed reduction communication to eliminate the feature differences caused by tensor parallelism and ensure that the feature dimension H of each input label remains intact. Finally, the shape of the full reduction local computation result output by each chip is [B*S / N, H]. In this way, both the parallel partitioning of the input label dimension and the integrity of the feature dimension are preserved, perfectly adapting to the input requirements of the residual connection normalization module.

[0090] In one exemplary embodiment, reference is made to Figure 4 As shown, in step 202, full reduction communication is performed on multiple AI chips. During the communication process, the attention local computation result is segmented based on the input label dimension, and the reduction computation is performed in parallel by each AI chip to obtain the full reduction local computation result output by each AI chip. This may include the following steps:

[0091] Step 402: Within any AI chip, the attention local calculation results output by each AI chip are uniformly divided based on the input label dimension to obtain N input label fragment data. The input label fragment data corresponding to the AI ​​chip is retained, and the remaining input label fragment data is distributed to the corresponding AI chip. The data shape of the input label fragment data is [B*S / N, H].

[0092] Step 404: Each AI chip performs reduction calculations in parallel on the locally retained and received input labeled fragment data, and outputs the corresponding full reduction local calculation results. The shape of the full reduction local calculation result data is [B*S / N, H].

[0093] In the embodiments of the present application, the input label dimension can be taken as the splitting reference (i.e. B*S in [B*S, H]), and the total input label quantity is split into N equal parts according to the number of artificial intelligence chips N by using the uniform non-overlapping splitting strategy, to obtain N input label split data, and the data shape is [B*S / N, H]. Each artificial intelligence chip only retains one input label split data corresponding to its own number, for example, the first artificial intelligence chip retains the first input label split data (corresponding to the first 192 input labels), the second artificial intelligence chip retains the second input label split data (corresponding to the 193th-384th input labels), and so on.

[0094] Subsequently, the artificial intelligence chip distributes the remaining N-1 input label split data not belonging to the chip to the corresponding artificial intelligence chip through a low-delay distributed communication protocol (such as PCIe (Peripheral Component Interconnect Express, Peripheral Component Interconnect Express)), for example, the first artificial intelligence chip distributes the second input label split data to the second artificial intelligence chip, and distributes the third input label split data (corresponding to the 385th-576th input labels) to the third artificial intelligence chip, and so on.

[0095] In an exemplary embodiment, the step of retaining the input label split data corresponding to the artificial intelligence chip and distributing the remaining input label split data to the corresponding artificial intelligence chip in step 402 can include the following steps:

[0096] For the ith artificial intelligence chip, the ith input label split data is retained, and the jth input label split data is distributed to the jth artificial intelligence chip, i is a positive integer less than or equal to N, j is a positive integer less than or equal to N, and i and j are not equal.

[0097] In the embodiments of the present application, i is a positive integer less than or equal to N, representing the number of the artificial intelligence chip currently performing the operation (hereinafter referred to as the ith chip for clarity), and the value range is i=1, 2, 3, …, N; j is a positive integer less than or equal to N, representing the number of the input label split data (hereinafter referred to as the jth split for clarity), and the value range is j=1, 2, 3, …, N, and i and j are not equal (i.e. the chip does not distribute the input label split data to itself).

[0098] For example, the first chip retains the first input label split data, the second chip retains the second input label split data, and so on.

[0099] The remaining N-1 input token slice data held by the i-th artificial intelligence chip, except for the i-th input token slice data, can be distributed to the corresponding j-th artificial intelligence chip according to the rule that the slice number is consistent with the chip number. For example, the first chip holds the first, second, third and fourth input token slice data, and after reserving the first input token slice data, the second input token slice data is distributed to the second chip, the third input token slice data is distributed to the third chip, and the fourth input token slice data is distributed to the fourth chip; the rest of the chips are similar, ensuring that the j-th input token slice data on each chip can be distributed to the j-th chip.

[0100] In this way, all input token slice data can be accurately allocated one by one, so that each artificial intelligence chip only retains one input token slice data responsible for itself, while receiving input token slice data corresponding to its own number from other N-1 artificial intelligence chips. Finally, each artificial intelligence chip holds all the relevant input token slice data required for complete reduction calculation, and there is no overlap or omission between input token slice data, completely covering the input token data of the complete input sequence.

[0101] It should be noted that the starting value of i and j (such as starting from 0 or 1) is not specifically limited in the embodiments of the present application, as long as the chip number and the slice number are one-to-one mapping and i≠j, which can be applied to the embodiments of the present application.

[0102] After completing the distribution of the input token slice data, the artificial intelligence chip can aggregate the feature information of all input token slice data through distributed reduction calculation. For example, after each artificial intelligence chip receives N-1 input token slice data from other chips, it integrates the 1 input token slice data retained locally, and holds N input token slice data. Each artificial intelligence chip starts reduction calculation in parallel (for example, the reduction calculation can be a reduction method of element-by-element accumulation), aggregates the feature information of N input token slice data, and obtains the corresponding full-reduction local calculation result.

[0103] For example, artificial intelligence chip 1 is responsible for the first input token slice data (corresponding to the first-192 input token), which can receive the feature data of the input token slice data (also corresponding to the first-192 input token) sent by other artificial intelligence chips 2, 3 and 4, and perform element-by-element accumulation on the four input token slice data held to eliminate the feature difference caused by tensor parallelism, to obtain the corresponding full-reduction local calculation result. After the reduction calculation is completed, each artificial intelligence chip outputs the corresponding full-reduction local calculation result, which has the same shape as the input token slice data, still [B*S / N, H], and the full-reduction local calculation results of all artificial intelligence chips are spliced according to the input token dimension, and the complete reduction result can be restored.

[0104] It should be noted that the specific way of reduction calculation (such as accumulation, mean value, maximum value) is not limited in the embodiments of the present application, and can be dynamically adjusted according to the model calculation requirements (such as attention feature aggregation logic), as long as the global aggregation of feature dimensions can be realized and the output result shape is [B*S / N, H], which can be applied to the embodiments of the present application.

[0105] In an exemplary embodiment, the hybrid expert module includes a routing operator, a distribution operator, a routing-expert operator, and an aggregation operator, as shown in Figure 5 As shown in FIG. 8, in step 208, the hybrid expert module performs hybrid expert reasoning calculation on the normalized local results output by each artificial intelligence chip to obtain the final hybrid expert reasoning result, which can include the following steps:

[0106] In step 502, the routing operator performs routing calculation on the normalized local result held by itself on each artificial intelligence chip to obtain the target expert index data output by each artificial intelligence chip.

[0107] In step 504, the distribution operator distributes the input marked feature data in the normalized local result to the corresponding routing-expert operator across chips based on the target expert index data output by each artificial intelligence chip.

[0108] In step 506, the routing-expert operator deployed on each artificial intelligence chip performs feature transformation calculation on the input marked feature data distributed to itself to output the corresponding expert local calculation result.

[0109] In step 508, the aggregation operator deployed on each artificial intelligence chip performs weighted accumulation on the different expert local calculation results corresponding to the same input marked feature data on the artificial intelligence chip where the aggregation operator is located to obtain the aggregation intermediate result of the input marked feature data, and synchronizes the aggregation intermediate result to the original artificial intelligence chip to which the input marked feature data belongs. Each artificial intelligence chip integrates the aggregation intermediate results corresponding to the input marked feature data that it is responsible for to obtain the hybrid expert reasoning result.

[0110] In the embodiments of the present application, the hybrid expert module includes a routing operator (also known as a Router operator), a distribution operator (also known as a Dispatch operator), a routing-expert operator (also known as a Router-Experts operator), and an aggregation operator (also known as a Combine operator). The routing operator is used to realize the dynamic matching of input marks and routing-expert operators, and reduces the consumption of invalid computing power by activating the routing-expert operator sparsely.

[0111] Exemplarily, each artificial intelligence chip only performs routing calculation on the normalized local result held by itself, and a routing operator performs linear transformation and softmax normalization on each dimension feature vector of each input label based on pre-trained parameters to obtain matching weights of the input label corresponding to all E routing-expert operators. Further, a Top-K routing strategy (exemplarily, K=2, i.e., 2 routing-expert operators are activated for each input label) can be used to filter out the top K experts with the highest matching weights to generate target expert index data, and each element in the target expert index data represents the expert network number to be activated for the corresponding input label.

[0112] The distribution operator is used to implement distribution of the input label data according to the routing decision. Exemplarily, a mapping relationship between the expert numbers and the chip numbers can be established in advance. Exemplarily, the first and second experts are deployed on chip 1, the third and fourth experts are deployed on chip 2, the fifth and sixth experts are deployed on chip 3, and the seventh and eighth experts are deployed on chip 4. For multiple input labels on each artificial intelligence chip, the input labels that need to be distributed to the expert networks of other chips are filtered out based on the target expert index data, the filtered input label feature data is distributed to the corresponding artificial intelligence chip across chips through a low-latency distributed communication protocol, and only the feature data of the input label and the corresponding routing weight are transmitted. Each artificial intelligence chip retains the input labels that do not need to be distributed across chips (i.e., the target experts are deployed on the input labels of the chip), and directly inputs the local routing-expert operator for processing to avoid invalid data transfer.

[0113] The routing-expert operator is an operator integrating routing weight fusion and expert feature transformation functions. Exemplarily, the routing-expert operator on each artificial intelligence chip includes two independent expert networks (such as a network composed of two fully connected networks + an activation function), and each expert network only processes the input label feature data distributed to itself. The expert network in the routing-expert operator receives the input label data retained by the local chip and the input label data distributed by other chips, and performs expert network calculation on the input label feature data based on the routing weight corresponding to the input label. Exemplarily, the expert network first maps the low-dimensional feature to high-dimensional through the fully connected network, then introduces the nonlinear transformation operation through the activation function, and then maps back to the original low-dimensional through the output layer, and each expert network can output the corresponding expert local calculation result, which has the same shape as the input label and the result is fused with the weighting factor of the routing weight.

[0114] It should be noted that the specific structure of the expert network (such as the number of layers, the dimension of the hidden layer), the type of activation function is not limited in the embodiments of the present application, as long as the input label feature transformation and the output result shape consistent with the input can be applied to the embodiments of the present application.

[0115] The aggregation operator is used to aggregate multiple expert network computing results and restore the complete input token sequence to ensure that the output result is consistent with the input tensor shape. For example, for multiple expert local computing results of the same input token on each artificial intelligence chip, the aggregation operator can perform a weighted accumulation operation according to the routing weight, that is, element-wise addition of the expert local computing results to obtain the aggregation intermediate result of the input token. Further, the aggregation intermediate result is synchronized to the original artificial intelligence chip to which the input token belongs through a distributed communication protocol. For example, the original artificial intelligence chip 1 is responsible for an input token, and the aggregation intermediate result of the input token is calculated by the artificial intelligence chips 2 and 3. The aggregation intermediate result needs to be synchronized back to the artificial intelligence chip 1 to ensure that the original artificial intelligence chip holds the complete aggregation result of the input token.

[0116] After each artificial intelligence chip receives all the synchronized aggregation intermediate results, the aggregation intermediate results corresponding to each input token are spliced into a continuous feature sequence in the original order of the input token, and finally the complete mixed expert reasoning result is obtained, which has the same data shape as the initial current input tensor, that is, [B*S, K].

[0117] The mixed expert reasoning result can be directly input into the subsequent output layer of the model to generate a result (such as a reply content of intelligent continuation) that meets the task requirements, thereby completing the entire inference process of the mixed expert model.

[0118] In order to enable those skilled in the art to better understand the embodiments of the present application, the embodiments of the present application are described below through specific examples.

[0119] In the dense model inference scenario, due to the limitation of the video memory, the weights of the MLP module have to be divided and placed on different chips, which requires the MLP module to be TP parallel, which requires the input to the MLP module to be a complete B*S*H tensor data.

[0120] However, in the mixed expert model inference scenario, the MoE module replaces the original MLP module, and the weight TP becomes the expert EP (Expert Parallelism, expert parallelism). The Router-Expert operator (Router-Expert operator) needs to be inserted with a dispatch operator (Dispatch operator) and a combine operator (Combine operator) before and after for cross-machine / cross-chip communication. One Router-Expert operator corresponds to one MLP, and the MLP is calculated as follows Figure 6As shown, it is a series of operations between input tokens that are irrelevant, where Silu / Mul (i.e. activation function / multiplication operator) is Point Wise operation (i.e. element-wise operation), Gate-Linear / Up-Linear / Down-Linear (i.e. gated linear layer / up-dimensional linear layer / down-dimensional linear layer) is matrix multiplication, and the result of matrix multiplication is the dimension of each input token. According to the characteristics of matrix multiplication, it can be seen that each input token does not participate in the operation with each other. Therefore, each card can be completely dynamic according to the number of experts. Based on this principle, the Dispatch dynamic operation of the feature dimension of each input token can be used on the basis of the parallel of the Add / RMSNorm sequence, and the AllGather operation can be eliminated to realize the SP of the Add / RMSNorm in the true sense.

[0121] Referring to Figure 7 As shown, the process of using experts in parallel to realize the input token parallel calculation of the Add / RMSNorm module in the MoE model structure is shown, including:

[0122] After entering the whole model inference, different sequences of different batches are spliced (i.e. packing operation) together to obtain the current input tensor of [B*S, 2048], and after the attention module does TP4 calculation, AllReduce communication is performed to obtain the complete data on the feature dimension after the attention module. The attention module adopts a 4-way tensor parallel strategy and outputs the attention local calculation result with a data shape of [B*S, 2048] on four artificial intelligence chips respectively.

[0123] In the full reduction operator calculation process, all attention local calculation results with a shape of [B*S, 2048] output by the attention module are divided into four chips with input tokens as the granularity to do AllReduce operation. Each card only does 1 / 4 of the B*S input token data, and after doing it, only 1 / 4 of the B*S data is retained, i.e. the data shape of the full reduction local calculation result output by each chip is [B*S / 4, 2048].

[0124] In the residual connection normalization module, because the Add calculation is Point Wise operation, and the RMSNorm is normalized in the 2048 dimension with input tokens as the unit, each card only does 1 / 4 of the B*S data, so the data shape of the full reduction local calculation result output is [B*S / 4, 2048].

[0125] In the distribution operator, the data of the feature dimension of each input label is sent to the chip of the corresponding calculation expert for routing expert calculation, where the amount of data of the feature dimension of the input label calculated on each chip is determined by the result of the routing operator, and the number of input labels calculated by the four chips in the figure is B*S / x, B*S / y, B*S / z and B*S / t respectively, where the specific values of x, y, z and t are determined based on the result of the routing operator.

[0126] After the routing-expert calculation is completed, the aggregation operator accumulates the local calculation results of different routing experts corresponding to the same input label of each chip, and then transmits the data between different chips across chips. In order to facilitate the TP calculation of the attention module, the output of the aggregation operator needs to transmit the feature data of the feature dimension of all input labels of different chips across chips, and then accumulate the local calculation results of different experts corresponding to the same input label to obtain the final output result of the MoE module.

[0127] The optimization method of the mixed expert model provided in the embodiments of the present application reduces the AllReduce communication quantity after the attention module to 1 / 4, eliminates the AllGather communication operator after the residual connection normalization module, reduces the calculation quantity of the routing operator in the mixed expert module to 1 / 4, and reduces the communication quantity of the distribution operator to a certain extent.

[0128] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0129] Based on the same inventive concept, the embodiments of the present application also provide an optimization device of a mixed expert model for implementing the optimization method of the mixed expert model as described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more optimization device embodiments of the mixed expert model provided below can refer to the limitations of the optimization method of the mixed expert model described above, which will not be repeated here.

[0130] In one exemplary embodiment, as shown in Figure 8 An optimization device 800 of a hybrid expert model is provided, the hybrid expert model comprising an attention module, a full reduction operator, a residual connection normalization module and a hybrid expert module, comprising: an attention calculation module 802, a reduction calculation module 804, a normalization calculation module 806 and an inference module 808, wherein:

[0131] The attention calculation module 802 is configured to, in the calculation process of the hybrid expert model, adopt a preset tensor parallel strategy to calculate a current input tensor on multiple artificial intelligence chips in the attention module, to obtain an attention local calculation result output by each of the artificial intelligence chips.

[0132] The reduction calculation module 804 is configured to adopt a preset sequence parallel strategy to perform full reduction communication on the multiple artificial intelligence chips in the full reduction operator, and in the communication process, the attention local calculation result is divided based on an input label dimension, and a reduction calculation is performed in parallel by each of the artificial intelligence chips, to obtain a full reduction local calculation result output by each of the artificial intelligence chips.

[0133] The normalization calculation module 806 is configured to perform residual connection and normalization processing on the full reduction local calculation result output by each of the artificial intelligence chips in the residual connection normalization module on each of the artificial intelligence chips, to obtain a normalized local result output by the artificial intelligence chip.

[0134] The inference module 808 is configured to perform hybrid expert inference calculation on the normalized local result output by each of the artificial intelligence chips in the hybrid expert module, to obtain a final hybrid expert inference result.

[0135] By using the optimization device of the hybrid expert model provided in the embodiment, the full reduction operator divides the attention local calculation result into N parts based on the input label dimension, each artificial intelligence chip only holds 1 / N of the input label data, that is, the full reduction communication only performs reduction aggregation on the 1 / N input label data, so the communication amount is reduced to 1 / N of the original amount, greatly reducing the communication overhead between the artificial intelligence chips after the attention module. Moreover, since the hybrid expert module in the embodiment can directly perform inference based on the input label fragment data, the full reduction local calculation result output by the full reduction operator can directly obtain the normalized local result adapted to the hybrid expert module after residual connection normalization processing, without the need to set a global collection operator after the residual connection normalization module to perform data merging, that is, compared with the traditional technical solution, the global collection operator is eliminated in the embodiment, so the model inference efficiency can be greatly improved.

[0136] In one of the embodiments, the current input tensor is obtained by concatenating different input sequences in different batches, and the data shape of the current input tensor is [B*S, H], where B represents the number of batches, S represents the sequence length, and H represents the feature dimension size.

[0137] In one of the embodiments, the data shape of the attention local calculation result is [B*S, H], and the full-reduction local calculation result has a data shape of [B*S / N, H], where N represents the number of artificial intelligence chips.

[0138] In one of the embodiments, the full-reduction communication is performed on the plurality of artificial intelligence chips, and the attention local calculation result is divided based on the input label dimension during the communication, and the reduction calculation is performed in parallel by each of the artificial intelligence chips to obtain the full-reduction local calculation result output by each of the artificial intelligence chips, including:

[0139] In any of the artificial intelligence chips, the attention local calculation result output by each of the artificial intelligence chips is uniformly divided based on the input label dimension to obtain N input label segment data, the input label segment data corresponding to the artificial intelligence chip is retained, and the remaining input label segment data is distributed to the corresponding artificial intelligence chip, and the data shape of the input label segment data is [B*S / N, H];

[0140] Each of the artificial intelligence chips performs reduction calculation on the locally retained and received input label segment data in parallel, and outputs the corresponding full-reduction local calculation result, and the data shape of the full-reduction local calculation result is [B*S / N, H].

[0141] In one of the embodiments, the input label segment data corresponding to the artificial intelligence chip is retained, and the remaining input label segment data is distributed to the corresponding artificial intelligence chip, including:

[0142] For the i-th artificial intelligence chip, the i-th input label segment data is retained, and the j-th input label segment data is distributed to the j-th artificial intelligence chip, i is a positive integer less than or equal to N, j is a positive integer less than or equal to N, and i and j are not equal.

[0143] In one of the embodiments, the hybrid expert module includes a routing operator, a distribution operator, a routing-expert operator, and an aggregation operator, the hybrid expert module performs hybrid expert reasoning calculation on the normalized local results output by each of the artificial intelligence chips to obtain the final hybrid expert reasoning result, including:

[0144] The routing operator performs routing calculation on the normalized local result held by itself on each artificial intelligence chip to obtain target expert index data output by each artificial intelligence chip;

[0145] The distribution operator distributes the input mark feature data in the normalized local result to the corresponding routing-expert operator across chips based on the target expert index data output by each artificial intelligence chip;

[0146] The routing-expert operator deployed on each artificial intelligence chip performs feature transformation calculation on the input mark feature data distributed to itself and outputs corresponding expert local calculation result;

[0147] The aggregation operator deployed on each artificial intelligence chip performs weighted accumulation on different expert local calculation results corresponding to the same input mark feature data on the artificial intelligence chip where the aggregation operator is located to obtain an aggregation intermediate result of the input mark feature data, and synchronizes the aggregation intermediate result to the original artificial intelligence chip to which the input mark feature data belongs, and each artificial intelligence chip integrates the aggregation intermediate result corresponding to the input mark feature data responsible by itself to obtain a mixed expert reasoning result.

[0148] Each module in the optimization device of the mixed expert model can be realized by software, hardware, and a combination thereof, in whole or in part. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0149] In an exemplary embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram of the computer device can be as shown in Figure 9The computer device shown in the figure includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be realized through WIFI, mobile cellular network, near field communication (NFC) or other technologies. The computer program is executed by the processor to realize a hybrid expert model optimization method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0150] Those skilled in the art can understand that, Figure 9 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0151] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to realize the steps in each of the above method embodiments.

[0152] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to realize the steps in each of the above method embodiments.

[0153] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to realize the steps in each of the above method embodiments.

[0154] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0155] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0156] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of these technical features is deemed to be within the scope of the present application.

[0157] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An optimization method using a hybrid expert model, characterized in that, The hybrid expert model includes an attention module, a full reduction operator, a residual connection normalization module, and a hybrid expert module. The method includes: During the computation of the hybrid expert model, the attention module employs a pre-set tensor parallel strategy to compute the current input tensor on multiple AI chips, thereby obtaining the attention local computation results output by each AI chip. The full reduction operator adopts a pre-set sequence parallel strategy to perform full reduction communication on the multiple artificial intelligence chips. During the communication process, the attention local calculation result is segmented based on the input label dimension, and the reduction calculation is performed in parallel by each of the artificial intelligence chips to obtain the full reduction local calculation result output by each of the artificial intelligence chips. The residual connection and normalization module performs residual connection and normalization processing on the fully reduced local calculation results output by each of the artificial intelligence chips on each of the artificial intelligence chips, respectively, to obtain the normalized local results output by the artificial intelligence chips. The hybrid expert module performs hybrid expert inference calculations on the normalized local results output by each of the artificial intelligence chips to obtain the final hybrid expert inference result.

2. The method according to claim 1, characterized in that, The current input tensor is obtained by concatenating different input sequences from different batches. The data shape of the current input tensor is [B*S, H], where B represents the number of batches, S represents the sequence length, and H represents the feature dimension size.

3. The method according to claim 2, characterized in that, The data shape of the attention local computation result is [B*S, H], and the data shape of the full reduction local computation result is [B*S / N, H], where N represents the number of artificial intelligence chips.

4. The method according to claim 3, characterized in that, The process involves performing full reduction communication on the multiple AI chips. During communication, the attention local computation result is segmented based on the input label dimension, and reduction computation is performed in parallel by each AI chip to obtain the full reduction local computation result output by each AI chip, including: Within any AI chip, the attention local calculation results output by each AI chip are uniformly divided based on the input label dimension to obtain N input label fragment data. The input label fragment data corresponding to the AI ​​chip is retained, and the remaining input label fragment data is distributed to the corresponding AI chip. The data shape of the input label fragment data is [B*S / N, H]. Each of the aforementioned artificial intelligence chips performs reduction calculations in parallel on the locally retained and received input labeled fragment data, and outputs the corresponding full reduction local calculation results respectively. The shape of the full reduction local calculation result data is [B*S / N, H].

5. The method according to claim 4, characterized in that, The step of retaining the input tag fragment data corresponding to the artificial intelligence chip and distributing the remaining input tag fragment data to the corresponding artificial intelligence chip includes: For the i-th AI chip, the i-th input tag fragment data is retained, and the j-th input tag fragment data is distributed to the j-th AI chip. i is a positive integer less than or equal to N, j is a positive integer less than or equal to N, and i and j are not equal.

6. The method according to claim 3, characterized in that, The hybrid expert module includes routing operators, distribution operators, routing-expert operators, and aggregation operators. The hybrid expert module performs hybrid expert inference calculations on the normalized local results output by each of the AI ​​chips to obtain the final hybrid expert inference result, including: The routing operator performs routing calculations on the normalized local results it holds on each of the artificial intelligence chips to obtain the target expert index data output by each of the artificial intelligence chips. The distribution operator distributes the input label feature data in the normalized local result across chips to the corresponding routing-expert operator based on the target expert index data output by each of the artificial intelligence chips. The routing-expert operator deployed on each of the aforementioned artificial intelligence chips performs feature transformation calculations on the input labeled feature data distributed to it and outputs the corresponding expert local calculation results; The aggregation operator deployed on each of the aforementioned artificial intelligence chips performs weighted summation on the local calculation results of different experts corresponding to the same input labeled feature data on its own artificial intelligence chip to obtain the aggregated intermediate result of the input labeled feature data, and synchronizes the aggregated intermediate result to the original artificial intelligence chip to which the input labeled feature data belongs. Each of the aforementioned artificial intelligence chips integrates the aggregated intermediate results corresponding to the input labeled feature data it is responsible for to obtain the hybrid expert inference result.

7. An optimization apparatus using a hybrid expert model, characterized in that, The hybrid expert model includes an attention module, a full reduction operator, a residual connection normalization module, and a hybrid expert module. The device includes: An attention calculation module is used to perform calculations on the current input tensor on multiple AI chips during the calculation process of the hybrid expert model, using a preset tensor parallel strategy to obtain the attention local calculation results output by each AI chip. The reduction calculation module is used to perform full reduction communication on the multiple artificial intelligence chips using a preset sequence parallel strategy for the full reduction operator. During the communication process, the attention local calculation result is segmented based on the input label dimension, and the reduction calculation is performed in parallel by each of the artificial intelligence chips to obtain the full reduction local calculation result output by each of the artificial intelligence chips. The normalization calculation module is used by the residual connection normalization module to perform residual connection and normalization processing on the fully reduced local calculation results output by each of the artificial intelligence chips, respectively, to obtain the normalized local results output by the artificial intelligence chips. The inference module is used by the hybrid expert module to perform hybrid expert inference calculations on the normalized local results output by each of the artificial intelligence chips to obtain the final hybrid expert inference result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images