Optimization method, device and computer equipment of hybrid expert system

Abstract

The application relates to an optimization method and device of a hybrid expert system, a computer device and a readable storage medium. The method comprises the following steps: inputting an input sequence into a routing operator, outputting a first output tensor and a second output tensor, the first output tensor representing weight values of each feedforward network corresponding to each input mark in the input sequence, the weight value of the feedforward network in the first output tensor being 0 when the feedforward network is not activated, and the second output tensor representing the total number of activated feedforward networks and the indexes of the activated feedforward networks; inputting the input sequence, the first output tensor and the second output tensor into each feedforward network, outputting a third output tensor from the activated feedforward network, and performing accumulation sum processing on the third output tensor to obtain an output result of the hybrid expert system, which is consistent with the data shape of the input sequence. The method can improve the reasoning efficiency of the hybrid expert system in a reasoning scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a mixed expert system optimization method and device, computer equipment and readable storage medium. BACKGROUND

[0002] In the inference scenario, the mixed expert system (MOE) has been widely applied. The MOE selects different experts (Feed-Forward Network, FFN) to process input data intelligently, and exhibits unique advantages in improving the model expression capability.

[0003] At present, the conventional implementation of the MOE layer is to calculate the weights corresponding to each expert by means of a Router operator. By traversing and calculating each input token, the expert activated by the current token is determined. In this process, since the shape of the data calculated by the expert is different from the shape of the MOE layer before and after, a Permuation operator is used to convert the shape of the MOE layer into the shape required by the FFN; after the FFN calculation is completed, an Unpermutation operator is used to convert the shape of the FFN output back to the shape of the MOE layer, so as to realize the layer-by-layer calculation of the MOE layer.

[0004] However, in the inference scenario, the sequence length (seqlen) is usually short, and the number of tokens that need to be inferred is relatively small, which leads to only a small part of the experts being activated, and most of the experts do not need to participate in the calculation. In the related technology, as shown in FIG. 1, the calculation of the MOE layer covers four operators, namely, the Router operator, the Permuation operator, the FFN, and the Unpermuation operator. Among them, the Permuation and Unpermuation operators have a small read-write instruction granularity, and there is a serious performance bottleneck. This performance defect will significantly reduce the calculation efficiency of the entire MOE layer, and then adversely affect the overall performance of the inference model, thereby reducing the inference efficiency of the MOE in the inference scenario. Figure 1 Figure 1 Figure 1 Figure 1 SUMMARY

[0005] Therefore, it is necessary to provide a mixed expert system optimization method, device, computer equipment and readable storage medium capable of improving the inference efficiency of the mixed expert system in the inference scenario.

[0006] ​​​​In a first aspect, the present application provides an optimization method of a hybrid expert system, the hybrid expert system comprising a routing operator and a plurality of feedforward networks, comprising:

[0007] inputting an input sequence into the routing operator, outputting a first output tensor and a second output tensor, the first output tensor representing weight values of each of the feedforward networks corresponding to each input token in the input sequence, and the weight value of the feedforward network in the first output tensor being 0 when the feedforward network is not activated, and the second output tensor representing a total number of activated feedforward networks and indexes of the activated feedforward networks;

[0008] inputting the input sequence, the first output tensor and the second output tensor into each of the feedforward networks, outputting a third output tensor by the activated feedforward networks, and performing accumulation sum processing on the third output tensor to obtain an output result of the hybrid expert system, the output result being consistent with a data shape of the input sequence.

[0009] In an embodiment, a data shape of the first output tensor is [E, NS], where E represents a total number of feedforward networks, N represents a batch, and S represents a sequence length, and a data shape of the second output tensor is [1, 1+E], and a first address of the second output tensor is used to represent the total number of activated feedforward networks, and the remaining addresses are used to represent indexes of the activated feedforward networks.

[0010] In an embodiment, the routing operator comprises a matrix multiplication accumulation (MMA) operator and a top-k (Topk) operator, and a process of outputting the first output tensor and the second output tensor by the routing operator performing processing on the input sequence comprises:

[0011] inputting the input sequence into the MMA operator to perform matrix multiplication accumulation operation, and outputting a fourth output tensor, the fourth output tensor being a matrix multiplication accumulation result of the input sequence and a weight matrix corresponding to the MMA operator, a data shape of the input sequence being [1, NS, K], a data shape of the weight matrix being [K, E], and a data shape of the fourth output tensor being [1, NS, E], where K represents a feature dimension;

[0012] inputting the fourth output tensor into the Topk operator to output the first output tensor and the second output tensor.

[0013] In an embodiment, the inputting the input sequence, the first output tensor and the second output tensor into each of the feedforward networks, and outputting the third output tensor by the activated feedforward networks comprises:

[0014] Each of the feedforward networks determines whether it is an activated feedforward network based on the second output tensor, and when determining that it is an activated feedforward network, processes the input sequence based on the first output tensor to output a third output tensor.

[0015] In one of the embodiments, the processing of the input sequence based on the first output tensor to output a third output tensor comprises:

[0016] The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor;

[0017] The dot product result of the fifth output tensor and the first output tensor is output as a third output tensor, and the fifth output tensor and the third output tensor are consistent with the data shape of the input sequence.

[0018] In one of the embodiments, the feedforward network comprises a first MMA operator and an activation function, and a second MMA operator, and the activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor, which comprises:

[0019] The input sequence is input into the first MMA operator to perform matrix multiplication accumulation operation, and the operation result is input into the activation function to obtain a sixth output tensor, the data shape of the input sequence is [1, NS, K], wherein K represents the data dimension, the data shape of the weight matrix of the first MMA operator is [K, W], and the data shape of the sixth output tensor is [1, NS, W];

[0020] The sixth output tensor is input into the second MMA operator to perform matrix multiplication accumulation operation to obtain a fifth output tensor, and the data shape of the weight matrix of the second MMA operator is [W, K].

[0021] In a second aspect, the application further provides an optimization device of a hybrid expert system, the hybrid expert system comprising a routing operator and a plurality of feedforward networks, the device comprising:

[0022] A first calculation module is configured to input an input sequence into the routing operator to output a first output tensor and a second output tensor, the first output tensor representing the weight value of each of the feedforward networks corresponding to each input label in the input sequence, and the weight value of the feedforward network in the first output tensor being 0 when the feedforward network is not activated, and the second output tensor representing the total number of activated feedforward networks and the index of each of the activated feedforward networks;

[0023] The second calculation module is configured to input the input sequence, the first output tensor and the second output tensor into each of the feedforward networks, output a third output tensor from the activated feedforward network, and perform accumulation sum processing on the third output tensor to obtain an output result of the hybrid expert system, which is consistent with the data shape of the input sequence.

[0024] In one of the embodiments, the data shape of the first output tensor is [E, NS], where E represents the total number of feedforward networks, N represents a batch, and S represents a sequence length, and the data shape of the second output tensor is [1, 1+E], where the first address of the second output tensor is used to represent the total number of activated feedforward networks, and the remaining addresses are used to represent the indexes of the activated feedforward networks.

[0025] In one of the embodiments, the routing operator includes a matrix multiplication accumulation (MMA) operator and a top-k (Topk) operator, and the process of processing the input sequence to output the first output tensor and the second output tensor includes:

[0026] The input sequence is input into the MMA operator to perform matrix multiplication accumulation operation, and a fourth output tensor is output, which is the matrix multiplication accumulation result of the input sequence and a weight matrix corresponding to the MMA operator, the data shape of the input sequence is [1, NS, K], the data shape of the weight matrix is [K, E], and the data shape of the fourth output tensor is [1, NS, E], where K represents a feature dimension;

[0027] The fourth output tensor is input into the Topk operator to output the first output tensor and the second output tensor.

[0028] In one of the embodiments, the process of inputting the input sequence, the first output tensor and the second output tensor into each of the feedforward networks and outputting a third output tensor from the activated feedforward network includes:

[0029] Each of the feedforward networks determines whether it is an activated feedforward network based on the second output tensor, and when it is determined that it is an activated feedforward network, processes the input sequence based on the first output tensor to output a third output tensor.

[0030] In one of the embodiments, the process of processing the input sequence based on the first output tensor to output a third output tensor includes:

[0031] The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor.

[0032] The dot product result of the fifth output tensor and the first output tensor is output as a third output tensor, and the fifth output tensor and the third output tensor are consistent with the data shape of the input sequence.

[0033] In one of the embodiments, the feedforward network comprises a first MMA operator and an activation function, and a second MMA operator, the activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor, comprising:

[0034] The input sequence is input into the first MMA operator to perform matrix multiplication accumulation operation, and the operation result is input into the activation function to obtain a sixth output tensor, the data shape of the input sequence is [1, NS, K], wherein K represents the data dimension, the data shape of the weight matrix of the first MMA operator is [K, W], and the data shape of the sixth output tensor is [1, NS, W];

[0035] The sixth output tensor is input into the second MMA operator to perform matrix multiplication accumulation operation, and the operation result is output as the fifth output tensor, and the data shape of the weight matrix of the second MMA operator is [W, K].

[0036] In a third aspect, the application further provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor realizes the optimization method of the hybrid expert system of any one of the above when executing the computer program.

[0037] In a fourth aspect, the application further provides a computer readable storage medium, which stores a computer program, and the computer program realizes the optimization method of the hybrid expert system of any one of the above when executed by a processor.

[0038] In a fifth aspect, the application further provides a computer program product, comprising a computer program, and the computer program realizes the optimization method of the hybrid expert system of any one of the above when executed by a processor.

[0039] The above-mentioned hybrid expert system optimization method, apparatus, computer device, and readable storage medium input an input sequence into the hybrid expert system's routing operator, outputting a first output tensor and a second output tensor. The first output tensor represents the weight value of each feedforward network corresponding to each input tag in the input sequence. When the feedforward network is not activated, the weight value of the feedforward network in the first output tensor is 0. The second output tensor represents the total number of activated feedforward networks and the index of each activated feedforward network. The input sequence, the first output tensor, and the second output tensor are input into each feedforward network. The activated feedforward network outputs a third output tensor. The third output tensor is accumulated and summed to obtain the output result of the hybrid expert system, which is consistent with the data shape of the input sequence. The hybrid expert system optimization method, apparatus, computer device, and readable storage medium provided in the embodiments of the present application change the output of the routing operator in the hybrid expert system so that the subsequent feedforward network layer can directly obtain the activation status corresponding to the global input token. By combining the total number of activated feedforward networks output and the index of the activated feedforward network, it is possible to calculate and output only the activated feedforward network, and obtain a calculation result consistent with the shape of the input sequence data. In other words, the hybrid expert system provided in the embodiments of the present application eliminates the Permuation and Unpermuation operators, thereby avoiding the large amount of inefficient data transposition and handling caused by the Permuation and Unpermuation operators, which can significantly reduce the consumption of inefficient computing power, improve the computational efficiency of the hybrid expert system, and thus improve the reasoning efficiency of the hybrid expert system in reasoning scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 is a schematic diagram of a hybrid expert system in related art in one embodiment;

[0042] Figure 2 A schematic flow chart of an optimization method of a hybrid expert system in one embodiment;

[0043] Figure 3 Schematic diagram of the flow of routing operator operation in one embodiment;

[0044] Figure 4 1 is a schematic diagram of a flow chart of a routing operator performing operations based on a first output tensor in one embodiment;

[0045] Figure 5 Flow chart for the operation of a feedforward network in one embodiment;

[0046] Figure 6 Flow chart for the operation of a conventional hybrid expert system in another embodiment;

[0047] Figure 7 Flow chart for the operation of a routing operator in one embodiment;

[0048] Figure 8 Flow chart for the operation of a hybrid expert system in one embodiment;

[0049] Figure 9 Block diagram of an optimization device for a hybrid expert system in one embodiment;

[0050] Figure 10 Internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0052] It should be noted that the terms "first", "second" and the like used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application means two or more. The term "and / or" used in the present application means one of the options or any combination of a plurality of options.

[0053] In one exemplary embodiment, as shown in Figure 2 An optimization method of a hybrid expert system is provided, which is applied to a host side for illustration, and it can be understood that the host side can include a CPU (Central Processing Unit). The hybrid expert system includes a routing operator and a plurality of feedforward networks, and the method can include the following steps 202 to 204. Wherein:

[0054] In step 202, the input sequence is input into the routing operator, and a first output tensor and a second output tensor are output. The first output tensor represents the weight value of each feedforward network corresponding to each input tag in the input sequence. When the feedforward network is not activated, the weight value of the feedforward network in the first output tensor is 0. The second output tensor represents the total number of activated feedforward networks and the index of each activated feedforward network.

[0055] The optimization method of the hybrid expert system provided in the embodiment of the present application can be applied to artificial intelligence fields such as natural language processing and image recognition to improve the efficiency and accuracy of the model in processing complex tasks. The following embodiments will be explained by taking text classification in natural language processing tasks as an example. Assume that the input sequence is a text to be classified, and the text is divided into multiple input tags (which can also be expressed as tokens or input elements, etc.), which constitute an input sequence of length S [x1, x2, ..., x S ].

[0056] An input sequence can be fed into a routing operator. Based on pre-trained parameters, the routing operator analyzes each token in the input sequence and determines which of multiple feedforward networks is suitable for processing the token. For example, the routing operator can determine the activated feedforward network for each token by calculating the matching degree between each token and each feedforward network. The output of the routing operator includes a first output tensor and a second output tensor.

[0057] In an exemplary embodiment, the data shape of the first output tensor is [E, NS], where E represents the total number of feedforward networks, N represents the batch, and S represents the sequence length. The data shape of the second output tensor is [1, 1+E], and the first address of the second output tensor is used to represent the total number of activated feedforward networks, and the remaining addresses are used to represent the indexes of the activated feedforward networks.

[0058] That is, each element A in the first output tensor A ij Represents the activation state of the i-th feedforward network for the j-th token. When it is in the activated state for the j-th token, the A ij is the corresponding weight value. If it is in an unactivated state for the jth token, then A ij is 0.

[0059] The second output tensor B contains two parts of information, one is the total number of activated feedforward networks Y, and the other is the index of each activated feedforward network [I1, I2, ..., I Y ], where the first address B0 of the second output tensor is the total number Y of feedforward networks activated by each token of the input sequence in the E feedforward networks, and the rest are B1, B2, ..., B E+1, I K ], I i represents the index of the i-th activated feedforward network in all feedforward networks, for example, there are a total of 8 feedforward networks FFN, and the index of the feedforward network activated by input is 1, 3, 5, then the second output tensor is [3, 1, 3, 5, 0, 0, 0, 0, 0].

[0060] In step 204, the input sequence, the first output tensor and the second output tensor are input into each feedforward network, the third output tensor is output by the activated feedforward network, and the third output tensor is accumulated and summed to obtain the output result of the hybrid expert system, and the output result is consistent with the data shape of the input sequence.

[0061] In the embodiment of the application, the input sequence, the first output tensor and the second output tensor can be input into each feedforward network. Only the activated feedforward network (i.e. the feedforward network in [I1, I2,..., I Y ] ) can process the input sequence and output the third output tensor. The shape of the third output tensor output by each activated feedforward network is consistent with the data shape of the input sequence. All third output tensors are accumulated and summed, i.e. the elements at each position are added, to obtain the output result of the hybrid expert system, and the shape of the output result is also consistent with the data shape of the input sequence in the dimension, which can be used for subsequent text classification tasks, for example: the output result corresponding to each token is processed by a further classifier to determine the category to which the text belongs.

[0062] The optimization method of the mixed expert system described above inputs the input sequence into the routing operator of the mixed expert system, and outputs a first output tensor and a second output tensor, where the first output tensor represents, for each token in the input sequence, an activated feedforward network in the plurality of feedforward networks, the value of the activated feedforward network in the first output tensor is the corresponding weight, and the value of the unactivated feedforward network is 0, and the second output tensor represents the total number of activated feedforward networks and the index of each activated feedforward network. The input sequence, the first output tensor and the second output tensor are input into each of the feedforward networks, and a third output tensor is output by the activated feedforward network. The third output tensor is subjected to an accumulated sum processing, and an output result of the mixed expert system can be obtained, which is consistent with the data shape of the input sequence. The optimization method of the mixed expert system provided in the embodiment of the present application changes the output of the routing operator in the mixed expert system, so that the subsequent feedforward network layer can directly obtain the activation condition corresponding to the global input token. In combination with the total number of activated feedforward networks and the index of the activated feedforward network, only the activated feedforward network can be calculated and output, and an operation result consistent with the data shape of the input sequence can be obtained. That is, the mixed expert system provided in the embodiment of the present application eliminates the permutation and unpermutation operators, thereby avoiding a large amount of inefficient data transposition and transportation caused by the permutation and unpermutation operators, greatly reducing the consumption of invalid computing power, improving the computing efficiency of the mixed expert system, and further improving the inference efficiency of the mixed expert system in the inference scenario.

[0063] In an exemplary embodiment, the routing operator includes a matrix multiplication accumulation (MMA) operator and a top-k (Topk) operator. As shown in FIG. 3, the process of processing the input sequence by the routing operator to output the first output tensor and the second output tensor can include steps 302 to 304, where: Figure 3

[0064] In step 302, the input sequence is input into the MMA operator to perform matrix multiplication, and a fourth output tensor is output. The fourth output tensor is the matrix multiplication accumulation result of the input sequence and the weight matrix corresponding to the MMA operator. The data shape of the input sequence is [1, NS, K], the data shape of the weight matrix is [K, E], and the data shape of the fourth output tensor is [1, NS, E], where K represents a feature dimension.

[0065] In step 304, the fourth output tensor is input into the Topk operator, and the first output tensor and the second output tensor are obtained.

[0066] ​In the embodiments of the present application, the routing operator includes a matrix multiplication accumulation (MMA) operator and a top-k (Topk) operator, which processes the input sequence and outputs a first output tensor and a second output tensor in the following manner:

[0067] For example, assume that the shape of the input sequence is [1, NS, K], where N is a batch, S is a sequence length, NS represents folding the batch and the sequence length into one axis, and K represents a feature dimension. Meanwhile, the MMA operator has a pre-stored weight matrix with a dimension of [K, E]. The input sequence and the weight matrix are subjected to matrix multiplication accumulation operation in the MMA operator to obtain and output a fourth output tensor with a shape of [1, NS, E]. Each element P ij in the fourth output tensor represents a matching score of an i-th token in the input sequence and a j-th feedforward network, which reflects the suitability of the j-th feedforward network for processing the i-th token.

[0068] Further, the fourth output tensor is input into the Topk operator. The Topk operator is used to select k values with the highest scores (k is a pre-set threshold of the number of activated feedforward networks) from each row (i.e., all matching scores of a token corresponding to the feedforward networks) of the fourth output tensor and record the indexes corresponding to the values. After processing by the Topk operator, the first output tensor and the second output tensor are obtained. The shape of the first output tensor is [E, NS], and for the k maximum values selected from each row of the fourth output tensor, the corresponding weight values (which can be further converted according to the matching scores, for example, normalized) are filled in the corresponding positions of the first output tensor, and the remaining positions are filled with 0, so as to represent the tokens in the input sequence that activate the respective feedforward networks and the corresponding weights; the shape of the second output tensor is [1, 1+E], and the total number Y of the activated feedforward networks is filled in the first address, and the indexes of the respective activated feedforward networks are sequentially filled in the following positions, and the remaining positions are filled with 0.

[0069] The optimization method of the hybrid expert system provided in the embodiments of the present application can make the subsequent feedforward network layer directly obtain the activation condition of the global token by changing the output of the Topk operator in the routing operator, and can realize the calculation and output of only the activated feedforward network by combining the total number of the activated feedforward network and the index of the activated feedforward network, so as to obtain the operation result consistent with the shape of the input sequence data. That is, the Permuation and Unpermuation operators in the hybrid expert system can be eliminated, and thus a large amount of inefficient data transposition and transportation caused by the Permuation and Unpermuation operators can be avoided, the invalid computing power consumption can be greatly reduced, the computing efficiency of the hybrid expert system is improved, and thus the reasoning efficiency of the hybrid expert system in the reasoning scenario is improved.

[0070] In an exemplary embodiment, inputting the input sequence, the first output tensor and the second output tensor into each feedforward network, and outputting the third output tensor by the activated feedforward network can include the following steps:

[0071] Each feedforward network determines whether itself is the activated feedforward network based on the second output tensor, and processes the input sequence based on the first output tensor to output the third output tensor when it is determined that itself is the activated feedforward network.

[0072] In the embodiments of the present application, after the processing of the routing operator is completed, the first output tensor (with a shape of [E, NS]) and the second output tensor (with a shape of [1, 1+E]) can be obtained, and these data and the input sequence are input into the E feedforward networks.

[0073] Each feedforward network determines whether itself is the activated feedforward network based on the second output tensor. Exemplarily, for the jth feedforward network, the total number Y of the activated feedforward networks is first read from the first address of the second output tensor. Subsequently, the index values of the activated feedforward networks are sequentially read from the start position of the index information stored in the second output tensor. In the traversal process, the read index value is compared with the index j of itself: if the read index value is found to be equal to j in the traversal process, the jth feedforward network is immediately determined to be in the activated state, and the traversal is stopped; if Y index values are read continuously until the end of the traversal, and an index value equal to j has never appeared, the jth feedforward network is determined to be not activated, and the jth feedforward network does not participate in the subsequent processing.

[0074] When it is determined that itself is the activated feedforward network, the feedforward network processes the input sequence based on the first output tensor. The element A ij represents the weight of the ith feedforward network activated by the jth token in the input sequence. The feedforward network processes the input sequence according to A ij, the corresponding token in the input sequence is weighted, and the weighted token is subjected to feature extraction, transformation, etc. combined with the network structure and parameters of the activated feedforward network, and finally the third output tensor is output. For example, for the token x3 in the input sequence, if the second feedforward network is activated, and the corresponding element A 23 of the first output tensor is 0.8, the second feedforward network processes x3 with a weight of 0.8, and after internal multi-layer neuron calculation, outputs feature information related to token x3, which constitutes part of the third output tensor.

[0075] Each activated feedforward network processes the input sequence in the above manner and outputs the third output tensor. The shapes of the third output tensors output by all activated feedforward networks are consistent with the data shape of the input sequence. All third output tensors are subsequently accumulated and summed to obtain the output result of the hybrid expert system.

[0076] The optimization method of the hybrid expert system provided by the embodiments of the present application can directly obtain the activation status of the global token, combine the total number of activated feedforward networks and the index of the activated feedforward network, and implement calculation and output only for the activated feedforward network without a large amount of inefficient data transposition and handling, thereby greatly reducing the consumption of invalid computing power, improving the calculation efficiency of the hybrid expert system, and further improving the inference efficiency of the hybrid expert system in the inference scenario.

[0077] In an exemplary embodiment, referring to Figure 4 , processing the input sequence based on the first output tensor to output a third output tensor can include the following steps 402 to 404, wherein:

[0078] Step 402, the activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor;

[0079] Step 404, the dot product result of the fifth output tensor and the first output tensor is output as the third output tensor. The fifth output tensor and the third output tensor are consistent with the data shape of the input sequence.

[0080] In the embodiments of the present application, the activated feedforward network performs weighted processing on the input sequence to obtain and output a fifth output tensor. For example, the activated feedforward network performs weighted calculation on each token in the input sequence according to its own network parameters. Taking token x i in the input sequence as an example, the activated feedforward network performs weighted summation on each feature dimension of x i according to the pre-trained weight matrix to obtain a weighted vector representation, which contains the feature information of xi After the above weighting processing is performed on all tokens in the input sequence, a fifth output tensor with the same dimension as the input sequence is obtained, and each element in the fifth output tensor corresponds to the feature value of the token in the input sequence after the weighting processing.

[0081] Assuming that the shape of the input sequence tensor is [1, NS, K], the fifth output tensor with the shape of [1, NS, K] can be obtained.

[0082] Since the shapes of the fifth output tensor and the first output tensor are different, direct dot multiplication may cause dimension mismatch, and therefore the broadcasting mechanism in the K dimension can be used. When broadcasting in the K dimension, the system automatically expands or aligns the tensor dimensions, so that the two tensors meet the dot multiplication operation requirements in the K dimension. For example, for the feature vector (with the shape of [1, 1, K]) corresponding to each token in the fifth output tensor and the vector (with the shape of [E, 1]) in the corresponding position of the first output tensor, [E, 1] is expanded to [E, K] in the K dimension through broadcasting, so that the two can be multiplied by corresponding elements. For example, if the feature vector corresponding to the i-th token in the fifth output tensor is [f i1 , f i2 ,..., f iK ] (here, the vector corresponds to the 1st instance, the i-th NS position, and the K-dimensional feature in the [1, NS, K] tensor), and the vector in the corresponding position of the first output tensor is [a i1 , a i2 ,..., a iE ], after K-dimensional broadcasting, f i1 is multiplied by the corresponding element of the expanded [a i1 , a i2 ,..., a iE ], f i2 is multiplied by the corresponding element of the expanded [a i1 , a i2 ,..., a iE ], and so on, the multiplication operation of all K-dimensional elements is completed, and then the sum of the results is obtained to obtain a scalar value.

[0083] After the above dot multiplication operation is performed on all NS tokens in the input sequence, a third output tensor with the same data shape as the input sequence and the fifth output tensor is obtained, and the shape of the third output tensor is also [1, NS, K]. That is, through the foregoing process, the activated feedforward network processes the input sequence based on the first output tensor and outputs a third output tensor with the same data shape as the input sequence. The output result of the MOE can be obtained by accumulating the third output tensors output by the activated experts, and the data shape of the output result is the same as that of the input sequence, both being [1, NS, K].

[0084] In the embodiments of the application, a small amount of calculation of the feedforward network can eliminate the transpose transfer operation of Permuation and Unpermuation, and can greatly improve the calculation efficiency of the hybrid expert system.

[0085] In an exemplary embodiment, referring to Figure 5 The feedforward network includes a first MMA operator and an activation function, and a second MMA operator, and the activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor, including the following steps 502 to 504.

[0086] In step 502, the input sequence is input into the first MMA operator to perform matrix multiplication accumulation operation, and the operation result is input into the activation function to obtain a sixth output tensor. The data shape of the input sequence is [1, NS, K], where K represents the data dimension. The data shape of the weight matrix of the first MMA operator is [K, W], and the data shape of the sixth output tensor is [1, NS, W].

[0087] In step 504, the sixth output tensor is input into the second MMA operator to perform matrix multiplication accumulation operation, and the fifth output tensor is output. The data shape of the weight matrix of the second MMA operator is [W, K].

[0088] In the embodiments of the application, it is assumed that the data shape of the input sequence is [1, NS, K], and the data shape of the weight matrix of the first MMA operator is [K, W], where K is consistent with the third dimension of the input sequence, and W represents a new dimension of data after the first MMA operator processing, which is an intermediate dimension. When the input sequence and the weight matrix of the first MMA operator perform matrix multiplication operation, for the feature vector (shape [1, 1, K]) corresponding to each token in the input sequence, the multiplication operation is performed with the weight matrix [K, W]. According to the matrix multiplication rule, the K-dimensional vector is multiplied with the [K, W] matrix to obtain a W-dimensional vector. After performing the above multiplication operation on all NS tokens in the input sequence, an intermediate result with a shape of [1, NS, W] is obtained.

[0089] The intermediate result is input into the activation function, and the activation function performs nonlinear transformation on each element, for example, using Sigmoid function, ReLU function, etc., to map each element in the intermediate result to a specific interval, and finally obtain a sixth output tensor with a shape of [1, NS, W].

[0090] The data shape of the weight matrix of the second MMA operator is [W, K], where W is consistent with the third dimension of the sixth output tensor, and K represents that the data is restored to the same dimension as the input sequence after being processed by the second MMA operator. When the sixth output tensor and the weight matrix of the second MMA operator perform matrix multiplication accumulation operation, for the feature vector (shape [1, 1, W]) corresponding to each token in the sixth output tensor, multiplication operation is performed with the weight matrix [W, K]. According to the matrix multiplication rule, the W-dimensional vector is multiplied with the [W, K] matrix to obtain a K-dimensional vector. After performing the above multiplication operation on all NS tokens in the sixth output tensor, a fifth output tensor with a shape of [1, NS, K] is obtained, and the data shape of the fifth output tensor is consistent with the input sequence.

[0091] In the embodiments of the present application, by increasing a small amount of calculation of the feedforward network, the transpose transfer operation of Permuation and Unpermuation can be eliminated, and the calculation efficiency of the hybrid expert system can be greatly improved.

[0092] In order to enable those skilled in the art to better understand the embodiments of the present application, the embodiments of the present application are described below through specific examples.

[0093] Referring to Figure 6 As shown in the figure, in the traditional MOE system, a Router operator is used to calculate the weights of different experts, i.e., the weights of each feedforward network, in each MOE layer. The weights of the selected experts are calculated by traversing each input token to select the expert activated by the current token from many experts. In this process, since the shape calculated by each expert and the shape of the MOE layer before and after is different, a Permuation operator is used to convert the shape of the MOE layer to the shape required by the FFN, and after the FFN (expert calculation), an Unpermutation operator is used to convert the shape of the FFN output back to the shape of the MOE layer to calculate layer by layer.

[0094] In the inference scenario, the sequence is usually not too long, and the number of tokens that need to be inferred is relatively small at this time, which means that only a small part of experts are activated and most of them do not need to be calculated. In the traditional scheme, the calculation of the MOE layer includes four operators of Router, Permuation, FFN and Unpermutation. The performance of the Permuation and Unpermutation operators is poor due to small read-write instruction granularity, which will lead to low performance of the entire MOE layer, thereby affecting the performance of the inference model.

[0095] Referring to Figure 7As shown, the optimization method of the mixed expert system provided in the embodiments of the present application changes the output of the Router operator into a first output tensor (which can also be represented as mask out) and a second output tensor (which can also be represented as hitted expert). The first output tensor is the weight of all tokens and corresponding experts globally, and the second output tensor represents which experts are activated in total for all tokens globally (including the total number of activated experts). By adding the above two inputs to the input input of the Router, each FFN after that can be calculated by selecting the activated expert number to calculate the global tensor, and the weight in the first output tensor is used for accumulated output to obtain the output result of the final MOE layer.

[0096] Referring to Figure 8 As shown, the Router operator is composed of MMA and topK, the input of the MMA is an input sequence with a shape of [1, NS, K] and a weight matrix with a shape of [K, E], after matrix multiplication of the two, a tensor with a shape of [1, NS, E] is output, which is used as the input of the TopK, and the output obtained is a first output tensor with a shape of [E, NS] and a second output tensor with a shape of [1, 1+E], wherein [E, NS] represents which experts are activated among all NS tokens and all experts, and the value of the expert not activated is 0, and the corresponding weight is the value of the activated expert. And [1, 1+E] represents which experts are activated in total among all NS tokens, wherein the first address hitted_expert[0] records the total number of activated experts, and the remaining content represents the index number of the activated expert. Wherein, N is the batch, S is the sequence length (seqlen) n, NS represents that the batch and seqlen can be folded into one axis, K represents the feature dimension (hidden), and E represents the total number of experts.

[0097] From the MOE layer, the shape of the input and output is [1, NS, K], and the FFN is the activated expert in the E dimension, and the activated expert in the second output tensor respectively does the first MMA and swinGLU (Shifted Window based GLU, Shifted Window based GLU) in the FFN according to its own weight for the same input, and the shape of the output tensor is [1, NS, W], and the tensor is right multiplied by the weight of the second MMA, and the result is [1, NS, K], which is respectively multiplied with the first output tensor (and broadcast in the K dimension). The result is the final output after the topK expert is activated and weighted for each token.

[0098] The optimization method of the mixed expert system provided in the embodiments of the present application can improve the performance of the MOE layer by increasing a small amount of calculation and tensor output of the Router operator and appropriately increasing the calculation of the FFN in the decode scenario of reasoning, saving a large number of inefficient Permuation / Unpermuation operations, without reducing the model accuracy and with good generalization, and is applicable to any network including the MOE layer.

[0099] It should be understood that, although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0100] Based on the same inventive concept, the embodiments of the present application also provide an optimization device of a mixed expert system for implementing the optimization method of the mixed expert system as described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, and therefore the specific limitations in one or more optimization device embodiments of the mixed expert system provided below can refer to the limitations of the optimization method of the mixed expert system described above, which will not be repeated here.

[0101] In one exemplary embodiment, as shown in Figure 9 An optimization device 900 of a mixed expert system is provided, the mixed expert system comprising a routing operator and a plurality of feedforward networks, the device comprising: a first calculation module 902, a second calculation module 904, wherein:

[0102] The first calculation module 902 is configured to input an input sequence into the routing operator, and output a first output tensor and a second output tensor, the first output tensor representing a weight value of each of the feedforward networks corresponding to each input token in the input sequence, and the weight value of the feedforward network in the first output tensor being 0 when the feedforward network is not activated, and the second output tensor representing a total number of the activated feedforward networks and an index of each of the activated feedforward networks.

[0103] The second calculation module 904 is configured to input the input sequence, the first output tensor and the second output tensor into each of the feedforward networks, output a third output tensor from the activated feedforward network, and perform accumulation sum processing on the third output tensor to obtain an output result of the hybrid expert system, which is consistent with the data shape of the input sequence.

[0104] The optimization device of the hybrid expert system is configured to input an input sequence into a routing operator of the hybrid expert system, and output a first output tensor and a second output tensor, where the first output tensor represents, for each token in the input sequence, an activated feedforward network in a plurality of feedforward networks, the value of the activated feedforward network in the first output tensor is a corresponding weight, and the value of a non-activated feedforward network is 0, and the second output tensor represents the total number of activated feedforward networks and the index of each activated feedforward network. The input sequence, the first output tensor and the second output tensor are input into each of the feedforward networks, a third output tensor is output from the activated feedforward network, and accumulation sum processing is performed on the third output tensor, so that an output result of the hybrid expert system can be obtained, which is consistent with the data shape of the input sequence. The optimization device of the hybrid expert system provided in the embodiment of the present application changes the output of the routing operator in the hybrid expert system, so that the subsequent feedforward network layer can directly obtain the activation condition corresponding to the global input token, and the total number of activated feedforward networks and the index of the activated feedforward network are combined, so that only the activated feedforward network is calculated and output, and an operation result consistent with the data shape of the input sequence is obtained. That is, the hybrid expert system provided in the embodiment of the present application eliminates the permutation and unpermutation operators, thereby avoiding a large amount of inefficient data transposition and transportation caused by the permutation and unpermutation operators, greatly reducing the consumption of invalid computing power, improving the calculation efficiency of the hybrid expert system, and further improving the inference efficiency of the hybrid expert system in the inference scenario.

[0105] In one embodiment, the data shape of the first output tensor is [E, NS], where E represents the total number of feedforward networks, N represents a batch, and S represents a sequence length, and the data shape of the second output tensor is [1, 1+E], where the first address of the second output tensor is used to represent the total number of activated feedforward networks, and the remaining addresses are used to represent the index of the activated feedforward network.

[0106] In one embodiment, the routing operator includes a matrix multiplication accumulation (MMA) operator and a top-k (Topk) operator. The routing operator processes the input sequence, and the process of outputting the first output tensor and the second output tensor includes:

[0107] inputting the input sequence into the MMA operator to perform a matrix multiplication operation, and outputting a fourth output tensor, the fourth output tensor being a matrix multiplication accumulation result of the input sequence and a weight matrix corresponding to the MMA operator, a data shape of the input sequence being [1, NS, K], a data shape of the weight matrix being [K, E], and a data shape of the fourth output tensor being [1, NS, E], wherein K represents a feature dimension;

[0108] inputting the fourth output tensor into the Topk operator to obtain the first output tensor and the second output tensor.

[0109] In one of the embodiments, the inputting the input sequence, the first output tensor and the second output tensor into each of the feedforward networks, and outputting a third output tensor by the activated feedforward network, comprises:

[0110] Each of the feedforward networks determines whether itself is an activated feedforward network based on the second output tensor, and when determining that itself is an activated feedforward network, processes the input sequence based on the first output tensor and outputs a third output tensor.

[0111] In one of the embodiments, the processing the input sequence based on the first output tensor and outputting a third output tensor comprises:

[0112] The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor;

[0113] The dot product result of the fifth output tensor and the first output tensor is output as the third output tensor, and the fifth output tensor and the third output tensor are consistent with the data shape of the input sequence.

[0114] In one of the embodiments, the feedforward network comprises a first MMA operator and an activation function, and a second MMA operator, and the activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor, comprising:

[0115] inputting the input sequence into the first MMA operator to perform a matrix multiplication accumulation operation, and inputting the operation result into the activation function to obtain a sixth output tensor, a data shape of the input sequence being [1, NS, K], wherein K represents a data dimension, a data shape of a weight matrix of the first MMA operator being [K, W], and a data shape of the sixth output tensor being [1, NS, W];

[0116] The sixth output tensor is input into the second MMA operator for matrix multiplication accumulation operation, and a fifth output tensor is output, and a data shape of a weight matrix of the second MMA operator is [W, K].

[0117] The modules in the optimization apparatus of the hybrid expert system can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the modules.

[0118] In an exemplary embodiment, a computer device, which can be a terminal, has an internal structure as shown in Figure 10 The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to perform wired or wireless communication with external terminals. The wireless communication can be achieved through WIFI, mobile cellular network, near field communication (NFC), or other technologies. The computer program is executed by the processor to implement a hybrid expert system optimization method. The display unit of the computer device is configured to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0119] Those skilled in the art can understand that Figure 10 The structure shown in the above description is only a block diagram of part of the structure related to the scheme of the present application, and does not limit the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] In an embodiment, a computer device is also provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0121] In an embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0122] In an embodiment, a computer program product is provided, comprising a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0125] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0126] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A hybrid expert system optimization method, characterized in that: The hybrid expert system includes a routing operator and a plurality of feedforward networks, and the method includes: Inputting an input sequence into the routing operator, and outputting a first output tensor and a second output tensor, wherein the first output tensor represents the weight value of each feedforward network corresponding to each input tag in the input sequence. When the feedforward network is not activated, the weight value of the feedforward network in the first output tensor is 0, and the second output tensor represents the total number of activated feedforward networks and the index of each activated feedforward network; The input sequence, the first output tensor, and the second output tensor are input into each of the feedforward networks, the activated feedforward network outputs a third output tensor, and the third output tensor is accumulated and summed to obtain an output result of the hybrid expert system, where the output result is consistent with the data shape of the input sequence.

2. The method according to claim 1, characterized in that The data shape of the first output tensor is [E, NS], where E represents the total number of feedforward networks, N represents the batch, and S represents the sequence length. The data shape of the second output tensor is [1, 1+E]. The first address of the second output tensor is used to represent the total number of activated feedforward networks, and the remaining addresses are used to represent the indexes of the activated feedforward networks.

3. The method according to claim 1 or 2, characterized in that The routing operator includes a matrix multiplication and accumulation (MMA) operator and a top k value (Topk) operator. The routing operator processes the input sequence and outputs the first output tensor and the second output tensor, including: Input the input sequence into the MMA operator for matrix multiplication and accumulation, and output a fourth output tensor, where the fourth output tensor is the matrix multiplication and accumulation result of the input sequence and the weight matrix corresponding to the MMA operator, wherein the data shape of the input sequence is [1, NS, K], the data shape of the weight matrix is ​​[K, E], and the data shape of the fourth output tensor is [1, NS, E], where K represents the feature dimension, E represents the total number of feedforward networks, N represents the batch, and S represents the sequence length; The fourth output tensor is input into the Topk operator, and the first output tensor and the second output tensor are output.

4. The method according to claim 1 or 2, characterized in that Inputting the input sequence, the first output tensor, and the second output tensor into each of the feedforward networks, and outputting a third output tensor from the activated feedforward network, comprises: Each of the feedforward networks determines whether it is an activated feedforward network based on the second output tensor, and when it is determined that it is an activated feedforward network, processes the input sequence based on the first output tensor and outputs a third output tensor.

5. The method according to claim 4, characterized in that The processing of the input sequence based on the first output tensor to output a third output tensor includes: The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor; The dot product result of the fifth output tensor and the first output tensor is used as the third output tensor, and the fifth output tensor and the third output tensor are both consistent with the data shape of the input sequence.

6. The method according to claim 5, characterized in that The feedforward network includes a first MMA operator and an activation function, and a second MMA operator. The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor, including: Inputting the input sequence into the first MMA operator for matrix multiplication and accumulation operation, and inputting the operation result into the activation function to obtain a sixth output tensor, wherein the data shape of the input sequence is [1, NS, K], where K represents the feature dimension, N represents the batch, and S represents the sequence length, the data shape of the weight matrix of the first MMA operator is [K, W], and the data shape of the sixth output tensor is [1, NS, W], where W is an intermediate dimension, representing the new dimension of the data after being processed by the first MMA operator; The sixth output tensor is input into the second MMA operator for matrix multiplication and accumulation operation, and a fifth output tensor is obtained as an output. The data shape of the weight matrix of the second MMA operator is [W, K].

7. An optimization device for a hybrid expert system, characterized in that: The hybrid expert system includes a routing operator and a plurality of feedforward networks, and the device includes: a first computing module, configured to input an input sequence into the routing operator and output a first output tensor and a second output tensor, wherein the first output tensor represents a weight value of each feedforward network corresponding to each input tag in the input sequence; when the feedforward network is not activated, the weight value of the feedforward network in the first output tensor is 0; and the second output tensor represents the total number of activated feedforward networks and an index of each activated feedforward network; The second computing module is used to input the input sequence, the first output tensor and the second output tensor into each of the feedforward networks, the activated feedforward network outputs a third output tensor, and accumulates and sums the third output tensor to obtain an output result of the hybrid expert system, where the output result is consistent with the data shape of the input sequence.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images