Optimization method and device of hybrid expert system, computer equipment and readable storage medium

Abstract

The invention relates to an optimization method and device of a hybrid expert system, computer equipment and a readable storage medium. The method comprises the steps that an input sequence is input into a routing operator, a first output tensor and a second output tensor are obtained through output, the first output tensor represents the weight value of each input mark in the input sequence corresponding to each feed-forward network, when the feed-forward network is not activated, the weight value of the feed-forward network in the first output tensor is 0, and when the feed-forward network is not activated, the weight value of the feed-forward network in the second output tensor is 0; the second output tensor represents the total number of the activated feed-forward networks and the index of each activated feed-forward network; and inputting the input sequence, the first output tensor and the second output tensor into each feed-forward network, outputting a third output tensor by the activated feed-forward network, and performing accumulation and summation processing on the third output tensor to obtain an output result that the hybrid expert system is consistent with the data shape of the input sequence. By adopting the method, the reasoning efficiency of the hybrid expert system in a reasoning scene can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an optimization method, apparatus, computer equipment, and readable storage medium for a hybrid expert system. Background Art

[0002] In reasoning scenarios, mixed expert systems (MOEs) have been widely used. MOEs intelligently select different experts (feed-forward networks (FFNs)) to process input data, demonstrating unique advantages in improving model expressiveness.

[0003] Currently, the conventional implementation of the MOE layer uses a router operator to calculate the weights corresponding to each expert. This calculation is performed on each input token to determine the expert activated by the current token. During this process, because the data shape during expert calculation differs from the shape of the MOE layers before and after, a permutation operator is used to convert the MOE layer shape to the shape required by the FFN. After the FFN calculation is complete, the unpermutation operator is used to convert the FFN output shape back to the MOE layer shape, thus implementing the layer-by-layer calculation of the MOE layer.

[0004] However, in the reasoning scenario, the sequence length (seqlen) is usually short, and the number of tokens that need to be reasoned is relatively small, which results in only a small number of experts being activated and most experts not needing to participate in the calculation. Figure 1 As shown, the calculation of the MOE layer covers Router ( Figure 1 The symbol is a routing operator), Permuation ( Figure 1 Identified as permutation operator), FFN and Unpermuation ( Figure 1 The permutation and unpermutation operators (denoted as the inverse permutation operator in Figure 2) have four operators. The permutation and unpermutation operators suffer from severe performance bottlenecks due to their small read and write instruction granularity. This performance shortcoming significantly reduces the computational efficiency of the entire MOE layer, adversely affecting the overall performance of the inference model and reducing the MOE's inference efficiency in inference scenarios. Summary of the Invention

[0005] Based on this, it is necessary to provide an optimization method, device, computer equipment and readable storage medium for a hybrid expert system that can improve the reasoning efficiency of the hybrid expert system in reasoning scenarios in order to address the above technical problems.

[0006] In a first aspect, the present application provides an optimization method for a hybrid expert system, wherein the hybrid expert system includes a routing operator and multiple feedforward networks, including:

[0007] Inputting an input sequence into the routing operator, and outputting a first output tensor and a second output tensor, wherein the first output tensor represents the weight value of each feedforward network corresponding to each input tag in the input sequence. When the feedforward network is not activated, the weight value of the feedforward network in the first output tensor is 0, and the second output tensor represents the total number of activated feedforward networks and the index of each activated feedforward network;

[0008] The input sequence, the first output tensor, and the second output tensor are input into each of the feedforward networks, the activated feedforward network outputs a third output tensor, and the third output tensor is accumulated and summed to obtain an output result of the hybrid expert system, where the output result is consistent with the data shape of the input sequence.

[0009] In one embodiment, the data shape of the first output tensor is [E, NS], where E represents the total number of feedforward networks, N represents the batch, and S represents the sequence length. The data shape of the second output tensor is [1, 1+E], and the first address of the second output tensor is used to represent the total number of activated feedforward networks, and the remaining addresses are used to represent the indexes of the activated feedforward networks.

[0010] In one embodiment, the routing operator includes a matrix multiplication and accumulation (MMA) operator and a top k value (Topk) operator, and the routing operator processes the input sequence and outputs the first output tensor and the second output tensor, including:

[0011] Inputting the input sequence into the MMA operator for matrix multiplication and accumulation, and outputting a fourth output tensor, where the fourth output tensor is a matrix multiplication and accumulation result of the input sequence and a weight matrix corresponding to the MMA operator, wherein the data shape of the input sequence is [1, NS, K], the data shape of the weight matrix is [K, E], and the data shape of the fourth output tensor is [1, NS, E], where K represents the feature dimension;

[0012] The fourth output tensor is input into the Topk operator, and the first output tensor and the second output tensor are output.

[0013] In one embodiment, inputting the input sequence, the first output tensor, and the second output tensor into each of the feedforward networks, and outputting a third output tensor from the activated feedforward network, comprises:

[0014] Each of the feedforward networks determines whether it is an activated feedforward network based on the second output tensor, and when it is determined that it is an activated feedforward network, processes the input sequence based on the first output tensor and outputs a third output tensor.

[0015] In one embodiment, processing the input sequence based on the first output tensor to output a third output tensor includes:

[0016] The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor;

[0017] The dot product result of the fifth output tensor and the first output tensor is output as the third output tensor, and the fifth output tensor and the third output tensor are both consistent with the data shape of the input sequence.

[0018] In one embodiment, the feedforward network includes a first MMA operator and an activation function, and a second MMA operator. The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor, including:

[0019] Inputting the input sequence into the first MMA operator to perform a matrix multiplication and accumulation operation, and inputting the operation result into the activation function to obtain a sixth output tensor, wherein the data shape of the input sequence is [1, NS, K], where K represents the data dimension, the data shape of the weight matrix of the first MMA operator is [K, W], and the data shape of the sixth output tensor is [1, NS, W].

[0020] The sixth output tensor is input into the second MMA operator for matrix multiplication and accumulation operation, and a fifth output tensor is obtained as an output. The data shape of the weight matrix of the second MMA operator is [W, K].

[0021] In a second aspect, the present application further provides an optimization device for a hybrid expert system, wherein the hybrid expert system includes a routing operator and multiple feedforward networks, and the device includes:

[0022] a first computing module, configured to input an input sequence into the routing operator and output a first output tensor and a second output tensor, wherein the first output tensor represents a weight value of each feedforward network corresponding to each input tag in the input sequence; when the feedforward network is not activated, the weight value of the feedforward network in the first output tensor is 0; and the second output tensor represents the total number of activated feedforward networks and an index of each activated feedforward network;

[0023] The second computing module is used to input the input sequence, the first output tensor and the second output tensor into each of the feedforward networks, the activated feedforward network outputs a third output tensor, and accumulates and sums the third output tensor to obtain an output result of the hybrid expert system, where the output result is consistent with the data shape of the input sequence.

[0024] In one embodiment, the data shape of the first output tensor is [E, NS], where E represents the total number of feedforward networks, N represents the batch, and S represents the sequence length. The data shape of the second output tensor is [1, 1+E], and the first address of the second output tensor is used to represent the total number of activated feedforward networks, and the remaining addresses are used to represent the indexes of the activated feedforward networks.

[0025] In one embodiment, the routing operator includes a matrix multiplication and accumulation (MMA) operator and a top k value (Topk) operator, and the routing operator processes the input sequence and outputs the first output tensor and the second output tensor, including:

[0026] Inputting the input sequence into the MMA operator for matrix multiplication and accumulation, and outputting a fourth output tensor, where the fourth output tensor is a matrix multiplication and accumulation result of the input sequence and a weight matrix corresponding to the MMA operator, wherein the data shape of the input sequence is [1, NS, K], the data shape of the weight matrix is [K, E], and the data shape of the fourth output tensor is [1, NS, E], where K represents the feature dimension;

[0027] The fourth output tensor is input into the Topk operator, and the first output tensor and the second output tensor are output.

[0028] In one embodiment, inputting the input sequence, the first output tensor, and the second output tensor into each of the feedforward networks, and outputting a third output tensor from the activated feedforward network, comprises:

[0029] Each of the feedforward networks determines whether it is an activated feedforward network based on the second output tensor, and when it is determined that it is an activated feedforward network, processes the input sequence based on the first output tensor and outputs a third output tensor.

[0030] In one embodiment, processing the input sequence based on the first output tensor to output a third output tensor includes:

[0031] The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor;

[0032] The dot product result of the fifth output tensor and the first output tensor is output as the third output tensor, and the fifth output tensor and the third output tensor are both consistent with the data shape of the input sequence.

[0033] In one embodiment, the feedforward network includes a first MMA operator and an activation function, and a second MMA operator. The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor, including:

[0034] Inputting the input sequence into the first MMA operator to perform a matrix multiplication and accumulation operation, and inputting the operation result into the activation function to obtain a sixth output tensor, wherein the data shape of the input sequence is [1, NS, K], where K represents the data dimension, the data shape of the weight matrix of the first MMA operator is [K, W], and the data shape of the sixth output tensor is [1, NS, W].

[0035] The sixth output tensor is input into the second MMA operator for matrix multiplication and accumulation operation, and a fifth output tensor is obtained as an output. The data shape of the weight matrix of the second MMA operator is [W, K].

[0036] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the above hybrid expert system optimization methods when executing the computer program.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above hybrid expert system optimization methods.

[0038] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements any of the above hybrid expert system optimization methods when executed by a processor.

[0039] The above-mentioned hybrid expert system optimization method, apparatus, computer device, and readable storage medium input an input sequence into the hybrid expert system's routing operator, outputting a first output tensor and a second output tensor. The first output tensor represents the weight value of each feedforward network corresponding to each input tag in the input sequence. When the feedforward network is not activated, the weight value of the feedforward network in the first output tensor is 0. The second output tensor represents the total number of activated feedforward networks and the index of each activated feedforward network. The input sequence, the first output tensor, and the second output tensor are input into each feedforward network. The activated feedforward network outputs a third output tensor. The third output tensor is accumulated and summed to obtain the output result of the hybrid expert system, which is consistent with the data shape of the input sequence. The hybrid expert system optimization method, apparatus, computer device, and readable storage medium provided in the embodiments of the present application change the output of the routing operator in the hybrid expert system so that the subsequent feedforward network layer can directly obtain the activation status corresponding to the global input token. By combining the total number of activated feedforward networks output and the index of the activated feedforward network, it is possible to calculate and output only the activated feedforward network, and obtain a calculation result consistent with the shape of the input sequence data. In other words, the hybrid expert system provided in the embodiments of the present application eliminates the Permuation and Unpermuation operators, thereby avoiding the large amount of inefficient data transposition and handling caused by the Permuation and Unpermuation operators, which can significantly reduce the consumption of inefficient computing power, improve the computational efficiency of the hybrid expert system, and thus improve the reasoning efficiency of the hybrid expert system in reasoning scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 is a schematic diagram of a hybrid expert system in related art in one embodiment;

[0042] Figure 2 A schematic flow chart of an optimization method of a hybrid expert system in one embodiment;

[0043] Figure 3 Schematic diagram of the flow of routing operator operation in one embodiment;

[0044] Figure 4 1 is a schematic diagram of a flow chart of a routing operator performing operations based on a first output tensor in one embodiment;

[0045] Figure 5 A schematic diagram of a flow chart of operations performed by a feedforward network in one embodiment;

[0046] Figure 6 A schematic diagram of the operation process of a traditional hybrid expert system in another embodiment;

[0047] Figure 7 A schematic diagram of an improved routing operator optimization in one embodiment;

[0048] Figure 8 A schematic diagram of an optimized and improved hybrid expert system in one embodiment;

[0049] Figure 9 A structural block diagram of an optimization device of a hybrid expert system in one embodiment;

[0050] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] It should be noted that the terms "first", "second", etc. used in this application may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "including" and "having" used in this application and any variations thereof are intended to cover non-exclusive inclusions. The term "plurality" used in this application refers to two or more. The term "and / or" used in this application refers to one of the solutions or any combination of multiple solutions.

[0053] In an exemplary embodiment, Figure 2 As shown, a hybrid expert system optimization method is provided, which is applied to the host side for example. It is understood that the host side may include a CPU (Central Processing Unit). The hybrid expert system includes a routing operator and multiple feedforward networks. The method may include the following steps 202 to 204. Among them:

[0054] In step 202, the input sequence is input into the routing operator, and a first output tensor and a second output tensor are output. The first output tensor represents the weight value of each feedforward network corresponding to each input tag in the input sequence. When the feedforward network is not activated, the weight value of the feedforward network in the first output tensor is 0. The second output tensor represents the total number of activated feedforward networks and the index of each activated feedforward network.

[0055] The optimization method of the hybrid expert system provided in the embodiment of the present application can be applied to artificial intelligence fields such as natural language processing and image recognition to improve the efficiency and accuracy of the model in processing complex tasks. The following embodiments will be explained by taking text classification in natural language processing tasks as an example. Assume that the input sequence is a text to be classified, and the text is divided into multiple input tags (which can also be expressed as tokens or input elements, etc.), which constitute an input sequence of length S [x1, x2, ..., x S ].

[0056] An input sequence can be fed into a routing operator. Based on pre-trained parameters, the routing operator analyzes each token in the input sequence and determines which of multiple feedforward networks is suitable for processing the token. For example, the routing operator can determine the activated feedforward network for each token by calculating the matching degree between each token and each feedforward network. The output of the routing operator includes a first output tensor and a second output tensor.

[0057] In an exemplary embodiment, the data shape of the first output tensor is [E, NS], where E represents the total number of feedforward networks, N represents the batch, and S represents the sequence length. The data shape of the second output tensor is [1, 1+E], and the first address of the second output tensor is used to represent the total number of activated feedforward networks, and the remaining addresses are used to represent the indexes of the activated feedforward networks.

[0058] That is, each element A in the first output tensor A ij Represents the activation state of the i-th feedforward network for the j-th token. When it is in the activated state for the j-th token, the A ij is the corresponding weight value. If it is in an unactivated state for the jth token, then A ij is 0.

[0059] The second output tensor B contains two parts of information, one is the total number of activated feedforward networks Y, and the other is the index of each activated feedforward network [I1, I2, ..., I Y ], where the first address B0 of the second output tensor is the total number Y of feedforward networks activated by each token of the input sequence in the E feedforward networks, and the rest are B1, B2, ..., B E+1, is the index of the activated feedforward network [I1, I2, ..., I K ], I i Represents the index of the activated i-th feedforward network in all feedforward networks. For example, there are a total of 8 feedforward networks FFN, among which the indexes of the feedforward networks activated by the input are 1, 3, and 5. Then the second output tensor is [3, 1, 3, 5, 0, 0, 0, 0, 0].

[0060] In step 204, the input sequence, the first output tensor, and the second output tensor are input into each feedforward network. The activated feedforward network outputs a third output tensor, and the third output tensor is accumulated and summed to obtain the output result of the hybrid expert system. The output result is consistent with the data shape of the input sequence.

[0061] In the embodiment of the present application, the input sequence, the first output tensor and the second output tensor can be input into each feedforward network. Only the activated feedforward network (i.e., the index in [I1, I2, ..., I Y ]) processes the input sequence and outputs a third output tensor. The shape of the third output tensor output by each activated feedforward network is consistent with the data shape of the input sequence. All third output tensors are cumulatively summed, that is, the elements at each position are added together to obtain the output of the hybrid expert system. The shape of the output result is also consistent with the data shape of the input sequence in terms of dimension. This output result can be used for subsequent text classification tasks, for example, by processing the output result corresponding to each token through a further classifier to determine the category to which the text belongs.

[0062] The above-mentioned hybrid expert system optimization method inputs the input sequence into the routing operator of the hybrid expert system, and outputs a first output tensor and a second output tensor, wherein the first output tensor represents the activated feedforward network among multiple feedforward networks for each token in the input sequence. The value of the activated feedforward network in the first output tensor is the corresponding weight, and the value of the inactivated feedforward network is 0. The second output tensor represents the total number of activated feedforward networks and the index of each activated feedforward network. The input sequence, the first output tensor, and the second output tensor are input into each feedforward network, and the activated feedforward network outputs a third output tensor. The third output tensor is accumulated and summed to obtain the output result of the hybrid expert system, which is consistent with the data shape of the input sequence. The hybrid expert system optimization method provided in the embodiment of the present application changes the output of the routing operator in the hybrid expert system so that the subsequent feedforward network layer can directly obtain the activation status corresponding to the global input token. Combined with the total number of activated feedforward networks output and the index of the activated feedforward network, it is possible to calculate and output only the activated feedforward network, and obtain a calculation result consistent with the shape of the input sequence data. In other words, the hybrid expert system provided in the embodiment of the present application eliminates the Permuation and Unpermuation operators, thereby avoiding the large amount of inefficient data transposition and transportation caused by the Permuation and Unpermuation operators, which can significantly reduce the consumption of inefficient computing power, improve the computational efficiency of the hybrid expert system, and thus improve the reasoning efficiency of the hybrid expert system in reasoning scenarios.

[0063] In an exemplary embodiment, the routing operator includes a matrix multiplication accumulation (MMA) operator and a top k value (Topk) operator. Figure 3 As shown, the process of the routing operator processing the input sequence and outputting the first output tensor and the second output tensor may include steps 302 to 304, wherein:

[0064] Step 302: Input the input sequence into the MMA operator for matrix multiplication, and output a fourth output tensor. The fourth output tensor is the matrix multiplication accumulation result of the input sequence and the weight matrix corresponding to the MMA operator. The data shape of the input sequence is [1, NS, K], the data shape of the weight matrix is [K, E], and the data shape of the fourth output tensor is [1, NS, E], where K represents the feature dimension.

[0065] Step 304: Input the fourth output tensor into the Topk operator to obtain the first output tensor and the second output tensor.

[0066] In the embodiment of the present application, the routing operator includes an MMA (Matrix Multiplication Accumulation) operator and a Topk operator. The process of processing the input sequence and outputting the first output tensor and the second output tensor is as follows:

[0067] For example, assume that the shape of the input sequence is [1, NS, K], where N is the batch, S is the seqlen (sequence length), NS means folding batch and seqlen onto one axis, and K represents the feature dimension. At the same time, the MMA operator pre-stores a weight matrix with a dimension of [K, E]. The MMA operator performs a matrix multiplication and accumulation operation on the input sequence and the weight matrix to obtain and output a fourth output tensor with a shape of [1, NS, E]. Each element P in the fourth output tensor is ij It represents the matching score between the i-th token in the input sequence and the j-th feedforward network, which reflects the suitability of the j-th feedforward network for processing the i-th token.

[0068] Furthermore, the fourth output tensor is fed into the Topk operator. The Topk operator selects the k highest-scoring values (k is a pre-set threshold for the number of activated feedforward networks) from each row of the fourth output tensor (i.e., the matching scores of all feedforward networks corresponding to each token) and records the indices corresponding to these values. After processing by the Topk operator, the outputs are a first output tensor and a second output tensor. The first output tensor has a shape of [E, NS]. For each of the k largest values selected from each row of the fourth output tensor, the corresponding positions in the first output tensor are filled with the corresponding weight values (these weight values can be further converted based on the matching scores, such as by normalization). The remaining positions are filled with zeros, representing the tokens in the input sequence that activated each feedforward network and their corresponding weights. The second output tensor has a shape of [1, 1+E]. It begins with the total number of activated feedforward networks (Y), followed by the indices of each activated feedforward network, and the remaining positions are filled with zeros.

[0069] The hybrid expert system optimization method provided in the embodiment of the present application changes the output of the Topk operator in the routing operator so that the subsequent feedforward network layer can directly obtain the activation status corresponding to the global token. Combined with the total number of activated feedforward networks output and the index of the activated feedforward network, it is possible to calculate and output only the activated feedforward network, and obtain a calculation result consistent with the shape of the input sequence data. In other words, the Permuation and Unpermuation operators in the hybrid expert system can be eliminated, thereby avoiding the large amount of inefficient data transposition and transportation caused by the Permuation and Unpermuation operators, significantly reducing the consumption of inefficient computing power, improving the computational efficiency of the hybrid expert system, and thereby improving the reasoning efficiency of the hybrid expert system in reasoning scenarios.

[0070] In an exemplary embodiment, inputting the input sequence, the first output tensor, and the second output tensor into each feedforward network, and outputting the third output tensor from the activated feedforward network may include the following steps:

[0071] Each feedforward network determines whether it is an activated feedforward network based on the second output tensor, and when it is determined that it is an activated feedforward network, processes the input sequence based on the first output tensor and outputs a third output tensor.

[0072] In this embodiment of the present application, after completing the processing of the routing operator, a first output tensor (with a shape of [E, NS]) and a second output tensor (with a shape of [1, 1+E]) can be obtained, and these data and the input sequence are input into E feedforward networks together.

[0073] Each feedforward network determines whether it is an activated feedforward network based on the second output tensor. Exemplarily, for the j-th feedforward network, the total number Y of activated feedforward networks is first read from the first address of the second output tensor. Subsequently, starting from the starting position where the index information is stored in the second output tensor, the index values of the activated feedforward networks are read in sequence. During the traversal process, the read index value is compared with its own index j: if the read index value is found to be equal to j during the traversal process, the j-th feedforward network is immediately determined to be activated and the traversal is stopped; if the traversal continues until Y index values are read and no index value equal to j appears, it is determined that the j-th feedforward network is not activated, and the j-th feedforward network does not participate in subsequent processing.

[0074] When determining that it is an activated feedforward network, the feedforward network processes the input sequence based on the first output tensor. ij Represents the weight of the i-th feedforward network activated by the j-th token in the input sequence. The feedforward network is based on A ijThe weight value of the second feedforward network is used to weight the corresponding tokens in the input sequence, and combined with its own network structure and parameters, it performs feature extraction, transformation and other operations on the weighted tokens, and finally outputs the third output tensor. For example, for token x3 in the input sequence, if the second feedforward network is activated and the corresponding element A in the first output tensor is 23 is 0.8, the second feedforward network processes x3 with a weight of 0.8, and after internal multi-layer neuron calculations, it outputs feature information related to token x3, which constitutes part of the third output tensor.

[0075] Each activated feedforward network processes the input sequence in the same manner as described above and outputs a third output tensor. The shape of the third output tensors output by all activated feedforward networks is consistent with the data shape of the input sequence. All third output tensors are then accumulated and summed to obtain the output of the hybrid expert system.

[0076] The optimization method of the hybrid expert system provided in the embodiment of the present application is that the feedforward network layer can directly obtain the activation status corresponding to the global token, and combine the total number of activated feedforward networks and the index of the activated feedforward network output, without the need for a large amount of inefficient data transposition and transportation, to achieve calculation and output only for the activated feedforward network, thereby greatly reducing the consumption of invalid computing power, improving the computing efficiency of the hybrid expert system, and thereby improving the reasoning efficiency of the hybrid expert system in the reasoning scenario.

[0077] In an exemplary embodiment, referring to Figure 4 As shown, processing the input sequence based on the first output tensor and outputting a third output tensor may include the following steps 402 to 404, wherein:

[0078] Step 402: The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor.

[0079] Step 404: Output the dot product result of the fifth output tensor and the first output tensor as the third output tensor. Both the fifth output tensor and the third output tensor have the same data shape as the input sequence.

[0080] In the embodiment of the present application, the activated feedforward network performs weighted processing on the input sequence to obtain and output the fifth output tensor. For example, the activated feedforward network performs weighted calculation on each token in the input sequence based on its own network parameters. i For example, the activated feedforward network adjusts x according to the pre-trained weight matrix i The weighted sum of each feature dimension of x is obtained to obtain a weighted vector representation, which contains the weight of the feedforward network on x.i The initial processing results of the features. After performing the above weighted processing on all tokens in the input sequence, a fifth output tensor with the same dimension as the input sequence is obtained. Each element in this tensor corresponds to the weighted feature value of a token in the input sequence.

[0081] Assuming the shape of the input sequence tensor is [1, NS, K], you can get the fifth output tensor of shape [1, NS, K].

[0082] Since the shapes of the fifth output tensor and the first output tensor are different, direct dot multiplication will cause dimension mismatch problems, so the broadcast mechanism in the K dimension can be used. When broadcasting in the K dimension, the system automatically expands or aligns the tensor dimensions so that the two tensors meet the dot multiplication requirements in the K dimension. For example, for the feature vector (shape is [1, 1, K]) corresponding to each token in the fifth output tensor and the vector (shape is [E, 1]) at the corresponding position in the first output tensor, [E, 1] is expanded to [E, K] through broadcasting in the K dimension, so that the two can perform corresponding element multiplication. For example, if the feature vector corresponding to the i-th token in the fifth output tensor is [f i1 , f i2 ,...,f iK ] (where the vector corresponds to the first instance, the i-th NS position, and the K-dimensional feature in the [1, NS, K] tensor), the vector at the corresponding position in the first output tensor is [a i1 , a i2 ,...,a iE ], after K-dimensional broadcasting, f i1 With the expanded [a i1 , a i2 ,…, a iE ] Corresponding element multiplication, f i2 Multiply it with the corresponding element after expansion, and so on, complete the multiplication operation of all K-dimensional elements, and then sum the results to get a scalar value.

[0083] After performing the above dot product operation on all NS tokens in the input sequence, a third output tensor with the same shape as the input sequence and the fifth output tensor is obtained, also with the shape [1, NS, K]. In other words, through the above process, the activated feedforward network processes the input sequence based on the first output tensor and outputs a third output tensor with the same shape as the input sequence. The output of the MOE is then accumulated by adding the third output tensors of each activated expert. The shape of this output is the same as the input sequence, also with the shape [1, NS, K].

[0084] In the embodiment of the present application, by adding a small amount of calculation to the feedforward network, the transposition and transfer operations of Permuation and Unpermuation can be eliminated, which can greatly improve the computational efficiency of the hybrid expert system.

[0085] In an exemplary embodiment, referring to Figure 5 As shown, the feedforward network includes a first MMA operator and an activation function, and a second MMA operator. The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor, including the following steps 502 to 504, wherein:

[0086] Step 502: Input the input sequence into the first MMA operator to perform a matrix multiplication and accumulation operation. The operation result is input into the activation function to obtain a sixth output tensor. The data shape of the input sequence is [1, NS, K], where K represents the data dimension. The data shape of the weight matrix of the first MMA operator is [K, W], and the data shape of the sixth output tensor is [1, NS, W].

[0087] Step 504: Input the sixth output tensor into the second MMA operator for matrix multiplication and accumulation operation to obtain a fifth output tensor. The data shape of the weight matrix of the second MMA operator is [W, K].

[0088] In this embodiment of the present application, it is assumed that the data shape of the input sequence is [1, NS, K], and the data shape of the weight matrix of the first MMA operator is [K, W], where K is the same as the third dimension of the input sequence, and W represents the new dimension of the data after processing by the first MMA operator, which is an intermediate dimension. When matrix multiplication is performed on the input sequence and the weight matrix of the first MMA operator, the feature vector corresponding to each token in the input sequence (with a shape of [1, 1, K]) is multiplied by the weight matrix [K, W]. According to the matrix multiplication rules, the K-dimensional vector is multiplied by the [K, W] matrix to obtain a W-dimensional vector. After performing this multiplication operation on all NS tokens in the input sequence, an intermediate result with a shape of [1, NS, W] is obtained.

[0089] The intermediate result is input into the activation function, which performs a nonlinear transformation on each element, such as using a Sigmoid function or a ReLU function, to map each element in the intermediate result to a specific interval, and finally obtains the sixth output tensor with a shape of [1, NS, W].

[0090] The weight matrix of the second MMA operator has a data shape of [W, K], where W matches the third dimension of the sixth output tensor, and K represents the data restored to the same dimension as the input sequence after processing by the second MMA operator. When performing matrix multiplication and accumulation on the sixth output tensor and the weight matrix of the second MMA operator, the feature vector corresponding to each token in the sixth output tensor (with a shape of [1, 1, W]) is multiplied by the weight matrix [W, K]. According to the matrix multiplication rules, the W-dimensional vector is multiplied by the [W, K] matrix to produce a K-dimensional vector. After performing this multiplication on all NS tokens in the sixth output tensor, a fifth output tensor with a shape of [1, NS, K] is produced, which has the same data shape as the input sequence.

[0091] In the embodiment of the present application, by adding a small amount of calculation to the feedforward network, the transposition and transfer operations of Permuation and Unpermuation can be eliminated, which can greatly improve the computational efficiency of the hybrid expert system.

[0092] In order to enable those skilled in the art to better understand the embodiments of the present application, the embodiments of the present application are described below with reference to specific examples.

[0093] Reference Figure 6 As shown, in a traditional MOE system, a Router operator is used at each MOE layer to calculate the weights of different experts, or feedforward networks. The weight of the selected expert is calculated by traversing each input token, and the expert whose current token is activated is selected from among the many experts. However, in this process, because the shape of each expert's calculation differs from the shapes of the previous and next MOE layers, a Permuation operator is required to convert the MOE layer shape to the shape required by the FFN. After the FFN (expert calculation) is complete, an Unpermutation operator converts the FFN output shape back to the MOE layer shape, allowing layer-by-layer calculations to be performed.

[0094] In inference scenarios, sequences are typically short, and the number of tokens requiring inference is relatively small. This means that only a small subset of all experts are activated, while the majority require no computation. Traditionally, the MOE layer's computations include four operators: Router, Permuation, FFN, and Unpermuation. The Permuation and Unpermuation operators suffer from poor performance due to their small read and write instruction granularity, leading to poor performance across the entire MOE layer and, consequently, impacting the performance of the inference model.

[0095] Reference Figure 7As shown, the optimization method of the hybrid expert system provided in the embodiment of the present application changes the output of the Router operator into a first output tensor (which can also be expressed as mask out) and a second output tensor (which can also be expressed as hittedexpert). The first output tensor is the weight of all global tokens and corresponding experts, and the second output tensor indicates which experts are activated by all global tokens (including the total number of activated experts). By adding the above two inputs to the input of the Router, each subsequent FFN can calculate the global tensor by selecting the activated expert number, and accumulate the output according to the weight in the first output tensor to obtain the output result of the final MOE layer.

[0096] Reference Figure 8 As shown, the Router operator (Router) consists of MMA and topK. The MMA inputs are an input sequence of shape [1, NS, K] and a weight matrix of shape [K, E]. After completing the matrix multiplication, the output is a tensor of shape [1, NS, E]. This tensor serves as the input of TopK. The output is a first output tensor of shape [E, NS] and a second output tensor of shape [1, 1+E]. Among them, [E, NS] represents all NS tokens and which experts are activated among all experts. The value of inactivated experts is 0, and the value of activated experts is the corresponding weight. And [1, 1+E] represents which experts are activated among all NS tokens. Among them, the first address hitted_expert[0] records the total number of activated experts, and the rest of the content represents the index number of the activated expert. Among them, record N is the batch (batch), S is the sequence length (seqlen) n, NS means that batch and seqlen can be folded onto one axis, K represents the feature dimension (hidden), and E represents the total number of experts.

[0097] From the perspective of the MOE layer, the input and output shapes are both [1, NS, K], and the FFN is the activated expert in the E dimension. In the second output tensor, the activated expert performs the first MMA and swinGLU (Shifted Window based GLU) in the FFN on the same input according to its own weights. The shape of the output tensor is [1, NS, W]. This tensor is right-multiplied by the weight of the second MMA, and the result is [1, NS, K]. This result is dot-multiplied with the first output tensor (and broadcasted in the K dimension). The result is the final output after the weighted activation of the topK experts for each token.

[0098] By adopting the optimization method of the hybrid expert system provided in the embodiment of the present application, in the decoding scenario of inference, only a small amount of calculation and tensor output of the Router operator is added, and the FFN calculation is appropriately increased, which eliminates a large number of inefficient Permuation / Unpermuation operations, and can greatly improve the performance of the MOE layer. Not only can the model accuracy remain the same as before without degradation, but it also has good generalization. Any network including the MOE layer is applicable to this application.

[0099] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps. It is understandable that the various steps in different embodiments can be freely combined as needed, and the various non-contradictory schemes formed by the combination all fall within the scope of protection of this application.

[0100] Based on the same inventive concept, embodiments of the present application also provide a hybrid expert system optimization device for implementing the aforementioned hybrid expert system optimization method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more hybrid expert system optimization device embodiments provided below can be found in the above-described limitations of the hybrid expert system optimization method and will not be further elaborated here.

[0101] In an exemplary embodiment, Figure 9 As shown, a hybrid expert system optimization device 900 is provided. The hybrid expert system includes a routing operator and multiple feedforward networks. The device includes: a first calculation module 902 and a second calculation module 904, wherein:

[0102] A first computing module 902 is configured to input an input sequence into the routing operator and output a first output tensor and a second output tensor, wherein the first output tensor represents a weight value of each feedforward network corresponding to each input tag in the input sequence. When the feedforward network is not activated, the weight value of the feedforward network in the first output tensor is 0. The second output tensor represents the total number of activated feedforward networks and the index of each activated feedforward network.

[0103] The second computing module 904 is used to input the input sequence, the first output tensor and the second output tensor into each of the feedforward networks, and the activated feedforward network outputs a third output tensor, and accumulates and sums the third output tensor to obtain an output result of the hybrid expert system, where the output result is consistent with the data shape of the input sequence.

[0104] The optimization device of the hybrid expert system inputs an input sequence into the routing operator of the hybrid expert system and outputs a first output tensor and a second output tensor, wherein the first output tensor represents the activated feedforward network among multiple feedforward networks for each token in the input sequence, the value of the activated feedforward network in the first output tensor being the corresponding weight, and the value of the inactivated feedforward network being 0, and the second output tensor representing the total number of activated feedforward networks and the index of each activated feedforward network. The input sequence, the first output tensor, and the second output tensor are input into each feedforward network, and the activated feedforward network outputs a third output tensor. The third output tensor is accumulated and summed to obtain the output result of the hybrid expert system, which is consistent with the data shape of the input sequence. The optimization device for the hybrid expert system provided in the embodiment of the present application changes the output of the routing operator in the hybrid expert system so that the subsequent feedforward network layer can directly obtain the activation status corresponding to the global input token. Combined with the total number of activated feedforward networks output and the index of the activated feedforward network, it is possible to calculate and output only the activated feedforward network, and obtain a calculation result consistent with the shape of the input sequence data. In other words, the hybrid expert system provided in the embodiment of the present application eliminates the Permuation and Unpermuation operators, thereby avoiding the large amount of inefficient data transposition and transportation caused by the Permuation and Unpermuation operators, which can significantly reduce the consumption of inefficient computing power, improve the computational efficiency of the hybrid expert system, and thus improve the reasoning efficiency of the hybrid expert system in reasoning scenarios.

[0105] In one embodiment, the data shape of the first output tensor is [E, NS], where E represents the total number of feedforward networks, N represents the batch, and S represents the sequence length. The data shape of the second output tensor is [1, 1+E], and the first address of the second output tensor is used to represent the total number of activated feedforward networks, and the remaining addresses are used to represent the indexes of the activated feedforward networks.

[0106] In one embodiment, the routing operator includes a matrix multiplication and accumulation (MMA) operator and a top k value (Topk) operator, and the routing operator processes the input sequence and outputs the first output tensor and the second output tensor, including:

[0107] Input the input sequence into the MMA operator for matrix multiplication, and output a fourth output tensor, where the fourth output tensor is a matrix multiplication accumulation result of the input sequence and the weight matrix corresponding to the MMA operator, wherein the data shape of the input sequence is [1, NS, K], the data shape of the weight matrix is [K, E], and the data shape of the fourth output tensor is [1, NS, E], where K represents the feature dimension;

[0108] The fourth output tensor is input into the Topk operator, and the first output tensor and the second output tensor are output.

[0109] In one embodiment, inputting the input sequence, the first output tensor, and the second output tensor into each of the feedforward networks, and outputting a third output tensor from the activated feedforward network, comprises:

[0110] Each of the feedforward networks determines whether it is an activated feedforward network based on the second output tensor, and when it is determined that it is an activated feedforward network, processes the input sequence based on the first output tensor and outputs a third output tensor.

[0111] In one embodiment, processing the input sequence based on the first output tensor to output a third output tensor includes:

[0112] The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor;

[0113] The dot product result of the fifth output tensor and the first output tensor is output as the third output tensor, and the fifth output tensor and the third output tensor are both consistent with the data shape of the input sequence.

[0114] In one embodiment, the feedforward network includes a first MMA operator and an activation function, and a second MMA operator. The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor, including:

[0115] Inputting the input sequence into the first MMA operator to perform a matrix multiplication and accumulation operation, and inputting the operation result into the activation function to obtain a sixth output tensor, wherein the data shape of the input sequence is [1, NS, K], where K represents the data dimension, the data shape of the weight matrix of the first MMA operator is [K, W], and the data shape of the sixth output tensor is [1, NS, W].

[0116] The sixth output tensor is input into the second MMA operator for matrix multiplication and accumulation operation, and a fifth output tensor is obtained as an output. The data shape of the weight matrix of the second MMA operator is [W, K].

[0117] Each module in the optimization device of the hybrid expert system described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0118] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 10 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements an optimization method for a hybrid expert system. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0119] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0120] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0121] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0122] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0124] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0125] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0126] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A hybrid expert system optimization method, characterized in that: The hybrid expert system includes a routing operator and a plurality of feedforward networks, and the method includes: Inputting an input sequence into the routing operator, and outputting a first output tensor and a second output tensor, wherein the first output tensor represents the weight value of each feedforward network corresponding to each input tag in the input sequence. When the feedforward network is not activated, the weight value of the feedforward network in the first output tensor is 0, and the second output tensor represents the total number of activated feedforward networks and the index of each activated feedforward network; The input sequence, the first output tensor, and the second output tensor are input into each of the feedforward networks, the activated feedforward network outputs a third output tensor, and the third output tensor is accumulated and summed to obtain an output result of the hybrid expert system, where the output result is consistent with the data shape of the input sequence.

2. The method according to claim 1, characterized in that The data shape of the first output tensor is [E, NS], where E represents the total number of feedforward networks, N represents the batch, and S represents the sequence length. The data shape of the second output tensor is [1, 1+E]. The first address of the second output tensor is used to represent the total number of activated feedforward networks, and the remaining addresses are used to represent the indexes of the activated feedforward networks.

3. The method according to claim 1 or 2, characterized in that The routing operator includes a matrix multiplication and accumulation (MMA) operator and a top k value (Topk) operator. The routing operator processes the input sequence and outputs the first output tensor and the second output tensor, including: Inputting the input sequence into the MMA operator for matrix multiplication and accumulation, and outputting a fourth output tensor, where the fourth output tensor is a matrix multiplication and accumulation result of the input sequence and a weight matrix corresponding to the MMA operator, wherein the data shape of the input sequence is [1, NS, K], the data shape of the weight matrix is [K, E], and the data shape of the fourth output tensor is [1, NS, E], where K represents the feature dimension; The fourth output tensor is input into the Topk operator, and the first output tensor and the second output tensor are output.

4. The method according to claim 1 or 2, characterized in that Inputting the input sequence, the first output tensor, and the second output tensor into each of the feedforward networks, and outputting a third output tensor from the activated feedforward network, comprises: Each of the feedforward networks determines whether it is an activated feedforward network based on the second output tensor, and when it is determined that it is an activated feedforward network, processes the input sequence based on the first output tensor and outputs a third output tensor.

5. The method according to claim 4, characterized in that The processing of the input sequence based on the first output tensor to output a third output tensor includes: The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor; The dot product result of the fifth output tensor and the first output tensor is output as a third tensor, and both the fifth output tensor and the third output tensor have the same data shape as the input sequence.

6. The method according to claim 5, characterized in that The feedforward network includes a first MMA operator and an activation function, and a second MMA operator. The activated feedforward network performs weighted processing on the input sequence to obtain a fifth output tensor, including: Inputting the input sequence into the first MMA operator to perform a matrix multiplication and accumulation operation, and inputting the operation result into the activation function to obtain a sixth output tensor, wherein the data shape of the input sequence is [1, NS, K], where K represents the data dimension, the data shape of the weight matrix of the first MMA operator is [K, W], and the data shape of the sixth output tensor is [1, NS, W]. The sixth output tensor is input into the second MMA operator for matrix multiplication and accumulation operation, and a fifth output tensor is obtained as an output. The data shape of the weight matrix of the second MMA operator is [W, K].

7. An optimization device for a hybrid expert system, characterized in that: The hybrid expert system includes a routing operator and a plurality of feedforward networks, and the device includes: a first computing module, configured to input an input sequence into the routing operator and output a first output tensor and a second output tensor, wherein the first output tensor represents a weight value of each feedforward network corresponding to each input tag in the input sequence; when the feedforward network is not activated, the weight value of the feedforward network in the first output tensor is 0; and the second output tensor represents the total number of activated feedforward networks and an index of each activated feedforward network; The second computing module is used to input the input sequence, the first output tensor and the second output tensor into each of the feedforward networks, the activated feedforward network outputs a third output tensor, and accumulates and sums the third output tensor to obtain an output result of the hybrid expert system, where the output result is consistent with the data shape of the input sequence.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images