Neural network model processing method, device, equipment and storage medium
By introducing sparse expert layer and routing network into large-scale language models, dynamically selecting the processing path of neural network structures, the problem of increasing model computing overhead is solved, and efficient inference and training capabilities are achieved.
Patent Information
- Application Number
- CN202410917304.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-09
AI Technical Summary
As the model scale and parameter volume of large-scale language models increase, the computational overhead of training and inference increases, it is difficult to deal with diversified downstream tasks in real time.
A multi-layer neural network structure is adopted, including sparse expert layer and routing network, and the target expert network is determined through the gated network and routing network, and dynamically select whether to enter the next layer of neural network structure for processing, reducing unnecessary computing paths.
It improves the inference computing efficiency, reduces the training and inference computing overhead of neural network models, and can respond to diverse downstream tasks in real time.
Smart Images

Figure CN118468929B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the technical fields of artificial intelligence and large language models, and in particular to a neural network model processing method, device, equipment and storage medium. Background Art
[0002] In recent years, large-scale language models (LLMs) have achieved great success in natural language processing tasks. Large-scale language models usually use neural network architectures such as Transformer (a deep neural network structure) to learn powerful language representation and generation capabilities through pre-training on massive text data. Not only in the field of text processing, large-scale language models with Transformer structures are playing an increasingly important role in multimodality, perception and other fields. However, as the model size and number of parameters of large-scale language models increase, the computational overhead of training and inference also increases significantly, making it difficult to respond to diverse downstream task requirements in real time. Summary of the invention
[0003] The purpose of the embodiments of the present invention is to provide a neural network model processing method, device, equipment and storage medium. The specific technical solutions are as follows:
[0004] In a first aspect of the present invention, a neural network model processing method is provided, wherein the neural network model includes a multi-layer neural network structure, wherein the neural network structure includes at least a sparse expert layer and a routing network, wherein the sparse expert layer includes an expert network, and the method includes:
[0005] Receiving a vectorized word unit; the vectorized word unit is obtained by vectorizing a user input;
[0006] Determine a target expert network from the expert networks in the sparse expert layer through the gating network of the neural network structure;
[0007] Inputting the vectorized word unit into the target expert network of the sparse expert layer to calculate and obtain an expert mixed output;
[0008] Determine, by the routing network, the output probability of the expert mixture output entering the neural network structure of the next layer;
[0009] The expert mixed output is input into the neural network structure of the next layer for processing according to the output probability, or the neural network structure of the next layer is skipped.
[0010] Optionally, the sparse expert layer further includes a gating network, and after receiving the vectorized word unit, the method further includes:
[0011] A target expert network is determined from the expert networks in the sparse expert layer through the gating network of the neural network structure.
[0012] Optionally, determining a target expert network from the expert networks in the sparse expert layer by the gating network of the neural network structure comprises:
[0013] Acquire the routing network weight and random noise weight of the expert network of the neural network structure through the gating network of the neural network structure;
[0014] A noise activation value is calculated according to the vectorized word unit, the routing network weight and the random noise weight;
[0015] Calculate an activation noise activation value according to a preset activation function of the neural network structure and the noise activation value;
[0016] The activation noise activation values are sorted, and a target expert network is determined from the expert networks according to the sorting result of the activation noise activation values.
[0017] Optionally, determining a target expert network from the expert network according to the ranking result of the activation noise activation values includes:
[0018] The expert network with the top K rankings of the activation noise activation values is determined as the target expert network; wherein K is a positive integer.
[0019] Optionally, the step of inputting the vectorized word-unit into the target expert network of the sparse expert layer to calculate and obtain an expert mixed output includes:
[0020] Inputting the vectorized word unit into the target expert network for calculation to obtain an initial expert mixed output;
[0021] The expert mixture output is obtained according to the weight corresponding to the initial expert mixture output and the target expert network.
[0022] Optionally, determining, by the routing network, the output probability of the expert mixture output entering the neural network structure of the next layer includes:
[0023] Performing linear processing on the expert mixed output to obtain a target expert mixed output;
[0024] The target expert mixed output is input into the preset activation function of the routing network to obtain the output probability of the neural network structure entering the next layer.
[0025] Optionally, the routing network includes a binary routing network.
[0026] Optionally, the step of inputting the expert mixed output into the neural network structure of a next layer for processing or skipping the neural network structure of a next layer according to the output probability comprises:
[0027] When the output probability is greater than or equal to a preset probability threshold, inputting the expert mixed output into the neural network structure of the next layer for processing;
[0028] When the output probability is less than a preset probability threshold, the expert mixture output is controlled to skip the neural network structure of the next layer.
[0029] Optionally, the step of inputting the expert mixed output into the neural network structure of a next layer for processing or skipping the neural network structure of a next layer according to the output probability comprises:
[0030] According to a preset auxiliary loss function and the output probability, the expert mixed output is input into the neural network structure of the next layer for processing or the neural network structure of the next layer is skipped.
[0031] Optionally, the preset auxiliary loss function includes a preset first auxiliary loss function and a preset second auxiliary loss function, and the inputting the expert mixed output into the neural network structure of the next layer for processing or skipping the neural network structure of the next layer according to the preset auxiliary loss function and the output probability includes:
[0032] Obtaining the number of experts in the expert network of the neural network structure, the number of word units of the vectorized word units stored in each of the expert networks, the total number of word units of the vectorized word units stored in the neural network structure, and the average weight of the vectorized word units stored in each of the expert networks;
[0033] Input the number of experts in the expert network of the neural network structure, the number of word units of the vectorized word units stored in each of the expert networks, the total number of word units of the vectorized word units stored in the neural network structure, and the average weight of the vectorized word units stored in each of the expert networks into a preset first auxiliary loss function to obtain a first auxiliary loss value;
[0034] Obtaining the output probability of the neural network structure of the neural network model;
[0035] Inputting the output probability of the neural network structure of the neural network model into a preset second auxiliary loss function to obtain a second auxiliary loss value;
[0036] According to the output probability, the first auxiliary loss value, and the second auxiliary loss value, the expert mixed output is input into the neural network structure of the next layer for processing or the neural network structure of the next layer is skipped.
[0037] Optionally, the neural network structure includes an attention layer, and the step of inputting the target expert mixed output into a preset activation function of the routing network to obtain an output probability of the neural network structure entering a next layer includes:
[0038] Obtaining a weighted vectorized word-element obtained after the vectorized word-element is processed after the attention layer;
[0039] Performing linear processing on the weighted vectorized word-unit to obtain a target weighted vectorized word-unit;
[0040] The target expert mixed output is input into the preset activation function of the routing network to obtain a first output probability, and the target weighted vectorized word is input into the preset activation function of the routing network to obtain a second output probability;
[0041] The output probability of the neural network structure entering the next layer is calculated according to the first output probability and the second output probability.
[0042] Optionally, the noise activation value is calculated according to the vectorized word unit, the routing network weight and the random noise weight, including:
[0043] Obtain the vectorized word in a specified block range;
[0044] A noise activation value is calculated based on the vectorized word unit, the routing network weight and the random noise weight within a specified block range.
[0045] Optionally, the expert networks respectively have capacity values corresponding to the vectorized word units, and determining a target expert network from the expert networks in the sparse expert layer through the gating network of the neural network structure includes:
[0046] Determining the number of word units of the vectorized word units stored in each of the expert networks in the sparse expert layer through the gating network of the neural network structure;
[0047] A target expert network is determined from the expert network according to the word unit quantity and the capacity value.
[0048] Optionally, determining a target expert network from the expert network according to the word unit quantity and the capacity value includes:
[0049] The expert network in which the number of word units does not exceed the capacity value is determined as a target expert network.
[0050] Optionally, before determining the number of word units of the vectorized word units stored in each of the expert networks in the sparse expert layer through the gating network of the neural network structure, the method further comprises:
[0051] Obtaining a preset capacity factor, a preset number of experts, the number of experts in the expert network of the neural network structure, the number of word units of the vectorized word units stored in each of the expert networks, and the total number of word units of the vectorized word units stored in the neural network structure;
[0052] The capacity value of each of the expert networks is determined based on the preset capacity factor, the preset number of experts, the number of experts in the expert network of the neural network structure, the number of vectorized word units stored in each of the expert networks, and the total number of vectorized word units stored in the neural network structure.
[0053] Optionally, the neural network model includes at least a dense neural network.
[0054] Optionally, the neural network model includes at least a dense neural network.
[0055] In a second aspect of the present invention, a neural network model processing device is provided, wherein the neural network model includes a multi-layer neural network structure, wherein the neural network structure includes at least a sparse expert layer and a routing network, wherein the sparse expert layer includes an expert network, and the device includes:
[0056] A vectorized word unit receiving module, used for receiving vectorized word units; the vectorized word units are obtained by vectorizing user input;
[0057] An expert mixture output calculation module, used for inputting the vectorized word unit into the target expert network of the sparse expert layer to calculate and obtain an expert mixture output;
[0058] An output probability determination module, used to determine the output probability of the expert mixed output entering the neural network structure of the next layer through the routing network;
[0059] A processing module is used to input the expert mixed output into the neural network structure of the next layer for processing or skip the neural network structure of the next layer according to the output probability.
[0060] In a third aspect of the implementation of the present invention, there is also provided a communication device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor;
[0061] The processor is used to read the program in the memory to implement the neural network model processing as described in any one of the first aspects.
[0062] In a fourth aspect of the implementation of the present invention, there is also provided a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, which, when executed on a computer, enables the computer to implement the neural network model processing as described in any one of the first aspects.
[0063] In the fifth aspect of the implementation of the present invention, a computer program product is also provided, including a computer program / instruction, which, when executed by a processor, implements the neural network model processing as described in any one of the first aspects.
[0064] In an embodiment of the present invention, the neural network model includes a multi-layer neural network structure, and the neural network structure includes at least a sparse expert layer and a routing network. The sparse expert layer includes an expert network. When the neural network model receives user input and performs vectorization processing to obtain a vectorized word unit, the vectorized word unit is input into one or more target expert networks in the sparse expert layer to calculate and obtain an expert mixed output. Then, the output probability of the expert mixed output entering the neural network structure of the next layer can be determined through the routing network. Subsequently, the expert mixed output can be input into the neural network structure of the next layer for processing or the neural network structure of the next layer can be skipped according to the output probability. The embodiment of the present invention can improve the efficiency of reasoning calculation. The vectorized word unit can choose whether to process the next layer of the neural network structure of the neural network model according to the result after the routing network. In this way, the vectorized word unit does not need to enter all neural network structures for processing, shortening the path in the depth of the neural network model reasoning, reducing the computational overhead of the training and reasoning of the neural network model, so that it can respond to diverse downstream task requirements in real time. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 is a flowchart of a neural network model processing method provided by an embodiment of the present invention;
[0066] Figure 2 It is a schematic diagram of the Transformer-decode structure of a neural network model;
[0067] Figure 3 It is a schematic diagram of a Transformer-decode structure of a neural network model provided by an embodiment of the present invention;
[0068] Figure 4 is a schematic diagram of a routing network in a neural network model provided by an embodiment of the present invention;
[0069] Figure 5 is a structural schematic diagram of a neural network model processing device provided by an embodiment of the present invention;
[0070] Figure 6It is a schematic diagram of a communication device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0071] In order to enable those skilled in the art to better understand the embodiments of the present invention, some technical terms or nouns involved in the embodiments of the present invention are explained below.
[0072] LLM: Large Language Model, large language model;
[0073] MoE: Mixture of Expert, Mixture of Experts (mixed experts);
[0074] KV cache: A commonly used optimization method for LLM reasoning. It refers to caching the KV matrix values generated by historical tokens in the transformer block to computing devices or storage during the reasoning process.
[0075] Transformer: A deep neural network architecture.
[0076] Attention: attention mechanism or attention module, the core module in transformer;
[0077] Token: refers to the information of a single word understood by the computer, which can be understood as a standard word or Chinese phrase;
[0078] FFN: Feedforward Neural Network, i.e., feedforward neural network or feedforward module, the core module in transformer;
[0079] Router: Routing, in the embodiment of the present invention, refers to a method for selecting an expert network.
[0080] In order to balance model performance and computational efficiency, the MoE (Mixture of Expert) model based on the transformer structure was proposed and received widespread attention. The MoE model consists of multiple expert networks and a gating network. Given an input, the MoE model first uses the gating network to generate a weight for each expert network, and then weights the outputs of these expert networks as the final output. In this way, the MoE model can model the diversity of input data and select relevant expert networks based on input adaptability.
[0081] The core of the MoE network based on the transformer structure includes: (1) a gating network that selects expert networks; (2) multiple different expert networks. The gating network receives token vectors (word vectors\vectorized word units) or activation vectors (activation vectors) and outputs the probability values of selecting different expert networks; multiple different expert networks are used to process different token vectors. When the token vector selects the topK (the first K most probable) expert networks, the token vector is sent to the corresponding expert network (FFN) for calculation to obtain a new vector representation value.
[0082] A dynamic neural network refers to a neural network whose network structure, weights or parameters can be adjusted in real time as time or input data changes. A dynamic neural network can dynamically update its own structure according to different inputs and adapt to different scenarios and tasks, thereby improving the flexibility and performance of the model. The MoE network is actually a type of dynamic neural network. In dynamic networks, sparse routing is an important technology. Sparse routing reduces the amount of network computation and the number of parameters by sparsely sparsely connecting the network, that is, only activating some connections or nodes, thereby improving the efficiency and speed of the network. The Router (gated network) in MoE is a sparse routing technology.
[0083] At present, when the MoE network based on the transformer structure processes user input (usually text), it first tokenizes the input (words) and then enters the Embedding layer. The Embedding layer vectorizes the token and inputs it into the first layer of the transformer structure. The first layer of the transformer first performs self-attention calculation on the vectorized token (with the historical token, that is, the token before the current position in the sequence); after the calculation, each token will pass through the Router layer to determine which expert network will process it; after the expert network is selected, each token enters the expert network for nonlinear change processing (matrix multiplication + activation), and then calculates the weighted sum of the expert network output according to the expert weight of each token given by the Router layer; this completes the calculation of the first layer, and then enters the second layer until the last layer of the model, and performs the same calculation; the final output is obtained and converted into the probability of the token id through the Embedding layer. Among them, the token id refers to the process of mapping the token in the text to the integer identifier.
[0084] (1) Tokens can autonomously select expert networks in each layer, which is helpful for different expert networks to process different types of tokens; however, at the vertical level, tokens cannot actively choose whether to be processed by the transformer layer, so all tokens must be calculated layer by layer.
[0085] (2) Calculating all tokens layer by layer can improve the accuracy of the model, but for most simple tasks or simple inputs, calculating all tokens layer by layer will be a huge waste; that is, the strategy or cost of MoE network reasoning will not change according to the difficulty of the task.
[0086] In response to the above problems, the embodiments of the present invention are innovative based on the MoE-transformer structure. It relates to the fields of artificial intelligence and large language models, and in particular to a MoE model based on a transformer structure and a dynamic neural network technology, which are used to optimize the training and reasoning efficiency of large language models. One of the core concepts of the embodiments of the present invention is to introduce the idea of dynamic neural networks at a deep level and combine it with MoE, thereby improving the training efficiency and reasoning efficiency of large language models involving neural network models.
[0087] Reference Figure 1 , shows a flowchart of a neural network model processing method provided by an embodiment of the present invention, wherein the neural network model includes a multi-layer neural network structure, wherein the neural network structure includes at least a sparse expert layer and a routing network, wherein the sparse expert layer includes an expert network, and the method may include:
[0088] Step 101, receiving a vectorized word unit; the vectorized word unit is obtained by vectorizing a user input.
[0089] In a specific implementation, the neural network model may at least include a mixture of experts model (MoE). Of course, in addition to the mixture of experts model, the embodiment of the present invention is also applicable to other neural network models such as a dense neural network model (dense neural network), and the embodiment of the present invention does not need to be limited to this. For the sake of convenience, the following mainly takes MoE as an example for explanation.
[0090] MoE is based on the transformer-decode (neural network encoder) structure. Figure 2, is a schematic diagram of the Transformer-decode structure of a neural network model. The Transformer-decode structure includes multiple layers (Nx) of transformer blocks (neural network structure / transformer layer), and each transformer block includes a Masked Multi-Head Attention (MHA, Masked Multi-Head Self-Attention) and a Feedforward Layer (FFN, Feedforward Neural Network). The MoE of the embodiment of the present invention replaces the FFN with a sparse expert layer on this basis, as shown in the following figure. Figure 3 , which is a schematic diagram of a Transformer-decode structure of a neural network model provided in an embodiment of the present invention, wherein the sparse expert layer mainly includes two parts: (1) a gating network (Router); (2) a sparse FFN layer (sparse expert layer). Specifically, the Router is used to determine which token vectors are sent to which expert network. For example, in Figure 2 In the example, the token vector "x1" may be sent to the second expert network, while the token vector "x2" may be sent to the first expert network. Of course, sometimes a token vector can be sent to multiple expert networks. The routing method of the token vector is a key point in the use of MoE, because the router (routing network) consists of learned parameters and is pre-trained with the rest of the network. Each FFN layer has the same structure as the feedforward layer in the transformer structure. The MoE optimization scheme for dynamically selecting transformer layers at a deep level proposed in an embodiment of the present invention is to add a binary routing network (gating) behind the FFN layer. The routing network includes activation functions (activation 1 and activation 2). The specific structure is as follows: Figure 4 As shown, in an embodiment of the present invention, after the neural network model receives the token vector (token) converted from the user input in the input embedding layer, the routing network can determine whether the current token vector enters the attention and MoE layers of the next layer, that is, the transformer layer for processing.
[0091] Step 102: input the vectorized word unit into the target expert network of the sparse expert layer to calculate and obtain an expert mixed output.
[0092] In one embodiment of the present invention, the sparse expert layer further includes a gating network, and after receiving the vectorized word, the method further includes:
[0093] A target expert network is determined from the expert networks in the sparse expert layer through the gating network of the neural network structure.
[0094] In an embodiment of the present invention, the sparse expert layer may include multiple expert networks. Since the expert networks are all limited by the capacity value of the token vector, it is necessary to select one or more expert networks to process the token vector from the multiple expert networks. Specifically, after the neural network model receives the token vector, the gating network of the neural network structure can be used to determine in which expert network the token vector is processed, wherein the expert network that processes the token vector is the target expert network, and the target expert network can obtain an expert mixed output after processing the token vector. Among them, the expert mixed output obtained after processing by the target expert network is a new token vector.
[0095] Step 103: Determine the output probability of the expert mixture output entering the neural network structure of the next layer through the routing network.
[0096] Step 104: input the expert mixed output into the neural network structure of the next layer for processing or skip the neural network structure of the next layer according to the output probability.
[0097] In an embodiment of the present invention, the expert mixture output of the target expert network can be sent to the routing network, and then the routing network further obtains the output probability of the expert mixture output entering the transformer layer of the next layer based on the expert mixture output. Based on the output probability, it can be determined whether the expert mixture output is input to the transformer layer of the next layer for processing or the transformer layer of the next layer is skipped. The neural network model of the embodiment of the present invention further increases the number of parameters and expands the depth of the model. By skipping the layer to select or not to perform data processing, the amount of calculation for each token is reduced. The number of transformer layers can be further increased in depth to expand the number of parameters and improve the upper limit of the capacity of neural network models such as LLM. In addition, the embodiment of the present invention uses the next layer selection mechanism of dual routing fusion (gating network and routing network), which is conducive to making full use of the learned knowledge and dependencies, and can improve the accuracy of the neural network model to a certain extent.
[0098] In an embodiment of the present invention, the neural network model includes a multi-layer neural network structure, and the neural network structure includes at least a sparse expert layer and a routing network. The sparse expert layer includes an expert network. When the neural network model receives user input and performs vectorization processing to obtain a vectorized word unit, the vectorized word unit is input into one or more target expert networks in the sparse expert layer to calculate and obtain an expert mixed output. Then, the output probability of the expert mixed output entering the neural network structure of the next layer can be determined through the routing network. Subsequently, the expert mixed output can be input into the neural network structure of the next layer for processing or the neural network structure of the next layer can be skipped according to the output probability. The embodiment of the present invention can improve the efficiency of reasoning calculation. The vectorized word unit can choose whether to process the next layer of the neural network structure of the neural network model according to the result after the routing network. In this way, the vectorized word unit does not need to enter all neural network structures for processing, shortening the path in the depth of the neural network model reasoning, reducing the computational overhead of the training and reasoning of the neural network model, so that it can respond to diverse downstream task requirements in real time.
[0099] In one embodiment of the present invention, determining a target expert network from the expert network in the sparse expert layer through the gating network of the neural network structure includes:
[0100] Acquire the routing network weight and random noise weight of the expert network of the neural network structure through the gating network of the neural network structure;
[0101] A noise activation value is calculated according to the vectorized word unit, the routing network weight and the random noise weight;
[0102] Calculate an activation noise activation value according to a preset activation function of the neural network structure and the noise activation value;
[0103] The activation noise activation values are sorted, and a target expert network is determined from the expert network according to the sorting results of the activation noise activation values. The step of determining the target expert network from the expert network according to the sorting results of the activation noise activation values comprises: determining the expert network with the top K rankings of the sorting results of the activation noise activation values as the target expert network; wherein K is a positive integer.
[0104] In the neural network model of the embodiment of the present invention, the gating network in the neural network model can determine the target expert network by inputting a token vector, thereby selecting whether to enter the next transformer layer processing through the output of the target expert network combination.
[0105] In one example of the present invention, several formulas for determining the target expert network are described as follows:
[0106] Formula 1
[0107] Formula 2
[0108] Formula 3
[0109] In the above formula 1, x represents the token vector (vectorized word), that is, the vector input to each sparse expert layer and routing network, i represents the number of expert networks, is the routing network weight, is the random noise weight, are the logits of the N expert networks (i.e., the logits of the N experts can be used as activations generated by each expert network), is a random value from a normal distribution, is the noise activation value. In the above formula 2, is based on The value of is taken as the largest topK, and the values of other positions are changed to negative infinity. In the above formula 3, Represents random noise weight After that, select the topK expert networks (target expert networks), that is, the weight of each expert network output, K is a positive integer, is the activation function.
[0110] When completing the expert network allocation of the entire sequence, the topK expert networks corresponding to each token are first screened out, and then the specified token is selected according to the token value with the highest weight on each expert network, thereby avoiding the problem of discarding tokens at the end of the sequence; and the embodiment of the present invention selects tokens according to the expert network, which can ensure load balancing between tokens to a certain extent.
[0111] In one embodiment of the present invention, the step of inputting the vectorized word into the target expert network of the sparse expert layer to calculate and obtain an expert mixed output includes:
[0112] Inputting the vectorized word unit into the target expert network for calculation to obtain an initial expert mixed output;
[0113] The expert mixture output is obtained according to the weight corresponding to the initial expert mixture output and the target expert network.
[0114] In one example of the present invention, the formula for calculating the expert mixture output is described as follows:
[0115] Formula 4
[0116] In formula 4, It means inputting x (token vector, same as formula 1) into the expert network numbered i to perform FFN calculation, and then obtaining the final MoE output, i.e., expert mixed output, based on the output of the TopK expert network (i.e., the target expert network) and its corresponding weights.
[0117] In one embodiment of the present invention, determining the output probability of the expert mixture output entering the neural network structure of the next layer through the routing network includes:
[0118] Performing linear processing on the expert mixed output to obtain a target expert mixed output;
[0119] The target expert mixed output is input into the preset activation function of the routing network to obtain the output probability of the neural network structure entering the next layer.
[0120] In one example of the present invention, the formula for calculating the output probability is described as follows:
[0121] Formula 5
[0122] In the above formula 5, the final MoE output (i.e., expert mixture output) is linearly transformed ( ) to obtain the target expert mixed output, and put it into the sigmiod function (activation function) in the routing network to obtain the output probability of entering the next layer of the transformer layer .
[0123] In one embodiment of the present invention, the routing network includes a binary classification routing network, and the step 105 of inputting the expert mixed output into the neural network structure of the next layer for processing or skipping the neural network structure of the next layer according to the output probability includes:
[0124] When the output probability is greater than or equal to a preset probability threshold, inputting the expert mixed output into the neural network structure of the next layer for processing;
[0125] When the output probability is less than a preset probability threshold, the expert mixture output is controlled to skip the neural network structure of the next layer.
[0126] In an embodiment of the present invention, the routing network includes a binary classification routing network, which is located after each transformer layer completes the MoE layer calculation, receives the activation of each token vector, and calculates whether it needs to enter the next transformer layer for processing.
[0127] Specifically, the preset probability threshold may be 0.5. If it is greater than or equal to 0.5, Input to the next transformer layer for conventional processing. If If it is less than 0.5, Input to the next transformer layer in the next layer and The activation function is used for processing, which can shorten the depth path of neural network model reasoning and improve the efficiency of neural network model reasoning.
[0128] In one embodiment of the present invention, the step 104 of inputting the expert mixed output into the neural network structure of the next layer for processing or skipping the neural network structure of the next layer according to the output probability includes:
[0129] According to a preset auxiliary loss function and the output probability, the expert mixed output is input into the neural network structure of the next layer for processing or the neural network structure of the next layer is skipped.
[0130] In the embodiment of the present invention, the main features of the MoE optimization scheme for dynamic selection of transformer layers are: (1) A binary classification routing network, which is located after each transformer layer completes the MoE layer calculation, receives the activation of each token, and calculates whether it needs to enter the next transformer layer for processing. (2) An auxiliary loss function is added so that the token can pass through at least N layers, which helps to obtain relatively accurate output. (3) The coupling processing of binary classification gating and expert gating is used to deduce whether to pass through the transformer calculation of the next layer.
[0131] With respect to the added auxiliary loss function, in one embodiment of the present invention, the step of inputting the expert mixed output into the neural network structure of the next layer for processing or skipping the neural network structure of the next layer according to the preset auxiliary loss function and the output probability includes:
[0132] Obtaining the number of experts in the expert network of the neural network structure, the number of word units of the vectorized word units stored in each of the expert networks, the total number of word units of the vectorized word units stored in the neural network structure, and the average weight of the vectorized word units stored in each of the expert networks;
[0133] Input the number of experts in the expert network of the neural network structure, the number of word units of the vectorized word units stored in each of the expert networks, the total number of word units of the vectorized word units stored in the neural network structure, and the average weight of the vectorized word units stored in each of the expert networks into a preset first auxiliary loss function to obtain a first auxiliary loss value;
[0134] Obtaining the output probability of the neural network structure of the neural network model;
[0135] Inputting the output probability of the neural network structure of the neural network model into a preset second auxiliary loss function to obtain a second auxiliary loss value;
[0136] According to the output probability, the first auxiliary loss value, and the second auxiliary loss value, the expert mixed output is input into the neural network structure of the next layer for processing or the neural network structure of the next layer is skipped.
[0137] In an embodiment of the present invention, a first auxiliary loss function is used to ensure load balancing among experts, and a second auxiliary loss function is used to ensure that the output of the sigmiod activation function on each transformer layer is greater than a preset probability threshold (e.g., 0.5) to ensure a certain quantity.
[0138] In one example of the present invention, the formulas of two auxiliary loss functions for calculating the first auxiliary loss value and the second auxiliary loss value are described as follows:
[0139] Formula 6
[0140] Formula 7
[0141] Among them, the actual meanings of the variables in the above formula 6, i.e. the first auxiliary loss function, are: E: the number of experts; ce: the number of tokens stored in the buffer (storage) of a certain expert network (i.e. the number of word units, generally refers to the number of tokens received when the expert is the first expert); S: the total number of tokens (i.e. the total number of word units); me: the avg(weight) of the token vectors stored in the buffer of a certain expert network on the expert (i.e. the average weight, the token consideration range is also those tokens that use the expert network as the first expert network). The actual meanings of the variables in the above formula 7, i.e. the second auxiliary loss function, are: output represents the result of the sigmiod activation function of the token in each layer. By optimizing the loss (loss) of formula 2 to the minimum, it can be encouraged to keep the number of skipped layers roughly consistent with the number of calculated layers. If more layers are allowed to calculate, 1 will be slightly increased. Conversely, if fewer layers are allowed, 1 will be slightly decreased.
[0142] In an embodiment of the present invention, after obtaining the first auxiliary loss value and the second auxiliary loss value through the first auxiliary loss function and the second auxiliary loss function, it can be determined based on the output probability, and the first auxiliary loss value and the second auxiliary loss value whether the expert mixture output enters the next transformer layer for processing or skips the next transformer layer.
[0143] The auxiliary loss functions designed in the embodiment of the present invention, namely the first auxiliary loss function and the second auxiliary loss function, help to balance the range of the number of layers selected for each token, neither too many nor too few, and can dynamically select the network parameters required according to the token, so as to obtain the output of neural network models such as LLM more efficiently and stably.
[0144] In one embodiment of the present invention, the neural network structure includes an attention layer, and the output probability of the neural network structure entering the next layer by inputting the target expert mixed output into a preset activation function of the routing network includes:
[0145] Obtaining a weighted vectorized word-element obtained after the vectorized word-element is processed after the attention layer;
[0146] Performing linear processing on the weighted vectorized word-unit to obtain a target weighted vectorized word-unit;
[0147] The target expert mixed output is input into the preset activation function of the routing network to obtain a first output probability, and the target weighted vectorized word is input into the preset activation function of the routing network to obtain a second output probability;
[0148] The output probability of the neural network structure entering the next layer is calculated according to the first output probability and the second output probability.
[0149] In a specific implementation, the two gating functions (i.e., the gating network and the routing network) in the same transformer layer correspond to the selection of the expert network and the selection of whether to process the next layer, respectively. The above formulas 3 and 5 both perform nonlinear processing on the token, that is, the corresponding logits are obtained through a matrix calculation. The difference is that formula 3 is for the output of the attention layer (attention layer) of the neural network mode, and formula 5 is for the fusion output of the MoE-FFN layer. The embodiment of the present invention takes into account whether the next layer of transformer is actually processed, and important information can also be obtained from the output of the attention layer. Therefore, in each layer, if the token is to be processed by attention+MoE, the MoE-router can also perform the output selected by the next layer. This output is fused with the output generated by formula 5 to determine whether to process the next layer.
[0150] In one example of the present invention, another formula for calculating the output probability is described as follows:
[0151]
[0152]
[0153] Among them, the actual meaning of each variable in formula 8 is: After the representative token is output by the attention layer, it is processed by nonlinear transformation, see Formula 1; the sigmoid activation function obtains the binary classification result (output probability), and the meaning of the result is consistent with Formula 5, that is, the embodiment of the present invention can use the above Formula 9 to replace the above Formula 5, and can also determine the output probability, and then can achieve the purpose of determining whether to perform the next layer processing based on the output probability.
[0154] In one embodiment of the present invention, the step of calculating the noise activation value according to the vectorized word unit, the routing network weight and the random noise weight includes:
[0155] Obtain the vectorized word in a specified block range;
[0156] A noise activation value is calculated based on the vectorized word unit, the routing network weight and the random noise weight within a specified block range.
[0157] In the embodiment of the present invention, other methods may also be used to calculate the noise activation value, namely: Specifically, in one example of the present invention, the improved The formula is as follows:
[0158]
[0159] Formula 10
[0160] in, To specify the block range, in the above formula 10 The calculation is not only related to the current x value (token), but also to the logits corresponding to each expert network based on the x in the previous chunksize range. The logic of this is that each token does not exist in isolation, and which expert network processes the token is determined by the previous tokens.
[0161] In one embodiment of the present invention, the expert networks respectively have capacity values corresponding to vectorized word units, and determining a target expert network from the expert networks in the sparse expert layer by the gating network of the neural network structure includes:
[0162] Determining the number of word units of the vectorized word units stored in each of the expert networks in the sparse expert layer through the gating network of the neural network structure;
[0163] The expert network in which the number of word units does not exceed the capacity value is determined as a target expert network.
[0164] In the specific implementation, the probability of sending a token to an expert network (expert) cannot be controlled. In actual operation, some expert networks may receive more tokens, while some expert networks may receive fewer tokens. This phenomenon is called uneven expert network load. Even if the above-mentioned chunkwise level (chunksize is the block range that defines the data block, and chunkwise is based on these data blocks) routing mechanism is used, this expert network load imbalance phenomenon may still occur. Therefore, in the embodiment of the present invention, the token capacity value can be set for each expert network to avoid a large number of tokens at the end of the sequence being discarded.
[0165] In the embodiment of the present invention, a capacity value is set for each expert network. If the number of tokens (number of word units) currently received by the expert network exceeds the capacity value of the expert network, the expert network will no longer receive tokens, and the extra tokens are overflow.
[0166] In one embodiment of the present invention, before determining the number of word units of the vectorized word units stored in each of the expert networks in the sparse expert layer through the gating network of the neural network structure, the method further includes:
[0167] Obtaining a preset capacity factor, a preset number of experts, the number of experts in the expert network of the neural network structure, the number of word units of the vectorized word units stored in each of the expert networks, and the total number of word units of the vectorized word units stored in the neural network structure;
[0168] The capacity value of each of the expert networks is determined based on the preset capacity factor, the preset number of experts, the number of experts in the expert network of the neural network structure, the number of vectorized word units stored in each of the expert networks, and the total number of vectorized word units stored in the neural network structure.
[0169] In the embodiment of the present invention, the capacity value of each expert network is shown in the following formula: S represents the total number of tokens, E represents the number of experts, K represents the number of expert networks that process each token, Represents the capacity factor. For example, if there are 8 tokens and 4 expert networks, in the case of ideal load balancing, each expert network should receive 8 / 4 = 2 tokens. Considering that the top2 expert network is used here, the capacity factor is 1. is 2, so the upper limit of tokens received by each expert network is preferably (8 / 4)*2 = 4.
[0170] In an example of the present invention, the formula for the capacity value of the expert network is as follows:
[0171] Formula 11
[0172] Specifically, when a token is assigned to the corresponding expert network, it has exceeded the capacity of the expert network, and an overflow will occur. When an overflow occurs, not all tokens will be processed normally by the expert network. This overflow operation is called drop tokens. Usually, the number of discarded tokens will be at the end of the sequence. This is an asymmetric information loss (the tokens at the back end of the sequence have not been parsed by any expert network). Of course, this problem can be alleviated by adjusting the capacity of the expert network, but excessive adjustment of the capacity of the expert network will cause serious matrix sparsity problems. Therefore, the embodiment of the present invention can also choose to use the first auxiliary loss function to ensure load balancing between expert networks according to actual needs.
[0173] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0174] Reference Figure 5 , shows a structural block diagram of a neural network model processing device provided by an embodiment of the present invention, wherein the neural network model includes a multi-layer neural network structure, wherein the neural network structure includes at least a sparse expert layer and a routing network, wherein the sparse expert layer includes an expert network, and the device includes:
[0175] A vectorized word unit receiving module 501 is used to receive vectorized word units; the vectorized word units are obtained by vectorizing user input;
[0176] An expert mixture output calculation module 502 is used to input the vectorized word unit into the target expert network of the sparse expert layer to calculate and obtain an expert mixture output;
[0177] An output probability determination module 503, used to determine the output probability of the expert mixed output entering the neural network structure of the next layer through the routing network;
[0178] The processing module 504 is used to input the expert mixed output into the neural network structure of the next layer for processing or skip the neural network structure of the next layer according to the output probability.
[0179] In one embodiment of the present invention, the sparse expert layer further includes a gating network, and the device further includes: a target expert network determination module, which is used to:
[0180] A target expert network is determined from the expert networks in the sparse expert layer through the gating network of the neural network structure.
[0181] In one embodiment of the present invention, the target expert network determination module is used to:
[0182] Acquire the routing network weight and random noise weight of the expert network of the neural network structure through the gating network of the neural network structure;
[0183] A noise activation value is calculated according to the vectorized word unit, the routing network weight and the random noise weight;
[0184] Calculate an activation noise activation value according to a preset activation function of the neural network structure and the noise activation value;
[0185] The activation noise activation values are sorted, and a target expert network is determined from the expert networks according to the sorting result of the activation noise activation values.
[0186] In one embodiment of the present invention, the target expert network determination module is used to:
[0187] The expert network with the top K rankings of the activation noise activation values is determined as the target expert network; wherein K is a positive integer.
[0188] In one embodiment of the present invention, the expert mixture output calculation module 502 is used to:
[0189] Inputting the vectorized word unit into the target expert network for calculation to obtain an initial expert mixed output;
[0190] The expert mixture output is obtained according to the weight corresponding to the initial expert mixture output and the target expert network.
[0191] In one embodiment of the present invention, the output probability determination module 503 is used to:
[0192] Performing linear processing on the expert mixed output to obtain a target expert mixed output;
[0193] The target expert mixed output is input into the preset activation function of the routing network to obtain the output probability of the neural network structure entering the next layer.
[0194] In an embodiment of the present invention, the routing network includes a binary routing network, and the processing module 504 is used to:
[0195] When the output probability is greater than or equal to a preset probability threshold, inputting the expert mixed output into the neural network structure of the next layer for processing;
[0196] When the output probability is less than a preset probability threshold, the expert mixture output is controlled to skip the neural network structure of the next layer.
[0197] In one embodiment of the present invention, the processing module 504 is used to: input the expert mixed output into the neural network structure of the next layer for processing or skip the neural network structure of the next layer according to a preset auxiliary loss function and the output probability.
[0198] In an embodiment of the present invention, the preset auxiliary loss function includes a preset first auxiliary loss function and a preset second auxiliary loss function, and the processing module 504 is used to:
[0199] Obtaining the number of experts in the expert network of the neural network structure, the number of word units of the vectorized word units stored in each of the expert networks, the total number of word units of the vectorized word units stored in the neural network structure, and the average weight of the vectorized word units stored in each of the expert networks;
[0200] Input the number of experts in the expert network of the neural network structure, the number of word units of the vectorized word units stored in each of the expert networks, the total number of word units of the vectorized word units stored in the neural network structure, and the average weight of the vectorized word units stored in each of the expert networks into a preset first auxiliary loss function to obtain a first auxiliary loss value;
[0201] Obtaining the output probability of the neural network structure of the neural network model;
[0202] Inputting the output probability of the neural network structure of the neural network model into a preset second auxiliary loss function to obtain a second auxiliary loss value;
[0203] According to the output probability, the first auxiliary loss value, and the second auxiliary loss value, the expert mixed output is input into the neural network structure of the next layer for processing or the neural network structure of the next layer is skipped.
[0204] In one embodiment of the present invention, the neural network structure includes an attention layer, and the output probability determination module 504 is used to:
[0205] Obtaining a weighted vectorized word-element obtained after the vectorized word-element is processed after the attention layer;
[0206] Performing linear processing on the weighted vectorized word-unit to obtain a target weighted vectorized word-unit;
[0207] The target expert mixed output is input into the preset activation function of the routing network to obtain a first output probability, and the target weighted vectorized word is input into the preset activation function of the routing network to obtain a second output probability;
[0208] The output probability of the neural network structure entering the next layer is calculated according to the first output probability and the second output probability.
[0209] In one embodiment of the present invention, the target expert network determination module is used to:
[0210] Obtain the vectorized word in a specified block range;
[0211] A noise activation value is calculated based on the vectorized word unit, the routing network weight and the random noise weight within a specified block range.
[0212] In one embodiment of the present invention, the expert networks respectively have capacity values corresponding to the vectorized word units, and the target expert network determination module is used to:
[0213] Determining the number of word units of the vectorized word units stored in each of the expert networks in the sparse expert layer through the gating network of the neural network structure;
[0214] The expert network in which the number of word units does not exceed the capacity value is determined as a target expert network.
[0215] In one embodiment of the present invention, the device further includes a capacity value determination module, which is used to:
[0216] Obtaining a preset capacity factor, a preset number of experts, the number of experts in the expert network of the neural network structure, the number of word units of the vectorized word units stored in each of the expert networks, and the total number of word units of the vectorized word units stored in the neural network structure;
[0217] The capacity value of each of the expert networks is determined based on the preset capacity factor, the preset number of experts, the number of experts in the expert network of the neural network structure, the number of vectorized word units stored in each of the expert networks, and the total number of vectorized word units stored in the neural network structure.
[0218] In one embodiment of the present invention, the neural network model at least includes an expert mixture model.
[0219] In one embodiment of the present invention, a dense neural network.
[0220] The embodiment of the present invention also provides a communication device, such as Figure 6 As shown, it includes a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104.
[0221] Memory 1103, used for storing computer programs;
[0222] The processor 1101, when used to execute the program stored in the memory 1103, can implement the following steps:
[0223] Receiving a vectorized word unit; the vectorized word unit is obtained by vectorizing a user input;
[0224] Determine a target expert network from the expert networks in the sparse expert layer through the gating network of the neural network structure;
[0225] Inputting the vectorized word unit into the target expert network of the sparse expert layer to calculate and obtain an expert mixed output;
[0226] Determine, by the routing network, the output probability of the expert mixture output entering the neural network structure of the next layer;
[0227] The expert mixed output is input into the neural network structure of the next layer for processing according to the output probability, or the neural network structure of the next layer is skipped.
[0228] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits together, which are all well known in the art, so this article will not further describe them. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor can be transmitted on a wired medium or on a wireless medium through an antenna. Further, the antenna also receives data and transmits the data to the processor. The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory can be used to store data used by the processor when performing operations.
[0229] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0230] The communication interface is used for communication between the above terminal and other devices.
[0231] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0232] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0233] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the neural network model processing method described in any of the above embodiments.
[0234] In another embodiment provided by the present invention, a computer program product containing instructions is also provided, which, when executed on a computer, enables the computer to execute the neural network model processing method described in any of the above embodiments.
[0235] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0236] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0237] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
Claims
1. A neural network model processing method, characterized in that: The neural network model includes a multi-layer neural network structure, the neural network structure includes at least a sparse expert layer and a routing network, the sparse expert layer includes an expert network and a gating network, and the method includes: Receiving a vectorized word unit; the vectorized word unit is obtained by vectorizing a user input; Determine a target expert network from the expert networks in the sparse expert layer through the gating network of the neural network structure; Inputting the vectorized word unit into the target expert network of the sparse expert layer to calculate and obtain an expert mixed output; Determine, by the routing network, the output probability of the expert mixture output entering the neural network structure of the next layer; Inputting the expert mixed output into the neural network structure of the next layer for processing or skipping the neural network structure of the next layer according to the output probability; Wherein, determining the target expert network from the expert network in the sparse expert layer through the gating network of the neural network structure includes: Acquire the routing network weight and random noise weight of the expert network of the neural network structure through the gating network of the neural network structure; A noise activation value is calculated according to the vectorized word unit, the routing network weight and the random noise weight; Calculate an activation noise activation value according to a preset activation function of the neural network structure and the noise activation value; sorting the activation noise activation values, and determining a target expert network from the expert network according to the sorting result of the activation noise activation values; The step of determining the output probability of the expert mixture output entering the neural network structure of the next layer through the routing network includes: Performing linear processing on the expert mixed output to obtain a target expert mixed output; Inputting the target expert mixed output into the preset activation function of the routing network to obtain the output probability of the neural network structure entering the next layer; The step of inputting the expert mixed output into the neural network structure of the next layer for processing or skipping the neural network structure of the next layer according to the output probability comprises: According to a preset auxiliary loss function and the output probability, the expert mixed output is input into the neural network structure of the next layer for processing or the neural network structure of the next layer is skipped.
2. The method according to claim 1, characterized in that The step of determining a target expert network from the expert network according to the ranking result of the activation noise activation values comprises: The expert network whose activation noise activation values are ranked in the top K positions is determined as the target expert network; wherein K is a positive integer.
3. The method according to claim 1, characterized in that The step of inputting the vectorized word unit into the target expert network of the sparse expert layer to calculate and obtain an expert mixed output includes: Inputting the vectorized word unit into the target expert network for calculation to obtain an initial expert mixed output; The expert mixture output is obtained according to the weight corresponding to the initial expert mixture output and the target expert network.
4. The method according to claim 1, characterized in that: The routing network includes a binary routing network.
5. The method according to claim 4, characterized in that The step of inputting the expert mixed output into the neural network structure of the next layer for processing or skipping the neural network structure of the next layer according to the output probability comprises: When the output probability is greater than or equal to a preset probability threshold, inputting the expert mixed output into the neural network structure of the next layer for processing; When the output probability is less than a preset probability threshold, the expert mixture output is controlled to skip the neural network structure of the next layer.
6. The method according to claim 1, characterized in that The preset auxiliary loss function includes a preset first auxiliary loss function and a preset second auxiliary loss function, and the step of inputting the expert mixed output into the neural network structure of the next layer for processing or skipping the neural network structure of the next layer according to the preset auxiliary loss function and the output probability includes: Obtaining the number of experts in the expert network of the neural network structure, the number of word units of the vectorized word units stored in each of the expert networks, the total number of word units of the vectorized word units stored in the neural network structure, and the average weight of the vectorized word units stored in each of the expert networks; Input the number of experts in the expert network of the neural network structure, the number of word units of the vectorized word units stored in each of the expert networks, the total number of word units of the vectorized word units stored in the neural network structure, and the average weight of the vectorized word units stored in each of the expert networks into a preset first auxiliary loss function to obtain a first auxiliary loss value; Obtaining the output probability of the neural network structure of the neural network model; Inputting the output probability of the neural network structure of the neural network model into a preset second auxiliary loss function to obtain a second auxiliary loss value; According to the output probability, the first auxiliary loss value, and the second auxiliary loss value, the expert mixed output is input into the neural network structure of the next layer for processing or the neural network structure of the next layer is skipped.
7. The method according to claim 1, characterized in that The neural network structure includes an attention layer, and the output probability of the neural network structure entering the next layer by inputting the target expert mixed output into the preset activation function of the routing network includes: Obtaining a weighted vectorized word-element obtained after the vectorized word-element is processed after the attention layer; Performing linear processing on the weighted vectorized word-unit to obtain a target weighted vectorized word-unit; The target expert mixed output is input into the preset activation function of the routing network to obtain a first output probability, and the target weighted vectorized word is input into the preset activation function of the routing network to obtain a second output probability; The output probability of the neural network structure entering the next layer is calculated according to the first output probability and the second output probability.
8. The method according to claim 1, characterized in that The calculating the noise activation value according to the vectorized word unit, the routing network weight and the random noise weight includes: Obtain the vectorized word in a specified block range; A noise activation value is calculated based on the vectorized word unit, the routing network weight and the random noise weight within a specified block range.
9. The method according to claim 1, characterized in that: The expert networks respectively have capacity values corresponding to the vectorized word units, and determining a target expert network from the expert networks in the sparse expert layer by the gating network of the neural network structure includes: Determining the number of word units of the vectorized word units stored in each of the expert networks in the sparse expert layer through the gating network of the neural network structure; A target expert network is determined from the expert network according to the word unit quantity and the capacity value.
10. The method according to claim 9, characterized in that The step of determining a target expert network from the expert network according to the word unit quantity and the capacity value includes: The expert network in which the number of word units does not exceed the capacity value is determined as a target expert network.
11. The method according to claim 9, characterized in that Before determining the number of word units of the vectorized word units stored in each of the expert networks in the sparse expert layer through the gating network of the neural network structure, the method further includes: Obtaining a preset capacity factor, a preset number of experts, the number of experts in the expert network of the neural network structure, the number of word units of the vectorized word units stored in each of the expert networks, and the total number of word units of the vectorized word units stored in the neural network structure; The capacity value of each of the expert networks is determined based on the preset capacity factor, the preset number of experts, the number of experts in the expert network of the neural network structure, the number of vectorized word units stored in each of the expert networks, and the total number of vectorized word units stored in the neural network structure.
12. The method according to claim 1, characterized in that The neural network model includes a mixture of experts model.
13. The method according to claim 1, characterized in that The neural network model also includes a dense neural network.
14. A neural network model processing device, characterized in that: The neural network model includes a multi-layer neural network structure, the neural network structure includes at least a sparse expert layer and a routing network, the sparse expert layer includes an expert network and a gating network, and the device includes: A vectorized word unit receiving module, used for receiving vectorized word units; the vectorized word units are obtained by vectorizing user input; a target expert network determination module, configured to determine a target expert network from the expert networks in the sparse expert layer through the gating network of the neural network structure; An expert mixture output calculation module, used for inputting the vectorized word unit into the target expert network of the sparse expert layer to calculate and obtain an expert mixture output; An output probability determination module, used to determine the output probability of the expert mixed output entering the neural network structure of the next layer through the routing network; A processing module, used for inputting the expert mixed output into the neural network structure of the next layer for processing or skipping the neural network structure of the next layer according to the output probability; Wherein, the target expert network determination module is used to: Acquire the routing network weight and random noise weight of the expert network of the neural network structure through the gating network of the neural network structure; A noise activation value is calculated according to the vectorized word unit, the routing network weight and the random noise weight; Calculate an activation noise activation value according to a preset activation function of the neural network structure and the noise activation value; sorting the activation noise activation values, and determining a target expert network from the expert network according to the sorting result of the activation noise activation values; The output probability determination module is used to: Performing linear processing on the expert mixed output to obtain a target expert mixed output; Inputting the target expert mixed output into the preset activation function of the routing network to obtain the output probability of the neural network structure entering the next layer; The processing module is used to: input the expert mixed output into the neural network structure of the next layer for processing or skip the neural network structure of the next layer according to a preset auxiliary loss function and the output probability.
15. A communication device, characterized in that: include: A transceiver, a memory, a processor, and a program stored on the memory and executable on the processor; The processor is used to read the program in the memory to implement the neural network model processing method as described in any one of claims 1-13.
16. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, it implements the neural network model processing method as described in any one of claims 1-13.
17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by the processor, the neural network model processing method as described in any one of claims 1-13 is implemented.
