A data processing method and apparatus

Abstract

A data processing method comprises: performing word segmentation on an input sequence, and mapping a word segmentation result to a first vocabulary index; and processing the first vocabulary index through a first neural network model to obtain an inference result; wherein, in a first hybrid expert layer of the first neural network model, based on the first vocabulary index, a query is performed in a first lookup table stored in a memory to obtain an output of each expert network in the first hybrid expert layer, the first lookup table comprises a mapping relationship between an index in a vocabulary and a result of performing inference on an embedding representation of the index by each expert network in the first hybrid expert layer, and the vocabulary is a vocabulary set used by the first neural network model. The method can realize that expert networks in a hybrid expert layer do not need to be loaded in an inference process, reduces memory occupation and communication volume, eliminates the calculation of the expert networks in the hybrid expert layer, and improves inference efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a data processing method and apparatus. Background Technology

[0002] With sufficient training data, the performance of large language models (LLMs) typically improves with increasing model size. However, increasing model size can also slow down inference, impacting user experience. To address this challenge, an increasing number of LLM architectures are adopting a mixture of experts (MoE) architecture. In the MoE architecture, multiple feed-forward networks (FFNs) are introduced as "expert networks," and a "gating network" is used to determine which subset of experts to activate in each inference iteration, rather than activating the entire model. This design allows the model to significantly reduce computational costs while maintaining a large number of parameters.

[0003] While the MoE architecture effectively reduces computational burden, the total number of model parameters remains unchanged. GPU memory is a critical resource for storing activation parameters and intermediate computation results during model inference, but its capacity is finite. This means that the GPU memory requirements during inference are still unsustainable. For example, although the Mixtral-8x7B model activates only 13 billion parameters in a single inference iteration, its total number of parameters is as high as 46 billion, making it impossible to fully load the model at FP16 precision on a single 80GB A100 graphics processing unit (GPU).

[0004] In related technologies, expert parameters are offloaded to larger storage devices (such as CPU memory, disk, or cloud storage) and dynamically loaded into GPU memory during each inference step to reduce GPU memory usage. This mechanism enables models to achieve more efficient inference with limited hardware resources. However, while this storage and loading mechanism effectively reduces GPU memory requirements, it also introduces additional complexity and latency. Therefore, achieving low GPU memory usage and low communication overhead during inference has become a pressing technical problem to be solved. Summary of the Invention

[0005] This application provides a data processing method, apparatus, computing device cluster, computer-readable storage medium, and computer program product containing instructions, which can achieve low video memory usage and low communication overhead during inference.

[0006] In a first aspect, this application provides a data processing method, comprising: segmenting an input sequence into words and mapping the segmentation results to a first vocabulary index; processing the first vocabulary index through a first neural network model to obtain an inference result; wherein, in a first hybrid expert layer of the first neural network model, based on the first vocabulary index, a query is performed in a first lookup table stored in a memory to obtain the outputs of each expert network in the first hybrid expert layer, the first lookup table including the mapping relationship between the index in the vocabulary and the inference result of the embedding representation of the index by each expert network in the first hybrid expert layer, and the vocabulary is the vocabulary set used by the first neural network model.

[0007] In this way, by implementing the functionality of the expert networks in the first hybrid expert layer of the first neural network model through a first lookup table, the outputs of each expert network in the first hybrid expert layer can be directly retrieved from the first lookup table during inference, thus eliminating the need to load the expert networks in the first hybrid expert layer during inference. Since the expert networks in the first hybrid expert layer are not loaded during inference, the memory usage and communication overhead caused by loading expert networks are reduced. Furthermore, because the outputs of the expert networks in the first hybrid expert layer are retrieved from the lookup table, without the need for expert network computation during inference, expert network computation is eliminated, improving inference efficiency.

[0008] In one possible implementation, the first neural network model further includes a second hybrid expert layer. The memory stores a second lookup table, which includes a mapping between indices in the vocabulary and the inference results of each expert network in the second hybrid expert layer based on the embedded representations of those indices. This facilitates lookup table retrieval and reduces query complexity. Additionally, it allows for partitioned storage of the lookup table, improving storage convenience.

[0009] In one possible implementation, after querying a first lookup table stored in memory based on a first vocabulary index to obtain the outputs of each expert network in the first hybrid expert layer, the method further includes: performing a weighted calculation on the outputs of each expert network in the first hybrid expert layer to obtain a first output; and calculating the output of the first hybrid expert layer based on the first output and the input of the first hybrid expert layer. In this way, reasoning can be implemented in the first hybrid expert layer, and the outputs of all expert networks in the first hybrid expert layer can be used during the reasoning process, which is equivalent to all expert networks in the first hybrid expert layer participating in the calculation, thereby improving the accuracy of the reasoning.

[0010] In one possible implementation, before performing a weighted calculation on the outputs of each expert network in the first hybrid expert layer to obtain the first output, the method further includes: processing the input of the first hybrid expert layer based on the gating network in the first hybrid expert layer to obtain the weights of each expert network in the first hybrid expert layer, wherein the weights of each expert network in the first hybrid expert layer are used to perform a weighted calculation on the outputs of each expert network in the first hybrid expert layer. This allows the weights of each expert network in the first hybrid expert layer to be calculated.

[0011] In one possible implementation, the output of the first hybrid expert layer is calculated based on the first output and the input of the first hybrid expert layer, including: processing the input of the first hybrid expert layer based on the shared expert network in the first hybrid expert layer to obtain the output of the shared expert network; and calculating the output of the first hybrid expert layer based on the first output, the output of the shared expert network, and the input of the first hybrid expert layer. In this way, reasoning can be implemented in the first hybrid expert layer.

[0012] In one possible implementation, each expert network in the first hybrid expert layer is logically defined and implemented through a first lookup table.

[0013] In one possible implementation, the method further includes: segmenting the input sample into words and mapping the segmentation results to a second vocabulary index; processing the second vocabulary index through a second neural network model to obtain predicted values ​​related to the input sample, wherein, in the second neural network model, the embedding representation of the second vocabulary index is input to each expert network in a third hybrid expert layer to obtain the output of each expert network in the third hybrid expert layer, where the third hybrid expert layer is any hybrid expert layer in the second neural network model; updating the parameters of the second neural network model based on the difference between the predicted values ​​and the ground truth values ​​related to the input sample; if the parameters of the second neural network model meet the requirements, inputting the embedding representation of the vocabulary index into each expert network in the third hybrid expert layer to obtain the first result of the inference performed by each expert network in the third hybrid expert layer on the embedding representation of the index; and generating a lookup table related to the third hybrid expert layer based on the index in the vocabulary and the first result, and deleting the expert networks in the third hybrid expert layer to obtain the first neural network model. In this way, the desired neural network model can be obtained by first training a neural network model, then parameterizing the expert networks in the neural network model, and finally deleting the expert networks from the trained neural network model.

[0014] Secondly, this application provides a data processing apparatus, including an acquisition module and a processing module. The acquisition module is used to acquire an input sequence. The processing module is used to segment the input sequence into words and map the segmentation results to a first vocabulary index. The processing module is further used to process the first vocabulary index through a first neural network model to obtain inference results. Specifically, in the first hybrid expert layer of the first neural network model, a query is performed based on the first vocabulary index in a first lookup table stored in memory to obtain the outputs of each expert network in the first hybrid expert layer. The first lookup table includes the mapping relationship between the index in the vocabulary and the inference results of the embedding representation of the index by each expert network in the first hybrid expert layer. The vocabulary is the set of words used by the first neural network model.

[0015] In one possible implementation, the first neural network model further includes a second hybrid expert layer; wherein a second lookup table is stored in the memory, the second lookup table including the mapping relationship between the indexes in the vocabulary and the results of reasoning on the embedded representations of the indexes by each expert network in the second hybrid expert layer.

[0016] In one possible implementation, after the processing module performs a lookup in the first lookup table stored in the memory based on the first vocabulary index to obtain the outputs of each expert network in the first hybrid expert layer, it is further configured to: perform a weighted calculation on the outputs of each expert network in the first hybrid expert layer to obtain a first output; and calculate the output of the first hybrid expert layer based on the first output and the input of the first hybrid expert layer.

[0017] In one possible implementation, before the processing module performs a weighted calculation on the outputs of each expert network in the first hybrid expert layer to obtain the first output, it is further configured to: process the input of the first hybrid expert layer based on the gating network in the first hybrid expert layer to obtain the weights of each expert network in the first hybrid expert layer, wherein the weights of each expert network in the first hybrid expert layer are used to perform a weighted calculation on the outputs of each expert network in the first hybrid expert layer.

[0018] In one possible implementation, when the processing module calculates the output of the first hybrid expert layer based on the first output and the input of the first hybrid expert layer, it specifically performs the following: processes the input of the first hybrid expert layer based on the shared expert network in the first hybrid expert layer to obtain the output of the shared expert network; and calculates the output of the first hybrid expert layer based on the first output, the output of the shared expert network, and the input of the first hybrid expert layer.

[0019] In one possible implementation, each expert network in the first hybrid expert layer is logically defined and implemented through a first lookup table.

[0020] In one possible implementation, the processing module is further configured to: segment the input sample into words and map the segmentation results to a second vocabulary index; process the second vocabulary index through a second neural network model to obtain predicted values ​​related to the input sample, wherein, in the second neural network model, the embedding representation of the second vocabulary index is input to each expert network in the third hybrid expert layer to obtain the output of each expert network in the third hybrid expert layer, where the third hybrid expert layer is any hybrid expert layer in the second neural network model; update the parameters of the second neural network model based on the difference between the predicted values ​​and the true values ​​related to the input sample; if the parameters of the second neural network model meet the requirements, input the embedding representation of the vocabulary index into each expert network in the third hybrid expert layer to obtain the first result of the inference performed by each expert network in the third hybrid expert layer on the embedding representation of the index; and, based on the index in the vocabulary and the first result, generate a lookup table related to the third hybrid expert layer and delete the expert networks in the third hybrid expert layer to obtain the first neural network model.

[0021] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0022] Fourthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the computing device cluster may include one or more computing devices.

[0023] Fifthly, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0024] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a MoE architecture provided in an embodiment of this application;

[0026] Figure 2 This is a schematic diagram of a MoLE architecture provided in an embodiment of this application;

[0027] Figure 3 This is a schematic diagram illustrating a training method for the MoLE layer in a MoLE architecture provided in an embodiment of this application;

[0028] Figure 4 This is a schematic diagram illustrating the training steps of a MoLE architecture model provided in an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of a data processing method provided in an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0031] Figure 7 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0032] Figure 8 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0033] Figure 9 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0034] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0035] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0036] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0037] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0038] For example, the basic idea of ​​the MoE architecture is to replace each FFN layer in the traditional Transformer model with a Hybrid Expert layer, namely the MoE layer. Figure 1 As shown, a neural network model typically consists of at least one MoE layer. A MoE layer usually comprises two key components: expert networks and a gating / router network. There can be, but is not limited to, multiple expert networks. These expert networks can have the same or different structures, and they are trained to handle different types of tasks or data. Each expert network is an independent sub-network (usually an FFN), and only a portion of the expert networks are activated to participate in the processing during actual computation. By having multiple expert networks share the computation of different subsets of data, the model can achieve the representational power of a large number of parameters with lower computational overhead during pre-training. The gating network is responsible for dynamically selecting which expert networks to activate based on the features of the input data. The gating network generally uses a simple feedforward network with softmax to compute the weights of each expert network. After training, the gating network gradually learns to route similar inputs to the better-performing expert network.

[0039] exist Figure 1 In neural networks, when processing data, the input sequence is first segmented into a series of strings (tokens). Then, these tokens are further converted into a numerical form that the model can understand, namely a vocabulary index (or "identification ID"). This process is called "vocabulary mapping." The vocabulary is a predefined mapping table of strings to integers that contains all the tokens the model can recognize. For example, a vocabulary index can be used to represent the index number of a token within the vocabulary.

[0040] After the vocabulary mapping is completed, the resulting IDs are then fed into the embedding layer, where they are converted into vector representations (or "embedded representations"). The purpose of the embedding layer is to transform discrete IDs into continuous vectors that capture semantic information. After embedding, the resulting vector representations are fed into the first MoE layer, and the data is processed sequentially according to the connection order of the MoE layers until the inference result is obtained.

[0041] Each MoE layer typically includes an attention layer. The attention mechanism allows the model to focus on the most important parts of the input data to the MoE layer. This is very useful for understanding long sequences or processing complex tasks. After passing through the attention layer, the output is fed into a gating network. The gating network uses some form of gating mechanism or probability distribution to determine the weights of each expert network. These weights represent the contribution of each expert network to processing the current input data. These weights determine which expert networks should process the data, for example, by selecting... Figure 1 The diagram shows FFN2 and FFN3. After the gating network completes the expert network selection, the output of the Attention layer is fed into the selected expert network for processing. Since the expert networks are independent, they can process the input data in parallel, thus greatly improving the model's processing speed. After processing its assigned input data, the selected expert network outputs a result. These results are then weighted and summed according to the weights assigned by the gating network to obtain the final output. Additionally, in some MoE architectures, there may be an optional shared feedforward neural network (Shared FFN) (or "shared expert network") that is activated in all cases. This network can further process the integrated output or work in parallel with the expert network's output to enhance the model's performance. For example, in... Figure 1 In this process, the outputs of the Attention layer, the expert network, and the Shared FFN can be added together to obtain the final output of the MoE layer.

[0042] exist Figure 1 In this context, for a single MoE layer, each expert network is typically an FFN (though it can be replaced with other neural network modules). See further... Figure 1 A MoE layer contains M expert networks, denoted as... and a gating network, denoted as Introducing the Shared Expert Network (FFN) shared Given input Here, h represents the result of processing by the Attention layer within the MoE layer. This is the output of the MoE layer. The calculation process is as follows:

[0043]

[0044] {g j} j∈G =SoftMax({h·r j} j∈G (2)

[0045] h′=∑j∈G (g j FFN j (h))+FFN shared (h)+h (3)

[0046] Where G represents the index of the activated expert network; g j Let be the weight of the j-th expert network.

[0047] As can be seen from the above calculation process of the MoE layer, the computational efficiency of the MoE layer lies in the fact that only K expert networks participate in the calculation in formula (3). However, since it is impossible to determine which expert networks need to participate in the calculation before formula (1) is completed, this means that either all expert networks need to be stored in the video memory, or the required K expert networks need to be temporarily loaded into the video memory after formula (1) is calculated. However, both solutions bring deployment challenges. Taking Mixtral-8x7B as an example, it contains 32 MoE layers, each with 8 experts, but each token only activates 2 expert networks. Although the actual number of parameters activated by each token is about 13 billion, the total number of parameters of the model reaches 45 billion. When deployed at FP16 precision, the model needs more than 92GB of video memory to accommodate all the parameters that may be activated. If the expert network parameters are temporarily loaded, the size of each expert network is about 176MB. During the single-step decoding process, up to 11.3 billion parameters need to be loaded, which will result in a large number of parameter transfers. If the expert network parameters are offloaded to the central processing unit (CPU) memory and an A100 GPU with a PCIe 4.0 x16 interface is used, the data transfer speed between CPU memory and GPU memory is limited by the PCIe bandwidth. The dynamic nature of the MoE architecture further exacerbates this latency, as each decoding step requires frequent loading of different expert network parameters from CPU memory, resulting in a 0.7-second latency per inference step. If the expert network parameters are offloaded to disk, the disk's read / write speed is much lower than that of memory and GPU memory. Therefore, when frequent loading of expert network parameters is required, the low speed of the disk will significantly slow down the model's execution speed, leading to a latency of over 10 seconds per inference step, which is clearly unacceptable. More importantly, the selection of expert networks is dynamically determined by the gating network. During batch parallel generation, different samples may select different expert networks. This requires loading all selected expert networks (or even all expert networks) into GPU memory when the batch size is large, which not only increases GPU memory usage but also further exacerbates communication latency.

[0048] The reason expert networks need to be loaded into GPU memory is that they are required to participate in computation, which relies on heterogeneous computing accelerators. Heterogeneous computing accelerators refer to heterogeneous computing resources other than the CPU, specifically designed to accelerate specific types of computational tasks, such as GPUs, tensor processing units (TPUs), and neural processing units (NPUs). In other words, if the expert network does not participate in computation, it does not need to be loaded, thus avoiding significant communication overhead. Based on this concept, this application introduces a mixture of lookup experts (MoLE) architecture, a novel MoE architecture where the functionality of the expert network can be implemented during inference using computation-free lookup tables. This eliminates the need to load the expert network during inference, resulting in low GPU memory usage and low communication overhead, while simultaneously improving performance.

[0049] For example, Figure 2 A schematic diagram of a MoLE architecture provided in an embodiment of this application is shown. This MoLE architecture can be applied to Transformer models as well as non-Transformer models, depending on the specific circumstances, and is not limited here. Figure 2 As shown, in the MoLE architecture, a neural network model mainly consists of at least one MoLE layer. For example, a MoLE layer can also be called a "hybrid expert layer." A MoLE layer typically consists of two key parts: a gating / router network and a look-up table (LUT) used to implement the functions of the expert networks in that MoLE layer. The LUT of a MoLE layer contains: the IDs of each item in the vocabulary, and the mapping relationships between the outputs of these expert networks when the output of the Embedding layer is used as the input of each expert network. The LUT in the l-th MoLE layer can be represented as: i represents the i-th ID; M represents the number of expert networks associated with the l-th MoLE layer; j represents the j-th expert network associated with the l-th MoLE layer; The LUT is the output of the j-th expert network when the processing result of the i-th ID by the Embedding layer is used as the input of the j-th expert network; |γ| is the size of the vocabulary. For example, the LUT can be stored in a storage location other than video memory, such as a disk, CPU memory, or a cloud server. It should be understood that the lookup table described in this embodiment can also be replaced with other tables with similar or identical functions, and the replaced scheme is still within the protection scope of this application. Furthermore, the expert network in the MoLE layer can be logically defined and implemented through a corresponding lookup table, or the expert network in the MoLE layer can also be a real entity that does not participate in the computation but is computed through a corresponding lookup table.

[0050] Since the expert network functionality within a MoLE layer is implemented using a LUT (Low-Level Underscore), and the LUT contains the IDs of each ID in the vocabulary, along with the mapping relationships between the outputs of these expert networks when the output of the Embedding layer is used as input, the LUT can replace the LUT-related expert network. This eliminates the need to load the expert network during inference. Furthermore, because the LUT already contains the processing results associated with each ID, inference can be performed by directly indexing the LUT based on the ID and loading the results into the heterogeneous computing accelerator as the output of the corresponding expert network. This eliminates the computation of the expert network, reduces computational load, and also reduces memory usage and communication overhead, thereby reducing inference latency.

[0051] Continue reading Figure 2 In each MoLE layer, the gating network primarily determines the weights of each expert network associated with that layer using some form of gating mechanism or probability distribution. Alternatively, the gating network can dynamically select which expert networks to activate based on their determined weights. In the MoLE architecture, since expert networks don't need to be loaded, all expert networks can be selected for use in each data processing step, thus improving computational accuracy compared to selecting only a subset.

[0052] exist Figure 2 In this context, a LUT in a MoLE layer is associated with M expert networks and a gating network, denoted as... Introducing the Shared Expert Network (FFN) shared Given input Here, h represents the result of processing by the Attention layer within the MoLE layer. This is the output of the MoLE layer. The calculation process is as follows:

[0053]

[0054] Among them, g j Let be the weight of the j-th expert network associated with the MoLE layer. Let i be the i-th ID. The output of the j-th expert network is the result of processing the i-th ID using the Embedding layer as the input of the j-th expert network. It is obtained from the LUT based on the i-th ID.

[0055] Formulas (3) and (4) select all expert networks related to the MoLE layer for calculation. When selecting a subset of expert networks related to the MoLE layer for calculation, the selection method shown in formulas (1) and (2) above can be used to select the corresponding expert networks. At the same time, when performing a lookup from the LUT, the output results related to the selected expert network can be selected from the LUT.

[0056] exist Figure 2 In this context, when there are multiple MoLE layers, these layers can be cascaded, but are not limited to; other methods of connection can also be chosen, and there are no restrictions here. Furthermore, other neural network modules can be added between multiple MoLE layers, depending on the specific situation, and there are no restrictions here either. Figure 2 Shared Expert Network (FFN) in the Middle MoLE Layer shared The choice can be made according to the actual situation, and no limitation is made here. For example, the head network may, but is not limited to, mainly consisting of convolutional layers. Furthermore, the Attention layer in the MoLE layer can also be replaced with other network layers, and the replaced scheme is still within the scope of this application. For example, the Attention layer can be replaced with a recurrent neural network (RNN) or other network layer capable of capturing the dependencies between data. In some embodiments, in Figure 2 A head network can also be added. This head network processes the output of the MoLE layer to obtain the inference result. Of course, the head network can also be replaced with other neural network models; the replacement scheme is still within the scope of this application.

[0057] for Figure 2 The MoLE architecture shown can be obtained through pre-training. Figure 3 As shown, during the training process of the MoLE architecture, the MoLE layer and the MoE layer have similar structures, each containing M real expert networks that actually participate in the computation, denoted as... and a gating network, denoted as Introducing the Shared Expert Network (FFN) sharedThe expert network is always activated under any input and does not accept weights from the gating network. Since the expert network is transformed into a LUT after training, the main differences between the MoLE and MoE architectures are as follows:

[0058] First, LUTs are computationally inactive, eliminating the need for sparse activations of expert networks to reduce computational costs. Therefore, the MoLE architecture can activate all expert networks, not just K. The computation formula for the gated network is:

[0059]

[0060] Where h is the result of processing by the Attention layer in the MoLE layer.

[0061] Secondly, since a LUT is essentially a finite input-output pair mapping, the key to implementing the functionality of an expert network using a LUT is ensuring that it has only a finite number of possible inputs. To this end, the MoLE architecture uses the output of the embedding layer as the input to the expert network. After training, the embedding layer is only associated with its input ID, meaning that the input to the expert network is only associated with the input IDs of the embedding layer, thus being restricted to a finite set. For example... Figure 3 As shown, the calculation formula for each MoLE layer is:

[0062]

[0063] Among them, g j Let be the weight of the j-th expert network. i is the i-th ID. h is the result of processing by the Attention layer in the MoLE layer.

[0064] During training, all expert networks are activated and receive gradients. Therefore, no auxiliary loss needs to be added to prevent model crashes or maintain training stability. For example, the MoLE architecture can be trained using cross-entropy loss modeled in language.

[0065] To facilitate understanding, the following will be combined with Figure 3 and Figure 4 This section provides a detailed introduction to the training process of the MoLE architecture model. Specifically, the MoLE layers in the MoLE architecture model during training contain real-world expert networks that actually participate in the computation. For example... Figure 4 As shown, the training process of the MoLE architecture model may include:

[0066] S401. Perform word segmentation on the input sample and map the segmentation results to the corresponding vocabulary index.

[0067] In this embodiment, the input sample can be segmented into a series of strings based on word segmentation methods such as regular expressions, rules, or deep learning. Then, the segmentation results are mapped to a predefined vocabulary to convert the segmentation results into numerical form, thereby obtaining the vocabulary index corresponding to the segmentation results. For example, this vocabulary index can also be called a "second vocabulary index".

[0068] S402. By processing the second vocabulary index in the MoLE architecture model, the predicted value related to the input sample is obtained. In the MoLE architecture model, the embedding representation of the second vocabulary index is input to each expert network in the third hybrid expert layer to obtain the output of each expert network in the third hybrid expert layer. The third hybrid expert layer is any MoLE layer in the MoLE architecture model.

[0069] In this embodiment, the second vocabulary index can be embedded using the Embedding layer in the MoLE architecture model to convert it into an embedded representation. This embedded representation captures the semantic information of the words and is a numerical form that the model can understand and process.

[0070] Furthermore, the embedding representation of the second vocabulary index is input into each expert network in each MoLE layer of the MoLE architecture model for processing, so as to obtain the output of each expert network in each MoLE layer. Here, the MoLE layers used require the implementation of the functions of the expert networks in these MoLE layers through LUTs. In other words, the step of selecting which MoLE layer's expert network functions to implement through a LUT determines which MoLE layer's expert network the embedding representation of the vocabulary index is input into for processing. For unselected MoLE layers, processing can continue according to the forward propagation method in the model. For example, a selected MoLE layer can be called a "third hybrid expert layer". In some embodiments, in each MoLE layer, an Attention layer can first process the data input to the MoLE layer. When the selected MoLE layer is the first MoLE layer, the input to the Attention layer is the embedding representation of the second vocabulary index. For example, in... Figure 3 In this model, the input to the first MoLE layer is the output of the Embedding layer. When the MoLE layer is not the first MoLE layer, its input can be the output of the previous MoLE layer. For example, in... Figure 3In this model, the input of the second MoLE layer is the output of the first MoLE layer. In some embodiments, in any MoLE layer of the MoLE architecture model, after calculating the output of the expert network in that MoLE layer, the weights of the expert network in that MoLE layer can be calculated using a corresponding gating network. Then, the output of the expert network in the MoLE layer is weighted, and the output of the MoLE layer is calculated based on the weighted calculation result and the input of that MoLE layer. Where the MoLE layer contains a shared expert network, the output of the MoLE layer can also be calculated based on the weighted calculation result, the processing result of the shared expert network on the input of the MoLE layer, and the input of that MoLE layer. For example, in... Figure 3 In the first MoLE layer shown, the outputs of FFN1, FFN2, FFN3, and FFN4 are calculated with their respective weights, and then the results are added together. At the same time, the sum of the results, the output of the Attention layer, and the output of the Shared FFN are added together to obtain the output of the first MoLE layer.

[0071] S403. Update the parameters of the MoLE architecture model based on the difference between the predicted values ​​and the true values ​​associated with the input samples.

[0072] In this embodiment, a loss value can be obtained by calculating the ground truth value and the predicted value associated with the input sample using loss functions such as cross-entropy. Then, the parameters in the MoLE architecture model are updated with the goal of minimizing this loss value to train the MoLE architecture model. For example, in Figure 3 In this process, parameters in each MoLE layer can be updated.

[0073] After training the MoLE architecture model, it can be used directly for inference like other LLMs. However, to further reduce memory overhead, the expert network functionality can be implemented using a LUT. Since the number of vocabulary indices used by the model is limited, approximately 50,000 to 100,000, for each possible input vocabulary index i, the corresponding expert network FFN is pre-computed. j The output is calculated using the following formula:

[0074]

[0075] In other words, the embedding representations of the indexes in the vocabulary can be input into each hybrid expert layer in the trained MoLE architecture model, so that each expert network in the hybrid expert layer can infer the embedding representations of the indexes, thereby obtaining the output (or "processing result") of the embedding representations of each expert network in the hybrid expert layer relative to the indexes.

[0076] In practice, only one forward propagation is needed, using the Embedding layer to process the results of all vocabulary indexes and then passing them to the expert network FFN. j This allows for the retrieval of all vocabulary indexes. Thus, the entries in the LUT of the l-th MoLE layer can be represented as:

[0077]

[0078] Where i is the index of the i-th vocabulary; M is the number of expert networks associated with the l-th MoLE layer; and j is the j-th expert network associated with the l-th MoLE layer. Let |γ| be the output of the j-th expert network when the processing result of the i-th vocabulary index by the Embedding layer is used as the input of the j-th expert network; |γ| is the size of the vocabulary. In other words, after one forward propagation, a LUT related to each hybrid expert layer can be generated based on the indexes in the vocabulary and the processing results of the embedding representations of the indexes by each expert network in the hybrid expert layer. This achieves reparameterization of the expert networks, that is, the functionality of the expert networks in the hybrid expert layer is implemented through LUTs.

[0079] After implementing the functionality of the expert network in the hybrid expert layer using a lookup table (LUT), the LUT can be offloaded to a storage device (e.g., disk or CPU memory). Then, the corresponding expert networks in the MoLE architecture model can be removed to obtain the desired MoLE architecture model. In other words, after training the MoLE architecture model, the lookup tables corresponding to the MoLE layers that require expert network functionality can be calculated first. Then, the expert networks contained in the MoLE layers that require expert network functionality can be removed, resulting in the desired MoLE architecture model. Figure 2 The described MoLE architecture model. For example, the MoLE architecture model used during training can be called the "second neural network model"; the model obtained after removing the expert network from the trained second neural network model can be called the "first neural network model". The computation of one MoLE layer can be represented as:

[0080]

[0081] In this way, when using the MoLE architecture model for inference, the pre-computed output in the LUT can be looked up, avoiding the need for computation in the video memory, thus greatly reducing video memory overhead.

[0082] The following section introduces the process of inference using the MoLE architecture model.

[0083] For example, Figure 5 This illustration shows a flowchart of a data processing method provided in an embodiment of this application. It is understood that this method can be executed by any device, apparatus, platform, or cluster of devices with computing and processing capabilities. For example, the method can be executed by a data processing apparatus, which can be implemented by software and / or hardware, and can be, but is not limited to, configured in an electronic device or server. Figure 5 As shown, the data processing method may include the following steps:

[0084] S501. Perform word segmentation on the input sequence and map the segmentation results to the index of the first vocabulary.

[0085] In this embodiment, the input sequence can be segmented into a series of strings. Then, the segmentation results are mapped to a predefined vocabulary to convert the segmentation results into numerical form, thereby obtaining the first vocabulary index corresponding to the segmentation results. Here, the vocabulary is the set of words used by the first neural network model. The input sequence can be text or an image, etc., without limitation. The input sequence can be user-inputted or transmitted from other devices, depending on the actual situation, without limitation.

[0086] S502. The first vocabulary index is processed by the first neural network model to obtain the reasoning result; wherein, in the first hybrid expert layer of the first neural network model, a query is performed in the first lookup table stored in the memory based on the first vocabulary index to obtain the output of each expert network in the first hybrid expert layer. The first lookup table includes the mapping relationship between the index in the vocabulary and the reasoning result of the embedding representation of the index by each expert network in the first hybrid expert layer. The vocabulary is the vocabulary set used by the first neural network model.

[0087] In this embodiment, the first vocabulary index can be processed by a first neural network model to obtain the inference result. This first neural network model may include at least one hybrid expert layer. For example, the hybrid expert network layer in the first neural network model may be, but is not limited to, […]. Figure 2 The MoLE layer shown.

[0088] During the processing of the first vocabulary index by the first neural network model, in the first hybrid expert layer of the first neural network model, a query can be performed based on the first vocabulary index in a first lookup table stored in memory to obtain the outputs of each expert network in the first hybrid expert layer. This first lookup table can include the mapping relationship between the index in the vocabulary and the results of inference performed by each expert network in the first hybrid expert layer on the embedded representation of the index. The vocabulary is the set of words used by the first neural network model. In other words, in the first hybrid expert layer, the output of the corresponding expert network can be obtained simply by querying the first lookup table, without loading the expert network or having it participate in the computation. This eliminates the computation of the expert network during the inference process of the first neural network model, reducing computational load, memory usage, and communication overhead, thereby reducing inference latency. In some embodiments, the expert networks in the first hybrid expert layer can be real or logically defined; regardless of the method, during the inference process of the first neural network model, the expert networks in the first hybrid expert layer can be non-computational, with the output of the corresponding expert network obtained by querying the first lookup table. In addition, the lookup tables associated with the hybrid expert layer can be stored on disk or in the CPU memory, depending on the specific circumstances, and are not limited here.

[0089] Furthermore, after obtaining the outputs of each expert network in the first hybrid expert layer, a weighted calculation can be performed on the outputs of each expert network in the first hybrid expert layer to obtain the first output. Then, based on the first output and the input of the first hybrid expert layer, such as by addition, the output of the first hybrid expert layer is calculated. The weights of each expert network in the first hybrid expert layer can be obtained by processing the input of the first hybrid expert layer using a gating network. For example, the input of the first hybrid expert layer can first be processed through a network layer in the first hybrid expert layer used to capture the dependencies between data, such as an attention layer; then, the corresponding processing result can be processed through a gating network to obtain the weights of each expert in the first hybrid expert layer. Alternatively, when the first hybrid expert layer includes a shared expert network, the input of the first hybrid expert layer can also be processed using this shared expert network to obtain the output of the shared expert network. Finally, the output of the first hybrid expert layer is calculated by adding the first output, the output of the shared expert network, and the input of the first hybrid expert layer.

[0090] Therefore, by implementing the functions of the expert networks in the first hybrid expert layer of the first neural network model through a first lookup table, the outputs of each expert network in the first hybrid expert layer can be directly retrieved from the first lookup table during inference, thus eliminating the need to load the expert networks in the first hybrid expert layer during inference. Since the expert networks in the first hybrid expert layer are not loaded during inference, the memory usage and communication overhead caused by loading expert networks are reduced. Furthermore, because the outputs of the expert networks in the first hybrid expert layer are retrieved from the lookup table, without the need for expert network computation during inference, expert network computation is eliminated, improving inference efficiency.

[0091] In some embodiments, the first neural network model may include a second hybrid expert layer in addition to the first hybrid expert layer. In this case, the functionality of the expert networks in the second hybrid expert layer can be implemented using a second lookup table. This second lookup table may include a mapping between indexes in the vocabulary and the results of reasoning based on the embedded representations of those indexes by the various expert networks in the second hybrid expert layer. This second lookup table can be stored in a corresponding memory.

[0092] In some embodiments, the first neural network model can be obtained based on a pre-trained neural network model. The process of obtaining the first neural network model can be found in the aforementioned training process of the MoLE architecture model, as well as the processing of the MoLE architecture model after training; these will not be elaborated upon here.

[0093] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments and features described in the above embodiments can be combined according to actual conditions, and the combined solutions are still within the protection scope of this application.

[0094] Next, based on the methods in the above embodiments, a data processing apparatus provided in this application will be described.

[0095] For example, Figure 6 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application is shown. Figure 6As shown, the data processing device 600 includes an acquisition module 601 and a processing module 602. The acquisition module 601 is used to acquire an input sequence. The processing module 602 is used to segment the input sequence into words and map the segmentation results to a first vocabulary index. The processing module 602 is also used to process the first vocabulary index through a first neural network model to obtain inference results. Specifically, in the first hybrid expert layer of the first neural network model, a query is performed based on the first vocabulary index in a first lookup table stored in memory to obtain the outputs of each expert network in the first hybrid expert layer. The first lookup table includes the mapping relationship between the index in the vocabulary and the inference results of the embedding representation of the index by each expert network in the first hybrid expert layer. The vocabulary is the set of words used by the first neural network model.

[0096] In some embodiments, the first neural network model further includes a second hybrid expert layer; wherein a second lookup table is stored in the memory, the second lookup table including the mapping relationship between the index in the vocabulary and the reasoning results of the embedding representation of the index by each expert network in the second hybrid expert layer.

[0097] In some embodiments, after the processing module 602 obtains the outputs of each expert network in the first hybrid expert layer by performing a query in the first lookup table stored in the memory based on the first vocabulary index, it is further configured to: perform a weighted calculation on the outputs of each expert network in the first hybrid expert layer to obtain a first output; and calculate the output of the first hybrid expert layer based on the first output and the input of the first hybrid expert layer.

[0098] In some embodiments, before performing weighted calculations on the outputs of each expert network in the first hybrid expert layer to obtain the first output, the processing module 602 is further configured to: process the input of the first hybrid expert layer based on the gated network in the first hybrid expert layer to obtain the weights of each expert network in the first hybrid expert layer, wherein the weights of each expert network in the first hybrid expert layer are used to perform weighted calculations on the outputs of each expert network in the first hybrid expert layer.

[0099] In some embodiments, when the processing module 602 calculates the output of the first hybrid expert layer based on the first output and the input of the first hybrid expert layer, it is specifically used to: process the input of the first hybrid expert layer based on the shared expert network in the first hybrid expert layer to obtain the output of the shared expert network; and calculate the output of the first hybrid expert layer based on the first output, the output of the shared expert network and the input of the first hybrid expert layer.

[0100] In some embodiments, each expert network in the first hybrid expert layer is logically defined and implemented through a first lookup table.

[0101] In some embodiments, the processing module 602 is further configured to: segment the input sample into words and map the segmentation results to a second vocabulary index; process the second vocabulary index through a second neural network model to obtain predicted values ​​related to the input sample, wherein, in the second neural network model, the embedding representation of the second vocabulary index is input to each expert network in the third hybrid expert layer to obtain the output of each expert network in the third hybrid expert layer, the third hybrid expert layer being any hybrid expert layer in the second neural network model; update the parameters of the second neural network model based on the difference between the predicted values ​​and the true values ​​related to the input sample; if the parameters of the second neural network model meet the requirements, input the embedding representation of the vocabulary index to each expert network in the third hybrid expert layer to obtain a first result of the inference performed by each expert network in the third hybrid expert layer on the embedding representation of the index; and, based on the index in the vocabulary and the first result, generate a lookup table related to the third hybrid expert layer and delete the expert networks in the third hybrid expert layer to obtain a first neural network model.

[0102] In some embodiments, Figure 6 Both the acquisition module 601 and the processing module 602 shown can be implemented in software or in hardware. For example, the implementation of the acquisition module 601 will be described below. Similarly, the implementation of the processing module 602 can refer to the implementation of the acquisition module 601.

[0103] As an example of a software functional unit, module 601 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 601 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0104] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0105] As an example of a hardware functional unit, the acquisition module 601 may include at least one computing device, such as a server. Alternatively, the acquisition module 601 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0106] The multiple computing devices included in the acquisition module 601 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 601 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 601 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0107] It should be noted that, in other embodiments, the acquisition module 601 can be used to execute any step in the data processing method described in the above embodiments, and the processing module 602 can also be used to execute any step in the data processing method described in the above embodiments. Furthermore, the steps implemented by the acquisition module 601 and the processing module 602 can be specified as needed, and different steps in the data processing method described in the above embodiments can be implemented by the acquisition module 601 and the processing module 602 respectively. Figure 6 The data processing device 600 shown has all the functions of the data processing device.

[0108] This application also provides a computing device 700. For example... Figure 7As shown, the computing device 700 includes a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, the memory 706, and the communication interface 708 communicate with each other via the bus 702. The computing device 700 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 700.

[0109] The 702 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus 704 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 704 may include a path for transmitting information between various components of the computing device 700 (e.g., memory 706, processor 704, communication interface 708).

[0110] Processor 704 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0111] The memory 706 may include volatile memory, such as random access memory (RAM). The processor 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0112] The memory 706 stores executable program code, and the processor 704 can call the program code in the memory 706 to implement the aforementioned functions. Figure 6 The functions of the acquisition module 601 and processing module 602 shown are used to implement the data processing method described in the above embodiments. That is, the memory 706 stores instructions for executing the data processing method described in the above embodiments.

[0113] Alternatively, the memory 706 may store executable code, which the processor 704 can call to implement the aforementioned functions. Figure 6 The data processing apparatus 600 shown in the diagram performs the functions of the data processing method described in the above embodiments. That is, the memory 706 stores instructions for executing the data processing method described in the above embodiments.

[0114] The communication interface 708 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 700 and other devices or communication networks.

[0115] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0116] like Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 706 of one or more computing devices 700 in the computing device cluster may store the same instructions for performing the data processing methods described in the above embodiments.

[0117] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the data processing method described in the above embodiments. In other words, a combination of one or more computing devices 700 can jointly execute instructions for executing the data processing method described in the above embodiments.

[0118] It should be noted that the memory 706 in different computing devices 700 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 6 This illustrates some of the functions of the data processing device 600 shown. Specifically, the instructions stored in the memory 706 of different computing devices 700 can implement the functions of one or more modules in the acquisition module 601 and processing module 602.

[0119] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 One possible implementation is shown. For example... Figure 9As shown, two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 706 in computing device 700A stores instructions for executing the functions of the acquisition module 601. Simultaneously, the memory 706 in computing device 700B stores instructions for executing the functions of the processing module 602.

[0120] It should be understood that Figure 9 The functions of the computing device 700A shown can also be performed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be performed by multiple computing devices 700.

[0121] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 8 and Figure 9 The connection method of the computing device cluster is different in that the memory 706 of one or more computing devices 700 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0122] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the aforementioned data processing method. In other words, a combination of one or more computing devices 700 can jointly execute the instructions for executing the aforementioned data processing method.

[0123] It should be understood that each step of the above method embodiments can be accomplished by hardware logic circuits or software instructions in a processor.

[0124] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When executed by a cluster of computing devices including at least one computing device, the computer program instructions cause the cluster of computing devices to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0125] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices containing at least one computing device, cause the cluster of computing devices to perform the methods in the above embodiments.

[0126] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0127] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0128] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0129] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method includes: The input sequence is segmented into words, and the segmentation results are mapped to the index of the first vocabulary. The first vocabulary index is processed by the first neural network model to obtain the inference result; In the first hybrid expert layer of the first neural network model, a query is performed in the first lookup table stored in the memory based on the first vocabulary index to obtain the output of each expert network in the first hybrid expert layer. The first lookup table includes the mapping relationship between the index in the vocabulary and the reasoning results of each expert network in the first hybrid expert layer on the embedded representation of the index. The vocabulary is the vocabulary set used by the first neural network model.

2. The method according to claim 1, characterized in that, The first neural network model also includes: a second hybrid expert layer; The memory stores a second lookup table, which includes a mapping relationship between the indexes in the vocabulary and the reasoning results of each expert network in the second hybrid expert layer on the embedded representation of the indexes.

3. The method according to claim 1 or 2, characterized in that, After obtaining the outputs of each expert network in the first hybrid expert layer by performing a query in the first lookup table stored in the memory based on the first vocabulary index, the method further includes: The first output is obtained by weighting the outputs of each expert network in the first hybrid expert layer. The output of the first hybrid expert layer is calculated based on the first output and the input of the first hybrid expert layer.

4. The method according to claim 3, characterized in that, Before performing a weighted calculation on the outputs of each expert network in the first hybrid expert layer to obtain the first output, the method further includes: Based on the gating network in the first hybrid expert layer, the input of the first hybrid expert layer is processed to obtain the weights of each expert network in the first hybrid expert layer. The weights of each expert network in the first hybrid expert layer are used to perform weighted calculations on the outputs of each expert network in the first hybrid expert layer.

5. The method according to claim 3 or 4, characterized in that, The step of calculating the output of the first hybrid expert layer based on the first output and the input of the first hybrid expert layer includes: Based on the shared experts in the first hybrid expert layer, the input of the first hybrid expert layer is processed to obtain the output of the shared experts; The output of the first hybrid expert layer is calculated based on the first output, the output of the shared expert, and the input of the first hybrid expert layer.

6. The method according to any one of claims 1-5, characterized in that, Each expert network in the first hybrid expert layer is logically defined and implemented through the first lookup table.

7. The method according to claims 1-6, characterized in that, The method further includes: The input sample is segmented into words, and the segmentation results are mapped to the index of the second vocabulary. The second vocabulary index is processed by the second neural network model to obtain the predicted value related to the input sample. In the second neural network model, the embedding representation of the second vocabulary index is respectively input to each expert network in the third hybrid expert layer to obtain the output of each expert network in the third hybrid expert layer. The third hybrid expert layer is any hybrid expert layer in the second neural network model. The parameters of the second neural network model are updated based on the difference between the predicted value and the true value associated with the input sample. If the parameters of the second neural network model meet the requirements, the embedding representations of the indexes in the vocabulary are respectively input to each expert network in the third hybrid expert layer to obtain the first result of the inference of the embedding representations of the indexes by each expert network in the third hybrid expert layer. Based on the indexes in the vocabulary and the first result, a lookup table related to the third hybrid expert layer is generated, and the expert networks in the third hybrid expert layer are deleted to obtain the first neural network model.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire the input sequence; The processing module is used to segment the input sequence into words and map the segmentation results to the first vocabulary index; The processing module is further configured to process the first vocabulary index through the first neural network model to obtain inference results. In the first hybrid expert layer of the first neural network model, a query is performed in the first lookup table stored in the memory based on the first vocabulary index to obtain the outputs of each expert network in the first hybrid expert layer. The first lookup table includes the mapping relationship between the index in the vocabulary and the inference results of each expert network in the first hybrid expert layer performing inference on the embedded representation of the index. The vocabulary is the vocabulary set used by the first neural network model.

9. The apparatus according to claim 8, characterized in that, The first neural network model also includes: a second hybrid expert layer; The memory stores a second lookup table, which includes a mapping relationship between the indexes in the vocabulary and the reasoning results of each expert network in the second hybrid expert layer on the embedded representation of the indexes.

10. The apparatus according to claim 8 or 9, characterized in that, After the processing module performs a query in the first lookup table stored in the memory based on the first vocabulary index to obtain the outputs of each expert network in the first hybrid expert layer, it is further used for: The first output is obtained by weighting the outputs of each expert network in the first hybrid expert layer. The output of the first hybrid expert layer is calculated based on the first output and the input of the first hybrid expert layer.

11. The apparatus according to claim 10, characterized in that, Before the processing module performs a weighted calculation on the outputs of each expert network in the first hybrid expert layer to obtain the first output, it is also used for: Based on the gating network in the first hybrid expert layer, the input of the first hybrid expert layer is processed to obtain the weights of each expert network in the first hybrid expert layer. The weights of each expert network in the first hybrid expert layer are used to perform weighted calculations on the outputs of each expert network in the first hybrid expert layer.

12. The apparatus according to claim 10 or 11, characterized in that, When the processing module calculates the output of the first hybrid expert layer based on the first output and the input of the first hybrid expert layer, it is specifically used for: Based on the shared experts in the first hybrid expert layer, the input of the first hybrid expert layer is processed to obtain the output of the shared experts; The output of the first hybrid expert layer is calculated based on the first output, the output of the shared expert, and the input of the first hybrid expert layer.

13. The apparatus according to any one of claims 8-12, characterized in that, Each expert network in the first hybrid expert layer is logically defined and implemented through the first lookup table.

14. The apparatus according to claims 8-13, characterized in that, The processing module is further configured to: The input sample is segmented into words, and the segmentation results are mapped to the index of the second vocabulary. The second vocabulary index is processed by the second neural network model to obtain the predicted value related to the input sample. In the second neural network model, the embedding representation of the second vocabulary index is respectively input to each expert network in the third hybrid expert layer to obtain the output of each expert network in the third hybrid expert layer. The third hybrid expert layer is any hybrid expert layer in the second neural network model. The parameters of the second neural network model are updated based on the difference between the predicted value and the true value associated with the input sample. If the parameters of the second neural network model meet the requirements, the embedding representations of the indexes in the vocabulary are respectively input to each expert network in the third hybrid expert layer to obtain the first result of the inference of the embedding representations of the indexes by each expert network in the third hybrid expert layer. Based on the indexes in the vocabulary and the first result, a lookup table related to the third hybrid expert layer is generated, and the expert networks in the third hybrid expert layer are deleted to obtain the first neural network model.

15. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-7, wherein the cluster of computing devices includes at least one computing device.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-7, wherein the computing device cluster includes at least one computing device.

Images