Neural network model acceleration method, PNM device and GPU device

Abstract

The invention provides a neural network model acceleration method, PNM equipment and GPU equipment. The neural network model is a neural network model adopting a MoE architecture, under the MoE architecture, the expert model is divided into a hot expert model and a cold expert model according to the number of activated times, the hot expert model is stored in external equipment, the acceleration method is executed by PNM equipment, and the cold expert model is stored in the external equipment. The acceleration method comprises the following steps: calculating by utilizing a selected cold expert model in cold expert models stored in the PNM equipment to obtain cold expert model output; the cold expert model output from the PNM device is transmitted to an external device, the external device including a GPU device. According to at least one example embodiment of the invention, the acceleration method of the neural network model can reduce the amount of data transmitted to the GPU equipment, thereby solving the bottleneck problem of data transmission.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence acceleration, and more specifically, to a method for accelerating neural network models, a PNM device, and a GPU device. Background Technology

[0002] In recent years, the number of parameters (including weights and biases) in large-scale neural networks (e.g., large language models (LLMs)) has expanded to hundreds of billions. However, these densely activated models require a significant amount of computation.

[0003] Large-scale neural networks such as LLMs can employ a hybrid expert (MoE) architecture, where expert models are stored in the video memory of a graphics processing unit (GPU). Alternatively, expert models under the MoE architecture can be offloaded to host-side (or host device) storage devices, such as dynamic random access memory (DRAM) or disks, thereby providing more storage space for the model.

[0004] However, storing the expert model under the MoE architecture in the GPU's video memory means that the parameter storage space of the expert model will occupy most of the storage space of the entire neural network model, and a single GPU will hardly be able to provide enough storage space for the neural network model.

[0005] Furthermore, using host-side storage devices to store expert models under the MoE architecture can provide more storage space. However, during inference, the GPU can only perform the next calculation after loading the parameters of the expert models under the MoE architecture from the host DRAM to the GPU. This requires transferring a large number of parameters of the activated expert models from DRAM to the GPU, which significantly increases access overhead. Therefore, this approach has the following drawbacks: First, due to the large number of selected expert model parameters, a large amount of data needs to be transferred, resulting in a data transfer bottleneck; second, because the GPU is idle while waiting for expert model data, GPU computing power is wasted, leading to low inference efficiency for large-scale neural networks.

[0006] The above information is provided as background information only and does not imply that the above information constitutes prior art in this disclosure. Summary of the Invention

[0007] One of the objectives of this disclosure is to provide a method for accelerating neural network models, which can reduce the cost per unit storage space of expert models.

[0008] One of the objectives of this disclosure is to provide a method for accelerating neural network models that can reduce the amount of data transmitted to GPU devices, thereby solving the data transmission bottleneck problem.

[0009] One of the objectives of this disclosure is to provide a method for accelerating neural network models, which can reduce the overall inference time of the neural network model.

[0010] According to a first aspect of this disclosure, a method for accelerating a neural network model is provided. The neural network model is a neural network model using a MoE architecture. Under the MoE architecture, expert models are divided into hot expert models and cold expert models based on the number of activations. The hot expert models are stored in an external device. The acceleration method is executed by a PNM device. The acceleration method includes: performing calculations using a selected cold expert model from the cold expert models stored in the PNM device to obtain a cold expert model output; and transmitting the cold expert model output from the PNM device to an external device, including a GPU device.

[0011] Optionally, the step of transmitting the cold expert model output may include transmitting the cold expert model output to the GPU device without transmitting the model parameters of the cold expert model.

[0012] Optionally, the PNM device may include multiple PNM devices, and there may be multiple cold expert models, which may be stored separately on multiple PNM devices.

[0013] Optionally, expert models with co-occurrence relationships among multiple cold expert models can be stored separately on different PNM devices in multiple PNM devices.

[0014] Optionally, the acceleration method may further include: identifying at least one cold expert model stored in a PNM device as a hot expert model during the computation of the neural network model; transmitting at least one cold expert model to a GPU device; deleting at least one cold expert model from the PNM device; and receiving at least one expert model dynamically determined as a cold expert model during the computation of the neural network model from the GPU device, wherein the at least one expert model is deleted from the GPU device after being provided to the PNM device.

[0015] Optionally, the step of using the selected cold expert models stored in the PNM device to perform computation to obtain the output of the cold expert model may include: identifying dense parts in the input data of the cold expert model; identifying important neurons in the cold expert model whose neuron importance metric exceeds a preset threshold to obtain the output of the cold expert model, wherein the cold expert model participating in the computation is determined based on cold expert selection information received from an external device, the cold expert selection information being provided by a gating network in the MoE architecture, the gating network being deployed in the GPU device.

[0016] Optionally, the PNM device may be a CMM-DC device, and the external device may also include the host device.

[0017] According to a second aspect of this disclosure, a PNM device is provided, comprising: a storage module configured to store cold expert models in a neural network model based on a MoE architecture, wherein the expert models of the MoE architecture are divided into hot expert models and cold expert models according to the number of times they are activated, and the hot expert models are stored in an external device; a computation module configured to perform computation using a selected cold expert model from the cold expert models to obtain a cold expert model output; and an input / output module configured to transmit the cold expert model output to an external device, including a GPU device.

[0018] Optionally, the input / output module can be further configured to transmit the output of the cold expert model to the GPU device without transmitting the model parameters of the cold expert model.

[0019] Optionally, the PNM device may include multiple PNM devices, and there may be multiple cold expert models, which may be stored separately on multiple PNM devices.

[0020] Optionally, expert models with co-occurrence relationships among multiple cold expert models can be stored separately on different PNM devices in multiple PNM devices.

[0021] Optionally, the computation module may be further configured to: identify at least one cold expert model stored in the PNM device as a hot expert model during the computation of the neural network model; the input / output module may be further configured to transmit at least one cold expert model to the GPU device; the computation module may be further configured to delete at least one cold expert model from the PNM device; the input / output module may receive from the GPU device at least one expert model that is dynamically determined to be a cold expert model during the computation of the neural network model, wherein at least one expert model is deleted from the GPU device after being provided to the PNM device.

[0022] Optionally, the computation module can be further configured to: identify dense portions in the input data of the cold expert model; identify important neurons in the cold expert model whose neuron importance metric exceeds a preset threshold, in order to obtain the output of the cold expert model, wherein the cold expert model participating in the computation is determined based on cold expert selection information received from an external device, the cold expert selection information being provided by a gating network in the MoE architecture, the gating network being deployed in a GPU device.

[0023] Alternatively, the PNM device may be a CMM-DC device.

[0024] According to a third aspect of this disclosure, a method for accelerating a neural network model is provided. The neural network model is a neural network model using a MoE architecture. Under the MoE architecture, expert models are divided into hot expert models and cold expert models based on the number of activations. The acceleration method includes: selecting a hot expert model through a gating network stored in a GPU device; performing calculations using the hot expert model to obtain the output of the hot expert model; and calculating the output of each transformation layer of the neural network model based on the output of the hot expert model, the output of the cold expert model received from an external device, and the weights determined by the gating network. The external device includes a PNM device.

[0025] Optionally, the acceleration method may also include transmitting cold expert selection information to the PNM device, the cold expert selection information being provided by a gating network.

[0026] Optionally, the acceleration method may further include: identifying at least one thermal expert model stored in a GPU device as a cold expert model during the computation of the neural network model; transmitting at least one thermal expert model to a PNM device; and receiving at least one expert model dynamically determined as a thermal expert model from the PNM device.

[0027] Optionally, the PNM device may include multiple PNM devices, and there may be multiple cold expert models. The multiple cold expert models are stored separately in multiple PNM devices. The acceleration method further includes: identifying co-occurrence relationships that reflect the activation of multiple cold expert models together; obtaining a graph including multiple nodes and edges, where the nodes of the graph represent expert models and the edges of the graph represent co-occurrence relationships; calculating the sum of all weights of each node in the multiple nodes; sorting the sum of weights of each node; assigning different colors according to the sorted order; and assigning different PNM devices to the multiple cold expert models based on the different colors.

[0028] Optionally, the step of assigning different colors according to the order may include: for the first node among multiple nodes, collecting the set of colors already colored for the adjacent nodes of the first node, and assigning the first node an unused color not included in the color set.

[0029] Optionally, in response to the absence of available colors, all colors can be iterated and the influence factor of each color can be calculated, and the first node can be assigned the color with the smallest influence factor.

[0030] According to a fourth aspect of this disclosure, a GPU device is provided, comprising: a storage module storing a thermal expert model of a neural network model employing a MoE architecture, wherein, under the MoE architecture, the expert models are divided into thermal expert models and cold expert models based on the number of times they are activated; a gating network selecting thermal expert models to participate in the computation; and a computation module performing computation using the selected thermal expert models to obtain thermal expert model outputs, and calculating the output of each transformation layer of the neural network model based on the thermal expert model outputs, cold expert model outputs received from an external device, and the weights of the cold expert model outputs and thermal expert model outputs determined by the gating network, wherein the external device includes a PNM device.

[0031] Optionally, the GPU device may also include an input / output module configured to transmit cold expert selection information provided by the gating network to the PNM device.

[0032] Optionally, the GPU device may also include a control module configured to: identify at least one hot expert model stored in the GPU device as a cold expert model during the computation of the neural network model; the input / output module is further configured to transmit at least one hot expert model to the PNM device; and receive at least one expert model dynamically determined as a hot expert model from the PNM device.

[0033] Optionally, the PNM device may include multiple PNM devices, and there may be multiple cold expert models, which are stored separately in multiple PNM devices. The control module is further configured to: identify co-occurrence relationships that reflect the activation of multiple cold expert models together; obtain a graph including multiple nodes and edges, where the nodes of the graph represent expert models and the edges of the graph represent co-occurrence relationships; calculate the sum of all weights of each node in the multiple nodes; sort the sum of weights of each node; assign different colors according to the sorted order; and assign different PNM devices to the multiple cold expert models based on the different colors.

[0034] Optionally, the control module can be further configured to: for a first node among multiple nodes, collect the set of colored colors of the adjacent nodes of the first node, and assign an unused color not included in the color set to the first node.

[0035] Optionally, the control module can be further configured to: in response to the absence of available colors, traverse all colors and calculate the influence factor of each color, and assign the color with the smallest influence factor to the first node.

[0036] According to a fifth aspect of this disclosure, a method for managing a neural network model is provided. The neural network model is a neural network model using the MoE architecture. The management method is executed by a GPU device or a host device. The management method includes: determining the usage of expert models under the MoE architecture during the training process of the neural network model; and pre-dividing the expert models into cold expert models and hot expert models according to the usage, wherein the cold expert models will be pre-stored in a PNM device and the hot expert models will be pre-stored in a GPU device.

[0037] Optionally, the management method may also include: during the inference computation of the neural network model, statistically analyzing the activation status of the expert models; dynamically determining whether each expert model in the expert models is a hot expert model or a cold expert model based on the activation status of the expert models; and dynamically adjusting the position of the expert models between the GPU device and the PNM device.

[0038] According to a sixth aspect of this disclosure, a computer system is provided, the computer system including the above-described PNM device and the above-described GPU device.

[0039] According to a seventh aspect of this disclosure, a host device is provided, comprising a non-transitory computer-readable recording medium configured to store instructions; and a processor configured to execute instructions to cause the processor to perform: identifying co-occurrence relations reflecting the co-activation of multiple cold expert models; obtaining a graph comprising multiple nodes and edges, wherein the nodes of the graph represent expert models and the edges of the graph represent co-occurrence relations; calculating the sum of all weights of each of the multiple nodes; sorting the sum of weights of each node; assigning different colors according to the sorted order; and assigning different PNM devices to the multiple cold expert models based on the different colors.

[0040] Optionally, the processor may be further configured to: for a first node among a plurality of nodes, collect a set of colored colors of the neighboring nodes of the first node, and assign an unused color not included in the color set to the first node.

[0041] Optionally, the processor can be further configured to, in response to the absence of a color available, traverse all colors and calculate the influence factor for each color, and assign the color with the smallest influence factor to the first node.

[0042] The acceleration method according to embodiments of this disclosure can improve the utilization efficiency of GPU devices and increase the inference speed of neural network models. Attached Figure Description

[0043] The above and other objects and features of exemplary embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, which exemplarily illustrate the embodiments, wherein: Figure 1An example of the application of the MoE architecture in LLM is shown; Figure 2 The MoE architecture diagram of the neural network model is shown; Figure 3 A schematic diagram illustrating the use of an expert model under the MOE architecture according to at least one example embodiment of the present disclosure is shown. Figure 4 This is a schematic diagram showing a comparison between the number of model parameters and the number of output parameters of the expert model according to this disclosure; Figure 5 This is a flowchart illustrating an acceleration method according to at least one example embodiment of the present disclosure; Figure 6 This is a flowchart illustrating an acceleration method according to at least one example embodiment of the present disclosure; Figure 7 This is a schematic diagram illustrating the assignment of cold expert models with co-occurrence relationships to different CMM-DCs according to at least one example embodiment of the present disclosure; Figure 8A This is an architectural diagram illustrating a computer system according to at least one example embodiment of the present disclosure; and Figure 8B This is an architectural diagram illustrating a neural network system according to at least one example embodiment of the present disclosure. Detailed Implementation

[0044] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, examples of which are illustrated in the drawings, wherein the same reference numerals always refer to the same parts. The embodiments will now be described with reference to the accompanying drawings in order to explain this disclosure.

[0045] The following detailed description is provided to aid in obtaining a full understanding of the methods, apparatus, and / or systems described herein. However, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein; equivalent substitutions or changes may be made, except for operations that must occur or be performed in a specific order. Furthermore, for clarity and conciseness, descriptions of content well-known in the art will be omitted or simplified.

[0046] Unless otherwise specified, the same reference numerals generally refer to the same elements (e.g., components, steps, and methods). Reference numerals described in previous embodiments that reappear in later embodiments may be omitted. Furthermore, technical features described in different or the same embodiments can be combined in any way, as long as the combined embodiment or technical solution is complete and can solve the technical problems of this application or achieve the technical effects described or not described in this disclosure but which can be determined based on the complete technical solution described above.

[0047] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner. A brief description of the terminology used in this disclosure is provided below.

[0048] CXL (Compute Express Link) is an open standard for high-speed, low-latency interconnects between host processors and devices such as memory extenders, smart memory modules, and accelerators, providing high-speed, high-capacity connectivity. CXL memory expansion technology allows servers to expand memory capacity to tens of terabytes while increasing bandwidth to hundreds of gigabytes per second.

[0049] Processing Near Memory (PNM) devices are devices that perform computational tasks near or inside memory to reduce latency and power consumption caused by data transfer between the processor and memory.

[0050] A CMM-DC (CXL Memory Module – DRAM & Compute) device is a device that integrates memory with accelerators or computing circuitry connected to that memory. By utilizing CMM-DC for data computation, it reduces data migration between the host CPU and memory. Because computation is performed closer to memory (i.e., the processor is closer to memory, PNM), data transfer between the host CPU and memory is reduced, thereby lowering (or eliminating) the bottleneck of data transfer between the host CPU and memory when processing large-scale artificial intelligence (AI) models. CMM-DC can improve data processing speed and shorten training time. A CMM-DC device is a type of PNM device.

[0051] The CMM-DC device pool includes multiple CMM-DC devices, and these devices transmit data or communicate with each other via a switch.

[0052] This disclosure uses PNM devices, such as CMM-DC devices, to store cold expert models, providing greater storage space for model parameters of expert models under the MoE architecture that require hundreds of GB.

[0053] This disclosure offloads the computation of cold expert models, which have fewer activations or less frequent computations, to a PNM device such as a CMM-DC device, and only transmits the output parameters of the cold expert model computations to the GPU, instead of transmitting the entire expert model parameters. This significantly reduces data transfer volume, thereby resolving the data transfer bottleneck, improving GPU utilization efficiency, and increasing model inference speed.

[0054] The acceleration scheme disclosed herein is applicable not only to LLMs, but also to other large-scale neural networks, such as convolutional neural networks (CNNs), generative adversarial networks (GANs), graph neural networks (GNNs), and so on.

[0055] The host device disclosed herein may include a host CPU (or CPU device). The CPU, as the main control unit of the system, is responsible for task scheduling, data management, and overall process control. The GPU or GPU device is connected to the CPU or CPU device in the host device via a high-speed interface (e.g., Peripheral Component Interconnect High Speed ​​(PCIe)). As an example, the GPU device may be a separate device from the host device or it may be part of the host device. The host device may also connect one or more CMM-DC devices via a CXL interface to expand memory capacity and implement near-memory data preprocessing.

[0056] Figure 1 This illustrates an example of the application of the MoE architecture in LLM. Figure 2 The MoE architecture diagram of the neural network model is shown.

[0057] Reference Figure 1 In LLM, the input tokens or words are passed from one layer to the next and processed through multiple transform layers 110. Each transform layer contains self-attention and a feedforward network. For example, self-attention 111 can be used to capture the relationships between words and contextual information in the input sequence, while the feedforward network (FFN) 112 can be used to perform non-linear transformations on the output of self-attention to enhance the model's expressive power and capture more complex features.

[0058] Reference Figure 1 In Transformer models, the MoE architecture is typically used after the self-attention of each transform layer, replacing the original feedforward network.

[0059] Reference Figure 2 In the MoE architecture, the input data is first fed into a gating network (211), which selects an appropriate expert model for processing based on the characteristics of the input data. For example... Figure 2As shown, gating network 211 selects the third expert (k=3), meaning that for the current input, the third expert is considered the most suitable expert model (or simply expert, specifically referring to a sub-neural network (e.g., a feedforward network)) to process that input. In addition to determining which specific expert model (or simply expert) is activated or participates in the computation, the input-side gating network 211 also determines the number of experts. The output-side gating network 212 can be used to determine the weights of each expert model.

[0060] Reference Figure 2 The selected expert model processes the input data and generates corresponding outputs. Finally, the outputs of all expert models are aggregated by another gating network to form the final output result. In other words, the gating network 212 on the output side can obtain the output of each layer based on the outputs of all expert models and their corresponding weights (determined by the gating network). In this way, the MoE architecture can combine the advantages of multiple expert models to provide more accurate and professional predictions or decisions.

[0061] As an example, expert models in the MoE architecture can be classified as cold expert models or hot expert models. Cold expert models in the MoE architecture are offloaded to the PNM device, while hot expert models, gating networks, etc., can still be stored or deployed on the GPU device. As an example, gating networks can also be stored or deployed on the host device.

[0062] Figure 3 A schematic diagram illustrating the use of an expert model under the MOE architecture according to at least one example embodiment of the present disclosure is shown.

[0063] Reference Figure 3 Only a small number of popular expert models will be activated to participate in the calculation of most tokens, while the remaining unpopular experts will be activated to participate in the calculation of a significantly smaller number of tokens.

[0064] Reference Figure 3 There are a total of 128 experts, of which 122 (122 = 25.4 + 72.5 + 24.6, accounting for 95%) are only activated on average to participate in the calculation of 0-7 tokens.

[0065] This demonstrates that most token computation is performed using a small number of hot expert models. Therefore, expert models can be categorized and their positions adjusted. Specifically, hot and cold expert models can be distinguished based on the number or frequency of their activation or use. For example, expert models with activation or usage counts greater than or equal to a preset threshold can be identified as hot expert models, while those with fewer activations or uses below the preset threshold can be identified as cold expert models. For instance, classification can be pre-executed or performed by a hot-cold expert manager, which can be deployed on GPU devices or host devices.

[0066] As mentioned above, the model parameters of existing expert models occupy a large portion of the model storage space. According to at least one example embodiment of this disclosure, cold expert models under the MoE architecture are offloaded to a PNM device, such as a CMM-DC device, thereby providing more storage space for expert model parameters that require hundreds of gigabytes. Since the cold expert model is stored on the PNM device, its computation is also performed on the PNM device. The PNM device only transmits the output of the cold expert model and does not need to transmit the model parameters (e.g., weights and biases) to GPU devices, etc. This provides more storage space for the model parameters of the cold expert model, solving the data transmission bottleneck problem.

[0067] Figure 4 This is a schematic diagram illustrating a comparison of the number of model parameters and the number of output parameters of an expert model according to at least one example embodiment of the present disclosure.

[0068] Reference Figure 4 The expert model of a certain LLM is a feedforward network (FFN), where the input and output dimensions of each expert model are 7168, the intermediate hidden layer dimension is 2048, the number of model parameters transmitted to the GPU by an expert model is 2×2048×7168, but the output of the expert model only has 7168 parameters.

[0069] Therefore, if only the output of the cold expert model is transmitted and not the model parameters (i.e., only the output of the cold expert model is transmitted to the GPU device, instead of transmitting all the parameters of the cold expert model), the amount of data transmitted can be only 1 / 4096 of the original data transmission amount, thus solving the data transmission bottleneck problem.

[0070] Figure 5 This is a flowchart illustrating an acceleration method according to at least one example embodiment of the present disclosure.

[0071] Reference Figure 5The acceleration method according to at least one example embodiment of the present disclosure can be performed by a PNM device (e.g., a CMM-DC device). The acceleration method according to at least one example embodiment of the present disclosure may include steps S510 and S520.

[0072] In step S510, the selected cold expert model from the cold expert models stored in the PNM device is used to perform calculations to obtain the cold expert model output.

[0073] Based on the number of times they are activated, the expert models under the MoE architecture are pre-divided into hot expert models and cold expert models. Hot expert models are stored in GPU devices, while cold expert models are stored in PNM devices (e.g., CMM-DC devices).

[0074] Multiple cold expert models can be pre-stored in the PNM device (e.g., a CMM-DC device), referencing... Figure 3 The number of cold expert models pre-stored in the PNM device (e.g., a CMM-DC device) can be 122. The number of hot expert models pre-stored in the GPU device can also be multiple, see [reference]. Figure 3 There can be up to seven thermal expert models pre-stored in the GPU device.

[0075] Which cold expert model is selected or activated can be determined by the gating network under the MoE architecture, specifically by the gating network on the input side. As an example, the gating network can be deployed or stored on a GPU device. As another example, both the input-side and output-side gating networks can be deployed or stored on the GPU device. The GPU device can transmit the identifier (ID) of the cold expert model determined by the gating network to the PNM device, which can then select the appropriate cold expert model based on the received ID. However, this disclosure is not limited to this; the gating network can also be deployed or stored on a CPU device.

[0076] In step S520, the cold expert model output is transmitted to an external device. Here, the external device is relative to the PNM device, and may include a GPU device, or even a host device.

[0077] Reference Figure 4 When the cold expert model is stored on the PNM device, the output of the cold expert model (e.g., 7168 parameters) can be transmitted to the CPU device, instead of the model parameters of the cold expert model (2×2048×7168 parameters) transmitted to the GPU device. Therefore, the output of the cold expert model can be transmitted to the GPU device without transmitting the model parameters of the cold expert model.

[0078] As an example, there can be multiple PNM devices. For instance, a PNM device can be a CMM-DC device, and multiple PNM devices can be implemented by a device pool that includes multiple CMM-DC devices. In this case, multiple cold expert models can be stored separately on multiple PNM devices. This can improve computational speed.

[0079] Furthermore, expert models that co-occur in multiple cold expert models can be stored separately on different PNM devices across multiple PNM devices. Here, expert models that co-occur refer to expert models that are simultaneously selected, simultaneously activated, or simultaneously used.

[0080] During the inference process, the location of the expert model (e.g., storage location or deployment location) can be adjusted based on changes in the frequency of its use. Specifically, in the context of... When reasoning with a token, if a certain cold expert model is selected, 0.1 If the cold expert model is migrated from the PNM device (such as the CMM-DC device) to the GPU after a certain number of times (i.e., a preset threshold), the expert model with the lowest usage frequency (less than 0.1Tr) on the GPU device will be migrated to the PNM device. In other words, the entire expert model (including the model parameters of the expert model) can be dynamically migrated.

[0081] As an example, the acceleration method according to at least one example embodiment of this disclosure may further include: transmitting to a GPU device at least one cold expert model from expert models pre-divided into cold expert models stored in a PNM device, the at least one cold expert model being dynamically determined as a hot expert model during the computation of the neural network model; and / or receiving from a GPU device at least one hot expert model from expert models pre-divided into hot expert models stored in a GPU device, the at least one hot expert model being dynamically determined as a cold expert model during the computation of the neural network model, while simultaneously deleting at least one migrated cold expert model from the PNM device and at least one migrated hot expert model from the GPU device.

[0082] However, this disclosure is not limited to migrating expert models between GPU devices and PNM devices, but can also migrate cold expert models between different PNM devices. For example, the storage location of cold expert models in multiple PNM devices can be adjusted, and when the co-occurrence frequency of cold expert models in the same PNM device increases to a preset number, at least one of the expert models in a certain PNM device whose co-occurrence frequency exceeds the preset number can be dynamically migrated to other PNM devices.

[0083] The steps of using the selected cold expert model from the cold expert models stored in the PNM device to perform calculations to obtain the output of the cold expert model may include: selecting the dense part of the input data of the cold expert model and selecting the important neurons of the cold expert model for calculation to obtain the output of the cold expert model. Here, the important neurons are neurons whose neuron importance metric exceeds a preset threshold.

[0084] The dense portion of the attention vector can be selected for computation; that is, the dense portion of the input data in a cold expert model can be chosen for computation. This dense portion can be the parts of the input data that are more critical to the task or contain more information. For example, in attention-based models (such as the Transformer), the dense portion can refer to those parts that score higher in the attention weight distribution. For instance, when processing text, certain words may be more important for understanding the meaning of the entire sentence, and therefore they will have higher weights in the attention distribution. The input portions corresponding to these high-weight positions can be considered "dense portions."

[0085] As an example, the top K maximum values ​​in the attention vector can be selected as the dense part. Here, K can be 1 / 10 of the dimension of the attention vector. When computing the input to the hidden layer, only the top K maximum values ​​in the attention vector can be selected, and matrix multiplication can be performed using the partial hidden layer parameters corresponding to these K positions.

[0086] In addition, when selecting important neurons in the cold expert model, the activation function can be used to select important neurons from the cold expert model to participate in the calculation.

[0087] For example, the L1 norm can be used as a criterion. The L1 norm is the sum of the absolute values ​​of all elements in a vector. Therefore, when storing the parameters of a cold expert model, the L1 norm of the weight vector of each neuron from the hidden layer to the output is calculated. The larger the L1 norm of a neuron, the larger its weight and the greater its contribution to the output. Since this norm is calculated only once, the computational cost is negligible. Then, when calculating the output of each expert model, the L1 norm of the activation vector is calculated. When calculating neurons from the hidden layer to the output, only neurons with an L1 norm greater than a preset value are selected for calculation, and the rest are padded with 0. This further improves the inference speed of the model.

[0088] PNM devices As described above, the acceleration method according to at least one example embodiment of this disclosure can be executed by a PNM device.

[0089] A PNM device according to at least one example embodiment of this disclosure may include a storage module, a computing module, and an output module. The storage module is used to store cold expert models, which are expert models that are less frequently activated in the MoE architecture, but may play a crucial role under specific inputs despite their low activation frequency. The type of storage module is not specifically limited.

[0090] As an example, the computation module can selectively call and execute certain cold expert models from among the cold expert models to complete the computation task. The computation module can use the selected cold expert model to perform calculations and obtain the cold expert model output.

[0091] As an example, the output module can transmit cold expert model output to external devices. Additionally, the PNM device may include a control module capable of performing management and scheduling functions, such as receiving and parsing task requests or various instructions from external sources to determine which cold expert models are activated. The type of storage module is not particularly limited and may include multiple components, such as one or more volatile memories (e.g., DRAM) and one or more non-volatile memories (e.g., solid-state memories, such as NAND).

[0092] Figure 6 This is a flowchart illustrating an acceleration method according to at least one example embodiment of the present disclosure.

[0093] The acceleration method according to at least one example embodiment of the present disclosure may be executed by a GPU device, and the acceleration method according to at least one example embodiment of the present disclosure may include steps S610 and S620.

[0094] In step S610, the thermal expert model selected by the gating network under the MoE architecture stored in the GPU device is used to perform calculations to obtain the thermal expert model output.

[0095] As an example, in the MoE architecture, GPU devices can store and compute hot expert models selected by the gating network. As mentioned above, hot expert models here refer to those expert models that are frequently activated and highly relevant to the current input. Based on the number of times they are activated, the expert models in the MoE architecture are pre-divided into hot expert models and cold expert models, and the specific division method is as described above, which will not be repeated here.

[0096] The gating network on the input side can generate expert selection signals based on the features of the input data, selecting several hot expert models from a large number of expert models. These selected hot expert models can be pre-loaded into the high-speed video memory of the GPU device, enabling fast access and parallel computing.

[0097] In step S620, the output of each transform layer of the neural network model is calculated based on the output of the hot expert model, the output of the cold expert model received from an external device, and the weights determined by the gating network. The GPU device can perform a weighted aggregation of the hot expert model output and the cold expert model output received from the PNM device (the weights can be determined by the gating network on the output side) to obtain the output of each transform layer. Here, receiving the cold expert model output from the PNM device means that the source of the cold expert model output is the PNM device, rather than necessarily receiving the cold expert model output directly from the PNM device. As an example, the cold expert model output can be received from a data transfer unit. Preferably, the GPU device can directly receive the cold expert model output from the PNM device.

[0098] The external devices mentioned here are in contrast to GPU devices, and can include PNM devices.

[0099] The acceleration method according to at least one example embodiment of the present disclosure may further include transmitting cold expert selection information to a PNM device, wherein the cold expert selection information may be determined by a gating network (e.g., an input-side gating network) and the gating network may be deployed in the GPU device similarly to a thermal expert model.

[0100] The GPU device disclosed herein can migrate expert models that are dynamically determined to be cold expert models to the PNM device, and can also receive expert models that are dynamically determined to be hot expert models from the PNM device.

[0101] Similarly, the acceleration method according to at least one example embodiment of this disclosure may further include: transmitting to a PNM device at least one hot expert model from expert models pre-divided into hot expert models stored in a GPU device, wherein the at least one hot expert model is dynamically determined as a cold expert model during the computation (i.e., inference computation) of the neural network model; and / or receiving from a PNM device at least one cold expert model from expert models pre-divided into cold expert models stored in a PNM device, wherein the at least one cold expert model is dynamically determined as a hot expert model during the computation of the neural network model. Therefore, the acceleration method according to at least one example embodiment of this disclosure may further include dynamically determining cold and hot expert models. As an example, the pre-division of cold and hot expert models in this disclosure may be performed during training, and the dynamic adjustment of cold and hot expert models may be performed during computation. However, this is merely an example, and the embodiments of this disclosure are not limited thereto.

[0102] In addition to migrating expert models between GPU devices and PNM devices, expert models can also be migrated between PNM devices. This migration between PNM devices can be managed by a hot / cold expert manager, which is preferably deployed on a GPU device or a host device, but this disclosure is not limited thereto. As an example, the hot / cold expert manager can be implemented via the CPU running the software.

[0103] The acceleration method according to at least one example embodiment of the present disclosure may further include: assigning different PNM devices to multiple cold expert models using a graph coloring strategy.

[0104] The steps of assigning different PNM devices to multiple cold expert models using a graph coloring strategy may include: constructing a graph reflecting the co-occurrence relationships that reflect the activation of multiple cold expert models together, where the nodes of the graph represent expert models and the edges of the graph represent co-occurrence relationships.

[0105] The steps of assigning different PNM devices to multiple cold expert models using a graph coloring strategy may further include: calculating the sum of all weights for each of the multiple nodes. In the graph, each node represents a cold expert model, and the edges connecting the expert models record the number of times the two experts are selected simultaneously, i.e., the weights.

[0106] The steps of assigning different PNM devices to multiple cold expert models using a graph coloring strategy may also include: sorting the sum of the weights of each node; assigning different colors according to the sorted order, thereby assigning different PNM devices to multiple cold expert models.

[0107] As an example, suppose there are Cm PNM devices (e.g., there are Cm CMM-DC devices, where Cm is a positive integer). Different colors can be assigned to adjacent nodes (i.e., experts used at the same time) (i.e., different CMM-DC devices are assigned) to avoid putting cold expert models used at the same time into the same CMM-DC device.

[0108] The step of assigning different colors according to the order may include: for a first node among a plurality of nodes, collecting a set of already colored colors of the adjacent nodes of the first node, and assigning the first node an unused color not included in the set of colors.

[0109] For example, for node v, collect the set S of colored colors of node v's neighboring nodes, prioritizing colors that have not been used before. If no color is available, iterate through all colors and calculate the influence factor for each color (i.e., the sum of the weights of nodes connected to that node and using that color), and assign the color with the smallest influence factor to node v. If multiple colors have the same influence factor, prioritize the color with the smallest influence factor.

[0110] Figure 7 This is a schematic diagram illustrating the assignment of cold expert models with co-occurrence relationships to different CMM-DCs according to at least one example embodiment of the present disclosure.

[0111] Reference Figure 7 We construct an expert co-occurrence network, where each node represents a cold expert model, and edges represent the number of times two cold expert models are activated simultaneously. For example, in the graph, expert 1 and expert 2 were selected together once, so there is an edge with a weight of 1 between them. Next, we calculate the sum of the weights of each node, which is the sum of the weights of all its adjacent edges. For example, the sum of the weights of expert 2 is 6 + 2 + 1 = 9. We then sort all the nodes in descending order of their total weights, resulting in the sorted list of experts: 2, 7, 9, 12, 4, 1.

[0112] Then, colors are assigned to each expert in sorted order (i.e., PNM devices are assigned). For the first node (expert 2), the set of colors already colored by its neighboring nodes is collected. If an unused color exists, it is selected first. For example, for expert 2, cyan from cyan, green, and yellow (color set S2) can be selected; for expert 7, green from green and yellow (color set S7) can be selected. If a node has no available colors, the influence factor for each color needs to be calculated, which is the sum of the weights of nodes connected to that node and using that color. The color with the smallest influence factor is selected. For example, for expert 25, all three colors are used by neighboring nodes (i.e., experts 34, 19, and 56), and no color is available, the influence factors of these three colors can be calculated, and the color with the smallest influence factor is selected; that is, for expert 25, yellow can be selected.

[0113] Therefore, the above steps can be repeated until all experts have been assigned colors. This ensures that concurrently used cold experts are assigned to different PNM devices as much as possible, thus avoiding resource conflicts.

[0114] Furthermore, although not shown, the steps of dynamically adjusting the expert model described above can also be performed by a GPU device. Therefore, the acceleration method according to at least one example embodiment of this disclosure may also include the step of dynamically adjusting the position of the expert model according to changes in the frequency of expert model usage during inference computation.

[0115] For example, as described above, the acceleration method of at least one example embodiment of this disclosure can adjust the storage location of the hot and cold expert model, for example, when... When reasoning with a token, if a certain cold expert model is selected, 0.1 Next, migrate it from the PNM device to the GPU device; at the same time, migrate the expert model with the lowest usage frequency and less than 0.1Tr on the GPU device to the PNM device.

[0116] The pre-partitioning and dynamic adjustment of the aforementioned cold expert model and hot expert model can be performed by the cold and hot expert manager. The cold and hot expert manager can be deployed on GPU devices or host devices. Preferably, the cold and hot expert manager can be deployed in the same way as the gating network.

[0117] Specifically, the management method for a neural network model according to at least one example embodiment of this disclosure can be executed by a cold / hot expert manager deployed in a GPU device or a host device. The management method of this disclosure may include: determining the usage of expert models under the MoE architecture during the training process of the neural network model, and pre-dividing the expert models into cold expert models and hot expert models based on the usage, wherein the cold expert models will be pre-stored in a PNM device and the hot expert models will be pre-stored in a GPU device.

[0118] The management method according to at least one example embodiment of the present disclosure may further include: during the inference computation of the neural network model, statistically analyzing the activation status of expert models; dynamically determining whether each expert model in the expert models is a hot expert model or a cold expert model based on the activation status of the expert models; and dynamically adjusting the position of the expert models between the GPU device and the PNM device.

[0119] Furthermore, the management method according to at least one example embodiment of this disclosure may also include dynamically adjusting the location of cold expert models among PNM devices. For example, at least one of at least two cold expert models with a co-occurrence relationship stored in the same PNM device may be migrated to other PNM devices. When the number of PNM devices is insufficient, at least one of at least two cold expert models in the same device with a co-occurrence rate exceeding a preset threshold may be migrated to other PNM devices.

[0120] GPU devices The GPU device disclosed herein may include a memory and a processor. The memory may store programs or instructions, and the processor may execute the programs or instructions to cause the processor to perform the acceleration method and / or the management method described above in at least one example embodiment of this disclosure. The type of memory in the GPU device is not specifically limited, and the memory may be a non-transitory computer-readable storage medium.

[0121] host device The host device disclosed herein may include a memory and a processor. The memory may store programs or instructions, and the processor may execute the programs or instructions to cause the processor to perform the management methods described above. The memory herein may be a non-transitory computer-readable storage medium.

[0122] Computer System The computer system disclosed herein may include the aforementioned PNM device and the aforementioned GPU device. As an example, the computer system disclosed herein may also include the aforementioned host device (or CPU device).

[0123] Specific reference Figure 8A , Figure 8A This is an architectural diagram illustrating a computer system according to at least one example embodiment of the present disclosure. Figure 8B This is an architectural diagram illustrating a neural network system according to at least one example embodiment of the present disclosure.

[0124] Reference Figure 8A The computer system 900 disclosed herein may include a GPU device (hereinafter referred to as GPU) 910 and a PNM device 920. The PNM device 920 may be a CMM-DC device. The CMM-DC device pool may include three CMM-DC devices (hereinafter referred to as CMM-DC), namely, CMM-DC 1, CMM-DC 2 and CMM-DC 3.

[0125] See Figure 8B The GPU (specifically, GPU 910) internally deploys a gating network and all hot experts. During MoE (Mixture of Experts) inference, the gating network first calculates the activation score of each expert based on the input token and then selects a subset of experts according to a predefined strategy. Models identified as hot experts—those with high activation frequencies or those frequently used recently—are loaded into the GPU's memory, where forward computation is performed to generate the hot expert's output.

[0126] Reference Figure 8A and Figure 8B The PNM device 920 is responsible for storing and computing cold experts. As an example, the PNM device 920 may include a storage module, a computing module, and an input / output module. The storage module can store cold expert models from a neural network model based on the MoE architecture. The cold expert outputs generated by the PNM device 920 can be transmitted to the GPU device 910 via the input / output module and aggregated with the outputs of hot experts. The computing module (e.g., refer to...) Figure 8BThe cold expert computation accelerometer (PNM) can perform calculations using a selected cold expert model from the cold expert models to obtain the cold expert model output. The computation module can be implemented through hardware components such as logic chips. During the computation of the neural network model, the computation module can also identify at least one cold expert model stored in the PNM device as a hot expert model and remove it from the PNM device.

[0127] When a cold expert stored in PNM device 920 is dynamically reclassified as a hot expert, that expert can be migrated from PNM device 920 to GPU device 910. For example, the input / output module of PNM device 920 can transfer the cold expert to GPU device 910. Conversely, when a hot expert stored in GPU device 910 is dynamically reclassified as a cold expert, that expert can be offloaded from GPU device 910 to PNM device 920.

[0128] The GPU device disclosed herein may include a storage module, a gating network, and a computing module. The storage module stores hot expert models; the gating network selects hot expert models to participate in computation; the input / output module transmits and receives expert models and transmits cold expert selection information provided by the gating network; the computing module performs computations using the selected hot expert models to obtain hot expert model outputs, and calculates the output of each transformation layer of the neural network model based on the hot expert model outputs, cold expert model outputs received from external devices, cold expert model outputs determined by the gating network, and the weights of the hot expert model outputs. Although not shown, the GPU device may also include an input / output module and a control module. The input / output module transmits cold expert selection information provided by the gating network to the PNM device. The control module coordinates with the hot and cold expert manager to efficiently manage and allocate hot and cold expert models, optimizing the use and performance of computing resources.

[0129] During the computation of the neural network model, the control module can identify at least one hot expert model stored in the GPU device as a cold expert model. Additionally, the control module can assign different PNM devices to multiple cold expert models as described above. As an example, different PNM devices can also be assigned to multiple cold expert models via a hot / cold expert manager deployed on the CPU device.

[0130] For details, please refer to Figure 8BThe attention vector from the GPU's self-attention module 821 is transmitted to the logic chip 832 of the CMM-DC. The logic chip 832 can use the cold expert model 831 stored in the CMM-DC to perform accelerated computation to obtain the output of the cold expert model (hereinafter referred to as cold expert). The output of the cold expert model can be transmitted to the GPU device 910. The expert computation module 822 in the GPU device 910 receives the attention vector output by the self-attention module 821 and performs feedforward computation using the hot expert 823 activated by the gating network to obtain the output of the hot expert model. Then, these hot expert model outputs can be weighted and aggregated with the cold expert model outputs from the logic chip 832 to generate the final output of the current Transformer layer, which is then passed to the next Transformer layer. Both the self-attention module 821 and the expert computation module 822 can be implemented through a combination of hardware and software. Specifically, the processor in the GPU device 910 can load software instructions corresponding to the self-attention computation and the hot expert computation to execute these operations or processes.

[0131] Additionally, refer to Figure 8B The Cooling and Hot Expert Manager 810 can operate independently of the GPU and CMM-DC; for example, it can be deployed on a host device. The Cooling and Hot Expert Manager 810 can adjust the storage location of the cooling and hot expert model between the CPU and the CMM-DC, as well as the storage location of the cold expert model across multiple CMM-DCs.

[0132] Therefore, the hot and cold expert manager can place expert models on GPUs and multiple CMM-DCs for specific models and / or user preferences, and dynamically adjust their positions to achieve load balancing based on expert activation.

[0133] Specifically, expert models can be stored on different devices based on their usage during model training. During inference, a hot and cold expert manager can track the activation status of each expert model to reflect its popularity. The positions of the expert models can then be dynamically adjusted.

[0134] Through the above scheme, CMM-DC1, CMM-DC2, CMM-DC3, etc., can work collaboratively. These CMM-DCs can be connected via a high-speed network, enabling rapid data transmission and sharing. When a cold expert on a CMM-DC is activated, other CMM-DCs can also provide necessary support. Therefore, by rationally allocating the computing and storage resources of hot and cold experts, combined with efficient management strategies, high-speed computing can be achieved.

[0135] It should be understood that the units, modules, etc., according to the exemplary embodiments of this disclosure can be implemented as hardware components and / or software components. Those skilled in the art can implement the respective units / modules, for example, using field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), according to the processes performed by the defined units / modules.

[0136] The acceleration method according to exemplary embodiments of this disclosure can be written as a computer program, code segment, instruction, or any combination thereof, and recorded, stored, or attached to one or more non-transitory computer-readable storage media. The computer-readable storage medium is any data storage device capable of storing data read by a computer system. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray and / or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0137] The acceleration method according to at least one example embodiment of this disclosure can provide higher memory capacity and lower total cost of ownership. Migrating cold expert models to PNM devices such as CMM-DC devices reduces the total cost of ownership by significantly lowering the storage cost per GB compared to the storage cost of high-bandwidth memory (HBM) for GPUs.

[0138] The acceleration method according to at least one example embodiment of the present disclosure can reduce the computational complexity of each expert model to be much lower than that of previous neural network models, offload the computation of cold expert models to PNM devices, thereby reducing the amount of data transmitted to the GPU and solving the data transmission bottleneck.

[0139] The acceleration method according to at least one example embodiment of this disclosure utilizes a cold expert computing accelerator to reduce computational complexity and a cold / hot expert manager to improve GPU utilization and accelerate parallel computing on PNM devices. The cold expert computing accelerator and the cold / hot expert manager can reduce overall inference time.

[0140] While specific terminology has been used to describe various embodiments of this disclosure, the specification and drawings are to be regarded as illustrative rather than restrictive in order to aid in understanding this disclosure. Various modifications and changes can be made by those skilled in the art (e.g., different features in different embodiments may be combined) without departing from the broader spirit and scope of this disclosure. Therefore, the scope of this disclosure is not limited by the specific embodiments and examples, but by the claims and their equivalents.

Claims

1. A method for accelerating a neural network model, characterized in that, The neural network model is a neural network model using the MoE architecture. Under the MoE architecture, based on the number of activations, the expert model is divided into a hot expert model and a cold expert model. The hot expert model is stored in an external device. The acceleration method is executed by the PNM device, and the acceleration method includes: The selected cold expert model from the cold expert models stored in the PNM device is used to perform calculations to obtain the cold expert model output; The cold expert model output from the PNM device, including a GPU device, is transmitted to the external device.

2. The method for accelerating a neural network model according to claim 1, characterized in that, The step of transmitting the output of the cold expert model includes: transmitting the output of the cold expert model to the GPU device without transmitting the model parameters of the cold expert model.

3. The method for accelerating a neural network model according to claim 1, characterized in that, The PNM device includes multiple PNM devices, and the cold expert model is multiple, with the multiple cold expert models stored separately on multiple PNM devices.

4. The method for accelerating a neural network model according to claim 3, characterized in that, Expert models with co-occurrence relationships in multiple cold expert models are stored separately on different PNM devices in multiple PNM devices.

5. The method for accelerating a neural network model according to claim 1, characterized in that, The acceleration method further includes: During the computation of the neural network model, at least one cold expert model stored in the PNM device is identified as a hot expert model; The at least one cold expert model is transmitted to the GPU device; Delete the at least one cold expert model from the PNM device; and The GPU device receives at least one expert model that is dynamically identified as a cold expert model during the computation of the neural network model. The at least one expert model is removed from the GPU device after being provided to the PNM device.

6. The method for accelerating a neural network model according to claim 1, characterized in that, The steps of performing calculations using a selected cold expert model from the cold expert models stored in the PNM device to obtain the cold expert model output include: Identify the dense components in the input data of the cold expert model; Identify important neurons in the cold expert model whose importance metric exceeds a preset threshold to obtain the output of the cold expert model. The cold expert model involved in the computation is determined based on cold expert selection information received from the external device. This cold expert selection information is provided by a gating network in the MoE architecture, which is deployed in the GPU device.

7. The method for accelerating a neural network model according to claim 1, characterized in that, The PNM device is a CMM-DC device, and the external device also includes a host device.

8. A PNM device, characterized in that, The PNM device includes: The storage module is configured to store cold expert models in a neural network model based on the MoE architecture, wherein the expert models of the MoE architecture are divided into hot expert models and cold expert models according to the number of times they are activated, and the hot expert models are stored in an external device. The calculation module is configured to perform calculations using the selected cold expert model from the cold expert models to obtain the cold expert model output; An input / output module is configured to transmit the output of the cold expert model to an external device, including a GPU device.

9. A method for accelerating a neural network model, characterized in that, The neural network model is a neural network model using the MoE architecture. Under the MoE architecture, expert models are divided into hot expert models and cold expert models based on the number of activations. The acceleration method includes: A thermal expert model is selected using a gating network stored in the GPU device; The thermal expert model is used to perform calculations to obtain the thermal expert model output. Based on the output of the thermal expert model, the output of the cold expert model received from an external device, and the weights determined by the gating network, the output of each transformation layer of the neural network model is calculated, wherein the external device includes a PNM device.

10. The method for accelerating a neural network model according to claim 9, characterized in that, The acceleration method further includes transmitting cold expert selection information to the PNM device, the cold expert selection information being provided by the gating network.

11. The method for accelerating a neural network model according to claim 9, characterized in that, The acceleration method further includes: During the computation of the neural network model, at least one hot expert model stored in the GPU device is identified as a cold expert model; Transmit the at least one thermal expert model to the PNM device; and Receive at least one expert model that is dynamically determined as a thermal expert model from the PNM device.

12. The method for accelerating a neural network model according to claim 9, characterized in that, The PNM device includes multiple PNM devices, and the cold expert model is multiple, with the multiple cold expert models stored separately in the multiple PNM devices. The acceleration method further includes: Identify co-occurrence relationships that reflect the activation of multiple cold expert models together; Obtain a graph comprising multiple nodes and edges, wherein the nodes of the graph represent expert models and the edges of the graph represent co-occurrence relationships; Calculate the sum of all weights for each node in a set of nodes; Sort the nodes by the sum of their weights; Assign different colors according to the order in which they are arranged; Different PNM devices are assigned to multiple cold expert models based on different colors.

13. The method for accelerating a neural network model according to claim 12, characterized in that, The steps for assigning different colors according to the order in which they are arranged include: For the first node among multiple nodes, collect the set of colors already colored for the neighboring nodes of the first node. Assign an unused color to the first node that is not included in the color set.

14. The method for accelerating a neural network model according to claim 13, characterized in that, If no color is available, it iterates through all colors and calculates the influence factor for each color, and assigns the color with the smallest influence factor to the first node.

15. A GPU device, characterized in that, The GPU device includes: The storage module stores the hot expert model of the neural network model using the MoE architecture. Under the MoE architecture, the expert model is divided into the hot expert model and the cold expert model according to the number of times it is activated. Gated networks select thermal expert models to participate in the computation. The computation module performs calculations using a selected thermal expert model to obtain the thermal expert model output, and calculates the output of each transformation layer of the neural network model based on the thermal expert model output, the cold expert model output received from an external device, and the weights of the cold expert model output and the thermal expert model output determined by the gating network. The external device includes a PNM device.

16. A method for managing neural network models, characterized in that, The neural network model is a neural network model using the MoE architecture, and the management method is executed by the GPU device or the host device. The management method includes: Determine the usage of the expert model under the MoE architecture during the training process of the neural network model; Based on the usage, the expert model is pre-divided into a cold expert model and a hot expert model. The cold expert model will be pre-stored in the PNM device, and the hot expert model will be pre-stored in the GPU device.

17. The method for managing neural network models according to claim 16, characterized in that, The management method also includes: During the inference calculation process of the neural network model, the activation status of the expert model is statistically analyzed; Based on the activation status of the expert models, it is dynamically determined whether each expert model in the expert models is a hot expert model or a cold expert model; The position of the expert model is dynamically adjusted between the GPU device and the PNM device.

18. A computer system, characterized in that, The computer system includes the PNM device according to claim 8 and the GPU device according to claim 15.

19. A host device, characterized in that, The host device includes: A non-transitory computer-readable recording medium configured to store instructions. The processor is configured to execute the instructions to cause the processor to perform: Identify co-occurrence relationships that reflect the activation of multiple cold expert models together; Obtain a graph comprising multiple nodes and edges, wherein the nodes of the graph represent expert models and the edges of the graph represent co-occurrence relationships; Calculate the sum of all weights for each node in a set of nodes; Sort the nodes by the sum of their weights; Assign different colors according to the order in which they are arranged; Different PNM devices are assigned to multiple cold expert models based on different colors.

Images