Function-Based Activation of Memory Hierarchies

Abstract

A 3D compute in-memory accelerator system and method for efficient inference of Mixture of Experts (MoE) neural network models are provided. The system includes multiple compute in-memory cores, each including multiple hierarchies of in-memory computational cells. One or more hierarchies of in-memory computational cells correspond to expert submodels of the MoE model. One or more expert submodels are selected for activation propagation based on a function-based routing, and the corresponding hierarchies of experts are activated based on the function. In one embodiment, the function is a hash-based hierarchical selection function used for dynamic routing of input and output activations. In one embodiment, the function is applied to select a single expert or multiple experts using an input database or layer activation-based MoE for activation of a single hierarchical layer. Furthermore, the system can be configured as a multi-model system with single expert model selection or a multi-model system with multiple expert selection.

Description

[Technical Field]

[0001] The present disclosure relates to deep learning machine learning models, and more specifically to a neural network model system that includes a hierarchy of compute-in-memory structures configured to implement multiple expert sub-models, denoted as a Mixture of Experts (MoE), for performing fast and energy-efficient inference on models with billions of parameters. [Background technology]

[0002] Deep learning refers to the use of multi-layer machine learning model architectures that use multiple layers to progressively extract high-level features from raw inputs. These machine learning models are typically artificial neural networks, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), trained with representation learning for use in artificial intelligence applications. Learning can be supervised, semi-supervised, or unsupervised.

[0003] It has been shown that the performance of modern deep learning architectures improves simply by scaling up the model size (i.e., the number of parameters) and dataset size. This framework faces two main challenges: training and inference for models of this size is extremely expensive (on the order of millions to train a single model); and models with many parameters require large amounts of DRAM, preventing fast, energy-efficient inference.

[0004] The first challenge is typically addressed via a mixture of experts (MoE) system. Mixture of experts is an ensemble learning method in the field of neural networks, known for its ability to scale models to billions of parameters without incurring additional computational costs. In an MoE model, a predictive modeling task is decomposed into subtasks, and an expert model is trained for each subtask. The main aim of an MoE system is to route data fed into a neural network to expert subjects. That is, a mixture of experts system implements a routing algorithm that routes hidden layer activations to designated experts for forward computation. This allows the number of experts to be scaled arbitrarily by simply increasing the number of experts while keeping computational complexity low. Computational complexity remains low because, for example, only one expert is selected, rather than all experts.

[0005] Despite the advantages that MoE systems bring, current architectures still require a huge amount of parameters, preventing fast and energy-efficient inference.

[0006] The main reason inference for large models is slow is the need to constantly pass weights between memory and the GPU / CPU, or the need for very large amounts of dynamic random access memory (DRAM) to store all the necessary weights. MoE systems alleviate this problem to some extent because only a subset of experts are active at any one time, and therefore not all weights need to be used. However, most systems are so large that using MoE does not eliminate the problem. Summary of the Invention

[0007] A system, method, and computer program product are provided for performing fast and energy-efficient inference for MoE systems by performing computations in-memory using a 3D memory architecture.

[0008] The present disclosure relates to deep learning machine learning models, and more particularly to a neural network model system including a hierarchy of compute-in-memory structures configured to implement multiple expert sub-models, denoted as a mixture of experts (MoE), with circuitry that provides dynamic routing using function-based hierarchical selection.

[0009] In one embodiment, fast inference in the MoE system using a compute-in-memory structure implements a hash function.

[0010] In one aspect, a three-dimensional compute-in-memory (CiM) system for efficient inference of a mixture of experts (MoE) model is provided, comprising: a 3D memory system having multiple tiers of compute-in-memory devices, where each tier includes a memory system associated with an expert of a mixture of experts (MoE) neural network model; a control processor for selecting, in response to received input, one or more tiers of the compute-in-memory devices associated with an expert in the network model layer; and the control processor configuring a computation path in the selected one or more of the multiple tiers associated with the expert, the computation path processing the input at the network model layer and generating one or more output activations for use in a further neural network model layer.

[0011] In one embodiment, the 3D memory system is a non-volatile memory system, such as a resistive memory (memristor) system or a 3D NAND flash memory, although it will be appreciated that other types of non-volatile memory systems may be employed.

[0012] In one embodiment, when running a deep neural network model, an input-based or layer activation-based function is used to select an MoE expert for neural network processing. If the MoE allows for the selection of a computational path via a function, and thus the expert to be used is determined using the input, the tiles / layers to be activated can be pre-selected. Furthermore, different layers hold different expert weights, and layers are dynamically switched on depending on the input / intermediate activations. If layer activations are to be used, the expert selected to process these activations can be determined only when these activations arrive.

[0013] The use of input-based or layer activation-based functions allows the selection of single or multiple experts using input-based MoEs or single or multiple experts using layer activation-based MoEs for activation of single or multiple layers.

[0014] The provided 3D CiM system can include crossbar memory, which acts as an accelerator to boost the performance of matrix-vector operations during processing of deep neural network models.

[0015] In a further aspect, a 3D Compute-in-Memory (CiM) system is provided. The 3D CiM system includes a plurality of processing tiles, where each of one or more processing tiles corresponds to a layer of a deep neural network model, and the layers receive an input dataset for inference processing using the deep neural network model. The processing tiles include a three-dimensional (3D) multi-tier configuration of compute-in-memory devices, where each tier includes a crossbar array of memory cells including a memory system associated with a neural network model processing operation. The 3D CiM system also includes a control processor circuit for selecting one or more tiers of compute-in-memory devices associated with the neural network model in response to the received input dataset, where the control processor circuit configures a computation path in one or more of the multiple tiers associated with a selected expert, the computation path processes the received input with the network model, and generates one or more outputs for use by a CiM processing unit associated with a further layer of the neural network model.

[0016] Furthermore, Compute-in-Memory (CiM) systems can be configured as multi-model systems with a single expert choice or multiple expert choices.

[0017] In yet another aspect, a method for operating a compute-in-memory system is provided, comprising: configuring one or more processing tiles based on received input data, each processing tile associated with a layer of a neural network model, each processing tile including a three-dimensional (3D) multi-tier configuration of compute-in-memory devices configured to process data associated with an expert of a mixed-experts neural network model; and a control processor circuit for iteratively selecting one or more tiers of compute-in-memory devices associated with the expert in response to a received input data set. The method further comprises selecting one or more tiers of compute-in-memory devices associated with the expert in the network model layer in response to the received input, and configuring, using the control processor, a computational path at one or more selected tiers of the plurality of tiers associated with the expert, wherein the computational path processes the input at the MoE network model layer and generates one or more output activations for use in a further neural network model layer.

[0018] Further features, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings, where like reference numbers indicate identical or functionally similar elements. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram of an exemplary Compute-in-Memory (CiM) hardware and memory system accelerator implementing the MoE neural network model architecture, according to an embodiment of the present invention;

[0020] [Figure 2A]1 illustrates the operations and signal / data flow involved by tiles in a 3D Compute-in-Memory (CiM) hardware and memory system accelerator tile implementing processes associated with the MoE deep neural network model architecture, according to an embodiment of the present invention; [Figure 2B] 1 illustrates the operations and signal / data flow involved by tiles in a 3D Compute-in-Memory (CiM) hardware and memory system accelerator tile implementing processes associated with the MoE deep neural network model architecture, according to an embodiment of the present invention; [Figure 2C] 1 illustrates the operations and signal / data flow involved by tiles in a 3D Compute-in-Memory (CiM) hardware and memory system accelerator tile implementing processes associated with the MoE deep neural network model architecture, according to an embodiment of the present invention; [Figure 2D] 1 illustrates the operations and signal / data flow involved by tiles in a 3D Compute-in-Memory (CiM) hardware and memory system accelerator tile implementing processes associated with the MoE deep neural network model architecture, according to an embodiment of the present invention; [Figure 2E] 1 illustrates the operations and signal / data flow involved by tiles in a 3D Compute-in-Memory (CiM) hardware and memory system accelerator tile implementing processes associated with the MoE deep neural network model architecture, according to an embodiment of the present invention;

[0021] [Figure 3] 1 illustrates one embodiment of a hierarchy activation circuit that selects a word line by generating a word line (WL) signal to activate a CiM memory hierarchy corresponding to a selected expert of the MoE model, the selected expert of the MoE model being present in a single or multiple hierarchies;

[0022] [Figure 4] shows a first embodiment of a 3D CiM accelerator system configured to select a single expert using input-based MoE for single or multiple tier activation;

[0023] [Figure 5] Conceptually demonstrate the process in the CiM accelerator system for selecting a single expert using input-based MoE for activation of a single hierarchy;

[0024] [Figure 6] 1 shows a second embodiment of a 3D CiM accelerator system configured to select a single expert using layer activation-based MoE for activation of a single layer;

[0025] [Figure 7] Fig. 6 conceptually illustrates the process in the CiM accelerator system for selecting a single expert using layer activation-based MoE for the activation of a single layer;

[0026] [Figure 8A] 1 illustrates a further embodiment of a 3D CiM accelerator system configured to select multiple experts using input-based MoE for a single tier of activation; [Figure 8B] 1 illustrates a further embodiment of a 3D CiM accelerator system configured to select multiple experts using input-based MoE for a single tier of activation;

[0027] [Figure 9]Shown is a multi-expert 3D CiM accelerator system tile containing a stack of CiM hierarchies implementing one or more expert models corresponding to an MoE feed-forward network (FFN);

[0028] [Figure 10A] 1 illustrates a further embodiment of a 3D CiM accelerator system configured to select multiple experts using activation-based MoE for a single tier of activation; [Figure 10B] 1 illustrates a further embodiment of a 3D CiM accelerator system configured to select multiple experts using activation-based MoE for a single tier of activation;

[0029] [Figure 11] 1 illustrates an embodiment of a 3D CiM accelerator system configured to implement multiple different models and to select a single model of a multi-model system;

[0030] [Figure 12] Conceptually illustrates the process in the multi-model 3D CiM accelerator system in Fig. 11 for selecting a single model of a multi-model system;

[0031] [Figure 13A] shows a further embodiment of a 3D CiM accelerator system configured to select multiple experts (models) present at different tiers of a 3D multi-model CiM system; [Figure 13B] shows a further embodiment of a 3D CiM accelerator system configured to select multiple experts (models) present at different tiers of a 3D multi-model CiM system;

[0032] [Figure 14]illustrates a method implemented in any processing block in or associated with a virtual tile of the 3D CiM MoE accelerator system of various embodiments; and

[0033] [Figure 15] 1 illustrates an exemplary computing system for controlling dynamic MoE switching in various embodiments of a 3D CiM accelerator system. DETAILED DESCRIPTION OF THE INVENTION

[0034] Research has shown that the performance of current AI models improves by scaling model and dataset sizes. However, this creates problems on traditional hardware (e.g., CPUs, GPUs) as large models cannot be trained without sparsification.

[0035] The main reason inference for large models is slow is the need to constantly pass weights between memory and the GPU / CPU, or the need for very large amounts of dynamic random access memory (DRAM) to store all the necessary weights. MoE systems alleviate this problem to some extent because only a subset of experts are active at any one time, and therefore not all weights need to be used. However, most systems are so large that using MoE does not eliminate the problem.

[0036] In particular, convolution-based AI systems require significant improvements in computational speed and throughput, and in-memory computing is one approach that can be used to accelerate deep learning inference and training.

[0037] FIG. 1 shows a compute-in-memory (CiM) accelerator system 10 that enables fast and energy-efficient inference for MoE systems by performing deep learning neural network computations in-memory using a three-dimensional (3D) non-volatile memory architecture.

[0038] As shown in FIG. 1 , the CiM accelerator system 10 includes one or more digital processors 15 that issue control signals for controlling the operation of many applications involving data stored in a non-volatile memory (NVM) subsystem 20, including memory devices such as DRAM. A system communication bus 12 is provided for passing data between the memory 20 and the computational unit 15 when performing operations such as Kalman filtering, fast Fourier transforms, and linear equation solvers. According to certain aspects of the present disclosure, the system communication bus 12 is further connected to a CiM device system 25 having a control unit, such as a microprocessor 28, and multiple CiM tiles 40, each of which includes a 3D compute-in-memory block including multiple layers 45 of 3D CiM memory devices and associated computational circuitry used to control neural network operations, such as in-place matrix-vector multiply (MVM) operations, in the layers. The microprocessor 28 can be configured to manage computational paths for performing CiM MoE neural network operations, such as input / output and / or computations, involving the layers in the tiles. In one embodiment, depending on the configuration of the MoE model, output data such as intermediate activations generated as a result of MVM operations performed in a layer of a tile, such as tile 41, may be controlled for input / transfer to another CiM device corresponding to the same or a different expert, e.g., a layer in the same tile or a different tile (e.g., tile 42), via data transfer using local data bus 47. In one embodiment, layer activations are inputs to multiple different layers of a neural network model and are typically vectors of floating-point / integer elements.

[0039] As shown in FIG. 1, each 3D CiM device layer in a tile 40 can be implemented as a resistive memory array, e.g., by leveraging analog memory capabilities and Kirchhoff's circuit laws.

number

number

number

number

number

number

number

number

[0040] Each 3D CiM device in tile 40 provides large weight storage capacity for densely storing millions of parameters per CiM tile, enabling efficient and fast inference (and potentially training) of models of billion-parameter size in multi-tile CiM accelerators.

[0041] In one embodiment, the present disclosure proposes to implement large-scale models using the MoE architecture by programming each of the experts into different tiers of the 3D memory.

[0042] As used herein, the term "virtual tile" refers to a tile in which an expert model resides. A virtual tile spans one or more physical tiles 40. Each physical tile 40 has multiple layers 45 that form a 3D CiM system. Because an expert can have multiple neural network layers, these layers are mapped to layers within different physical tiles. For example, an expert model with two layers may be mapped to the first layer (layer 1) of a first tile and the first layer (layer 2) of a second tile. Additionally, two experts may reside in a virtual tile, e.g., a first expert residing in layer 1 of a first physical tile (e.g., the first layer of the expert model) and layer 1 of a second physical tile (e.g., the second layer of the expert model), and a second expert residing in the second layer of the first physical tile, e.g., layer 2 (e.g., the second expert of one neural network model layer). The output of the experts is transmitted to the next layer.

[0043] Given newly arriving data, a layer of the virtual tile can be appropriately selected to process the data. Generally, when input data arrives at a virtual tile (e.g., in the form of hidden layer activations from a previous layer), an expert is selected. This method activates a corresponding layer in each physical tile in the virtual tile depending on the expert index. For example, in response to input data, expert selection selects expert 1, which then activates layer 1 in both physical tiles. Data is first sent to layer 1 in physical tile 1, then to layer 1 in physical tile 2. The output of the virtual tile is then sent to the next layer in the neural network.

[0044] In the MoE model, hidden activations generated in the neural network model layer are

number

number

number

number

number

number

number

number

number

number

[0045] 2A-2E illustrate the operations and signal / data flow involved in an exemplary 3D Compute-in-Memory (CiM) hardware and memory system accelerator 100 in a hierarchy of virtual tiles implementing an MoE neural network model architecture, according to one embodiment of the present invention.

[0046] In an embodiment, the compute in-memory system accelerator 100 configures multiple in-memory cores, each with more than one tier of in-memory computational cells, with more than one expert present in the tier. This configuration includes selecting one or more experts for activation propagation based on a function, such as a hash function. In one embodiment, based on this function, the corresponding tier of experts is activated. That is, the MoE enables the selection of a computation path via the function f(), thus pre-selecting the tier to be activated. Different tiers have different expert weights, and tiers are dynamically switched on depending on input / intermediate activations.

[0047] FIG. 2A particularly illustrates a compute in-memory accelerator system 100 including tiles with multiple in-memory cores that employ function-based activation of memory hierarchies for efficient inference of mixture of experts (MoE) models.

[0048] In FIG. 2A , a CiM accelerator system 100 includes multiple tiles 102, each corresponding to a successive layer of a deep neural network (DNN). For non-limiting illustrative purposes, FIG. 2A illustrates a hypothetical tile including a sequence of tiles T−1, T, and T+1. In one embodiment, each tile 102 includes components including a compute-in-memory (CiM) structure that forms a three-dimensional compute-in-memory system 105 used to accelerate deep learning inference and training. Each three-dimensional compute-in-memory system 105 includes multiple memory cell tiers 106, each tier 106 having an addressable two-dimensional CiM array including a crossbar configuration of memory cells 50 for handling neural network processes at a particular neural network layer. In one embodiment, the memory cells can store weights associated with a particular neural network model and / or MoE. In one embodiment, the weights reside entirely within the 3D memory architecture. Because the weights are all present in the 3D CiM architecture and the method performs computations in-memory, the bottleneck of passing data between memory and computation units is completely eliminated. The compute-in-memory accelerator system 100 of Figure 2A consequently enables fast and highly energy-efficient inference of models containing billions of parameters.

[0049] As shown in FIG. 2A, the first step in the processing method is the arrival of data 125 at tile T in the expert selection function logic 110. In one embodiment, the input data 125 arrives from another tile / processing block and includes data “x” that includes a vector of floating-point / integer elements. However, the input data 125 may also be a single number (e.g., a vector with a single element). For example, a programmer / user may specify a desire to use the first expert / layer of the MoE model. In one embodiment, the input data 125 may be intermediate activations “h” received from a previous layer, i.e., from the same or another layer / processing block. For example, as shown in FIG. 2A, data 125 generated by scaling and accumulating (gating) circuitry 120 in the previous tile T-1 (previous DNN model layer) is received by the expert selection function logic 110 for processing in tile T (e.g., the next DNN model layer).

[0050] 2B, an expert selection function f() is executed in the expert selection function circuit 110. Execution of the expert selection function f() results in the generation of one or more expert indices and corresponding gating values. The expert indices are used as inputs to corresponding hierarchy activation circuits 115, which are used to generate word line signals used to select a particular CiM hierarchy 106 of the determined expert model. That is, in the CiM accelerator system 100, the CiM structure 105 in a tile 102 includes word line drivers WL0, WL1, ..., WL2, ..., WL3, ..., WL4, ..., WL5, ..., WL6, ..., WL7, ..., WL8, ..., WL9, ..., WL10, ..., WL11, ..., WL12, ..., WL13, ..., WL14, ..., WL15, ..., WL16, ..., WL17, ..., WL18, ..., WL19, ..., WL20, ..., WL21, ..., WL22, ..., WL33, ..., WL44, ..., WL55, ..., WL66, ..., WL70, ..., WL81, ..., WL90, ..., WL102, ..., WL113, ..., WL124, ..., WL135, ..., WL146, ..., WL157, ..., WL167, ..., WL178, ..., WL189, ..., WL200, ..., WL2112, ..., WL225, ..., WL230, ..., WL310, ..., WL420, ..., WL430, ..., WL540, ..., WL650, ..., WL760, ..., WL870, ..., WL90, ..., WL191, ..., WL192, ... K-1 Associated with the layer 106 is an associated layer activation circuit 115 based on a word line driver (WLD) 113 denoted as . Different layers 106 can have different expert weights since the layers can be associated with different MoE models.

[0051] The layer activation circuit 115 includes functionality for dynamically switching (activating) layers. For example, layers are dynamically switched on depending on the received input x / intermediate activation h. In one embodiment, the MoE model enables computational path selection via an expert selection function “f” circuit 115, which implements logic for selecting / activating the CiM layers 106. In one embodiment, the expert selection function “f” circuit 115 implements logic that can select and activate layers in advance. One example of this function “f” is a hash function. Another example of f(x) may be a softmax function followed by an argmax function, such as argmax(softmax(x*fixed_weight_matrix)), where the fixed weight matrix is ​​a matrix learned in a different process and is fixed, in the sense that it is determined before training the model, for example. As an example, a multilayer neural network may use multiple weight matrices chained with some nonlinearity.

[0052] As further shown in system 100, the CiM system 105 in the processing tile 102 has associated therewith gating circuitry 120 and peripheral circuitry 107 that can function to scale and accumulate DNN model layer outputs. The peripheral circuitry 107 can include analog-to-digital converters, digital-to-analog converters, registers, memory, buffers, filters, etc. for performing neural network processing operations in the tile.

[0053] As a result, as shown in third processing step 135 of Figure 2C, based on the expert index 132 from the expert selection function logic circuit 110, the hierarchy activation circuitry 115 activates the target hierarchy. For example, as shown in Figure 2C, in response to input 132, memory hierarchy word line WL0111 is set to a high voltage value to activate the expert model in the corresponding CiM memory hierarchy 106A for use in calculating and generating the model layer output activations.

[0054] 2D illustrates a fourth processing step in system 100, in which gating circuitry 120 receives gating values ​​138 generated by expert selection function logic 110 and analog-to-digital converted intermediate output activations 140. Gating circuitry 120 uses gating values ​​138 to process and scale model layer outputs with experts in tiers, such as tier 106A, before accumulating them.

[0055] To select other experts within the MoE model, further processing steps are performed in a similar manner. For example, as shown in FIG. 2E, based on further input 133 from expert selection function logic 110, hierarchy activation circuitry 115 activates another (second or further) target hierarchy. For example, as shown in FIG. 2E, memory hierarchy wordline WL1121 corresponding to second CiM memory hierarchy 106B is set to a high voltage, activating hierarchy 106B and generating further intermediate output activations 141. In each further processing step shown in FIG. 2E, gating values ​​139 generated by expert selection function logic 110 are applied to output activations 141 that have undergone analog-to-digital conversion. The results are stored in gating circuitry 120, which scales and stores the model layer output. In some embodiments, hierarchy 106A, 106B may be associated with the same MoE expert or different experts.

[0056] After all other selected experts have been run in a similar manner as shown in Figures 2A-2E, intermediate output activations 142 are calculated and these activations are navigated to the next network layer, which may reside in one or more tiles, for example, propagated as input activations 143 to the next virtual tile T+1 102, for processing in the subsequent neural network layers.

[0057] 3 illustrates one embodiment of hierarchy activation circuitry 115 that generates word lines WL to activate the CiM memory hierarchy 106 corresponding to the MoE model expert. In one embodiment, hierarchy activation circuitry 115 generates word line pulses WL0, WL1, ..., WL K-1 In one embodiment, a word line (WLD) pulse 210 is generated for a matrix vector multiplication operation in hierarchical activation logic applied in hierarchical activation circuitry 115.

[0058] In one embodiment, the pulse generator circuit 200 is configured to receive digital input data (e.g., floating-point / integer) and convert it into a waveform, which serves as the input for an analog matrix-vector multiplication. In one example, the duration of the waveform 210 represents the input number. As a non-limiting example, assuming the input number can take values ​​from 0 to 128 and the maximum possible duration is 128 ns, the number 64 can be represented as a pulse with a fixed amplitude and a duration of 64 ns. An alternative is to modulate the amplitude with a fixed duration. Assume the amplitude can be modulated between 0 V and 0.5 V. Then, the number 64 can be represented by a pulse amplitude of 0.25 V.

[0059] In operation, pulse generation circuit 200 receives input data, e.g., a vector of values, and generates output waveform pulse 210. In one example, output waveform pulse 210 is of a pulse duration proportional to the time-sliced ​​n-bit enables. Output waveform pulse 210 is input to 1×K switch / demultiplexer 220, which

number

[0060] Referring to FIG. 4, in a first embodiment, the 3D CiM accelerator system 300 is configured to select a single expert using an input-based MoE for activation of a single layer or multiple layers. In this input-based MoE embodiment, inputs to the model are used / processed to define which experts / layers will be activated. The inputs to the model are received at the start of model inference, so that experts / layers for all layers can be predetermined as needed. In one embodiment, it is assumed that there is one expert activated per layer.

[0061] As shown in FIG. 4, for example, any processing block 310 in the virtual tile 202 may

number

number

number

number

number

number

[0062] 5 conceptually illustrates the process in the CiM accelerator system 300 for selecting a single expert using input-based MoE for activation of a single layer. In FIG. 5, the MoE expert system 400 selects an expert from the intermediate activation sequence received at the layer.

number

number

number

number

number

number

number

number

number

[0063] Referring back to FIG. 4, the expert index 325 is the input for all tiles, e.g., tile T+1.

number

number

[0064] Referring to FIG. 6, in a second embodiment, the 3D CiM accelerator system 350 is configured to select a single expert using a layer activation-based MoE for the activation of a single layer. In this layer activation-based MoE embodiment, activations received from a previous layer are used / processed to determine which expert / layer will be activated. In this embodiment, the expert index is calculated dynamically / on-the-fly; or, in other words, the expert index is not known before the previous layer is executed. In one embodiment, it is assumed that there is one expert activated per layer.

[0065] As shown in FIG. 6, the expert selection function logic processing block 360 selects the received activation sequence

number

number

number

number

number

[0066] 7 conceptually illustrates the process in the CiM accelerator system 350 of FIG. 6 for selecting a single expert using layer activation-based MoE for activation of a single layer. In FIG. 7, the MoE expert system 500 selects an expert for the intermediate activation sequence.

number

[0067] In the embodiment of Figures 6 and 7, the expert selection function applied

number

number

number

number

number

number

number

number

[0068] Referring back to Figure 6, the expert index 375 is the previous layer activation for the current tile, such as tile T.

number

[0069] 8A and 8B illustrate a further embodiment of a 3D CiM accelerator system 600 configured to select multiple experts using an input-based MoE for activation of a single layer. In this input-based MoE embodiment, input to the model is used / processed to define the multiple experts / layers that will be activated. The input to the model is received at the start of model inference, so the experts / layers for all layers can be predetermined as needed. In some embodiments, more than one layer / expert is utilized in each tile. Each expert contributes to the tile's output in proportion to its gating value. Which layer / expert will be used for a layer is determined according to the model input.

[0070] As shown in FIG. 8A, for example, any processing block 610 in the virtual tile 202 may

number

number

number

number

number

number

number

[0000] , is used to sequentially activate expert models in one or more layers to process the current input data. In one embodiment, the same input is passed through different experts / layers. Processing in the layers produces further intermediate output activations 641 that are scaled, accumulated with previous outputs, and stored for use in the next iteration of the input sequence.

[0071] As a result, in the next processing sequence of sequentially activating expert models in one or more layers as shown in FIG. 8B, the next (e.g., second) of the generated expert indexes, i.e., f(x l In response to )2, hierarchical activation circuitry 615 generates a next word line signal, such as word line 612 (e.g., WL3), to activate a further hierarchical layer associated with the second MoE expert, e.g., hierarchical layer 616B shown in FIG. 8B, to process / generate further intermediate output activations 643. FIG. 8B also shows a corresponding generated gating value 649 generated by the expert selection function logic in response to a second input x2 of the input sequence received at optional processing block 610 for application to output activations 643 after they have been converted from analog to digital. This result is propagated to and accumulated in gating circuitry 620, which scales, accumulates, and stores this model layer output. As a result, in layer L, input x_1 generates activation (h_1) L Generate (h_1) L The expert to process is determined based on x_1. For example, given experts 1 and 2 are selected. Then, (h_1) L is first passed through Expert 1. The result is multiplied by the gating value and stored. Then the same activation (h_1) L is passed through Expert 2. The result is multiplied by the corresponding gating value and added to the stored output. The final output is then sent to the next layer / tile.

[0072] In this further embodiment, the process shown in FIGS. 8A and 8B is iterative, i.e., the expert selection function logic applied at any processing block 610 and the input sequence x l The process is repeated for as many experts / layers as required. In each iteration, the generated output activations, which are scaled and accumulated in the gating circuitry 620, are sent to the next layer in the tile 602 or another physical tile in the virtual tile, which then processes the input x l The index generated in response to l ) output sequence. Furthermore, in this embodiment, the gating value is determined by the expert selection function logic. A non-limiting example of the gating function is softmax(input*fixed_weight_matrix), where fixed_weight_matrix represents the matrix learned in different processes.

[0073] In this further embodiment, the iterative process shown in FIGS. 8A and 8B illustrates that for each input

number

number

number

number

[0074] where the gating function is the activation

number

number

number

[0075] In this embodiment, the input sequence

number

number

number

number

number

number

number

number

[0076] The Multi-Expert 3D CiM Accelerator System tile 603 shown in Figure 9 shows a stack of CiM layers 616 that implement one or more expert models corresponding to the MoE FFN.

number

number

number

number

number

number

number

number

number

number

number

number

[0077] 10A and 10B illustrate a further embodiment of a 3D CiM accelerator system 700 configured to select multiple experts using activation-based MoE for the activation of a single layer. In this activation-based MoE embodiment, activations received from a previous layer are used / processed to determine which expert / layer of a tile 702 will be active. In this embodiment, the expert index is calculated dynamically / on-the-fly; i.e., the expert index is not known before the previous layer is executed. In some embodiments, a single expert is activated per layer. In some embodiments, more than one layer / expert is utilized in each tile. Each expert contributes to the tile's output in proportion to its gating value. Which layer / expert will be used for a layer is determined according to the input activations.

[0078] As shown in FIG. 10A, the expert selection function logic processing circuit 660 in the tile 702 selects the received activation sequence resulting from processing in the previous layer, for example.

number

number

number

number

number

number

[0079] For example, the first generated expert index, e.g., f(h l In response to )1, hierarchical activation circuit 715 generates a first word line signal, such as word line 711 (e.g., WL2), to activate a first hierarchical layer associated with the first MoE expert, e.g., hierarchical layer 716A shown in CiM stack 105 of FIG. 10A, to process / generate further output activations 741. FIG. 10A also shows a corresponding generated gating value 739 generated by the expert selection function logic in response to a first input h1 of the input activations received at expert selection function logic processing block 660 for application to the output activations 741 after they have been converted from analog to digital. This result is propagated to and accumulated in gating circuitry 720, which scales and accumulates this model layer output. 10A and 10B, the generated output activations scaled and accumulated in the gating circuitry 720 are transmitted to the next tier in the tile 702, which is activated based on the next expert index, i.e., f(h2), generated in response to the second input activation h2 of the input sequence received in the expert selection processing block 660. That is, in this embodiment, the current input activation sequence

number

number

[0080] As a result, in the next processing sequence of sequentially activating expert models in one or more layers as shown in FIG. 10B, the next (e.g., second) of the generated expert indexes, i.e., f(h l In response to )2, the hierarchical activation circuit 715 generates a next word line signal, such as word line 712 (e.g., WL3), to activate a further hierarchical layer associated with the second MoE expert, e.g., hierarchical layer 716B shown in FIG. 10B, to activate the next iteration of the input activation sequence (e.g., the next input h l) for use in the expert selection function logic 660. Figure 10B also shows the corresponding generated gating values ​​749 that are generated by the expert selection function logic in response to a second set of input activations h2 received at the expert selection processing block 660 for application to the output activations 743 after they have been converted from analog to digital. This result is propagated to and accumulated in the gating circuitry 620, which scales and accumulates this model layer output.

[0081] In this further embodiment, the process shown in FIGS. 10A and 10B is iterative, i.e., the expert selection function logic and input activation sequence h applied at process block 660 are l The process is repeated for as many experts / layers as required by the gating circuitry 620. In each iteration, the generated output activations, which are scaled, accumulated, and stored in the gating circuitry 620, may be transmitted to the next layer in the tile 602 or another physical tile, which then selects the input activations h of the input sequence received in the expert selection processing block 660. l The index generated in response to l ) output sequence. Furthermore, in this embodiment, the gating value is determined by the expert selection function logic. A non-limiting example of the gating function is softmax(input*fixed_weight_matrix), where fixed_weight_matrix represents the matrix learned in different processes.

[0082] In this further embodiment, the iterative process shown in FIGS. 10A and 10B illustrates that each input activation

number

number

number

number

[0083] where the gating function is the activation

number

number

number

[0084] In a further embodiment, the 3D CiM accelerator system implements multiple models that exist at different hierarchical levels.

[0085] Referring to FIG. 11 , in a further embodiment, the 3D CiM accelerator system 800 is configured to implement multiple different models and select a single expert for the multi-model system. In this input-based multi-model system, inputs to the models are used / processed to define which expert / tier will be active. The inputs to the models are received at the start of model inference, so that the experts / tiers for all tiers can be predetermined as needed. In one embodiment, it is assumed that there is one model activated per tier.

[0086] As shown in FIG. 11, for example, any processing block 810 in the virtual tile 202 may

number

number

number

number

number

number

[0087] Figure 12 conceptually illustrates the process in the multi-model 3D CiM accelerator system 800 of Figure 11 for selecting a single expert for a multi-model system.

number

number

number

number

number

number

number

number

number

number

number

[0088] Referring back to FIG. 11, in one embodiment, the input sequence

number

number

number

number

number

[0089] Generated expert index 825 is the input for all tiles

number

number

[0090] 13A and 13B show a further embodiment of a 3D CiM accelerator system 850 configured to select multiple experts (models) present at different tiers of a 3D multi-model CiM system. In this activation-based MoE embodiment, activations received from previous tiers are used / processed to determine which expert / tier of a tile 702 will be active. In this embodiment, the expert index is calculated dynamically / on-the-fly. Here, the method can select a model according to user input and activate the tier within each tile corresponding to the particular model.

[0091] 13A, in a further embodiment, the 3D CiM accelerator system 850 is configured to implement multiple different models (experts) and select multiple experts for the multi-model system. In this input-based multi-model system, the input sequence

number

number

number

number

number

[0092] As shown in FIG. 13A, an optional processing block 810 processes an input sequence

number

number

number

number

number

number

number

number

number

number

number

[0093] As a result, in the next processing sequence of sequentially activating expert models in one or more layers as shown in FIG. 13B, the next (e.g., second) of the generated expert indexes, i.e., f(x l In response to WL2, the hierarchy activation circuit 815 generates a next word line signal, such as word line 862 (e.g., WL3), to activate a further hierarchy associated with the second model, e.g., hierarchy 816B shown in FIG. 13B, to process the input data. Additionally, via data flow 830, the expert model selection output

number

number

[0094] In this further embodiment, the process shown in Figures 13A and 13B is iterative, i.e., the expert selection function logic applied at any processing block 810 and the input sequence x l The process is repeated for as many models / layers as required by the input x th input sequence received at any processing block 810. In each iteration, the generated output activations, which are scaled and accumulated in the gating circuitry 820, are transmitted to the next layer in tile 852 or another physical tile 853, which then scales the input x th input sequence received at any processing block 810. l The index generated in response to l ) output sequence. Furthermore, in this embodiment, the gating value is determined by the expert selection function logic. A non-limiting example of the gating function is softmax(input*fixed_weight_matrix), where fixed_weight_matrix represents the matrix learned in different processes.

[0095] In this further embodiment, the iterative process shown in FIGS. 13A and 13B illustrates that each input

number

number

number

number

[0096] where the gating function is the activation

number

number

[0097] FIG. 14 illustrates a method 900 implemented in any processing block in or associated with a tile of a 3D CiM MoE accelerator system. Such a method may be implemented, for example, by a control process implementing expert selection function logic in a tile or by a control process implementing expert selection function logic in any processing block, such as in the embodiments illustrated in FIGS. 4, 8A, 8B, 11, etc. Method 900 begins at 902 with receiving an input data set at the expert selection function logic in any processing block or tile to initiate MoE model inference. The input data set may include input data and / or intermediate activations generated by previous processing iterations in the same or different tile. As illustrated in method 905, the system invokes an expert selection logic function (e.g., a hash function) to apply the hash function to the input data set in the tile. At 910, a determination is made as to whether the expert selection logic function results in a hard selection value (e.g., a single expert selection index value). If the expert selection logic function results in a single hard selection value, the process proceeds to 913 where a further determination is made as to whether multiple experts / models should be invoked to process the input data. If the system determines at 913 that only a single expert / model should be invoked to process the input data, the process proceeds to 915 where a single model / expert at a single tier of tiles is activated (based on the hard selection expert selection function value) to process the input data set. Otherwise, if it is determined at 913 that multiple experts / models should be invoked to process the input data, the process proceeds to 920 of Figure 14 where the system activates models / experts at tiers in multiple tiles. Logic employed in any processing block or expert selection function logic block that receives an input data set can determine which of multiple physical tiles in the virtual tile to use to process the same input data set.Alternatively, the expert selection function result may be transmitted for reception at all tiles or a subset of tiles in the system, with each tile processing the hard selection value independently to determine whether to activate the corresponding expert / hierarchy in its respective tile. At each of the determined plurality of tiles, a hierarchy associated with each model / expert is selected (based on the hard selection expert selection function value), as well as processing of the received input data.

[0098] Referring back to 910 of FIG. 14 , if it is determined that the applied expert selection logic function results in soft-selected values ​​(e.g., more than one expert-selected index value), the process proceeds to 930, where a further determination is made as to whether multiple experts / models should be invoked to process the received input data. If the system determines at 930 that only a single expert / model should be invoked to process the input data in a single tile, the process proceeds to 935, where a single model / expert is activated in the first tier of the tile (based on the first soft-selected expert index value result) to process any activation / output and input data sets scaled and received from the previous network layer. Then, in the tile, the system iteratively activates one or more next tiers (based on the remaining corresponding soft-selected expert index values) to process the input data set.

[0099] Otherwise, if it is determined at 930 that multiple experts / models should be invoked to process the input data, the process proceeds to 940 of FIG. 14, where the system activates a model / expert in each of the multiple physical tiles of the virtual tile. Logic employed in any processing block or expert selection function logic block that receives the input data set can determine which of the multiple other physical tiles will process the same input data set. Alternatively, the expert selection function result may be transmitted to all tiles or a subset of tiles in the system for reception, with each respective tile independently processing its respective soft-selection expert selection index value and activating the corresponding expert / tier in its respective tile. In each of the determined multiple physical tiles, the system iteratively activates one or more tiers (based on the remaining corresponding soft-selection expert index values), also scaling and processing any activations / outputs received from the previous neural network model layer and the received input data set.

[0100] FIG. 15 illustrates an exemplary computing system according to the present invention for providing control unit function-based activation of memory hierarchies in the manner described in FIG. 14 to control dynamic MoE switching in a 3D CiM accelerator system. It should be understood that the illustrated computing system is merely one example of a suitable processing system and is not intended to suggest any limitation as to the functionality or scope of use of embodiments of the present invention. For example, the illustrated system may be operational with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the illustrated system may include, but are not limited to, integrated circuits, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

[0101] In some embodiments, the computer system may be described in the general context of computer system-executable instructions embodied as program modules stored in memory 16 and executed by the computer system. Generally, program modules 10 may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks and / or implement particular input data and / or data types in accordance with the methods described herein with respect to FIG.

[0102] Components of a computer system may include, but are not limited to, one or more processors or processing units 12, memory 16, and a bus 14 operatively coupling various system components, including memory 16, to the processor 12. In some embodiments, the processor 12 may execute one or more modules 10 loaded from memory 16, where the program modules embody software (program instructions) that cause the processor to perform one or more method embodiments of the present invention. In some embodiments, the modules 10 may be programmed into integrated circuits of the processor 12 and loaded from memory 16, a storage device 18, a network 24, and / or combinations thereof.

[0103] Bus 14 may represent one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0104] The computer system may include a variety of computer system readable media, which may be any available media that can be accessed by the computer system and may include both removable and non-removable media, volatile and non-volatile media.

[0105] Memory 16 (sometimes referred to as system memory) may include computer-readable media in the form of volatile memory, such as random access memory (RAM), cache memory, and / or other forms. The computer system may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 18 may be provided for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to removable, non-volatile magnetic disks (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to removable, non-volatile optical disks, such as CD-ROMs, DVD-ROMs, or other optical media. In such cases, each may be connected to bus 14 by one or more data media interfaces.

[0106] The computer system may communicate with one or more external devices 26, such as a keyboard, pointing device, display 28, etc.; one or more devices that allow a user to interact with the computer system; and / or any device (e.g., a network card, modem, etc.) that allows the computer system to communicate with one or more other computing devices. Such communication may occur via an input / output (I / O) interface 20. Furthermore, the computer system may communicate with one or more networks 24, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via a network adapter 22. As shown, the network adapter 22 communicates with other components of the computer system via bus 14. Although not shown, other hardware and / or software components may be used in conjunction with the computer system. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

[0107] The present invention may be a system, method, and / or computer program product, at any conceivable level of technical detail integration. The computer program product may include one or more computer-readable storage media having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0108] A computer-readable storage medium may be a tangible device that can hold and store instructions for use in an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridge structures in grooves that record instructions, and any suitable combination of the foregoing. Computer-readable storage medium, as used herein, should not be construed as being a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over a wire.

[0109] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium into each computing / processing device, or may be downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0110] Computer-readable program instructions for carrying out operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for configuring integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk or C++ or the like, and procedural programming languages ​​such as the "C" programming language or similar. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field programmable gate arrays, or programmable logic arrays (PLAs), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0111] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0112] These computer-readable program instructions may be provided to a computer processor or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the computer processor or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer-readable storage medium having the instructions stored thereon comprises a program product including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0113] The computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable apparatus, or other device to perform a series of operable steps to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0114] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may actually be accomplished as a single step and may be executed in parallel, substantially parallel, partially, or fully overlapping in time, or the blocks may possibly be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by special-purpose hardware-based systems that perform the specified functions or operations or execute a combination of special-purpose hardware and computer instructions.

[0115] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best explain the principles of the embodiments, practical applications, or technical improvements to technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0116] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly dictates otherwise. It will be further understood that, as used herein, the terms "comprises" and / or "comprising" may specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Corresponding structures, materials, operations, and equivalents of all elements in the following claims are intended to include any structure, material, or operation for performing a function in combination with other claimed elements as specifically claimed. The description of the invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. These embodiments have been chosen and described in order to best explain the principles and practical applications of the invention and to enable others skilled in the art to understand the invention in various embodiments with various modifications suited to the particular uses contemplated.

Claims

1. a 3D memory system having multiple tiers of compute-in-memory devices, where each tier includes a memory system associated with an expert of a mixture of experts (MoE) neural network model; a control processor for selecting, in response to the received input, one or more tiers of the compute-in-memory device associated with the expert in a neural network model layer; wherein the control processor configures computational paths at the selected one or more layers of the plurality of layers associated with the expert, the computational paths processing the inputs at the neural network model layers and generating output activations for use in further neural network model layers. A Compute-in-Memory (CiM) system comprising:

2. To select the one or more tiers associated with an expert, the control processor: implementing first logic for applying a function to the received input to generate an output expert-selected value; activating a hierarchy of compute-in-memory devices associated with the selected expert in the neural network model layer in response to the expert selection value. The CiM system according to claim 1 , configured as follows:

3. The CiM system of claim 2 , wherein the received input comprises model input data for inference processing by the selected expert in the neural network model layer.

4. 3. The CiM system of claim 2, wherein the activated layer is associated with a current layer of the MoE neural network model, and the received input comprises hidden layer activation values ​​generated as a result of processing in a previous neural network model layer.

5. 3. The CiM system of claim 2, wherein the selected expert at the neural network model layer generates hidden layer activation values ​​for propagation to a next neural network model layer at a different tier of a compute-in-memory device associated with the same selected expert or at a different tile associated with the same selected expert of the configured computation path.

6. 3. The CiM system of claim 2, wherein one or more selected experts in the neural network model layer process the same received input, and each of the one or more selected experts generates one of a model output or a hidden layer activation value in response to the processing of the received input, the generated hidden layer activation value for propagation to a different tier or different tile of a compute-in-memory device, or for propagation to a next neural network model layer not associated with an expert.

7. gating circuitry for scaling and accumulating the generated hidden layer activation values ​​associated with inference processing by the experts at the one or more layers associated with the experts at one or more tiles; wherein the gating circuitry further propagates the resulting values ​​of the accumulated hidden layer activation values ​​to a next network model layer; The CiM system of claim 2 further comprising:

8. a pulse signal generator for generating a pulse signal having a characteristic associated with the value of the received input, wherein the control processor implements the pulse signal when performing a matrix vector multiplication operation on the selected hierarchy of compute-in-memory devices associated with the selected expert; The CiM system of claim 2 further comprising:

9. 3. The CiM system of claim 2, wherein each of the tiers of compute-in-memory devices includes a non-volatile memory system configured as a crossbar array of memory storage cells implemented for matrix-vector multiplication operations associated with the MoE neural network model expert.

10. a plurality of processing tiles, wherein a layer of a deep neural network model corresponds to one or more tiles of the plurality of processing tiles, the layer receiving input data for inference processing using the deep neural network model, and each of the plurality of processing tiles: a three-dimensional (3D) multi-tier configuration of a compute-in-memory device, where each tier includes a crossbar array of memory cells that includes a memory system associated with a neural network model; a control processor circuit for selecting one or more tiers of compute-in-memory devices associated with the neural network model in response to the received input data; wherein the control processor circuit configures computational paths in one or more of the plurality of tiers associated with a selected model, the computational paths processing the received input data in the network model and generating one or more outputs for use in a further neural network model. Including, A Compute-in-Memory (CiM) system comprising:

11. To select the one or more layers associated with the neural network model, the control processor circuitry: applying a function to the received input data to generate an output model selection value; activating a hierarchy of compute-in-memory devices associated with the neural network model in response to the model selection value. The CiM system according to claim 10, configured as follows:

12. The CiM system of claim 11 , wherein the received input data comprises model input data for inference processing by the selected neural network model.

13. 12. The CiM system of claim 11, wherein the activated layer is associated with a current layer of the neural network model, and the received input data includes hidden layer activation values ​​generated as an output result of processing in a layer of a compute-in-memory device at a previous neural network model layer.

14. gating circuitry for scaling and accumulating the generated hidden layer activation values ​​associated with the inference processing results in one or more tiers associated with the model layer, wherein the gating circuitry further propagates the accumulated activation result values ​​to one or more different layers of the same model in the same tier or a different tier in one of the same processing tile or a different processing tile; The CiM system of claim 11 further comprising:

15. configuring one or more processing tiles based on the received input data, where each processing tile is associated with a layer of the neural network model, and each processing tile comprises: a three-dimensional (3D) multi-tier configuration of compute-in-memory devices configured to process data associated with experts of a mixture of experts (MoE) neural network model; and a control processor circuit for iteratively selecting one or more tiers of compute-in-memory devices associated with said expert in response to received input data; Includes; using the control processor to configure a computational path at one or more selected layers of the plurality of layers associated with the expert, wherein the computational path processes the input at a layer of the neural network model and generates one or more output activations for use at a further layer of the neural network model; 1. A method for operating a compute-in-memory system, comprising:

16. constructing the one or more layers in response to the received input data, applying a function to the received input data in a first expert selection logic circuit to generate an output expert selection value; activating a hierarchy of compute-in-memory devices associated with the selected expert in the MoE network model layer in response to the expert selection value.

16. The method of claim 15, comprising:

17. The received input data includes model input data for inference processing by one or more selected experts in the neural network model layer, and the method further comprises: generating, in each of the one or more selected experts, one of a model output or a hidden layer activation value in response to the inference processing of the received input data, wherein the generated hidden layer activation value is for propagation to a different tier or a different tile of a compute-in-memory device or for propagation to a next neural network model layer not associated with the expert; 17. The method of claim 16, comprising:

18. scaling and accumulating the generated hidden layer activation values ​​associated with the inference process by the expert in one or more layers associated with the expert in one or more tiles; and propagating the accumulated hidden layer activation values ​​to the same expert at a different layer in either the same processing tile or a different processing tile, or propagating the output activations to a next network model layer.

20. The method of claim 17, further comprising:

19. generating, in the first expert selection logic circuit, a gating value in response to the received input data, wherein the gating value is used to scale the output activations prior to the propagation; 20. The method of claim 18, further comprising:

20. The activated layer is associated with a current layer of the MoE neural network model, and the received input data is Hidden layer activation values ​​that are generated as a result of processing in and propagated from previous neural network model layers 17. The method of claim 16, comprising:

Images