Function-based memory level activation

Abstract

A 3D in-memory computing accelerator system (100) and method for efficiently inferring a hybrid expert (MoE) neural network model. The system includes a plurality of in-memory computing cores (102), each memory core including a plurality of layers of in-memory computing units (106). One or more layers of the in-memory computing unit correspond to expert sub-models of the MoE model. One or more expert sub-models for activation propagation are selected according to a function-based routing (115) based on which a corresponding expert layer is activated. In one embodiment, the function is a hash-based layer selection function for dynamic routing of input and output activation. In an embodiment, the function is applied to select a single expert or multiple experts for single-level activation using input data based or using MoE based on layer activation. Further, the system is configured as a multi-model system with a single expert model selection or a multi-model system with multiple expert selections.

Description

Technical Field

[0001] The present disclosure relates to deep learning machine learning models, and more particularly, to a neural network model system comprising a hierarchy of in-memory computing devices configured to implement multiple expert sub-models, referred to as mixtures of experts (MoEs), to perform fast and energy-efficient inference on models with billions of parameters. Background Art

[0002] Deep learning refers to the use of a multi-layer machine learning model architecture that uses multiple layers to progressively extract higher-level features from the raw input. These machine learning models are typically artificial neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc., trained using representation learning for use in artificial intelligence applications. Learning can be supervised, semi-supervised, or unsupervised.

[0003] It has been shown that the performance of modern deep learning architectures improves by simply scaling up the model size (i.e., number of parameters) and the dataset size. This paradigm poses two major challenges: training and inference on models of this size are extremely expensive (training a single model costs on the order of millions); and fast and energy-efficient inference is not possible when the model has a large number of parameters because it requires large amounts of DRAM.

[0004] The first challenge is usually addressed by a mixture of experts (MoE) system. Mixture of experts is an ensemble learning technique in the field of neural networks, known for its ability to scale models up to billions of parameters without incurring additional computational cost. In a MoE model, the predictive modeling task is decomposed into subtasks and an expert model is trained on each subtask. The main idea of ​​a MoE system is to route the data fed into a neural network to a set of experts. That is, a mixture of experts system implements a routing algorithm that routes the hidden layer activations to a designated expert for forward computation. By doing so, one can arbitrarily scale up the number of experts by simply increasing the number of experts while keeping the computational complexity low. The computational complexity is kept low because not all experts are selected, but only one expert is selected.

[0005] Despite the advantages of MoE systems, modern architectures still require a large number of parameters, which makes fast and energy-efficient inference impossible.

[0006] The main reason large models are slow to infer is that either the weights have to be constantly transferred back and forth between memory and the GPU / CPU, or a large amount of dynamic random access memory (DRAM) is required to store all the required weights. MoE systems alleviate this problem to some extent, because only a subset of the experts are active at a time, so not all weights have to be used. However, most systems are too large to eliminate this problem even with MoE. Summary of the invention

[0007] A system, method, and computer program product for performing fast and power-efficient inference of MoE systems by performing computations in memory using a 3D memory architecture.

[0008] The present disclosure relates to deep learning machine learning models, and more particularly, to a neural network model system comprising an in-memory computational fabric layer configured to implement multiple expert sub-models represented as a mixture of experts (MoE), wherein a circuit system provides dynamic routing using function-based hierarchical selection.

[0009] In an embodiment, a hash function is implemented using fast inference in a MoE system using an in-memory compute structure.

[0010] In one aspect, a 3-dimensional in-memory computing (CiM) system for efficient inference of a mixture of experts (MoE) model is provided. The system includes: a 3D memory system including multiple levels of in-memory computing devices, each level including a memory system associated with an expert of a mixture of experts (MoE) neural network model; a control processor for selecting one or more levels of the in-memory computing devices associated with the experts at the network model layer in response to received input; the control processor configures a computing path at the selected one or more levels associated with the experts in the multiple levels, the computing path being used to process the input at the network model layer and generate one or more output activations for use at another neural network model layer.

[0011] In an embodiment, the 3D memory system is a non-volatile memory system, such as a resistive memory (memristor) system or a 3D NAND flash memory. It is understood that other types of non-volatile memory systems may also be used.

[0012] In an embodiment, when running a deep neural network model, a MoE expert for neural network processing is selected using a function based on input or layer activation. MoE allows the calculation path to be selected by function, so if the input is used to determine the expert to be used, the processing block / layer to be activated can be selected a priori. In addition, different layers have weights for different experts, and the layers are switched dynamically based on the input / intermediate activations. If layer activations are to be used, the experts selected to process these activations can only be determined after they arrive.

[0013] Using input-based or layer-activation-based functions enables selecting single or multiple experts using input-based MoE, or selecting single or multiple experts using layer-activation-based MoE for single-level / multi-level activations.

[0014] The provided 3D CiM system may include crossbar memory and act as an accelerator to improve the performance of matrix-vector operations during deep neural network model processing.

[0015] In another aspect, a 3D compute-in-memory (CiM) system is provided. The 3D CiM system includes a plurality of processing blocks, each one or more processing blocks corresponding to a layer of a deep neural network model that receives an input data set for inference processing using the deep neural network model, the processing blocks comprising: a multi-level configuration of a three-dimensional (3D) in-memory compute device, each level comprising a crossbar array of memory cells, which includes a memory system associated with a neural network model processing operation; a control processor circuit for selecting one or more levels of the in-memory compute device associated with the neural network model in response to the received input data set; and the control processor circuit configures a computational path at one or more levels of the plurality of levels associated with the selected expert, the computational path for processing the input received at the network model and generating one or more outputs for use at a CiM processing unit associated with another layer of the neural network model.

[0016] Furthermore, the Computing-in-Memory (CiM) system can be configured as a multi-model system with a single expert selection or multiple expert selections.

[0017] In another aspect, a method for operating an in-memory computing system is provided. The method includes: configuring one or more processing blocks based on received input data, each processing block being associated with a layer of a neural network model, each processing block comprising: a multi-level configuration of a three-dimensional (3D) in-memory computing device configured to process data associated with an expert of a mixture of experts neural network model; and a control processor circuit for iteratively selecting one or more levels of the in-memory computing device associated with the expert in response to the received input data set; selecting one or more levels of the in-memory computing device associated with the expert at a network model layer in response to the received input; and configuring a computational path at one or more selected levels associated with the expert among the multiple levels using the control processor, the computational path being used to process the input at the MoE network model layer and generate one or more output activations for use at another neural network model layer.

[0018] Further features as well as the structure and operation of various embodiments will be described in detail below with reference to the accompanying drawings.In the drawings, like reference numbers indicate identical or functionally similar elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of an example of a compute-in-memory (CiM) hardware and memory system accelerator that implements a MoE neural network model architecture according to an embodiment of the present invention;

[0020] Figure 2A-2E Depicts operations and signal / data flows involving processing blocks at a 3D Compute-in-Memory (CiM) hardware and memory system accelerator processing block implementing processes associated with a MoE deep neural network model architecture, according to an embodiment of the present invention;

[0021] Figure 3 An embodiment of a hierarchy activation circuit is shown that selects a word line by generating a word line (WL) signal to activate a CiM memory hierarchy corresponding to a selected expert of a MoE model, the selected expert of the MoE model residing on a single or multiple hierarchies;

[0022] Figure 4 A first embodiment of a 3D CiM accelerator system is depicted, which is configured to select a single (multiple) expert for single-level or multi-level activation using an input-based MoE;

[0023] Figure 5 Conceptually depicts a process at a CiM accelerator system for selecting a single (multiple) expert for single-level activation using an input-based MoE;

[0024] Figure 6A second embodiment of a 3D CiM accelerator system is depicted, the 3D CiM accelerator system being configured to select a single(s) expert(s) for single-layer level activation using a layer-activation based MoE;

[0025] Figure 7 Conceptually depicted Figure 6 The CiM accelerator system is used to select a single (multiple) expert for single-layer activation processing using a layer-activation-based MoE;

[0026] Fig. 8A and Figure 8B Another embodiment of a 3D CiM accelerator system configured to select multiple experts for single-level activation using an input-based MoE is depicted;

[0027] Fig. 9 depicts a multi-expert 3D CiM accelerator system processing block comprising a stack of CiM levels implementing one or more expert models corresponding to a MoE feed-forward network (FFN);

[0028] Fig. 10A and Fig. 10B depicts additional embodiments of a 3D CiM accelerator system configured for selecting multiple experts for single-level activation using an activation-based MoE;

[0029] Fig.11 An embodiment of a 3D CiM accelerator system implementing a plurality of different models and configured for selecting a single model in a multi-model system is depicted;

[0030] Fig.12 Conceptually depicted Fig.11 A multi-model 3D CiM accelerator system for selecting a single model in the multi-model system;

[0031] Fig.13A and Fig. 13B depicts a further embodiment of a 3D CiM accelerator system configured for selecting a plurality of experts (models) residing at different levels of a 3D multi-model CiM system;

[0032] Fig.14 Describes methods implemented at a virtual processing block or at any processing block associated therewith of a 3D CiM MoE accelerator system of various embodiments; and

[0033] Fig.15 An example computing system for controlling dynamic MoE switching in various embodiments of a 3D CiM accelerator system is shown. DETAILED DESCRIPTION

[0034] Research has shown that current AI models can improve performance by scaling up model and dataset sizes. However, this poses a problem on traditional hardware (e.g., CPU, GPU) because large models cannot be trained without sparsification.

[0035] The main reason large models are slow to infer is that either the weights have to be constantly transferred back and forth between memory and the GPU / CPU, or a large amount of dynamic random access memory (DRAM) is required to store all the required weights. MoE systems alleviate this problem to some extent, because only a subset of the experts are active at a time, so not all weights have to be used. However, most systems are too large to eliminate this problem even with MoE.

[0036] Especially in AI systems based on convolution operations, computing speed and throughput need to be greatly improved. In-memory computing is a way to accelerate deep learning inference and training.

[0037] Figure 1 An in-memory computing accelerator system 10 is shown that enables fast and energy-efficient inference for MoE systems by performing deep learning neural network computations in-memory using a three-dimensional (3D) non-volatile memory architecture.

[0038] like Figure 1As shown, the CiM accelerator system 10 includes one or more digital processors 15 that issue control signals to control operations for a number of applications involving data stored in a non-volatile memory (NVM) subsystem 20, which includes memory devices, such as DRAM. A system communication bus 12 is provided for transferring data back and forth between the memory 20 and the computational unit 15 when performing operations (e.g., Kalman filtering, fast Fourier transforms, linear equation solvers). According to one aspect of the present disclosure, further connected to the system communication bus 12 is a CiM device system 25, which has a control unit (such as a microprocessor 28) and a plurality of CiM processing blocks 40, wherein each processing block 40 includes a 3D in-memory computation block, which includes multiple levels 45 of a 3D CiM memory device, and also includes associated computational circuitry for controlling neural network operations at the levels (e.g., in-situ matrix vector multiplication (MVM) operations). The microprocessor 28 can be configured to manage computational paths for performing CiM MoE neural network operations, such as input / output and / or computations, involving the levels at the processing block. In an embodiment, depending on the configuration of the MoE model, output data (such as intermediate activations generated due to MVM operations performed at a level of a processing block (e.g., processing block 41)) may be controlled to be input / transmitted to another CiM device corresponding to the same or different (e.g., at a level at the same processing block or a different processing block (e.g., processing block 42)) expert via data transfer using a local data bus 47. In an embodiment, the layer activations are inputs to different layers of the neural network model and are typically vectors of floating point / integer elements.

[0039] like Figure 1 As shown, each 3D CiM device level at processing block 40 includes a resistive memory array, for example, a two-dimensional array 50 of memristive devices in a crossbar configuration, which is suitable for executing computational primitives by utilizing analog memory capabilities and Kirchhoff's circuit laws, such as having For example, at a two-dimensional array 50 of memristive devices, an MVM operation can be calculated as

[0040]

[0041] in and is the value of the input data vector, which can be mapped to the Vin voltage value 52, A 11 -A 22 The value corresponds to the conductance (resistance) value or "weight" stored at the corresponding memristive device 51 at the crossbar array 50 at the level, and and is the output value converted from the sensed output current 53 read from the array 50 at this level. Since all weights reside in the 3D memory architecture and the calculations are performed within the memory, the bottleneck of transferring data back and forth between the memory and the computing unit is completely eliminated. In an embodiment, the size of the 3D CiM cross-bar memory can correspond to one expert at a single level. However, larger experts may require 3D CiM cross-bar memories at multiple levels, for example, two or more levels. That is, experts can span a single processing block as well as multiple processing blocks.

[0042] Each 3D CiM device at processing block 40 provides significant weight storage capacity to store millions of parameters per CiM processing block at high density, enabling efficient and fast inference (and potentially training) of billion-parameter sized models on multi-processing block CiM accelerators.

[0043] In an embodiment, the present disclosure proposes to implement a large-scale model using a MoE architecture by programming each expert into a different level of 3D memory.

[0044] As used herein, the term "virtual processing block" refers to a processing block where an expert model resides. A virtual processing block spans one or more physical processing blocks 40. Each physical processing block 40 has multiple levels 45 that form a 3D CiM system. Since an expert can have multiple neural network layers, these layers are mapped to levels in different physical processing blocks. For example, an expert model with two layers can be mapped to the first level (level 1) of a first processing block and the first level (level 2) of a second processing block. Additionally, two experts can reside on a virtual processing block, for example, the first expert resides on level 1 of the first physical processing block (e.g., the first layer of the expert model) and level 1 of the second physical processing block (e.g., the second layer of the expert model), while the second expert resides on the second level of the first physical processing block (e.g., the second expert of a neural network model layer). The output of the expert is sent to the next layer.

[0045] Given newly arrived data, the layers of the virtual processing blocks can be selected accordingly to process the data. Typically, an expert is selected when input data (e.g., in the form of hidden layer activations from a previous layer) arrives at a virtual processing block. Based on the expert index, the method activates the corresponding layers in each physical processing block in the virtual processing block. For example, in response to the input data, the expert selects expert 1 and then activates layer 1 in both physical processing blocks. The data is first sent to layer 1 in physical processing block 1 and then to layer 1 in physical processing block 2. The output of the virtual processing block is then sent to the next layer in the neural network.

[0046] In the MoE model: Given the hidden activation h generated at the neural network model layer, the output of the MoE layer is given by Given, where K is the number of experts, e i (h) is the output from the ith expert, g i is the i-th gating value. However, since the gating function may route the input to the same subset of experts and cause instability / inefficiency, a MoE scheme is implemented, where a hash function is applied to the input and the hash is calculated on each input token instead of calculating the hash on the hidden state, and the expert index is calculated based on the input and the corresponding specific feed-forward network (FFN) is selected as the expert. That is, using the function: The a priori determined experts are the output, where f(x) is the expert index (multiple indices) determined by applying the expert selection function f(x) on the input x, ranging from 0 to K-1.

[0047] Figure 2A-2E Depicted are operations and signal / data flows involving an example 3D compute-in-memory (CiM) hardware and memory system accelerator 100 at the level of virtual processing blocks implementing a MoE neural network model architecture in accordance with an embodiment of the present invention.

[0048] In an embodiment, the computational in-memory system accelerator 100 is configured with multiple memory cores, each memory core having more than one level of in-memory computational units, and more than one expert residing in the level. The configuration includes selecting one or more experts for activation propagation based on a function (e.g., a hash function). In an embodiment, the level of the corresponding expert is activated based on the function. That is, the MoE allows the selection of a computation path via the function f(), so the level to be activated can be selected a priori. Different levels have weights for different experts, and the levels are switched dynamically based on the input / intermediate activations.

[0049] Figure 2A Specifically shown is an in-memory computing accelerator system 100 including a processing block having multiple memory cores, employing function-based memory hierarchy activation to efficiently infer a mixture of experts (MoE) model.

[0050] exist Figure 2A In FIG. 1 , the CiM accelerator system 100 includes a plurality of processing blocks 102, each of which corresponds to a successive layer of a deep neural network (DNN). For non-limiting illustrative purposes, Figure 2AA virtual processing block is shown including a sequence of processing blocks T-1, T, T+1. In an embodiment, each processing block 102 includes the following components: a compute-in-memory (CiM) structure forming a 3-dimensional compute-in-memory system 105 for accelerating deep learning inference and training. Each 3-dimensional compute-in-memory system 105 includes multiple memory cell levels 106, each level 106 having an addressable 2-dimensional CiM array, including a cross-strip configuration of memory cells 50 for processing neural network processes at a specific neural network layer. In an embodiment, the memory cells may store weights associated with a specific neural network model and / or MoE. In an embodiment, all weights reside in the 3D memory architecture. Since all weights reside in the 3D CiM architecture and the method performs computations in memory, the bottleneck of transferring data back and forth between memory and compute units is completely eliminated. Therefore, Figure 2A The in-memory computing accelerator system 100 is capable of quickly and very energy-efficiently inferring models containing billions of parameters.

[0051] like Figure 2A As shown, in the processing method, the first step is that data 125 arrives at the processing block T at the expert selection function logic circuit 110. In one embodiment, the input data 125 arrives from another processing block / processing blocks and includes data "x", which includes a vector of floating point / integer elements. However, the input data 125 can also be a single number (e.g., a vector with a single element). For example, a programmer / user can specify the first expert / level at which the MoE model is desired to be used. In an embodiment, the input data 125 can be an intermediate activation "h" received from a previous layer, that is, from the same or another level / processing block. For example, as Figure 2A As shown, data 125 generated at a previous processing block T-1 (previous DNN model layer) at the scaling and accumulation (gating) circuit system 120 is received at the expert selection function logic circuit 110 for processing at processing block T (e.g., the next DNN model layer).

[0052] In the second processing step 130 ( Figure 2B ), at the expert selection function circuit 110, the expert selection function f() is executed. Execution of the expert selection function f() results in the generation of one or more expert indexes and corresponding gating values. The expert index is used as an input to a corresponding level activation circuit 115 for generating a wordline signal for selecting a particular CiM level 106 of the determined expert model. That is, in the CiM accelerator system 100, associated with the CiM structure 105 at the processing block 102 is an associated level activation circuit 115, which is based on a wordline driver (WLD) 113, for example, shown as a wordline driver WL 0 , WL 1, ..., WL K-1 , which is connected to the corresponding memory hierarchy 106 to activate the corresponding memory hierarchy to process the received input. Since the hierarchy can be associated with different MoE models, different hierarchies 106 can hold weights of different experts.

[0053] The layer activation circuit 115 includes a function for dynamically switching (activating) layers. For example, the layer is dynamically switched according to the received input x / intermediate activation h. In an embodiment, the MoEs model is able to select a computation path via an expert selection function "f" circuit 115, and the "f" circuit 115 implements the logic for selecting / activating the CiM layer 106. In an embodiment, the expert selection function "f" circuit 115 implements the logic that can a priori select and activate the layer. An example of this function "f" is a hash function. Another example of f(x) can be a softmax function followed by an argmax function, as shown below:

[0054] argmax(softmax(x*fixed_weight_matrix)).

[0055] Here, a fixed weight matrix is ​​a matrix that is learned in a different process, i.e., fixed in the sense that it is determined before training the model. For example, a multi-layer neural network may use multiple weight matrices linked by some nonlinearity.

[0056] As further shown in system 100, associated with CiM system 105 at processing block 102 is peripheral circuitry 107 and gating circuitry 120, which can be used to scale and accumulate DNN model layer outputs. Peripheral circuitry 107 can include analog / digital converters, digital / analog converters, registers, memories, buffers, filters, etc., for performing neural network processing operations at the processing block.

[0057] Therefore, if Figure 2C As shown in the third processing step 135 in FIG. 1 , based on the expert index 132 from the expert selection function logic circuit 110, the level activation circuit system 115 activates the level of interest. For example, in response to the input 132, such as Figure 1 C, the memory level word line WL 0 111 is set to a high voltage value to activate the expert model at the corresponding CiM memory level 106A for calculating and generating model layer output activations.

[0058] Figure 2DA fourth processing step in system 100 is depicted, where gating circuitry 120 receives gating values ​​138 generated at expert selection function logic circuitry 110 and analog-to-digital converted intermediate output activations 140. Gating circuitry 120 performs processing using gating values ​​138 to scale and then accumulate model layer outputs at an expert at a level (e.g., level 106A).

[0059] Further processing steps are performed in a similar manner to select other experts in the MoE model. Figure 2E As shown, based on the additional input 133 from the expert selection function logic circuit 110, the level activation circuit system 115 activates another (second or additional) level of interest. Figure 2E As shown, the memory level word line WL corresponding to the second CiM memory level 106B 1 121 is set to a high voltage to activate stage 106B and generate a further intermediate output activation 141. In each further processing step, as Figure 2E As shown, the gating value 139 generated at the expert selection function logic circuit 110 is applied to the output activation 141 that has been analog-to-digital converted. The result is accumulated at the gating circuit system 120, which scales and accumulates the model layer output. In one embodiment, the levels 106A, 106B can be associated with the same MoE expert or different experts.

[0060] In with Figure 2A-2E After executing all other selected experts in a similar manner as shown in , the intermediate output activation 142 is calculated and the activation is navigated to the next network layer which may reside on one or more processing blocks, for example, as input activation 143 to the next virtual processing block T+1 102 for processing at a subsequent neural network layer.

[0061] Figure 3 One embodiment of a hierarchy activation circuit system 115 is shown, which generates a word line WL for activating a CiM memory hierarchy 106 corresponding to an expert of a MoE model. In one embodiment, the hierarchy activation circuit system 115 includes a pulse generation circuit 200, which generates a word line pulse WL 0 , WL 1 , ..., WL K-1 , whose duration or amplitude is commensurate with the input data. In one embodiment, the word line (WLD) pulse 210 is generated for a matrix-vector multiplication operation in the hierarchy activation logic applied at the hierarchy activation circuitry 115 .

[0062] In an embodiment, the pulse generator circuit 200 is configured to receive digital input data (e.g., floating point / integer) and convert it into a waveform that is used as an input for analog matrix-vector multiplication. In the example, the duration of the waveform 210 represents the input number. As a non-limiting example, assuming that the input number can take values ​​between 0 and 128 and the maximum possible duration is 128ns, the number 64 can be represented as a pulse with a fixed amplitude and a duration of 64ns. An alternative is to modulate the amplitude with a fixed duration. Assume that the amplitude can be modulated between 0V and 0.5V. Then, the number 64 can be represented by a pulse amplitude of 0.25V.

[0063] In operation, the pulse generation circuit 200 receives input data (e.g., a vector of values) and generates an output waveform pulse 210. In one example, the pulse duration of the output waveform pulse 210 is proportional to the n-bit enable of the time slice. The output waveform pulse 210 is input to the 1×K switch / demultiplexer 220, which further asserts the word line signal WL based on the m-bit MoE index determined by the following formula 0 , WL 1 , ..., WL K-1 :

[0064]

[0065] Where K is the number of selectable CiM levels in the 3D memory. In one embodiment, the m-bit MoE index is the output of the function f(x). For example, assuming there are six (6) bit experts, each at a different level, then a 3-bit index is sufficient to represent each expert. An index value of 000 will translate to expert 0, an index value of 001 will translate to expert 1, ..., and an index value of 101 will translate to expert 5. The m-bit MoE expert index value controls the switch / demultiplexer 220 to select which word line WL associated with each level receives the generated pulse 210 and applies the output waveform to that word line. In this case, the expert selection function f logic circuit 110 outputs multiple indices, and each matrix-vector multiplication is performed sequentially.

[0066] refer to Figure 4 In a first embodiment, the 3D CiM accelerator system 300 is configured to use an input-based MoE to select a single (multiple) expert to activate a single layer or multiple layers. In this input-based MoE embodiment, the inputs to the model are used / processed to define which experts / layers will be activated. The inputs to the model are received at the beginning of model inference, so the experts / layers of all layers can be determined in advance if necessary. In an embodiment, it is assumed that a single expert is activated per layer.

[0067] like Figure 4As shown, an arbitrary processing block 310 (eg, at virtual processing block 202) is shown that receives an input sequence x l 305, the block is an expert choice or arbitrary implementation of a hash function f(x l ). Block 310 receives the sequence x l As input and produces output f(x l ) 325. The block can perform this operation in the analog or digital domain. Any processing block 310 can be part of the virtual processing block 302 or can physically reside in another part of the accelerator system 300. The activation circuit system 315 at the processing block T 302 associated with the neural network layer receives the output f(x l ) and the corresponding received activation h received from the previous network model layer l 326 to generate a wordline signal (eg, wordline 311 ) to activate the tier or tiers associated with the MoE expert to generate an additional output activation 341 .

[0068] Figure 5 Conceptually depicted is the process at the CiM Accelerator system 300 for selecting a single (multiple) expert for single-level activation using an input-based MoE. Figure 5 , a MoE expert system 400 is shown, in response to an intermediate activation sequence [h 1 ,h 2 ,...,h L ]. The choice of which FFN (expert) is determined by f(x), i.e., the input to the neural network (not the hidden layer activations). Each FFN model is a 3D CiM structure 410 composed of layers 415 of memory cells. 1 ,x 2 ,...,x L ] at any processing block 310, wherein (N is a natural number greater than or equal to 1), the function f is applied to the input sequence x to produce a single value (hard selection) 420 for selecting a single expert / level. In one embodiment, given an input sequence x = [x 1 , x 2 , ..., x L ], generate the index sequence [f(x1), f(x2), ..., f(xL)], and generate the corresponding gate value [g f(x1) (h1), g f(x2) (h2), ..., g f(xL) (hL)] = [{1}, {1}, ..., {1}], that is, all generated gating values ​​are equal to 1. Here, and

[0069] Back to Figure 4 , based on the input x l Expert indices 325 are determined for all processing blocks (e.g., processing block T+1), i.e., the same arbitrary processing block 310 determines the expert indices for all virtual processing blocks (e.g., processing block T, processing block T+1, etc.), and a data stream 330 of expert indices is communicated from the arbitrary processing block 305 to the hierarchy activation circuitry 315 at the other processing blocks (e.g., processing block T+1). However, there may be alternative implementations, for example, the input sequence x may be provided to all physical processing blocks of the virtual processing block simultaneously, and each physical processing block may determine its own activation hierarchy. Figure 4 As further shown, in the example, based on the input x l The received expert index 325 is used to activate the expert model at the level (e.g., level 306A), which processes the data and any activations and generates additional intermediate output activations 341. Figure 4 As further shown, Figure 4 It is the corresponding generated gating value (=1) 339 generated by the expert selection function logic at any processing block 310 for application to the output activation 341 after the output activation 341 has been analog-to-digital converted. The result is accumulated at the gating circuit system 320, which scales and accumulates the model layer output.

[0070] refer to Figure 6 In a second embodiment, the 3D CiM accelerator system 350 is configured to use a layer activation based MoE to select a single (multiple) expert for a single layer level activation. In this layer activation based MoE embodiment, the activations received from the previous layer are used / processed to determine which experts / layers will be active. In this embodiment, the expert indices are dynamically / on-the-fly calculated; in other words, they are unknown before the previous layer is executed. In an embodiment, it is assumed that a single expert is activated per layer.

[0071] like Figure 6 As shown, the expert selection function logic processing block 360 selects the activation sequence h=[h 1 ,h 2 ,...,h L ] generates an expert index or selection. Specifically, in response to applying the hash function f to the activation sequence (i.e., f(h l )) and generate the expert index 361. The layer activation block 315 at the processing block T 302 associated with the neural network layer receives the activation h l The sequence and corresponding received activation h received from the previous layer in the expert selection function logic processing l The sequence f(h l) 375 to generate a wordline signal (e.g., wordline 361) to activate a tier (e.g., tier 306A) associated with the MoE expert, thereby generating an additional output activation 341.

[0072] Figure 7 Conceptually depicted Figure 6 The CiM accelerator system 350 is used to select a single expert for single-layer activation processing using MoE based on layer activation. Figure 7 , the MoE expert system 500 is shown, in response to receiving an intermediate activation sequence [h 1 ,h 2 , ..., h L ] and activated multiple feed-forward neural network models 505. Each FFN model is a 3D CiM structure 510 composed of a hierarchy 515 of memory cells.

[0073] exist Figure 6 and Figure 7 In the embodiment of , the applied expert selection function f produces a single value (hard selection) 520 to select an expert / level. Given the previous layer, the layer output is represented as [h 1 ,h 2 ,...,h L ],in The expert selection function logic processing block 360 generates a series of indices [f(h1), f(h2), ..., f(hL)] and corresponding gate values ​​[g f(h1) (h 1 ), g f(h2) (h2),…,g f(hL) (h L )] = [{1}{1},...,{1}], i.e., all generated gating values ​​are equal to 1. Given the activations from the previous layer, a specific expert (level in the 3D architecture) is selected according to [f(h1),f(h2),...,f(hL)]. Here, and

[0074] review Figure 6 , the expert index 375 is based on the previous layer activation h of the current processing block (e.g., processing block T). l The expert index 375 is used to activate an expert model at the level (eg, level 306A), which processes the data and any activations and generates additional intermediate output activations 341 . Figure 6Further shown is a corresponding generated gating value (=1) 339 generated at the expert selection function logic processing block 360 for application to the output activation 341 after the output activation 341 has been analog-to-digital converted. The result is propagated to and accumulated at the gating circuit system 320, which scales and accumulates the model layer output.

[0075] Fig. 8A and Figure 8B Depicted is an additional embodiment of a 3D CiM accelerator system 600 configured for selecting multiple experts for single layer activation using input-based MoE. In this input-based MoE embodiment, the inputs to the model are used / processed to define multiple experts / layers that will be active. The inputs to the model are received at the beginning of model inference, so the experts / layers for all layers can be determined in advance if necessary. In an embodiment, multiple layers / experts are used in each processing block. Each expert contributes to the output of the processing block, which contribution is proportional to its gating value. It is determined which layers / experts will be used for that layer based on the model input.

[0076] like Fig. 8A As shown, an arbitrary processing block 610 (eg, at virtual processing block 202) is shown that receives an input sequence x l 605, the block is an expert selection or hash function f(x l ). Block 610 receives the sequence x l As input and output the corresponding sequence of expert indices f(x l )625, where l = 1, 2, ..., L. This block can perform this operation in the analog or digital domain. Any processing block 610 can be part of the same physical processing block 602 or can physically reside in another part of the accelerator system 600. The layer activation circuit system 615 at the processing block T602 associated with the neural network layer sequentially receives the output f(x l ) and receives activations h from the previous iteration (the activations h received from the previous layers of the model) l 626 to generate a corresponding hierarchical activation wordline signal. For example, in response to the first generated expert index, for example, f(x l ) 1 ,, the hierarchical activation circuit 615 generates a first word line signal, for example, a word line 611 (eg, WL 2 ) to activate the first level associated with the first MoE expert, e.g. Fig. 8A , to process / generate additional output activations 641 . Fig. 8A 610 is shown in FIG. 62 , in response to a first input x of an input sequence received at any processing block 610 by the expert selection function logic.1 The corresponding generated gating value 639 is used to apply to the output activation 641 after analog-to-digital conversion of the output activation 641. The result is propagated to and accumulated at the gating circuit system 620, which scales and accumulates the model layer output. Fig. 8A , Figure 8B In the embodiment shown in FIG. 1 , the generated output activations scaled and accumulated at the gating circuit system 620 are sent to the next level at the physical processing block 602 to be activated based on the second input x in response to the input sequence received at any processing block 610. 2 The next expert index generated (i.e., f(x 2 That is, in this embodiment, based on the current input sequence x l The received expert index 625 is used to sequentially activate the expert model at one or more levels to process the current input data. In one embodiment, the same input passes through different experts / levels. The processing at the level generates additional intermediate output activations 641 for scaling, accumulation with previous outputs and storage for use in the next iteration of the input sequence.

[0077] Therefore, if Figure 8B As shown, in the next processing sequence of sequentially activating the expert models at one or more levels, in response to the next (eg, second) generated expert index, i.e., f(x l ) 2 , the hierarchical activation circuit 615 generates the next word line signal, for example, word line 612 (eg, WL 3 ) to activate another level associated with the second MoE expert, e.g. Figure 8B Level 616B is shown for processing / generating additional intermediate output activations 643 . Figure 8B Also shown is the second input x of the input sequence received by the expert selection function logic in response to any processing block 610. 2 The corresponding generated gating value 649 is used to apply to the output activation 643 after the output activation 643 is analog-to-digital converted. The result is propagated to and accumulated at the gating circuit system 620, which scales, accumulates and stores the model layer output. Therefore, at layer L, the input x_1 produces an activation (h_1) L . Used to process (h_1) L The experts of are determined based on x_1. For example, in the given case, experts 1 and 2 are selected. Then, first (h_1) L Pass it to expert 1. Multiply the result by the gate value and store it. Then, pass the same activation (h_1) LPassed to Expert 2. The result is multiplied by the corresponding gated value and added to the stored output. The final output is then sent to the next layer / processing block.

[0078] In this further embodiment, Fig. 8A , Figure 8B The process shown in is iterative, that is, for the input sequence x l and as many experts / levels as required by the expert selection function logic applied at any processing block 610. In each iteration, the generated output activations scaled and accumulated at the gating circuit system 620 are sent to the next level at the processing block 602 or another physical processing block of the virtual processing block, and the next level is based on the next expert index in the index output sequence (i.e., f(x l )) is activated, the next expert index responding to the input x of the input sequence received at any processing block 610 l Furthermore, in this embodiment, the gating value is determined in the expert selection function logic. A non-limiting example of a gating function is:

[0079] softmax(input * fixed_weight_matrix)

[0080] Where fixed_weight_matrix represents the matrix learned in different processes.

[0081] exist Fig. 8A , Figure 8B In another embodiment described in the iterative process shown, for each input x, the expert selection logic function f(x) generates a list of values ​​(soft selection) to select multiple experts / levels. As an illustrative, non-limiting example, the expert selection logic function f(x) generates an example output as follows:

[0082] f(x) = [{2, 3}]

[0083] With gating function output: g {2，3} (h) = [{0.1, 08}] means that in order to process the activation h, two experts 2 and 3 are selected, and the corresponding weight coefficient of the output of expert 2 is = 0.1 ( Fig. 8A The corresponding weight coefficient of the output of expert 3 is = 0.8 ( Figure 8B ). Thereafter, the method computes the output activation as

[0084] In this embodiment, given an input sequence [x 1 , x 2 , ..., x L], expert selection function logic calculation [f(x1), f(x2), ..., f(xL)], expert index [e f(x1) (h1), e f(x2) (h2), e f(xL) (hL)] and the corresponding gating (weight) value [g f(x1) (h1), g f(x2) (h2), g f(xL) (hL)]), where L is the sequence length and hl is the input activation 626 from the previous layer. In such an embodiment, when a block is processed using a single 3D CiM accelerator system, the system feeds the input activation h 626 to the jth level 616 (e.g., through a different expert) and adds the weighted output to the calculated output 658 of the physical processing block.

[0085] Fig. 9 The multi-expert 3D CiM accelerator system processing block 603 shown in FIG. 6 depicts a stack of CiM layers 616 that implement one or more expert models corresponding to the MoE FFN. At processing block 603, given a single input x l The iterative process of (where l = 1, 2, ..., k) consists of applying the function f(xl) and using the input activations hl to produce the expert index e j 650 is used to activate the j-th level 616 (where j=0, 1, ... k-1). The j-th level includes f(xl) (hl) determines the MoE expert model. The applied function f(xl) and the input activation hl also produce the MoE expert model according to g f(xl) (hl) determines the corresponding output weight or gate value g j 652. In the first iteration, the layer output 658 of the processing block is initialized to a zero vector. For each value of f(xl) = [{f(xl)1, f(xl)2, ...}], the system activates the corresponding j-th layer (0 to k-1) and passes the input to the layer for processing to obtain the corresponding output 655. The output 655 is compared with the corresponding gate value g j multiplied and added to the memory of the processing block 603. Finally, the system returns the accumulated output 658 of the physical processing block. Once the iteration passes through the sequence x l , the scaled, accumulated and stored output activations 658 can then be passed to the next layer of the model for further processing.

[0086] Fig. 10A and Fig. 10BDepicted is an additional embodiment of a 3D CiM accelerator system 700 configured for selecting multiple experts for single-layer activation using an activation-based MoE. In this activation-based MoE embodiment, activations received from a previous layer are used / processed to determine which experts / layers of a processing block 702 will be active. In this embodiment, the expert indexes are dynamically / on-the-fly calculated; i.e., they are not known prior to execution of the previous layer. In one embodiment, a single expert is activated per layer. In one embodiment, multiple layers / experts are used in each processing block. Each expert contributes to the output of the processing block, which contribution is proportional to its gating value. Determination of which layers / experts to use for the layer is based on the input activations.

[0087] like Fig. 10A As shown, the expert selection function logic processing circuit 660 at processing block 702 selects the activation sequence h=[h 1 ,h 2 , ..., h L ] (e.g., generated by processing of a previous layer) to generate an expert index or selection. Specifically, in response to applying a hash function f to the activation sequence (f(h l )) to generate a first expert index 711. The layer activation block 715 at the processing block T 702 associated with the neural network layer receives the activation sequence h and the sequence f(h) based on the corresponding received activation h. l ) 725 to generate a first wordline signal 711, the received activation h generated when the expert select function logic processes the corresponding received activation h received from the previous layer. The signal 711 activates the level associated with the MoE expert (e.g., level 716A) for generating a first set of output activations 741.

[0088] For example, in response to a first generated expert index (e.g., f(h l ) 1 ) The hierarchical activation circuit 715 generates a first word line signal, for example, a word line 711 (eg, WL 2 ) to activate the first level associated with the first MoE expert, e.g. Fig. 10A , for processing / generating additional output activations 741 . Fig. 10A Further shown is the first input h in the input activation received at the expert selection function logic processing block 660 in response to the expert selection function logic. 1 The corresponding generated gating value 739 is used to apply to the output activation 741 after analog-to-digital conversion of the output activation 741. The result is propagated to and accumulated at the gating circuit system 720, which scales and accumulates the model layer output. Fig. 10A , Fig. 10B In the embodiment shown in FIG. 1 , the generated output activations scaled and accumulated at the gating circuit system 720 are sent to the next level at the processing block 702 to be activated based on the second input activation h in response to the input sequence received at the expert selection processing block 660. 2 The next expert index generated (i.e., f(h 2 That is, in this embodiment, based on the current input activation sequence h l The received expert index 725 is used to sequentially activate the expert model at one or more levels to process the current generated activation set received from the memory, which stores the sequence h l The output activations (not shown) of the immediately preceding iteration of processing are the result of processing at this level. The processing at this level generates additional intermediate output activations 741 to be scaled, accumulated, and stored for use in the next iteration of the input activation sequence. If the expert spans multiple levels, i.e., level 1 of expert 1 resides on level 1 and level 2, then the output from level 1 will be fed back into level 2 of the same physical processing block. The output of level 2 will then be scaled using the gating value and then stored.

[0089] Therefore, if Fig. 10B As shown, in the next processing sequence of sequentially activating the expert models at one or more levels, in response to the next (e.g., second) generated expert index, i.e., f(h l ) 2 , the hierarchical activation circuit 715 generates the next word line signal, for example, word line 712 (eg, WL 3 ) to activate another level associated with a second MoE expert, e.g. Fig. 10B The layer 716B shown is used to process / generate another intermediate output activation 743 for the next iteration of the input activation sequence (e.g., the next input h l ). Fig. 10B Further shown is the activation of the second input set h by the expert selection function logic in response to the second input received at the expert selection processing block 660. 2 The corresponding generated gating value 749 is applied to the output activation 743 after the output activation 743 has been analog-to-digital converted. The result is propagated to and accumulated at the gating circuit system 620, which scales and accumulates the model layer output.

[0090] In this further embodiment, Fig. 10A , Fig. 10B The process shown in is iterative, i.e., for the input activation sequence h land the expert selection function logic applied at processing block 660. In each iteration, the generated output activations scaled, accumulated, and stored at gating circuit system 620 may be transferred to the next level at processing block 602 or another physical processing block, where the next level is selected based on the next expert index of the output index sequence (i.e., f(h l )) is activated, the next expert index is activated in response to the input sequence received by the expert selection processing block 660 l In addition, in this embodiment, the gating value is determined in the expert selection function logic. A non-limiting example of a gating function is:

[0091] softmax(input * fixed_weight_matrix)

[0092] Where fixed_weight_matrix represents the matrix learned in different processes.

[0093] Fig. 10A , Fig. 10B In a further embodiment described in the iterative process shown, for each input activation h, the expert selection logic function f(h) generates a list of values ​​(soft selection) to select multiple experts / levels. As an illustrative, non-limiting example, the expert selection logic function f(h) generates an example output as follows:

[0094] f(x) = [{2, 3}]

[0095] With gating function output: g {2，3} (h) = [{0.1, 0.8}], indicating that in order to process activation h, two experts 2 and 3 are selected, and the corresponding weight coefficient of the output of expert 2 is = 0.1 ( Fig. 10A The corresponding weight coefficient of the output of expert 3 is = 0.8 ( Fig. 10B ). Thereafter, the method calculates the output activation as

[0096] In further embodiments, the 3D CiM accelerator system implements multiple models residing at different levels.

[0097] refer to Fig.11 In another embodiment, the 3D CiM accelerator system 800 implements multiple different models and is configured to select a single (multiple) expert in a multi-model system. In such an input-based multi-model system, the inputs to the models are used / processed to define which experts / layers will be active. The inputs to the models are received at the beginning of model inference, so the experts / layers for all layers can be determined in advance if necessary. In an embodiment, it is assumed that each layer has a single activated model.

[0098] like Fig.11 As shown, an arbitrary processing block 810 (eg, at virtual processing block 202) is shown that receives an input sequence x l 805, the block is an expert selection or hash function f(x l ). Block 810 receives the sequence x l As input and output f(x l ) 825. This block can perform this operation in the analog or digital domain. Any processing block 810 can be part of processing block 802 or can physically reside in another part of accelerator system 800. The layer activation circuit system 815 at processing block T802 associated with the neural network layer receives the output f(x l ) and the corresponding received activation h resulting from processing in the previous neural network model layer l 826 to generate a wordline signal (eg, wordline 811) to activate a level associated with the model (eg, level 806A) for generating a model output (eg, output activation 841).

[0099] Fig.12 Conceptually describes Fig.11 The multi-model 3D CiM accelerator system 800 is used to select a single (multiple) expert in the multi-model system. Fig.12 , a multi-model 3D CiM accelerator system 800 is shown, wherein in response to receiving an input data sequence x l =[x 1 , x 2 , ..., x L ] and activate a single model, where (N is a natural number greater than or equal to 1). Given an input sequence x 805, a function f() is applied to the input sequence x to determine the model e 827 associated with the model to be used to process the input. Unlike other embodiments, these models are clearly separate and do not interact, i.e., each model (expert) operates completely independently. In one embodiment, given K models and an input sequence x l , the function f generates a single integer f(xl)∈{0,...,k-1} (hard choice) that is used to infer the input data sequence x lA single value (hard choice) 827 is generated to select the model for execution at a single level 806. For example: the system 800 consists of multiple models for language translation from multiple input languages ​​(e.g., De, Fr, It, Sp) to a single output language (En). In this case, there will be 4 different models, each implementing the mapping (input language -> En). The function f() can be a neural network that infers the input language and selects the corresponding model.

[0100] review Fig.11 In the embodiment, given an input sequence x=[x 1 , x 2 , ..., x L ], applying the function f() generates a single value (hard selection) 825 for selecting a single expert / level. In one embodiment, any processing block 810 generates an index sequence [f(x1), f(x2), ..., f(xL)] and a corresponding gating value 839 [g f(x1) (h1), g f(x2) (h2), ..., g f(xL) (hL)] = [{1}, {1}, ..., {1}] That is, all generated gating values ​​are equal to 1. Here, and

[0101] Based on the input x for all processing blocks l The generated expert index 825 is determined. That is, in one embodiment, the same arbitrary processing block 810 determines the expert index of the virtual processing block, that is, all physical processing blocks (processing block T, processing block T+1, etc.), and the data stream 830 of the expert index is transmitted from the arbitrary processing block 805 to the hierarchy activation circuit system 815 at the current processing block and / or other processing blocks (e.g., processing block T+1). In an alternative embodiment, the input sequence x can be provided to all processing blocks 802 at the same time, and each processing block can determine its own activation hierarchy. Fig.11 As further shown, in the example, based on the input x l The received expert index 825 is used to activate the expert model at the level (e.g., level 806A), which processes the data and any activations and generates additional intermediate output activations 841. Fig.12 As further shown, the corresponding generated gating value (=1) 839 generated by the expert selection function logic at any processing block 810 is applied to the output activation 841 after the output activation 841 has been analog-to-digital converted. The result is accumulated at the gating circuit system 820, which scales, accumulates and stores the model layer output.

[0102] Fig.13A and Fig. 13B Depicted is an additional embodiment of a 3D CiM accelerator system 850 configured to select multiple experts (models) residing at different levels of a 3D multi-model CiM system. In this activation-based MoE embodiment, activations received from a previous layer are used / processed to determine which experts / levels in a processing block 702 will be active. In this embodiment, the expert index is dynamically / on-the-fly calculated. Here, the method can pick a model based on user input and activate the layer corresponding to a specific model in each processing block.

[0103] refer to Fig.13A In another embodiment, the 3D CiM accelerator system 850 implements multiple different models (experts) and is configured to select multiple experts of a multi-model system. In this input-based multi-model system, given an input sequence [x 1 , x 2 , ..., x L ],in An expert selection function f is applied to determine the index of the model that needs to be used to process this input (soft selection). The models are clearly separated and do not interact with each other. Given K models corresponding to multiple processing blocks 852 and an input sequence, the function f generates an index list of models for inference. Function f can also select all models K. Gating is only applied to the final output of the inference. This embodiment can be used in cases involving an ensemble of neural networks.

[0104] like Fig.13A As shown, it is shown that any processing block 810 receives an input sequence x l 805, the block is an expert selection or hash function f(x l ). Block 810 receives an input data sequence x l As input and output expert index f(x l ) 825, where l = 1, 2, ..., L. This block can perform this operation in the analog or digital domain. Any processing block 810 can be part of the same processing block 852 or can physically reside in another part of the accelerator system 850. The layer activation circuit system 815 at the processing block T 852 associated with the neural network layer sequentially receives the output f(x l ) and the activation h received from the previous layer processing l 826, to generate a corresponding hierarchical activation wordline signal. For example, at processing block 852, in response to a first generated expert index, such as f(x l ) 1 , the hierarchical activation circuit 815 generates a first word line signal, for example, a word line 861 (eg, WL 2 ) to activate the first level associated with the first model, for example, Fig.13A The layer 816A shown in FIG. 8 is used to process / generate the additional output activation h l In addition, via data flow 830, the model selection output f(x l ) 1 The data is simultaneously transmitted to the hierarchical activation circuit 815 of another processing block (e.g., another processing block K 853) to activate the same word line and select another first model (e.g., the model running at the hierarchical level 817A at the processing block 853) to process the input data sequence x l Multiple models at different processing blocks can be selected to run against this input data. Fig.13A Further shown is the process by which the expert select function logic responds to the first input x of the input sequence received at any processing block 810. 1 The corresponding generated gating value 839 is then applied to the output activation 841 after it has been analog-to-digital converted at another processing block 853. The result is propagated to and accumulated at the gating circuitry 820 at the last processing block K 853, which scales and accumulates the model layer outputs from all processing blocks. Fig.13A , Fig. 13B In the embodiment shown in FIG. 8 , the generated output activations scaled and accumulated at the gating circuit system 820 are sent to the next level at the processing block 852 to be activated based on the second input x in response to the input sequence received at any processing block 810. 2 The next expert index generated (i.e., f(x 2 That is, in this embodiment, based on the current input sequence x l The received expert index 825 is used to sequentially activate the expert model at one or more levels to process the current input data, except for the expert model from the stored sequence h l In addition to the set of generated activations received by the memory of the output (not shown) activation results of the immediately preceding iterative processing at 826. The processing at this level also generates additional intermediate output activations 841 to be scaled, accumulated and stored for use in the next iteration of the input sequence.

[0105] Therefore, in Fig. 13B In the next processing sequence of sequentially activating one or more levels of expert models as shown, in response to the next (e.g., second) generated expert index, i.e., f(x l ) 2 , the hierarchical activation circuit 815 generates the next word line signal, for example, word line 862 (eg, WL 3 ) to activate another level associated with the second model, e.g. Fig. 13BThe layer 816B shown is used to process the input data. In addition, through the data flow 830, the expert model selects the output f(x l ) 2 The hierarchical activation circuit 815 is simultaneously transmitted from any processing block to other processing blocks (e.g., processing block K 853) to activate the same word line and select another second model (e.g., the model running at the hierarchical level 817B at processing block 853) to process the input data sequence x l . Fig. 13B Further shown is the selection of a second input x by the expert selection function logic in response to the input sequence received at any processing block 810. 2 The corresponding gating value 849 is generated, which is applied to the output activation 843 after analog-to-digital conversion of the output activation 843. The result is propagated to and accumulated at the gating circuit system 820, which scales, accumulates and stores the model layer output.

[0106] Fig.13A , Fig. 13B The process shown in is iterative, that is, for the input sequence x l and the expert selection function logic applied at any processing block 810 are repeated for as many models / layers as required. At each iteration, the generated output activations scaled and accumulated at the gating circuit system 820 are sent to the next layer at processing block 852 or another processing block 853, and the next layer is based on the next expert index of the index output sequence (i.e., f(x l )) is activated, the next expert index responds to the input x of the input sequence received at any processing block 810 l Furthermore, in this embodiment, the gating value is determined in the expert selection function logic. A non-limiting example of a gating function is:

[0107] softmax(input * fixed_weight_matrix)

[0108] Where fixed_weight_matrix represents the matrix learned in different processes.

[0109] Fig.13A , Fig. 13B In another embodiment described in the iterative process shown, for each input x, the expert selection logic function f(x) generates a list of values ​​(soft selection) to select multiple experts / levels. As an illustrative, non-limiting example, the expert selection logic function f(x) generates an example output as follows:

[0110] f(x) = [{2, 3}]

[0111] With gating function output: g {2，3} (h) = [{0.1, 0.8}] means that in order to process the activation h, two experts 2 and 3 are selected, and the corresponding weight coefficient of the output of expert 2 = 0.1 (such as Fig.13A The weight coefficient of the corresponding output of expert 3 = 0.8 (as shown in Fig. 13B shown).

[0112] Fig.14 A method 900 is depicted implemented at a processing block of a 3D CiM MoE accelerator system or at any processing block associated therewith. This method may be implemented by controlling the processing to implement expert selection function logic at the block or by controlling the processing to implement expert selection function logic at any processing block, e.g. Figure 4 , Fig. 8A , Figure 8B , Fig.11 In the embodiments shown in the above, method 900 begins by receiving an input data set at an expert selection function logic at any processing block or processing tile to start MoE model inference at 902. The input data set may include input data and / or intermediate activations generated by previous processing iterations at the same or different processing blocks. As shown in method 905, the system calls an expert selection logic function (e.g., a hash function) to apply the hash function to the input data set at the processing block. At 910, it is determined whether the expert selection logic function results in the generation of a hard selection value (e.g., a single expert selection index value). If a single hard selection value is generated by the expert selection logic function, the process proceeds to 913, where it is further determined whether multiple experts / models are to be called to process the input data. At 913, if the system determines that only a single expert / model is to be called to process the input data, the process proceeds to 915 to activate a single model / expert at a single level of the processing block (based on the hard selection expert selection function value) to process the input data set. Otherwise, at 913, if it is determined that multiple experts / models are to be called to process the input data, the process proceeds to 920 ( Fig.14 ), where the system activates models / experts at the hierarchy at multiple processing blocks. The logic employed at any processing block or expert selection function logic block that receives the input data set can determine another processing block from the multiple physical processing blocks of the virtual processing block to process the same input data set. Alternatively, the expert selection function result can be sent to receive at all processing blocks or a subset of processing blocks in the system, and each processing block will independently process the hard selection value to determine whether to activate the corresponding expert / hierarchy at the corresponding processing block. At each processing block in the determined multiple processing blocks, the hierarchy associated with each model / expert will be selected (based on the hard selection expert selection function value) to process the received input data.

[0113] review Fig.14 In 910, if it is determined that the applied expert selection logic function results in the generation of soft selection values ​​(e.g., more than a single expert selection index value), the process proceeds to 930, where it is further determined whether multiple experts / models are to be called to process the received input data. At 930, if the system determines that only a single expert / model is to be called to process the input data at a single processing block, the process proceeds to 935 to activate a single model / expert at the first level of the processing block (based on the first soft selection expert index value result) to process the input data set and any activations / outputs scaled and received from the previous network layer. Then, at the processing block, the system iteratively activates one or more next levels (based on the remaining corresponding soft selection expert index values) to process the input data set.

[0114] Otherwise, at 930, if it is determined that multiple experts / models are to be invoked to process the input data, the process continues to Fig.14 940, wherein the system activates a model / expert at each of a plurality of physical processing blocks of a virtual processing block. The logic employed at any processing block or expert selection function logic block that receives an input data set can determine which of the other plurality of physical processing blocks to process the same input data set. Alternatively, the expert selection function result can be sent to receive all processing blocks or a subset of processing blocks in the system, and each corresponding processing block will independently process the corresponding soft selection expert selection index value to activate the corresponding (multiple) expert / (multiple) layers at the corresponding processing block. At each of the determined plurality of physical processing blocks, the system will iteratively activate one or more layers (based on the remaining corresponding soft selection expert index values) to process the received input data set and any activations / outputs scaled and received from previous neural network model layers.

[0115] Fig.15 An example computing system according to the present invention is shown, which can provide Fig.14Activation of a memory hierarchy based on control unit functionality as shown in the method described in the foregoing to control dynamic MoE switching in a 3D CiM accelerator system. It should be understood that the depicted computer system is only an example of a suitable processing system and is not intended to imply any limitation on the scope of use or functionality of embodiments of the present invention. For example, the system shown can operate with many other general or special computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the system shown in the figure may include, but are not limited to, integrated circuits, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, small computer systems, large computer systems, and distributed cloud computing environments including any of the above systems or devices, etc.

[0116] In some embodiments, computer systems may be described in the general context of computer system executable instructions, which are embodied as program modules stored in memory 16 and executed by the computer system. In general, program modules 10 may include routines, programs, objects, components, logic, data structures, etc., which are described herein in terms of Fig.14 The described methods perform specific tasks and / or implement specific input data and / or data types.

[0117] The components of the computer system may include, but are not limited to, one or more processors or processing units 12, memory 16, and bus 14, which operably couples various system components (including memory 16) to processor 12. In some embodiments, processor 12 may execute one or more modules 10 loaded from memory 16, wherein the program module(s) are embodied as software (program instructions) that causes the processor to perform one or more method embodiments of the present invention. In some embodiments, module 10 may be programmed into an integrated circuit of processor 12, loaded from memory 16, storage device 18, network 24, and / or a combination thereof.

[0118] Bus 14 may represent one or more of a variety of types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using a variety of bus architectures, such as, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0119] The computer system may include a variety of computer system readable media. Such media may be any available media that can be accessed by the computer system and may include volatile and nonvolatile media, removable and non-removable media.

[0120] Memory 16 (sometimes referred to as system memory) may include computer-readable media in the form of volatile memory, such as random access memory (RAM), cache, and / or other forms. The computer system may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 18 may be provided to read and write non-removable, non-volatile magnetic media (e.g., a "hard drive"). Although not shown, a disk drive for reading and writing a removable, non-volatile disk (e.g., a "floppy disk") may be provided, as well as an optical drive for reading or writing a removable, non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media). In this case, each may be connected to bus 14 via one or more data media interfaces.

[0121] The computer system may also communicate with one or more external devices 26, such as a keyboard, pointing device, display 28, etc.; one or more devices that enable a user to interact with the computer system; and / or any device that enables the computer system to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 20. In addition, the computer system may communicate with one or more networks 24, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via a network adapter 22. As shown, the network adapter 22 communicates with other components of the computer system via bus 14. Although not shown, other hardware and / or software components may be used in conjunction with the computer system. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0122] The present invention may be a system, method and / or computer program product at any possible level of technical detail integration. The computer program product may include one (or more) computer-readable storage media having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0123] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punched card or a raised structure in a groove on which instructions are recorded), and any suitable combination of the above. The computer-readable storage medium used herein should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0124] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0125] The computer-readable program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and procedural programming languages, such as "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, an electronic circuit system (e.g., including a programmable logic circuit system, a field programmable gate array, or a programmable logic array (PLA)) may be personalized electronic circuit systems by utilizing the state information of the computer-readable program instructions, thereby executing computer-readable program instructions to perform aspects of the present invention.

[0126] This article describes aspects of the present invention with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to an embodiment of the present invention. It should be understood that each block in the flow chart and / or block diagram and the combination of blocks in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0127] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device create means for implementing the functions / actions specified in the flowchart and / or (multiple) block diagram blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium, which may instruct a computer, a programmable data processing device, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in the flowchart and / or (multiple) block diagram blocks.

[0128] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in the flowchart and / or (multiple) block diagram blocks.

[0129] The flow chart and block diagram in the figure show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each block in the flow chart or block diagram can represent a part of a module, segment or instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the function indicated in the block may not occur in the order indicated in the figure. For example, two blocks displayed continuously can actually be completed as a step, executed simultaneously, executed substantially simultaneously, executed in a manner that overlaps part or all of the time, or sometimes blocks can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flow chart and the combination of blocks in the block diagram and / or flow chart can be implemented by a dedicated hardware system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0130] The description of various embodiments of the present invention is for illustrative purposes only and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found on the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0131] The terms used herein are only for the purpose of describing specific embodiments, rather than to limit the present invention. As used herein, the singular forms "one", "an" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the terms "include" and / or "comprise" when used in this specification specify the existence of stated features, integers, steps, operations, elements and / or components, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. The corresponding structures, materials, actions and equivalents of all elements in the following claims are intended to include any structure, material or action for performing functions in combination with other elements claimed for protection specifically claimed. A description of the present invention has been given for the purpose of illustration and description, but the description is not exhaustive or limits the present invention to the disclosed form. Without departing from the scope and spirit of the present invention, many modifications and changes are obvious to those of ordinary skill in the art. The embodiments are selected and described in order to best explain the principles and practical applications of the present invention, and to enable other persons of ordinary skill in the art to understand the various embodiments of the present invention with various modifications, which are suitable for the specific purposes envisioned.

Claims

1. A computing-in-memory CiM system, comprising: a 3D memory system comprising a plurality of levels of in-memory computing devices, each level comprising a memory system associated with an expert of a mixture of experts MoE neural network model; a control processor to select, in response to received input, one or more levels of the in-memory computing device associated with the expert at the network model layer; The control processor configures computational paths at the selected one or more of the plurality of levels associated with the expert, the computational paths being used to process the input at the network model layer and generate output activations for use at additional neural network model layers.

2. The CiM system of claim 1 , wherein to select the one or more levels associated with an expert, the control processor is configured to: implementing first logic for applying a function to said received input to generate an output expert selection value; In response to the expert selection value, a level of the in-memory computing device associated with the selected expert at the network model layer is activated.

3. The CiM system of claim 2, wherein the received input comprises: Model input data for inference processing by the selected expert at the neural network model layer.

4. The CiM system of claim 2, wherein the activated level is associated with a current layer of the MoE neural network model, wherein the received input comprises: Hidden layer activation values ​​generated as a result of processing at the previous neural network model layer.

5. The CiM system of claim 2, wherein the selected expert at the neural network model layer generates hidden layer activation values ​​for propagation to a next neural network model layer, the next neural network model layer being located at a different level of an in-memory computing device associated with the same selected expert, or at a different processing block associated with the same selected expert of the configured computing path.

6. The CiM system of claim 2, wherein one or more selected experts at the neural network model layer process the same received input, each of the one or more selected experts generating one of the following: a model output or a hidden layer activation value in response to processing of the received input, the generated hidden layer activation value being used to propagate to different levels or different processing blocks of a computing device in a memory, or to propagate to a next neural network model layer not associated with the expert.

7. The CiM system according to claim 2, further comprising: A gating circuit system is selected to scale and accumulate the generated hidden layer activation values ​​associated with the inference processing performed by the expert at one or more levels associated with the expert at one or more processing blocks; the gating circuit system also propagates the accumulated hidden layer activation result values ​​to the next network model layer.

8. The CiM system according to claim 2, further comprising: A pulse signal generator for generating a pulse signal having characteristics associated with the value of the received input, the control processor implementing the pulse signal when performing a matrix-vector multiplication operation at the level selected and associated with the selected expert in the in-memory computing device.

9. The CiM system of claim 2, wherein each of the hierarchies of in-memory computing devices comprises a non-volatile memory system configured as a crossbar array of memory storage cells, the crossbar array being implemented for matrix-vector multiplication operations associated with experts of the MoE neural network model.

10. A computing-in-memory CiM system, comprising: A plurality of processing blocks, wherein a layer of a deep neural network model corresponds to one or more processing blocks of the plurality of processing blocks, wherein the layer receives input data for performing inference processing using the deep neural network model, and the processing block comprises: a three-dimensional 3D multi-level configuration of a computing-in-memory device, each level comprising a crossbar array of memory cells, the crossbar array comprising a memory system associated with a neural network model; controlling a processor circuit to select one or more levels of a computing device in memory associated with the neural network model in response to receiving the input data; The control processor circuit configures computational paths at one or more of the plurality of levels associated with the selected model, the computational paths being used to process the received input data at the network model and generate one or more outputs for use at another neural network model.

11. The CiM system of claim 10, wherein to select the one or more levels associated with the neural network model, the control processor circuit is configured to: applying a function to said received input data to generate an output model selection value; In response to the model selection value, a level of the in-memory computing device associated with the neural network model is activated.

12. The CiM system of claim 11, wherein the received input data comprises model input data for inference processing by the selected neural network model.

13. The CiM system of claim 11, wherein the activated level is associated with a current layer of the neural network model, and the received input data comprises: Hidden layer activation values ​​generated as an output result of processing at the level of the in-memory computing device at the previous neural network model layer.

14. The CiM system according to claim 11, further comprising: A gating circuit system for scaling and accumulating the generated hidden layer activation values ​​associated with the inference processing results at one or more levels associated with the model layers, and the gating circuit system also propagates the accumulated activation result values ​​to one or more of the following: different layers at the same level in the same model, or different layers at different levels and located in one of the following: the same processing block or different processing blocks.

15. A method for operating an in-memory computing system, comprising: One or more processing blocks are configured according to the received input data, each processing block is associated with a layer of the neural network model, and each processing block includes: A three-dimensional 3D multi-layer configuration of computing devices in a computer memory configured to process data associated with an expert of a mixture of experts neural network model; and controlling the processor circuit to iteratively select one or more levels of the computing device in memory associated with the expert in response to the received input data; The computational path is configured, using the control processor, at a selected one or more of the plurality of levels associated with the expert, the computational path being used to process the input at the MoE network model layer and generate one or more output activations for use at another neural network model layer.

16. The method of claim 15, wherein selecting the one or more levels in response to received input data comprises: applying, at a first expert selection logic circuit, a function to said received input data to generate an output expert selection value; Responsive to the expert selection value, a level of an in-memory computing device associated with the selected expert of the MoE network model layer is activated.

17. The method of claim 16, wherein the received input data comprises: Model input data for inference processing by one or more selected experts at the neural network model layer, the method further comprising: At each of the one or more selected experts, one of the following is generated: a model output or hidden layer activation value of the processing in response to the received input, and the generated hidden layer activation value is used to propagate to different levels or different processing blocks of the computing device in the memory, or to propagate to the next neural network model layer not associated with the expert.

18. The method according to claim 17, further comprising: scaling and accumulating, at one or more levels associated with the expert at one or more processing blocks, the generated hidden layer activation values ​​associated with the inference processing performed by the expert; as well as The accumulated hidden layer activation values ​​are propagated to the same expert at a different level at one of the following: the same processing block or a different processing block, or the output activation values ​​are propagated to the next network model layer.

19. The method according to claim 18, further comprising: A gating value is generated at the first expert selection logic circuit in response to the received input data, the gating value being used to scale the output activation value prior to the propagation.

20. The method of claim 16, wherein the activated level is associated with a current layer of the MoE neural network model, and the received input data comprises: Hidden layer activation values ​​generated as a result of processing at previous neural network model layers and propagated from previous neural network model layers.

Images