Method and apparatus for improved generative ai models

Abstract

An improved neuron model for a spiking neural network is provided. The neuron model includes a soma synapse and a dendrite synapse. The soma synapse is used to receive an input token, and the dendrite synapse is used to receive context data. The context data is used to modulate the behavior of the spiking neuron. The neuron model can be used to implement an attention mechanism in an SNN and to implement a hybrid expert architecture.

Description

[0001] Cross-citation of related applications This application claims priority to U.S. Application No. 18 / 413,957, filed January 16, 2024. Technical Field

[0002] This invention relates to generative AI models. Specifically, this invention relates to generative AI models that integrate dendritic dynamics in spiking neural networks. Background Technology

[0003] Generative artificial intelligence (AI) models have recently garnered significant attention, with examples such as the advent of ChatGPT. While the potential of such models is increasingly evident, current models suffer from numerous drawbacks, including large scale (e.g., billions or even trillions of parameters), high computational costs in both training and inference phases, high power consumption, high infrastructure costs, high memory requirements, and high latency, particularly during inference. Summary of the Invention

[0004] The purpose of this invention is to provide a method for improving the performance of generative AI models.

[0005] In a first aspect, a method for implementing a spiking neural network (SNN) is provided, comprising: creating at least one layer of spiking neurons, wherein each spiking neuron includes a cell body input of dimension d, at least one dendritic input of dimension d, and an output of dimension d; providing input terms through the cell body input; and providing contextual data through the at least one dendritic input, wherein the output of each of the spiking neurons is modulated by the contextual data.

[0006] In a second aspect, a computing device is provided, comprising a processor and a memory, wherein the processor and the memory are configured to create at least one layer of spiking neurons, wherein each spiking neuron comprises a cell body input of dimension d, at least one dendritic input of dimension d, and an output of dimension d; input terms are provided through the cell body input; and context data is provided through the at least one dendritic input, wherein the output of each of the spiking neurons is modulated by the context data.

[0007] In a third aspect, a computer-readable medium is provided having executable code stored thereon for execution by a processor of a computing device, the executable code including instructions for: creating at least one layer of spiking neurons, wherein each of the spiking neurons includes a cell body input of dimension d, at least one dendritic input of dimension d, and an output of dimension d; providing input terms through the cell body input; and providing context data through the at least one dendritic input, wherein the output of each of the spiking neurons is modulated by the context data.

[0008] Specifically, a neural network can consist of layers of spiking neurons, each spiking neuron having a cell body input of dimension d, at least one dendritic input of dimension d, and an output of dimension d. The dimension can correspond to the embedding dimension. Input terms can be provided as cell body inputs, and contextual data is provided through dendritic inputs, allowing each neuron to use contextual data when computing its output.

[0009] In at least some implementations of the first, second, or third aspect, the at least one dendritic input comprises N basal dendritic inputs and N apical dendritic inputs, and the maximum input sequence length is N.

[0010] Dendritic inputs can be divided into apical dendritic inputs and basal dendritic inputs. The number of such inputs can represent the maximum input sequence length.

[0011] In at least some implementations of the first, second, or third aspect, the context data includes an input sequence.

[0012] The input sequence can be provided as contextual data so that neurons can perceive the entire input sequence.

[0013] In at least some implementations of the first, second, or third aspect, the input lexical includes a query for selected lexical elements of the input sequence.

[0014] Input terms can include queries, for example, when calculating attention.

[0015] In at least some implementations of the first, second, or third aspect, the N basal dendritic inputs are used to receive N keys, and the N apical dendritic inputs are used to receive N values, wherein each of the N keys and each of the N values ​​is derived from N lexical terms in the input sequence.

[0016] When calculating attention, keys and values ​​can be provided via dendritic inputs.

[0017] In at least some implementations of the first, second, or third aspect, the N basal dendritic inputs and the N apical dendritic inputs are shared dendrites, and the output of the shared dendrites is received by each of the spiking neurons.

[0018] Some dendritic inputs can be shared by spiking neurons within the layer.

[0019] In at least some implementations of the first, second, or third aspects, the output of each of the spiking neurons is determined by the following:

[0020] Among them, out i It is the output of neuron i, Q i It is the selected query, K j and V j Let represent the N keys and the N values, and sim be the similarity function.

[0021] The output of a neuron can be calculated based on the above formula.

[0022] In at least some implementations of the first, second, or third aspect, the similarity function is one of an exponential function and a cosine similarity function.

[0023] Different similarity functions can be used.

[0024] In at least some implementations of the first, second, or third aspect, the output of the shared dendrite includes a scalar determined by:

[0025] and the vector determined by the following:

[0026] Among them, K j and V j This represents the N keys and the N values, and It is a non-negative function representing the underlying features.

[0027] Shared dendrites can perform computations for multiple neurons, thus avoiding redundant computations.

[0028] In at least some of the implementations of the first, second, or third aspect, (x) = elu(x) + 1, where elu is an exponential linear unit function.

[0029] The exponential linear unit function can be used to calculate .

[0030] In at least some implementations of the first, second, or third aspects, the output of each of the spiking neurons is determined by the following:

[0031] Where S is the scalar and V is the vector.

[0032] Spiking neurons can implement the attention mechanism based on the above formula. Attached Figure Description

[0033] The invention will be better understood with reference to the accompanying drawings, in which: Figure 1A This is a simplified diagram of a pyramidal neuron; Figure 1B This is a diagram of a point neuron model; Figure 1C This is a diagram of a dendritic neuron model; Figure 2 It is a graphical representation of the behavior of the LIF neuron model; Figure 3 It is a graphical representation of a spiking neural network (SNN); Figure 4 This is a graphical representation of the dendritic neuron model provided in at least some embodiments of the present invention; Figure 5 This is a block diagram of a method for handling attention in an SNN provided by at least some embodiments of the present invention; Figure 6 This is a graphical representation of the dendritic neuron model provided in at least some embodiments of the present invention; Figure 7 This is a graphical representation of the dendritic neuron model provided in at least some embodiments of the present invention; Figure 8 This is a graphical representation of the shared dendritic model provided in at least some embodiments of the present invention; Figure 9 These are graphical representations of neuron models using shared dendrites provided by at least some embodiments of the present invention; Figure 10 This is a graphical representation of a neuron layer with shared dendrites provided in at least some embodiments of the present invention; Figure 11 It is a graphical representation of a neural network that implements a hybrid expert architecture; Figure 12 This is a graphical representation of a dendritic SNN architecture for implementing hybrid expert applications provided by at least some embodiments of the present invention; Figure 13This is a graphical representation of a neuron model for implementing hybrid expert applications provided by at least some embodiments of the present invention; Figure 14 This is a block diagram of an exemplary computing device for implementing embodiments of the present invention. Detailed Implementation

[0034] This invention relates to methods and apparatus for improving the efficiency of generative AI models.

[0035] Various methods have been employed to improve the efficiency and reduce the cost of generative AI models: Optimization methods: These strategies include techniques such as model compression and innovative learning techniques. The main goal is to achieve acceptable performance while using faster and less complex models.

[0036] Novel Algorithmic Approaches: The initial Transformer architecture, built on attention mechanisms, offered powerful functionality but was limited by quadratic complexity in terms of memory and computation. The demands on memory and computation increase with the length of the input sequence. Innovations such as linear attention, non-attention, low-rank, and sparse transformers aim to improve power efficiency by reducing computational load and memory usage. Despite their promise, optimal solutions in this field remain under investigation.

[0037] Energy-efficient computing—neuromorphic computation (NC): This strategy utilizes spiking neural networks (SNNs) for data representation and processing. NC promises to achieve energy efficiency by using low-precision data and cost-effective computation, but it still lags behind traditional deep learning networks, such as artificial neural networks (ANNs), in terms of performance. Both the hardware and software involved in NC are still in the experimental stage.

[0038] Among these methods, NC stands out due to its inherent efficiency. NC can also be combined with other methods to provide a way to optimize power consumption in large AI models. To date, the only implemented SNN-based language model is SpikeGPT. Although SpikeGPT is relatively small in scale (260 million parameters) compared to typical ANN language models (billions of parameters), it represents an important step towards integrating SNN-based generative AI. However, to achieve performance comparable to the ANN Transformer, SNN Transformers must find ways to scale up.

[0039] SpikeGPT significantly reduces computational requirements by utilizing a receptance-weighted key-value (RWKV) Transformer (an attention-free transformer that scales linearly with the length of the input sequence) and binary synaptic operations to reduce computational complexity and improve power efficiency. This allows SpikeGPT to reduce the number of synaptic operations by 22 times when implemented on neuromorphic hardware, resulting in power consumption reduced to 1 / 22 of the original amount.

[0040] However, SpikeGPT still has many shortcomings, namely: Performance limitations are caused by (i) the approximation of native backpropagation of ANN during training, and (ii) the use of an attentionless Transformer architecture (i.e., RWKV), which results in a significant performance reduction compared to standard attention mechanisms. Compared to the size limitations of typical large models with billions of parameters; It is not purely based on the same paradigm as SNN because (i) it is more of a quantized version of the large language model (LLM) than an impulsive version, and (ii) the multiply-accumulate (MAC) computation it includes is not native to SNN; This is not biologically plausible.

[0041] In this invention, the following terms are given the following definitions: Generative Artificial Intelligence: Generative AI is a category of deep learning models specifically designed to generate new content across various domains, including text, audio, and images. It's important to note that generative AI models aim to generate data that closely resembles content created by humans.

[0042] Large language models (LLMs): LLMs are a specialized category of generative AI models. They are designed to understand and generate human-like text by leveraging a broad architecture called the Transformer. It's worth noting that ChatGPT exemplifies the application of LLMs. These models excel at processing and interpreting the context and semantics within data.

[0043] Transformer: Transformers are a class of neural network architectures that utilize a unique processing method called "attention" to effectively integrate dependencies in sequential data. Their uniqueness lies in considering the relationships between elements regardless of their positional distance in the input or output sequence. Transformers, with their attention mechanism, have revolutionized various natural language processing tasks.

[0044] Attention Mechanism: Attention is a component of the Transformer network architecture. It manages and quantifies the dependencies between input and output elements (general attention), as well as the dependencies within input elements (self-attention). In deep neural networks, the attention mechanism attempts to selectively focus on a few relevant things while ignoring others that are irrelevant. It prioritizes and emphasizes relevant information, acting as a spotlight to enhance the overall model performance.

[0045] Spiking Neural Networks (SNNs): SNNs represent the third generation of deep learning, designed to mimic brain function while significantly reducing energy consumption. In SNNs, information transmission and signal processing rely on discrete "spiking" signals, represented as binary values ​​(0 or 1). Compared to the continuous-value processing used in ANNs, this spike-based approach drastically reduces power consumption. This energy efficiency stems from several inherent factors in SNNs, including the utilization of sparse spikes, transmission of low-precision data, cost-effective computation, and an event-driven asynchronous processing model. In SNNs, neurons are the basic computational units that mimic biological neurons in the human brain.

[0046] Biological neurons: Biological neurons are the basic building blocks of an organism's nervous system. They operate by processing information through the generation of discrete electrical impulses called pulses or action potentials. These impulses play a crucial role in encoding and efficiently transmitting data, and their timing and frequency are key factors. Biological neurons have several key components and characteristics. One important characteristic is the membrane potential, which indicates the electrical charge on the neuron's cell membrane. Synapses are neuronal components that act as connections between neurons, used for signal transmission. Dendrites are structural branches on neurons that collect and process incoming signals. Dendrites can play an important role in information integration, computation, and decision-making within neural networks.

[0047] Attention mechanisms constitute the core process of Transformer models. They provide adaptive weighting based on the dependencies between elements (lexical units) in sequence data.

[0048] ANNs use a traditional attention mechanism (standard attention), which operates in a non-parametric manner (i.e., without learnable parameters). It consists of three inputs: key (K), value (V), and query (Q). In standard attention, Q and K... T The dot product operation (where T is the transpose operation) is followed by a softmax operation to produce an adaptive weight matrix called the attention map. The attention map indicates the degree of association between each word and other words. The final step involves matrix multiplication between the attention map and V, which produces embedding vectors representing the words from the input sequence and their relationships with other words in the input sequence.

[0049] Specifically, the input sequence consists of words, each of which is converted into an embedding vector in a d-dimensional space for processing by the neural network. Typically, d is quite large. While such high-dimensional spaces are not easily visualized by human thought, they possess mathematical properties similar to the more familiar three-dimensional space. Therefore, such vectors can be considered to be relatively close to each other, or conversely, relatively far apart.

[0050] Before applying the attention mechanism, input words are embedded into a first vector, as are all other words in the input sequence. Then, an attention map is computed to calculate the similarity between the first vector and all other vectors from the input sequence. When the attention map is applied to the first vector, it proportionally moves the first vector towards the other vectors in the input sequence in d-dimensional space based on the similarity between the first vector and the other vectors. In this sense, attention can be thought of as gravity, where closer objects have a greater gravitational pull than distant objects.

[0051] Adapting standard attention mechanisms to SNNs is impractical. SNNs require all data transferred between memory and processing units to be in the form of binary "pulses" (0 or 1). Standard attention, however, uses the dot product of matrices with values ​​between 0 and 1, which produces a matrix with a very limited number of 1s. Therefore, a new data representation is needed for attention mechanisms to match the (pulse) nature of SNNs.

[0052] Ideally, an SNN should exhibit two fundamental characteristics: event-driven processing (computation is triggered only when the input is non-zero) and binary impulse communication (data between processing units must be in binary format).

[0053] Several approaches have emerged to integrate spiking neurons into Transformers. The first type, known as "hybrid computation," involves replacing some neurons in a Transformer with spiking neurons to handle various tasks while retaining the operations required for multiply-accumulate (MAC), such as dot product and softmax. Examples of such works include SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks (arXiv:2302.13939 [cs.CL]) by Zhu et al.; Spikeformer: A Novel Architecture for Training High-Performance Low-Latency Spiking Neural Network (arXiv:2211.10686 [cs.NE]) by Li et al.; Spikformer: When spiking neural network meets transformer (International Conference on Learning Representations, 2023, hereinafter referred to as "Zhou et al.") by Zhou et al.; and Spikingformer: Spike-driven Residual Learning for Transformer-based Spiking Neural Networks (Spikingformer: Spike-driven Residual Learning for Transformer-based Spiking Neural Networks) by Zhou et al. Spiking Neural Network), arXiv:2304.11954 [cs.NE] (hereinafter referred to as "Zhou et al. 2"), all of which are incorporated herein by reference.

[0054] It should be noted that Zhou et al. eliminated the MAC operation by removing the softmax operation. This results in large integers in the output, requiring additional scaling for normalization to mitigate gradient vanishing. Therefore, these methods often fail to fully utilize energy efficiency and may not be suitable for neuromorphic chips.

[0055] To address these issues, Yao et al. published Spike-driven Transformer, arXiv:2307.01694 [cs.NE] (hereinafter referred to as "Yao et al.") (incorporated hereby). Two different approaches were proposed: one utilizes spiking neuron layers to generate a binary attention map for query-key interactions; the other involves key-value interactions followed by spiking neuron layers to generate adaptive weight vectors for the query. The latter approach results in linear attention, which is more efficient in terms of memory and computation, similar to linear attention in ANNs. While both approaches are closer to pure SNN models, they still require additional spiking neuron layers within the attention process. Furthermore, attention computation is not as intuitive as standard attention.

[0056] Another innovative SNN-Transformer approach applies attention mechanisms to the hidden state level of neurons, such as membrane potential, which is a simulated or continuous value rather than the raw binary spiking data. Membrane potential is an intrinsic state of spiking neurons, serving as a suitable variable for attention mechanisms. In their paper, *Attention Spiking Neural Networks*, arXiv:2209.13929 [cs.CV] (hereinafter referred to as "Yao et al. 2") (incorporated hereby), the authors proposed a multi-scale attention mechanism in SNN models, including temporal, channel, and spatial dimensions, used to determine the "time," "object," and "location" to be focused on, respectively. Channel and spatial attention act on the membrane potential of the Leaky Integrate-and-Fire (LIF) neuron model. These intrinsic attentions enhance the processing of relevant information while mitigating the effects of interfering noise. Importantly, even though attention processing at membrane potentials involves MAC operations, it significantly increases the sparsity of neuronal activity (output impulses) by over 80%, even improving accuracy compared to standard SNN attention (attention to impulse data, similar to the method of Zhou et al.). By taking into account both MAC increments and impulse count decrements, attention to the membrane state of neurons produces a greatly enhanced overall energy efficiency.

[0057] Expanding the language model with more data, computing power, and parameters significantly enhances the overall performance of LLM. However, expansion has become increasingly costly and energy-intensive. Since a large portion of the parameters in the transformer architecture reside in dense layers (typically two neuron layers following each attention layer), increasing the number of parameters can be achieved by scaling up the size of these layers.

[0058] A practical approach to scaling transformer size while controlling energy consumption involves model sparsification. This means that for processing a single term, only a portion of the model is activated. This concept aligns with the concept of mixture of experts (MoE), where instead of using a single large dense layer, multiple smaller dense layers (i.e., "experts") are used. A router (typically another small dense layer) decides which expert to activate for each term.

[0059] Shen et al., in *Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models* (arXiv:2305.14705 [cs.CL]), have demonstrated that each expert can learn a unique task during training. This approach allows the model to benefit from an increased parameter count while avoiding excessive computational demands.

[0060] For example, Du et al.'s *GLaM: Efficient Scaling of Language Models with Mixture-of-Experts* (International Conference on Machine Learning, 2022, incorporated herein by reference) increased the number of parameters to 1.2 trillion, but only activated a subnetwork with 96.6 billion (8% of the 1.2 trillion) parameters. This is approximately seven times larger than GPT-3, but consumes only one-third the energy used to train GPT-3, halves the floating-point operations required for inference, and even outperforms GPT-3. This approach has been integrated into newer language models. However, MoE models pose a significant challenge in terms of memory requirements, limiting the number and size of experts that can be incorporated into the model. Increasing the number of experts may improve performance across a range of tasks, but it also requires storing additional parameters during training and inference. It should be noted that the concept of MoE is still unexplored in SNNs. However, it provides a pathway that warrants further research.

[0061] SNNs are known to suffer from problems such as vanishing gradients and dead neurons, especially in deeper networks. These problems arise when neurons fail to receive a sufficient number of spurs at their inputs to produce any output signal. MoE (Modal of Equilibrium) can serve as a potential remedy in such cases because it allows for the dynamic allocation of processing resources to meet the specific needs of different tasks or inputs. The flexibility offered by MoE has the potential to mitigate these problems by enabling neurons to make meaningful contributions to the overall computation of the network, and by horizontally scaling the model's capacity rather than scaling it through deeper layers.

[0062] Therefore, the changes in attention mechanisms and the adoption of dense layer structures in SNNs represent significant steps in the field of generative AI. These developments not only expand the capabilities of SNNs but also lay the foundation for more efficient and powerful neural network models that can excel in complex tasks while conserving computational resources and energy.

[0063] Despite these advances, integrating SNNs with LLMs still presents several unresolved challenges and limitations: Maintaining pulse-based signal transmission: A fundamental challenge is the need to keep all signal transmissions as pulses throughout the network.

[0064] Avoid MAC operations: MAC operations have high computational overhead and high power consumption.

[0065] Maintaining the efficiency and effectiveness of the standard ANN attention mechanism in SNNs is a complex task. Standard ANN attention differs significantly from typical SNN computational models.

[0066] Scalability challenges associated with gradient explosion / vanishing and dead neurons: Creating a scalable SNN model to avoid problems such as gradient vanishing / exploding and dead neurons, especially in deeper networks, remains a major hurdle.

[0067] Among the methods discussed above, Yao et al.2 stand out as an attempt to address some of these challenges. For computational attention, membrane processing appears to have great potential in reducing power consumption, decreasing the number of pulses to be transmitted, mitigating the effects of noise and / or redundant pulses, and combating gradient vanishing / exploding. The authors propose that using skip connections (residual connections) between membrane potentials rather than between pulse data can solve the gradient vanishing / exploding problem common in SNNs. Membrane residual connections may enable networks to scale up to near the size of models with billions of parameters, gaining the benefits of larger model sizes. However, it should be noted that Yao et al.2 introduces a custom-designed attention mechanism that differs significantly from standard ANN attention. Furthermore, their method primarily focuses on self-attention, where the input sequence is transformed into an output sequence, and each output term is the result of an input term and its dependencies on all other terms. The attention process in Yao et al.2 propagates within the membrane potential space but still involves MAC operations. While their models demonstrate an efficiency advantage on GPU processors due to the reduced number of spurs, this suggests they may not be suitable for implementation on neuromorphic chips specifically designed for predefined neuron models, such as LIF neurons (e.g., BrainScaleS, Neurogrid, and TrueNorth). However, neuromorphic chips that allow for neuron model customization may offer potential compatibility (e.g., SpiNNaker and Loihi). These customizable neuromorphic chips operate digitally, whereas analog neuromorphic chips, despite their reputation for energy efficiency, do not offer configurable neuron models. Therefore, to fully realize the potential of SNN-LLM, ideally, neurons should be equipped with (analog) neuromorphic chips that incorporate attention mechanisms.

[0068] Most SNN research focuses on network architecture and learning mechanisms, maintaining predefined neuron models such as LIF neurons. However, recent research has explored the idea of ​​parameterized neuron models, enabling neurons to adapt and specialize based on their unique learning experiences. This approach bears a striking resemblance to the heterogeneous nature of neurons in the brain, where each neuron learns different biological traits through its learning process.

[0069] Parametric neuron models in the prior art typically consider biological features that generally revolve around membrane-related characteristics, including the membrane potential leakage rate that varies over time, the threshold required for the membrane potential to trigger an output pulse, and the method for resetting the membrane potential when the pulse is generated (see Yao et al., Glif: A unified gated leaky integrate-and-fire neuron for spiking neural networks, Advances in Neural Information Processing Systems, 2022, hereinafter referred to as "Yao et al. 3") and Fang et al., Incorporating learnable membrane time constant to enhance learning of spiking neural networks, Proceedings of the IEEE / CVF international conference on computer vision). (vision), 2021, all of which are incorporated herein by reference. Models equipped with learnable parameterized neurons exhibit faster learning capabilities. In Yao et al.3, a LIF-based neuron model was introduced that enables each neuron to balance leakage, reset, and weighted integral between two biological features at each time step, thereby facilitating the implementation of time-dependent pulse coding schemes such as rank-ordered coding or phase-ordered coding.

[0070] However, these advances have overlooked two key biological aspects. First, attention mechanisms—a behavior intuitively consistent with neuronal characteristics—have not yet been integrated or proposed as a dynamic aspect of neuronal models. While the concept of parameterized or gated neuronal models allows neurons to acquire specialized features, attention as a non-parametric process requires the ability to dynamically adapt within neuronal models. Second, all these works, whether in the context of ANNs or SNNs, treat neurons as point-like models, ignoring the complex structure of dendrites. Dendrites are complex, providing many features validated by neuroscience, but many aspects remain undiscovered. Comprehensive neuronal models must include dendritic behavior, thus offering the potential to introduce “intrinsic attention” into models of parameterized neurons.

[0071] This invention provides models that seek to overcome these shortcomings. Specifically, this invention provides models that integrate dendritic dynamics. In some embodiments, dendritic dynamics are used to achieve intrinsic attention within a neuron model.

[0072] This invention draws inspiration from the complex neural structures discovered in biological systems, particularly pyramidal neurons. Combined with... Figure 1A A pyramidal neuron is shown. (Example) Figure 1A As shown, neuron 100 includes a cell body 120. The cell body 120 is the central region of the neuron, where all received signals are integrated and processed. Neuron 100 also includes synapses 110 that transmit signals between neurons, and dendrites 130 that are responsible for collecting and locally processing afferent signals from the synapses.

[0073] These neurons are ubiquitous across a wide range of species and possess a variety of dendrites with unique chemical and morphological characteristics. These dendrites provide neurons with several key advantages, but are missing in common neuron models used in ANNs and SNNs.

[0074] refer to Figure 1B This is a point-like model of neurons. For example... Figure 1B As shown, neuron 142 receives input from synapse 140 and computes a nonlinear function 144 for its output 146. It should be noted that the point model of the neuron lacks any dendrites.

[0075] exist Figure 1C The image illustrates a model of a dendritic neuron, also known as a multicompartment neuron. Figure 1C As shown, neuron 152 also receives input from synapse 150 and computes a nonlinear function 154 for its output 156. However, neuron 152 also includes dendritic dynamics 158, which receives input from dendrite 160, which itself receives input from dendritic synapse 162. Using... Figure 1C The dendritic neuron model in the paper supports the processing of information from various spatial and temporal sources.

[0076] Like traditional point neurons, all synapses of dendritic neurons have plastic connection strength (learnable weights), which allows for different weighting of different inputs, including feedforward data from cell body synapses and contextual data from dendritic synapses.

[0077] Dendritic dynamics introduces an inherent nonlinearity into neural computation, opening up opportunities for more complex data processing. Dendrites can actively and passively influence cell body activity, including the ability to control neuronal hidden states (such as membrane potential) in linear and nonlinear ways. This allows AI models to better capture complex relationships and patterns within input data.

[0078] D-SNN combines SNN with dendritic models for additional information integration and nonlinear processing. Due to the nature of sparse spiking signals and the low-cost accumulation (AC) operation, SNN can significantly reduce power consumption. While adding dendrites to the spiking neuron model increases neuron-level complexity, neural networks with dendritic neurons require fewer parameters for specific desired performance. Furthermore, dendritic models may converge faster and achieve similar or better performance than non-dendritic models. Increasing the computational power of a single neuron improves the model's energy efficiency.

[0079] Compared to traditional synaptic neurons used in models such as SpikeGPT, D-SNN models introduce more complex computations (additional information integration and nonlinearity) to neurons. The nonlinearity of a neuron represents its ability to learn nonlinear patterns within the data. This additional nonlinearity enables neurons to learn complex patterns more skillfully. Therefore, neurons can be modeled as specifically implementing intrinsic attention and the MoE (Modal of Energy) of active dendrites, similar to those in pyramidal neurons. To achieve this, a novel neuron model called leaky integrate modulated firing (LIMF) is proposed and described below.

[0080] SpikeGPT relies on synaptic connections, faces scalability challenges, and includes the potential for dead neurons in deep networks. In contrast, the model based on this invention uses the localization of dendritic computation to facilitate efficient scalability. The intrinsic modulation mechanism of neurons according to this invention ensures that neurons fire when the current feed input is consistent with the context data, thereby reducing the occurrence of dead neurons.

[0081] Contextual data serves as an additional source of information. For example, for attention purposes, context includes key-value pairs, and for MoE, it includes routers that identify the most suitable expert. Furthermore, dendritic models naturally support parallel processing, making them more scalable as model size increases. This scalability remains promising even at scales exceeding 1 billion parameters, overcoming the limitations SpikeGPT faces when scaling to larger models.

[0082] LIF neurons Combination Figure 2 The behavior of LIF neurons is illustrated. Specifically, Figure 2 A graph of voltage versus time is shown. Voltage represents the voltage potential of the neuron membrane. Typically, starting from time T0 and voltage V0, the voltage leaks over time, as shown in section 210. The leakage follows the differential equation shown in Equation 1, where, It is a constant.

[0083]

[0084] For each input pulse passing through the neuron's synapse, the neuron's membrane potential (or hidden state) increases by an amount 220 proportional to the synaptic weight. Between pulses, the voltage (or hidden state) decays again according to Equation 1. If the voltage (or hidden state) reaches a threshold voltage 240, the neuron fires a pulse, as shown in segment 230. In some cases, the threshold voltage 240 can be dynamic.

[0085] After the pulse, the voltage resets to zero until a subsequent pulse is received, at which point the voltage will increase by 220.

[0086] The time 260 when the pulse occurs can be recorded on the pulse sequence 250. The pulse sequence is a time sequence of neuronal firing.

[0087] Spiking Neural Network Now for reference Figure 3 The diagram illustrates a basic SNN 300. The SNN 300 comprises multiple layers 330, each layer 330 consisting of neurons 340. Each neuron in a given layer is connected to all neurons in subsequent layers via synapses 350.

[0088] When input 310 is in the input layer (i.e., Figure 3 When the input is received at the leftmost layer (the layer in the middle), the input is processed at each neuron 340 in the input layer, and the output of each neuron 340 in the input layer is received at each neuron 340 in the next layer via synapses 350. Synapses 350 can be weighted so that some neurons have a greater influence on other neurons than others. Each layer 330 processes data until it reaches the output layer. The output of the output layer is considered as the output 320 of the SNN 300 to the input 310.

[0089] Since SNN 300 is a spiking neural network, all data consists of pulses and is represented on a pulse sequence, as described above. Figure 2 As discussed. Therefore, input 310 includes a pulse sequence, and so does output 320.

[0090] Specifically, the behavior of neuron 340 is typically as follows: Figure 2 As described, when a pulse is received, the hidden state of a neuron increases, and between pulses, the hidden state of the neuron decreases. When the hidden state reaches a threshold, the neuron fires (i.e., sends a pulse to the neuron in the next layer).

[0091] Attention mechanism This invention provides a novel approach to attention mechanisms based on the emerging understanding that biological neurons possess an intrinsic attention mechanism, a concept that has recently emerged through collaborative research involving MIT, IBM, and Harvard Medical School. These studies hypothesize the existence of a biological neural network, comprising neurons and astrocytes (non-neuronal brain cells), capable of performing core computations of a transformer model (i.e., the attention mechanism).

[0092] This invention utilizes the dendritic morphology of pyramidal neurons, based on the hypothesis that astrocytes facilitate rapid data communication, enabling the transfer of contextual data between different regions of the brain. This hypothesis is based on evidence that mainstream data reaches the neuronal cell body synapse and simultaneously extends to the dendritic components. The dendrites use this data to create adaptive weighting (or modulation) for the neuron, thereby effectively achieving attention mechanisms. Furthermore, the dendrites can retrieve contextual information from memory, such as in Shin et al.'s "Memories off the top of your head," Science, October 28, 2021, Vol. 374, No. 6567, pp. 538-539, providing an alternative to mainstream data processing for associative memory concepts.

[0093] Self-attention is a fundamental technique in sequence-to-sequence (seq2seq) tasks, such as machine translation and chatbot applications. It aims to capture the dependencies between terms within a sequence of terms. Unlike traditional recurrent neural networks (RNNs), which create a single context, attention generates a separate context for each input term, as shown in Equation 2. In other words, instead of providing a general context for the entire sequence, the attention mechanism computes a specific context for each term. This context is determined as a weighted combination of all input terms, with weights reflecting the importance of each term. Each context Y... i Calculated as all input tokens x j Weighted sum:

[0094] Among them, y i Let x represent the i-th context word. j w represents all input tokens ij This represents the weight (or attention score) of word wj when calculating yi. The weights capture the dependencies between specific input words i and j and can be calculated using Equation 3 below, which is the expression for the softmax function:

[0095] The attention method described above originated from RNNs, where the input sequence includes hidden states stored in memory.

[0096] In some more recent attention methods, each lexical unit is transformed into a key vector, a query vector, and a value vector through different learned weight matrices. The dot product between the query vector of a given lexical unit and the key vector of another lexical unit produces a value proportional to the importance of the relationship between the given lexical unit and the other lexical unit. This operation on the lexical sequence can be represented as a matrix multiplication QK between the query matrix and the transposed key matrix. T Here, Q is the query matrix, where row i contains the query vector for term i, and K is the key matrix, where row j contains the key vector for term j. For both matrices Q and K, the column size is equal to the embedding size (i.e., the vector size). The resulting matrix is ​​an N×N matrix, where N is the number of terms in the input sequence, and is used as the attention map.

[0097] The resulting attention map is then multiplied by a value matrix V, where each row i contains the value vector of word i, which produces N vectors of the same dimension as the value vectors.

[0098] This method, first published by Vaswani et al. in "Attention is all you need," Advances in Neural Information Processing Systems, 2017 (incorporated hereby cited), can be represented by Equation 4 below, where d is the dimension of the key vector (i.e., the embedding size):

[0099] According to at least some embodiments of the present invention, a neuron model that can intrinsically implement attention is proposed. This neuron model requires multiple inputs, including a single query vector (Q), all key vectors (K), and all value vectors (V).

[0100] To effectively implement the attention mechanism within the dendritic neuron model, Equation 5 can be used:

[0101] Equation 5 calculates the attention vector out of word i. i The sim function is... Various forms can be used, such as exponential or cosine similarity. To handle attention in the hidden states of spiking neurons, Equation 5 can be modified to Equation 6, as shown below:

[0102] To apply this equation to a lexical unit with an embedding size of d, d dendritic neurons can be used. Therefore, It is a vector of size d, where each element is a single value representing the hidden state of one of the neuron's cell bodies. There are N vectors, each with a single word dimension of d, representing the dendritic hidden state of the base dendrites. There are N additional vectors, each with a single word dimension of d, representing the dendritic hidden state of the apical dendrite.

[0103] Now for reference Figure 4 The illustration shows a pyramidal neuron model provided by at least some embodiments of the present invention.

[0104] exist Figure 4 In the diagram, neuron 400 is shown to include cell body synapses 410. For example, each synapse 410 can receive impulses from neurons in the preceding layer. Furthermore, each synapse 410 can be weighted such that the hidden state of neuron 400 is differently affected by impulses from different neurons.

[0105] Neuron 400 also includes basal dendrites 430 and apical dendrites 440, reflecting the structure of a biological pyramidal neuron. The basal dendrites 430 and apical dendrites 440 can receive inputs 432 and 442, respectively, in the form of pulse sequences. Dendritic dynamics 450a and 450b then influence the hidden state of neuron 400, as described in more detail below. According to at least some embodiments of the invention, this influence on the hidden state is used to perform attention within the neuron itself.

[0106] Specifically, the input 432 received by the basal dendrites 430 may include the key K of each word in the input sequence. j The input 442 received by the apical dendrite 440 may include the value V of each word in the input sequence. j For j=1 to N, where N is the number of terms in the input sequence, weighted accordingly by the corresponding synapses. The query Q is received via somatic synapse 410. i The embedded lexical units serve as feedforward signals from the previous layer.

[0107] Since neuron 400 now has query Q i and each key K j Sum V j Therefore, it can use Equation 6 to calculate the attention of the i-th word in the input sequence. Specifically, the output 460 of the nonlinear function 420 of neuron 400 can be interpreted as a representation of the words from the input sequence, with attention information already added. In other words, output 460 carries information about the i-th word, as well as information about the relationship between the i-th word and every other word in the input sequence.

[0108] Now for reference Figure 5 The illustrations depict methods for computational attention provided by at least some embodiments of the present invention.

[0109] The method begins at block 500 and proceeds to block 510, where input tokens are embedded. Input tokens can be received from prompts via a user interface (such as a chatbot), but the present invention is not limited in this respect.

[0110] Embedded input terms refer to elements of an input sequence represented in a format that a neural network can process. For example, in an ANN, terms are embedded into real-valued vectors, while in a SNN, terms are embedded into impulse sequences. Embeddings typically require a trained neural network to produce meaningful values, meaning that similar input terms produce similar real-valued vectors or similar impulse sequences.

[0111] For example, in the case of a chatbot, the input sequence would be a sequence of words, where each word can be considered a lexical. In this case, the embedding values ​​of words like "great" and "fantastic" should be similar because these words have similar meanings and are used in similar contexts.

[0112] For simplicity, we should assume Figure 5 The method is performed entirely on an SNN. However, this is not necessarily the case, and the invention is not limited in this respect. For example, some layers of a neural network may be spiking layers, while others may not. Assuming there is a suitable interface between spiking and non-spiking layers, the invention contemplates any combination of spiking and non-spiking layers, except for attention layers performed on an SNN.

[0113] Once the lexical is embedded, the method proceeds to block 520, where individual lexicals are fed via somatic synapses to create Q. i .

[0114] The method then proceeds to block 530, where each lexical term of the input sequence is fed to the basal and apical synapses to create K synapses from j=1 to N. j and V j .

[0115] Specifically, according to at least some embodiments of the present invention, each dendritic neuron can perform attention for a specific query at a time. Therefore, the query Q of lexical i... i Data is fed through the dendritic neurons via cell body synapses. The K value for each term is fed through the basal dendrites, and the V value for each term is fed through the apical dendrites. At this stage, since the dendritic neurons are spiking neurons, the data is input as a spiking sequence. Therefore, if the Q, K, and V values ​​have not yet been formatted as spiking sequences, they will be converted into spiking sequences before being fed to the dendritic neurons.

[0116] The output of a dendritic neuron is a vector representing word i, to which an attention map has been applied. This can be conceptualized in an N-dimensional space where all words reside, with similar words closer to each other and dissimilar words further apart. By applying attention, words move in this N-dimensional space toward the other words in the sequence that are most relevant to them. Details on how this vector is computed are described below.

[0117] Based on this output term, the method can proceed to block 540, where further processing occurs. For example, the term can then be passed to the decoder, where it will be used to generate an output sequence in response to the input sequence; however, the invention is not limited in this respect.

[0118] The method then proceeds to block 550 and ends.

[0119] Now for reference Figure 6 The illustration shows how dendritic neurons can respond to a given query Q. i Attention is calculated using a series of keys K and a series of values ​​V.

[0120] like Figure 6 As shown, neuron 600 includes cell body hidden state 610 (in Equation 6) ), basal hidden state 620 (in Equation 6) ) and the top hidden state 630 (in Equation 6) These hidden states act as a combination. Figure 2 The described membrane potential plays a role in causing the potential to leak over time. Furthermore, the membrane potential increases when a pulse enters through each of the somatic synapse 611, the basal dendritic synapse 621, and the apical dendritic synapse 631.

[0121] As discussed above, query data can be transmitted via pulses on the cell body synapse 611, bonds can be transmitted via pulses on the basal dendritic synapse 621, and values ​​can be transmitted via pulses on the apical dendritic synapse 631.

[0122] According to at least some embodiments, a given lexical with an embedding vector of dimension d can be associated with d cell body synapses of a neuron.

[0123] In non-spiking neural networks, each term can be represented as a vector with an embedding dimension of d. In contrast, in SNNs, data is represented as binary impulses. However, continuous values ​​can be converted into impulse sequences, also known as multi-time-step windows. As discussed above, impulse sequences consist of time series of impulse occurrences.

[0124] According to at least some embodiments of the invention, there are d neurons, each comprising d somatic synapses associated with a d×d learnable weight matrix for processing a single word, and two sets of dendrites (base and apex), each having N dendrites, where N is the length of the input sequence in terms of words. Each of the N base dendrites and each of the N apex dendrites is associated with the d×d learnable weight matrix. When a signal is received through a synapse, each vector of dimension d (i.e., a single word received through a somatic synapse, N words received through base dendrites, and N words received through apex dendrites) is processed by the corresponding d×d learnable weight matrix to generate a vector of dimension d from somatic hidden state 610, a vector of N dimensions d from base hidden state 620, and a vector of N dimensions d from apex hidden state 630.

[0125] Therefore, each of the d cell body synapses 611 influences the cell body hidden state 610. The hidden state 610 is also a vector of dimension d.

[0126] The basal hidden state 620 connects to N dendrites, where each dendrite includes d synapses. Therefore, the basal hidden state 620 can be considered as N A matrix d, where each row is a vector of dimension d. For the same reason, the top hidden state 630 is also N. d matrix.

[0127] At block 650, a similarity function (denoted as "SIM") is performed on each vector of the cell body hidden state 610 and the basal hidden state 620. The cell body hidden state 610 encodes the query, and the basal hidden state 620 encodes all keys. The similarity function is performed separately on the query and each key vector to produce N scalar values.

[0128] At block 670, N vectors V are received from the top tree synapse 631. j Each of these can be multiplied by N scalar values. Specifically, for j = 1 to N, V j Multiply by s j , where s j From the query and key K j The scalar obtained by executing the similarity function. This produces N vectors of dimension d, which can be combined at block 680 to produce a single vector of dimension d.

[0129] At block 660, N scalar values ​​s j Add them together. Then, divide that sum by the vector of dimension d produced at block 680 to produce a normalized vector of dimension d at block 690.

[0130] At block 640, the normalized vector of dimension d can then be output to the next layer. Specifically, the normalized vector of dimension d is converted into a pulse sequence and then transmitted through d output synapses.

[0131] therefore, Figure 6 Neuron 600 can implement Equation 6 above, simultaneously receiving data as pulses and outputting data as pulses. The output of neuron 600 can be interpreted as words (Q) from the sequence. i The vector representation of ), which has been applied with attention.

[0132] For an input sequence of length N, the attention layer can include N groups of neurons, each group being a combination of d neurons, for example... Figure 6 The neuron 600 described in the text. In other words, it contains N... A neural network with d neurons in its attention layer can accommodate an input sequence of up to N words.

[0133] According to at least some embodiments of the present invention, neurons in an SNN can achieve linear attention through dendritic dynamics, thereby further optimizing the efficiency and adaptability of the SNN in language modeling tasks.

[0134] Attention mechanisms are powerful tools for modeling dependencies within sequences. However, their implementation often comes at the cost of high computational cost and involves quadratic memory requirements, posing scalability and efficiency challenges in both ANNs and SNNs. To address this, extensive research in the ANN field has focused on implementing attention mechanisms using methods with sub-quadratic complexity. One such method is linear attention, which attempts to decompose the dot product between the query Q and the key K vectors, as illustrated in Transformers are RNNs: Fast autoregressive transformers with linear attention (International conference on machine learning, 2020, incorporated herein by reference).

[0135] In this embodiment, the concept of linear attention is used to reduce the computational requirements of the attention mechanism.

[0136] Extend Equation 6 and employ the kernel trick, which is used in Equation 6 and in Figure 6 The nonlinear similarity function illustrated in the diagram can be replaced by some linear functions. In this case, the kernel denoted as k(a,b) can be represented as the low-level feature representation. A nonnegative function of , such that This effectively transforms the nonlinear kernel into a linear inner product operation. Functions encapsulate the essence of the kernel and can take various forms; for example, those proposed by Katharopoulos et al. (x) = elu(x) + 1, where elu is an exponential linear unit function. Based on this, equation 6 can be rewritten as equation 7:

[0137] In the above text, and The interaction between them is decoupled so that from and This separation allows for the creation of neuronal models where the processes of the cell body and dendrites are no longer interdependent, as will be described in detail below.

[0138] The dendritic components of a process applied to the entire sequence can be executed once, and then the process modulates the cell body portion of each neuron. This can be considered as shared dendrites, a phenomenon also present in biological neurons.

[0139] Implementing the neuron model in Equation 7 Figure 7 The image is shown in the middle.

[0140] Specifically, such as Figure 7 As shown, the neuron model 700 includes a cell body hidden state 710, a basal hidden state 720, and a apical hidden state 730. These hidden states serve as a basis for... Figure 2 The membrane potential described plays a role, causing the potential to leak over time.

[0141] As discussed above, query data can be transmitted via pulses on the cell body synapse 711, bonds can be transmitted via pulses on the basal dendritic synapse 721, and values ​​can be transmitted via pulses on the apical dendritic synapse 731.

[0142] At block 750, the query can be performed. Transformation. Specifically, Transform a vector v of dimension d into a vector of dimension d. (v). In at least some embodiments, (x) = elu(x) +1, where elu is the exponential linear unit, although the definition is not limited to this particular form.

[0143] Similarly, at block 752, each key All passed Perform the transformation. The key to the transformation. ( They can be added together at block 753. At block 751, the transformed key... ( The sum (e.g., using dot product) and the query of the transformation. ( Multiply by , to produce the first scalar value.

[0144] exist Figure 7 In the example, at block 754, the key of each transformation ( ) and value Multiply (e.g., using a dot product) to produce N scalar values, which are then added together at block 755.

[0145] The sum of these N scalar values ​​can then be used to multiply the transformed query at block 756. ( The scalar value from block 751 is used to divide the resulting vector at block 757. The result from block 757 is then output at block 740.

[0146] Therefore, linear attention mechanisms can be used in situations such as Figure 7 Implemented on the neuron model shown.

[0147] As discussed above, Figure 7 The neuron model can also be modified to provide shared dendrites, thereby avoiding duplication of the same operations and saving processing time and resources. Implementing the shared dendrite model of Equation 7 in... Figure 8 and Figure 9 It is described in the text.

[0148] Figure 8 The image shows a shared dendrite 800. The shared dendrite 800 includes components from... Figure 7 All dendritic elements of the neuron model.

[0149] Specifically, the dendrite 800 includes a basal hidden state 820 and a apical hidden state 830. These hidden states act as a combination... Figure 2 The membrane potential described plays a role, causing the potential to leak over time.

[0150] As discussed above, bonds can be transmitted via pulses on the basal tree synapses 821, and values ​​can be transmitted via pulses on the apical tree synapses 831. For example, the basal tree synapse 821 can receive N 1×d vectors, where each 1×d vector corresponds to a bond. Similarly, the apical tree synapse 831 can receive N 1×d vectors, where each 1×d vector corresponds to a value.

[0151] Each of the N basal dendrites and each of the N apical dendrites is associated with a d×d learnable weight matrix. The received input is transformed through the learnable weights in the d×d matrix, such that the received 1×d vector is multiplied by its corresponding d×d weight matrix, thereby generating a 1×d vector for each of the N 1×d vectors.

[0152] At block 852, there are N keys. Each of the passages The transformation produces an N×d matrix (a vector with N dimensions and d). Then the key of the transformation... ( They can be added together at block 853. Then, a vector of dimension d can be output at 856, representing the transformation key φ. The sum of ) is used by neurons.

[0153] exist Figure 8 In the example, at block 854, for j=1 to N, then the key of each transformation is... ( Multiply by (e.g., using a dot product) the corresponding vector This produces N scalar values, which can be summed at block 855 to produce a scalar value at block 857 for the neuron to use as an output.

[0154] Specifically, multiple neurons can connect to dendrite 800 to receive outputs 856 and 857. Because the values ​​of outputs 856 and 857 can be calculated only once for multiple neurons, greater efficiency is achieved.

[0155] Figure 9 The image shows one such neuron. Specifically, Figure 9 Neurons include those from Figure 7 The cell body elements of neurons, and can also be obtained from, for example Figure 8 Dendrite 800 and other dendrite receivers receive input.

[0156] like Figure 9 As shown, neuron 900 includes a cell body hidden state 910. This hidden state serves as a binding... Figure 2 The membrane potential described plays a role, causing the potential to leak over time.

[0157] As discussed above, query data can be transmitted via pulses at cell somatic synapse 911. Specifically, N queries Q i Each query in (i=1 to N) can be transmitted to its own neuron 900. As combined with the above... Figure 6 The discussed approach uses a d×d learnable weight matrix to receive queries. Contextual information from the entire input sequence, such as... Figure 8Shared dendrites, such as dendrite 800, are provided. Specifically, context information may include outputs 856 and 857 from shared dendrite 800.

[0158] At block 950, query The hidden state can be achieved through Transform to produce ( At block 951, the key of the transformation. ( ) is received via input 953 from shared dendrites and with the query of the transformation. ( Multiply them (e.g., using a dot product) to produce the first scalar value.

[0159] At block 956, the key AND value The sum of the N scalar values ​​obtained by multiplication is received via input 955 from the shared dendrites and can be used for multiplication by a transform query. ( ).exist Figure 9 In the example, the first scalar value from block 951 is then used to divide the resulting vector at block 957. The result of block 957 (i.e., a vector of dimension d) is then output at block 940.

[0160] Figure 9 The diagram illustrates the LIMF neuron model, an extension of the LIF neuron model, which includes two additional modulation inputs, denoted as 953 and 955.

[0161] Therefore, linear attention mechanisms can be implemented in both neuron models and shared dendrite models, such as... Figure 8 and Figure 9 As shown.

[0162] Figure 10 The diagram shows... Figure 9 The neurons shown are shared by neuron layers, such as... Figure 8 Dendrites, such as those shown in the image.

[0163] Specifically, such as Figure 10 As shown, a spiking neural network 1000 is illustrated. The SNN 1000 includes shared dendrites 1010 and N neurons 1 to N, 1040. The shared dendrites 1010 receive contextual information in the form of key and value for each term in the input sequence and perform the operations discussed above to produce outputs 1020 and 1030. These outputs correspond to... Figure 8 The outputs are 856 and 857.

[0164] Outputs 1020 and 1030 are connected to each neuron 1040, ensuring that the computation required to produce these output values ​​is performed only once. Each neuron 1040 receives a corresponding query Q via its membrane potential. i Then the neuron can perform binding. Figure 9 The computation described is used to produce their respective outputs, which can be interpreted as the representation of the i-th lexical in a sequence to which linear attention has been applied.

[0165] Mixture of experts (MoE) According to at least some embodiments of the present invention, a dendritic neuron model is used to implement the MoE architecture. The MoE architecture in a neural network typically includes a routing layer, the purpose of which is to select an expert subnetwork to perform a specific task.

[0166] refer to Figure 11 The diagram illustrates a traditional MoE architecture using neural networks. Figure 11 As shown, the neural network 1100 includes a routing layer 1120 and multiple expert sublayers 1130a, 1130b, 1130c, and 1130d. Although Figure 11 Only four expert sub-layers are included, but this is not intended to limit and is shown for illustrative purposes only.

[0167] When input term 1110 is received by routing layer 1120, the routing layer selects one of the expert sublayers to process the input term. Specifically, the routing layer has been trained to select a suitable expert based on training data, such that when input term 1110 is received, routing layer 1120 selects the most suitable expert to process input term 1110. Figure 11 In the example, expert sublayer 1130c is selected, so expert sublayer 1130c processes the input word 1110 to produce the output word 1140.

[0168] Now for reference Figure 12 The illustration shows the MoE architecture in an SNN according to at least some embodiments of the present invention.

[0169] like Figure 12 As shown, the SNN includes: a first layer 1206, which includes F LIMF neurons 1208 sharing a dendritic input 1203; and a second layer 1207, which includes d LIMF neurons 1209 that also share a dendritic input 1203.

[0170] Dendritic input 1203 may include dendrites with d synapses for receiving contextual data. For example, the contextual data may include a lexical of dimension d, where d is the embedding dimension.

[0171] According to at least one embodiment, the dendritic input 1203 consists of two parts, which can be combined when both parts use the same input data, i.e., words representing the dimension d of the context data. In the first part, the context data is processed by M d×F learnable weight matrices, where M is the number of experts in the MoE architecture and F is the number of neurons in the first layer 1206. The context data can be processed by each of the M d×F matrices to produce a vector of dimension F. One of the vectors of dimension F is then selected by a selection function. In one embodiment, the selection function can be a k-Winners-Take-All (kWTA) function to select k experts, or k=1 to select one expert. Optionally, the selection function can select the vector with the maximum amplitude (i.e., the maximum function). The selected vector is then output from the dendrite 1203 and provided as input 1204 to the first layer 1206 as context data. The processing of the context data by the first layer 1206 is described below. Figure 13 describe.

[0172] In the second part of the dendritic input 1203, the context data is processed by M d×d learnable weight matrices, where M is the number of experts in the MoE architecture, to produce M d-dimensional vectors. One of these M d-dimensional vectors is then selected by a selection function such as kWTA or a maximum function and output from dendritic 1203 as input 1205 to the second layer 1207 as context data. The processing of this context data by the second layer 1207 is described below. Figure 13 describe.

[0173] Neurons 1208 from the first layer 1206 further receive input terms 1201 via their respective cell body synapses. The neurons 1208 of the first layer 1206 each output a single impulse event, which is provided to each neuron 1209 of the second layer 1207, thus providing an F-dimensional vector as input to each neuron 1209. Similarly, each neuron 1209 of the second layer 1207 outputs 1210, thus providing a d-dimensional vector as the output of the MoE architecture.

[0174] Now for reference Figure 13 It describes in detail Figure 12 The processing of the first and second layers.

[0175] Corresponding to Figure 12 The input words of input 1201 can be provided through the cell body synapses 1311 of the first layer 1301. The input words have dimension d and are received through the hidden state 1310. A d×F learnable weight matrix can be applied to the input words to create an F-dimensional vector α.

[0176] Context input 1320 received from, for example, from Figure 12 The input to dendrite 1203 is the F-dimensional vector β selected by dendrite selection, such as 1204. Specifically, since there are F neurons in the first layer, each element of the F-dimensional vector can be provided to a single neuron in the first layer.

[0177] The context input 1321 can be provided as a vector where all elements are equal to 1. This allows you to essentially disable a feature of the neuron model and make it suitable for MoE applications.

[0178] The F-dimensional vector α received from the cell somatic synapse 1311 can then be processed at block 1330 to compute... (α). At block 1331, (α) Multiply by β to produce the scalar value γ at block 1333. (α) Multiply the vector by 1 and keep it unchanged. Then, at block 1332, (α) Divide by γ to produce an F-dimensional vector, which is output at 1340 by the first layer 1301.

[0179] The process then continues into the second layer 1302, where the output 1340 is fed into the hidden state 1350 via the cell somatic synapse 1341 and through an F×d matrix of learnable weights to produce a d-dimensional vector α2. The context input 1342 receives data from... Figure 12 The input of dendrite 1203 is the d-dimensional vector β2 of dendrite selection, such as 1205.

[0180] The d-dimensional vector α2 received from cell somatic synapse 1341 can then be processed at block 1360 to compute... (α2). At block 1361, (α2) multiplied by β2 produces the scalar value γ2, and at block 1363, (α2) is multiplied by an all-1 vector and remains unchanged. Then, at block 1362, (α2) is divided by γ to produce a d-dimensional vector, which is output at 1370 by the second layer 1302. Output 1370 represents the output of the MoE architecture.

[0181] Therefore, based on the above, the Mixture of Experts (MoE) architecture can be implemented on SNN.

[0182] The above functions can be implemented on any combination of one or more computing devices. Figure 14This is a block diagram of a computing device 1400 that can be used to implement the devices and methods disclosed herein. A particular device may utilize all or only a subset of the components shown, and the level of integration may vary from device to device. Furthermore, the device may contain multiple instances of components, such as multiple processing units, processors, memory, transmitters, receivers, etc. The computing device 1400 may include a central processing unit (CPU) 1410, memory 1420, mass storage device 1440, and peripherals 1430. Peripherals 1430 may include one or more input / output devices, such as speakers, microphones, mice, touchscreens, keypads, keyboards, printers, displays, network interfaces, etc. Communication between the CPU 1410, memory 1420, mass storage device 1440, and peripherals 1430 may occur via one or more buses 1450.

[0183] Bus 1450 can be any type or one of several bus architectures, including memory bus or memory controller, peripheral bus, video bus, etc. CPU 1410 can include any type of electronic data processor. Memory 1420 can include any type of system memory, such as static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), combinations thereof, etc. In one embodiment, memory 1420 can include ROM for booting and DRAM for storing programs and data during program execution.

[0184] Mass storage device 1440 may include any type of storage device for storing data, programs, and other information, and enabling access to this data, programs, and other information via a bus. For example, mass storage device 1440 may include one or more of solid-state drives, hard disk drives, disk drives, optical disk drives, etc.

[0185] The computing device 1400 may also include one or more network interfaces (not shown), which may include wired links such as Ethernet cables and / or wireless links for accessing nodes or different networks. The network interface enables the processing unit to communicate with remote units over a network. For example, the network interface may provide wireless communication via one or more transmitter / transmit antennas and one or more receiver / receive antennas. In one embodiment, the processing unit is coupled to a local area network (LAN) or wide area network (WAN) for data processing and communication with remote devices such as other processing units, the Internet, remote storage facilities, etc.

[0186] Based on the foregoing description of the embodiments, the guidance of the present invention can be implemented solely by using hardware or by using a combination of software and hardware. Software or other computer-executable instructions for implementing one or more embodiments or one or more portions thereof can be stored on any suitable computer-readable storage medium. The computer-readable storage medium can be a tangible or transient / non-transient medium, such as optical media (e.g., CDs, DVDs, Blu-ray discs, etc.), magnetic media, hard disks, volatile or non-volatile media, solid-state media, or any other type of storage medium known in the art.

[0187] Additional features and advantages of the present invention will be understood by those skilled in the art.

[0188] The structures, features, accessories, and alternatives of the specific embodiments described herein and illustrated in the accompanying drawings are intended to be generally applicable to all teachings of the invention, including all embodiments described and illustrated herein, provided they are compatible. In other words, the structures, features, accessories, and alternatives of the specific embodiments are not intended to be limited to that particular embodiment unless so indicated.

[0189] Furthermore, the foregoing detailed description is provided to enable any person skilled in the art to make or use one or more embodiments of the invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the spirit or scope of the guidance provided herein. Therefore, the methods, systems, and / or devices are not intended to be limited to the embodiments disclosed herein. The scope of the claims should not be limited by these embodiments, but should be given the broadest interpretation consistent with the overall description. Reference to an element in the singular, such as the article “a” or “an,” unless specifically stated otherwise, does not mean “one and only one,” but rather “one or more.” All structural and functional equivalents of the elements of the various embodiments described in this invention, whether known to or subsequently known to those skilled in the art, are intended to be covered by the elements of the claims.

[0190] Furthermore, nothing herein is intended to acknowledge prior art or general common sense. Moreover, any reference to or identification of any document in this application does not imply that such document is available as prior art, or that any reference constitutes part of general common sense in the art. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is expressly stated in the claims.

[0191] In the foregoing description, numerous details have been set forth to provide an understanding of the subject matter disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and variations to the details discussed above. The appended claims are intended to cover such modifications and variations.

[0192] Specifically, exemplary terms may include: A. A method for implementing a spiking neural network, comprising creating at least one layer of spiking neurons, wherein each spiking neuron comprises: a cell body input of dimension d; at least one dendritic input of dimension d; an output of dimension d; providing input terms through the cell body input; providing context data through the at least one dendritic input; wherein the output of each spiking neuron is modulated by the context data.

[0193] B. The method according to Clause A, wherein the context data causes the spiking neuron to perform a first task on context data of a first category, and the context data causes the spiking neuron to perform a second task on context data of a second category.

[0194] C. The method according to Clause B, wherein the context data includes the input sequence.

[0195] D. The method according to Clause B, wherein the context data includes the category of the input sequence.

[0196] E. The method according to Clause D further includes preprocessing the input sequence using a preprocessing layer to compute the category of the input sequence.

[0197] F. The method according to Clause B, wherein the at least one dendritic input is a shared dendrite, and the output of the shared dendrite is received by each spiking neuron.

[0198] G. A computing device for implementing a spiking neural network, comprising a processor; and a memory; wherein the processor and the memory are configured to: create at least one layer of spiking neurons, wherein each spiking neuron comprises: a cell body input of dimension d; at least one dendritic input of dimension d; an output of dimension d; provide input terms through the cell body input; and provide context data through the at least one dendritic input; wherein the output of each spiking neuron in the spiking neurons is modulated by the context data.

[0199] H. The computing device according to Clause G, wherein the context data causes the spiking neuron to perform a first task of a first category of context data, and the context data causes the spiking neuron to perform a second task of a second category of context data.

[0200] I. The computing device as described in Clause H, wherein the context data includes an input sequence.

[0201] J. The computing device as described in Clause H, wherein the context data includes the category of the input sequence.

[0202] K. The computing device according to Clause J, wherein the processor and the memory are further configured to preprocess the input sequence using a preprocessing layer to compute the category of the input sequence.

[0203] L. The computing device according to Clause H, wherein the at least one dendritic input is a shared dendrite, and the output of the shared dendrite is received by each of the peak neurons.

[0204] M. A computer-readable medium having stored thereon executable code for execution by a processor of a computing device, the executable code including instructions for: creating at least one layer of spiking neurons, wherein each spiking neuron includes: a cell body input of dimension d; at least one dendritic input of dimension d; an output of dimension d; providing input terms through the cell body input; providing context data through the at least one dendritic input; wherein the output of each spiking neuron is modulated by the context data.

[0205] N. A computer-readable medium according to Clause M, wherein the context data causes the spiking neuron to perform a first task of a first category of context data, and the context data causes the spiking neuron to perform a second task of a second category of context data.

[0206] O. A computer-readable medium pursuant to Clause N, wherein the context data includes an input sequence.

[0207] P. A computer-readable medium pursuant to Clause N, wherein the context data includes the category of the input sequence.

[0208] Q. The computer-readable medium according to Clause P, wherein the executable code further includes instructions for preprocessing the input sequence using a pre-layer to calculate the category of the input sequence.

[0209] R. A computer-readable medium according to Clause N, wherein the at least one dendritic input is a shared dendrite, and the output of the shared dendrite is received by each of the peak neurons.

Claims

1. A method for implementing a spiking neural network (SNN), characterized in that, include: Create at least one layer of spiking neurons, wherein each spiking neuron comprises: Cell body input with dimension d; At least one dendritic input of dimension d; Output of dimension d; Input terms are provided through the cell body input; Contextual data is provided through the at least one dendritic input; The output of each of the spiking neurons is modulated by the context data.

2. The method according to claim 1, characterized in that, The at least one dendritic input includes N basal dendritic inputs and N apical dendritic inputs, and the maximum input sequence length is N.

3. The method according to claim 2, characterized in that, The context data includes the input sequence.

4. The method according to claim 3, characterized in that, The input terms include queries for selected terms of the input sequence.

5. The method according to claim 4, characterized in that, The N basal dendritic inputs are used to receive N keys, and the N apical dendritic inputs are used to receive N values, wherein each of the N keys and each of the N values ​​is derived from N terms in the input sequence.

6. The method according to claim 5, characterized in that, The N basal dendritic inputs and the N apical dendritic inputs are shared dendrites, and the output of the shared dendrites is received by each of the spiking neurons.

7. The method according to claim 5 or 6, characterized in that, The output of each of the spiking neurons is determined by the following: Among them, out i It is the output of neuron i, Q i It is the selected query, K j and V j Let represent the N keys and the N values, and sim be the similarity function.

8. The method according to claim 7, characterized in that, The similarity function is one of the exponential function and the cosine similarity function.

9. The method according to claim 6, characterized in that, The output of the shared dendrites includes a scalar determined by the following: And the vector determined by the following: Among them, K j and V j This represents the N keys and the N values, and It is a non-negative function representing the underlying features.

10. The method according to claim 9, characterized in that, (x) = elu(x) + 1, where elu is an exponential linear unit function.

11. The method according to claim 10, characterized in that, The output of each of the spiking neurons is determined by the following: Where S is the scalar and V is the vector.

12. A computing device, characterized in that, include: processor; Memory; Wherein, the processor and the memory are used for Create at least one layer of spiking neurons, each of which comprises: Cell body input with dimension d; At least one dendritic input of dimension d; Output of dimension d; Input terms are provided through the cell body input; Contextual data is provided through the at least one dendritic input; The output of each of the spiking neurons is modulated by the context data.

13. The computing device according to claim 12, characterized in that, The at least one dendritic input includes N basal dendritic inputs and N apical dendritic inputs, and the maximum input sequence length is N.

14. The computing device according to claim 13, characterized in that, The context data includes the input sequence.

15. The computing device according to claim 14, characterized in that, The input terms include queries for selected terms of the input sequence.

16. The computing device according to claim 15, characterized in that, The N basal dendritic inputs are used to receive N keys, and the N apical dendritic inputs are used to receive N values, wherein each of the N keys and each of the N values ​​is derived from N terms in the input sequence.

17. The computing device according to claim 16, characterized in that, The N basal dendritic inputs and the N apical dendritic inputs are shared dendrites, and the output of the shared dendrites is received by each of the spiking neurons.

18. The computing device according to claim 17, characterized in that, The output of the shared dendrite includes a scalar determined by the following: and the vector determined by the following: Among them, K j and V j This represents the N keys and the N values, and It is a non-negative function representing the underlying features, where elu is an exponential linear unit function.

19. The computing device according to claim 17 or 18, characterized in that, The output of each of the spiking neurons is determined by the following: Where S is the scalar and V is the vector.

20. A computer-readable medium, characterized in that, It stores executable code for execution by the processor of a computing device, the executable code including instructions for the following operations: Create at least one layer of spiking neurons, wherein each spiking neuron comprises: Cell body input with dimension d; At least one dendritic input of dimension d; Output of dimension d; Input terms are provided through the cell body input; Contextual data is provided through the at least one dendritic input; The output of each of the spiking neurons is modulated by the context data.

Images