Agent orchestration of multiple expert chips implementing models-on-silicon architecture

Abstract

An agent chip in a multi-chip architecture orchestrates multiple specialized AI models embedded and / or etched on different chips. Implementing the agent chip effectively solves the problem of deploying multiple specialized AI models in a cost-effective and scalable manner by training and utilizing the agent chip to orchestrate multiple specialized AI models embedded on different models-on-silicon chips. Each models-on-silicon chip is optimized for a specific task or goal, and the agent chip coordinates and / or routes their activities to perform complex, multi-faceted tasks efficiently. Accordingly, the multi-chip architecture allows for efficient, scalable, and cost-effective machine learning inference, significantly reducing power consumption and latency.

Description

AGENT ORCHESTRATION OF MULTIPLE EXPERT CHIPS IMPLEMENTING MODELS-ON-SILICON ARCHITECTURECross-reference to Related Application(s)

[0001] This application claims priority to and / or receives benefit from US Non-Provisional Application No. 19 / 275,640, filed on 21 July 2025 and titled ‘AGENT ORCHESTRATION OF MULTIPLE EXPERT CHIPS IMPLEMENTING MODELS-ON-SILICON ARCHITECTURE” and US Provisional Application No. 63 / 681 ,692, filed on 9 August 2024 and titled “AGENT LARGE LANGUAGE MODELS CHIP WITH MODEL ON SILICON ARCHITECTURE”. The US Non-Provisional and the US Provisional application are hereby incorporated by reference in their entirety.Background

[0002] Deep neural networks (DNNs) including large language models (LLMs) are used extensively for a variety of artificial intelligence applications ranging from computer vision to speech recognition and natural language processing due to their ability to achieve high accuracy. However, the high accuracy comes at the expense of significant computation cost. DNNs have high computing demands especially when being scaled across multiple machine learning models as there can be a large number of operations as well as a large amount of data to read and write.Brief Description of the Drawings

[0003] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0004] FIG. 1 illustrates an exemplary chip architecture, according to some embodiments of the disclosure.

[0005] FIG. 2 illustrates exemplary details within the parts of the exemplary chip architecture, according to some embodiments of the disclosure.

[0006] FIG. 3 illustrates embedding an exemplary open-source model onto the chip, according to some embodiments of the disclosure.

[0007] FIG. 4 illustrates exemplary hardware blocks representing an exemplary open-source model, according to some embodiments of the disclosure.DOCKET NO.: AG2834-US-PCT

[0008] FIG. 5 illustrates a sequential read-only memory, according to some embodiments of the disclosure.

[0009] FIG. 6 illustrates a sequential read / write memory in an attention multiplier circuit, according to some embodiments of the disclosure.

[0010] FIG. 7A illustrates an exponent unit circuit, according to some embodiments of the disclosure.

[0011] FIG. 7B illustrates an exponent function, according to some embodiments of the disclosure.

[0012] FIG. 8A illustrates a sigmoid linear unit (SILU) activator circuit, according to some embodiments of the disclosure.

[0013] FIG. 8B illustrates a sigmoid linear unit function and a rectified linear unit (RELU) function, according to some embodiments of the disclosure.

[0014] FIG. 9 illustrates a weights multiplier circuit, according to some embodiments of the disclosure.

[0015] FIG. 10 illustrates an embedding dot unit circuit, according to some embodiments of the disclosure.

[0016] FIG. 11 illustrates bit cell area optimization, according to some embodiments of the disclosure.

[0017] FIG. 12 illustrates a weights multiplier circuit, according to some embodiments of the disclosure.

[0018] FIG. 13 illustrates a SoftMax circuit, according to some embodiments of the disclosure.

[0019] FIG. 14 illustrates an embedder circuit, according to some embodiments of the disclosure.

[0020] FIG. 15 illustrates a root mean square (RMS) normalizer circuit, according to some embodiments of the disclosure.

[0021] FIG. 16 illustrates a sampler circuit, according to some embodiments of the disclosure.

[0022] FIG. 17 illustrates a sampling comparator circuit, according to some embodiments of the disclosure.

[0023] FIG. 18A illustrates a rotary positional encoding circuit, according to some embodiments of the disclosure.

[0024] FIG. 18B illustrates a cosine function and a sine function, according to some embodiments of the disclosure.

[0025] FIG. 19A illustrates using multiple chips to implement a large transformer model, according to some embodiments of the disclosure.DOCKET NO.: AG2834-US-PCT

[0026] FIG. 19B illustrates using multiple chips to implement a large transformer model, according to some embodiments of the disclosure.

[0027] FIG. 20 illustrates hardware-based inferencing process with embedded large language model (LLM) and read-only memory (ROM), according to some embodiments of the disclosure.

[0028] FIG. 21 illustrates a matrix multiplication operation, according to some embodiments of the disclosure.

[0029] FIG. 22 illustrates an embedded weights fused multiply-add architecture, according to some embodiments of the disclosure.

[0030] FIG. 23A illustrates an agent chip orchestrating multiple expert chips, according to some embodiments of the disclosure.

[0031] FIG. 23B illustrates an agent chip orchestrating multiple expert chips, according to some embodiments of the disclosure.

[0032] FIG. 24 illustrates an implementation of an agent chip having a router neural network model, according to some embodiments of the disclosure.

[0033] FIG. 25 illustrates a system having expert chips and an agent chip orchestrating machine learning tasks being performed by the expert chips in a software development scenario, according to some embodiments of the disclosure.

[0034] FIG. 26 illustrates a data flow of the system illustrated in FIG. 25, according to some embodiments of the disclosure.

[0035] FIG. 27 illustrates a method for orchestrating expert chips, according to some embodiments of the disclosure.

[0036] FIG. 28 is a block diagram of an exemplary computing device, according to some embodiments of the disclosure.Detailed DescriptionTechnical problem

[0037] The problem being solved is the need for a cost-effective, dedicated solution for Al inference tasks. Huge Al models are capable of addressing any small-scale need (for example, audio to text, robotics, or the like). These huge models are expensive in power and performance and are therefore limited in terms of implementation. For example, a humanoid system may use a huge battery to perform simple tasks, and real-time response time can be difficult or close to impossible to achieve. Such systems may also require Internet connectivity to a cloud computing environment that implements the huge model and thus cannot autonomously execute in an isolated environment. Huge Al models have been implemented in software, but a software solution can be inefficient in terms of performanceDOCKET NO.: AG2834-US-PCTand energy (e.g., per token). Software solutions can be sufficient for conducting time-insensitive calculations, but not for applications that may demand real-time performance.

[0038] An example of a model that can carry out an inferencing task is a transformer-based neural network. An example of a transformer-based neural network that is used often is the LLM, which can be used to understand, generate, and manipulate human language. Some transformer-based neural network can operate on one or more modalities (e.g., audio, text, images, video, signals, etc.). Transformer-based neural networks are a type of deep learning model that can handle sequential data. Transformer-based neural networks can employ self-attention to weight the importance of different words in a sentence, or different tokens in a sequence of tokens, to capture context and relationships. Transformer-based neural networks can have millions to billions of trainable weights to capture the context and relationships. It is not trivial to implement these transformer-based neural networks on hardware, due to the extreme amounts of processing and the amount of weights involved in the processing.

[0039] While general-purpose solutions like Graphics Processing Units (GPUs), Tensor Processing Units (TPUs), and Central Processing Units (CPUs) can be utilized for both training and inference, they are not cost-effective for inference on a given model alone due to their inherent design to handle a wide range of tasks, including the repetitive loading of the LLM including its weights.

[0040] In a GPU-based solution, model weights are loaded from memory every time a machine learning inference task is performed. This process consumes significant power and time, particularly for complex models. GPUs are designed in a generic manner to handle a wide range of tasks, making them inefficient for dedicated tasks like inference on a pre-trained model alone.

[0041] In a field programmable gate array (FPGA) based solution, programmable hardware can be customized to perform specific tasks, including loading and handling LLM weights, to make machine learning inference more efficient. While FPGAs offer flexibility, they can require significant programming effort and expertise to be utilized effectively. They also have lower performance compared to dedicated hardware solutions and are not as power-efficient and not cost-effective.

[0042] In CPU-based solutions, CPUs can be programmed to perform machine learning inference tasks. CPUs are not suitable for large-scale matrix multiplications which can be essential for machine learning inference tasks. They also consume more power and are slower in comparison to dedicated solutions.

[0043] In the inferencing process with GPU acceleration, the user initiates the sequence by providing input data for analysis. This data undergoes tokenization and embedding generation, transforming it into a format suitable for machine learning models. The system then loads the pretrained model into memory, along with its associated weights, which are the learned parameters crucialDOCKET NO.: AG2834-US-PCTfor making predictions. Once the GPU is initialized, the model weights and embeddings are transferred to the High Bandwidth Memory (HBM), a specialized memory architecture designed for high-speed data transfer. The data is then shuttled from the HBM to the GPU cores, where the actual inferencing computations take place in parallel. After processing, the data is moved back to the HBM. A significant challenge in this workflow is the data transfer between the HBM and the GPU cores. While HBM offers high bandwidth, the repeated movement of data can create a bottleneck, leading to latency issues that can diminish the overall performance gains from GPU acceleration. Each transfer incurs a cost in time and energy, and when dealing with large datasets or complex models, these costs can accumulate, impacting the efficiency of the inferencing process. Optimizing data movement, reducing the frequency of transfers, and ensuring that the GPU cores have sufficient work to perform while data is in transit are critical considerations in maximizing the performance of GPU-accelerated machine learning inference.Overview of models-on-silicon

[0044] Various other solutions, while capable of performing machine learning inference tasks, are lacking in one aspect or another. To overcome at least some of these limitations, a dedicated, efficient, and cost-effective chip can be designed and implemented for machine learning inference. In particular, the chip can be designed to support and perform inference according to a transformer-based neural network, such as an open-source transformer-based neural network or an open-source LLM.

[0045] According to one aspect, the disclosed solution, referred to herein as models-on- silicon, introduces a groundbreaking chip architecture that is specifically designed to encapsulate the LLM weights and inference architecture directly onto the hardware. This unique models-on-silicon architecture design optimizes a given LLM by etching the weights onto the chip, eliminating the recurring task of loading these weights and model into GPUs every time.

[0046] According to one aspect, the models-on-silicon architecture utilizes a sequential readonly memory to store one or more weights of a transformer-based neural network. The weights of the transformer-based neural network are thus etched onto the sequential read-only memory and fixed onto the hardware. An application processor no longer has to load weights onto memory or compile a processing graph of a transformer-based neural network and load the compiled instructions onto the GPU. In some embodiments, the sequential read-only memory may power up an active word line and a next active word line and powers down one or more other word lines.

[0047] According to one aspect, the models-on-silicon architecture includes a memory to store a key-value cache for the transformer-based neural network. The memory to store the key-value cache may be a sequential read memory. The key-value cache may be a sequential write memory.

[0048] The one or more memories in the models-on-silicon architecture can be sequential and do not require random-access. Each line can be read in its designated time slot along with theDOCKET NO.: AG2834-US-PCToperation for it. This maximizes performance, simplifies routing, and enables quick access to data, weights, key-value cache, and / or activations.

[0049] According to one aspect, the models-on-silicon architecture facilitates placing one or more memories in close proximity to the custom-built circuits that are performing the logic operations. The architecture not only frees up the need to persistently retrieve an LLM's weights from a main memory (e.g., a large static random-access memory (SRAM)) for each computation but also allows the data to be strategically positioned in close proximity to the logic operations.

[0050] According to one aspect, the models-on-silicon architecture has one or more (custom- built) circuits to perform the logic operations and / or calculations of the transformer-based neural network. The custom-built or purpose-built circuits encapsulate operations of the inference architecture directly on hardware. Custom circuits can be highly efficient and have low-power consumption and smaller area.

[0051] According to one aspect, the one or more circuits include a read-only memory to store a look up table (LUT) having one or more precomputed values of an exponent function.

[0052] According to one aspect, the one or more circuits include a read-only memory to store a look up table having one or more precomputed values of a sigmoid linear unit function.

[0053] According to one aspect, the one or more circuits include a (custom-built) multiplier circuit to multiply an embedding value of an embedding vector of the transformer-based neural network and a weight value of a weight matrix of the transformer-based neural network. In some cases, the weight value can be read from a sequential read-only memory.

[0054] In some cases, the multiplier circuit is specifically designed to perform multiplication of an 8-bit floating-point (FP8) number and a 6-bit floating-point (FP6) number. For example, the weight value may be a 6-bit floating-point number, and the embedding value is an 8-bit floating-point number. In some cases, the multiplier circuit is specifically designed to perform multiplication of an FP8 number and a 4-bit floating-point (FP4) number. For example, the weight value may be a 4-bit floating-point number, and the embedding value is a 8-bit floating-point number. In some cases, the multiplier circuit is specifically designed to perform multiplication of an FP6 number and an FP4 number. For example, the weight value may be a 4-bit floating-point number, and the embedding value is a 6-bit floating-point number. In some cases, the multiplier circuit is specifically designed to perform multiplication of a 16-bit floating-point (FP16) number and a FP16 number.

[0055] According to one aspect, the multiplier circuit includes a multiplexer to allow the bypassing of the etched weight value and use a different weight value instead. In some cases, an application processor may selectively apply one or more weight values of a low-rank weight matrix that was generated by fine-tuning the transformer-based neural network. In such cases, the weight value toDOCKET NO.: AG2834-US-PCTbe used or processed in the multiplier circuit can be read from a read-write memory storing the one or more weight values of the low-rank weight matrix. In some cases, one or more etched weight values may have errors, and one or more repair weight values can be selectively applied in place of the etched weight values. In such cases, the weight value to be used or processed in the multiplier circuit can be read from a read-write memory storing one or more repair weight values for the transformer-based neural network.

[0056] According to one aspect, the one or more circuits include a tree adder circuit. According to one aspect, the one or more circuits include a tree comparator circuit. The tree / hierarchical structures facilitate processing a large number of inputs in parallel to produce a final output. The tree / hierarchical structures can perform processing in a feedforward manner without recursion. In some cases, the adders in the tree adder operate with wide bit-width numbers to avoid overflow.

[0057] According to one aspect, the models-on-silicon architecture includes a flow control circuit (also referred to as a sequencer, a sequencer circuit, an orchestrator circuit, etc.). The flow control circuit orchestrates the operations of a transformer-based neural network in a feedforward manner, as if following a predetermined timing sequence or recipe of operations. Because the models- on-silicon chip implements a predetermined inferencing task of a predetermined transformer-based neural network, the timing sequence of operations (including how many clock cycles each operation takes, the data flow between operations, etc.) is known or established ahead of time. The timing sequence can specify one or more operations of an inferencing task of the transformer-based neural network to be performed at a given clock cycle. The timing sequence may specify the overall sequence of operations to be performed. The timing sequence can specify the data being processed by a given operation. The timing sequence can specify the data being generated by a given operation. The flow control circuit may control gates, muxes, flip-flops, etc., to execute the timing sequence and orchestrate the (custom-built) circuits to perform the operations according to the timing sequence. The flow control circuit can control the data flow into and / or out of the one or more (custom-built) circuits. The flow control circuit can enable and / or disable the one or more (custom-built) circuits according to a predetermined timing sequence. The flow control circuit may include digital logic to generate control signals, timing signals, trigger signals, etc., which can be used to control one or more of: gates, muxes, flip-flops, and custom circuits. The signals can cause the one or more (custom-built) circuits to follow and execute operations of the transformer-based neural network, e.g., in a feedforward manner, according to the predetermined timing sequence.

[0058] According to one aspect, the models-on-silicon chip architecture embeds a feedforward-only transformer-based neural network. In comparison to other solutions, the models-on-DOCKET NO.: AG2834-US-PCTsilicon chip architecture avoid the need to implement software, complex program control or counters, or back propagation, since the model is only feedforward. The models-on-silicon chip architecture and the hardware execution timing sequence involve only forward pass.

[0059] The models-on-silicon chip encapsulates an LLM inferencing model on a single chip and includes a token interface that can demand low bandwidth per inferencing task into the system-on- a-chip (SoC). The models-on-silicon architecture ensures a highly scalable solution, as any number of SoCs can be connected in parallel to handle multiple batches of inference requests simultaneously with low overhead. The models-on-silicon design revolutionizes the way Al inference tasks are handled, making it both cost-effective and scalable.

[0060] One of the advantages of the disclosed solution is its cost-effectiveness. Unlike general-purpose GPUs, this chip is specifically designed to handle Al inference tasks, and thus, does not carry any overhead of unnecessary or general-purpose functionalities. This focus on specific tasks makes it a much more cost-effective solution. The disclosed solution enables faster machine learning inference and reduces power consumption, can offer offering a more efficient and environmentally friendly solution for artificial intelligence tasks.

[0061] This disclosed models-on-silicon solution solves the problem of cost, high power consumption, and time delay, in Al inference by integrating the LLM weights and model onto the hardware itself, effectively removing the need to load weights onto the GPU every load. In some embodiments, the chip includes custom-built circuits for matrix multiplication, allowing for efficient computation. By embedding the weights and the model onto the hardware, power consumption is significantly reduced, and inference tasks are completed faster, while cost is low. The disclosed solution can be visualized as a chip with multiple modules for computations and dedicated sections for weight storage. Various aspects can together contribute to increased performance, scale, reduction of power consumption and area on the chip, reduction in real-time compute calculations, and more.

[0062] By hardcoding the LLM weights and architecture onto the chip, the time and power to load these weights from memory are significantly reduced. As a result, inference tasks can be executed faster, providing a significant performance boost. The disclosed solution reduces power consumption by eliminating the need to repeatedly load weights and models from memory for each inference task. This makes the solution more power-efficient, reducing the overall operational cost, and making it a more environmentally friendly solution. Unlike general-purpose GPUs or FPGAs, this dedicated chip is specifically designed to handle Al inference tasks. Therefore, it does not carry any overhead of unnecessary or general-purpose functionalities, making it a more cost-effective solution. Due to encapsulation of a full LLM inferencing model on a single chip and a token interface, requiring a very low bandwidth per inferencing task into the SoC, a number of SoCs can be connected to in parallel toDOCKET NO.: AG2834-US-PCTsimultaneously handle multiple batches of inference requests with low overhead, making the disclosed solution scalable. Because the model and weights are hardcoded into the hardware, model integrity is assured and less susceptible to manipulation. The disclosed solution can be more secure. The power efficiency and performance boost offered by this invention make it ideal for real-time computing, such as edge computing, mobile and I nternet-of-Things (loT) applications where resources are limited, and low latency may be required.

[0063] Relative to solutions where model weights are stored in HBM, the models-on-silicon chip is much faster, with 150x better latency, because the data is located where it is used. In addition, the models-on-silicon chip is more power-efficient due to the use of sequential read-only memories with 3000x better power efficiency. Relative to solutions that support generic matrix-to-matrix multiplication, vector-to-matrix multiplication, and matrix-to-vector multiplication, the models-on-silicon chip implements a predefined matrix multiplier to perform vector dot product operations that multiply an FP8 valued vector and FP6 valued vector to enable optimization in the hardware bit level, save die area, enable faster operations, and reduce power. Relative to solutions that compute values for activations, the models-on-silicon chip implements predefined look up tables with values precalculated in advance to save compute calculations in real-time. Relative to solutions where the model definition has to be compiled and loaded to run the model, the models-on-silicon chip while being less flexible, can enable highly optimized hardware design, save die area, enable faster operation, and reduce power.

[0064] Applications that can potentially benefit from having a more efficient solution may include huge Al models with hundreds of billions of parameters deployed on GPUs, TPUs, CPUs and cloud computing environments, mid-to-small Al models with a few to a dozen billion parameters deployed in humanoid robots and personal computers, and tiny Al models with less than a billion parameters deployed on mobile devices. Use cases that can benefit from having a more efficient solution may include real-time speech-to-text, real-time text-to-speech, dictation, translation, personal assistance, LLM operating system, LLM supervisor activating experts like coding LLM and productivity LLM, autonomous robots with reasoning, humanoids, cars, appliances, smart carts, smart factories, video-to-tokens, generating video tokens for LLMs training at scale, etc.

[0065] FIGS. 1-22 detail the innovations with models-on-silicon chip and architecture.

[0066] In some variants of the models-on-silicon chip, the sequential read-only memory is replaced by a sequential read memory whose data can be written onto the memory more than once. The data on the sequential read memory, such as the weights and parameters of the transformerbased neural network, would be read sequentially by the circuits performing operations of the transformer-based neural network, e.g., one word line at a time. The operations utilizing the weights and parameters of the transformer-based neural network are analyzed, e.g., by a compiler or otherDOCKET NO.: AG2834-US-PCTsuitable software, to determine how to organize the weights and parameters in the sequential read memory such that they can be read sequentially and be supplied to the corresponding operation at specified time periods or cycles. The organized weights and parameters can be written to the sequential read memory on the models-on-silicon chip.Orchestrating and deploying a system of specialized model-on-chips as a further solution and technical advantages thereof

[0067] The problem being addressed is the need for an efficient and scalable solution to deploy multiple specialized Al models that can work together to perform complex tasks. Some solutions focus on embedding a single, LLM onto a chip, which, while efficient for specific tasks, falls short in some real-world applications where multiple specialized models are cooperating to perform different tasks. Such solutions involve etching a single model onto a chip, which is limiting. This approach is limited to the chips’ memory density allowing only a small range of capabilities of that one model, making it inefficient for scenarios that require diverse functionalities such as text-to-speech, summarization, question answering (QA), and program / project management. Many real-world applications involve tasks and complex use cases that cannot be efficiently handled by a single model.

[0068] For example, in software development, one model might be needed for code generation, another for testing, another for documentation, and yet another for program / project management. Similarly, in autonomous systems or a network of sensing devices, edge devices, or I nternet-of-Things devices, different models are used for various tasks, such as perception, decisionmaking, and control. In one example of a smart home network of devices, various electronic devices in the smart home network may benefit from executing one or more specialized models at the edge. A doorbell camera may execute a vision transformer-based neural network (ViT) to perform action segmentation on a captured video, a security home manager may utilize the embeddings produced by the ViT to execute a control model to determine movement of cameras and turning on lights to track activity captured in the video, and a speaker system with a voice-activated assistant may execute a speech-to-text model to process an utterance of a user asking about the video and a text-to-speech model to generate a voice message to a user in the home based on the results of the action segmentation performed on the video. The single model approach does not scale well when more models or more complex tasks are to be performed.

[0069] In one approach involving a GPU-based solution, a GPU is used where model weights are loaded from memory every time an inference task is performed. A potential drawback of this approach is that the process consumes significant power and time, particularly for complex models. GPUs are designed to handle a wide range of tasks, making them inefficient for dedicated tasks like inference on a pre-trained model alone. In one approach involving a CPU-based solution, CPUs areDOCKET NO.: AG2834-US-PCTused for machine learning inference tasks. One potential drawback is that CPUs are not suitable for large-scale matrix multiplications which are essential for machine learning inference tasks. They also consume more power and are slower in comparison to dedicated solutions. In one approach involving distributed systems, different models are deployed on different machines, but this can be challenging to manage and optimize. Some of the approaches mentioned above, while capable of performing machine learning inference tasks, lack in one aspect or another.

[0070] A proposed solution, referred to herein as, agent chip (or referred to as an agent LLM chip or agent models-on-silicon chip herein) in a multi-chip system and architecture can overcome these limitations by providing a dedicated, efficient, and cost-effective solution to support complex, multi-task machine learning inference. The agent chip in multi-chip architecture addresses some of the challenges and limitations described above by orchestrating multiple specialized Al models embedded and / or etched on different chips.

[0071] Implementing the agent chip effectively solves the problem of deploying multiple specialized Al models in a cost-effective and scalable manner by training and utilizing the agent chip to orchestrate multiple specialized Al models embedded on different models-on-silicon chips. Each models-on-silicon chip orchestrated by the agent chip is optimized for a specific task or goal, and the agent chip coordinates their activities to perform complex, multi-faceted tasks efficiently. Accordingly, the multi-chip architecture allows for efficient, scalable, and cost-effective machine learning inference, significantly reducing power consumption and latency.

[0072] The agent chip acts as a coordinator or a router, similar to how Mixture of Experts (MoE) models operate in transformer-based neural networks and LLMs. In this setup, the agent chip routes tasks (e.g., in the form of tokens and / or embeddings of the tokens) to one or more most appropriate specialized models-on-silicon chip based on the nature of the task and available chips in the system. The multi-chip with an agent chip represents a highly efficient and scalable approach to handling diverse and complex tasks, as each chip can focus on its specialized function or goal while the agent chip ensures optimal task distribution and resource utilization. The multi-chip system having the agent chip can be pre-trained as a whole, with the agent chip model being trained to select the best model for each task. Training the agent chip can involve applying machine learning to understand the strengths and weaknesses of each specialized model embedded on the models-on-silicon chips and making real-time decisions to optimize overall performance.

[0073] The models-on-silicon chips orchestrated by the agent chip, and in some embodiments, the agent chip itself, leverages the models-on-silicon architecture (as described and illustrated in FIGS. 1-22) and includes modules for matrix multiplication, allowing for efficient computation, and dedicated sections for weight storage, enabling fast and efficient retrieval of modelDOCKET NO.: AG2834-US-PCTweights during inference. The solution can be understood as a multi-chip system where each models- on-silicon chip is embedded and / or etched with a model optimized for a specific task, such as text-to- speech, summarization, QA, or project / program management. The models-on-silicon chips are realized using the models-on-silicon (model-on-chip or model-on-die) architecture and design (where the model can be up to 10B parameters) as illustrated in FIGS. 1-22. This multi-chip architecture allows for specialization, scalability, and flexibility, with the ability to easily update or expand by adding new chips with different models without the need to redesign the entire hardware setup. The agent chip can be retrained to accommodate changes in the multi-chip system (e.g., fewer or more models-on-silicon chips). The multi-chip architecture significantly reduces power consumption and latency because the models- on-silicon chips are highly power-efficient, and exchanging embeddings between the models-on-silicon chips can be computationally efficient, making the overall multi-chip system highly efficient and cost- effective.

[0074] The agent chip implements a router neural network model to fit the task of orchestrating the multi-chip system. This multi-chip architecture allows for:• Specialization: Each chip is etched with a model optimized for a specific task, such as text-to- speech, summarization, QA, or project / program management.• Scalability: The agent chip can manage any suitable number of models-on-silicon chips, each performing different tasks, making the system highly scalable and adaptable to various use cases.• Flexibility: The system can be updated or expanded, e.g., by adding new models-on-silicon chips with different models, without the need to redesign the entire hardware setup.

[0075] The multi-chip system can include an expert chip having a transformer-based neural network embedded on-chip and a sequential read memory storing one or more parameters of the transformer-based neural network. The expert chip can be realized using the models-on-silicon architecture illustrated in FIGS. 1-22. The multi-chip system further includes a further expert chip having a further transformer-based neural network embedded on-chip and a further sequential read memory storing one or more further parameters of the one or more further parameters of the further transformerbased neural network. The further expert chip can be realized using the models-on-silicon architecture illustrated in FIGS. 1-22. The multi-chip system can further include an agent chip implementing a router neural network model. The router neural network model can select and / or route, according to one or more yet further parameters of the router neural network model, one or more embeddings to one or more of the expert chip and the further expert chip.

[0076] The agent chip can be implemented as an integrated circuit, which can include a sequential read memory, one or more hardware circuits, a SoftMax circuit, and a top-K selection circuit.DOCKET NO.: AG2834-US-PCTThe sequential read memory (in some cases, a read-only memory) can store one or more parameters of a router neural network model. The one or more hardware circuits can implement one or more operations of the router neural network model, and process one or more input embeddings using the one or more parameters read from the sequential read memory. The SoftMax circuit can process one or more outputs from the one or more hardware circuits and produce probabilities. The top-K selection circuit can select one or more expert chips among a plurality of expert chips (e.g., a plurality of models- on-silicon chips realized using the models-on-silicon architecture illustrated in FIGS. 1-22). The plurality of expert chips can include an expert chip having a transformer-based neural network embedded / etched on-chip and a further expert chip having a further transformer-based neural network embedded / etched on-chip) based on one or more further outputs from the SoftMax circuit (e.g., the one or more probabilities). For example, K expert chips having the top probabilities may be selected and the input embeddings or a derivation thereof may be routed to the selected K expert chips. In some cases, K=1. In some cases, K=2. In some cases, K is greater than 1.

[0077] The agent chip can perform a method for orchestrating multi-task machine learning inference. The agent chip can receive one or more embeddings from an expert model. The expert model is among a plurality of expert chips (e.g., a plurality of models-on-silicon chips realized using the models-on-silicon architecture illustrated in FIGS. 1-22). The one or more embeddings are generated using one or more parameters stored in a sequential read memory of the expert chip. The agent chip can input the one or more embeddings into a router neural network model implemented on an agent chip. The router neural network model implemented on the agent chip can select one or more one or more expert chips among the plurality of expert chips to route the one or more embeddings. The selection can be performed according to one or more further parameters of the router neural network model implemented on the agent chip. The agent chip can output or route the one or more embeddings to the one or more selected expert chips.

[0078] The multi-chip solution having the agent chip enables faster machine learning inference and reduces power consumption, offering a more efficient and environmentally friendly solution for artificial intelligence tasks by leveraging specialized LLM models in a multi-chip architecture, orchestrated by the agent chip. By hardcoding the LLM weights and architecture onto multiple specialized models-on-silicon chips, the time and power required to load these weights from memory are significantly reduced. As a result, inference tasks can be executed faster, providing a significant performance boost. The solution reduces power consumption by eliminating the need to repeatedly load weights and models from memory for each inference task. The specialized chips operate efficiently, making the solution more power-efficient and reducing the overall operational cost, thus being more environmentally friendly. Unlike general-purpose GPUs or FPGAs, these dedicated models-DOCKET NO.: AG2834-US-PCTon-silicon chips are specifically designed to handle Al inference tasks. Therefore, they do not carry any overhead of unnecessary or general-purpose functionalities, making the solution more cost-effective. Due to the encapsulation of specialized LLM models on multiple chips and the use of a token or embeddings interface, the system utilizes very low bandwidth per inferencing task into SoC. Multiple SoCs can be connected in parallel to simultaneously handle numerous batches of inference requests with low overhead, enhancing scalability. As the models and weights are hardcoded into the hardware, model integrity is assured and less susceptible to manipulation, enhancing security. The power efficiency and performance boost offered by the multi-chip system make the solution ideal for edge computing, mobile and I nternet-of-Things applications where resources are limited and low latency is desirable.

[0079] In some embodiments, the router neural network model is referred to as a task management neural network model. The agent chip implementing the task management neural network model can communicate with one or more expert chips. Each expert chip has a transformer-based neural network model embedded on the expert chip, leveraging the models-on-silicon architecture. Each expert chip has one or more parameters that are trained for a computing task to be performed by the expert chip. The agent chip routes one or more tokens and / or one or more embeddings associated with the computing task to one or more selected expert chips of the one or more expert chips and receives one or more results of from the one or more selected expert chips.

[0080] In some embodiments, a chip in the multi-chip solution receives, processes, and / or outputs tokens. Tokens refer to basic units of text or data that the transformer-based neural network processes. When the model is processing text, tokens can correspond to a word, a sub-word, or a character. Input data is broken down into tokens, i.e., manageable units of data, through a process called tokenization, before being fed into the neural network.

[0081] In some embodiments, a chip in the multi-chip solution receives, processes, and / or outputs embeddings or token embeddings. Embeddings are dense vector representations of tokens. Embeddings can capture semantic, contextual, syntactic, and / or positional meaning of tokens in a way that can be interpreted by the neural network. A token can be mapped to a high-dimensional vector space, where tokens which have similar semantic meanings can be located closer to each other. Various operations in the neural network transform the embeddings as the embeddings progress through the neural network.Exemplary models-on-silicon chip architecture

[0082] FIG. 1 illustrates an exemplary chip architecture, according to some embodiments of the disclosure. FIG. 2 illustrates exemplary details within the parts of the exemplary chip architecture,DOCKET NO.: AG2834-US-PCTaccording to some embodiments of the disclosure. Models-on-silicon chip 100 is depicted in both figures to illustrate exemplary implementations.

[0083] A “models-on-silicon” chip 100 illustrated in FIGS. 1-2 may include one or more of: embedder circuit 102, RMS normalizer circuit 104, flow control circuit 106, sampler circuit 108, and one or more etched mind units 110 (etched mind units are referred to as EMUs). Exemplary implementations of embedder circuit 102 are illustrated in FIG. 14. Exemplary implementations of RMS normalizer circuit 104 are illustrated in FIG. 15. Exemplary implementations of sampler circuit 108 are illustrated in FIGS. 16-17.

[0084] An EMU of one or more etched mind units 110 may include one or more of: one or more rotary embedder circuits 112, one or more SI LU activator circuits 114, one or more SoftMax circuits 118, one or more embedding dot unit circuits (EDUs) 116, one or more attention dot unit circuits (ADUs) 120.

[0085] In one implementation, an EDU of the one or more embedding dot unit circuits may carry out a (4096-elements) dot product operation between FP8 embedding vector and FP6 weights vector stored in one or more ROMs 130, e.g., every cycle. The dot product operation can be performed using one or more tree adders 202 and one or more multipliers 204 in the EDU.

[0086] In one implementation, an ADU of the one or more attention dot unit circuits 120 may carry out a (128-elements) dot product operation between FP16 input vector and FP16 K or V vector cached in one or more SRAMs 140, e.g., every cycle. The dot product operation can be performed using one or more tree adders 206 and one or more multipliers 208 in the ADU.

[0087] Exemplary implementations of one or more rotary embedder circuits 112 are illustrated in FIGS. 18A-18B. Exemplary implementations of one or more SILU activator circuits 114 are illustrated in FIGS. 8A-8B. Exemplary implementations of one or more SoftMax circuits 118 are illustrated in FIG. 13. Exemplary implementations of one or more EDU circuits 116 are illustrated in FIGS. 9-10. Exemplary implementations of one or more ADU circuits 120 are illustrated in FIG. 6.

[0088] An EDU of one or more EDU circuits 116 can include one or more tree adders 202. The EDU may include one or more multipliers 204. A multiplier in one or more multiplier 204 may multiple two values, such as two floating-point values. For example, one or more multipliers 204 may include an FP4 / FP6 multiplier. One or more multipliers 204 may include an FP4 / FP8 multiplier, one or more multipliers 204 may include an FP6 / FP8 multiplier. One or more multipliers 204 may be specifically designed to perform multiplication of values or data having predetermined representations (e.g., FP4, FP6, FP8, FP12, INT8, etc.). One or more multipliers 204 may read data from one or more ROMs 130. One or more tree adders 202 may add multiplication results produced by one or more multipliers 204 together.DOCKET NO.: AG2834-US-PCT

[0089] An EMU of one or more etched mind units 110 may include one or more ROMs 130 that can store and provide data to one or more circuits performing logic operations in an EDU of EDU circuits 116. One or more ROMs 130 may include one or more sequential read-only memories, which may be placed in proximity to the circuits performing logic operations in the EDU. Exemplary implementations of the one or more ROMs 130 are illustrated in FIG. 5.

[0090] An ADU of one or more ADU circuits 120 can include one or more tree adders 206. The ADU may include one or more multipliers 208. A multiplier in one or more multiplier 204 may multiple two values, such as two floating-point values. For example, one or more multipliers 208 may include an FP16 / FP16 multiplier. One or more multipliers 208 may be specifically designed to perform multiplication of data having predetermined representations (e.g., FP4, FP6, FP8, FP12, FP16, INT8, etc.). One or more multipliers 208 may read data from one or more SRAMs 140. One or more tree adders 206 may add multiplication results produced by one or more multipliers 208 together.

[0091] An EMU of one or more etched mind units 110 may include one or more SRAMs 140 that can store and provide data to one or more circuits performing logic operations in an ADU of ADU circuits 120. One or more SRAMs 140 may include one or more sequential read / write memories, which may be placed in proximity to the circuits performing logic operations in the ADU.

[0092] In some embodiments, models-on-silicon chip 100 is a model-specific integrated circuit. The integrated circuit includes a sequential read-only memory (e.g., one or more ROMs 130) to store one or more weight values of a weight matrix of a transformer-based neural network. The integrated circuit includes one or more circuits to perform one or more operations of an inferencing task of the transformer-based neural network (e.g., various circuits illustrated in FIGS. 1-2). The integrated circuit includes a sequencer circuit to orchestrate the one or more circuits according to a predetermined timing sequence of the transformer-based neural network (e.g., flow control circuit 106).

[0093] Flow control circuit 106 (also referred to as a sequencer circuit) plays a role in orchestrating various circuits to execute operations according to a predetermined timing sequence. Advantageously, a transformer-based neural network operates in a feedforward manner. The sequence of operations of the transformer-based neural network corresponding to different layers of the neural network can be determined and mapped into a timing sequence of operations. The timing sequence of operations may include stages of operations, one following another. In a particular time slot or stage in the timing sequence, data can be moved in, processed, and moved out to be processed in the next / following time slot, in a feedforward, progressive manner. Flow control circuit 106 thus can implement digital logic to generate clock edges / signals (e.g., control signals, timing signals, enable signals, disable signals, trigger signals, etc.) to orchestrate operations to be performed according to the timing sequence. Flow control circuit 106 can control data flow into and / or out of the one or moreDOCKET NO.: AG2834-US-PCTcircuits. Flow control circuit 106 can enable and / or disable the one or more circuits according to a predetermined timing sequence.

[0094] According to one aspect, the models-on-silicon chip 100 illustrated in FIGS. 1-2 provides and implements at least a part of or an entire generative Al model (e.g., a transformer-based neural network, an LLM, etc.) in a single chip or integrated circuit. This involves integrating the generative Al model into a single chip, e.g., as illustrated as models-on-silicon chip 100 in FIGS. 1-2. The chip 100 receives tokens in and outputs tokens out. The entire architecture, weights, and flow of the generative Al model can be embedded into the chip 100.

[0095] In one exemplary implementation where chip 100 embeds a specific transformer-based neural network, there are 32 instances of EMUs 110 on models-on-silicon chip 100. In an EMU, there may be 4 instances of SILU activator circuit 114. An instance of SILU activator circuit 114 may include a look up table 220, e.g., a 96 Kilobyte (KB) look up table. In an EMU, there may be 4 instances of rotary embedder circuit 112. An instance of rotary embedder circuit 112 may include a look up table 230, e.g., 2KB look up table. In an EMU, there may be 8 instances of EDU circuit 116. In an EMU, there may be 16 instances of ADU circuit 120.

[0096] An instance of an EDU may include tree adder 202, e.g., a tree adder to add 4096 inputs. An instance of an EDU may include 4096 instances of multiplier 204. An instance of EDU may include 4096 instances of sequential read-only memory 130, e.g., 4.6 KB sequential read-only memory. A sequential read-only memory may be provided for an individual multiplier, e.g., in proximity to the multiplier. In total, one or more EDU circuits 116 may include 4.6 Gigabytes (GB) of sequential readonly memory, and 1 ,048,576 multiplier circuits and adder circuits.

[0097] An instance of an ADU may include tree adder 206, e.g., a tree adder to add 128 inputs. An instance of an ADU may include 128 instances of multiplier 208. An instance of ADU may include 128 instances of sequential read / write memory 140, e.g., 4 KB sequential read / write memory. A sequential read / write memory may be provided for an individual multiplier, e.g., in proximity to the multiplier. In total, one or more ADUs may include 256 Megabytes (MB) of sequential read / write memory, and 65,536 multiplier circuits and adder circuits.

[0098] According to one aspect, the chip 100 illustrated in FIGS. 1-2 has the actual components, blocks, and parts that make up the operations of an inference task of a transformer-based neural network model architecture. The chip 100 thus includes circuits that implement one or more transformer blocks. The circuits may implement various operations in a transformer block, e.g., SoftMax, attention, RMS normalizer, etc. For example, embedding the chip with an open-source model would mean that the way the hardware blocks are connected to each other on the chip would match the architecture of the open-source model.DOCKET NO.: AG2834-US-PCT

[0099] FIG. 3 illustrates embedding an exemplary open-source model onto the chip, according to some embodiments of the disclosure. As illustrated, the model includes one or more functional blocks, such as tokenizer 330, embedder 302, RMS normalizer 304 operating on weights vector 306, one or more transformers 308 (e.g., 32 transformer blocks), matrix multiply 310 operating on weight matrix 312, and sampler 314 (e.g., deterministic sampler). Some functional blocks of the model, such as embedder 302, RMS normalizer 304 operating on weights vector 306, one or more transformers 308, matrix multiply 310 operating on weight matrix 312, and sampler 314, as seen in FIG. 3 can be embedded as circuits onto the models-on-silicon chip 100, as illustrated in FIGS. 1-2.

[0100] Input data (e.g., input words) may be tokenized by tokenizer 330, and input tokens may be output by tokenizer 330. The input tokens (e.g., an input token may be represented as a 15-bit integer) may be provided as input to embedder 302. Embedder 302 may include one or more look up tables. Embedder 302 may output a vector (e.g., a vector having 4096 values). In some embodiments, the values of the vector are FP16 values. The vector may be provided as input to RMS normalizer 304. RMS normalizer 304 may perform the function:Jxi ’RMSiy4,096 2 j=° j + 10- 54,096

[0101] RMS normalizer 304 may read weights vector 306 (Wn3weights vector having 4096 values) from a sequential read-only memory. In some embodiments, the values of weights vector 306 are FP6 values. RMS normalizer 304 may output a vector (e.g., a vector having 4096 values). In some embodiments, the values of the vector are FP8 values. The vector may be processed by one or more transformers 308, which may output a vector (e.g., a vector having 4096 values) to be processed by matrix multiply 310. In some embodiments, the values of the vector of FP8 values. Matrix multiply 310 may read weight matrix 312 (icisweight matrix (e.g., a matrix having FP6 values) a sequential readonly memory. Matrix multiply 310 may perform matrix multiplication between the vector from one or more transformers 308 and weight matrix 312. Matrix multiply 310 may output a vector (e.g., a vector having 128,256 values). In some embodiments, the values of the vector may include FP16 values. The vector is passed onto sampler 314 to get an index of the largest number in the vector and output an output token (e.g., an output token may be represented as a 15-bit integer). The output token may be looped back as an input to embedder 302, since the model is auto-regressive. Timestep may increase by 1 to trigger the model to produce the next output token.

[0102] FIG. 4 illustrates exemplary hardware blocks or circuits representing and corresponding to an exemplary open-source model, according to some embodiments of the disclosure.DOCKET NO.: AG2834-US-PCTSpecifically, the one or more transformers 308 seen in FIG. 3 are depicted in greater detail in FIG. 4. The functional blocks of the one or more transformers 308 (e.g., representing one or more operations of an inferencing task of a transformer-based neural network) seen in FIG. 3, such as matrix multiply, rotary embedder, SoftMax, add, RMS normalizer, SI LU activator, and product, can be embedded onto the chip as the circuits as illustrated in FIGS. 1-2. Specifically, the functional blocks can be implemented in hardware as an EMD (e.g., one or more etched mind units 110 seen in FIGS. 1-2). In some implementations, there are 32 transformers, and thus the 32 transformers can be implemented in hardware as 32 EMDs. The weight vectors and matrices can be stored in sequential read-only memories (e.g., one or more ROMs 130) as depicted in FIGS. 1-2. The KV-cache can be stored in sequential read / write memories (e.g., one or more SRAMs 140) as depicted in FIGS. 1-2. The functional blocks of one or more transformers 308 thus can be directly implemented as circuits on the chip, and the sequencer circuit can configure the circuits corresponding to the functional blocks to operate according to the data and operational flow illustrated in FIG. 4. The circuits (e.g., hardware blocks) of the EMU are coupled to each other according to the data and operational flow as illustrated in FIG. 4.

[0103] A rotary embedder seen in FIG. 4 may implement the following functions: / (*i) = Xi ■ wr- xi+1• Wi

[0104] A SoftMax block seen in FIG. 4 may implement the following:

[0105] An add block seen in FIG. 4 may implement element-wise addition: f(x,y) = x + y

[0106] A product block seen in FIG. 4 may implement element-wise multiplication: f(x,y) = x - y

[0107] A SI LU activator block seen in FIG. 4 may implement the following:

[0108] The data and operational flow illustrated in FIG. 4 can include different groups of operations, e.g., group 402, group 404, group 406, group 408, and group 410, being performed or arranged in a feedforward manner. Group 402 includes two rotary embedders and three matrix multiply blocks. Group 402 may be embedded onto models-on-silicon chip 100 as one or more rotary embedder circuits 112 and one or more EDU circuits 116. Group 404 includes two matrix multiply blocks and aDOCKET NO.: AG2834-US-PCTSoftMax block. Group 404 may be embedded onto models-on-silicon chip 100 as one or more ADU circuits 120 and one or more SoftMax circuits 118. Group 406 includes a matrix multiply block, an add block, and an RMS normalizer block. Group 406 may be embedded onto models-on-silicon chip 100 as one or more EDU circuits 116, and RMS normalizer circuit 104. Group 408 includes three matrix multiply blocks, a SILU activator block, and a product block. Group 408 may be embedded onto models-on-silicon chip 100 as one or more EDU circuits 116 and one or more SILU activator circuits 114. Group 410 includes an add block and an RMS normalizer block. Group 408 may be embedded onto models-on-silicon chip 100 as one or more EDU circuits 116 and RMS normalizer circuit 104.Sequential read-only memory

[0109] FIG. 5 illustrates sequential read-only (SRO) memory, according to some embodiments of the disclosure. According to one aspect, the models-on-silicon chip has one or more instances of SRO memories. SRO memory is a type of memory storage, utilizing ROMs, that allows data to be read sequentially but not written or modified after the values have been etched onto the ROM. The rest of the ROM can be shutdown to reduce power and area. In some embodiments, the models-on-silicon chip has one or more SRO memories. The SRO memory powers up an active current word line and an active next word line at a time, while other word lines can be powered down. The active current word line refers to the word line having data being used or processed by a circuit to perform an operation during a time slot in the predetermined timing sequence. The active next word line refers to the word line having data being used or processed by the circuit to perform an operation during a further / next time slot in the predetermined timing sequence. The SRO memory can power down the rest of the word lines, or the rest of the word lines in the SRO memory can remain powered down. At the next clock or time slot, the active current word line is powered down, the active next word line is already powered up, and a further active next word line is powered up. At every clock or time slot, two word lines are powered up in the SRO memory. The two active word lines that are powered up gets moved by one word line down the SRO memory at every clock or time slot.

[0110] In some embodiments, one or more SRO memories may be provided on the chip to store various weight matrices for a transformer model:DOCKET NO.: AG2834-US-PCT

[0111] There may be 1 ,048,576 Weights ROMs (e.g., SRO memories) in models-on-silicon chip 100 illustrated in FIGS. 1-4. A ROM can hold weights in FP6 format. A ROM output can be a 6-bit value. A weights ROM can hold a specific weight matrix column, since a weights ROM can output a single number out of the 4096-element vector being multiplied in the EDU. A weights ROM can hold one of 256 weight matrix rows, since there are 256 EDUs working in parallel and producing 256 numbers per clock cycle. A ROM can hold matrix rows 1 , 257, ..., and another ROM can hold matrix rows 2, 258, and so forth. In some cases, a weights ROM can hold elements from (all) weights matrices in (all) layers, since a weights ROM sequentially outputs the number the matrix multiplier is using for (all) transformers and matrices, as the weights multipliers are shared across all layers and weights matrices. The weights ROM hold (only) the linear layers’ weights. There may be one or more dedicated ROMs for the embedder and RMS normalizer units.Sequential read / write memory in an attention multiplier circuit

[0112] FIG. 6 illustrates sequential read / write (SRW) memory used in attention multiplier circuit 600, according to some embodiments of the disclosure. According to one aspect, the models-on-DOCKET NO.: AG2834-US-PCTsilicon chip has one or more SRW memories. The SRW memory involves using an SRAM in a special configuration that it is not dynamically readable, but is built up sequentially to reduce power and area. An SRAM that can be read sequentially and / or written sequentially has drastically simplified logic and circuitry for reads and / or writes. An SRW memory can be used with or in an attention dot unit to supply weights to attention multiplier circuit 600. Attention multiplier circuit 600 may be a part of an ADU. In one implementation, the ADU having the attention multiplier circuit 600 may receive an input number and multiplies it by a number from SRAM (e.g., SRW memory) every clock cycle. 64 SRAMs can be used to store the 32 layers and K vs. V separately, so the SRAM can read lines sequentially.

[0113] According to one aspect, the SRW memory may be referred to as Key-Value Static Random-Access Memory (KV SRAM), which can store data in key-value pairs. KV SRAM can enable storing the attention history (e.g., cached keys and values) of a transformer block.

[0114] Referring back to FIG. 6, the models-on-silicon chip includes an attention dot unit (shown as attention multiplier) as illustrated by FIG. 6. The attention dot unit may receive an input number and multiplies it by a number from SRAM - every clock cycle. 64 SRAMs are used to store the 32 layers and K vs. V separately, so the SRAM can read lines sequentially.

[0115] In some embodiments, a models-on-silicon chip has a sequential read / write memory to store a key-value cache for the transformer-based neural network. To improve computational efficiency, one or more key-value caches can be included on-chip with the ADUs to enhance the performance of the transformer-based neural network by temporarily storing frequently accessed data. Keys and values computed in the attention mechanism can be cached to allow for rapid retrieval of information. In the context of transformer-based neural networks, the key typically represents a unique identifier for a specific input or query, while the value contains the corresponding output or computational result. This caching mechanism deals with dynamic data, and thus uses read / write memory, such as SRAM. The key-value cache can significantly reduce latency and computational overhead by avoiding redundant calculations and data fetching, thereby improving the efficiency and responsiveness of the model during inference. Because the cached keys and values can be written and read sequentially during inference, the SRAM implementation can be simplified by restricting reads and writes to be done in a sequential manner (obviating circuits that allow for random-access).

[0116] Attention multiplier circuit 600 may have the following exemplary specification:DOCKET NO.: AG2834-US-PCT

[0117] Attention multiplier circuit 600 may be included in an ADU to perform multiplication of two numbers (e.g., FP16 value and FP16 value), where one of the two numbers is read from the sequential read / write memory storing the key-value cache. As illustrated, attention multiplier circuit 600 includes 64 SRW memories 602, and decoder 604 may turn on one of the 64 SRW memories 602 to be used. Data is read from the active SRW memory serially, e.g., line by line. The data the active SRW memory is multiplied against the input by multiplier 606.DOCKET NO.: AG2834-US-PCT

[0118] Many instances of attention multiplier circuit 600 may be included in an ADU to perform element-wise multiplication, e.g., in parallel. The multiplication results of the instances of attention multiplier circuit 600 can be summed by a tree adder to form a vector dot product result. The ADU may perform many vector dot products to form a final matrix multiplication result.Activator circuits: exponent unit circuit and sigmoid linear unit activator circuit

[0119] In some embodiments, the models-on-silicon chip has one or more read-only memories to store one or more look up tables for approximating one or more functions, e.g., f(x). The look up tables can store precomputed values of a function, f(x). The precomputed values may correspond to one or more values or segments over a range of values of an input number, x. The input number, x, can be used as an index or address to look up and obtain a precomputed value, f(x), from the look up table. The precomputed values can be stored in a ROM. The functions that are a part of the transformer-based neural network are established ahead of time, and thus it is possible to construct look up tables with precomputed values. Compute calculations can be avoided during real-time inference, which saves power and reduces latency.

[0120] Examples of a function may include activation functions. Activation functions introduce non-linearity into the model, enabling it to learn complex patterns. An example of an activation function includes the RELU, which outputs the input directly if it is positive and zero otherwise, thus helping to mitigate the vanishing gradient problem. Another example of an activation function includes the SILU function, which maps input values to a range between 0 and 1 , is often used in binary classification tasks. Another example of an activation function includes the Hyperbolic Tangent (Tanh) function, similar to SILU but with outputs ranging from -1 to 1 , is useful for centering data. Another example of an activation function includes Leaky RELU, which allows a small gradient when the input is negative. Another example of an activation function includes the Swish function, defined as x-sigmoid(x), which has shown to improve model performance by providing smoother gradients and better convergence properties.

[0121] FIG. 7A illustrates exponent unit circuit 700, according to some embodiments of the disclosure. FIG. 7B illustrates an exponent function approximated by exponent unit circuit 700, according to some embodiments of the disclosure. Exponent unit circuit 700 includes a read-only memory to store a look up table 702 having one or more precomputed values of an exponent function:

[0122] In some cases, exponent unit circuit 700 includes mux control 704 and mux 706. Mux control 704 may check whether the input value meets a particular condition, and selects a particularDOCKET NO.: AG2834-US-PCTvalue to use as the output of exponent unit circuit 700. Mux control 704 may output a 2-bit value as selection signal for mux 706, to select one of four possible values to use as the output.

[0123] For example, if the most significant bits (MSBs) of the input are “00”, then the value of “1” is selected by mux 706 to use as the output. If the sign bit is 0 and the MSBs of the input are “11”, then the value of “Inf’ (positive infinity) is selected by mux 706 to use as the output. If the sign bit is 1 and the MSBs of the input are “11 ”, then the value of “0” is selected by mux 706 to use as the output. Otherwise, the value from look up table 702 is used as the output.

[0124] FIG. 8A illustrates a SILU activator circuit 800, according to some embodiments of the disclosure. FIG. 8B illustrates a sigmoid linear unit function and a RELU function, according to some embodiments of the disclosure. SILU activator circuit 800 includes a read-only memory to store a look up table 802 having one or more precomputed values of a SILU function:

[0125] In some cases, SILU activator circuit 800 includes mux control 804 and mux 806. Mux control 804 may check whether the input value meets a particular condition and selects a particular value to use as the output of SILU activator circuit 800. Mux control 804 may output a 2-bit value as selection signal for mux 806, to select one of three possible values to use as the output.

[0126] For example, if the sign bit is 0 and the MSBs of the input are “11 ”, then the input is selected by mux 806 and passed on to use as the output. If the sign bit is 1 and the MSBs of the input are “11”, then the value of “0” is selected by mux 806 to use as the output. Otherwise, the value from look up table 802 is used as the output.Weights multiplier circuit in embedding dot unit circuit

[0127] One operation of an inferencing task of a transformer-based neural network involves multiplying an embedding vector with a weight matrix. The embedding vector can represent a particular token, and various weight matrices of the transformer-based neural network are used to transform the embedding vector as the embedding vector progresses through the transformer-based neural network. The embedding vector is a vector representation of a token, and can be a dense, high-dimensional vector that encodes various types of information about the token, such as semantic information, syntactic information, contextual information, and positional information about the token. The weight matrix has weight values which have been learned through training to transform an embedding vector to extract patterns and relationships in the data.

[0128] Because the vector-to-matrix multiplication operation to be performed in models-on- silicon is known, the one or more circuits can include a custom-built embedding dot unit circuit that can perform the multiplication of the embedding vector with a weight matrix with low-power. The custom-DOCKET NO.: AG2834-US-PCTbuilt embedding dot unit circuit can be designed to perform vector dot products. Multiplying an embedding vector having 1 by X elements with a weight matrix having X by Y elements involves calculating Y vector dot products and producing an output vector having Y elements (the output vector having the Y vector dot products). Each vector dot product is a dot product of the embedding vector with a column vector of the weight matrix (or a row vector of the weight matrix).

[0129] To calculate the vector dot product, element-wise multiplication of values in the embedding vector and values in a column / row vector of the weight matrix is performed, and the multiplication results are added together to form a value in the output vector. A number of multiplier circuits multiplying two floating-point numbers (e.g., an embedding value in the embedding vector and a weight value in the weight matrix) can be implemented to perform the element-wise multiplication of values for the vector dot product, e.g., in parallel. A tree adder circuit can be implemented to sum the multiplication results. Because the multiplication operation of an embedding value in the embedding vector with a weight value of the weight matrix is established ahead of time, a custom-built multiplier circuit to multiply the embedding value and the weight value may be implemented, such as a multiplier circuit that performs a specific task of FP8xFP6 multiplication (e.g., the embedding value may be an FP8 value, and the weight value may be an FP6 value).

[0130] According to one aspect, the models-on-silicon chip illustrated in FIGS. 1-4 has optimized physical layout and design. Matrix multiplications are predefined and known, and digital circuits, such as the EDU, can be designed and implemented to perform a specific type of matrix multiplication. Also, the format of the values being operated on are also predefined and known, so custom-built multiplier circuits can be designed and implemented to perform a specific type of multiplication of two values. For example, weights multiplier circuit 900 illustrated in FIG. 9 to be used in an EDU may be predefined and built with one specific task in mind (e.g., FP8xFP6 multiplication). In addition, at least SRO memory 904 is placed in proximity to multiplication circuit 908.

[0131] In some embodiments, the models-on-silicon chip includes weights multiplier circuit 900 (e.g., many instances of weights multiplier circuit 900). Weights multiplier circuit 900 can multiply an embedding value of an embedding vector of the transformer-based neural network and a weight value of a weight matrix of the transformer-based neural network. Weights multiplier circuit 900 may include multiplication circuit 908 to perform multiplication of an FP6 number (e.g., a weight value) and an FP8 number (an embedding value). Multiplication circuit 908 is designed with one specific task, to multiply an FP8 value and an FP6 value. The custom circuitry of multiplication circuit 908 means that the circuitry is simpler and consumes less power than other generic multiplication circuits.

[0132] Weights multiplier circuit 900 includes SRO memory 904 to store weights (e.g., weight values of a weight matrix). In some embodiments, weights multiplier circuit 900 may include SRAM 902.DOCKET NO.: AG2834-US-PCTSRAM 902 may include a small read / write memory to store additional weight values that can be used in place of the etched weight values on SRO memory 904 (e.g., thus bypassing the etched weight values on SRO memory 904).

[0133] In some embodiments, SRAM 902 may store one or more weight values of a low-rank weight matrix. The transformer-based neural network may have pre-trained weights that are stored and etched in SRO memory 904. The transformer-based neural network may be fine-tuned using a Low- Rank Adaptation (LoRA) technique, where a low-rank weight matrix (a much smaller matrix than the original weight matrix) can be trained and updated so that the transformer-based neural network can perform a specific task. One or more tree adders 202 may add multiplication results produced by one or more multipliers 204 together.

[0134] In LoRA, the original weight matrix W can be decomposed into smaller low-rank matrices A and B, where W=B-A. A low-rank weight matrix may be based on the original weight matrix W. A low-rank weight matrix may approximate the original weight matrix W. A low-rank weight matrix may capture significant features of the original weight matrix W while discarding less important features. A low-rank weight matrix may be a compressed version of the original weight matrix W. A low-rank weight matrix may have fewer linearly independent rows or columns when compared to the original weight matrix W. During fine-tuning, the weight values of the low-rank, smaller weights matrices A and B are updated, and not the weight values of the original weight matrix W. The weight values of the low- rank weight matrix can be stored in SRAM 902 to offer some flexibility for the models-on-silicon chip to implement a fine-tuned transformer-based neural network. In some implementations, a 2% LoRA update can be implemented to offer some flexibility. An application processor may write one or more weight values of the low-rank matrix onto SRAM 902.

[0135] In some embodiments, SRAM 902 may store one or more repair weight values. If there are one or more errors or faulty values in SRO memory 904 (the errors or faulty values can occur when values are being etched onto SRO memory 904), the errors or faulty values can be corrected by storing correct values, e.g., one or more repair weight values, in SRAM 902. The one or more repair weight values may correct one or more etched weight values.

[0136] Weights multiplier circuit 900 may include mux 906, SRAM 902, and SRO memory 904. Mux 906 can be used to select an output from SRAM 902 or an output from SRO memory 904 to be used as an input to multiplication circuit 908. Advantageously, mux 906 allows bypassing of a value read from SRO memory 904, and using the value from SRAM 902 to be used instead as the input to multiplication circuit 908. If selected by mux 906, multiplication circuit 908 may perform multiplication of a weight that is read from SRO memory 904. If selected by mux 906, multiplication circuit 908 mayDOCKET NO.: AG2834-US-PCTperform multiplication of a weight that is read from SRAM 902, such as a weight value of a low-rank weight matrix, or a repair weight value.

[0137] FIG. 10 illustrates embedding dot unit circuit 1000, according to some embodiments of the disclosure. According to one aspect, the models-on-silicon chip includes one or more instances of embedding dot unit circuit 1000. Embedding dot unit circuit 1000 can perform elements dot product operation between an embedding vector (e.g., FP8 embedding vector) and a weights vector (e.g., FP6 weights vector read from SRO memory) every cycle. Embedding dot unit circuit 1000 may include one or more instances (e.g., 4096 instances) of weights multiplier circuit 900. The instances of weights multiplier circuit 900 may perform multiplication in parallel. The outputs (e.g., 4096 outputs) may be added together by tree adder circuit 1002 of embedding dot unit circuit 1000. Embedding dot unit circuit 1000 may include tree adder circuit 1002 to add one or more multiplication results produced by one or more instances of weights multiplier circuit 900. In an implementation that adds 4096 numbers together, tree adder circuit 1002 may include 12 layers of adders and a total of 4095 adders. To sum all the multiplication results and receive a fused multiple add effect, tree adder circuit 1002 can implement a tree or hierarchical structure (and not a recursive structure) to add multiple input simultaneously and efficiently. In some embodiments, tree adder circuit 1002 uses a special fixed-point adder with a relatively large number of bits (e.g., 20 bits, 21 bits, ... 32 bits), and uses a sampler 1004 to resample the final sum into a floating-point representation. Embedding dot unit circuit 1000 may generate an FP16 output. Using a large number of bits in tree adder circuit 1002 can prevent overflow during many stages / layers of adding.Power and clock gating

[0138] According to one aspect, the models-on-silicon chip can implement power / clock gating of one or more hardware components / blocks when not in use. In addition, using purpose-built SRO memories and SRW memories, it is possible to shut most of the memory off when only one line is needed for a given operation. In some cases, power and clock gating can be implemented by a sequencer circuit (e.g., flow control circuit 106 of FIGS. 1-2).Bit cell area optimization

[0139] FIG. 11 illustrates bit cell area optimization, according to some embodiments of the disclosure. According to one aspect, the models-on-silicon chip illustrated in FIGS. 1 -4 benefits from reduced bit cell area. Due to relaxed performance requirement and architecture enabled circuit optimization, the area of a bit cell in ROM can be reduced. The models-on-silicon chip has array efficiency (AE) between 80-85%, which may translate to 1 .5x density gain.Custom multiplier circuitsDOCKET NO.: AG2834-US-PCT

[0140] FIG. 12 illustrates a weights multiplier circuit, according to some embodiments of the disclosure. According to one aspect, a weights multiplier implements tailor made optimized hardware for specific floating-point multiplication. In contrast to the multiplication circuit 908 of FIG. 9, the logic shown in FIG. 12 implements multiplying a FP4 input by a FP8 input.

[0141] It is envisioned by the disclosure that various custom floating-point multiplication logic can be implemented for performing floating-point multiplication on the models-on-silicon chip (e.g., FP4xFP8, FP6xFP8, FP16xFP16, etc.).SoftMax circuit

[0142] FIG. 13 illustrates SoftMax circuit 1300, according to some embodiments of the disclosure. According to one aspect, the models-on-silicon chip includes a hardware implementation of the SoftMax function, e.g.,:

[0143] SoftMax circuit 1300 depicted in FIG. 13 includes look up table implementation of a SoftMax function and is not a compute-oriented solution. SoftMax circuit 1300 receives an input vector of t FP16 elements (1 <t<512) and return the SoftMax normalized vector of the same size. SoftMax circuit 1300 receives 16 numbers per cycle for up to 32 cycles and returns 16 numbers per cycle for up to 32 cycles. SoftMax circuit 1300 can have the following exemplary specification:DOCKET NO.: AG2834-US-PCT

[0144] SoftMax circuit 1300 may be included in an ADU to perform SoftMax on an input vector (e.g., FP16 vector) and to output a SoftMax-ed vector (e.g., FP16 vector). SoftMax circuit 1300 may include ROM 1302 storing a look up table comprising one or more precomputed values of an exponent X function: (x) = e^. SoftMax circuit 1300 may include ROM 1304 storing a look up table comprising one or more precomputed values of a reciprocal function: SoftMax circuit 1300may include tree adder 1306 to add a number of values (e.g., 18 values) together simultaneously.Maximizing floating-point range

[0145] According to one aspect, the models-on-silicon chip maximizes floating-point range. The chip may implement predefined floating-point tables and ranges that do not have Inf (infinity) nor NaN (not a number) numbers. The predefined tables and ranges can be used because the data into each module is controlled, which enables a non-overflow process, and enables maximizing the range of numbers.Embedder circuit

[0146] FIG. 14 illustrates embedder circuit 1400, according to some embodiments of the disclosure. A models-on-silicon chip includes a hardware implementation to produce an embedding vector (e.g., 4096 FP16 elements) of the input token. Embedder circuit 1400 can return 256 elementsDOCKET NO.: AG2834-US-PCTevery clock cycle for 16 clocks cycles. As depicted, embedder circuit 1400 may include a number of ROMs to store look up tables. The example shown includes 256 ROMs storing 256 look up tables.Embedder circuit 1400 can have the following exemplary specification:DOCKET NO.: AG2834-US-PCTRMS normalizer circuit

[0147] FIG. 15 illustrates RMS normalizer circuit 1500, according to some embodiments of the disclosure. The models-on-silicon chip implements a hardware implementation of an RMS normalizer function:+ 10 4,096

[0148] RMS normalizer circuit 1500 can receive an input vector (e.g., 4096 FP16 elements) and return an RMS-normalized vector (e.g., 4096 elements in FP8 format). RMS normalizer circuit 1500 can receive 256 elements every clock for 16 clocks cycles. RMS normalizer circuit 1500 can have the following exemplary specification:DOCKET NO.: AG2834-US-PCT

[0149] RMS normalizer circuit 1500 may include tree adder 1502 to add a number of values (e.g., 256 values) together simultaneously. RMS normalizer circuit 1500 may include ROM 1504 storing a look up table comprising one or more precomputed values of the function: (x) =Sampler circuit

[0150] FIG. 16 illustrates sampler circuit 1600, according to some embodiments of the disclosure. FIG. 17 illustrates sampling comparator circuit 1602 that can be implemented in sampler circuit 1600, according to some embodiments of the disclosure. According to one aspect, the models- on-silicon chip implements a hardware implementation of a sampler to return a token (e.g., an index, such as a 32-bit index) corresponding to the largest number in an input vector (e.g., 32,000 elements input vector having logits). Sampler circuit 1600 may implement a deterministic sampler having zero temperature. Sampler circuit 1600 may have the following exemplary specification:DOCKET NO.: AG2834-US-PCT

[0151] Sampling comparator circuit 1602 may have the following exemplary specification:DOCKET NO.: AG2834-US-PCT

[0152] The models-on-silicon chip may include sampler circuit 1600 to return a token of the largest number in an input vector (e.g., the index in the input vector corresponding to the largest value the input vector).

[0153] In some embodiments, sampler circuit 1600 includes a tree comparator circuit having many layers of instances of sampling comparator circuit 1602 arranged in a tree structure or hierarchical structure to efficiently compare a large number of values (e.g., hundreds or thousands of values or more) simultaneously.Rotary embedder circuit

[0154] FIG. 18A illustrates a rotary positional encoding (RoPE) circuit 1800, according to some embodiments of the disclosure. FIG. 18B illustrates a cosine function and a sine function, according to some embodiments of the disclosure. The models-on-silicon chip implements a hardware implementation of a rotary positional encoder to produce rotary positional encoded embeddings. Circuit 1800 is implemented to provide the functionality of a sine cosine unit without the need to calculate / compute sine and cosine in real-time. The sine cosine unit has a look up table implementation. Rotary positional encoding circuit 1800 may include ROM 1802 to store a look up table hn comprising one or more precomputed values of a cosine function (e.g., (t) = cos (10 « ■ t ). Rotary positional encoding circuit 1800 may include ROM 1804 to store a look up table comprising one hn or more precomputed values of sine function (e.g., (t) = sin (10 « ■ t)).Scaling the models-on-silicon architecture

[0155] In some embodiments, an apparatus can include a processing circuit implementing an application (e.g., a user application) and can receive input data and generate one or more input tokens. The apparatus can further include an inferencing circuit, such as a models-on-silicon chip as describedDOCKET NO.: AG2834-US-PCTherein. The inferencing circuit can receive the one or more input tokens and output one or more output tokens. In some embodiments, the processing circuit receives one or more output tokens generated by the inferencing circuit.

[0156] The models-on-silicon architecture is modular and can be scaled to implement larger transformer-based neural networks.

[0157] FIG. 19A illustrates using multiple chips to implement a large transformer model, according to some embodiments of the disclosure. FIG. 19B illustrates using multiple chips to implement a large transformer model, according to some embodiments of the disclosure. According to one aspect, models-on-silicon architecture enables scaling through multi-chip implementation. To implement huge models such as models with more than 1 trillion parameters, multiple instances of the models-on-silicon chips can be arranged together in the various manners illustrated in FIGS. 19A-B. For example, transformer output of 4096 vectors of one chip can be passed using a general-purpose input / output (GPIO) output to another chip, and so on. Many chips can be coupled together to form a larger transformer model architecture and scale as needed.

[0158] Referring to FIG. 19A, multiple models-on-silicon chips can be stacked, where chip 1902 may embed one subset of transformers, e.g., transformers 1-16, of a transformer-based neural network, and chip 1904 can embed a further subset of transformers, e.g., transforms 17-32, of the transformer-based neural network. Chip 1904 (e.g., a further inferencing circuit) can receive the one or more output tokens from chip 1902 (e.g., the inferencing circuit) and output one or more further output tokens. The one or more further output tokens can be fed back as input to chip 1902 in an autoregressive manner.

[0159] Referring to FIG. 19B, multiple models-on-silicon chips can be parallelized (e.g., implementing tensor parallelism), where chip 1906 may perform processing of a subset of embedding values, e.g., embedding values 1-2048, of embedding vector having 4096 elements, and chip 1908 may perform processing of a further subset of embedding values, e.g., embedding values 2049-4096, of embedding vector having 4096 elements.Hardware-based inferencing process

[0160] FIG. 20 illustrates hardware-based inferencing process with embedded LLM and ROM, according to some embodiments of the disclosure. According to one aspect, the process of using the models-on-silicon chip to implement a model such as a transformer model is different from the traditional inferencing process involving a GPU.

[0161] The process of using the models-on-silicon chip 100 begins in 2002 with user 2082 providing input data for inferencing. User 2082 may provide input data to application processor 2084 (sometimes referred to as a host processor) implementing a user application.DOCKET NO.: AG2834-US-PCT

[0162] In 2004, application processor 2084 may tokenize the input data and transform the input data into tokenized embeddings.

[0163] In 2006, the tokenized embeddings are passed onto models-on-silicon chip 100. In some embodiments, the input data as one or more tokens can be loaded into models-on-silicon chip 100 as a vector of tokens, or a vector of token embeddings.

[0164] Unlike traditional setups using GPUs, the model and its weights are already embedded in the ROM of models-on-silicon chip 100. The step of loading models or weights from external sources is eliminated.

[0165] In 2008, the models-on-silicon chip 100 performs inference and executes a transformer-based neural network. The tokenized embeddings are processed by models-on-silicon, using the weights of the model, which are read directly from the embedded ROM (e.g., SRO memory). This means that the information used for the inferencing process is available on models-on-silicon chip 100 itself, leading to faster data retrieval and processing. The information is retrieved from the ROM, and it is moved to one or more circuits for processing and execution. The one or more circuits are coupled to form a feedforward network within models-on-silicon chip 100. The feedforward network handles the inferencing computations and operations and is orchestrated by a sequencer circuit to perform operations according to a timing sequence to generate one or more output tokens. The models-on-silicon chip 100 computes the output token. If a next output token is to be generated, the output token can be fed back to models-on-silicon chip 100 as an input to generate a next output token in an auto-regressive manner.

[0166] In 2010, after processing, one or more output tokens are directed back to the application processor 2084.

[0167] Notably, the input and output interfaces of models-on-silicon (interfacing with application processor 2084) are very low bandwidth interfaces. Since the (entire) inference model architecture and weights are embedded in the SoC, the only data being input and output are tokens. Usually, each token is the size of 2 Bytes (based on the vocabulary size).

[0168] In 2012, the application processor 2084 may process the one or more output tokens and generate user output representing the inferencing result back to user 2082.

[0169] This approach of embedding the model and its weights in the hardware models-on- silicon chip 100 significantly streamlines the inferencing process, reducing latency and increasing efficiency, as it eliminates the need for external memory and data transfer. By hardcoding or etching the weights and model onto models-on-silicon chip 100 itself, it eliminates the need to load these weights from random-access memory for each task, thereby reducing power consumption and improvingDOCKET NO.: AG2834-US-PCTprocessing speed. The design of models-on-silicon chip 100 enables it to handle the complex calculations for machine learning inferencing tasks in real-time applications.Enhanced matrix multiplication operations

[0170] In some embodiments, the models-on-silicon chip 100 implements Embedded Weights and models Fused Multiply-Add Architecture (EWFMAA) to perform matrix multiplication operations. This architecture can be designed specifically to perform Fused Multiply-Add (FMA) operations with embedded weights and models, significantly enhancing the efficiency of matrix operations in machine learning tasks.

[0171] The solution may implement a series of cores, each providing a matrix processing array which performs the operation D = A*B- ^, where A, B, C and D are FP16 matrices. The operation is illustrated in FIG. 21 . A feature of this architecture is that the weight matrix B is hardcoded directly onto the chip, eliminating the need to load these weights from external random-access memory for each inference task.

[0172] Exemplary logic for implementing EWFMAA is illustrated in FIG. 22. The flow of operations within the EWFMAA is as follows: (1) the hardcoded weights are retrieved, (2) the input data matrix A & B for the inference task are loaded, (3) each core having multiplier 2202 and adder 2204 performs the FMA operation D = A*B- ^, where D is FP16 matrix, and C is an accumulator, (4) process continues until the dot operation is complete.

[0173] The architecture with its embedded weights, model and optimized transformer operations such as FMA operations, normalization, activation and SoftMax provides a highly efficient and powerful solution for inference tasks. It significantly reduces power consumption and enhances processing speed, making it ideal for applications demanding real-time inference and low-power consumption.Agent chip as orchestrator of specialized model-on-chips

[0174] FIG. 23A illustrates agent chip 2304 orchestrating multiple expert chips, according to some embodiments of the disclosure. Electronic system 2300 includes a plurality of expert chips and agent chip 2304 orchestrating the plurality of expert chips.

[0175] In some embodiments, the plurality of expert chips include a plurality of models-on- silicon chips (realized by the architecture illustrated in FIGS. 1-22). The expert chips can perform a variety of specialized tasks and are equipped with transformer-based neural networks and parameters that are trained to perform those specialized tasks.

[0176] For example, an expert chip in the plurality of expert chips can include a transformerbased neural network embedded or etched on-chip and a sequential read memory storing one or more parameters of the transformer-based neural network. A further expert chip in the plurality of expertDOCKET NO.: AG2834-US-PCTchips can include a further transformer-based neural network embedded on-chip and a further sequential read memory storing one or more further parameters of the one or more further parameters of the further transformer-based neural network.

[0177] Following the models-on-silicon architecture, the sequential read memory of the expert chip can be a read-only memory, and the further sequential read memory of the further expert chip can be a read-only memory. Also following the models-on-silicon architecture, the expert chip can include a predefined matrix multiplier to perform vector dot product operations between a vector having values of a predetermined precision and a further vector having further values of a further predetermined precision, and the further expert chip can include a further predefined matrix multiplier to perform vector dot product operations between a vector having values of a predetermined precision and a further vector having further values of a further predetermined precision.

[0178] In various examples and embodiments herein, these expert chips are referred to as expert A chip, expert B chip, expert C chip, {specialized task} chip, {specialized role} chip, and so forth, to indicate that the expert chip is tailored to and / or specializes in performing a specialized task or role. In the illustration of electronic system 2300, the plurality of expert chips as illustrated include one or more expert A chips 2306, expert B chip 2308, and expert C chip 2310.

[0179] Agent chip 2304 is communicably coupled to individual expert chips. In some embodiments, the agent chip communicates with the individual expert chips via inter-processor communication. In some embodiments, the agent chip communicates with the individual expert chips via GPIO connections. In some embodiments, the agent chip communications with the individual expert chips via networked communication (e.g., wired communication, wireless communication, or a combination of wired and wireless communications).

[0180] In some cases, two or more expert chips may be communicably coupled to each other according to a suitable network topology to facilitate expert chip to expert chip communication. The two or more expert chips and potentially the agent chip may communicate with each other through a mesh network. In a mesh network, each chip is connected to multiple other chips, creating a web-like structure. The two or more expert chips and potentially the agent chip may communicate with each other through a (shared) communication bus. In a bus network, the chips are connected to a single central cable, known as the bus. Data sent from any chip is available to all other chips on the network, but only the intended recipient processes the data. The two or more expert chips and potentially the agent chip may communicate with each other through a star network. In a star network, all chips are connected to a central hub or switch, such as the agent chip. The hub (e.g., the agent chip) can act as a repeater or router for data flow, improving the performance and reliability of the network. The two or more expert chips and potentially the agent chip may communicate with each other through aDOCKET NO.: AG2834-US-PCTbroadcast network. The two or more expert chips and potentially the agent chip may communicate with each other through a ring network. In a ring network, each chip is connected to two other chips, forming a circular data path. The two or more expert chips and potentially the agent chip may communicate with each other through a tree network. A tree network is a hybrid model that combines characteristics of bus and star networks. The tree network can include groups of star-configured networks connected to a linear bus backbone. The two or more expert chips and potentially the agent chip may communicate with each other through a hybrid network combining one or more different types of network topology.

[0181] For electronic system 2300, the plurality of expert chips can include a plurality of expert chips specializing in the same task (e.g., a plurality of expert A chips 2306). Having more than one expert chips specializing in the same task can mean that different inputs can be processed in parallel by the expert chips to increase throughput of electronic system 2300.

[0182] For electronic system 2300, the plurality of expert chips can include a plurality of expert chips specializing in different / diverse tasks (e.g., expert A chip 2306, expert B chip 2308, and expert C chip 2310). Having more than one expert chips specializing in different tasks can mean that electronic system 2300 can perform various subtasks to accomplish complex multi-task machine learning inference.

[0183] In some embodiments, the one or more parameters of the transformer-based neural network embedded / etched onto the expert chip are determined through training the transformer-based neural network to perform a task, and the one or more further parameters of the further transformerbased neural network embedded / etched onto the further expert chip are determined through training the further transformer-based neural network to perform a further task. In one example, the task is different from the further task (e.g., tasks performed by one or more expert A chips 2306, expert B chip 2308, and expert C chip 2310 are different from each other). In one example, the task and the further task are the same (e.g., tasks performed by expert A Chips 2306 are the same). In some embodiments, the task and the further task include one or more of: document retrieval-augmented generation, question and answering, classification, text-to-speech, speech-to-text, and summarization. In some embodiments, the task and the further task include one or more of: planning and requirements analysis, architecture design, project coordination, coding, and testing. A more detailed example is illustrated in FIGS. 25-26.

[0184] Agent chip 2304 implements a router neural network model (e.g., an example implementation is illustrated in FIG. 24). The router neural network model to route, according to one or more yet further parameters of the router neural network model, one or more embeddings to one or more of the expert chip and the further expert chip. The router neural network model can receive one or more output embeddings from the expert chip and forward the one or more output embeddings to theDOCKET NO.: AG2834-US-PCTfurther expert chip based on the one or more yet further parameters of the router neural network model. The one or more yet further parameters of the router neural network model are determined through training the agent chip and the plurality of expert chips (e.g., the expert chip, and the further expert chip) coupled together as a system. The one or more yet further parameters of the router neural network model are determined through training the router neural network model and one or more transformer-based neural network models embedded on the one or more expert chips coupled together as a system. The one or more yet further parameters of the router neural network can learn, through training the system as a whole, to determine which expert chip(s) are most suited to perform the next specialized task to accomplish multi-task machine learning inference. The one or more yet further parameters enable the router neural network model to intelligently select and activate one or more expert chips most suited to process a given set of tokens and / or a given set of embeddings.

[0185] In some embodiments, electronic system 2300 can include one or more integrated circuits and / or one or more processors that do not necessarily adopt the models-on-silicon architecture. The one or more integrated circuits and / or the one or more processors may be included to assist an expert chip to perform the specialized task. For example, expert C chip 2310 may, as part of performing a specialized task or role, utilize the transformer-based neural network embedded / etched thereon to generate one or more actions to be executed by processor 2312. In some cases, expert C chip 2310 may, as part of performing a specialized task or role, utilize the transformer-based neural network embedded / etched thereon to evaluate the result of the one or more actions executed by processor 2312.

[0186] One exemplary use case of electronic system 2300 having the multi-chip architecture may include an Al operating system (OS). The use case is illustrated as electronic system 2390 of FIG. 23B. Agent chip 2304 can direct tasks (e.g., in the form of token(s) and / or embedding(s)) to specialized models-on-silicon expert chips, each trained and / or fine-tuned for specific functions like document retrieval, question answering, classification, audio processing, summarization, and translation. Agent chip 2304 selects the appropriate expert chip with the fitting model embedded on the chip for the task, ensuring efficient and specialized processing with low latency and power consumption.

[0187] In one implementation, agent chip 2304 can receive tokens from a host machine or application processor, shown as host 2320 in FIG. 23B. The tokens can be generated by host 2320 based on user input. Agent chip 2304 can direct tasks to specialized models-on-silicon expert chips, e.g., document retrieval-augmented generation (RAG) expert LLM chip 2330, question answering expert LLM chip 2332, classification expert LLM chip 2334, text-to-speech / speech-to-text LLM chip 2336, summarization expert LLM chip 2338 and translation expert LLM chip 2340. An expert chip can be fine-tuned for a specific function or computing task. Exemplary functions or computing tasks include:DOCKET NO.: AG2834-US-PCT• Document RAG expert LLM chip 2330 for document RAG, which combines a pretrained LLM with an external data retrieval system to generate enhanced responses by incorporating information from relevant documents outside the model's original training data• Question answering expert LLM chip 2332 for answering queries or questions about a topic and producing an answer• Classification expert LLM chip 2336 for categorizing or classifying data.• Text-to-speech / speech-to-text LLM chip 2338 for audio-to-text or text-to-audio processing or conversion• Summarization expert LLM chip 2338 for condensing information and generating a summary• Translation expert LLM chip 2340 for language translation between different languages

[0188] Agent chip 2304 selects the appropriate models-on-silicon expert chip for the task at hand, sends the request (e.g., in the form of token(s) and / or embedding(s)), and receives the computed answer. Agent chip 2304 can relay the computed answer back to host 2320, which then outputs the answer to the user. This solution as seen in electronic system 2390 enables efficient and specialized processing for a wide range of tasks, and can have very low latency and power.Implementing a router neural network model on the agent chip

[0189] FIG. 24 illustrates an implementation of agent chip 2304 having a router neural network model, according to some embodiments of the disclosure. Electronic system 2400 includes a plurality of expert chips (e.g., expert A chip 2306, expert B chip 2308, and expert C chip 2310) and agent chip 2304 orchestrating the plurality of expert chips. The router neural network model of agent chip 2304 can be tasked to determine which one or more of the expert chips to route or forward tokens and / or embeddings to accomplish complex multi-task machine learning inference.

[0190] In some embodiments, one or more input tokens may be provided to agent chip 2304 as tokens in 2410. Agent chip 2304 may output one or more output tokens as tokens out 2412. In some embodiments, one or more input tokens may be provided to expert A chip 2306 as tokens in 2410. Expert A chip 2306 (or a different expert chip) may output one or more output tokens as tokens out 2412.

[0191] Expert A chip 2306 may receive one or more embeddings 2472 from agent chip 2304. Expert A chip 2306 may output embeddings 2474 to agent chip 2304. Expert B chip 2308 may receive one or more embeddings 2462 from agent chip 2304. Expert B chip 2308 may output embeddings 2464 to agent chip 2304. Expert C chip 2310 may receive one or more embeddings 2452 from agent chip 2304. Expert C chip 2310 may output embeddings 2454 to agent chip 2304.DOCKET NO.: AG2834-US-PCT

[0192] In some embodiments, the router neural network model includes one or more neural network layers 2402. The one or more neural network layers 2402 can have one or more trainable parameters, such as weights, that impact the results produced by one or more neural network layers 2402. The trainable parameters can be stored in weights memory 2440. One or more neural network layers 2402 can process one or more input tokens and / or one or more embeddings and produce one or more outputs according to one or more parameters of one or more neural network layers 2402. As discussed with FIGS. 23A-B, the trainable parameters can be determined by training the agent chip with the expert chips coupled together as a whole.

[0193] In some embodiments, the router neural network model includes SoftMax operator 2404 to compute one or more scores or probabilities based on the one or more outputs produced by one or more neural network layers 2402.

[0194] In some embodiments, the router neural network model includes top-K selection operator 2406 to select one or more expert chips among the plurality of expert chips. The selection can be performed based on the one or more scores or probabilities computed by SoftMax operator 2404. Based on the selected expert chip(s), agent chip 2304 can route tokens and / or embeddings to the selected expert chip(s) for further processing.

[0195] The illustrated router neural network model represents one exemplary implementation. One or more neural network layers 2402 can include one or more linear projection layers. One or more neural network layers 2402 can include multi-player perceptron model with one or more hidden layers and a non-linear activation layer. One or more neural network layers 2402 can include an attention mechanism where the input to the router neural network model is used as a query and each expert chip has a learned key vector. The attention mechanism may compute attention scores between the query and respective key vector to measure the affinity (or suitability) of each vector. The result can be scaled according to the value vector (for context). One or more neural network layers 2402 can include one or more linear projection layers and an attention mechanism, and the outputs can be combined through a weighted sum (where weights can be fixed or learned).

[0196] In some embodiments, agent chip 2304 adopts one or more components of the models-on-silicon architecture as illustrated in FIGS. 1-22. Agent chip 2304 includes an integrated circuit (e.g., a chip or a die). The integrated circuit can include a sequential read memory, one or more hardware circuits, a SoftMax circuit, and a top-K selection circuit.

[0197] The sequential read memory can store one or more parameters of the router neural network model. For example, weights memory 2440 can include a sequential read memory, where the weights stored on the sequential read memory or organized such that weights can be read one word line at a time and provided to one or more operations being performed by one or more neural networkDOCKET NO.: AG2834-US-PCTlayers 2402 for a given period or cycle. In some cases, the sequential read memory is read-only. Working with SRO memory is illustrated in FIG. 5. The one or more parameters of the router neural network model can be determined through training agent chip 2304 and the expert chips coupled together as a system. The one or more parameters of the router neural network model can be determined through training the router neural network model and the specialized transformer-based neural networks coupled together as a system.

[0198] The one or more hardware circuits can be provided to carry out operations of one or more neural network layers 2402. In other words, the operations of one or more neural network layers 2402 can be executed by the hardware circuits. The hardware circuits can process one or more input embeddings using the one or more parameters read from the sequential read memory. In cases where one or more neural network layers 2402 may implement an attention mechanism, the hardware circuits can include a predefined matrix multiplier to perform vector dot product operations between a vector having values of a predetermined precision and a further vector having further values of a further predetermined precision. In those cases, the hardware circuits can include one or more components illustrated as part of one or more etched mind units 110 of FIGS. 1-22 (e.g., efficient multipliers, look up tables, tree adders, dot units, etc.).

[0199] The SoftMax circuit to process one or more outputs from the one or more hardware circuits. In some implementations, the SoftMax circuit includes a look up table having precalculated values of a SoftMax function. The SoftMax circuit is illustrated as one or more SoftMax circuits 118 herein (e.g., SoftMax circuit 1300 of FIG. 13).

[0200] The top-K selection circuit can select one or more expert chips among the plurality of expert chips. The plurality of expert chips can include an expert chip having a transformer-based neural network etched on-chip and a further expert chip having a further transformer-based neural network etched on-chip based on one or more further outputs from the SoftMax circuit. In some implementations, the top-K selection circuit can implement a sampler circuit to efficiently select top-K scores and / or probabilities, e.g., as illustrated as sampler circuit 1600 of FIG. 16.Software development use case of agent chip in multi-chip architecture

[0201] One exemplary use case may include software development using multi-chip architecture. Software development can include many different specialized tasks and / or roles which can be performed by LLMs. This use case transforms software development by carrying out technical tasks with specialized models-on-silicon expert chips. Each models-on-silicon chip is trained and / or finetuned for tasks such as planning, designing architecture, project coordination, coding, and testing. Tokens and / or embeddings can move between these specialized chips, emulating a software development workflow, enabling collaborative and efficient software creation.DOCKET NO.: AG2834-US-PCT

[0202] FIG. 25 illustrates system 2500 having expert chips and agent chip 2304 orchestrating machine learning tasks being performed by the expert chips in a software development scenario, according to some embodiments of the disclosure. FIG. 26 illustrates a data flow of system 2500 illustrated in FIG. 25, according to some embodiments of the disclosure. Software development can be carried out by the expert chips realized using the models-on-silicon architecture as illustrated by FIGS. 1-22.

[0203] Before embedding an expert model onto silicon, the expert model can be trained and / or fine-tuned with a dedicated task. In this example, the expert chips include product manager chip 2502, architect chip 2504, project manager chip 2506, software engineer chip 2508, and quality assurance engineer chip 2510. Different models can be trained and / or fine-tuned on selected training data that fits the task assigned to be performed by the model. The model, once trained and / or finetuned for the specialized task, can be embedded and / or etched onto a models-on-silicon chip (as illustrated by FIGS. 1-22).

[0204] The transformer-based neural network model embedded onto product manager chip 2502 can be pre-trained and / or fine-tuned to perform a product management task, involving taking an input prompt describing a software product (e.g., prompt 2580 from user 2582) and defining a requirement document having one or more product requirements. The project management task can involve planning and requirements analysis.

[0205] The transformer-based neural network model embedded onto architect chip 2504 can be pre-trained and / or fine-tuned to perform software architect role, involving taking a requirement documents (e.g., from product manager chip 2502) and creating a system design. The software architect role can involve architecture design.

[0206] The transformer-based neural network model embedded onto project manager chip 2506 can be pre-trained and / or fine-tuned to perform a project management role, involving taking a system design (e.g., from architect chip 2504), planning one or more tasks, and executing the one or more tasks. The project management role can involve project coordination.

[0207] The transformer-based neural network model embedded onto software engineer chip 2508 can be pre-trained and / or fine-tuned to perform a software engineer role, involving taking a task (e.g., from project manager chip 2506), and generating code for the task. In some cases, the software engineer role can involve compiling the code. In some cases, the software engineer role can involve building a software package for the code. In some embodiments, software engineer chip 2508 can be communicably coupled to one or more processors 2566 to perform compilation and / or create software builds. In some cases, the software engineer role can involve employing coding standards and bestDOCKET NO.: AG2834-US-PCTpractices to write high-quality code, leveraging one or more pre-trained models on various programming languages.

[0208] The transformer-based neural network model embedded onto quality assurance engineer chip 2510 can be pre-trained and / or fine-tuned to perform a QA engineer role, involving taking code (or compiled executable, or software package) (e.g., from software engineer chip 2508), and testing the code. In some cases, the QA engineer role can involve testing the code, the compiled executable, or the software package. In some embodiments, quality assurance engineer chip 2510 can be communicably coupled to one or more processors 2568 to implement a computing environment for execute and test the code, the compiled executable, or the software package. In some cases, the QA engineer role can use automated testing frameworks and tools (e.g., implemented on one or more processors 2568) to perform thorough testing, define unit tests, and identify defects to ensure quality.

[0209] In some cases, a plurality of instances of architect chip 2504 can be included as part of system 2500 to perform separate system design tasks for different requirements specified by product manager chip 2502. Agent chip 2304 can intelligently partition the requirement document and route them to the plurality of instances of architect chip 2504.

[0210] In some cases, a plurality of instances of project manager chip 2506 can be included as part of system 2500 to perform separate project management tasks for different subsystems of the system design specified by architect chip 2504. Agent chip 2304 can intelligently partition the system design and route subsystems to the plurality of instances of project manager chip 2506.

[0211] In some cases, a plurality of instances of software engineer chip 2508 can be included as part of system 2500 to perform separate coding tasks for different tasks or parts of the project defined by project manager chip 2506. Agent chip 2304 can intelligently partition the project and route tasks and / or parts to the plurality of instances of software engineer chip 2508. The code, compiled executable, or software package generated by an instance of software engineer chip 2508 may be forwarded to a corresponding instance of quality assurance engineer chip 2510 for testing.

[0212] Tokens and / or embeddings can move in and out between these specialized expert chips, emulating a software development workflow. The data flow between the specialized expert chips can be orchestrated by agent chip 2304. The expert chips can collaboratively create new software, leveraging their specialized capabilities to complete the development lifecycle efficiently.

[0213] In some cases, the interaction between software engineer chip 2508 and quality assurance engineer chip 2510 can iterate until all tests are passed or a certain passing rate is reached. In some cases, the interaction between project manager chip 2506 and software engineer chip 2508 can iterate until all tasks are completed. In some cases, the interaction between architect chip 2504 and project manager chip 2506 can iterate until all subsystems of the system design are completed. InDOCKET NO.: AG2834-US-PCTsome cases, the interaction between product manager chip 2502 and architect chip 2504 can iterate until all product requirements of the requirement document are met. The iteration process can be managed and orchestrated by agent chip 2304.

[0214] One of the chips in system 2500, e.g., product manager chip 2502, quality assurance engineer chip 2510, or agent chip 2304 can output product 2584 (e.g., a tested software build) as a final result produced by system 2500.

[0215] The use case, as illustrated by system 2500 of FIG. 25, can enable collaborative creation. The collaborative creation process in a multi-model-on-chip architecture involves seamless interaction between various specialized chips. Each models-on-silicon chip is dedicated to a specific task within the software development lifecycle. Agent chip 2304 can orchestrate this collaboration by selecting and routing tasks to the appropriate specialized chip based on the input task. Agent chip 2304 can orchestrate iteration processes between the chips.

[0216] Tokens and / or embeddings, representing pieces of data or instructions, move between the chips in a sequence that emulates or simulates the software development workflow. Communications between chips can be facilitated by inter-chip communication or networked communications. After completing the specialized task, an expert chip can send its output back to the agent chip 2304. Agent chip 2304 can determine the next appropriate chip(s) and forwards the data accordingly. For instance, the product manager chip 2502 generates a requirements document and sends it to agent chip 2304, which then routes it to architect chip 2504 for designing the system architecture. Some tasks may be processed in parallel or in a distributed fashion to improve efficiency, such as simultaneous coding by multiple instances of software engineer chip 2508 chips for different modules or subsystems of the system architecture. The expert chips can iterate over their tasks, incorporating internal and / or external feedback and making adjustments as necessary to refine the output. Each expert chip is embedded or etched with a model that has been pre-trained and fine-tuned for a specific role or task within the software development process. Leveraging these specialized capabilities involves optimizing a model for its specific function, ensuring high efficiency and accuracy in performing its designated role or task. The models which are embedded or etched on the chips are pre-trained on domain-specific data, providing them with the expertise needed for their tasks. For example, the quality assurance engineer chip 2510 is trained on extensive testing data to identify and fix bugs effectively. The algorithms within each model can be tailored to perform the tasks associated with their roles. For instance, the architect chip 2504 can use design patterns and principles to create robust software architectures.DOCKET NO.: AG2834-US-PCT

[0217] Agent chip 2304 can serve as the central orchestrator of the multi-chip architecture. It is pre-trained with routing neural network model that determines which specialized models-on-silicon chip is best suited to perform the next task.

[0218] Referring to FIG. 26, user 2582 may input a prompt to product manager chip 2502 in 2602, and product manager chip 2502 may utilize market analysis data and user feedback to create detailed product requirements and plans based on the prompt. In 2604, the output generated by product manager chip 2502 is forwarded to agent chip 2304.

[0219] In 2606, agent chip 2304 routes the output to architect chip 2504, and architect chip 2504 may define a system design. In 2608, the output generated by architect chip 2504 is forwarded to agent chip 2304.

[0220] In 2610, agent chip 2304 routes the output to project manager chip 2506, and project manager chip 2506 may define a plan and tasks. In 2612, the output generated by project manager chip 2506 is forwarded to agent chip 2304.

[0221] In 2614, agent chip 2304 routes the output to software engineer chip 2508, and software engineer chip 2508 may write code. In 2616, the output generated by software engineer chip 2508 is forwarded to agent chip 2304.

[0222] In 2618, agent chip 2304 routes the output to quality assurance engineer chip 2510, and quality assurance engineer chip 2510 may perform testing of the code. In 2620, the output generated by quality assurance engineer chip 2510 is forwarded to agent chip 2304.

[0223] In 2622, agent chip 2304 may determine that revision to the code is desired based on the output produced by quality assurance engineer chip 2510 and routes the output to software engineer chip 2508 to revise the code accordingly. Software engineer chip 2508 may generate revised code. In 2624, the revised code generated by software engineer chip 2508 is forwarded to agent chip 2304. In 2626, agent chip 2304 routes the output (having the revised code) to quality assurance engineer chip 2510, and quality assurance engineer chip 2510 may perform testing of the revised code. In 2628, the output generated by quality assurance engineer chip 2510 is forwarded to agent chip 2304. The output may indicate to agent chip 2304 that the next task to be performed is to determine if all project tasks are completed.

[0224] In 2630, agent chip 2304 routes the output to project manager chip 2506 to determine if project tasks are completed. In 2632, the output generated by project manager chip 2506 is forwarded to agent chip 2304. The output may indicate to agent chip 2304 that the next task to be performed is to determine if the system design is completed.

[0225] In 2634, agent chip 2304 routes the output to architect chip 2504 to determine if the system design is completed. In 2636, the output generated by project manager chip 2506 is forwardedDOCKET NO.: AG2834-US-PCTto agent chip 2304. The output may indicate to agent chip 2304 that the next task to be performed is to determine if the project requirements are met.

[0226] In 2638, agent chip 2304 routes the output to product manager chip 2502 to determine if the project requirements are met. In 2640, the output generated by product manager chip 2502 is forwarded to agent chip 2304.

[0227] FIG. 26 illustrates exemplary interactions and data flow between user 2582 and the plurality of expert chips, and it is envisioned that some interactions may be omitted and / or some interactions may be added.Methods for orchestrating expert chips

[0228] FIG. 27 illustrates method 2700 for orchestrating expert chips, according to some embodiments of the disclosure. Method 2700 can be performed by agent chip 2304 as described and illustrated herein.

[0229] In 2702, agent chip 2304 may receive one or more embeddings from an expert model. The expert model can be among a plurality of expert chips and having a sequential read memory to store one or more parameters used to generate the one or more embeddings.

[0230] In 2704, the one or more embeddings can be input into a router neural network model implemented on agent chip 2304.

[0231] In 2706, agent chip 2304 can select one or more expert chips among the plurality of expert chips to route the one or more embeddings according to one or more further parameters of the router neural network model.

[0232] In 2708, agent chip 2304 can output or route the one or more embeddings to the one or more selected expert chips.

[0233] In some embodiments, method 2700 may further include determining the one or more further parameters of the router neural network model through training the agent chip and the plurality of expert chips coupled together as a system.

[0234] In some embodiments, agent chip 2304 adopts the models-on-silicon architecture illustrated in FIGS. 1-22. The one or more further parameters of the router neural network can be retrieved from a sequential read memory. In some embodiments, the sequential read memory is readonly.

[0235] In some embodiments, agent chip 2304 can perform orchestration of the expert chips, or cause the expert chips to route their outputs to selected expert chips. Agent chip 2304 can receive one or more further embeddings generated by the one or more selected chips. The one or more further embeddings can be input into the router neural network model implemented on agent chip 2304. Agent chip 2304 can select one or more further expert chips among the plurality of expert chips to route theDOCKET NO.: AG2834-US-PCTone or more further embeddings according to the one or more further parameters of the router neural network model. Agent chip 2304 can output the one or more further embeddings to the one or more selected further expert chips.Exemplary computing device

[0236] FIG. 28 is a block diagram of an apparatus or a system, e.g., an exemplary computing device 2800, according to some embodiments of the disclosure. One or more computing devices 2800 may be used to implement the functionalities described with the FIGS, and herein. A number of components are illustrated in the FIG. 28. can be included in the computing device 2800, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 2800 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single SoC die. Additionally, in various embodiments, the computing device 2800 may not include one or more of the components illustrated in FIG. 28, and the computing device 2800 may include interface circuitry for coupling to the one or more components. For example, the computing device 2800 may not include a display device 2806, and may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 2806 may be coupled. In another set of examples, the computing device 2800 may not include an audio input device 2818 or an audio output device 2808 and may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which an audio input device 2818 or audio output device 2808 may be coupled.

[0237] Computing device 2800 may include a processing device 2802 (e.g., one or more processing devices, one or more of the same types of processing device, one or more of different types of processing device). Processing device 2802 may include electronic circuitry that process electronic data from data storage elements (e.g., registers, memory, resistors, capacitors, quantum bit cells) to transform that electronic data into other electronic data that may be stored in registers and / or memory. Examples of processing device 2802 may include a CPU, a GPU, a quantum processor, a machine learning processor, an artificial intelligence processor, a neural network processor, an artificial intelligence accelerator, an application specific integrated circuit (ASIC), an analog signal processor, an analog computer, a microprocessor, a digital signal processor, a FPGA, a TPU, a data processing unit (DPU), etc.

[0238] In some embodiments, computing device 2800 may include the multi-chip solution as described herein, having agent chip 2304 and a plurality of expert chips (depicted as instances of expert chip 2880). The multi-chip solution can interface with processing device 2802 to accelerate inference, in particular complex multi-task machine learning inference.DOCKET NO.: AG2834-US-PCT

[0239] In some embodiments, computing device 2800 may include a subset of the chips in the multi-chip solution. Computing device 2800 may include agent chip 2304, while another instance of computing device 2800 may include one or more expert chips 2880. The specialized chip(s) can accelerate inference, in particular complex multi-task machine learning inference, within a larger computing system having many instances of computing device 2800.

[0240] Computing device 2800 may include a memory 2804, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., ROM), HBM, flash memory, solid state memory, and / or a hard drive. Memory 2804 includes one or more non- transitory computer-readable storage media. In some embodiments, memory 2804 may include memory that shares a die with the processing device 2802.

[0241] In some embodiments, memory 2804 includes one or more non-transitory computer- readable media storing instructions executable to perform operations described with the FIGS, and herein. Memory 2804 may store instructions that generate inputs to the multi-chip solution. Memory 2804 may store instructions that process outputs from multi-chip solution. The instructions stored in the one or more non-transitory computer-readable media may be executed by processing device 2802.

[0242] In some embodiments, memory 2804 may store data, e.g., data structures, binary data, bits, metadata, files, blobs, etc., as described with the FIGS, and herein. Data may include inputs to the multi-chip solution (e.g., input tokens). Data may include outputs from the multi-chip solution (e.g., output tokens). Memory 2804 can store training data used to train the router neural network model and the expert models described and illustrated herein. Memory 2804 may store parameters which are updated through a training process and subsequently loaded or etched onto the chips of the multi-chip solution.

[0243] In some embodiments, the computing device 2800 may include a communication device 2812 (e.g., one or more communication devices). For example, the communication device 2812 may be configured for managing wired and / or wireless communications for the transfer of data to and from the computing device 2800. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not. The communication device 2812 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to asDOCKET NO.: AG2834-US-PCT"3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication device 2812 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High-Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. The communication device 2812 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). Communication device 2812 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication device 2812 may operate in accordance with other wireless protocols in other embodiments. The computing device 2800 may include an antenna 2822 to facilitate wireless communications and / or to receive other wireless communications (such as radio frequency transmissions). Computing device 2800 may include receiver circuits and / or transmitter circuits. In some embodiments, the communication device 2812 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication device 2812 may include multiple communication chips. For instance, a first communication device 2812 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication device 2812 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV- DO, or others. In some embodiments, a first communication device 2812 may be dedicated to wireless communications, and a second communication device 2812 may be dedicated to wired communications.

[0244] Computing device 2800 may include power source / power circuitry 2814. The power source / power circuitry 2814 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 2800 to an energy source separate from the computing device 2800 (e.g., DC power, AC power, etc.).

[0245] Computing device 2800 may include a display device 2806 (or corresponding interface circuitry, as discussed above). The display device 2806 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.DOCKET NO.: AG2834-US-PCT

[0246] Computing device 2800 may include an audio output device 2808 (or corresponding interface circuitry, as discussed above). The audio output device 2808 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0247] Computing device 2800 may include audio input device 2818 (or corresponding interface circuitry, as discussed above). The audio input device 2818 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0248] Computing device 2800 may include a GPS device 2816 (or corresponding interface circuitry, as discussed above). The GPS device 2816 may be in communication with a satellite-based system and may receive a location of the computing device 2800, as known in the art.

[0249] Computing device 2800 may include sensor 2830 (or one or more sensors). The computing device 2800 may include corresponding interface circuitry, as discussed above). Sensor 2830 may sense physical phenomenon and translate the physical phenomenon into electrical signals that can be processed by, e.g., processing device 2802. Examples of sensor 2830 may include: capacitive sensor, inductive sensor, resistive sensor, electromagnetic field sensor, light sensor, camera, imager, microphone, pressure sensor, temperature sensor, vibrational sensor, accelerometer, gyroscope, strain sensor, moisture sensor, humidity sensor, distance sensor, range sensor, time-of- flight sensor, pH sensor, particle sensor, air quality sensor, chemical sensor, gas sensor, biosensor, ultrasound sensor, a scanner, etc.

[0250] Computing device 2800 may include another output device 2810 (or corresponding interface circuitry, as discussed above). Examples of the other output device 2810 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, haptic output device, gas output device, vibrational output device, lighting output device, home automation controller, or an additional storage device.

[0251] Computing device 2800 may include another input device 2820 (or corresponding interface circuitry, as discussed above). Examples of the other input device 2820 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0252] Computing device 2800 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a netbook computer, a personal digital assistant (PDA), a personal computer, a remote control, wearable device, headgear, eyewear, footwear, electronic clothing, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, aDOCKET NO.: AG2834-US-PCTmonitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, an loT device, or a wearable computer system. In some embodiments, the computing device 2800 may be any other electronic device that processes data.Select examples

[0253] Example 1 provides an electronic system, including an expert chip having a transformer-based neural network embedded on-chip and a sequential read memory storing one or more parameters of the transformer-based neural network; a further expert chip having a further transformer-based neural network embedded on-chip and a further sequential read memory storing one or more further parameters of the one or more further parameters of the further transformer-based neural network; and an agent chip implementing a router neural network model, the router neural network model to route, according to one or more yet further parameters of the router neural network model, one or more embeddings to one or more of the expert chip and the further expert chip.

[0254] Example 2 provides the electronic system of example 1 , where the router neural network model is to: receive one or more output embeddings from the expert chip; and forward the one or more output embeddings to the further expert chip based on the one or more yet further parameters of the router neural network model.

[0255] Example 3 provides the electronic system of example 1 or 2, where the router neural network model includes one or more neural network layers.

[0256] Example 4 provides the electronic system of any one of examples 1 -3, where the router neural network model includes a SoftMax operator to compute one or more probabilities.

[0257] Example 5 provides the electronic system of any one of examples 1 -4, where the router neural network model includes a top-K selection operator to select one or more of the expert chip and the further expert chip.

[0258] Example 6 provides the electronic system of any one of examples 1 -5, where the agent chip includes a yet further sequential read memory storing the one or more yet further parameters of the router neural network model.

[0259] Example 7 provides the electronic system of example 6, where the yet further sequential read memory is read-only.

[0260] Example 8 provides the electronic system of any one of examples 1 -7, where the agent chip communicates with the expert chip and the further expert chip via inter-processor communication.

[0261] Example 9 provides the electronic system of any one of examples 1-7, where the agent chip communicates with the expert chip and the further expert chip via networked communication.

[0262] Example 10 provides the electronic system of any one of examples 1 -9, where the sequential read memory is read-only.DOCKET NO.: AG2834-US-PCT

[0263] Example 11 provides the electronic system of any one of examples 1-10, where the further sequential read memory is read-only.

[0264] Example 12 provides the electronic system of any one of examples 1-11 , where the expert chip includes a predefined matrix multiplier to perform vector dot product operations between a vector having values of a predetermined precision and a further vector having further values of a further predetermined precision.

[0265] Example 13 provides the electronic system of any one of examples 1-12, where the further expert chip includes a further predefined matrix multiplier to perform vector dot product operations between a vector having values of a predetermined precision and a further vector having further values of a further predetermined precision.

[0266] Example 14 provides the electronic system of any one of examples 1-13, where the one or more yet further parameters of the router neural network model are determined through training the router neural network model, the transformer-based neural network, and the further transformerbased neural network coupled together as a system.

[0267] Example 15 provides the electronic system of any one of examples 1-14, where: the one or more parameters of the transformer-based neural network are determined through training the transformer-based neural network to perform a task; and the one or more further parameters of the further transformer-based neural network are determined through training the further transformer-based neural network to perform a further task.

[0268] Example 16 provides the electronic system of example 15, where the task is different from the further task.

[0269] Example 17 provides the electronic system of example 15, where the task and the further task are the same.

[0270] Example 18 provides the electronic system of any one of examples 15-17, where the task and the further task include one or more of: document retrieval-augmented generation, question and answering, classification, text-to-speech, speech-to-text, and summarization.

[0271] Example 19 provides the electronic system of any one of examples 15-17, where the task and the further task include one or more of: planning and requirements analysis, architecture design, project coordination, coding, and testing.

[0272] Example 20 provides an integrated circuit, including a sequential read memory to store one or more parameters of a router neural network model; one or more hardware circuits to process one or more input embeddings using the one or more parameters read from the sequential read memory; a SoftMax circuit to process one or more outputs from the one or more hardware circuits; and a top-K selection circuit to select one or more of an expert chip having a transformer-based neuralDOCKET NO.: AG2834-US-PCTnetwork etched on-chip and a further expert chip having a further transformer-based neural network etched on-chip based on one or more further outputs from the SoftMax circuit.

[0273] Example 21 provides the integrated circuit of example 20, where the sequential read memory is read-only.

[0274] Example 22 provides the integrated circuit of example 20 or 21 , where the one or more parameters of the router neural network model are determined through training the router neural network model, the transformer-based neural network, and the further transformer-based neural network coupled together as a system.

[0275] Example 23 provides the integrated circuit of any one of examples 20-22, where the one or more hardware circuits include a predefined matrix multiplier to perform vector dot product operations between a vector having values of a predetermined precision and a further vector having further values of a further predetermined precision.

[0276] Example 24 provides the integrated circuit of any one of examples 20-23, where the SoftMax circuit includes a look up table having precalculated values of a SoftMax function.

[0277] Example 25 provides a method for orchestrating multi-task machine learning inference, including receiving one or more embeddings from an expert model, the expert model being among a plurality of expert chips and having a sequential read memory to store one or more parameters used to generate the one or more embeddings; inputting the one or more embeddings to a router neural network model implemented on an agent chip; selecting one or more expert chips among the plurality of expert chips to route the one or more embeddings according to one or more further parameters of the router neural network model; and outputting the one or more embeddings to the one or more selected expert chips.

[0278] Example 26 provides the method of example 25, further including determining the one or more further parameters of the router neural network model through training the router neural network model and transformer-based neural networks being embedded on respective expert chips coupled together as a system.

[0279] Example 27 provides the method of example 25 or 26, where the one or more further parameters of the router neural network model are retrieved from a sequential read memory.

[0280] Example 28 provides the method of example 27, where the sequential read memory is read-only.

[0281] Example 29 provides the method of any one of examples 25-28, further including receiving one or more further embeddings generated by the one or more selected expert chips; inputting the one or more further embeddings into the router neural network model implemented on the agent chip; selecting one or more further expert chips among the plurality of expert chips to route theDOCKET NO.: AG2834-US-PCTone or more further embeddings according to the one or more further parameters of the router neural network model; and outputting the one or more further embeddings to the one or more further selected expert chips.

[0282] Example 30 provides an electronic system, including an agent chip implementing a task management neural network model; and one or more expert chips communicating with the agent chip, the one or more expert chips including an expert chip having a transformer-based neural network model embedded on the expert chip and one or more parameters determined by training the transformer-based neural network model to perform a computing task; where the agent chip routes one or more tokens associated with the computing task to the expert chip and receives a result of the computing task from the expert chip.

[0283] Example 31 provides the electronic system of example 30, where the agent chip routes the one or more tokens associated with the computing task by: selecting the expert chip, based on, at least the one or more tokens associated with the computing task and one or more yet further parameters of the task management neural network model, from the one or more expert chips to route the one or more tokens; and sending the one or more tokens to the expert chip.

[0284] Example 32 provides the electronic system of example 31 , where selecting the expert chip includes applying one or more neural network layers on the one or more tokens; applying a SoftMax operator to compute one or more probabilities; and applying a top-K selection operator on the one or more probabilities to select the expert chip.

[0285] Example 33 provides the electronic system of any one of examples 30-32, where the one or more expert chips further include: a further expert chip having a further transformer-based neural network model embedded on the further expert chip and one or more further parameters determined by training the further transformer-based neural network model to perform a further computing task.

[0286] Example 34 provides the electronic system of example 33, where the computing task and the further computing task are the same.

[0287] Example 35 provides the electronic system of example 33, where the computing task is different from the further computing task.

[0288] Example 36 provides the electronic system of any one of examples 33-35, where the agent chip routes the result of the computing task to the further expert chip and receives a further result of the further computing task from the further expert chip.

[0289] Example 37 provides the electronic system of example 36, where the agent chip routes the result of the computing task by: selecting the further expert chip, based on, at least the result of the computing task and one or more yet further parameters of the task management neural network model,DOCKET NO.: AG2834-US-PCTfrom the one or more expert chips to route the result of the computing task; and sending the result of the computing task to the further expert chip.

[0290] Example 38 provides the electronic system of any one of examples 30-35, where one or more yet further parameters of the task management neural network model are determined through training the task management neural network model, the transformer-based neural network model, and one or more further transformer-based neural network models coupled together as a system.

[0291] Example 39 provides the electronic system of any one of examples 30-38, where the agent chip communicates with the expert chip via inter-processor communication.

[0292] Example 40 provides the electronic system of any one of examples 30-38, where the agent chip communicates with the expert chip via networked communication.

[0293] Example 41 provides an apparatus including means for performing a method according to any one of examples 25-29.Variations and other notes

[0294] Although the operations of the example method shown in and described with reference to some of the FIGS, are illustrated as occurring once each and in a particular order, it will be recognized that the operations may be performed in any suitable order and repeated as desired. Additionally, one or more operations may be performed in parallel. Furthermore, the operations illustrated in some of the FIGS, may be combined or may include more or fewer details than described.

[0295] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. These modifications may be made to the disclosure in light of the above detailed description.

[0296] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without the specific details and / or that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0297] Further, references are made to the accompanying drawings that form a part hereof, and in which are shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be madeDOCKET NO.: AG2834-US-PCTwithout departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0298] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the disclosed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0299] For the purposes of the present disclosure, the phrase “A or B” or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase ‘A, B, or C” or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0300] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives “first,” “second,” and “third,” etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0301] In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0302] The terms “substantially,” “close,” “approximately,” “near,” and “about,” generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0303] In addition, the terms “comprise,” “comprising,” “include,” “including,” “have,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, or device, that comprises a list of elements is not necessarily limited to only those elementsDOCKET NO.: AG2834-US-PCTbut may include other elements not expressly listed or inherent to such method, process, or device. Also, the term “or” refers to an inclusive “or” and not to an exclusive “or.”

[0304] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description and the accompanying drawings.DOCKET NO.: AG2834-US-PCT

Claims

Claims1 . An integrated circuit, comprising: a sequential read memory to store one or more parameters of a router neural network model; one or more hardware circuits to process one or more input embeddings using the one or more parameters read from the sequential read memory; a SoftMax circuit to process one or more outputs from the one or more hardware circuits; and a top-K selection circuit to select one or more of an expert chip having a transformer-based neural network etched on-chip and a further expert chip having a further transformer-based neural network etched on-chip based on one or more further outputs from the SoftMax circuit.

2. The integrated circuit of claim 1 , wherein the sequential read memory is read-only.

3. The integrated circuit of claim 1 or 2, wherein the one or more parameters of the router neural network model are determined through training the router neural network model, the transformer-based neural network, and the further transformer-based neural network coupled together as a system.

4. The integrated circuit of any one of claims 1 -3, wherein the one or more hardware circuits include a predefined matrix multiplier to perform vector dot product operations between a vector having values of a predetermined precision and a further vector having further values of a further predetermined precision.

5. The integrated circuit of any one of claims 1 -4, wherein the SoftMax circuit includes a look up table having precalculated values of a SoftMax function.

6. A method for orchestrating multi-task machine learning inference, comprising: receiving one or more embeddings from an expert model, the expert model being among a plurality of expert chips and having a sequential read memory to store one or more parameters used to generate the one or more embeddings; inputting the one or more embeddings into a router neural network model implemented on an agent chip; selecting one or more expert chips among the plurality of expert chips to route the one or more embeddings according to one or more further parameters of the router neural network model; and outputting the one or more embeddings to the one or more selected expert chips.DOCKET NO.: AG2834-US-PCT7. The method of claim 6, further comprising: determining the one or more further parameters of the router neural network model through training the router neural network model and transformer-based neural networks being embedded on respective expert chips coupled together as a system.

8. The method of claim 6 or 7, wherein the one or more further parameters of the router neural network model are retrieved from a sequential read memory.

9. The method of claim 8, wherein the sequential read memory is read-only.

10. The method of any one of claims 6-9, further comprising: receiving one or more further embeddings generated by the one or more selected expert chips; inputting the one or more further embeddings into the router neural network model implemented on the agent chip; selecting one or more further expert chips among the plurality of expert chips to route the one or more further embeddings according to the one or more further parameters of the router neural network model; and outputting the one or more further embeddings to the one or more further selected expert chips.

11. An electronic system, comprising: an agent chip implementing a task management neural network model; and one or more expert chips communicating with the agent chip, the one or more expert chips comprising an expert chip having a transformer-based neural network model embedded on the expert chip and one or more parameters determined by training the transformer-based neural network model to perform a computing task; wherein the agent chip routes one or more tokens associated with the computing task to the expert chip and receives a result of the computing task from the expert chip.

12. The electronic system of claim 11 , wherein the agent chip routes the one or more tokens associated with the computing task by: selecting the expert chip, based on, at least the one or more tokens associated with the computing task and one or more yet further parameters of the task management neural network model, from the one or more expert chips to route the one or more tokens; andDOCKET NO.: AG2834-US-PCTsending the one or more tokens to the expert chip.

13. The electronic system of claim 12, wherein selecting the expert chip comprises: applying one or more neural network layers on the one or more tokens; applying a SoftMax operator to compute one or more probabilities; and applying a top-K selection operator on the one or more probabilities to select the expert chip.

14. The electronic system of any one of claims 11-13, wherein the one or more expert chips further include: a further expert chip having a further transformer-based neural network model embedded on the further expert chip and one or more further parameters determined by training the further transformer-based neural network model to perform a further computing task.

15. The electronic system of claim 14, wherein the computing task and the further computing task are the same.

16. The electronic system of claim 14, wherein the computing task is different from the further computing task.

17. The electronic system of any one of claims 14-16, wherein the agent chip routes the result of the computing task to the further expert chip and receives a further result of the further computing task from the further expert chip.

18. The electronic system of claim 17, wherein the agent chip routes the result of the computing task by: selecting the further expert chip, based on, at least the result of the computing task and one or more yet further parameters of the task management neural network model, from the one or more expert chips to route the result of the computing task; and sending the result of the computing task to the further expert chip.

19. The electronic system of any one of claims 11-16, wherein one or more yet further parameters of the task management neural network model are determined through training the task management neural network model, the transformer-based neural network model, and one or more further transformer-based neural network models coupled together as a system.DOCKET NO.: AG2834-US-PCT20. The electronic system of any one of claims 11-19, wherein the agent chip communicates with the expert chip via inter-processor communication.21 . The electronic system of any one of claims 11-19, wherein the agent chip communicates with the expert chip via networked communication.

22. An electronic system, comprising: an expert chip having a transformer-based neural network embedded on-chip and a sequential read memory storing one or more parameters of the transformer-based neural network; a further expert chip having a further transformer-based neural network embedded on-chip and a further sequential read memory storing one or more further parameters of the one or more further parameters of the further transformer-based neural network; and an agent chip implementing a router neural network model, the router neural network model to route, according to one or more yet further parameters of the router neural network model, one or more embeddings to one or more of the expert chip and the further expert chip.

23. The electronic system of claim 22, wherein the router neural network model is to: receive one or more output embeddings from the expert chip; and forward the one or more output embeddings to the further expert chip based on the one or more yet further parameters of the router neural network model.

24. The electronic system of claim 22 or 23, wherein: the one or more parameters of the transformer-based neural network are determined through training the transformer-based neural network to perform a task; and the one or more further parameters of the further transformer-based neural network are determined through training the further transformer-based neural network to perform a further task.

25. The electronic system of claim 24, wherein the task is different from the further task.DOCKET NO.: AG2834-US-PCT

Images