A core particle computing system, device and application for an intelligent agent large language model

Abstract

This invention belongs to the field of intelligent agent large language model service technology, and discloses a chip computing system, device and application for intelligent agent large language model. Through the collaborative design of hardware architecture and software scheduling, it systematically solves the core technical challenges faced by intelligent agent large language model services. Specifically, it includes a hardware and software collaborative implementation scheme for resource partitioning, key-value cache management and runtime scheduling of pre-filling, decoding and tool call related workloads in a multi-chip encapsulation and inconsistent memory access environment.

Description

Technical Field

[0001] This invention belongs to the field of intelligent agent large language model service technology, and in particular relates to a core computing system, device and application for intelligent agent large language models. Background Technology

[0002] In recent years, deep learning-based natural language processing technology has made groundbreaking progress. The next generation of artificial intelligence technologies, represented by large-scale pre-trained language models (hereinafter referred to as large language models), is profoundly changing the way information is processed, knowledge is acquired, and human-computer interaction occurs. Through unsupervised pre-training on massive amounts of text data, large language models have acquired rich linguistic knowledge, world knowledge, and reasoning abilities. Their parameter scale has grown from hundreds of millions in the early days to hundreds of billions or even trillions today, and the model's capabilities have also undergone a qualitative leap.

[0003] The application of large language models has permeated many aspects of social life and industrial production, including intelligent dialogue, document writing, translation, code generation and review, legal document analysis, medical diagnosis assistance, and scientific literature review. Model inference—the process of generating an output response given input text—is the computational foundation supporting all applications.

[0004] The reasoning process in large language models is typically divided into two stages. For example... Figure 1 As shown, the first stage is the pre-filling stage, where the model processes all contextual information from the user input at once, calculates the attention representation for each word through large-scale matrix operations, and stores the intermediate results as a key-value cache. This stage is characterized by computational intensity and high parallelism, making it suitable for acceleration using a large number of computing cores. The second stage is the decoding stage, where the model generates output words one by one. Each word generation requires accessing all previously accumulated key-value caches for attention calculation. This stage is characterized by memory access intensity and low parallelism; the performance bottleneck lies primarily in memory bandwidth rather than computational power. The hardware resource requirements of these two stages differ fundamentally, and this difference provides significant optimization opportunities for the hardware architecture design of the inference system.

[0005] As the capabilities of large language models continue to improve, their application paradigm is undergoing a fundamental shift from single-step stateless reasoning to multi-step stateful interaction. In the traditional large language model reasoning model, a user submits a request, and the model generates a complete response; each interaction is independent and does not retain state information across dialogues. However, complex tasks in the real world—such as multi-source information retrieval and synthesis, multi-step scientific experimental design, and automated execution of complex processes—often cannot be completed through single-step reasoning and require the model to engage in continuous, multi-turn interactions with external tools and the environment.

[0006] Driven by this demand, large language models for intelligent agents have emerged. These models use a large language model as the core decision-making tool, enabling it to act as an intelligent agent capable of autonomous reasoning and planning, invoking external tools (such as search engines, databases, calculators, code executors, etc.), obtaining feedback, and continuing to reason and act based on the feedback results until the user-specified complex task is completed. Currently, various intelligent agent frameworks and interaction paradigms have been proposed to define the interaction methods between intelligent agents and external tools and users, and are widely used in scenarios such as personalized intelligent assistants, integrated circuit design assistance, enhanced retrieval generation, and automated scientific discovery.

[0007] like Figure 2 As shown, a single conversation in an intelligent agent large language model involves multiple rounds of iterative interaction between the user and the agent. Each iteration typically includes three phases: (1) a pre-filling phase, where the model understands the user's request and feedback information from the previous round of tool calls; (2) a decoding phase, where the model generates the response text and action plan for the current round; and (3) a tool call phase, where the model calls external tools through the application programming interface to obtain new information. The return result of the tool call is appended to the dialogue context for the next round of iteration. In subsequent iterations, the incremental pre-filling mechanism reuses existing key-value caches and only processes newly added input content, thereby avoiding redundant computation.

[0008] The multi-round iterative working mode described above makes the workload characteristics of intelligent agent large language models significantly different from traditional single-step inference. On the one hand, as the user prompts, model-generated lexical units, and tool-returned information accumulated in each iteration, the key-value cache continuously grows, leading to a surge in demand for memory capacity and bandwidth. On the other hand, the duration and computational load of each iteration are highly unpredictable because the response time and amount of returned data from tool calls vary. Furthermore, the frequent switching between the pre-filling and decoding phases means that the hardware needs to adapt quickly to both computationally intensive and memory-intensive workloads, posing a severe challenge to traditional statically configured hardware architectures.

[0009] In the semiconductor manufacturing field, as integrated circuit process nodes continue to shrink, the manufacturing area of ​​a single chip is gradually approaching the physical limits of photolithography and yield. For large chips with an area exceeding 600 square millimeters, manufacturing yield drops sharply, and costs rise significantly. Traditional single-chip integration routes face severe physical and economic barriers. Against this backdrop, chip-to-chip packaging architecture has emerged as a new integration technology solution. Figure 3 As shown, it effectively overcomes the area limitations of single-chip manufacturing by modularizing functions, dividing them into multiple smaller chips, and integrating them into a single module at the packaging level.

[0010] The core concept of chip-integration architecture is "decomposition and re-integration"—breaking down functions that would otherwise be integrated onto a single large chip into multiple independently manufactured chips, which are then recombined at the packaging level using advanced packaging technologies (such as silicon bridge interconnects and organic substrate fan-out). The advantages of this approach include: different modules can utilize their respective optimal process nodes, reducing costs; small individual chip area, resulting in high manufacturing yield; and flexible system scalability through adjustments to the number of chips. Current mainstream high-performance graphics processors and AI accelerators widely adopt chip-integration solutions.

[0011] Another key advantage of die-to-die packaging is the provision of a high-bandwidth in-package interconnect network. Components communicate with each other via standardized die-to-die interconnect protocols (such as the Universal Die-to-Die Interconnect Interface Standard), offering bandwidths several orders of magnitude higher than traditional external package networks. Figure 4 This is a schematic diagram of the non-uniform memory access system in a core-granular architecture: each core is equipped with local high-bandwidth memory (low-latency access), and accessing remote core memory requires traversing the encapsulated network (latency increases with distance); for example... Figure 4 As shown, the kernel-granule architecture inherently supports Non-Uniform Memory Access (NUMA) systems, where each kernel can be tightly coupled with its local high-bandwidth memory, forming a low-latency local access path. However, when accessing remote memory on other kernels is required, data must be transmitted through the encapsulated network, and the transmission latency depends on the physical distance and network topology between the source and target kernels. Efficiently managing data locality and reducing remote memory access overhead in the kernel-granule architecture is one of the core challenges in kernel-granule system design.

[0012] The above three technological backgrounds demonstrate that large language models are evolving from single-step reasoning to multi-step agent interaction. The unique workflow of large language models presents entirely new design requirements for hardware architecture. Chip-and-granular architecture, with its scalability and high-bandwidth interconnect characteristics, provides an ideal hardware foundation for building large-scale inference systems. However, existing chip-and-granular architectures lack specific design and optimization for the workload characteristics of large language models. Systematic innovation at both the architectural partitioning and dynamic scheduling levels is needed to fully leverage the advantages of chip-and-granular architecture to meet the dynamic resource requirements of large language models.

[0013] Based on the above background, the existing technical solutions still have the following drawbacks:

[0014] The unique workflow of large language models for intelligent agents presents unprecedented challenges to key-value cache management and workload scheduling. Through in-depth analysis of a typical intelligent agent retrieval enhancement generation workflow, the following main problems and shortcomings of existing technical solutions can be summarized:

[0015] First, the continuous growth of the key-value cache leads to unpredictable memory pressure. In agent-based large language models, each processed lexical unit (regardless of whether it comes from user input, model generation, or tool call feedback) generates a corresponding key-value cache, which is persistently stored for attention calculation in subsequent decoding stages. Since agent dialogue involves multiple iterations, the key-value cache grows steadily with each iteration, exhibiting cumulative expansion. For example... Figure 5 As shown in (a), statistical analysis of the typical agent retrieval enhancement generation workflow indicates that all dialogues involve three to six rounds of retrieval iterations, with the dialogue context continuously expanding during the iteration process. Although the lexical distribution in the pre-filling stage is generally concentrated within a certain range, it exhibits a significant long-tail characteristic, making runtime memory requirements extremely difficult to predict. Taking the current mainstream 175 billion parameter scale model as an example, when the sequence length exceeds 8192, the key-value cache alone requires more than 36 gigabytes of memory space, exceeding the single-card memory capacity of most commercial graphics processors.

[0016] Second, the workload of intelligent agents leads to highly dynamic imbalances in resource demands. Large language models for intelligent agents frequently alternate between a computationally intensive pre-filling phase and a memory-intensive decoding phase. Analysis of actual operational data reveals that the pre-filling phase is dominated by large-scale matrix multiplication operations, with an arithmetic intensity (the number of floating-point operations per byte of data transmission) exceeding 70, exhibiting typical computationally constrained characteristics. The decoding phase, dominated by vector-matrix multiplication operations, has an arithmetic intensity of only about 1, exhibiting typical memory bandwidth constrained characteristics. Interspersed between these two phases are tool call waiting intervals of varying durations, during which the hardware is idle or underutilized. Actual statistics show that in existing systems, up to approximately 18% of agent requests are preempted due to resource contention, and approximately 14% of memory is wasted due to improper allocation. As the pre-filling phase iterates, it gradually accounts for a larger proportion of the total latency, further exacerbating the dynamic changes in resource demands.

[0017] Third, traditional graphics processing unit (GPU) architectures face inherent memory capacity bottlenecks. Current mainstream large language model inference systems rely on GPUs as their core computing hardware. However, the on-chip high-bandwidth memory capacity of GPUs is fixed and limited. When faced with the continuously growing key-value cache in agent dialogue, it quickly encounters the dual limitations of insufficient memory capacity and bandwidth bottlenecks. Although academia and industry have proposed various key-value cache optimization techniques, including reducing cache precision through quantization to decrease storage volume, reducing the number of cache entries that need to be accessed through sparse attention, and improving memory utilization through paging mechanisms, these methods are mainly designed for single-step inference scenarios and are insufficient to effectively address the fundamental problem of the cumulative growth of key-value cache across iterations in large language models for agents.

[0018] Fourth, existing split-inference service architectures have significant limitations. To optimize the execution efficiency of the pre-filling and decoding stages separately, academia and industry have proposed split-inference schemes, deploying pre-filling and decoding tasks on different computing nodes. While this approach improves resource utilization in a single stage, it faces severe data transmission overhead when applied to large language models for intelligent agents. Because agent dialogue requires frequent switching between pre-filling and decoding, a large amount of key-value cache data needs to be repeatedly transferred between the two sets of computing nodes. In traditional GPU cluster-based schemes, nodes communicate via an encapsulated external network, resulting in limited bandwidth and high latency; the transmission of key-value cache data becomes the main bottleneck for end-to-end latency.

[0019] Fifth, static resource mapping strategies in the chip-granular architecture cannot adapt to dynamic workloads. Although the chip-granular architecture provides scalable performance and high-bandwidth in-package interconnects, most existing chip-granular systems employ static workload mapping strategies, which assign fixed computational roles and memory usage to each chip during system initialization. This static strategy cannot adapt to the highly dynamic nature of large language model workloads for intelligent agents, leading to some chips being overloaded while others are idle during peak load periods. Traffic is unevenly distributed across the interconnect links between chips, and the advantages of high-bandwidth in-package interconnects are not fully utilized. Furthermore, existing scheduling strategies are mainly designed for graphics processor clusters and lack scheduling schemes specifically considering the non-uniform memory access characteristics in the chip-granular architecture. Summary of the Invention

[0020] The purpose of this invention is to provide a chip computing system, device, and application for large language models of intelligent agents, so as to solve the problems existing in the prior art.

[0021] In a first aspect, to achieve the above objectives, the present invention provides a core-particle computing system for large language models of intelligent agents, comprising:

[0022] At least one pre-filled region, the pre-filled region being used for computationally intensive pre-filled phase computation tasks;

[0023] A decoding area is placed adjacent to the pre-filled area, and the decoding area is used to handle memory bandwidth intensive decoding stage tasks;

[0024] A swing area, located between the pre-filled area and the decoding area, is used to convert the swing area into a decoding area or a pre-filled area according to workload requirements;

[0025] A scheduler is configured to dynamically schedule the pre-filled area, the decoding area, and the oscillation area according to a dialogue request.

[0026] The pre-filled area, decoding area, and swing area each include several instances, and each instance includes multiple adjacent cores.

[0027] Optionally, each of the cores is configured with corresponding high-bandwidth memory, which is used to form a low-latency local access path.

[0028] Optionally, the core chip specifically includes:

[0029] A host processor, which is used to run control logic, scheduling algorithms and system management tasks;

[0030] The computing core array includes multiple computing cores, which are used to perform matrix multiplication operations and batch vector matrix multiplication operations; each computing core is equipped with a single instruction multiple data stream processing unit for performing nonlinear operations such as normalization and activation functions;

[0031] Local static random access memory (SRAM) temporary storage space, which is used to buffer intermediate computation data;

[0032] A secondary cache, which provides a low-latency data sharing channel for multiple computing cores;

[0033] An interconnection interface is used to connect the various cores to enable high-speed data transmission between the cores.

[0034] An encapsulated internal network router is used for routing and forwarding data packets in a two-dimensional mesh topology.

[0035] A data transfer engine is used to perform batch data transfer between local static random access memory (SRAM) temporary storage space, secondary cache, and high-bandwidth memory.

[0036] Optionally, the processing procedure of the scheduler specifically includes:

[0037] Topology-aware attribution: When the scheduler receives a new dialogue request, it selects a pair of physically adjacent decoding attribution instances and pre-filled attribution instances in the decoding area and the pre-filled area to obtain an attribution instance pair. The attribution instance pair serves as the fixed attribution location of the current dialogue. The computational tasks and key-value cache storage of the current dialogue throughout its entire lifecycle are bound to the selected attribution instance pair.

[0038] Active cache prefetching: While the decoding phase tasks are being executed, the scheduler predicts the next phase tasks based on the current decoding progress and tool call prediction signals, and uses the direct memory access engine to prefetch the key-value cache from the decoding home instance to the pre-filled home instance in the background transmission.

[0039] Adaptive resource conversion: The scheduler monitors the average utilization of the pre-filled area and the decoding area. When the average utilization of the pre-filled area exceeds the preset high water level threshold, the scheduler converts idle swing instances into pre-filled instances. When the average utilization of the decoding area exceeds the high water level threshold, the scheduler converts idle swing instances into decoding instances.

[0040] Secondly, to achieve the above objectives, this invention provides an application of a core-particle computing system for intelligent agent large language models in the conversation process of intelligent agent large language models, specifically including:

[0041] Dialogue arrival and attribution assignment: After a new dialogue request is received, an attribution instance pair is selected for the new dialogue request; the initial pre-filling of the dialogue is performed on the pre-filled attribution instance, and the generated key-value cache is transferred to the local high-bandwidth memory of the decoding attribution instance for storage; the decoding stage is performed on the decoding attribution instance, and the response is generated word by word.

[0042] Tool Invocation and Active Prefetching: When a tool invocation instruction is generated during the decoding phase, decoding is paused and the tool returns. At the same time, the key-value cache is prefetched from the decoding home instance to the prefilled home instance. After the tool returns, incremental prefilling is started on the prefilled home instance, and the newly added key-value cache is then sent back to the decoding home instance to enter the next round of decoding.

[0043] Dynamic resource adjustment: Continuously monitor the load of each region and dynamically adjust the role of the swing zone instance based on the real-time resource requirements of the pre-filled area and the decoding area to ensure overall system resource balance;

[0044] Dialogue End and Resource Release: After the dialogue ends, the key-value cache space allocated for the dialogue by the owner instance is released, and the load count of the decoding owner instance is decremented by one.

[0045] Thirdly, to achieve the above objectives, the present invention provides a computer device, including the chip computing system for intelligent agent large language models as described in the first aspect.

[0046] The technical effects of this invention are as follows:

[0047] This invention systematically analyzes and models the workload characteristics of intelligent agent large language models, identifies and quantifies two key bottlenecks: the cumulative growth of key-value cache and the highly dynamic fluctuation of resource requirements in multi-iteration operations, and provides a data foundation and design basis for architecture design and scheduling optimization.

[0048] This invention proposes a partitioned core-particle architecture scheme for large language models of intelligent agents. The core-particle network is innovatively divided into three functional regions: pre-filling region, decoding region, and swing region. Each region is optimized with differentiated hardware configuration according to the characteristics of its workload. At the same time, dynamic resource reallocation is realized during runtime through the swing region, which takes into account both the efficiency improvement brought by specialization and the adaptability brought by flexibility.

[0049] This invention designs a fine-grained dynamic workload scheduling method for dialogue, which includes three core mechanisms: topology-aware attribution, proactive cache prefetching, and adaptive resource conversion. It can dynamically adjust resource allocation, optimize data transmission paths, and hide key-value cache transmission latency according to the actual workload at runtime, thereby achieving high throughput and low latency intelligent agent large language model services.

[0050] The effectiveness of the present invention has been verified through systematic evaluation. Compared with the existing best discrete chip architecture baseline, it achieves a 2.33x increase in throughput and an average 58% reduction in end-to-end dialogue latency. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0053] Figure 1 A schematic diagram of the two-stage process for reasoning in a large language model;

[0054] Figure 2 A schematic diagram illustrating the workflow of a large language model for an intelligent agent;

[0055] Figure 3 This is a schematic diagram of chip integration;

[0056] Figure 4 This is a schematic diagram of the non-uniform memory access system in the chip architecture of this invention: each chip is equipped with local high-bandwidth memory (low-latency access), and accessing remote chip memory requires passing through the encapsulated network (latency increases with distance).

[0057] Figure 5The workload analysis includes (a) the timeline of multiple iterations and changes in hardware utilization; and (b) the statistics of the number of tokens (including long-tail features) and the number of iterations in the pre-filling, incremental pre-filling and decoding stages.

[0058] Figure 6 The diagram shows the overall design of the chip architecture in this embodiment of the invention, where (a) is a top view of the package, showing the 8×12 chip array and high-bandwidth memory distribution; (b) is the internal microarchitecture of a single chip (computing core array, cache, router, data transport engine); and (c) is a cross-sectional view of the silicon bridge interconnect package.

[0059] Figure 7 The diagram shows the partitioned resource management scheme, design space exploration results, and simulator structure in this embodiment of the invention. (a) is a schematic diagram of the partitioned resource management scheme, showing the strip layout of the pre-filled area (blue), the decoding area (green), and the swing area (orange); (b) is the design space exploration results: the trend of throughput variation with the area ratio of each region and the tensor parallelism configuration; (c) is a schematic diagram of the simulator structure.

[0060] Figure 8 This is a schematic diagram of the scheduling mechanism in an embodiment of the present invention, wherein (a) is a topology-aware attribution mechanism, which selects a physically adjacent decode-pre-filled attribution instance pair for a new dialogue; (b) is an active cache prefetching timing mechanism, in which key-value cache transmission and decoding computation are performed in parallel; and (c) is an adaptive resource switching mechanism, in which swiveling instances dynamically switch roles according to load. In the figure, P-inst., D-inst., and S-inst. represent pre-filled instances, decoded instances, and swiveling instances, respectively.

[0061] Figure 9 For the performance comparison of different hardware architecture configurations in the embodiments of the present invention, the four indicators in the figure are: System Capacity (effective throughput, in the number of requests per second), System Congestion, User-Perceived Latency, and Traffic Balance (measured by the standard deviation of total traffic).

[0062] Figure 10For the scheduling strategy ablation experiment in this embodiment of the invention, the groups on the horizontal axis are superimposed from left to right as follows: Homing (homing assignment), +Proactive & Spill (+active caching and peer overflow), +Adaptive (+adaptive resource conversion). The three indicators on the vertical axis are: Norm. P95 TTFT (normalized 95th percentile first-word delay), Norm. P95 Queue Time (normalized 95th percentile queuing time), and Norm. Avg End-to-End Latency (normalized average end-to-end latency).

[0063] Figure 11 This is a comparison of throughput scalability under different load levels in this embodiment of the invention. The horizontal axis OfferedLoad of Incoming Conversations (RPS) represents the incoming load of input conversations (requests per second), and the vertical axis AchievedGoodput (RPS) represents the effective throughput (requests per second). Detailed Implementation

[0064] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0065] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0066] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0067] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0068] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0069] Example 1

[0070] like Figure 1 - Figure 11 As shown, this embodiment provides a chip computing system for intelligent agent large language models. Through the collaborative design of hardware architecture and software scheduling, it systematically solves the core technical challenges faced by intelligent agent large language model services. Specifically, it includes a hardware and software collaborative implementation scheme for resource partitioning, key-value cache management and runtime scheduling of pre-filling, decoding and tool call related workloads in a multi-chip encapsulation and inconsistent memory access environment.

[0071] Workload analysis and design motivation of intelligent agent large language model:

[0072] To fundamentally understand the workload characteristics of large language models for intelligent agents and provide a data-driven basis for architecture design, this embodiment first conducts a comprehensive analysis of a typical intelligent agent retrieval enhancement generation workflow. Using an open-source intelligent agent retrieval question-answering workflow as the experimental object, a large language model backbone network with approximately 7 billion parameters was run on mainstream graphics processing unit hardware, and complete execution trajectory data was collected and analyzed.

[0073] The analysis revealed the following three core observations:

[0074] First, the workload of intelligent agents generates highly dynamic resource demands. Each dialogue dynamically alternates between a computationally intensive pre-filling phase and a memory-intensive decoding phase, with the pre-filling phase gradually accounting for a larger proportion of the total latency as iterations progress. Frequent phase switching means that statically configured hardware resources cannot simultaneously meet two distinctly different computational needs, leading to severe resource waste or performance bottlenecks.

[0075] Secondly, the growth of the key-value cache causes unpredictable memory pressure. Statistical data shows that all dialogues involve multiple rounds of iteration, with the dialogue context expanding repeatedly during these iterations. The number of lexical units in the pre-filling stage varies greatly, exhibiting a long-tail distribution. Since each generated lexical unit needs to have a corresponding entry retained in the key-value cache, the cache size steadily accumulates with each iteration, and the rate of increase is difficult to predict in advance.

[0076] Third, inter-stage data transfer constitutes a significant performance bottleneck. In a split inference architecture, key-value caches need to be transferred between computational units dedicated to pre-filling and those dedicated to decoding. For agent-based large language models, due to frequent stage switching triggered by tool calls, the key-value cache transfer frequency is much higher than in traditional single-step inference, and the proportion of transfer latency in the total latency increases significantly.

[0077] Based on the above three observations, this embodiment proposes the following core design principles: (1) Hardware resources need to have the ability to dynamically reallocate during runtime to adapt to the dynamic resource demand changes between pre-filling and decoding; (2) The scheduling strategy needs to be dialogue-centric, perceive data locality, and restrict the transmission path of key-value cache to short-distance links with high bandwidth and low latency; (3) The system should proactively predict the trend of workload changes, prepare data and resources in advance, and hide the unavoidable transmission delay within the effective computation time window.

[0078] Overall Design of Core-Particle Architecture for Large Language Models of Intelligent Agents: Based on the above design principles, this embodiment proposes a multi-core-particle module architecture for large language models of intelligent agents. For example... Figure 6 As shown, this architecture integrates multiple homogeneous computing cores within a two-dimensional mesh topology, forming an approximately square encapsulation layout. In one specific implementation, the module contains 96 cores arranged in 8 rows and 12 columns. Regarding memory, the architecture employs a non-uniform memory access system, with each core equipped with a dedicated high-bandwidth memory module. Local memory access latency is extremely low, while remote memory access latency depends on the transmission path length of the network within the encapsulation within the two-dimensional mesh.

[0079] The internal microstructure of each core particle, such as Figure 6 As shown in (b), it includes the following core modules:

[0080] (1) Computing Core Array: Each core integrates 16 computing cores. Each core contains a 128x128 systolic array matrix engine, which is capable of performing large-scale matrix multiplication operations (for parallel computing in the pre-filling stage) and batch vector matrix multiplication operations (for word-by-word generation in the decoding stage). In addition, each core is equipped with a dedicated single instruction multiple data stream processing unit for performing nonlinear operations such as normalization and activation functions.

[0081] (2) Local storage hierarchy: Each computing core is equipped with 1 megabyte of local static random access memory (SRAM) for buffering intermediate computation data. At the core level, there is a 16-megabyte shared L2 cache to provide a low-latency data sharing channel for multiple cores.

[0082] (3) Interconnection Interface: Each die is equipped with 4 inter-die communication interfaces, with a single interface bandwidth of 500 gigabytes per second, for high-speed data transmission between dies. Each die has a built-in in-encapsulation network router, which is responsible for routing and forwarding data packets in a two-dimensional mesh topology.

[0083] (4) Data transfer engine: Each core is equipped with a direct memory access engine with a bandwidth of 1 terabyte per second, which can autonomously complete batch data transfer between local temporary storage space, secondary cache and high bandwidth memory without occupying the execution resources of the computing core.

[0084] (5) Host processor: Each chip integrates a general-purpose processor core for running control logic, scheduling algorithms and system management tasks.

[0085] In this embodiment, the topology of the chip array is not limited to an 8-row, 12-column two-dimensional grid. It can also be configured as other N-row, M-column regular or near-regular array structures depending on the model size, packaging area, and target throughput.

[0086] Partition resource management solution:

[0087] This embodiment proposes a partitioned resource management scheme that divides the chip network into three functional regions, such as... Figure 7 As shown in (a), the three functional regions are the pre-filled region, the decoding region, and the swing region. Each region consists of several instances, and each instance is constructed from multiple adjacent cores. The design concept and functional definition of each region are as follows:

[0088] (1) Pre-filled region: This region is specifically designed for performing computationally intensive pre-filled stage computations. Each instance in the pre-filled region is aggregated from multiple cores to provide sufficient parallel computing power. This region employs a large tensor parallelism configuration, enabling multiple cores to collaboratively perform matrix multiplication operations at the same layer. The optimal parallelism size was determined to be 8 through design space exploration. This is because the computational requirements of the pre-filled stage far outweigh the communication requirements, and a larger parallelism can significantly improve computational throughput, while the full reduction communication overhead between cores is negligible under the high-bandwidth interconnect within the package.

[0089] (2) Decoding Zone: This zone is specifically designed to handle memory-bandwidth-intensive decoding tasks. Each instance in the decoding zone consists of fewer cores, forming a compact, small-scale computing unit to enable flexible concurrent task scheduling. This zone employs a small tensor parallelism configuration, with an optimal value of 4. This is because the decoding phase requires processing decoding tasks from multiple different dialogues simultaneously. The smaller instance size allows the system to run more decoding task instances concurrently, reducing the synchronization overhead and communication bottlenecks of a single task, and effectively improving the overall concurrent throughput of the decoding phase.

[0090] (3) Swing Zone: One of the key innovations of this embodiment. The swing zone is located between the pre-filling zone and the decoding zone. Instances within it can dynamically switch roles during runtime according to actual workload requirements—either acting as pre-filling instances to perform pre-filling calculations or acting as decoding instances to perform decoding tasks. This flexible role-switching capability allows the system to dynamically adjust the resource allocation ratio between pre-filling and decoding according to real-time load, effectively coping with the highly dynamic nature of the agent's workload and avoiding situations where resources are excessive in one stage and insufficient in another stage when the load fluctuates.

[0091] In terms of physical layout, the three regions are arranged in a striped sequence of "decoding region - pre-filled region - swing region - pre-filled region". This layout design takes into account the following factors: First, placing the decoding region and the pre-filled region adjacently ensures that the key-value cache only needs to be transmitted through short-distance, high-bandwidth links, minimizing the latency of cross-stage data exchange. Second, setting up two sets of pre-filled regions reflects the workload profiling results—the total computational load of the pre-filled sequence is much greater than that of the decoding sequence, and it gradually becomes dominant as iterations increase, thus requiring more hardware resources. Finally, the swing region is located between the pre-filled region and the decoding region, ensuring that it remains physically adjacent to the corresponding region regardless of its role, reducing data migration overhead during role switching.

[0092] Regarding parallelism configuration, this embodiment explores the design space systematically, and under constraints of a fixed total number of cores and mesh size, jointly optimizes the area ratio and tensor parallelism configuration of each region, such as... Figure 7 As shown in (b), the optimization objective is to maximize the system throughput under a representative agent workload. Constraints include the sum of the areas of each region equaling the total number of cores, and the tensor parallelism being a valid factor of the number of cores. The final optimal configuration is as follows: pre-filled instances use 4 columns and 2 rows, totaling 8 cores, with a parallelism of 8; decoding instances use 2 columns and 2 rows, totaling 4 cores, with a parallelism of 4; and oscillating instances use 3 columns and 2 rows, totaling 6 cores.

[0093] This embodiment innovatively divides the chip network into three functional regions: a pre-filling region, a decoding region, and a swinging region. This enables differentiated hardware configuration optimization for the distinct computational characteristics of the pre-filling and decoding stages. At the same time, the dynamic role switching capability of the swinging region enables runtime resource reallocation, balancing specialized efficiency with flexible adaptability.

[0094] In practice, the area ratio, number of instances, and tensor parallelism of the pre-filled region, decoding region, and oscillating region in this embodiment are not fixed. Different partition ratios, instance granularities, and parallel configurations can be adopted for different model parameter scales, context lengths, tool call frequencies, and service level targets. The local high-bandwidth storage medium in this embodiment is not limited to high-bandwidth memory and can be replaced with other on-chip or near-memory structures suitable for high-throughput, low-latency access, depending on the implementation conditions; the inter-die interconnect is also not limited to a specific die interconnect protocol.

[0095] A Dialogue-Oriented Dynamic Workload Scheduling Method: This embodiment proposes a dialogue-oriented dynamic workload scheduling method. This method uses the dialogue as the basic unit of scheduling and comprehensively considers multiple optimization objectives such as data locality, resource balancing, and transmission latency concealment. The scheduling method includes three core mechanisms:

[0096] Topology-aware attribution mechanism: The core idea of ​​the topology-aware attribution mechanism is that for each newly arrived dialogue request, a pair of physically adjacent instances are selected in the decoding area and the pre-filling area as the fixed "home" of the dialogue. The dialogue's computational tasks and key-value cache storage throughout its entire lifecycle are bound to this pair of instances. Specifically, as... Figure 8 As shown in (a), when the scheduler receives a new dialogue request, it first selects the decoding instance with the lowest current load in the decoding area as the decoding home instance for the dialogue; then, within the physical neighborhood of the decoding instance, it selects the pre-filled instance with the largest available capacity and the closest topological distance as the paired pre-filled home instance.

[0097] The key advantage of this mechanism lies in the fact that the key-value cache for decoding tasks is always stored in the local high-bandwidth memory of the decoding home instance. Frequent memory accesses during the decoding phase are all low-latency local accesses, fully leveraging the performance advantages of high-bandwidth memory. When switching to the pre-filling phase is required, the key-value cache only needs to be transferred from the decoding home instance to the physically adjacent pre-filling home instance, completed via a short-distance, high-bandwidth link within the encapsulation. The transmission latency is far lower than that of cross-cluster, out-of-encapsulation transmissions. By binding dialogues to fixed instance pairs, the need for random migration of the key-value cache between arbitrary instances is eliminated, significantly reducing global data transfer volume and network congestion.

[0098] This embodiment selects a physically nearby decode-prefill instance pair as a fixed home location for each dialogue, restricting the transmission path of the key-value cache to a short-distance, high-bandwidth link, fundamentally solving the latency bottleneck of long-distance transmission of the key-value cache in the split inference architecture.

[0099] Proactive Cache Prefetching Mechanism: This mechanism addresses the frequent stage switching issues triggered by tool calls in large language models for intelligent agents by pre-transmitting key-value caches to hide transmission latency. In traditional solutions, after a tool call returns a result, the system needs to transfer the key-value cache from the decoded home instance to the pre-filled home instance before incremental prefilling computation can begin. This transmission wait time occurs frequently in the agent workflow, accumulating to constitute a significant performance penalty.

[0100] The active caching strategy proposed in this embodiment is as follows: Figure 8As shown in (b): During the decoding phase, the scheduler anticipates upcoming phase transitions based on the current decoding progress and tool call prediction signals. Utilizing the direct memory access engine, it prefetches the key-value cache from the decoding home instance to the pre-filled home instance via background transmission. This background transmission runs in parallel with the decoding computation, without consuming core computing resources or interfering with ongoing decoding tasks. When the tool call result actually arrives, the pre-filled home instance already possesses a complete copy of the key-value cache, allowing for immediate incremental pre-filling computation, completely eliminating idle time spent waiting for transmission completion.

[0101] The timing of prefetching is determined by a sliding window mechanism: the scheduler continuously estimates the remaining time of the current decoding phase and compares it with the time required for key-value buffer pretransmission. Prefetching is triggered when the estimated transmission time is less than the remaining decoding time. This adaptive triggering mechanism dynamically adjusts the prefetching timing based on the key-value buffer size and decoding speed of different conversations, achieving a balance between sufficient transmission and avoiding premature bandwidth consumption.

[0102] This embodiment eliminates the waiting idle time during phase switching by predicting the upcoming phase switch during the decoding stage and transmitting the key value cache in advance, thus hiding the unavoidable transmission delay within the effective calculation time window, significantly reducing the end-to-end dialogue latency.

[0103] Adaptive resource switching mechanism: The adaptive resource switching mechanism fully utilizes the role-switching capability of the swing zone to achieve dynamic resource reallocation at runtime. For example... Figure 8 As shown in (c), the scheduler continuously monitors runtime metrics for each instance in the pre-filled and decoded regions, including compute core utilization, memory usage, pending task queue length, and network link bandwidth utilization within the encapsulation. Based on these metrics, the scheduler makes resource conversion decisions using a dual-threshold strategy.

[0104] When the average utilization rate of the pre-filled region exceeds a preset high-water mark threshold (e.g., 90%), indicating strong demand for pre-filling computation and resource scarcity, the scheduler converts idle oscillating instances into pre-filled instances, enhancing the computational capacity of the pre-filled region. Similarly, when the average utilization rate of the decoding region exceeds the high-water mark threshold, the scheduler converts idle oscillating instances into decoding instances, increasing the concurrent processing capacity of the decoding region. During this role conversion process, necessary data migration is completed in the background through the direct memory access engine. This flexible resource conversion capability ensures that the overall hardware utilization of the system remains at a high level, effectively smoothing the dynamic fluctuations of the agent's workload and avoiding the common problem of resource idleness and bottlenecks in static resource allocation schemes.

[0105] This embodiment uses real-time load monitoring and a dual-threshold decision-making strategy to dynamically switch the role of the swing zone instance between pre-filled computing resources and decoding computing resources, effectively smoothing the dynamic fluctuations of the agent's workload and maintaining the overall high resource utilization of the system.

[0106] In practice, the topology-aware attribution, proactive cache prefetching, and adaptive resource conversion mechanisms in this embodiment can be implemented using rule-based thresholding, statistical prediction, or learning-based control strategies. Their triggering conditions can be determined based on queuing length, key-value cache size, link congestion level, tool call return prediction time, or a combination of multiple indicators.

[0107] Overall architecture and workflow of the scheduling system: The three scheduling mechanisms mentioned above work together to form a complete scheduling system. The data flow and control flow of the entire scheduling system are as follows:

[0108] (1) Dialogue arrival and attribution assignment: After a new dialogue request arrives, the topology-aware attribution mechanism selects a decoding attribution instance and a pre-filled attribution instance pair for it. The initial pre-filling of the dialogue is performed on the pre-filled attribution instance, and the generated key-value cache is transferred to the local high-bandwidth memory of the decoding attribution instance for storage. Subsequently, the decoding phase is performed on the decoding attribution instance, generating a response word by word.

[0109] (2) Tool Invocation and Active Prefetching: When a tool invocation instruction is generated during the decoding phase, decoding is paused and the tool returns. At the same time, the active caching mechanism begins to prefetch the key-value cache from the decoding home instance to the pre-filled home instance. After the tool returns, incremental prefilling is immediately started on the pre-filled home instance, and the newly added key-value cache is then sent back to the decoding home instance to enter the next round of decoding.

[0110] (3) Dynamic resource adjustment: Throughout the service process, the adaptive resource conversion mechanism continuously monitors the load of each region and dynamically adjusts the role of the swing zone instance according to the real-time resource requirements of pre-filling and decoding to ensure the overall resource balance of the system.

[0111] (4) End of dialogue and release of resources: After the dialogue is completed, the key-value cache space allocated by the host instance for the dialogue is released, the load count of the decoding host instance is decremented by one, and resources are freed up for subsequent new dialogue requests.

[0112] The following is a specific implementation example of this embodiment:

[0113] The system employs an 8x12 two-dimensional mesh topology, integrating a total of 96 homogeneous cores. Each core integrates 16 computing cores (including a 128x128 systolic array matrix engine), 16 megabytes of L2 cache, four 500 gigabytes per second inter-die interfaces, and a 1 terabyte per second direct memory access engine. The 96 cores are arranged in a striped sequence of "decoding area—pre-filling area—swinging area—pre-filling area". Pre-filling instances consist of 8 cores in 4 columns and 2 rows (parallelism 8), decoding instances consist of 4 cores in 2 columns and 2 rows (parallelism 4), and swinging instances consist of 6 cores in 3 columns and 2 rows.

[0114] The evaluation used a large language model with approximately 8 billion parameters, running test workloads derived from an agent-based question-answering workflow. The baseline setup employed a decoupled core architecture with static resource allocation and fixed workload mapping. Evaluation results: This implementation achieved a 2.33x throughput improvement and an average 58% reduction in end-to-end dialogue latency. The pre-filled and decoding areas maintained high utilization under various workloads. Proactive cache prefetching reduced stage switching latency by over 60%, and topology-aware attribution reduced cross-instance data transfer by approximately 40%.

[0115] To verify the effectiveness of the system in this embodiment, a comprehensive system evaluation was conducted on a representative agent retrieval enhancement generation task using a mainstream large language model with approximately 8 billion parameters, and compared with the following three baseline schemes: (1) a split graphics processor scheme (marked as DS-GPU in the figure), which is a scheme that uses split inference on a traditional graphics processor cluster; (2) a split chip scheme (marked as DS-Chiplet in the figure), which is a scheme that uses static split inference on a chip architecture; and (3) a partitioned chip scheme (marked as WSC-Chiplet in the figure), which is a scheme that uses static partitioning on a chip architecture but without dynamic scheduling. The scheme in this embodiment is marked as ACES in the figure. The evaluation results show that this embodiment has the following significant beneficial effects:

[0116] First, it significantly increases system throughput. For example... Figure 9 As shown, compared to the baseline solution of the discrete core-granular architecture, the system in this embodiment achieves a throughput improvement of up to 2.33 times, while significantly reducing system congestion, user-perceived latency, and traffic imbalance. This improvement stems from three technical aspects: the partitioned resource management scheme ensures that pre-filling and decoding each utilize optimally configured hardware resources, avoiding resource contention; the adaptive switching mechanism of the swing zone dynamically matches workload demands, maximizing overall system resource utilization; and topology-aware attribution reduces unnecessary cross-instance data transfer, increasing the proportion of effective computation time.

[0117] Second, each scheduling mechanism has a verifiable marginal contribution. For example... Figure 10As shown, by gradually adding various scheduling strategies through ablation experiments, the independent contribution of each mechanism can be clearly observed: the introduction of the attribution allocation mechanism alone can achieve a 13% increase in throughput and a 22% reduction in first-word latency; on this basis, after adding the active cache prefetch and peer overflow mechanisms, the throughput is further increased by more than 50%; finally, after adding the adaptive resource conversion mechanism, the queuing time is reduced by another 48%, and the end-to-end latency is reduced to 28% of the static baseline.

[0118] Third, it significantly reduces end-to-end dialogue latency. This embodiment achieves an average 58% reduction in end-to-end dialogue latency. The proactive cache prefetching mechanism is a key contributor to this latency reduction—by hiding the key-value cache transmission latency within the decoding computation time window, it eliminates the idle time that must be waited for data transmission to complete during phase switching in traditional schemes. Topology-aware attribution ensures that the key-value cache is transmitted only between physically adjacent instance pairs, further shortening the absolute time required for each transmission.

[0119] Fourth, it possesses good system scalability. For example... Figure 11 As shown, with the gradual increase in system load (request arrival rate), the throughput of the system in this embodiment maintains a steady increase, demonstrating load-bearing capacity and scalability superior to the baseline scheme. The chip architecture itself can linearly expand computing and storage resources by increasing the number of chips. The partition resource management and scheduling strategy of this embodiment is compatible with chip array configurations of different sizes and can adaptively adjust the area ratio and parallelism parameters of each region.

[0120] Fifth, it significantly reduces data transmission costs. Compared to traditional discrete inference schemes based on GPU clusters, this embodiment utilizes high-bandwidth interconnects within the chip package, providing an order-of-magnitude bandwidth increase compared to the external network. The topology-aware affiliation strategy further restricts communication to physically adjacent chips, minimizing the number of transmission hops and queuing delays in the internal network and reducing overall communication power consumption.

[0121] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A chip computing system for large language models of intelligent agents, characterized in that, include: At least one pre-filled region, the pre-filled region being used for computationally intensive pre-filled phase computation tasks; A decoding area is placed adjacent to the pre-filled area, and the decoding area is used to handle memory bandwidth intensive decoding stage tasks; A swing area, located between the pre-filled area and the decoding area, is used to convert the swing area into a decoding area or a pre-filled area according to workload requirements; A scheduler is configured to dynamically schedule the pre-filled area, the decoding area, and the oscillation area according to a dialogue request. The pre-filled area, decoding area, and swing area each include several instances, and each instance includes multiple adjacent cores.

2. The chip computing system for large language models of intelligent agents according to claim 1, characterized in that, Each of the aforementioned chips is equipped with corresponding high-bandwidth memory, which is used to form a low-latency local access path.

3. The chip computing system for large language models of intelligent agents according to claim 2, characterized in that, The core chip specifically includes: A host processor, which is used to run control logic, scheduling algorithms and system management tasks; The computing core array includes multiple computing cores, which are used to perform matrix multiplication operations and batch vector matrix multiplication operations; each computing core is equipped with a single instruction multiple data stream processing unit for performing nonlinear operations such as normalization and activation functions; Local static random access memory (SRAM) temporary storage space, which is used to buffer intermediate computation data; A secondary cache, which provides a low-latency data sharing channel for multiple computing cores; An interconnection interface is used to connect the various cores to enable high-speed data transmission between the cores. An encapsulated internal network router is used for routing and forwarding data packets in a two-dimensional mesh topology. A data transfer engine is used to perform batch data transfer between local static random access memory (SRAM) temporary storage space, secondary cache, and high-bandwidth memory.

4. The chip computing system for large language models of intelligent agents according to claim 3, characterized in that, The processing procedure of the scheduler specifically includes: Topology-aware attribution: When the scheduler receives a new dialogue request, it selects a pair of physically adjacent decoding attribution instances and pre-filled attribution instances in the decoding area and the pre-filled area to obtain an attribution instance pair. The attribution instance pair serves as the fixed attribution location of the current dialogue. The computational tasks and key-value cache storage of the current dialogue throughout its entire lifecycle are bound to the selected attribution instance pair. Active cache prefetching: While the decoding phase tasks are being executed, the scheduler predicts the next phase tasks based on the current decoding progress and tool call prediction signals, and uses the direct memory access engine to prefetch the key-value cache from the decoding home instance to the pre-filled home instance in the background transmission. Adaptive resource conversion: The scheduler monitors the average utilization of the pre-filled area and the decoding area. When the average utilization of the pre-filled area exceeds the preset high water level threshold, the scheduler converts idle swing instances into pre-filled instances. When the average utilization of the decoding area exceeds the high water level threshold, the scheduler converts idle swing instances into decoding instances.

5. The application of the chip computing system for large language models of intelligent agents as described in any one of claims 1-4 in the conversation process of large language models of intelligent agents.

6. The application according to claim 5, characterized in that, Specifically, it includes: Dialogue arrival and attribution assignment: After a new dialogue request is received, an attribution instance pair is selected for the new dialogue request; the initial pre-filling of the dialogue is performed on the pre-filled attribution instance, and the generated key-value cache is transferred to the local high-bandwidth memory of the decoding attribution instance for storage; the decoding stage is performed on the decoding attribution instance, and the response is generated word by word. Tool Invocation and Active Prefetching: When a tool invocation instruction is generated during the decoding phase, decoding is paused and the tool returns. At the same time, the key-value cache is prefetched from the decoding home instance to the prefilled home instance. After the tool returns, incremental prefilling is started on the prefilled home instance, and the newly added key-value cache is then sent back to the decoding home instance to enter the next round of decoding. Dynamic resource adjustment: Continuously monitor the load of each region and dynamically adjust the role of the swing zone instance based on the real-time resource requirements of the pre-filled area and the decoding area to ensure overall system resource balance; Dialogue End and Resource Release: After the dialogue ends, the key-value cache space allocated for the dialogue by the owner instance is released, and the load count of the decoding owner instance is decremented by one.

7. A computer device, characterized in that, Including the chip computing system for large language models of intelligent agents as described in any one of claims 1-4.

Images