Pre-filling decoding coincidence execution LLM calculation method based on high-computing-power wafer-level chip

Abstract

The invention relates to a pre-filling decoding coincidence execution LLM calculation method based on a high-computing-power wafer-level chip. The method comprises the following steps: in response to an inference request triggered in an input box of a large language model, searching a pre-filling instance and a corresponding decoding instance from all instances corresponding to a wafer-level chip according to an instance configuration table and a corresponding demand configuration table; the instance configuration table and the corresponding demand configuration table are both constructed on the basis of the existing wafer-level chip architecture; performing pre-filling operation on the reasoning request by using the pre-filling instance, and storing data required by the reasoning request into a cache corresponding to the pre-filling instance; performing decoding operation on data required by the reasoning request based on the decoding instance, and taking an obtained decoding result as a reasoning result of the large language model; and outputting a reasoning result through an output box of the large language model. By adopting the method, reasoning of the large language model can be carried out by utilizing the wafer-level chip.

Description

Technical Field

[0001] This application relates to the field of large language model technology, and in particular to a pre-filled decoding overlap execution LLM calculation method based on a high-computing-power wafer-level chip. Background Technology

[0002] With the continuous development of Large Language Models (LLMs), the number of parameters in LLMs has grown to hundreds of billions, significantly improving the generalization ability of the models, but also posing unprecedented challenges to the computing, storage and communication resources of the underlying hardware.

[0003] Graphics Processing Unit (GPU) clusters can provide inference services for LLMs using either a unified or separate mapping strategy. However, the size of GPU clusters is limited by scalability constraints, and the communication bandwidth between cluster nodes is relatively low. Wafer-level chips, utilizing advanced packaging methods, can overcome the limitations of GPU clusters. However, significant differences exist between wafer-level chips and GPU clusters, and the mapping strategies used in GPU clusters cannot be migrated to wafer-level chips.

[0004] Therefore, how to utilize wafer-level chips for reasoning of large language models has become an urgent problem to be solved. Summary of the Invention

[0005] Therefore, it is necessary to provide a pre-filled decoding overlap execution LLM calculation method based on a high-computing-power wafer-level chip to address the above-mentioned technical problems, which can utilize wafer-level chips for reasoning of large language models.

[0006] Firstly, this application provides a reasoning method for a large language model, applied to a wafer-level chip with a deployed large language model, comprising:

[0007] In response to an inference request triggered within the input box of the large language model, the pre-filled instance and the corresponding decoding instance are searched from all instances corresponding to the wafer-level chip according to the instance configuration table and the corresponding requirement configuration table; both the instance configuration table and the corresponding requirement configuration table are built on the basis of the existing wafer-level chip architecture.

[0008] Pre-populated instances are used to pre-populate inference requests, storing the data required by the inference requests into the cache corresponding to the pre-populated instances.

[0009] Based on the decoding instance, the data required for the inference request is decoded, and the decoding result is used as the inference result of the large language model.

[0010] The inference results are output through the output box of the large language model.

[0011] In one embodiment, according to the instance configuration table and the corresponding demand configuration table, a pre-filled instance is searched from all instances corresponding to the wafer-level chip, including:

[0012] Based on the instance configuration table and the corresponding requirement configuration table, sort the pre-filled memory usage of all instances in the wafer-level chip to obtain the pre-filled instance sorting result;

[0013] Based on the sorting results of pre-filled instances, search for pre-filled instances from all instances corresponding to the wafer-level chip.

[0014] In one embodiment, the pre-filled memory usage of all instances in the wafer-level chip is sorted according to the instance configuration table and the corresponding demand configuration table to obtain the pre-filled instance sorting result, including:

[0015] Based on the instance configuration table and the corresponding requirement configuration table, obtain the number of pre-filled requests to be processed for each instance in the wafer-level chip and the time of inference requests to be processed for each instance.

[0016] Sort the quantities of each pre-filled item in ascending order to obtain the sorted quantity result;

[0017] The time of each pending inference request is sorted from earliest to latest based on the quantity sorting result to obtain the pre-filled instance sorting result.

[0018] In one embodiment, searching for pre-filled instances from all instances corresponding to the wafer-level chip according to the pre-filled instance sorting result includes:

[0019] Get the first instance in the pre-filled instance sorting results and use the first instance as the pre-filled instance in the wafer-level chip; where the first instance is the instance with the smallest memory usage in the pre-filled instance sorting results.

[0020] In one embodiment, according to the instance configuration table and the corresponding requirement configuration table, the decoding instance corresponding to the pre-filled instance is searched from all instances corresponding to the wafer-level chip, including:

[0021] Based on the pre-filled instances and instance configuration table in the wafer-level chip, at least one candidate decoding instance is searched from all instances corresponding to the wafer-level chip;

[0022] According to the corresponding requirement configuration table, time increment evaluation is performed on each candidate decoding instance to obtain the time increment evaluation result for each candidate instance.

[0023] The candidate instance corresponding to the smallest time increment evaluation result is used as the decoding instance corresponding to the pre-filled instance.

[0024] In one embodiment, according to the pre-filled instances and instance configuration table in the wafer-level chip, at least one candidate decoding instance is searched from all instances corresponding to the wafer-level chip, including:

[0025] Calculate the number of cross-instance relationships between a pre-filled instance and any other instance in a wafer-level chip;

[0026] Instances with a cross-instance number less than or equal to a preset threshold are selected as candidate decoding instances in the wafer-level chip; the preset threshold represents the maximum number of cross-instance instances that can be accessed via cache without congestion.

[0027] In one embodiment, time increment evaluation is performed on each candidate decoding instance according to the corresponding requirement configuration table to obtain the time increment evaluation result for each instance, including:

[0028] Based on the requirement configuration table corresponding to each candidate decoding instance, the prediction time increment delay time for each candidate decoding instance to perform decoding operations on the data required for the inference request is determined.

[0029] The prediction time increment delay time corresponding to each candidate decoding instance is used as the time increment evaluation result.

[0030] In one embodiment, based on the demand configuration table corresponding to each candidate decoding instance, the prediction time increment delay time for each candidate decoding instance to perform decoding operations on the data required for the inference request is determined, including:

[0031] For any candidate decoding instance, based on the requirement configuration table corresponding to the candidate decoding instance, obtain the first predicted decoding time for the candidate decoding instance to perform decoding operations on the data required by the existing inference request; and obtain the second predicted decoding time for the candidate decoding instance to perform decoding operations on the data required by the existing inference request and the data required by the inference request.

[0032] The time difference between the second prediction decoding time and the first prediction decoding time is used as the prediction time increment delay time corresponding to the candidate decoding instance.

[0033] In one embodiment, the configuration process for the instance configuration table and the corresponding requirement configuration table includes:

[0034] Based on the hardware parameters of the existing wafer-level chip architecture, a configuration table of multiple candidate instances is determined;

[0035] The configuration of each candidate instance configuration table is pre-filled and decoded in a completely overlapping manner to obtain the requirement configuration table corresponding to each candidate instance configuration table;

[0036] The requirement configuration table corresponding to each candidate instance configuration table is used to guide instance operation, and the chip throughput corresponding to each candidate instance configuration table is estimated.

[0037] The candidate instance configuration table with the lowest chip throughput is used as the instance configuration table, and the requirement configuration table corresponding to the candidate instance configuration table with the lowest chip throughput is used as the requirement configuration table corresponding to the instance configuration table.

[0038] Secondly, this application also provides a reasoning apparatus for a large language model, comprising:

[0039] The lookup module is used to respond to inference requests triggered within the input box of the large language model. It searches for pre-filled instances and corresponding decoding instances from all instances corresponding to the wafer-level chip according to the instance configuration table and the corresponding requirement configuration table. Both the instance configuration table and the corresponding requirement configuration table are built on the basis of the existing wafer-level chip architecture.

[0040] The pre-filling module is used to pre-fill inference requests using pre-filled instances, storing the data required by the inference request into the cache corresponding to the pre-filled instance;

[0041] The decoding module is used to decode the data required for the inference request based on the decoding instance, and use the decoded result as the inference result of the large language model.

[0042] The output module is used to output the inference results through the output box of the large language model.

[0043] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any embodiment of the reasoning method of the large language model in the first aspect described above.

[0044] Fourthly, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the content of any embodiment of the reasoning method for the large language model in the first aspect described above.

[0045] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the content of any embodiment of the reasoning method for the large language model in the first aspect described above.

[0046] The aforementioned pre-filled decoding overlap execution LLM computation method based on high-performance wafer-level chips responds to inference requests triggered within the input box of the large language model. It searches for pre-filled instances and corresponding decoding instances from all instances corresponding to the wafer-level chip, according to the instance configuration table and the corresponding requirement configuration table. Both the instance configuration table and the corresponding requirement configuration table are built on the existing wafer-level chip architecture. The pre-filled instances are used to pre-fill the inference request, storing the data required by the inference request in the cache corresponding to the pre-filled instances. The decoding instances are used to decode the data required by the inference request, and the resulting decoding result is used as the inference result of the large language model. Finally, the inference result is output through the output box of the large language model. This method constructs an instance configuration table and a corresponding requirement configuration table based on the existing wafer-level chip architecture. When responding to large language model inference requests, it can quickly match pre-filled instances and decoding instances. The pre-filled instances store the data required for inference in advance in a high-speed cache, while the decoding instances are dedicated to performing data decoding and outputting results. This not only achieves precise adaptation between the wafer-level chip and the large language model inference process, significantly reducing data access latency and task scheduling overhead, but also improves the targeting and efficiency of inference computation through clear instance division of labor. This accelerates the inference response speed of the large language model, optimizes the efficiency of inference result output, and fully leverages the hardware architecture advantages of the wafer-level chip, improving overall resource utilization. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the wafer-level chip architecture in one embodiment;

[0049] Figure 2 This is a schematic diagram of the reasoning process of a unified LLM service system in one embodiment;

[0050] Figure 3 This is a schematic diagram of the reasoning process of a decomposed LLM service system in one embodiment.

[0051] Figure 4 This is a schematic diagram of the execution flow of a split LLM service system in one embodiment;

[0052] Figure 5 This is a schematic diagram of the structure of a wafer-level chip in one embodiment;

[0053] Figure 6This is a diagram illustrating the application environment of a reasoning method for a large language model in one embodiment.

[0054] Figure 7 This is a flowchart illustrating the reasoning method of a large language model in one embodiment;

[0055] Figure 8 This is a flowchart illustrating the reasoning method of a large language model in one embodiment;

[0056] Figure 9 This is a flowchart illustrating the reasoning method of a large language model in one embodiment;

[0057] Figure 10 This is a schematic diagram of the queues of various instances in a wafer-level chip in one embodiment;

[0058] Figure 11 This is a flowchart illustrating the reasoning method of a large language model in one embodiment;

[0059] Figure 12 This is a flowchart illustrating the reasoning method of a large language model in one embodiment;

[0060] Figure 13 This is a flowchart illustrating the reasoning method of a large language model in one embodiment;

[0061] Figure 14 This is a flowchart illustrating the reasoning method of a large language model in one embodiment;

[0062] Figure 15 This is a flowchart illustrating the reasoning method of a large language model in one embodiment;

[0063] Figure 16 This is a flowchart illustrating the reasoning method of a large language model in one embodiment;

[0064] Figure 17 This is a flowchart illustrating the reasoning method of a large language model in one embodiment;

[0065] Figure 18 This is a structural block diagram of the reasoning device for a large language model in one embodiment. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0067] Before providing a detailed introduction to the technical solution of this application, let me first briefly introduce the technical background of this application.

[0068] In recent years, large language models (LLMs) have become a significant driving force in artificial intelligence, widely applied in scenarios such as dialogue, code generation, and intelligent agents. The parameter scale of LLMs has continued to grow to the hundreds of billions, significantly improving the model's generalization ability, but also posing unprecedented challenges to the computing, storage, and communication resources of the underlying hardware. Due to the slowdown in the progress of photolithography technology, the improvement of transistor density no longer meets the expectations of Moore's Law. Coupled with the limitation of photomask size (typically not exceeding 858 mm²), single-chip devices can no longer efficiently support the inference needs of LLMs. Against this backdrop, how to efficiently provide LLM inference services through multi-chip systems has gradually become a focus of attention.

[0069] Currently, GPU clusters are widely regarded as the most commonly used multi-chip systems due to their relatively mature software ecosystem, and they are widely used in LLM inference services. However, GPU clusters have the following problems: (1) Low computing power density: GPU clusters are composed of existing GPU products, and their external packaging and related auxiliary components bring additional volume overhead, which is a certain disadvantage in computing power density compared with dedicated chips. Therefore, the scale of GPU clusters is more limited by the scale wall, that is, because the interconnect bandwidth between nodes will decrease with the increase of communication link length, when the number of system nodes expands to a critical point, the communication quality is difficult to guarantee, and further expansion will become inefficient. In the LLM inference service scenario, this critical point usually corresponds to a cluster scale of no more than about 40 servers (i.e., about 320 GPUs). Therefore, the low computing power density combined with the scale limitation makes it difficult for GPU clusters to compete with dedicated multi-chip systems with higher integration in terms of overall computing power. (2) Low communication bandwidth between cluster nodes: Even with the most advanced communication protocols, the communication bandwidth between nodes is far lower than that of on-chip metal interconnects. (3) Worse acceleration of neural network models: Since GPUs are general-purpose devices, their inference acceleration effect on neural network models is not as good as that of dedicated devices such as Neural Network Processing Units (NPUs) and Tensor Processing Units (TPUs). The main reason is that devices such as NPUs can control more fine-grained operations, thereby achieving better hardware resource utilization.

[0070] During the wafer-level chip design process, advanced packaging methods (such as CoWoS, Embedded Multi-Die Interconnect Bridge (EMIB), and fan-out packaging technology) can be used to achieve high-density integration of multiple NPU chips and high-speed device-to-device communication (D2D) interconnection, providing a multi-chip system solution that can address the problems existing in the aforementioned GPU cluster systems. Compared with the most advanced GPU clusters currently available, it can achieve tens of times more computing power and storage capacity, as well as nearly ten times the D2D interconnect bandwidth.

[0071] Figure 1 This is a schematic diagram of a wafer-level chip architecture, which comprises three layers: the wafer, the NPU chip, and the computing core. The wafer consists of multiple NPU chips, employing a mesh topology communication structure and using a D2D interconnect interface for communication between the NPU chips. The NPU chip consists of a computing chip composed of multiple computing cores and Dynamic Random Access Memory (DRAM), communicating via a Network-on-Chip (NoC). The computing core mainly consists of an array of computing units, vector computing units, a controller, and an on-chip network communication interface.

[0072] Wafer-level chips (WLS) offer two key architectural advantages: fine-grained operational control and high D2D interconnect bandwidth. Firstly, WLS employs a chip-level architecture similar to an NPU, supporting dynamic tensor decomposition and product engineering (PE) level execution control. This enables fine-grained parallelism, improves pipeline overlap, reduces computational redundancy, and enhances hardware resource utilization. Secondly, the D2D interconnect bandwidth is achieved through an intermediary layer directly connecting the compute chip and DRAM, minimizing signal attenuation and latency, resulting in faster data transmission, shorter communication distances, and higher signal integrity. However, WLS also faces topological and physical limitations. Two-dimensional (2D) mesh topologies suffer from tail latency due to varying communication hop counts, leading to inconsistent data arrival times. Furthermore, the shape of each instance is constrained by collective communication requirements.

[0073] Because wafer-level chips represent a novel type of large-scale multi-chip system, they differ significantly from existing GPU clusters or dedicated accelerators. Currently, there is a lack of LLM inference methods tailored to this new hardware architecture, resulting in the underutilization of its architectural advantages (such as fine-grained parallel control and high D2D interconnect bandwidth), while its inherent limitations (such as tail latency and collective communication constraints caused by two-dimensional mesh topologies) are difficult to overcome effectively. Therefore, designing scheduling and optimization methods that are aware of architectural characteristics and fully utilize hardware potential to truly transform the massive computing, storage, and communication resources provided by wafer-level chips into a significant improvement in end-to-end LLM inference performance remains a pressing problem.

[0074] In related technologies, LLM inference methods are designed for GPU clusters and are divided into two main categories: unified LLM service systems and decomposed LLM service systems.

[0075] For a unified LLM service system, the two key stages of batched inference requests are processed simultaneously within a single GPU node (typically configured with 8 GPUs): the prefill stage and the decode stage. In this system, inference requests are processed in batches. For example, if a GPU node's hardware resources support parallel processing of two requests, the prefill stage for both requests will be completed in parallel on that node, followed immediately by their decode stage. Subsequently, the node will schedule the next two requests and execute them in the same order. In other words, the execution mode within each node is serial batching with intra-batch parallelism. Simultaneously, multiple GPU nodes work independently and in parallel, with each node processing request batches without interference, thereby achieving an overall throughput improvement at the system level.

[0076] Migrating a unified LLM service system directly to wafer-level chips is relatively easy. Specifically, a two-dimensional array (such as 2*4 or 4*2 NPUs) composed of multiple NPU chip units can be viewed as a functionally equivalent computing node, with a logical role similar to a single GPU node. In this way, scheduling and task partitioning strategies originally based on GPU nodes can be directly applied to wafer-level chips. Through this analogy, the pre-filling and decoding processes of batch processing requests can be uniformly scheduled and executed in parallel within the NPU array.

[0077] However, in the inference process of large language models, the pre-filling stage only needs to process the complete input sequence once, while the decoding stage requires multiple rounds of iterative generation, and its cumulative time is often significantly longer than that of the pre-filling stage. When the request arrival rate is low, a single instance can sequentially complete the pre-filling and decoding process of a batch of requests, and then receive and process subsequent batches of requests, in which case there is no obvious conflict between the two. However, when the request arrival rate becomes high, the situation becomes complicated. Figure 2 This diagram illustrates the inference process of a unified LLM service system. While the instance is still processing the decoding phase of batch n, a new request for batch n+1 may have already arrived, causing pre-filling-decoding interference. In this situation, a trade-off must be made between prioritizing pre-filling and prioritizing decoding. If prioritizing pre-filling is chosen, the decoding process of the current batch n (Bn) must be forced to pause, waiting for the pre-filling phase of batch n+1 (Bn+1) to complete before resuming. This directly extends the overall decoding time of batch n, thus reducing decoding performance. If prioritizing decoding is chosen, the request for batch n+1 is forced to wait until the decoding of batch n is completely completed. While this ensures the decoding efficiency of the previous batch, the pre-filling phase of batch n+1 is delayed, resulting in a longer perceived response delay for the user and reduced pre-filling performance. It should be noted that in online LLM service scenarios, this typically occurs during a period of high request rates.

[0078] Furthermore, different instances of the unified LLM service system run independently, with almost no communication between instances. This design results in the high D2D bandwidth provided by the wafer-level chip not being fully utilized, and its advantages in cross-chip data transfer cannot be translated into actual performance improvements. Moreover, even when executing batch requests within a single instance, the fine-grained operation features supported by the wafer-level chip are not fully utilized. Specifically, intra-batch operations lack sufficient fine-grained parallelism and pipeline overlap optimization, resulting in inefficient scheduling and utilization of chip computing resources and memory bandwidth, thus limiting overall resource utilization and failing to unlock performance potential.

[0079] For decomposed LLM service systems, the inference process also includes a pre-filling phase and a decoding phase. Figure 3This diagram illustrates the inference process of a decomposed LLM service system. In the pre-filling phase, upon receiving an inference request, the LLM processes all input tokens in parallel and generates the first output token in the initial iteration. The computation process in this phase is similar to traditional forward propagation, primarily involving computationally intensive operations. The decoding phase takes the last generated token and its corresponding key-value (KV) cache as input and iteratively generates subsequent tokens until a termination token is generated. The frequent access to the KV cache in this phase places high demands on memory bandwidth and capacity, making it a memory bottleneck stage.

[0080] Because the two stages of the LLM inference process have different properties, the state-of-the-art LLM service system designed for GPU clusters adopts a separate mapping strategy to improve service quality. Specifically, the GPU cluster is divided into multiple groups, and each group deploys an LLM instance (the instance is divided into two types: pre-filled instance and decoding instance, and each type of instance has multiple groups). The corresponding execution strategy is applied to the specific processing stage (pre-filled is computationally intensive and the batch size is small; decoding is memory-intensive and the batch size is large). Figure 4 This is a schematic diagram of the execution flow of a split LLM service system. Its central controller is responsible for allocating requests to the pre-fill pool and the decoding pool; the machines in the pre-fill pool are responsible for executing pre-fill instances, and the machines in the decoding pool are responsible for executing decoding instances.

[0081] To migrate a discrete LLM service system to a wafer-level chip, different NPU chip arrays need to be configured as either pre-filled instances or decoded instances. Since the KV cache is transferred between pre-filled instances and their nearest decoded instances, the decoded instances are placed in the middle, and the pre-filled instances are placed on either side. Figure 5 This is a schematic diagram of a wafer-level chip structure. As shown in the diagram, each pre-fill instance is paired with its nearest decoding instance, for example, pre-fill instance 1 - decoding instance 1, pre-fill instance 3 - decoding instance 1, pre-fill instance 2 - decoding instance 2, and pre-fill instance 4 - decoding instance 2. Each request is executed on a pair of instances. For example, if request R is assigned to pre-fill instance 1 for the pre-filling stage, then it will subsequently be executed on decoding instance 1 for the decoding stage.

[0082] However, in a split system, the primary responsibility of pre-filled instances is to perform parallel computation of the input sequence and generate key-value (KV) caches. These KV caches are then transferred to the decoding instance to support subsequent iterative inference. Since the KV caches are almost entirely transferred to the decoding instance, after completing computation, the pre-filled instance's local memory resources are rarely used for storing data other than model weights, resulting in a significant amount of DRAM storage resources being idle. For example, with a large language model like the LLaMA3-70B model and 8 NPUs, the pre-filled instance's memory utilization rate is less than 30%. For smaller LLMs, the memory utilization is even lower. In scenarios with high user request rates, the KV cache becomes a significant factor limiting service quality. Lower available DRAM capacity means that fewer user requests can be processed in parallel.

[0083] In addition, under the discrete system, the topology and physical layout constraints of the wafer-level chip will be further amplified. The following is an example of executing the LLaMA-30B model on the wafer-level chip. Under this large language model, through experiments and analysis, it can be found that the optimal size of both pre-filled instances and decoding instances is 4 NPU chips (2×2 array), and according to the balance of request traffic, the ratio of pre-filled instances to decoding instances is about 3:1, that is, 9 pre-filled instances and 3 decoding instances need to be configured. Under this instance division, the system will face two prominent problems: (1) Due to the influence of resource allocation, some pre-filled instances cannot be physically adjacent to the decoding instances, resulting in significant tail latency. For example, pre-filled instances 4 and 9 are not adjacent to any decoding instances, and their KV caches must be transmitted through multiple hops to reach the target decoding instance (decoding instance 3), which ultimately results in a communication latency much higher than that of pre-filled instances that are directly adjacent to the decoding instances. (2) Since each pre-filled instance only transmits its KV cache to its nearest decoding instance, this may lead to an imbalance in the decoding workload. For example, in the above configuration, decoding instance 1 and decoding instance 2 each only need to handle tasks from two pre-filled instances, while decoding instance 3 needs to handle requests from five pre-filled instances simultaneously. This load imbalance can cause some decoding instances to become bottlenecks, further limiting the overall system throughput and response latency.

[0084] In summary, the problem with unified systems is the existence of pre-filling-decoding interference, while the problem with decomposed systems is low memory resource utilization.

[0085] To address the aforementioned issues, this application provides a pre-filling and decoding concurrent execution LLM computation method based on high-performance wafer-level chips. This method fully leverages the fine-grained operation advantages of wafer-level chips to achieve logical separation and collaborative management of the pre-filling and decoding stages within a single instance. Specifically, each instance maintains two independent queues: one for processing pre-filling tasks and one for processing decoding tasks. By introducing a fine-grained pipeline orchestration mechanism within the instance, the system can dynamically schedule pre-filling and decoding batches, ensuring that their single-iteration execution times remain approximately equal. Furthermore, this device fully utilizes the high D2D interconnect bandwidth of wafer-level chips, allowing the transmission of KV caches (cross-instance scheduling requests) between instances. This cross-instance KV cache communication not only improves the flexibility of resource scheduling but also enables dynamic workload balancing globally, effectively alleviating bottleneck problems caused by uneven request allocation among some instances. The technical solution of this application will be described in detail below.

[0086] The reasoning method for large language models provided in this application can be applied to, for example... Figure 6 The application environment shown is as follows. This wafer-level chip with a large language model consists of multiple NPU chips. By grouping these NPU chips, multiple instances are obtained. Each instance can be used for pre-filling operations or decoding operations.

[0087] In one exemplary embodiment, such as Figure 7 As shown, a reasoning method for large language models is provided, which is then applied to... Figure 6 Taking a wafer-level chip with a large language model deployed in it as an example, the explanation includes the following steps S101 to S104. Wherein:

[0088] S101, in response to an inference request triggered within the input box of the large language model, searches for pre-filled instances and corresponding decoding instances from all instances corresponding to the wafer-level chip according to the instance configuration table and the corresponding requirement configuration table; both the instance configuration table and the corresponding requirement configuration table are built on the basis of the existing wafer-level chip architecture.

[0089] The instance configuration table refers to the size configuration information of each instance in the existing wafer-level chip architecture. Multiple instances are obtained by configuring multiple NPU chips in the existing wafer-level chip architecture. The corresponding requirement configuration table refers to the configuration for overlapping pre-filling requests and decoding requests under each instance. Both the instance configuration table and the corresponding requirement configuration table are configured before the wafer-level chip processes inference requests.

[0090] In this embodiment, when a user inputs an inference request through the input box of the large language model, the inference request enters the wafer-level chip. The wafer-level chip can determine the memory usage of each instance according to the instance configuration table and the corresponding requirement configuration table. Based on the memory usage of each instance, it finds the instance with the least memory usage from all instances and uses it as the pre-filled instance in the wafer-level chip. Then, based on the number of instances that cross between other instances and the pre-filled instance, it finds the decoding instance corresponding to the pre-filled instance from the other instances in the wafer-level chip.

[0091] Alternatively, the wafer-level chip can input the instance configuration table and the corresponding demand configuration table into a preset instance allocation model. The instance allocation model can extract features from the instance configuration table and the corresponding demand configuration table, and allocate pre-filled instances and corresponding decoding instances from all instances corresponding to the wafer-level chip based on the extracted feature information and inference requests. This application embodiment does not limit the specific method for searching for pre-filled instances and corresponding decoding instances from all instances corresponding to the wafer-level chip.

[0092] S102, use the pre-filled instance to perform a pre-filling operation on the inference request, and store the data required by the inference request into the cache corresponding to the pre-filled instance.

[0093] In this embodiment, after obtaining the pre-filled instance, the pre-filled instance in the wafer-level chip can parse the inference request to determine the data information required for the inference request. For example, if the inference request is to query orders for a certain period, the data information required for the inference request may include: user address, query time range, and data type, etc. Then, a data read request is initiated to the order database through the distributed storage interface. This data read request is generated according to the data required by the inference request to extract the data required by the inference request from the order database, encapsulate the data required by the inference request into a key-value pair format, and use the cache corresponding to the pre-filled instance to store the encapsulated data required by the inference request in the specified address range.

[0094] It should be noted that pre-filled instances can process all input tokens in parallel and generate the first token and the data required for inference requests in the cache.

[0095] S103, based on the decoding instance, perform decoding operations on the data required for the inference request, and use the obtained decoding result as the inference result of the large language model.

[0096] In this embodiment, after obtaining the decoding instance, the decoding instance in the wafer-level chip can receive the pre-filling operation completion signal of the pre-filling instance, and then read the data required for the inference request from the cache of the pre-filling instance through the internal high-speed bus of the wafer-level chip. Then, it performs a decoding operation on the data required for the inference request to obtain the decoding result corresponding to the inference request. The decoding result corresponding to this inference request is the inference result of the large language model.

[0097] It should be noted that the decoding operation includes multiple decoding stages. After the first decoding stage is performed on the data required for the inference request, a second token and new decoded data are generated. The new decoded data is stored in the cache corresponding to the decoding instance. If the decoding stage terminates, the decoding result is output directly. If the decoding stage does not terminate, a second decoding stage is performed on the data required for the inference request, and so on, until the decoding stage terminates.

[0098] S104 outputs the inference results through the output box of the large language model.

[0099] In this embodiment of the application, after obtaining the inference result, the format of the inference result can be optimized, and the format-optimized inference result can be transmitted to the output box associated with the input box of the large language model at the front end. The output box can then display the format-optimized inference result to the user.

[0100] In the aforementioned inference method of the large language model, in response to an inference request triggered within the input box of the large language model, a pre-filled instance and a corresponding decoding instance are searched from all instances corresponding to the wafer-level chip according to the instance configuration table and the corresponding requirement configuration table. Both the instance configuration table and the corresponding requirement configuration table are built on the basis of the existing wafer-level chip architecture. The pre-filled instance is used to pre-fill the inference request, storing the data required by the inference request into the cache corresponding to the pre-filled instance. The data required by the inference request is decoded based on the decoding instance, and the resulting decoding result is used as the inference result of the large language model. The inference result is output through the output box of the large language model. This method constructs an instance configuration table and a corresponding requirement configuration table based on the existing wafer-level chip architecture. When responding to large language model inference requests, it can quickly match pre-filled instances and decoding instances. The pre-filled instances store the data required for inference in advance in a high-speed cache, while the decoding instances are dedicated to performing data decoding and outputting results. This not only achieves precise adaptation between the wafer-level chip and the large language model inference process, significantly reducing data access latency and task scheduling overhead, but also improves the targeting and efficiency of inference computation through clear instance division of labor. This accelerates the inference response speed of the large language model, optimizes the efficiency of inference result output, and fully leverages the hardware architecture advantages of the wafer-level chip, improving overall resource utilization.

[0101] The following example illustrates the specific steps of retrieving pre-filled instances from all instances corresponding to the wafer-level chip based on the instance configuration table and the corresponding requirement configuration table. Figure 8 As shown, it includes:

[0102] S201, according to the instance configuration table and the corresponding requirement configuration table, sort the pre-filled memory usage of all instances in the wafer-level chip to obtain the pre-filled instance sorting result.

[0103] The pre-filled instance sorting result represents the sorting result of the processing speed of the pre-filled operations under each instance.

[0104] In this embodiment, the corresponding requirement configuration table includes the pre-fill queue QPi and the decoding queue QD1 for that instance. The wafer-level chip can determine the existing pre-fill quantity in each instance according to the instance configuration table and the corresponding requirement configuration table. Based on the existing pre-fill quantity in each instance, the pre-fill memory usage of each instance is determined. The pre-fill instances are then sorted according to their pre-fill memory usage to determine the pre-fill instance sorting result corresponding to the wafer-level chip.

[0105] S202, based on the sorting results of pre-filled instances, search for pre-filled instances from all instances corresponding to the wafer-level chip.

[0106] In this embodiment, the pre-filled instance ranked first in the pre-filled instance sorting result can quickly complete the existing pre-filling operation, while the other pre-filled instances are processed more slowly. Therefore, the wafer-level chip can use the instance ranked first in the pre-filled instance sorting result as the pre-filled instance.

[0107] In one embodiment, the specific content of searching for pre-filled instances from all instances corresponding to the wafer-level chip according to the pre-filled instance sorting result includes:

[0108] Get the first instance in the pre-filled instance sorting results and use the first instance as the pre-filled instance in the wafer-level chip; where the first instance is the instance with the smallest memory usage in the pre-filled instance sorting results.

[0109] Since the first instance is the one with the smallest memory footprint in the pre-filled instance sorting results, the wafer-level chip can directly use the first instance in the pre-filled instance sorting results as the pre-filled instance after obtaining the pre-filled instance sorting results.

[0110] In the aforementioned reasoning method for large language models, the pre-filled memory usage of all instances in the wafer-level chip is sorted according to the instance configuration table and the corresponding requirement configuration table to obtain a pre-filled instance sorting result. Based on this sorting result, pre-filled instances are then searched from all instances corresponding to the wafer-level chip. This method sorts the pre-filled memory usage of all instances in the wafer-level chip using the instance configuration table and the corresponding requirement configuration table for each instance. Based on the memory usage sorting result, instances that best meet the memory requirements of the inference task are matched first, reducing memory resource waste and data access conflicts during pre-filling operations, and further improving the targeting of data pre-storage and the utilization rate of hardware resources during large language model inference.

[0111] In one embodiment, such as Figure 9 As shown, this document describes the sorting operation performed on the pre-filled memory usage of all instances in the wafer-level chip according to the instance configuration table and the corresponding requirement configuration table to obtain the pre-filled instance sorting results. The specific content includes:

[0112] S301, according to the instance configuration table and the corresponding requirement configuration table, obtain the number of pre-filled items to be processed for each instance in the wafer-level chip and the time of inference requests to be processed in each instance.

[0113] Figure 10 The diagram shows the queues for each instance in a wafer-level chip. As can be seen from the diagram, instance 1 includes a pre-fill queue QP1 and a decoding queue QD1; instance 2 includes a pre-fill queue QP2 and a decoding queue QD2; instance 3 includes a pre-fill queue QP3 and a decoding queue QD3; instance 4 includes a pre-fill queue QP4 and a decoding queue QD4; instance 5 includes a pre-fill queue QP5 and a decoding queue QD5; and instance 6 includes a pre-fill queue QP6 and a decoding queue QD6.

[0114] In this embodiment of the application, the number of pre-filled items to be processed for each instance refers to the number of pre-filled operations that have already been performed in that instance. Figure 10 The number of boxes under each instance of QPi represents the number of pre-filled items to be processed. In the figure, QP1, QP2, QP3, QP5, and QP6 correspond to 1 pre-filled item to be processed, and QP4 corresponds to 2 pre-filled items to be processed.

[0115] The time for pending inference requests in each instance refers to the request time for existing pre-filled operations within that instance. Figure 10 In each instance, the request sequence number of QPi represents the request time. The larger the request sequence number, the later the request arrived. In the figure, the request sequence number of QP1 is 1, which means that the request time was earlier.

[0116] The wafer-level chip can parse the instance configuration table and the corresponding demand configuration table to determine the memory usage of each instance. Then, by further analyzing the memory usage of each instance, it can determine the number of pre-filled requests to be processed for each instance and the time of inference requests to be processed in each instance.

[0117] S302, sort the quantities of each pre-filled item to be processed in ascending order to obtain the quantity sorting result.

[0118] In this embodiment, a larger number of pre-filled instances indicates a higher memory usage for the corresponding instances, and a smaller number of pre-filled instances indicates a lower memory usage for the corresponding instances. Therefore, instances with fewer pre-filled instances can be sorted in ascending order, with those having more pre-filled instances sorted first, to obtain a sorting result. For example, if the sorting result is QP1, QP2, QP3, QP5, and QP6 first, followed by QP4, the order of QP1, QP2, QP3, QP5, and QP6 cannot be determined solely from the sorting result.

[0119] S303, sort the time of each pending inference request in order from earliest to latest based on the quantity sorting result to obtain the pre-filled instance sorting result.

[0120] In this embodiment, since the quantity sorting result cannot determine the sorting result of multiple instances with the same number of pre-filled instances to be processed, the wafer-level chip further sorts them according to the inference request time of each instance to be processed, placing instances with smaller request numbers first and instances with larger request numbers last, thus obtaining the pre-filled instance sorting result. This pre-filled instance sorting result is: QP1, QP2, QP3, QP5, QP6, QP4.

[0121] As can be seen from the above sorting process, the sorting priority of the number of pre-filled items to be processed is greater than the sorting priority of the inference request time to be processed.

[0122] Additionally, it's important to emphasize that upon receiving a new inference request, a corresponding new pre-fill request arises. The wafer-level chip can directly allocate this new pre-fill request to the first instance in the pre-fill instance sorting result and update the sorting order to ensure efficient allocation of subsequent requests. For each QPi queue, its internal pre-fill requests must be processed according to a first-come, first-served strategy.

[0123] In the inference method of the aforementioned large language model, the number of pending pre-filled instances and the pending inference request time for each instance in the wafer-level chip are obtained according to the instance configuration table and the corresponding requirement configuration table. The pending pre-filled instances are then sorted in ascending order to obtain a quantity sorting result. Finally, the pending inference request times are sorted in ascending order based on the quantity sorting result to obtain the pre-filled instance sorting result. This method accurately obtains the number of pending pre-filled instances and the pending inference request time for each instance in the wafer-level chip through the instance configuration table and the corresponding requirement configuration table. It first sorts the pending pre-filled instances in ascending order, and then, based on this result, sorts them again in ascending order by pending inference request time to obtain the pre-filled instance sorting result. This enables multi-dimensional and accurate filtering of pre-filled instances, avoiding resource allocation imbalances caused by single-dimensional sorting.

[0124] The above embodiments are introductions to finding pre-filled instances. The following describes the process of finding decoding instances. Because different instances operate differently, a request that completes the pre-filling stage in QPi may not necessarily continue decoding in QDi. Utilizing the high D2D bandwidth of the wafer-level chip, we can allocate decoding requests to more suitable decoding instances.

[0125] In one embodiment, such as Figure 11 As shown, the specific content of searching for the decoding instance corresponding to the pre-filled instance from all instances corresponding to the wafer-level chip according to the instance configuration table and the corresponding requirement configuration table includes:

[0126] S401, according to the pre-filled instances and instance configuration table in the wafer-level chip, search for at least one candidate decoding instance from all instances corresponding to the wafer-level chip.

[0127] In this embodiment, the wafer-level chip has a D2D bandwidth exceeding the DRAM bandwidth. This means that, in the absence of D2D link contention, cross-NPU chip DRAM read / write operations are primarily limited by the DRAM bandwidth itself. Therefore, once pre-filled instances are identified within the wafer-level chip, the schedulable range of decoding instances is limited by the instance performing its pre-filling phase, the D2D bandwidth, and the DRAM bandwidth. Thus, the wafer-level chip can search for instances with a smaller instance span than the pre-filled instances from all instances corresponding to the wafer-level chip as candidate decoding instances. For example, instances 2 and 3 can be selected as candidate decoding instances.

[0128] S402, perform time increment evaluation on each candidate decoding instance according to the corresponding requirement configuration table, and obtain the time increment evaluation result for each candidate instance.

[0129] In this embodiment, for each candidate decoding instance, the memory usage of that candidate decoding instance is determined. Based on the memory usage of that candidate decoding instance, the time required for that candidate decoding instance to perform decoding operations on inference requests is predicted, and this time is determined as the time increment evaluation result corresponding to that candidate instance.

[0130] S403, the candidate instance corresponding to the smallest time increment evaluation result is used as the decoding instance corresponding to the pre-filled instance.

[0131] In this embodiment, after obtaining the time increment evaluation result corresponding to each candidate instance, the wafer-level chip can find the smallest time increment evaluation result from multiple time increment evaluation results, and use the candidate instance corresponding to the smallest time increment evaluation result as the decoding instance corresponding to the pre-filled instance.

[0132] In the inference method of the aforementioned large language model, at least one candidate decoding instance is searched from all instances corresponding to the wafer-level chip according to the pre-filled instances and instance configuration table in the wafer-level chip. A time increment evaluation is performed on each candidate decoding instance according to the corresponding requirement configuration table to obtain the time increment evaluation result for each candidate instance. The candidate instance with the smallest time increment evaluation result is selected as the decoding instance corresponding to the pre-filled instance. This method accurately selects at least one candidate decoding instance based on the pre-filled instances and instance configuration table in the wafer-level chip, and then performs a time increment evaluation on the candidate decoding instances based on the requirement configuration table corresponding to each instance, selecting the candidate instance with the smallest time increment evaluation result as the decoding instance matching the pre-filled instance. This achieves optimal collaborative adaptation between the pre-filled instances and the decoding instances, effectively reducing data transmission latency between instances and the additional time consumption of task scheduling.

[0133] In one embodiment, such as Figure 12 As shown, the specific content of finding at least one candidate decoding instance from all instances corresponding to the wafer-level chip according to the pre-filled instances and instance configuration table in the wafer-level chip is described below. This specific content includes:

[0134] S501 calculates the number of cross-instance relationships between a pre-filled instance and any other instance in a wafer-level chip.

[0135] In this embodiment, the wafer-level chip can determine the position of each instance within the wafer-level chip based on an instance configuration table. When pre-filled instances are obtained within the wafer-level chip, the number of cross-instance relationships between a pre-filled instance and any other instance can be determined according to the position of each instance and the position of the pre-filled instance. Figure 10For example, the number of cross-instances between pre-filled instance 1 and instance 2 is 1, and the number of cross-instances between pre-filled instance 1 and instance 5 is 2. The number of cross-instances between a pre-filled instance and any other instance can be represented as Distance(insi,insj), which is the number of instances along the shortest path between the binding centers.

[0136] S502, instances corresponding to the number of cross-instances less than or equal to a preset threshold are selected as candidate decoding instances in the wafer-level chip; the preset threshold represents the maximum number of cross-instances that can be accessed via cache between cross-instances without congestion.

[0137] The preset threshold can be represented as D2D_Bandwidth / DRAM_Bandwidth, which represents the maximum distance for cross-instance DRAM access without D2D congestion in time-division multiplexing mode. Here, D2D_Bandwidth represents the bandwidth for direct transmission between instances, and DRAM_Bandwidth represents the data transfer rate between the memory module and the processor.

[0138] In this embodiment, the wafer-level chip can compare the number of instances between a pre-filled instance and any other instance with a preset threshold. If Distance(insi,insj) <= D2D_Bandwidth / DRAM_Bandwidth, the instance can be used as a candidate decoding instance. For example, assuming D2D_Bandwidth / DRAM_Bandwidth = 2 and the pre-filled instance is 1, instances 1, 2, 3, 4, and 5 can be used as candidate decoding instances.

[0139] In the inference method of the aforementioned large language model, the number of cross-instances between a pre-filled instance and any other instance in the wafer-level chip is calculated. Instances with a number of cross-instances less than or equal to a preset threshold are selected as candidate decoding instances in the wafer-level chip. The preset threshold represents the maximum number of cross-instances that can be accessed via cache without congestion. This method, by calculating the number of cross-instances between a pre-filled instance and any other instance in the wafer-level chip and selecting instances with a number less than or equal to the preset threshold as candidate decoding instances, avoids cache access congestion and data transmission bandwidth bottlenecks caused by an excessive number of cross-instances. Furthermore, the quantization constraint of the preset threshold ensures efficient cross-instance communication between candidate decoding instances and pre-filled instances, reducing latency and packet loss risks during data transmission between instances.

[0140] The following example illustrates the specific details of obtaining the time increment evaluation result for each candidate decoding instance based on the corresponding requirement configuration table. Figure 13As shown, it includes:

[0141] S601, based on the requirement configuration table corresponding to each candidate decoding instance, determine the prediction time increment delay time for each candidate decoding instance to perform decoding operations on the data required for the inference request.

[0142] In this embodiment, for any candidate decoding instance, the wafer-level chip can determine the decoding memory space occupied in that candidate decoding instance according to the requirement configuration table corresponding to the candidate decoding instance. Based on the decoding memory space, it can determine the prediction time increment delay required after decoding the data required for the inference request. Alternatively, the wafer-level chip can also determine a first prediction time by decoding the data required for an existing inference request using the candidate decoding instance, and determine a second prediction time by decoding both the data required for the existing inference request and the data required for the inference request using the candidate decoding instance. The difference between the second prediction time and the first prediction time is calculated to obtain the prediction time increment delay corresponding to the candidate decoding instance.

[0143] S602, the prediction time increment delay time corresponding to each candidate decoding instance is used as the time increment evaluation result.

[0144] In this embodiment of the application, after obtaining the predicted time increment delay time corresponding to each candidate decoding instance, the predicted time increment delay time corresponding to each candidate decoding instance can be directly used as the corresponding time increment evaluation result.

[0145] In the inference method of the aforementioned large language model, based on the requirement configuration table corresponding to each candidate decoding instance, the prediction time increment delay time for each candidate decoding instance to perform decoding operations on the data required for the inference request is determined; the prediction time increment delay time corresponding to each candidate decoding instance is used as the time increment evaluation result. This method, through the requirement configuration table corresponding to each candidate decoding instance, can achieve a quantitative evaluation of the decoding efficiency of the candidate decoding instances, ensuring that the evaluation results are highly consistent with the actual decoding scenario.

[0146] In one embodiment, such as Figure 14 As shown, the specific details of the prediction time increment delay time for each candidate decoding instance to perform decoding operations on the data required for the inference request, determined based on the requirement configuration table corresponding to each candidate decoding instance, include:

[0147] S701, for any candidate decoding instance, based on the requirement configuration table corresponding to the candidate decoding instance, obtain the first predicted decoding time for the candidate decoding instance to perform decoding operations on the data required by the existing inference request; and obtain the second predicted decoding time for the candidate decoding instance to perform decoding operations on the data required by the existing inference request and the data required by the inference request.

[0148] In this embodiment, for any candidate decoding instance, the wafer-level chip can determine the existing inference requests within that candidate decoding instance through the demand configuration table corresponding to that candidate decoding instance. The candidate decoding instance performs decoding operations on the data required by the existing inference requests to determine the first predicted decoding time Ti under the workload. Furthermore, the candidate decoding instance performs decoding operations on both the data required by the existing inference requests and the data required by the inference requests to determine the second predicted decoding time Ti′ under the workload.

[0149] S702, the time difference between the second prediction decoding time and the first prediction decoding time is used as the prediction time increment delay time corresponding to the candidate decoding instance.

[0150] In this embodiment, after obtaining the first prediction decoding time Ti and the second prediction decoding time Ti′, the wafer-level chip can calculate the difference between the second prediction decoding time Ti′ and the first prediction decoding time Ti, which can be expressed as: This difference is the prediction time increment delay time corresponding to the candidate decoding instance.

[0151] In the inference method of the aforementioned large language model, for any candidate decoding instance, based on the requirement configuration table corresponding to the candidate decoding instance, the first predicted decoding time for the candidate decoding instance to decode the data required for the existing inference request is obtained; and the second predicted decoding time for the candidate decoding instance to decode the data required for the existing inference request and the data required for the inference request is obtained; the time difference between the second predicted decoding time and the first predicted decoding time is used as the predicted time increment delay time corresponding to the candidate decoding instance. For any candidate decoding instance, this method, based on its corresponding requirement configuration table, obtains the first predicted decoding time for the instance to decode the existing inference request data, and then obtains the second predicted decoding time for it to simultaneously decode the existing inference request data and the current inference request data. The time difference between the two is used as the predicted time increment delay time corresponding to the candidate decoding instance. This method accurately captures the additional delay cost after the candidate instance undertakes the current decoding task through quantitative comparison, ensuring the accuracy of the time increment evaluation results.

[0152] In a detailed embodiment, such as Figure 15 As shown, the reasoning method of this large language model includes:

[0153] S801, in response to an inference request triggered within the input box of the large language model, obtains the number of pre-filled instances to be processed and the time of inference requests to be processed in each instance in the wafer-level chip according to the instance configuration table and the corresponding requirement configuration table.

[0154] S802, sort the quantities of each pre-filled item to be processed in ascending order to obtain the quantity sorting result;

[0155] S803, sort the time of each pending inference request in order from earliest to latest, based on the quantity sorting result, to obtain the pre-filled instance sorting result;

[0156] S804, Obtain the first instance in the pre-filled instance sorting result, and use the first instance as the pre-filled instance in the wafer-level chip;

[0157] S805, calculates the number of cross-instance relationships between a pre-filled instance and any other instance in a wafer-level chip;

[0158] S806, the instances corresponding to the number of cross-instances that are less than or equal to a preset threshold are used as candidate decoding instances in the wafer-level chip;

[0159] S807, for any candidate decoding instance, based on the requirement configuration table corresponding to the candidate decoding instance, obtain the first predicted decoding time for the candidate decoding instance to perform decoding operations on the data required by the existing inference request; and obtain the second predicted decoding time for the candidate decoding instance to perform decoding operations on the data required by the existing inference request and the data required by the inference request.

[0160] S808, the time difference between the second prediction decoding time and the first prediction decoding time is used as the prediction time increment delay time corresponding to the candidate decoding instance;

[0161] S809, the prediction time increment delay time corresponding to each candidate decoding instance is used as the time increment evaluation result;

[0162] S810 uses pre-filled instances to pre-fill inference requests, storing the data required by the inference requests into the cache corresponding to the pre-filled instances;

[0163] S811, based on the decoding instance, performs decoding operations on the data required for the inference request, and uses the obtained decoding result as the inference result of the large language model;

[0164] S812 outputs the inference results through the output box of the large language model.

[0165] Figure 16This is a flowchart illustrating the reasoning method of a large language model. The method includes: S11: Triggering a reasoning request through the input box of the large language model; S12: Allocating pre-filling requests to pre-filling instances based on the instance configuration table and the corresponding requirement configuration table, and performing pre-filling operations through the pre-filling instances; S13: Allocating decoding requests to decoding instances based on the instance configuration table and the corresponding requirement configuration table, and performing decoding operations through the decoding instances; S14: Determining whether the decoding operation has terminated. If yes, outputting the result; otherwise, continuing to execute S13 until the decoding operation terminates.

[0166] In the specific configuration process, two key issues need to be addressed. One is determining the optimal instance arrangement and the size of each instance, i.e., the instance configuration table, within the existing wafer-level chip architecture. The other is determining the optimal configuration scheme for each instance that supports pre-filling-decoding overlapping execution, i.e., the corresponding requirement configuration table.

[0167] The following example will be used to explain in detail the configuration process of the instance configuration table and the corresponding requirement configuration table. Figure 17 As shown, the process includes:

[0168] S901 determines multiple candidate instance configuration tables based on the hardware parameters of the existing wafer-level chip architecture as constraints.

[0169] In this embodiment, under the existing wafer-level chip architecture, the total number of chips and the global topology of the existing wafer-level chip architecture are used as constraints to list all feasible instance shapes (such as 2×2, 2×3, 2×4, etc.), resulting in all candidate instance configuration tables (ICLs). Each instance configuration table represents a unique combination of different instance shapes and their quantities. In step S902, a configuration process involving pre-filling and decoding that completely overlaps is performed on each candidate instance configuration table to obtain the requirement configuration table corresponding to each candidate instance configuration table.

[0170] S902, for each candidate instance configuration table, obtains the optimal pre-filled configuration and decoding configuration.

[0171] During configuration, pre-filling can employ a block-based pre-filling technique, setting the block size to the average length of input tokens in the test dataset and processing one block at a time. This eliminates the need to consider the batch dimension of pre-filling requests, thus reducing exploration complexity. Decoding configuration can use zero-padding to align tokens of varying lengths within a batch with the longest token, using that length as the batch decoded token count for efficient matrix-based post-processing. Based on this, the optimal configuration that achieves complete overlap between pre-filling and decoding is found by traversing key parameters (such as the number of decoded tokens and batch size).

[0172] S903 uses the requirement configuration table corresponding to each candidate instance configuration table to guide instance operation and estimates the chip throughput corresponding to each candidate instance configuration table.

[0173] S904, the candidate instance configuration table with the lowest chip throughput is used as the instance configuration table, and the requirement configuration table corresponding to the candidate instance configuration table with the lowest chip throughput is used as the requirement configuration table corresponding to the instance configuration table.

[0174] In this embodiment, for each candidate instance configuration table ICL[i], the optimal pre-filled configuration and decoding configuration corresponding to the candidate instance configuration table are used to guide the execution of each instance, and the single-chip throughput of each instance is independently evaluated through workload operation. It should be emphasized that this stage does not involve inter-instance scheduling, but only focuses on the performance measurement within the instance. Then, a replication strategy is used to estimate the average per-chip throughput of the candidate instance configuration table ICL[i], and the candidate instance configuration table with the highest average per-chip throughput among all candidate instance configuration tables is selected as the globally optimal configuration, i.e., the instance configuration table, and the corresponding demand configuration table is obtained. For example, Figure 6 It includes an instance configuration table consisting of six 2×4 chip arrays.

[0175] In the aforementioned large language model inference method, multiple candidate instance configuration tables are determined based on the hardware parameters of the existing wafer-level chip architecture as constraints. For each candidate instance configuration table, pre-filling and decoding are performed in a fully overlapping manner to obtain the corresponding requirement configuration table. The requirement configuration table for each candidate instance configuration table guides instance execution, estimating the chip throughput for each candidate instance configuration table. The candidate instance configuration table with the lowest chip throughput is selected as the instance configuration table, and the requirement configuration table corresponding to the candidate instance configuration table with the lowest chip throughput is selected as the requirement configuration table corresponding to the instance configuration table. This method ensures the feasibility and architecture adaptability of the configuration scheme through hardware parameter constraints, maximizes the compression of inference task time overhead through overlapping pre-filling and decoding, and accurately locks the optimal configuration combination by minimizing chip throughput. This effectively avoids resource waste or inference bottlenecks caused by improper configuration, significantly improving the task execution efficiency, resource utilization rationality, and overall operational stability of wafer-level chips in large language model inference scenarios.

[0176] In one exemplary embodiment, such as Figure 18 As shown, a reasoning device for a large language model is provided, including: a lookup module 11, a pre-filling module 12, a decoding module 13, and an output module 14, wherein:

[0177] The lookup module 11 is used to respond to inference requests triggered in the input box of the large language model, and to look up pre-filled instances and corresponding decoding instances from all instances corresponding to the wafer-level chip according to the instance configuration table and the corresponding requirement configuration table; both the instance configuration table and the corresponding requirement configuration table are built on the basis of the existing wafer-level chip architecture.

[0178] Pre-filling module 12 is used to pre-fill inference requests using pre-filling instances, storing the data required by the inference requests into the cache corresponding to the pre-filling instances;

[0179] Decoding module 13 is used to perform decoding operations on the data required for the inference request based on the decoding instance, and use the obtained decoding result as the inference result of the large language model;

[0180] Output module 14 is used to output the inference results through the output box of the large language model.

[0181] In an exemplary embodiment, the search module includes: a sorting unit and a first search unit, wherein:

[0182] The sorting unit is used to sort the pre-filled memory usage of all instances in the wafer-level chip according to the instance configuration table and the corresponding requirement configuration table, and obtain the pre-filled instance sorting result.

[0183] The first lookup unit is used to search for pre-filled instances from all instances corresponding to the wafer-level chip according to the sorting results of the pre-filled instances.

[0184] The various modules in the reasoning apparatus of the aforementioned large language model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0185] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any embodiment of the reasoning method for the large language model described above.

[0186] In one embodiment, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the content of any embodiment of the reasoning method for the large language model described above.

[0187] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the content of any embodiment of the reasoning method for the large language model described above.

[0188] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0189] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A reasoning method for a large language model, characterized in that, The method, applied to a wafer-level chip with a large language model deployed, includes: In response to an inference request triggered within the input box of the large language model, a pre-filled instance and a corresponding decoding instance are searched from all instances corresponding to the wafer-level chip according to the instance configuration table and the corresponding requirement configuration table; the instance configuration table and the corresponding requirement configuration table are both built on the basis of the existing wafer-level chip architecture; The inference request is pre-populated using the pre-populated instance, and the data required by the inference request is stored in the cache corresponding to the pre-populated instance. Based on the decoding instance, the data required for the inference request is decoded, and the resulting decoding result is used as the inference result of the large language model. The inference result is output through the output box of the large language model.

2. The method according to claim 1, characterized in that, Based on the instance configuration table and the corresponding requirement configuration table, pre-filled instances are searched from all instances corresponding to the wafer-level chip, including: According to the instance configuration table and the corresponding requirement configuration table, the pre-filled memory usage of all instances in the wafer-level chip is sorted to obtain the pre-filled instance sorting result; Based on the sorting results of the pre-filled instances, the pre-filled instance is searched from all instances corresponding to the wafer-level chip.

3. The method according to claim 2, characterized in that, The step involves sorting the pre-filled memory usage of all instances in the wafer-level chip according to the instance configuration table and the corresponding demand configuration table to obtain the pre-filled instance sorting result, including: According to the instance configuration table and the corresponding requirement configuration table, obtain the number of pre-filled requests to be processed for each instance in the wafer-level chip and the time of inference requests to be processed in each instance. Sort the quantities of each pre-filled item to be processed in ascending order to obtain the quantity sorting result; The time of each pending inference request is sorted from earliest to latest based on the quantity sorting result to obtain the pre-filled instance sorting result.

4. The method according to claim 2, characterized in that, The step of searching for the pre-filled instance from all instances corresponding to the wafer-level chip according to the sorting result of the pre-filled instances includes: Obtain the first instance in the pre-filled instance sorting result and use the first instance as the pre-filled instance in the wafer-level chip; wherein, the first instance is the instance with the smallest memory usage in the pre-filled instance sorting result.

5. The method according to any one of claims 1-4, characterized in that, The step of searching for the decoding instance corresponding to the pre-filled instance from all instances corresponding to the wafer-level chip according to the instance configuration table and the corresponding requirement configuration table includes: Based on the pre-filled instances in the wafer-level chip and the instance configuration table, at least one candidate decoding instance is searched from all instances corresponding to the wafer-level chip; According to the corresponding requirement configuration table, time increment evaluation is performed on each candidate decoding instance to obtain the time increment evaluation result for each candidate instance. The candidate instance corresponding to the smallest time increment evaluation result is used as the decoding instance corresponding to the pre-filled instance.

6. The method according to claim 5, characterized in that, The step of searching for at least one candidate decoding instance from all instances corresponding to the wafer-level chip according to the pre-filled instances in the wafer-level chip and the instance configuration table includes: Calculate the number of cross-instances between a pre-filled instance and any other instance in the wafer-level chip; The instances corresponding to the number of cross-instances less than or equal to a preset threshold are selected as candidate decoding instances in the wafer-level chip; the preset threshold represents the maximum number of cross-instances that can be accessed via cache without congestion.

7. The method according to claim 5, characterized in that, The step of performing time increment evaluation on each candidate decoding instance according to the corresponding requirement configuration table to obtain the time increment evaluation result for each instance includes: Based on the requirement configuration table corresponding to each candidate decoding instance, the prediction time increment delay time for each candidate decoding instance to perform decoding operations on the data required for the inference request is determined. The prediction time increment delay time corresponding to each candidate decoding instance is used as the time increment evaluation result.

8. The method according to claim 7, characterized in that, The step of determining the prediction time increment delay for each candidate decoding instance to perform decoding operations on the data required for the inference request, based on the demand configuration table corresponding to each candidate decoding instance, includes: For any candidate decoding instance, based on the requirement configuration table corresponding to the candidate decoding instance, a first predicted decoding time is obtained for the candidate decoding instance to perform decoding operations on the data required by the existing inference request; and a second predicted decoding time is obtained for the candidate decoding instance to perform decoding operations on the data required by the existing inference request and the data required by the inference request. The time difference between the second predicted decoding time and the first predicted decoding time is used as the predicted time increment delay time corresponding to the candidate decoding instance.

9. The method according to any one of claims 1-4, characterized in that, The configuration process for the instance configuration table and the corresponding requirement configuration table includes: Using the hardware parameters of the existing wafer-level chip architecture as constraints, a configuration table of multiple candidate instances is determined; The configuration of each candidate instance configuration table is pre-filled and decoded in a completely overlapping manner to obtain the requirement configuration table corresponding to each candidate instance configuration table; The requirement configuration table corresponding to each candidate instance configuration table is used to guide instance operation, and the chip throughput corresponding to each candidate instance configuration table is estimated. The candidate instance configuration table with the lowest chip throughput is used as the instance configuration table, and the demand configuration table corresponding to the candidate instance configuration table with the lowest chip throughput is used as the demand configuration table corresponding to the instance configuration table.

10. A reasoning device for a large language model, characterized in that, The device includes: The lookup module is used to respond to inference requests triggered within the input box of the large language model, and to search for pre-filled instances and corresponding decoding instances from all instances corresponding to the wafer-level chip according to the instance configuration table and the corresponding requirement configuration table; the instance configuration table and the corresponding requirement configuration table are both built on the basis of the existing wafer-level chip architecture. The pre-filling module is used to pre-fill the inference request using the pre-filling instance, and store the data required by the inference request into the cache corresponding to the pre-filling instance; The decoding module is used to perform decoding operations on the data required for the inference request based on the decoding instance, and use the obtained decoding result as the inference result of the large language model; The output module is used to output the inference result through the output box of the large language model.

Images