Inference System and Method Based on Large Language Model Using Multi-Chip Parallel Computing

Abstract

This invention provides an inference system and method for a large language model based on multi-chip parallel computing, relating to the field of artificial intelligence technology. The system includes: a pre-processing module for processing input instruction information to generate inference data, wherein the inference data is in matrix form; and an expert parallel module for sending the inference data to accelerator chips and determining the sub-inference data processed by the corresponding activated expert units of each accelerator chip, so that the activated expert units perform calculations based on the corresponding sub-inference data and determine the complete parallel computing result data. This invention broadcasts input data to all accelerator chips, each accelerator chip selects the corresponding input data for calculation according to the set activated expert units, and obtains the same complete calculation result through global reduction operations between all accelerator chips, improving the overall performance and efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and for example to a reasoning system and method for a large language model based on multi-chip parallel computing. Background Technology

[0002] Large Language Model (LLM) inference is the process by which a large language model, based on received input information, engages in logical thinking, connects knowledge, and analyzes and judges to ultimately generate new, reasonable, coherent content that fits the context. In recent years, large language models based on the transformer architecture have developed rapidly. Models such as GPT3 (Generative Pre-trained Transformer 3), Qianwen, and Deepseek-R1 have become increasingly large in scale and their performance has improved significantly. Dialogue applications based on various large language models are also becoming more widespread. The transformer is a deep learning architecture entirely based on self-attention mechanisms. Deepseek-R1 and the latest qwen3-235B both employ a Mixed Expert Model (MOE) structure. Compared to traditional dense models, the MOE structure contains multiple independent network units, each maintaining independent weight parameters and referred to as an "expert." During the network's inference process, the network selects and activates different "expert" units in the MOE structure in real time based on the input, thereby achieving better model quality and inference efficiency. The MOE (Model-View-Engine) structure exhibits high sparsity during inference. While it typically contains a large number of "experts," in actual inference, only a small number of these "experts" are activated for each input, leaving the majority inactive. Furthermore, the activated "experts" differ for different inputs, making it difficult for accelerator hardware to uniformly process batches of input data during inference, thus impacting performance. On the other hand, due to the large number of "expert" units, MOE networks are typically very large; for example, Deepseek-R1 has a weight scale of 671 bytes, and qwen3-235B has a weight scale of 235 bytes. Considering both network size and inference performance, the inference process of large language models usually requires multiple accelerator chips to operate in parallel for acceleration, achieving extremely high total storage capacity and total data transmission bandwidth.

[0003] In related technologies, the current MOE structure operation process involves first calculating the activation expert unit corresponding to the input term and the weight of each expert through the routing layer, and then distributing the input feature data corresponding to the input term to the corresponding activation expert unit. After the activation expert unit completes its calculation on the input feature data corresponding to the input term, it multiplies the result by the expert's weight, and finally sums the weighted results of all experts to obtain the output feature of the MOE structure. For the current MOE structure, expert parallelism (EP) is typically used for parallel acceleration. Typical multi-accelerator chip parallel acceleration schemes can be found in [reference needed]. Figure 1 The diagram shows the reasoning process. Figure 1 The system employs three accelerator chips, Worker0, Worker1, and Worker2, for parallel acceleration of the Expert Processing (EP) algorithm, with one expert placed on each accelerator chip. For the feature input data corresponding to the six parallel input tokens (Token0, Token1, Token2, Token3, Token4, and Token5), the CPU (Central Processing Unit) extracts the feature input data corresponding to each expert from the feature input data corresponding to the input tokens, concatenates them (full-pair, fully variable-length communication), and then sends them to the accelerator chips Expert0, Expert1, and Expert2 where the expert resides. Figure 1 In the scenario shown, Expert0 processes a large number of feature input data points for each token, resulting in expert overload. Expert2 processes a smaller number of feature input data points for each token, resulting in expert underload. After computation, the CPU collects the results from the accelerator chip (all-pair, all-variable-length communication) and places them into the output feature data positions corresponding to each word in the output result data Out0, Out1, Out2, Out3, Out4, and Out5. This CPU operation is complex and time-consuming, impacting inference performance. Summary of the Invention

[0004] The present invention aims to provide an inference system and method for a large language model based on multi-chip parallel computing.

[0005] According to one aspect of the present invention, a reasoning system based on a large language model using multi-chip parallel computing is proposed, comprising:

[0006] The pre-calculation module is used to process the input instruction information to generate the data to be reasoned, wherein the data to be reasoned is in matrix form;

[0007] The expert parallel module includes multiple accelerator chips, each of which is equipped with an activated expert unit. The expert parallel module determines the sub-inference data to be processed by each activated expert unit based on the data to be inferred, so that the activated expert unit can perform calculations based on the corresponding sub-inference data and determine the parallel calculation result data.

[0008] According to one aspect of the present invention, a reasoning method for a large language model based on multi-chip parallel computing is proposed, comprising: generating data to be reasoned based on input instruction information, wherein the data to be reasoned is in matrix form; determining sub-reasoning data processed by activation expert units corresponding to multiple accelerator chips, so that the activation expert units perform calculations based on the corresponding sub-reasoning data; and determining parallel computing result data based on the calculation results of the activation expert units.

[0009] According to one aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory storing a computer program, which, when executed by the processor, causes the processor to perform the method described above.

[0010] According to one aspect of the present invention, a non-transitory computer-readable medium is provided, on which readable instructions are stored, which, when executed by a processor, cause the processor to perform the method described above.

[0011] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention.

[0012] Beneficial effects:

[0013] Through the embodiments provided by this invention, the differentiated data distribution and collection operations for each accelerator chip, originally handled by the CPU, are transformed into a highly efficient, parallel, and fully interchangeable communication process autonomously completed within the module through the collaborative scheduling mechanism of the expert parallel module. This significantly simplifies the control complexity of the host CPU for heterogeneous data communication, avoids the time-consuming operations of sorting, splicing, and rearranging massive amounts of token data one by one by the CPU, thereby greatly reducing system communication overhead and CPU burden, and effectively improving the overall inference throughput and efficiency of large-scale language models on multi-chip platforms. In traditional multi-chip inference systems, since the token data allocated to each accelerator chip is different in content and scale, the CPU needs to finely control the interconnection network to perform point-to-point data transmission for each chip, resulting in extremely complex communication scheduling. At the same time, the CPU needs to manually extract and splice subsets from the global data according to the expert affiliation of each token and distribute them to the corresponding chip. After calculation, the results need to be manually reassembled. This series of operations brings huge overhead to the CPU and becomes the main bottleneck of system performance. Compared to existing technologies, this invention achieves automatic routing and result aggregation of token data through an integrated expert parallel module. It broadcasts input data to all accelerator chips, and each accelerator chip selects the corresponding input data for calculation according to the set activated expert unit. This completely frees the CPU from heavy data handling and scheduling tasks, allowing the CPU to focus on high-level task scheduling, thereby significantly improving the system's parallel efficiency and inference speed. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without exceeding the scope of protection claimed by the present invention.

[0015] Figure 1 This is a schematic diagram of the inference process of the prior art MOE structure provided in the embodiments of the present invention;

[0016] Figure 2 A block diagram of an inference system for a large language model based on multi-chip parallel computing, provided in an embodiment of the present invention;

[0017] Figure 3 The hardware structure for EP parallelism provided in the embodiments of the present invention;

[0018] Figure 4 A flowchart illustrating the computation process of a large language model based on multi-chip parallel computing, provided for an embodiment of the present invention;

[0019] Figure 5 A schematic diagram illustrating the process of determining sub-inference data of the 6×8 matrix-form data to be inferred, provided in an embodiment of the present invention;

[0020] Figure 6 A schematic diagram illustrating the process of determining the expert computation result of 6×8 matrix data to be reasoned, as provided in an embodiment of the present invention;

[0021] Figure 7 This is a schematic diagram illustrating the process of integrating parallel computing result data provided in an embodiment of the present invention;

[0022] Figure 8 A flowchart illustrating the computation method for a large language model based on multi-chip parallel computing provided in an embodiment of the present invention;

[0023] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0024] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0025] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0026] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0028] It should be understood that although terms such as first, second, and third may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Thus, the first component discussed below may be referred to as the second component without departing from the teachings of the inventive concept. As used herein, the term "and / or" includes any one and all combinations of one or more of the associated listed items.

[0029] All English abbreviations used in this invention and their corresponding Chinese translations:

[0030] English abbreviation: LLM; English full name: Large Language Model; Chinese full name: Large Language Model;

[0031] English abbreviation: GPT3; English full name: Generative Pre-trained Transformer 3; Chinese full name: Generative Pre-trained Transformer 3;

[0032] English abbreviation: MOE; English full name: Mixed Expert Models; Chinese full name: Mixed Expert Models;

[0033] English abbreviation: EP; English full name: Expert Parallel; Chinese full name: Expert Parallel;

[0034] English abbreviation: CPU; English full name: Central Processing Unit; Chinese full name: Central Processing Unit;

[0035] English abbreviation: FFN; English full name: Feed-Forward Network; Chinese full name: Feed-Forward Network.

[0036] The specific implementation can refer to the following embodiments.

[0037] Figure 2 It is a block diagram of an inference system for a large language model based on multi-chip parallel computing provided by an embodiment of the present invention. As Figure 2 shown, the system 20 includes: a pre-computation module 21 and an expert parallel module 22.

[0038] The pre-computation module 21 is used to process the input instruction information to generate data to be inferred, where the data to be inferred is in matrix form.

[0039] In this invention, the pre-processing module 21 is equipped with a preprocessing structure and an attention mechanism structure for preprocessing externally input instruction information. Users can send questions or other content as instruction information to the large language model through manual input, voice input, etc. The pre-processing module 21 can process these text or voice messages, converting them into data that can be used for subsequent inference calculations, serving as the data to be inferred. The data to be inferred in this invention is in matrix form.

[0040] The expert parallel module 22 includes multiple accelerator chips 220, each of which is equipped with an activated expert unit. The expert parallel module 22 determines the sub-inference data to be processed by each activated expert unit based on the data to be inferred, so that the activated expert unit performs calculations based on the corresponding sub-inference data and determines the parallel calculation result data.

[0041] In this invention, the expert parallel module 22 includes at least multiple accelerator chips 220, and the data to be inferred generated by the pre-computation module 21 serves as the unified input for each accelerator chip 220. Each accelerator chip 220 may be equipped with an activated expert unit. By processing the data to be inferred, it can be determined which data should be processed by which activated expert unit, thereby dividing the data to be inferred into multiple sets of sub-inference data, which are also in matrix form. The activated expert units in the accelerator chips 220 perform calculations based on the processable sub-inference data to obtain the calculation results of each accelerator chip 220, thereby determining the complete parallel computation result data corresponding to the calculation results of all accelerator chips 220.

[0042] In some implementations, multiple integration methods for calculation results can be pre-set. The calculation results obtained by each activated expert unit are then processed according to the integration method to obtain parallel calculation result data.

[0043] The system of this invention, through the collaborative scheduling mechanism of the expert parallel module, transforms the differentiated data distribution and collection operations, originally handled by the CPU for each accelerator chip, into a highly efficient, parallel, and fully interchangeable communication process autonomously completed within the module. This significantly simplifies the host CPU's control complexity for heterogeneous data communication and avoids the time-consuming operations of sorting, splicing, and rearranging massive amounts of token data one by one. This greatly reduces system communication overhead and CPU burden, effectively improving the overall inference throughput and efficiency of large-scale language models on multi-chip platforms. In traditional multi-chip inference systems, because the token data allocated to each accelerator chip differs in content and scale, the CPU needs to finely control the interconnect network to perform point-to-point data transmission for each chip, resulting in extremely complex communication scheduling. Simultaneously, the CPU must manually extract and splice subsets from the global data based on the expert affiliation of each token and distribute them to the corresponding chip, and then manually reassemble the results after calculation. This series of operations incurs huge overhead for the CPU, becoming a major bottleneck in system performance. Compared to existing technologies, this invention achieves automatic routing and result aggregation of token data through an integrated expert parallel module. It broadcasts input data to all accelerator chips, and each accelerator chip selects the corresponding input data for calculation according to the set activated expert unit. This completely frees the CPU from heavy data handling and scheduling tasks, allowing the CPU to focus on high-level task scheduling, thereby significantly improving the system's parallel efficiency and inference speed.

[0044] According to some embodiments, multiple accelerator chips 220 in the expert parallel module 22 can interact with each other; the accelerator chip 220 is used to detect the expert identifier generated based on the data to be inferred, and to determine the activated expert unit corresponding to the sub-inference data according to the expert identifier, and to allocate each sub-inference data to the accelerator chip 220 with the corresponding activated expert unit according to its corresponding expert identifier.

[0045] In this invention, gate operations can be used to generate expert identifiers, and the accelerator chips 220 can exchange information. Each accelerator chip 220 is equipped with an activated expert unit, and each activated expert unit can process different types of data. Each accelerator chip 220 needs to select and retain the corresponding sub-inference data that its activated expert unit can process, based on the complete expert identifier of the data to be processed. Based on this, the allocation of sub-inference data is achieved.

[0046] The accelerator chips of this invention can autonomously and in parallel determine their expert tasks and data destinations by detecting expert identifiers generated based on the data to be inferred. This eliminates the need for complex global scheduling and data sorting by a central controller (such as a CPU), fundamentally changing the traditional data flow model that is centrally controlled by the host CPU. The identifier-based matching mechanism allows data to be sent directly from the source chip to the correct target chip in one step, eliminating the intermediate steps of data aggregation to the CPU and sorting before distribution, as in traditional solutions. This significantly reduces communication hops and transmission latency, improving data flow efficiency.

[0047] According to some embodiments, the accelerator chip 220 is also used to perform its corresponding operation and split the operation result into expert operation result and sub-result to be fused corresponding to the data to be inferred. Each accelerator chip 220 sends the expert operation result to the corresponding other accelerator chips 220 and receives the expert operation result sent by the other accelerator chips 220. The sub-result to be fused and the expert operation result sent by the other accelerator chips 220 are fused to generate parallel computing result data.

[0048] In this invention, the activated expert unit on the accelerator chip 220 can perform calculations on the sub-inference data to obtain the calculation results. The calculation results are in matrix form and are divided into two parts: the expert calculation results and the sub-results to be fused. The expert calculation results are the data obtained by the activated expert unit, while the sub-results to be fused are generally 0 and are calculated by other accelerator chips 220. The expert calculation results calculated on the accelerator chip 220 are sent to other accelerator chips, and the accelerator chip 220 also receives expert calculation results sent by other accelerator chips 220.

[0049] For any accelerator chip 220, the sub-result to be fused is fused with the expert computation results calculated by other activated expert units. The zero-marked matrix part of the sub-result to be fused is covered by the expert computation results with actual data, generating parallel computation result data.

[0050] In this invention, after completing local expert calculations, each accelerator chip immediately and asynchronously sends its independently transmittable "expert calculation results" to the target chip. Simultaneously, it processes the "sub-results to be fused" in parallel within its own chip. This maximizes the overlap between communication latency on the critical path and local computation time, significantly hiding cross-chip communication latency and greatly improving hardware utilization and overall system throughput. The final parallel computation result data is generated by each chip locally and in parallel fusing necessary intermediate results from other chips with its own partial results, rather than aggregating all data to a single node for serial fusion. This distributed fusion mode completely eliminates the data aggregation bottlenecks and single-point latency caused by traditional centralized fusion architectures, making the result generation speed positively correlated with the number of chips, resulting in excellent system scalability.

[0051] According to some embodiments, reference Figure 2 The accelerator chip 220 further includes a first on-chip cache unit 2201 and a second on-chip cache unit 2202, used to cache sub-inference data during algorithm operation in the accelerator chip 220; a third on-chip cache unit 2203, used to cache expert identifiers of lexical units corresponding to activated expert units to generate an identifier sequence; a computation unit 2204, used to perform a preset gate operation on the data to be inferred to determine the expert identifier of the activated expert unit corresponding to the lexical unit of the sub-inference data, and to perform the calculation process of the algorithm operation during the sub-inference data processing, wherein the lexical unit is the smallest semantic unit processed by the system; an address generation unit 2205, used to generate read and write addresses of the corresponding data in the first on-chip cache unit according to the algorithm operation; and a direct memory access unit 2206, used to read sub-inference data from the device memory into multiple accelerator chips 220, and to write the expert operation results of multiple accelerator chips 220 into the device memory.

[0052] In this invention, lexical units are used to represent the smallest semantic units that the model can understand and process. They are a series of discrete indices obtained by segmenting and mapping the "data to be reasoned" using a word segmenter. (See reference...) Figure 3 The hardware structure shown includes a first on-chip cache unit 2201 and a second on-chip cache unit 2202 for caching input data during algorithm operations by the accelerator. For binocular computation, each of the first and second on-chip cache units caches one input data path. For example, for a linear operation (i.e., a linear transformation operation), the first on-chip cache unit 2201 caches feature data, and the second on-chip cache unit 2202 caches weight data. Simultaneously, the first and second on-chip cache units 2201 and 2202 output the cached data to the computation unit 2204 according to the data arrangement format corresponding to the algorithm operation for computation.

[0053] The third on-chip cache unit 2203 is used to cache the input token index corresponding to each activation expert unit generated in the MOE structure; the address generation unit 2205 is used to generate the read and write address of the corresponding data in the first on-chip cache unit 2201 according to the algorithm operation; the direct content access unit is used to read data from the device memory into the accelerator chip 220, or write the result from the accelerator chip 220 into the device memory.

[0054] This invention achieves complete separation of computational data flow and control routing flow in physical storage and access paths by setting up independent first and second on-chip cache units dedicated to caching input / output sub-inference data, and a third on-chip cache unit dedicated to caching expert identifiers (routing information). This avoids contention for cache resources between two different types of data, ensuring data supply stability and high bandwidth, and providing a continuous, non-blocking data pipeline for the core computing unit. The arithmetic unit can focus on pure algorithmic computation, unaffected by tasks such as data handling and address generation. The address generation unit is dedicated to complex memory access pattern calculations (such as matrix partitioning and transposition in Transformers), freeing the core computing unit from tedious address calculations. The direct memory access unit is dedicated to handling high-speed data exchange between off-chip and device memory, achieving overlap between computation and I / O, eliminating the overhead caused by task switching and resource contention in traditional general-purpose processors, and optimizing the performance of each critical path. All critical data (input, output, routing identifiers) are cached in the on-chip cache, avoiding frequent access to the high-latency, high-power device memory. Lightweight but frequently accessed expert identifiers are cached independently on a third-chip cache unit, enabling the routing logic to quickly read decision information with extremely low latency and power consumption. This is crucial for achieving low-latency token-level dynamic routing.

[0055] According to some embodiments, the operation unit 2204 is further configured to select a lexical identifier of a target lexical that matches the corresponding activated expert unit from the third on-chip cache unit 2203 based on the expert number of the currently executing activated expert unit, and multiply the lexical identifier by the length of the sub-inference data corresponding to the target lexical to determine the first address offset of the sub-inference data corresponding to the target lexical in the data to be inferred; the address generation unit 2205 sequentially generates the first basic address of each data in each row of the sub-inference data according to the linear transformation mode of the algorithm operation; the operation unit 2204 is further configured to add the first basic address and the first address offset to determine the read address, wherein the read address is the address of the sub-inference data relative to the data to be inferred stored in the first on-chip cache unit 2201; the first on-chip cache unit 2201 reads the corresponding sub-inference data from the direct memory access unit 2206 according to the read address, and sends the sub-inference data to the operation unit 2204; the operation unit 2204 also performs expert calculations based on the sub-inference data to generate expert calculation results.

[0056] In this invention, the inference data output by the pre-computation module 21 serves as the same input data for each accelerator chip in the MOE section. The MOE section may contain multiple such... Figure 4 The calculation process shown is as follows: Figure 4 Taking two accelerator chips 220 as an example, the inference process of the two accelerator chips 220 is described. During the entire inference process of the large language model, the pre-processing module 21 and the expert parallel module 22 will work in a loop multiple times. That is, the calculation result of the expert parallel module 22 is used as the input data of the pre-processing module 21 to continue a new round of calculation. For... Figure 4 The computation process can involve three layers of operations. Each of the two accelerator chips 220 executes a preset gate operation, i.e., a gating operation to obtain the activation expert unit identifier corresponding to each sub-inference data. Then, using the FFN (Feed-Forward Network) of the activation expert unit, the fused gather (collection / aggregation) + Expert (expert computation) + scatter (distribution) operations are run respectively. Specifically, the first layer of linear fusion operation of gather and Expert is first run, i.e., the corresponding data is directly retrieved from the input data according to the index of the input data corresponding to the cached activation expert unit through the above-mentioned EP parallel hardware structure, and then given to the computation module for linear operation.

[0057] exist Figure 4In the process, when chip A performs the first-level linear calculation for expert 0, it directly extracts the sub-inference data corresponding to expert 0 from the input data to be inferred and performs the calculation. When chip B performs the first-level linear calculation for expert 1, it directly extracts the sub-inference data corresponding to expert 1 from the input data to be inferred and performs the calculation, thus implementing the gather operation. Next, the intermediate layer operations of expert 0 and expert 1, i.e., the Expert operation, are run. Then, the final layer linear and scatter fusion operation of Expert is performed. That is, when chip A performs the final layer linear operation for expert 0, it directly writes the result data to the position corresponding to the expert calculation result for expert 0, while filling the position of the parallel calculation result data corresponding to expert 1 with 0, meaning the matrix part corresponding to the sub-result to be fused is 0. Similarly, when chip B performs the final layer linear operation for expert 1, it directly writes the result data to the position corresponding to the expert calculation result for expert 1, while filling the position of the parallel calculation result data corresponding to expert 0 with 0, meaning the matrix part corresponding to the sub-result to be fused is 0. Finally, the two accelerator chips perform a global reduction communication operation, collecting each other's data and adding it to their own data. This involves fusing the results to be fused with the expert computation results sent by other accelerator chips, thereby obtaining the same MOE structure results for each chip.

[0058] In the specific implementation process, you can refer to Figure 5 Assuming the input data is a 6x8 matrix, with each row corresponding to one token input data, such as... Figure 5 As shown on the left, the data to be inferred corresponds to the input data of token numbers 0-5 along the column direction. Assume that after gating, the expert identifiers corresponding to the activated expert units for each token's input data are 0, 1, 1, 0, 0, 1, as follows... Figure 5 The left-hand Expert column shows the process. First, the activation expert unit ID corresponding to each token obtained from the gating operation is cached in the third on-chip cache unit 2203. Then, during the first-level linear execution of the Expert operation, the token (lexicon) matching the activation expert unit is selected from the third on-chip cache unit 2203 based on the Expert number currently being executed on the accelerator chip. This token is output as the target lexicon ID (i.e., lexicon identifier). For example, if chip A executes expert 0, then the third on-chip cache unit 2203 of chip A will sequentially select the IDs of the tokens with activation expert unit 0 for output, i.e., outputting 0, 3, and 4 sequentially. The token IDs output by the third on-chip cache unit 2203 are passed through the right side of the first on-chip cache unit 2201 (see reference). Figure 3The multiplier of the multiplier is multiplied by the length (len) of the input data corresponding to each token to obtain the address offset (offset) of the input data corresponding to the token to be read in the complete input data matrix. Figure 5 Each line of data in the middle has a length of 8, corresponding to len=8. Then, the address generation unit 2205 to the right of the first on-chip cache unit 2201 (see reference) Figure 3 Following the linear operation calculation pattern, the base address of each data element in a row of the input data matrix is ​​generated sequentially. Figure 5 The input data matrix has addresses 0, 1, 2, 3, 4, 5, 6, 7.

[0059] The base address generated by the address generation unit 2205 and the address offset generated by the multiplier are respectively input into the adder on the right side of the first on-chip cache unit 2201 (see reference). Figure 3 In the above steps, the numbers are added together to obtain the read address (raddr) of the first on-chip cache unit 2201. This address is the address of the input data corresponding to the target token in the complete input data matrix stored in the first on-chip cache unit 2201. For example... Figure 5 The addresses of the input data for token number 3, which is activated as expert unit 0, are 24, 25, 26, 27, 28, 29, 30, and 31. The first on-chip cache unit 2201 reads the corresponding input data from the cache according to the input read address and outputs it to the arithmetic unit for linear operation to generate the result of the linear operation.

[0060] The accelerator chip of this invention directly matches and selects the identifier of the target term from the on-chip cache based on the currently executing expert number, and calculates the precise location (first address offset) of the required data in global memory in real time. This achieves dynamic data addressing at the term level based on expert ID, enabling precise and waste-free extraction of each term data element that the current expert needs to process from complex global data. This avoids the bandwidth waste and storage pressure caused by irrelevant data mixed in during traditional batch transmission. Tasks such as identifier matching, offset calculation, and basic address generation, which originally required complex logical judgments and calculations from the CPU or general-purpose cores, are decentralized to a dedicated address generation unit and the control logic of the accelerator chip. This completely liberates the computing unit (operation unit) from tedious address calculation and data management tasks, allowing it to continuously operate in a highly efficient computing state, thereby significantly improving the overall computing efficiency and energy efficiency of the hardware. In expert parallelism, the data (term sequence) processed by each expert is irregularly and discontinuously distributed in global memory. This invention synthesizes read addresses in real time through hardware, enabling the direct memory access unit to initiate efficient and targeted read requests based on these precise addresses. Compared to the traditional method of reading consecutive data blocks in the largest possible range and then filtering them by software, this greatly reduces the transmission of invalid data, allowing valuable memory bandwidth to be used entirely for moving valid data, and significantly improving memory access efficiency.

[0061] According to some embodiments, the operation unit 2204 is further configured to multiply the term identifier by the length of the expert operation result corresponding to the target term to determine the second address offset of the expert operation result in the parallel computing result data; the address generation unit 2205 sequentially generates the second basic address of each data in each row of the parallel computing result data according to the linear transformation mode of the algorithm operation; the operation unit 2204 is further configured to add the second basic address and the second address offset to determine the write address, wherein the write address is the address of the expert operation result in the parallel computing result data stored in the first on-chip cache unit.

[0062] In this invention, the implementation of the fusion operation of the last layer of linear and scatter in Expert can be found by referring to Figure 6 After the final linear operation, the calculation result for each Expert is obtained as follows: Figure 6 As shown on the left, when the calculation result is stored back into the first on-chip cache unit 2201, it is filled back into the corresponding row of the complete result data matrix according to its corresponding token number, such as... Figure 6As shown on the right. For example, the calculation result corresponding to token number 3 generated by expert 0 is written back to the 3rd row of the result data matrix. For chip A, which executes expert 0, the result data of the token corresponding to expert 1, which activates the expert unit, is filled with 0. Chip B is processed similarly.

[0063] In the specific implementation process, during the final linear execution of the Expert, the ID of the token matching the Expert number executed on the current accelerator chip is selected from the third on-chip cache unit 2203 and output. For example, if chip A executes Expert 0, then the third on-chip cache unit 2203 of chip A will sequentially select the ID of the token whose Expert unit is 0 and output it, i.e., output 0, 3, and 4 in sequence. This part is the same as the execution process in the above embodiment.

[0064] The token ID output by the third on-chip cache unit 2203 is processed by the multiplier on the left side of the first on-chip cache unit 2201 (see reference). Figure 3 Multiplying the length (len) of the result data corresponding to each token by the address offset (offset) of the result data corresponding to the token to be written in the complete result data matrix yields the address offset (offset) of the result data corresponding to the token to be written. Figure 6 Each line of data in the middle is 8 bytes long, corresponding to len=8. Then, the address generation unit 2205 to the left of the first on-chip cache unit 2201 (see reference) Figure 3 Following the linear operation computation pattern (i.e., the linear transformation pattern of the algorithm operation), the base address of each result in a certain row of the result data matrix is ​​generated sequentially. Figure 6 The resulting data matrix has addresses 0, 1, 2, 3, 4, 5, 6, 7 in sequence.

[0065] The base address generated by the address generation unit 2205 and the address offset generated by the multiplier are respectively input into the adder on the left side of the first on-chip cache unit 2201 (see reference). Figure 3 In the above steps, the sums are used to obtain the write address (waddr) of the first on-chip cache unit 2201. This address is the address of the result data corresponding to the target token in the complete result data matrix stored in the first on-chip cache unit 2201. For example... Figure 6 The addresses of the result data for token number 3, which is activated as expert unit 0, are: 24, 25, 26, 27, 28, 29, 30, 31.

[0066] The result writing process of this invention is symmetrical to the data reading process. The accelerator chip uses lexical identifiers to calculate the precise location (second address offset) in the global output tensor where each expert computation result should be written back. This ensures that fragmented results generated by distributed computing can be automatically and accurately reassembled into the correct location in the final output data, guaranteeing global data consistency at the hardware level and eliminating the need for the CPU to perform complex and error-prone data rearrangement operations afterward. Through precise address calculation based on lexical identifiers, each result is written to its unique location in the final output tensor, avoiding write conflicts, serialization, and bandwidth waste caused by multiple computation results being blindly written to the same memory region or cache line. The write operation is deterministic and ordered, greatly improving the efficiency of the storage subsystem in the write-back phase.

[0067] According to some embodiments, the arithmetic unit 2204 is also used to add the arithmetic result obtained from the arithmetic operation to the arithmetic result sent by other accelerator chips according to the write address, so as to generate parallel computing result data.

[0068] This invention can add the computation results of accelerator chip 220 to the computation results of other accelerator chips, for reference. Figure 7 In the specific implementation process, after the accelerator chips 220 complete their respective activation expert unit calculations, they perform a global reduction communication operation, collecting each other's result data and adding it to their own, so that both accelerator chips obtain complete MOE result data. Then, the attention mechanism structure of the later layers in the network structure, i.e., the front-end computation module 21, continues to work.

[0069] In this invention, each accelerator chip, upon receiving a write address, actively and in parallel adds (reduces) its own computation results with intermediate results from other chips belonging to the same target address. The final parallel computation result data is generated synchronously on all chips, rather than being first aggregated to a main chip for serial reduction and then broadcast. This avoids the single-point memory bandwidth bottleneck and additional communication latency present in traditional centralized reduction architectures, allowing the result synthesis speed to scale linearly with the number of chips. The result fusion operation (addition) is triggered instantly upon data arrival at the destination address, deeply integrated with the network transmission and write cache operations of the result data at the hardware level. This achieves an extremely streamlined process of transmitting, writing, and fusion simultaneously, completely hiding the reduction overhead within the data transmission path and achieving near-zero overhead global result synchronization.

[0070] The following describes method embodiments of the present invention, which can be implemented using the apparatus embodiments described above. For details not disclosed in the apparatus embodiments of the present invention, please refer to the method embodiments of the present invention.

[0071] Figure 8A flowchart illustrating the computation method for a large language model based on multi-chip parallel computing provided in an embodiment of the present invention. Figure 8 As shown, the computation method for a large language model based on multi-chip parallel computing includes steps S801, S802, and S803.

[0072] In step S801, data to be inferred is generated based on the input instruction information, wherein the data to be inferred is in matrix form.

[0073] In step S802, the sub-inference data processed by the activation expert unit corresponding to each of the multiple accelerator chips is determined so that the activation expert unit can perform calculations based on the corresponding sub-inference data.

[0074] In step S803, the parallel computing result data is determined based on the calculation results of the activated expert unit.

[0075] This invention employs a dataflow approach to achieve dynamic data splitting and merging in parallel execution of the MOE structure using EP (Execution Processing) methods. This avoids using the CPU for data transfer, improving the performance and efficiency of accelerator chips executing the MOE structure in parallel. The complex communication process of distributing input data to different accelerator chips according to corresponding activation expert units, and collecting results from different accelerator chips according to corresponding activation expert units, is simplified into a unified hardware operation. Each accelerator chip uses the same input data and obtains the same complete computation result through a global reduction operation across all accelerator chips. This significantly simplifies the process of executing the MOE structure in parallel using EP methods across multiple accelerator chips, improving overall performance and efficiency.

[0076] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 9 As shown, the electronic device 900 of this embodiment may include a memory 901 and a processor 902.

[0077] The memory 901 stores a computer program, which, when executed by the processor 902, causes the processor 902 to perform the method described in the above embodiments.

[0078] The processor 902 and the memory 901 are connected, for example, via a bus.

[0079] Optionally, the electronic device 900 may also include a transceiver. It should be noted that in practical applications, the transceiver is not limited to one, and the structure of the electronic device 900 does not constitute a limitation on the embodiments of the present invention.

[0080] Processor 902 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 902 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0081] A bus can include a pathway for transmitting information between the aforementioned components. The bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0082] The memory 901 can be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD. ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0083] The memory 901 stores application code that executes the present invention, and its execution is controlled by the processor 902. The processor 902 executes the application code stored in the memory 901 to implement the content shown in the foregoing method embodiments.

[0084] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0085] The electronic device in this embodiment can be used to execute the method of any of the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0086] The present invention also provides a non-transitory computer-readable storage medium having stored computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the method as described in the above embodiments.

[0087] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a non-transitory computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0088] The embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of the present invention, its specific implementation methods, and its application scope, are all within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A reasoning system for a large language model based on multi-chip parallel computing, characterized in that, include: A pre-calculation module is used to process the input instruction information to generate data to be reasoned, wherein the data to be reasoned is in matrix form; The expert parallel module includes multiple accelerator chips, each of which is equipped with an activated expert unit. The expert parallel module determines the sub-inference data to be processed by each activated expert unit based on the data to be inferred, so that the activated expert unit performs calculations based on the corresponding sub-inference data and determines the parallel calculation result data. The multiple accelerator chips in the expert parallel module can interact with each other. The accelerator chip is used to detect the expert identifier generated based on the data to be inferred and to determine the activated expert unit corresponding to the sub-inference data according to the expert identifier. It is also used to allocate each sub-inference data to the accelerator chip with the corresponding activated expert unit according to its corresponding expert identifier. The accelerator chip is also used to perform its corresponding operation and split the operation result into expert operation result and sub-result to be fused corresponding to the data to be inferred. Each accelerator chip sends the expert operation result to the corresponding other accelerator chip and receives the expert operation result sent by the other accelerator chip. The sub-result to be fused and the expert operation result sent by the other accelerator chip are fused to generate the parallel computing result data.

2. The system according to claim 1, characterized in that, The accelerator chip also includes: The first on-chip cache unit and the second on-chip cache unit are used to cache the sub-inference data when the accelerator chip is running the algorithm operation; The third on-chip cache unit is used to cache the expert identifiers of the lexical corresponding to the activated expert unit in order to generate an identifier sequence; The operation unit is used to perform a preset gate operation on the data to be reasoned, so as to determine the expert identifier of the activated expert unit corresponding to the lexical of the sub-reasoning data, and to perform the calculation process of the algorithm operation in the process of processing the sub-reasoning data, wherein the lexical is the smallest semantic unit processed by the system; The address generation unit is used to generate the corresponding read / write address of the data in the first on-chip cache unit according to the algorithm operation; A direct memory access unit is used to read the sub-inference data from the device memory into the plurality of accelerator chips, and to write the expert computation results of the plurality of accelerator chips into the device memory.

3. The system according to claim 2, characterized in that, The operation unit is further configured to select, according to the expert number of the currently executing activated expert unit, a lexical identifier of a target lexical that matches the corresponding activated expert unit from the third on-chip cache unit, and multiply the lexical identifier by the length of the sub-inference data corresponding to the target lexical to determine the first address offset of the sub-inference data corresponding to the target lexical on the data to be inferred. The address generation unit generates the first basic address of each data in each row of the sub-inference data sequentially according to the linear transformation pattern of the algorithm operation; The arithmetic unit is further configured to add the first base address and the first address offset to determine the read address, wherein the read address is the address of the sub-inference data relative to the data to be inferred stored in the first on-chip cache unit; The first on-chip cache unit reads the corresponding sub-inference data from the direct memory access unit according to the read address, and sends the sub-inference data to the arithmetic unit; The computing unit also performs expert calculations based on the sub-inference data to generate the expert calculation results.

4. The system according to claim 3, characterized in that, The computation unit is further configured to multiply the term identifier by the length of the expert computation result corresponding to the target term, so as to determine the second address offset of the expert computation result on the parallel computation result data; The address generation unit generates the second basic address of each data in each row of the parallel computing result data in sequence according to the linear transformation mode of the algorithm operation; The arithmetic unit is further configured to add the second base address and the second address offset to determine the write address, wherein the write address is the address of the parallel computation result data stored in the first on-chip cache unit for the expert computation result.

5. The system according to claim 4, characterized in that, The computing unit is also used to add the computing result obtained from the operation to the computing result sent by other accelerator chips according to the write address, so as to generate the parallel computing result data.

6. A reasoning method for a large language model based on multi-chip parallel computing, characterized in that, include: Based on the input instruction information, generate data to be reasoned, wherein the data to be reasoned is in matrix form; Sub-inference data is determined for each of the multiple accelerator chips, each corresponding to an activated expert unit, to process. The activated expert units perform calculations based on the corresponding sub-inference data. The multiple accelerator chips can interact with each other. Each accelerator chip detects an expert identifier generated based on the data to be inferred and determines the activated expert unit corresponding to the sub-inference data based on the expert identifier. It also allocates each sub-inference data to an accelerator chip equipped with a corresponding activated expert unit according to its corresponding expert identifier. The accelerator chip performs its corresponding computation and splits the computation result into an expert computation result corresponding to the data to be inferred and a sub-result to be fused. Each accelerator chip sends its expert computation result to other corresponding accelerator chips and receives expert computation results sent by other accelerator chips. The sub-result to be fused and the expert computation results sent by other accelerator chips are then fused to generate the parallel computing result data. Based on the calculation results of the activated expert unit, the parallel computing result data is determined.

7. An electronic device, characterized in that, include: processor; A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in claim 6.

8. A non-transitory computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in claim 6.

Images