Computing systems, data processing methods, apparatus, and media for high-bandwidth inference

Abstract

The disclosure provides a computing system, a data processing method, equipment and a medium for high-bandwidth inference, and relates to the technical field of integrated circuits. The computing system is used for performing a decoding stage of a Transformer-based model inference, and comprises a host processor for performing a decoding stage nonlinear operator, an offload subsystem for performing at least a part of a decoding stage linear operator, and a standard high-speed interface module for transmitting an input activation vector and an output result vector between the host processor and the offload subsystem. The offload subsystem comprises a weight lock storage array for storing a weight matrix in a static residence manner, an input vector streaming interface for streamingly receiving the input activation vector, a matrix vector multiplication calculation unit for performing a matrix vector multiplication operation on the input activation vector and the weight matrix, and a result processing module for reducing or arranging the operation result to obtain the output result vector.

Description

Technical Field

[0001] This disclosure relates to the field of integrated circuit technology, and in particular to a computing system, a data processing method, apparatus, and medium for high-bandwidth inference. Background Technology

[0002] Large-scale language models based on the Transformer architecture include a pre-filling stage and a decoding stage during inference. The decoding stage generates output per token in an autoregressive manner, and each token generation requires performing matrix-vector multiplication on the linear operators in the attention mechanism.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] According to one aspect of this disclosure, a computing system is provided for performing a decoding phase of Transformer-based model inference. The computing system includes: a main processor configured to execute nonlinear operators in the decoding phase; an offloading subsystem configured to execute at least a portion of linear operators in the decoding phase, the offloading subsystem including: a weight-locked storage array configured to statically store weight matrices corresponding to the at least a portion of the linear operators in the decoding phase; an input vector streaming interface configured to stream input activation vectors corresponding to the at least a portion of the linear operators; a matrix-vector multiplication unit configured to perform a matrix-vector multiplication operation on the input activation vectors and the weight matrix; a result processing module configured to reduce or reorganize the result of the matrix-vector multiplication operation to obtain an output result vector; and a standard high-speed interface module configured to transmit the input activation vectors and the output result vectors between the main processor and the offloading subsystem.

[0005] According to another aspect of this disclosure, a data processing method for high-bandwidth inference is provided for the decoding stage of Transformer-based model inference. The method includes: storing weight matrices corresponding to at least a portion of the linear operators in the decoding stage in a storage array of an execution unit, and maintaining the weight matrices in a statically resident manner during the decoding stage; streaming input activation vectors corresponding to the at least a portion of the linear operators to the execution unit via an external interface; performing a matrix-vector multiplication operation on the input activation vectors and the weight matrix within the execution unit; reducing or rearranging the result of the matrix-vector multiplication operation to obtain an output result vector; and outputting the output result vector via the external interface.

[0006] According to one or more embodiments of this disclosure, the weight matrix of at least a portion of the linear operators in the decoding stage is stored in a weight-locked storage array of the offloading subsystem in a static resident manner. The offloading subsystem performs matrix-vector multiplication of the input activation vector and the weight matrix and reduces or organizes the result within the subsystem. The standard high-speed interface module only transmits the input activation vector and the output result vector, thereby reducing the bandwidth requirement for weight matrix transfer between the main processor and external storage and alleviating the computational load of the main processor on the linear operators in the decoding stage.

[0007] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 A structural block diagram of a computing system according to an exemplary embodiment of the present disclosure is shown; Figure 2 A flowchart of a data processing method according to an exemplary embodiment of the present disclosure is shown; Figure 3 A structural block diagram of a computer device according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0009] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0010] The terminology used in the description of the various examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context expressly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0011] According to one aspect of this disclosure, a computational system is proposed for performing the decoding phase of Transformer-based model inference. Figure 1 This is a structural block diagram illustrating a computing system 100 according to an exemplary embodiment.

[0012] refer to Figure 1 The computing system 100 includes: a main processor 110 configured to execute nonlinear operators in the decoding stage; an offloading subsystem 120 configured to execute at least a portion of linear operators in the decoding stage, the offloading subsystem 120 including: a weight-locked storage array 121 configured to statically store weight matrices corresponding to the at least a portion of linear operators in the decoding stage; an input vector streaming interface 122 configured to stream input activation vectors corresponding to the at least a portion of linear operators; a matrix-vector multiplication unit 123 configured to perform matrix-vector multiplication on the input activation vectors and the weight matrices; a result processing module 124 configured to reduce or organize the result of the matrix-vector multiplication to obtain an output result vector; and a standard high-speed interface module 130 configured to transmit the input activation vectors and the output result vectors between the main processor 110 and the offloading subsystem 120.

[0013] Therefore, this disclosure stores the weight matrices of at least a portion of the linear operators in the decoding stage in a statically resident manner in the weight-locked storage array of the offloading subsystem. The offloading subsystem performs matrix-vector multiplication of the input activation vector and the weight matrix and reduces or organizes the results within the subsystem. The standard high-speed interface module only transmits the input activation vector and the output result vector, thereby reducing the bandwidth requirements for weight matrix transfer between the main processor and external storage and alleviating the computational load of the main processor on the linear operators in the decoding stage.

[0014] Transformer-based models are a class of neural network models with attention mechanisms at their core, widely used in tasks such as natural language processing and multimodal understanding. Transformer-based model inference can include a prefilling stage and a decoding stage. The prefilling stage processes all tokens in the input sequence in parallel to generate an initial context representation. The decoding stage generates output token by token in an autoregressive manner; when a new token is generated, a series of operator operations are performed on the activation data corresponding to the current token.

[0015] Operators in the decoding phase can be categorized into linear operators and nonlinear operators. Linear operators are those whose core operation is matrix-vector multiplication of the weight matrix and activation vectors. The weight matrix represents the model parameters corresponding to the linear operator and remains unchanged during inference. The input activation vector is intermediate data generated after the current token has been processed in the previous stage, serving as the input to the linear operator. Nonlinear operators are those that do not rely on matrix-vector multiplication as their core operation, including layer normalization, activation functions, attention calculations, and residual connections. In the decoding phase, since only one or a small number of tokens are processed per step, the computational mode of linear operators is manifested as matrix-vector multiplication between the weight matrix and low-dimensional activation vectors. The weight matrix remains unchanged between steps, while the input activation vector is dynamically updated with the new tokens generated in each step.

[0016] The following will describe in detail the various components of the computing system 100.

[0017] The main processor 110 is the processor in the computing system 100 that executes nonlinear operators during the decoding phase. During inference, the main processor 110 can be responsible for performing operations such as layer normalization, activation functions, attention calculation, and residual connections, and can further be responsible for control logic and data orchestration. In an exemplary embodiment, in each decoding step, the main processor 110 generates an input activation vector based on the current layer and the current token, sends it to the offloading subsystem 120, and after receiving the output result vector returned by the offloading subsystem 120, continues to execute subsequent nonlinear operators.

[0018] The offloading subsystem 120 is a hardware subsystem separate from the main processor 110 and configured to execute at least a portion of the linear operators in the decoding phase. The range of at least a portion of the linear operators can be determined based on the model structure and system configuration. This disclosure does not require that all computations in the decoding phase be offloaded to the offloading subsystem 120. The offloading subsystem 120 may include a single offloading chip, multiple offloading chips, or be implemented in other ways.

[0019] The aforementioned separation refers to the fact that the offloading subsystem 120 and the main processor 110 are physically independent semiconductor entities, and data transmission between them is achieved through the standard high-speed interface module 130. When the offloading subsystem 120 includes an offloading chip, the offloading chip can be an independent die or an independent chip. When the offloading chip is an independent die, the offloading subsystem 120 and the main processor 110 can be connected via in-package interconnects. When the offloading chip is an independent chip, the offloading subsystem 120 and the main processor 110 can be connected via a board-level interface. In this disclosure, the offloading chip is not limited to its specific physical form; in addition to the aforementioned dies and chips, it can also be implemented as a chiplet, a multi-chip module, or other semiconductor entities with independent computing and storage capabilities. When the offloading subsystem 120 includes multiple offloading chips, the topology and interconnection method of the multiple offloading chips will be further described later.

[0020] In some embodiments, the above-mentioned division of labor among operators is the main division of labor mode of the computing system 100 as a whole. It is not excluded that the main processor 110 executes some linear operators, nor is it excluded that the offloading subsystem 120 performs nonlinear operations such as auxiliary control, cache management or format conversion.

[0021] In some embodiments, the allocation of operators during the decoding phase to the main processor 110 or the offloading subsystem 120 can be determined based on the computational characteristics of the operators. Weight-intensive matrix-vector multiplication operators are suitable for offloading to the offloading subsystem 120 because their weight matrices remain unchanged during inference and can reside statically in the weight-locked storage array 121, while their inputs and outputs are low-dimensional vectors, suitable for transmission via the standard high-speed interface module 130. Nonlinear operators, low-weight-density operators, or high-state-dependent operators are suitable for execution on the main processor 110 side because these operators typically do not involve large-scale weight matrices or require reliance on dynamically updated state data at runtime. In one exemplary embodiment, layer normalization, activation functions, attention computation, residual connections, and KV cache updates are performed on the main processor 110 side, while Q-projection, K-projection, V-projection, and linear transformations in the feedforward network are performed on the offloading subsystem 120 side. Other operations that do not belong to either the linear or nonlinear operators described above can be flexibly allocated to the main processor 110 or the offloading subsystem 120 according to the system configuration.

[0022] The weight-locked storage array 121 is a storage module within the offloading subsystem 120, configured to store the weight matrices corresponding to at least a subset of linear operators in a statically residing manner during the decoding phase. Static residing means that after the weight matrices are written to the weight-locked storage array 121, they remain residing during the decoding phase and are not repeatedly loaded via an external interface. In other words, during the decoding phase, the weight matrices are continuously stored within the offloading subsystem 120, and the standard high-speed interface module 130 between the main processor 110 and the offloading subsystem 120 does not handle the transmission of the weight matrices.

[0023] The input vector streaming interface 122 is a module within the offloading subsystem 120 that receives input activation vectors. Streaming reception means that the reception of input activation vectors and the computation of the matrix-vector multiplication unit 123 can overlap in time; that is, while the input vector streaming interface 122 is receiving current data, the matrix-vector multiplication unit 123 can perform computations on the already received data, rather than waiting for all input activation vectors to be received before starting computation. During the decoding phase, whenever the main processor 110 completes the pre-level nonlinear processing of the current token at the current layer, it generates the corresponding input activation vector and sends it to the offloading subsystem 120 via the standard high-speed interface module 130. The input vector streaming interface 122 receives this input activation vector and provides it to the matrix-vector multiplication unit 123.

[0024] The matrix-vector multiplication calculation unit 123 is a computation module within the offloading subsystem 120 that performs matrix-vector multiplication operations. The matrix-vector multiplication calculation unit 123 reads the weight matrix from the weight-locked storage array 121 and performs a matrix-vector multiplication operation on the input activation vector provided by the input vector streaming interface 122 and the weight matrix. Since the weight matrix is ​​stored in the weight-locked storage array 121 within the offloading subsystem 120, the weight reading required for the matrix-vector multiplication operation is completed within the offloading subsystem 120, without going through the external interface between the main processor 110 and the offloading subsystem 120.

[0025] In some embodiments, the matrix-vector multiplication calculation unit 123 may employ a lightweight instruction control method, retaining only a small number of instruction primitives such as matrix-vector multiplication, local accumulation, vector loading, and result write-back, thereby reducing control overhead.

[0026] The result processing module 124 is the module within the offloading subsystem 120 that reduces or organizes the results of matrix-vector multiplication operations. Reduction refers to performing aggregation operations on multiple partial results to obtain complete output data. Organization refers to splicing, aligning, or reorganizing the calculation results to conform to the format requirements returned to the main processor 110. The result processing module 124 returns the reduced or organized output vector to the main processor 110 via the standard high-speed interface module 130.

[0027] The standard high-speed interface module 130 is an interface module connecting the main processor 110 and the offloading subsystem 120, configured to transmit input activation vectors and output result vectors between the two. Since the weight matrix is ​​stored statically within the offloading subsystem 120, and matrix-vector multiplication is performed within the offloading subsystem 120, the data transmitted by the standard high-speed interface module 130 is vector-level data—that is, input activation vectors and output result vectors—rather than matrix-level data. The dimension of the input activation vector depends on the input dimension of the current linear operator, and the dimension of the output result vector depends on the output dimension of the current linear operator; both are much smaller than the size of the weight matrix itself. This allows the computing system 100 to complete data transmission between the main processor 110 and the offloading subsystem 120 using the standard high-speed interface, without relying on advanced packaging techniques to repeatedly move large-scale weight matrices.

[0028] In some embodiments, the unloading subsystem 120 may further include a vector register module for temporarily storing input activation vectors, intermediate vectors, or result vectors. The vector register module provides temporary data buffers for the matrix-vector multiplication calculation unit 123 and the result processing module 124, enabling the reception of input data, calculation execution, and result output to be decoupled in time.

[0029] According to some embodiments, the unloading subsystem 120 may include an unloading chip, which may include multiple basic processing slices. Each basic processing slice may include a local partition of the weight-locked storage array and a sub-computation unit of the matrix-vector multiplication calculation unit. The input activation vector can be broadcast to the multiple basic processing slices. Each basic processing slice can read the corresponding weights from its local partition and have its sub-computation unit perform the matrix-vector multiplication operation to obtain a local result. The result processing module can reduce or organize the local results of the multiple basic processing slices to obtain the output result vector.

[0030] Therefore, by organizing the offloading chip into multiple basic processing slices and broadcasting the input activation vector to each slice for parallel computation, multiple slices can simultaneously perform matrix-vector multiplication operations on different parts of the weight matrix, thereby improving the computational parallelism and scalability of the weight capacity of the offloading chip.

[0031] In some embodiments, the unloading subsystem 120 may include one or more unloading chips. When the unloading subsystem 120 includes a single unloading chip, the weight-locked storage array 121, the input vector streaming interface 122, the matrix-vector multiplication calculation unit 123, and the result processing module 124 can all be located inside the unloading chip. When the unloading subsystem 120 includes multiple unloading chips, the above-mentioned functional modules can be distributed across multiple unloading chips, and their specific distribution will be further described later. The following embodiments illustrate the case where the unloading subsystem 120 is composed of a single unloading chip. Each unloading chip may include multiple basic processing slices, or it may not be divided into slices but perform matrix-vector multiplication operations as a whole.

[0032] A basic processing slice is a reusable hardware unit that can be assembled within the chip. Each basic processing slice contains a local partition of a weight-locked memory array 121 and a sub-computation unit of a matrix-vector multiplication calculation unit 123. The local partition is a local storage area of ​​the weight-locked memory array 121 within the slice, storing a portion of the weight matrix. The sub-computation unit is a local computation unit of the matrix-vector multiplication calculation unit 123 within the slice, responsible for performing matrix-vector multiplication operations on the weights stored in the local partition and the input activation vector. The local result is a partial result obtained after performing matrix-vector multiplication operations on a single slice.

[0033] Broadcasting refers to simultaneously sending the same input activation vector to multiple basic processing slices. Upon receiving the same input activation vector, each slice reads the corresponding weights from its respective local partition and performs matrix-vector multiplication independently by its own sub-computation unit. In some embodiments, each slice receives the broadcast input activation vector and outputs a local result, and there may be no horizontal data exchange between slices; therefore, the number of slices can be expanded through simple stacking. The result processing module 124 is located above the slice level and reduces or organizes the local results output by each slice at the chip level to obtain the output result vector.

[0034] According to some embodiments, local partitions of multiple basic processing slices can store different blocks of the same weight matrix or the weight matrices corresponding to multiple linear operators respectively.

[0035] Therefore, by flexibly configuring the weight content stored in the local partitions of each slice, the offloading chip can both distribute a single large-scale weight matrix to multiple slices for parallel computation, and deploy weight matrices of different linear operators in different slices to support the parallel execution of multiple operators, thus improving the flexibility of weight deployment.

[0036] When multiple basic processing slices store different blocks of the same weight matrix in their local partitions, each slice performs matrix-vector multiplication on the corresponding block of the weight matrix for the same input activation vector. The local results of each slice are reduced by the result processing module 124 to obtain the complete output result vector. When multiple basic processing slices store weight matrices corresponding to different linear operators in their local partitions, each slice can perform matrix-vector multiplication on the same input activation vector for different linear operators to obtain the output results of each linear operator.

[0037] In one exemplary embodiment, the weight matrix can be divided into blocks along the column direction, with different column blocks stored in local partitions of different slices. Each slice outputs a sub-vector of the result of its corresponding column block, and the result processing module 124 concatenates the sub-vectors of the result of each slice into a complete output result vector. In another exemplary embodiment, the weight matrix can be divided into blocks along the row direction, with different row blocks stored in local partitions of different slices. Each slice outputs a partial sum, and the result processing module 124 performs accumulation and reduction on the partial sums of each slice.

[0038] According to some embodiments, the offloading chip may further include a protocol parsing unit configured to classify data received via the standard high-speed interface module 130 and identify configuration information and input activation vectors therein. Each basic processing slice may further include a local routing unit configured to receive the data classified by the protocol parsing unit, distribute the configuration information to the local partitions of the corresponding slice, and distribute the input activation vectors to the sub-computing units of the corresponding slice.

[0039] Therefore, the received data is uniformly classified by the protocol parsing unit, and the classified data is distributed to the corresponding storage or computing resources by the local routing unit in each slice. This enables the offloading chip to distinguish different types of input data and route them to the correct target module, thereby improving the accuracy of on-chip data distribution.

[0040] The protocol parsing unit can be located on the interface side of the offloading chip and is shared across the entire chip. The protocol parsing unit receives data transmitted via the standard high-speed interface module 130 and classifies it according to the identification information in the data. Configuration information refers to control data used to set the weight deployment, address mapping, or operating mode of local partitions within a slice.

[0041] Local routing units reside within each basic processing slice, and each slice contains one local routing unit. A local routing unit receives data categorized by the protocol parsing unit and distributes it to different targets within the slice based on data type: configuration information is distributed to local partitions to complete weight deployment or parameter configuration, and input activation vectors are distributed to sub-computation units to perform matrix-vector multiplication operations.

[0042] According to some embodiments, local partitions of multiple basic processing slices can store the weight matrices corresponding to Q-projection, K-projection, and V-projection, respectively. The input activation vector can be broadcast to multiple basic processing slices to perform matrix-vector multiplication operations corresponding to Q-projection, K-projection, and V-projection in parallel, obtaining three local results. The result processing module 124 can be configured to organize and package the three local results into a unified result, and then send the unified result back to the main processor 110 via the standard high-speed interface module 130.

[0043] Therefore, by deploying the weight matrices of Q-projection, K-projection, and V-projection in different slices respectively, and performing three-way matrix-vector multiplication operations in parallel on the same input activation vector, the number of interactions between the main processor 110 and the offloading subsystem 120 for attention projection is reduced, thereby improving the throughput of attention projection computation.

[0044] Q-projection, K-projection, and V-projection are three linear operators in the attention mechanism, used to transform the input activation vector into a query vector, a key vector, and a value vector, respectively. In this embodiment, multiple basic processing slices are divided into three groups, and the local partitions of each group store the weight matrices corresponding to the Q-projection, K-projection, and V-projection, respectively. After the same input activation vector is broadcast to all slices, the three groups of slices execute their respective matrix-vector multiplication operations in parallel, obtaining three local results.

[0045] A unified result refers to a single result data formed by the result processing module 124 after organizing and packaging the three local results within the unloading subsystem. After aligning the three local results in time, the result processing module 124 packages them into a single result frame and sends it back to the main processor 110 via the standard high-speed interface module 130. In some embodiments, the three local results may also be sent back to the main processor 110 separately without being packaged.

[0046] In some embodiments, the offloading subsystem 120 may provide synchronization information to the main processor 110 to indicate the completion status of computation and result return. The synchronization information may include one or more of a result completion flag, a partial completion flag, a transaction count, or a synchronization barrier flag. The main processor 110 may determine, based on the synchronization information, whether to immediately consume the returned output result vector, whether to wait for all multiple results to be ready, or whether to continue initiating the transmission of the input activation vector for the next operator.

[0047] According to some embodiments, the standard high-speed interface module 130 may include at least one of UCIe, DDR, LPDDR, GDDR, PCIe, CXL, InfiniBand, RDMA, Ethernet, and optical interconnect interfaces.

[0048] UCIe is a chip-to-chip interconnect standard that supports high-speed data transfer between dies or packages. DDR is a synchronous dynamic memory interface standard. LPDDR is a DDR interface variant for low-power applications. GDDR is a DDR interface variant for high-bandwidth graphics applications. All of these interfaces are standard protocols with mature industry ecosystems and publicly available physical layer specifications and electrical characteristics. Since the data transmitted by the standard high-speed interface module 130 is vector-level data rather than matrix-level data, the bandwidth of these standard interfaces can meet the transmission requirements of input activation vectors and output result vectors. Therefore, the computing system 100 does not need to rely on advanced packaging technologies such as CoWoS to maintain the repeated handling of large-scale weight matrices, reducing the system's packaging complexity and manufacturing cost.

[0049] In some embodiments, the standard high-speed interface module 130 may also employ PCIe, CXL, InfiniBand, RDMA, Ethernet, optical interconnect, or other high-speed interface protocols with publicly available specifications. Classified by interface semantics, the aforementioned standard high-speed interfaces can be divided into memory semantic interfaces and packet-switching interfaces. Memory semantic interfaces access data using memory read / write semantics; the memory access modes of DDR, LPDDR, GDDR, and CXL belong to this category. Packet-switching interfaces address and route data using data packets as transmission units; the I / O modes of PCIe and CXL, and Ethernet belong to this category. Classified by physical connection level, the aforementioned standard high-speed interfaces can be implemented as intra-package interconnects, inter-package interconnects, or board-level interconnects.

[0050] According to some embodiments, the input vector streaming interface 122 can be configured to receive multiple input activation vectors, each of which can carry an independent transaction identifier. The matrix-vector multiplication unit 123 can be configured to perform corresponding matrix-vector multiplication operations on each input activation vector based on the transaction identifier. The result processing module 124 can be configured to reduce or organize the results of each matrix-vector multiplication operation based on the transaction identifier, and return the corresponding output result vector via the standard high-speed interface module 130.

[0051] Therefore, by assigning an independent transaction identifier to each input activation vector, the offloading subsystem 120 can distinguish between input data and output results from different tokens, support concurrent processing of activation vectors of multiple tokens, and improve the computational throughput of the offloading subsystem 120 in multi-token scenarios.

[0052] A transaction identifier is an identification information appended to each input activation vector, used to distinguish different computational tasks within the offloading subsystem 120. In an exemplary embodiment, the transaction identifier may include one or more of the following: a token number, a layer number, an operator number, a sequence number, a slice number, or return address information. The slice number specifies the target slice or slice group to which the input activation vector should be routed. The return address information indicates the target address or target buffer location when the output result vector is returned to the main processor 110.

[0053] When sending input activation vectors, the main processor 110 appends a transaction identifier to the data. The offloading subsystem 120 routes, schedules, and returns the results to different input activation vectors according to the transaction identifier.

[0054] In a multi-Token concurrent scenario, the main processor 110 can send the input activation vectors of multiple Tokens to the offloading subsystem 120 in sequence or in batches. The offloading subsystem 120 allocates computing resources and result buffers for different Tokens according to the transaction identifier of each input activation vector, and after the calculation is completed, it sends the output result vector corresponding to each Token, along with its transaction identifier, back to the main processor 110.

[0055] In some embodiments, the return order of the output result vectors corresponding to each Token can be consistent with the input order or inconsistent with the input order. When the return order is inconsistent with the input order, the offloading subsystem 120 can append a transaction identifier to the output result vector, and the main processor 110 matches the out-of-order returned output result vectors with the corresponding Tokens based on the transaction identifier.

[0056] According to some embodiments, the standard high-speed interface module 130 may employ one or more of credit-based flow control, virtual channels, priority arbitration, or time-division multiplexing to schedule the transmission of multiple input activation vectors and corresponding output result vectors.

[0057] Therefore, by adopting flow control and scheduling mechanisms in the standard high-speed interface module 130, the risk of interface congestion in multi-Token concurrent scenarios is reduced, and the utilization efficiency of interface bandwidth is improved.

[0058] Credit-based flow control requires the sender to hold a credit limit granted by the receiver before sending data. When the receiver's buffer resources are insufficient, credit granting ceases, thus preventing data overflow. Virtual channels involve dividing a physical link into multiple logical channels, allowing different types of data to be transmitted in different virtual channels to avoid mutual blocking. Priority arbitration assigns priorities to different data streams; when multiple data streams compete for the same transmission resource, the higher-priority data is transmitted first. Time-division multiplexing divides the interface's transmission time into multiple time slots, allowing different data to be transmitted in turn in different time slots.

[0059] In some embodiments, the standard high-speed interface module 130 may also employ a ready / valid handshake mechanism or a tag / response transaction mechanism. The ready / valid handshake mechanism means that the sender indicates data validity via a valid signal, and the receiver indicates readiness via a ready signal; data transmission is completed when both valid and ready signals are valid simultaneously. The tag / response transaction mechanism means that the sender attaches a transaction tag when sending data, and the receiver carries the corresponding tag in the returned result, enabling the sender to match the returned data with the original request.

[0060] According to some embodiments, the weight matrix can be encoded in floating-point format, fixed-point format, or mixed-precision format and stored in a weight-locked storage array. The floating-point format may include FP32, TF32, BF16, FP16, or FP8, and the fixed-point format may include INT16, INT8, INT4, INT2, or a binary format.

[0061] Floating-point formats use a sign bit, exponent bit, and mantissa bit to represent weight values. Different floating-point formats differ in the total bit width and the allocation of exponent and mantissa bits. In an exemplary embodiment, FP32 is a 32-bit floating-point format, TF32 is a 19-bit floating-point format, BF16 and FP16 are 16-bit floating-point formats, and FP8 is an 8-bit floating-point format. Fixed-point formats use a fixed integer bit width to represent weight values. In an exemplary embodiment, INT16, INT8, INT4, and INT2 use 16, 8, 4, and 2 bits to represent weight values, respectively. Binary formats use 1 bit to represent weight values, with each weight taking only two discrete values.

[0062] In some embodiments, for layers with high precision requirements, the weight matrix can be encoded and stored in FP16, BF16, or FP32 format. For layers with low precision requirements, the weight matrix can be encoded and stored in INT8, INT4, INT2, or binary format, thereby accommodating more weight parameters with the same storage capacity.

[0063] In some embodiments, when a mixed-precision format is used, different layers of the same model can use different encoding formats. Furthermore, weight data of different precision levels can be deployed in different subarrays of the weight-locked storage array, and the read and multiplication paths of each subarray can be configured according to the precision level it stores.

[0064] According to some embodiments, the result processing module 124 can be configured to truncate, round or re-encode the result of the matrix-vector multiplication operation and then output the output result vector via the standard high-speed interface module 130.

[0065] Therefore, by truncating, rounding or recoding the calculation results within the unloading subsystem 120, the data bit width of the output result vector is reduced, thereby reducing the amount of data transmitted by the standard high-speed interface module 130 and improving the utilization efficiency of the interface bandwidth.

[0066] Truncating refers to discarding low-order bits in the result data that exceed the target bit width. Rounding refers to approximating the result data according to rounding rules, mapping it to the nearest representable value within the target precision range. Recoding refers to converting the result data from one numerical encoding format to another. In an exemplary embodiment, the intermediate result of the matrix-vector multiplication operation is generated with higher internal precision. After the result processing module 124 performs truncating or rounding on the intermediate result, it outputs the result vector in the target precision format agreed upon with the main processor 110.

[0067] According to some embodiments, the offloading subsystem 120 may also include an asynchronous buffer module configured to decouple the interface clock domain of the standard high-speed interface module 130 from the computation clock domain of the matrix-vector multiplication calculation unit 123.

[0068] Therefore, by setting an asynchronous buffer module between the interface clock domain and the computing clock domain, the standard high-speed interface module 130 and the matrix-vector multiplication computing unit 123 can operate at their respective independent clock frequencies, avoiding data transmission blockage or computing pauses caused by clock frequency differences, and improving the operational stability of the computing system 100.

[0069] The interface clock domain is the clock domain in which the standard high-speed interface module 130 operates, and its clock frequency is specified by the interface protocol. The computation clock domain is the clock domain in which the matrix-vector multiplication computation unit 123 operates, and its clock frequency is determined by the computation logic within the offloading subsystem 120. When the clock frequencies of the two are different, the asynchronous buffer module buffers data between the two clock domains, enabling the input activation vector and the output result vector to be safely transferred between the two clock domains. In an exemplary embodiment, the asynchronous buffer module may be implemented using an asynchronous FIFO or a resilient buffer.

[0070] According to some embodiments, the weight-locked memory array 121 may include MRAM and / or SRAM.

[0071] Therefore, by constructing a weight-locked storage array 121 using MRAM and / or SRAM, the weight matrix can be stored in the offloading subsystem 120 in a manner suitable for high-bandwidth reading, supporting high-speed access to the weight data by the matrix-vector multiplication calculation unit 123.

[0072] MRAM is a non-volatile memory that retains stored data even when power is off and does not require dynamic refreshing, making it suitable for long-term storage of weight matrices and providing stable read timing. SRAM is a volatile memory with faster read and write speeds. In one exemplary embodiment, the weight-locked memory array 121 consists only of MRAM, utilizing the non-volatile characteristics and storage density advantages of MRAM to achieve long-term resident status of the weight matrix. In another exemplary embodiment, the weight-locked memory array 121 is composed of a combination of MRAM and SRAM, with MRAM used to store the main body of the weight matrix and SRAM used to cache frequently accessed weight data or intermediate data.

[0073] In some embodiments, the weight-locked memory array 121 may also be composed of Flash, RRAM, or PCM. Flash is a non-volatile memory that uses a floating gate or charge trap structure to store data. RRAM is a non-volatile memory based on the principle of resistance change. PCM is a non-volatile memory based on the resistance state change of a phase change material. All of the above storage media can be used to realize the long-term resident storage of the weight matrix.

[0074] According to some embodiments, at least some linear operators may include one or more of Q-projection, K-projection, V-projection, output projection, projection onto a feedforward network, gated projection, and projection under a feedforward network.

[0075] Therefore, by specifying the specific type of the unloaded linear operator, the computational range performed by the unloading subsystem 120 corresponds to the weight-intensive linear transformation operation in the decoding stage, which facilitates determining the unloading range based on the model structure and system configuration.

[0076] Q-projection, K-projection, and V-projection are linear operators in the attention mechanism that transform the input activation vector into a query vector, key vector, and value vector, respectively. Output projection is a linear operator in the attention mechanism that transforms the attention calculation result back to the model's hidden dimensions. Projection over a feedforward network is a linear operator in a feedforward network that transforms the input activation vector from the model's hidden dimensions to the feedforward network's internal dimensions. Gated projection is a linear operator in a gated feedforward network that executes in parallel with up-projection. Down-projection over a feedforward network is a linear operator in a feedforward network that transforms the internal dimensions back to the model's hidden dimensions.

[0077] In one exemplary embodiment, the computing system 100 may offload only the Q-projection, K-projection, and V-projection to the offloading subsystem 120 for execution. In another exemplary embodiment, the computing system 100 may offload all of the above linear operators to the offloading subsystem 120 for execution. The offloading scope may be determined based on the model structure, the capacity of the weight-locked storage array 121, and the system bandwidth configuration.

[0078] According to some embodiments, the main processor 110 may include at least one of a GPU, NPU, CPU, SoC processor, or edge AI processor.

[0079] Thus, by supporting multiple types of main processors, the computing system 100 can be adapted to different deployment scenarios and computing platforms, improving the system's versatility.

[0080] GPU is a graphics processing unit designed for massively parallel computing. NPU is a dedicated processor for neural network inference. CPU is a general-purpose processor. SoC (System-on-a-Chip) processor is a system-on-a-chip that integrates multiple functional modules onto the same chip. Edge AI processors are AI inference processors designed for terminal devices or edge computing scenarios. All of the above processor types can serve as the main processor 110, responsible for executing nonlinear operators in the decoding stage and working collaboratively with the offloading subsystem 120 through the standard high-speed interface module 130.

[0081] According to some embodiments, the offloading subsystem 120 may include multiple offloading chips, which can be connected by at least one of in-package interconnect, inter-package interconnect, board-level interconnect or network interconnect, and the weighted locked storage array and the matrix-vector multiplication calculation unit are both distributed in the multiple offloading chips.

[0082] Therefore, by organizing the offloading subsystem 120 into multiple offloading chips and interconnecting them, the weight storage capacity and computing power of the offloading subsystem 120 can be expanded by increasing the number of offloading chips, thereby improving the scalability of the system.

[0083] In some embodiments, the input vector stream interface 122 and the result processing module 124 may be located in one or more offload chips, or they may be independent of these offload chips. Each offload chip may further contain multiple basic processing slices. The topology between the multiple offload chips may be a bus, ring, star, mesh, or other topology.

[0084] In some embodiments, the offloading subsystem 120 may further include bridging nodes, switching nodes, or aggregation nodes for data forwarding, aggregation, or format conversion among multiple offloading chips.

[0085] In some embodiments, multiple offload chips can be connected via inter-chip interconnect interfaces such as UCIe, DDR, LPDDR, GDDR, PCIe, and CXL, or via vendor-defined interconnects such as NVLink, custom die-to-die interconnects, serial transceiver links, or other dedicated interconnect interfaces.

[0086] In some embodiments, the interconnection interface used between the multiple offloading chips may be the same as or different from the standard high-speed interface module used between the offloading subsystem 120 and the main processor 110. In an exemplary embodiment, the multiple offloading chips are interconnected via a UCIe interface, and the offloading subsystem 120 and the main processor 110 are connected via an Ethernet or RDMA interface.

[0087] According to another aspect of this disclosure, a data processing method for high-bandwidth inference is provided, specifically the decoding stage of Transformer-based model inference. For example... Figure 2 As shown, the data processing method 200 includes: step S201, storing the weight matrix corresponding to at least a portion of the linear operators in the decoding stage in the storage array of the execution unit, and keeping the weight matrix stored in a static resident manner during the decoding stage; step S202, streaming the input activation vector corresponding to at least a portion of the linear operators to the execution unit via an external interface; step S203, performing matrix-vector multiplication on the input activation vector and the weight matrix inside the execution unit; step S204, reducing or tidying the result of the matrix-vector multiplication to obtain an output result vector; and step S205, outputting the output result vector via an external interface.

[0088] Therefore, this disclosure stores the weight matrices corresponding to at least a portion of the linear operators in the decoding stage in a statically resident manner in the storage array of the execution unit. The matrix-vector multiplication operation between the input activation vector and the weight matrix and the result reduction or sorting are completed inside the execution unit. The external interface only transmits the input activation vector and the output result vector, thereby changing the data transmission object of the external interface from matrix-level data to vector-level data, reducing the external bandwidth requirements.

[0089] Furthermore, the partial sums generated during matrix-vector multiplication are aggregated within the execution unit without being transmitted through an external interface. The output vector is reduced within the execution unit before being output, further avoiding the use of external interface bandwidth by intermediate data.

[0090] The following describes each step of data processing method 200.

[0091] In step S201, the weight matrices corresponding to at least a portion of the linear operators in the decoding phase are stored in the storage array of the execution unit, and the weight matrices are stored in a statically resident manner during the decoding phase. After being written to the storage array, the weight matrices remain resident during the decoding phase and are not repeatedly loaded via an external interface.

[0092] In some embodiments, the static residency state of the weight matrix in the storage array can be independent of KV cache update operations during the decoding phase. The KV cache is used to store the key vectors and value vectors of the generated tokens and can be dynamically updated in each decoding step. Updates to the KV cache do not change the residency state of the weight matrix in the storage array.

[0093] In step S202, the input activation vectors corresponding to at least a portion of the linear operators are streamed into the execution unit via an external interface. During the decoding phase, after each current stage processing is completed, the input activation vector corresponding to the current Token is sent to the execution unit via the external interface.

[0094] In step S203, within the execution unit, a matrix-vector multiplication operation is performed on the input activation vector and the weight matrix. The execution unit reads the weight matrix from the storage array and performs a matrix-vector multiplication operation with the input activation vector. The weight reading required for this operation is completed internally by the execution unit, without going through an external interface.

[0095] In step S204, the result of the matrix-vector multiplication operation is reduced or rearranged to obtain the output result vector.

[0096] In step S205, the output result vector obtained in step S204 is output via an external interface.

[0097] In the aforementioned data processing method 200, since the weight matrix is ​​stored statically in the execution unit's storage array, the external interface no longer repeatedly transmits the complete weight matrix. Taking a linear operator with a model hidden dimension of 'd' as an example, the size of its weight matrix is ​​proportional to the square of 'd'. In traditional execution methods, each decoding step requires loading the complete weight matrix via the external interface, and the data transmission volume of the external interface is proportional to the square of 'd'. In data processing method 200, the weight matrix resides within the execution unit, and the external interface only transmits the input activation vector and the output result vector, both of which have dimensions proportional to 'd'. Therefore, the data transmission volume of the external interface is linearly proportional to 'd'. Thus, data processing method 200 reduces the data transmission volume of the external interface from a square-order magnitude related to the weight matrix size to a linear magnitude related to the vector dimension.

[0098] In some embodiments, the reduction in data transmission scale described above can be verified through algorithm-architecture co-simulation. The simulation model can decompose the decoding stage into normalization nodes, QKV projection nodes, attention nodes, residual nodes, and feedforward network nodes, and establish transaction-level models for the main processor, external interface links, protocol parsing unit, routing module, weight-locked storage array, vector register, matrix-vector multiplication calculation unit, result processing module, and result return path, respectively. Simulation parameters may include model hidden dimension, number of attention heads, feedforward network expansion factor, weight precision, number of basic processing slices, interface bit width, buffer depth, token concurrency, and transaction scheduling strategy.

[0099] During the simulation, the following execution methods can be modeled respectively: unloading a single linear operator to the execution unit for execution; merging multiple linear operators in the same layer for execution within the execution unit; or executing all linear operators in the same layer continuously within the execution unit, so that the data round trip between the main processor and the execution unit for each layer is compressed into a single input and a single output.

[0100] The simulation statistics can include the number of transactions across external interfaces, the flow of input activation vectors and output result vectors, the weight read volume within the execution unit slice, local reduction volume, buffer usage, wait cycle, and token-level end-to-end latency. The simulation verifies that when the weight matrix is ​​stored statically within the execution unit, the dominant data transfer term of the external interface changes from weight matrix transport to the transmission of input activation vectors and output result vectors. Furthermore, as the degree of linear operator fusion within the execution unit increases, the number of data round trips per layer between the main processor and the execution unit decreases accordingly. The data processing method 200 can be implemented in various hardware configurations. In one implementation, the execution unit is an offloading subsystem separate from the main processor, the storage array is a weight-locked storage array, and the external interface is a standard high-speed interface module. This implementation corresponds to the computing system 100 described above. Further implementation methods, optional embodiments, and technical details of each step of the data processing method 200 in this implementation can be found in the description of the functional units in the computing system 100 above.

[0101] In another implementation, the execution unit is a near-memory computing array, where the storage array is tightly coupled with the computing logic, the weight matrix is ​​stored in the storage array and matrix-vector multiplication is performed locally, and the external interface is the data path between the near-memory computing array and the main processor.

[0102] In another implementation, the execution unit is a dedicated matrix-vector multiplication unit within a GPU, NPU, CPU, or SoC processor; the storage array is a local storage structure within the processor used for weight residency; and the external interface is the data path between this dedicated unit and other functional modules within the processor. In this implementation, the weight matrix is ​​locked in the local storage structure within the processor, and the input activation vector is streamed into the processor to perform the matrix-vector multiplication operation.

[0103] The common feature of the above implementation methods is that the weight matrix is ​​stored statically in the execution unit's storage array, matrix-vector multiplication is performed internally within the execution unit, and the external interface only transmits the input activation vector and the output result vector. Regardless of the specific physical form of the execution unit, the data transmission objects of the external interface are vector-level data rather than matrix-level data.

[0104] According to some embodiments, the execution unit may be an offloading subsystem separate from the main processor, the storage array may be a weighted locked storage array, the external interface may be a standard high-speed interface module, and the main processor may be configured to execute the nonlinear operators of the decoding stage.

[0105] In this embodiment, step S201 corresponds to storing the weight matrix in a statically resident manner in the weight-locked storage array within the offloading subsystem. Step S202 corresponds to the main processor sending the input activation vector to the offloading subsystem via the standard high-speed interface module. Steps S203 and S204 correspond to the offloading subsystem performing matrix-vector multiplication and result reduction or sorting internally. Step S205 corresponds to the offloading subsystem sending the output result vector back to the main processor via the standard high-speed interface module. Further implementation methods and technical details of each step can be found in the above description of the functional units in the computing system 100.

[0106] In an exemplary embodiment, taking the execution flow of a certain layer in the decoding stage as an example, after the main processor completes the layer normalization and position encoding for that layer, it can send the current input activation vector to the offloading subsystem via a standard high-speed interface module. The offloading subsystem internally performs matrix-vector multiplication operations on the Q-projection, K-projection, and V-projection of that layer and sends the output vector back to the main processor. The main processor uses this result to perform nonlinear operations such as attention calculation and residual connection, and then sends the input activation vector of the next linear operator to the offloading subsystem. This process can be repeated in each layer until all linear operators for that layer have been executed.

[0107] According to some embodiments, the execution unit may include multiple basic processing slices, each of which may include a local partition of the storage array and a sub-computation unit that performs the matrix-vector multiplication operation. In step S203, performing the matrix-vector multiplication operation on the input activation vector and the weight matrix may include: broadcasting the input activation vector to the multiple basic processing slices, with each basic processing slice reading the corresponding weights from its local partition and having its sub-computation unit perform the matrix-vector multiplication operation to obtain a local result. In step S204, reducing or tidying the result of the matrix-vector multiplication operation to obtain an output result vector may include: reducing or tidying the local results of the multiple basic processing slices to obtain the output result vector.

[0108] Therefore, by organizing the execution unit into multiple basic processing slices and broadcasting the input activation vector to each slice for parallel computation, the matrix-vector multiplication operation in step S203 can be performed simultaneously on multiple slices, thereby improving computational parallelism.

[0109] In some embodiments, the local partitions of each basic processing slice can store a portion of the weight matrix, and each sub-computation unit can perform matrix-vector multiplication operations corresponding to its respective local partition on the same input activation vector. The local results output by each slice are reduced or rearranged in step S204 to form a complete output result vector. When the execution unit is the unloading subsystem 120 described above, the basic processing slices, local partitions, sub-computation units, and local results correspond to the corresponding modules in the computing system 100 described above.

[0110] According to some embodiments, the execution unit may further include a protocol parsing unit, and each basic processing slice may further include a local routing unit. The method may further include: within the execution unit, the protocol parsing unit classifies the data received via the external interface, identifying configuration information and the input activation vector therein; and the local routing unit in each basic processing slice distributes the configuration information to local partitions of the corresponding slice, and distributes the input activation vector to sub-computing units of the corresponding slice.

[0111] Therefore, by classifying the received data through the protocol parsing unit and distributing it by the local routing unit of each slice, the execution unit can distinguish different types of input data and accurately route them to the corresponding storage or computing resources, thereby improving the accuracy of on-chip data distribution.

[0112] After classifying the data received via the external interface, the protocol parsing unit distributes the configuration information and input activation vectors along different intra-chip paths. The local routing units of each slice send the configuration information to the local partition and the input activation vectors to the sub-computing units according to the data type. When the execution unit is the unloading subsystem 120 described above, the protocol parsing unit and the local routing unit correspond to the respective modules described above in the computing system 100.

[0113] According to some embodiments, the local partitions of the multiple basic processing slices can respectively store the weight matrices corresponding to Q-projection, K-projection, and V-projection. In step S203, performing matrix-vector multiplication on the input activation vector and the weight matrix may include: broadcasting the input activation vector to the multiple basic processing slices to perform the matrix-vector multiplication operations corresponding to Q-projection, K-projection, and V-projection in parallel, obtaining three local results. In step S204, reducing or tidying the results of the matrix-vector multiplication to obtain the output result vector may include: tidying up and packaging the three local results into a unified result, which is used as the output result vector.

[0114] Therefore, by deploying the weight matrices of Q-projection, K-projection, and V-projection in multiple slices respectively and performing three-way matrix-vector multiplication operations in parallel on the same input activation vector, the number of interactions between the execution unit and the outside world for attention projection is reduced, thereby improving the throughput of attention projection computation.

[0115] In some embodiments, multiple basic processing slices can be divided into three groups, each group responsible for matrix-vector multiplication operations of Q-projection, K-projection, and V-projection, respectively. After the same input activation vector is broadcast to all slices, the three groups of slices can execute their respective operations in parallel, obtaining three local results. The three local results can be time-aligned within the execution unit, packaged into a unified result, and output via an external interface. When the execution unit is the unloading subsystem 120 described above, the above process corresponds to the description of the parallel execution of QKV in the computing system 100 described above.

[0116] According to some embodiments, there can be multiple input activation vectors, and each input activation vector can carry an independent transaction identifier. In step S203, performing a matrix-vector multiplication operation on the input activation vector and the weight matrix may include: performing the corresponding matrix-vector multiplication operation on each input activation vector based on the transaction identifier. In step S204, reducing or simplifying the result of the matrix-vector multiplication operation to obtain the output result vector may include: reducing or simplifying the result of each matrix-vector multiplication operation based on the transaction identifier to obtain the output result vector.

[0117] Therefore, by assigning an independent transaction identifier to each input activation vector, the execution unit can distinguish between input data and computation results from different tokens, supporting concurrent processing of activation vectors of multiple tokens and improving computational throughput in multi-token scenarios.

[0118] In some embodiments, in a multi-Token concurrent scenario, input activation vectors corresponding to multiple Tokens are sent to the execution unit sequentially or in batches via an external interface. Each input activation vector may carry an independent transaction identifier. The execution unit internally performs computation scheduling on each input activation vector according to the transaction identifier, and after computation, outputs the output result vectors corresponding to each Token via the external interface according to the transaction identifier. When the execution unit is the unloading subsystem 120 described above, the content and processing method of the transaction identifier correspond to the description of the multi-Token concurrent mechanism in the computing system 100 above.

[0119] According to some embodiments, the weight matrix may employ BF16, FP16, INT8, INT4, or mixed-precision encoding. In step S205, outputting the output result vector via the external interface may include: truncating, rounding, or re-encoding the result of the matrix-vector multiplication operation, and then outputting the output result vector via the external interface.

[0120] Therefore, by using multiple precision encoding formats to store the weight matrix, and performing truncation, rounding, or re-encoding on the calculation results inside the execution unit before outputting them through the external interface, the weight capacity of the storage array is improved, the data bit width of the output result vector is reduced, and the amount of data transmitted through the external interface is reduced.

[0121] In some embodiments, the precision encoding format of the weight matrix can be determined according to the precision requirements of different linear operators or different model layers. Intermediate results of matrix-vector multiplication can be generated with higher internal precision. Before output in step S205, the execution unit can truncate, round, or re-encode the intermediate results to convert them into the target precision format before outputting them via the external interface. When the execution unit is the unloading subsystem 120 described above, the precision encoding and result processing methods correspond to the descriptions of the corresponding modules in the computing system 100 above.

[0122] According to some embodiments, the weight matrices corresponding to multiple linear operators in the same layer of the decoding stage can be stored separately in the execution unit. Step S203, performing a matrix-vector multiplication operation on the input activation vector and the weight matrix, may include: continuously executing multiple linear operators in the same layer within the execution unit in a pipelined or parallel manner, such that the number of data round trips between the main processor and the execution unit for the same layer is less than the number of linear operators in the same layer.

[0123] Therefore, by executing multiple linear operators in the same layer consecutively within the execution unit, the number of data round trips between the main processor and the execution unit is reduced, further reducing the transmission overhead of the external interface and the latency caused by round-trip communication.

[0124] Each layer in the decoding stage typically contains multiple linear operators. In the non-fusion execution mode, the input activation vector of each linear operator is sent from the main processor to the execution unit, and the output vector of each linear operator is returned to the main processor. The number of round trips between the main processor and the execution unit is equal to the number of linear operators in that layer. In some embodiments, the weight matrices corresponding to multiple linear operators in the same layer are stored in the execution unit. After the main processor sends the input activation vector of the layer to the execution unit, the execution unit can internally perform matrix-vector multiplication operations of multiple linear operators sequentially or in parallel, and only return the final result or necessary intermediate results to the main processor.

[0125] In an exemplary embodiment, the weight matrices of the Q-projection, K-projection, and V-projection of the same layer are all stored in the execution unit. After the main processor sends an input activation vector once, the execution unit performs the matrix-vector multiplication operation of the three projections in parallel internally. The main processor only needs to receive the output result once, and the number of round trips between the main processor and the execution unit is reduced from three to one.

[0126] In some embodiments, the multiple linear operators can be executed in a pipelined manner, meaning that the output of one linear operator is passed within the execution unit as the input of the next linear operator, without needing to go through an external interface. In other embodiments, the multiple linear operators can be executed in parallel, meaning that multiple linear operators simultaneously perform matrix-vector multiplication on the same input activation vector, and their respective outputs are converged within the execution unit before being output through an external interface.

[0127] According to one aspect of this disclosure, a computer device is provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0128] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0129] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0130] In the following text, combined with Figure 3 Illustrative examples describing such computer devices, non-transitory computer-readable storage media, and computer program products.

[0131] Figure 3 An example configuration of a computer device 300 that can be used to implement the methods described herein is shown.

[0132] Computer device 300 can be a variety of different types of devices. Examples of computer device 300 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0133] Computer device 300 may include at least one processor 302, memory 304, multiple communication interfaces 306, display device 308, other input / output (I / O) devices 310, and one or more mass storage devices 312 capable of communicating with each other, such as via system bus 314 or other suitable connections.

[0134] Processor 302 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 302 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 302 may be configured to acquire and execute computer-readable instructions stored in memory 304, mass storage device 312, or other computer-readable media, such as program code of operating system 316, program code of application program 318, program code of other program 320, etc.

[0135] Memory 304 and mass storage device 312 are examples of computer-readable storage media for storing instructions executed by processor 302 to perform the various functions described above. For example, memory 304 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 312 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 304 and mass storage device 312 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 302 as a specific machine configured to perform the operations and functions described in the examples herein.

[0136] Multiple programs may be stored on mass storage device 312. These programs include operating system 316, one or more application programs 318, other programs 320, and program data 322, and they may be loaded into memory 304 for execution. Examples of such application programs or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the following components / functions: the methods described herein (including any suitable steps of the methods), and / or other embodiments described herein.

[0137] Although illustrated as being stored in memory 304 of computer device 300, modules 316, 318, 320, and 322, or portions thereof, may be implemented using any form of computer-readable medium accessible by computer device 300. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0138] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by computer devices. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0139] One or more communication interfaces 306 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as WLAN) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth™ interface, Near Field Communication (NFC) interface, etc. Communication interface 306 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 306 can also provide communication with external storage devices (not shown), such as storage arrays, network-attached storage, storage area networks, etc.

[0140] In some examples, a display device 308, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 310 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0141] The technologies described herein can be supported by these various configurations of computer device 300, and are not limited to specific examples of the technologies described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on servers remote from computer device 300. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect computer device 300 to other computer devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality may be implemented partly on computer device 300 and partly through a platform that abstracts the functionality of the cloud.

[0142] Although this disclosure has been described and illustrated in detail in the accompanying drawings and the foregoing description, such description and illustration should be considered illustrative and suggestive, not restrictive; this disclosure is not limited to the disclosed embodiments. By studying the drawings, the disclosure, and the appended claims, those skilled in the art will be able to understand and implement variations of the disclosed embodiments in practice with respect to the claimed subject matter. In the claims, the word "comprising" does not exclude other elements or steps not listed, the indefinite article "a" or "an" does not exclude a plurality, the term "a plurality" means two or more, and the term "based on" should be interpreted as "at least partially based on". The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be beneficial.

Claims

1. A computing system for performing a decoding phase of Transformer-based model inference, the computing system comprising: The main processor is configured to execute the nonlinear operators of the decoding phase; An unloading subsystem, configured to execute at least a portion of the linear operators of the decoding phase, the unloading subsystem comprising: A weight-locked storage array is configured to store weight matrices corresponding to the at least a subset of linear operators in a statically residing manner during the decoding phase. An input vector streaming interface is configured to stream input activation vectors corresponding to at least a subset of the linear operators; A matrix-vector multiplication unit is configured to perform a matrix-vector multiplication operation on the input activation vector and the weight matrix; and The result processing module is configured to reduce or rearrange the result of the matrix-vector multiplication operation to obtain an output result vector; and A standard high-speed interface module is configured to transmit the input activation vector and the output result vector between the main processor and the offloading subsystem.

2. The computing system according to claim 1, wherein, The unloading subsystem includes an unloading chip, which comprises multiple basic processing slices. Each basic processing slice includes a local partition of the weight-locked storage array and a sub-computation unit of the matrix-vector multiplication calculation unit. The input activation vector is broadcast to the multiple basic processing slices. Each basic processing slice reads the corresponding weights from its local partitions and performs the matrix-vector multiplication operation by its sub-computation units to obtain a local result. The result processing module reduces or organizes the local results of the multiple basic processing slices to obtain the output result vector.

3. The computing system according to claim 2, wherein, The local partitions of the multiple basic processing slices respectively store different blocks of the same weight matrix or the weight matrices corresponding to multiple linear operators.

4. The computing system according to claim 2, wherein, The offloading chip also includes a protocol parsing unit configured to classify data received via the standard high-speed interface module, and identify configuration information and the input activation vector therein. Each basic processing slice also includes a local routing unit, which is configured to receive the data classified by the protocol parsing unit, distribute the configuration information to the local partitions of the corresponding slice, and distribute the input activation vector to the sub-computation units of the corresponding slice.

5. The computing system according to claim 2, wherein, The local partitions of the multiple basic processing slices respectively store the weight matrices corresponding to Q-projection, K-projection and V-projection. The input activation vector is broadcast to the multiple basic processing slices to perform matrix-vector multiplication operations corresponding to Q-projection, K-projection and V-projection in parallel to obtain three local results. The result processing module is configured to organize and package the three local results into a unified result, and send the unified result back to the main processor via the standard high-speed interface module.

6. The computing system according to any one of claims 1-5, wherein, The standard high-speed interface module includes at least one of UCIe, DDR, LPDDR, GDDR, PCIe, CXL, InfiniBand, RDMA, Ethernet, and optical interconnect interfaces.

7. The computing system according to any one of claims 1-5, wherein, The input vector streaming interface is configured to receive multiple input activation vectors, each carrying an independent transaction identifier. The matrix-vector multiplication calculation unit is configured to perform corresponding matrix-vector multiplication operations on each input activation vector based on the transaction identifier. The result processing module is configured to reduce or organize the results of each matrix-vector multiplication operation based on the transaction identifier and return the corresponding output result vector via the standard high-speed interface module.

8. The computing system according to claim 7, wherein, The standard high-speed interface module employs one or more of the following: credit-based flow control, virtual channels, priority arbitration, or time-division multiplexing, to schedule the transmission of the multiple input activation vectors and their corresponding output result vectors.

9. The computing system according to any one of claims 1-5, wherein, The weight matrix is ​​encoded in floating-point format, fixed-point format, or mixed-precision format and stored in the weight-locked storage array; wherein the floating-point format includes FP32, TF32, BF16, FP16, or FP8; and the fixed-point format includes INT16, INT8, INT4, INT2, or binary format.

10. The computing system according to claim 9, wherein, The result processing module is configured to truncate, round, or re-encode the result of the matrix-vector multiplication operation, and then output the output result vector via the standard high-speed interface module.

11. The computing system according to any one of claims 1-5, wherein, The offloading subsystem also includes an asynchronous buffer module configured to decouple the interface clock domain of the standard high-speed interface module from the computation clock domain of the matrix-vector multiplication calculation unit.

12. The computing system according to any one of claims 1-5, wherein, The weighted locked memory array includes MRAM and / or SRAM.

13. The computing system according to any one of claims 1-5, wherein, The at least some linear operators include one or more of the following: Q projection, K projection, V projection, output projection, projection over a feedforward network, gated projection, and projection under a feedforward network.

14. The computing system according to any one of claims 1-5, wherein, The offloading subsystem includes multiple offloading chips, which are connected by at least one of in-package interconnect, inter-package interconnect, board-level interconnect, or network interconnect. The weight-locked storage array and the matrix-vector multiplication calculation unit are both distributed in the multiple offloading chips.

15. A data processing method for high-bandwidth inference, used in the decoding stage of Transformer-based model inference, the method comprising: The weight matrices corresponding to at least a portion of the linear operators in the decoding stage are stored in the storage array of the execution unit, and the weight matrices are stored in a static resident manner during the decoding stage. The input activation vectors corresponding to at least a portion of the linear operators are streamed into the execution unit via an external interface; Inside the execution unit, Perform a matrix-vector multiplication operation on the input activation vector and the weight matrix; as well as The result of the matrix-vector multiplication operation is reduced or rearranged to obtain the output vector; and The output result vector is output via the external interface.

16. The method according to claim 15, wherein, The execution unit is an offloading subsystem separate from the main processor, the storage array is a weighted locked storage array, the external interface is a standard high-speed interface module, and the main processor is configured to execute the nonlinear operators of the decoding stage.

17. The method according to claim 15 or 16, wherein, The execution unit includes multiple basic processing slices, each of which includes a local partition of the storage array and a sub-computation unit that performs the matrix-vector multiplication operation. Specifically, within the execution unit, performing a matrix-vector multiplication operation on the input activation vector and the weight matrix includes: The input activation vector is broadcast to the multiple basic processing slices. Each basic processing slice reads the corresponding weights from its local partitions and performs the matrix-vector multiplication operation in its sub-computation units to obtain a local result. The reduction or simplification of the result of the matrix-vector multiplication to obtain the output vector includes: The local results of the multiple basic processing slices are reduced or sorted to obtain the output result vector.

18. The method of claim 17, wherein, The execution unit further includes a protocol parsing unit, and each basic processing slice further includes a local routing unit. The method further includes: Within the execution unit, the protocol parsing unit classifies the data received via the external interface, identifying configuration information and the input activation vector; and The configuration information is distributed to the local partitions of the corresponding slice by the local routing unit in each basic processing slice, and the input activation vector is distributed to the sub-computation unit of the corresponding slice.

19. The method of claim 17, wherein, The local partitions of the multiple basic processing slices respectively store the weight matrices corresponding to the Q projection, K projection, and V projection. Specifically, within the execution unit, performing a matrix-vector multiplication operation on the input activation vector and the weight matrix includes: The input activation vector is broadcast to the multiple basic processing slices to perform matrix-vector multiplication operations corresponding to Q-projection, K-projection, and V-projection in parallel, resulting in three local results. The result of the matrix-vector multiplication operation is reduced or rearranged to obtain the output vector, which includes: The three local results are organized and packaged into a unified result, which is used as the output result vector.

20. The method according to claim 15 or 16, wherein, There are multiple input activation vectors, each carrying an independent transaction identifier. Specifically, within the execution unit, performing a matrix-vector multiplication operation on the input activation vector and the weight matrix includes: Based on the transaction identifier, perform the corresponding matrix-vector multiplication operation on each input activation vector. The result of the matrix-vector multiplication operation is reduced or rearranged to obtain the output vector, which includes: Based on the transaction identifier, the results of each matrix-vector multiplication operation are reduced or rearranged to obtain the output result vector.

21. The method according to claim 15 or 16, wherein, The weight matrix is ​​encoded using BF16, FP16, INT8, INT4, or mixed precision. The step of outputting the output vector via the external interface includes: After truncating, rounding, or recoding the result of the matrix-vector multiplication operation, the output result vector is output through the external interface.

22. The method according to claim 16, wherein, The weight matrices corresponding to multiple linear operators in the same layer of the decoding stage are stored in the execution unit, and the execution unit contains... Performing matrix-vector multiplication on the input activation vector and the weight matrix includes: executing multiple linear operators in the same layer continuously within the execution unit in a pipelined or parallel manner, so that the number of data round trips between the main processor and the execution unit for the same layer is less than the number of linear operators in the same layer.

23. A computer device, comprising: The computing system according to any one of claims 1-14.

24. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the method of any one of claims 15-22.

25. A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 15-22.

Images