A routing-aware quantization heterogeneous inference method of hybrid expert large language model

Abstract

This application discloses a route-aware quantization heterogeneous inference method, apparatus, device, and medium for a hybrid expert large language model. The method is applied to a heterogeneous computing system including a first processor and a second processor. The first processor performs attention calculation and hybrid expert routing on the hidden state to obtain top-k expert identifiers and gating weights; it statistically analyzes activation amplitude features according to the token dimension; it determines the quantization format and granularity based on the expert identifier, gating weights, expert quantization sensitivity, and activation amplitude features; it quantizes the hidden state into low-bit activation data and generates scaling factors and metadata, then bins it according to experts; and it asynchronously transmits this data to the second processor. The second processor directly uses the low-bit activation data and scaling factors to perform expert matrix multiplication calculations without requantization, then weights and aggregates the data according to the gating weights before sending it back. This invention can simultaneously reduce cross-device transmission bandwidth and the overhead of repeated quantization by the second processor, thereby improving the efficiency of heterogeneous inference.

Description

Technical Field

[0001] This application relates to the field of large language model reasoning optimization technology, specifically to the quantization and heterogeneous reasoning of hybrid expert large language models, and particularly to a method, apparatus, device and computer-readable storage medium for route-aware quantization heterogeneous reasoning of hybrid expert large language models. Background Technology

[0002] As model size continues to increase, the number of parameters, activation data, and cached data during inference in large language models also increases rapidly. This leads to problems such as high GPU memory usage, high computational overhead, high cross-device communication volume, and high deployment costs. Especially in autoregressive generation scenarios, the model needs to generate output token by token. The inference process typically includes a pre-filling stage and a decoding stage. The pre-filling stage usually involves a large matrix computation, while the decoding stage often has a smaller batch or number of tokens, but requires frequent access to model weights, activation data, and key-value caches. Therefore, it is easily limited by memory bandwidth and data transfer overhead.

[0003] To reduce the storage and computational overhead of large language models, existing technologies have proposed various quantization methods. These methods mainly focus on the low-bit representation of model weights or matrix multiplication calculations themselves, which can reduce memory usage and computational overhead to a certain extent. On the other hand, as the context length increases, key-value caching has also become an important storage and bandwidth bottleneck in the inference of large language models. Existing technologies have proposed corresponding low-bit quantization methods for the numerical distribution characteristics of key-value caching.

[0004] This type of technique illustrates that, in addition to model weights, intermediate states and cached data during the inference process also need to be represented using low-bit values ​​to reduce storage and memory access pressure. Besides dense large language models, hybrid expert models are also widely used to expand model capacity and reduce the computational cost of single-token activation.

[0005] Hybrid expert models select a small number of experts for computation for each token at each layer through routing, thereby increasing model capacity while maintaining a low number of activation parameters. Although hybrid expert models reduce the activation computation for each token, they typically contain a large number of expert parameters, which can lead to problems such as high memory consumption, complex expert scheduling, and large overhead in data rearrangement from token to expert during deployment.

[0006] To address the issue of insufficient GPU memory for large models, existing systems have proposed model offloading or heterogeneous inference schemes. These schemes involve placing some expert weights in the CPU's memory, with the GPU performing some of the computation or scheduling logic, and the CPU performing the computations of the activated experts, thereby reducing the GPU memory pressure.

[0007] However, existing technologies still have the following problems: First, existing quantization methods typically aim to reduce model storage or matrix multiplication computation overhead, and rarely design the quantization process in a unified manner with the cross-device transmission process. In heterogeneous inference, if the hidden state calculated by the first processor is still transmitted to the second processor in floating-point format, it will generate a large cross-device bandwidth overhead. Especially in hybrid expert models, hidden state transmission between the first and second processors may occur at each layer, and this transmission overhead may offset the benefits of placing the expert computation on the second processor.

[0008] Second, a common practice is for the first processor to transfer the floating-point hidden state to the second processor, and the second processor to re-execute activation quantization according to the local low-bit matrix multiplication requirements. This results in repeated cross-device transmission and quantization at the second processor end, increasing inference latency and system overhead.

[0009] Third, in hybrid expert models, routing has already generated top-k expert identifiers and gating weights for each token. This information reflects which experts will process the token and the importance of different expert branches. However, existing quantization methods usually do not use routing results to determine cross-device transmission formats, nor do they select different quantization precisions for different tokens or expert branches based on the experts' quantization sensitivity, call frequency, or weight layout. Therefore, it is difficult to achieve a finer-grained balance between transmission bandwidth, computational efficiency, and model accuracy.

[0010] Fourth, in scenarios with multiple non-uniform memory access nodes, different experts may be distributed across different nodes. Existing technologies typically perform data rearrangement on the second processor based on expert identifiers, without performing quantization, bucketing, and transmission layout optimization in advance on the first processor based on routing results and expert node affiliation. This leads to additional data migration, cross-node access, or synchronization overhead. Therefore, there is an urgent need for a heterogeneous inference method that can integrate quantization, transmission, and expert computation to solve the above-mentioned technical problems. Summary of the Invention

[0011] To address the shortcomings of the prior art, this application provides a method, apparatus, device, and computer-readable storage medium for route-aware quantization heterogeneous reasoning using a hybrid expert large language model.

[0012] The technical effect to be achieved in this application is accomplished through the following solution: The first aspect of this application: A route-aware quantization heterogeneous reasoning method based on a hybrid expert large language model is provided, applicable to heterogeneous computing systems including a first processor and a second processor, comprising: The first processor performs attention calculation on the input hidden state of the current hybrid expert layer to obtain attention output, and performs hybrid expert routing on the attention output to obtain the hidden state of the expert module to be input, the top-k expert identifiers and gating weights corresponding to each token; The first processor statistically analyzes the activation amplitude characteristics of the hidden state according to the token dimension; Based on the top-k expert identifiers, the gating weights, the activation amplitude features, and the quantization sensitivity of each expert obtained from offline calibration, determine the quantization format and quantization granularity for each token or each token-expert branch; The first processor quantizes the hidden state into low-bit activation data according to the quantization format and quantization granularity, generates a scale factor and metadata corresponding to the low-bit activation data, and buckets the low-bit activation data according to the top-k expert identifier. The bucketed low-bit activation data, the scaling factor, and the metadata are asynchronously transmitted to the second processor; The second processor, based on the metadata, directly uses the low-bit activation data and the scaling factor to perform low-bit expert matrix multiplication calculations to obtain the expert outputs of each expert; The second processor performs weighted aggregation of the expert output according to the gating weights to obtain a hybrid expert output, and sends the hybrid expert output back to the first processor as the input of the next hybrid expert layer.

[0013] Preferably, the attention calculation includes: performing normalization, QKV projection, position encoding, KV cache read / write, attention score calculation, normalization exponent operation, and output projection sequentially on the input hidden state to obtain the attention output; the hybrid expert routing includes: performing routing calculation on the attention output, selecting the top-k active experts for each token, and outputting the gating weights corresponding to each of the top-k experts.

[0014] Preferably, the activation amplitude feature includes at least one of: maximum absolute value, root mean square value, outlier score, and outlier proportion; wherein, the maximum absolute value is the maximum absolute value of each element of the corresponding token in the hidden dimension, the root mean square value is the arithmetic square root of the mean of the squares of each element of the corresponding token in the hidden dimension, the outlier score is the ratio of the maximum absolute value to the root mean square value, and the outlier proportion is the proportion of the number of elements in the corresponding token whose absolute value exceeds a preset outlier threshold to the size of the hidden dimension; the statistical process of the activation amplitude feature is integrated with the calculation process of the quantization scaling factor.

[0015] Preferably, a token quantization risk score is calculated based on the activation amplitude characteristics and the quantization sensitivity of the current layer; an expert-weighted sensitivity is calculated based on the expert quantization sensitivity corresponding to the top-k expert identifier and the gating weight, and the overall quantization risk is obtained by combining the token quantization risk score; the overall quantization risk is compared with a preset first threshold and a second threshold, and a higher precision quantization format or finer quantization granularity is selected for tokens or token-expert branches with high quantization risk, while a lower bit count quantization format is selected for tokens or token-expert branches with low quantization risk.

[0016] Preferably, the quantization granularity includes one of token quantization, channel quantization, and group or block quantization; the bucketing of the low-bit activation data according to the top-k expert identifier includes: writing the low-bit activation data of each token into the bucket corresponding to the expert to which it is routed; when the second processor contains multiple non-unified memory access nodes, the low-bit activation data is further bucketed according to the non-unified memory access node where each expert is located; when the same token is routed to multiple experts, the hidden state is quantized once and the same low-bit activation data and scale factor are shared by multiple expert branches, and the metadata records the reference relationship of each expert branch to the low-bit activation data.

[0017] Preferably, the asynchronous transmission is implemented through page-locked memory, zero-copy memory, unified virtual address, or asynchronous direct memory access, and the low-bit activation data belonging to different non-unified memory access nodes are transmitted to the memory areas bound to the corresponding nodes respectively; the second processor pre-stores the weights of each expert and the weight scaling factors corresponding to the weights in low-bit format, and the second processor directly uses the low-bit activation data and the scaling factors, combined with the pre-stored weights and the weight scaling factors, to sequentially perform gate projection, up projection, activation function operation and down projection calculation to obtain the expert output.

[0018] Preferably, the backhaul includes: converting the hybrid expert output into a floating-point format and then backhauling it to the first processor; or, quantizing the hybrid expert output into a low-bit format and generating an output scaling factor and then backhauling it to the first processor, whereby the first processor performs inverse quantization in the next layer operator.

[0019] The second aspect of this application: A route-aware quantization heterogeneous inference device based on a hybrid expert large language model is provided, applicable to a heterogeneous computing system including a first processor and a second processor, comprising: The first processor computing module is configured to control the first processor to perform attention calculation on the input hidden state of the current hybrid expert layer to obtain attention output, and to perform hybrid expert routing on the attention output to obtain the hidden state of the expert module to be input, the top-k expert identifiers corresponding to each token, and the gating weights. The route-aware quantization decision module is configured to control the first processor to statistically analyze the activation amplitude features of the hidden state according to the token dimension; and is configured to determine the quantization format and quantization granularity for each token or each token-expert branch based on the top-k expert identifier, the gating weight, the activation amplitude features, and the quantization sensitivity of each expert obtained through offline calibration. The quantization activation generation and expert bucketing module is configured to control the first processor to quantize the hidden state into low-bit activation data according to the quantization format and quantization granularity, generate the scale factor and metadata corresponding to the low-bit activation data, and bucket the low-bit activation data according to the top-k expert identifier. An asynchronous transmission module is configured to asynchronously transmit the low-bit activation data, the scaling factor, and the metadata to the second processor; The second processor expert calculation module is configured to control the second processor to directly use the low-bit activation data and the scaling factor to perform low-bit expert matrix multiplication calculations based on the metadata, so as to obtain the expert outputs of each expert. The output aggregation and feedback module is configured to control the second processor to perform weighted aggregation of the expert output according to the gating weights to obtain a hybrid expert output, and to feed the hybrid expert output back to the first processor as the input of the next hybrid expert layer.

[0020] The third aspect of this application: An electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the route-aware quantization heterogeneous reasoning method of the hybrid expert large language model described in the first aspect of this application.

[0021] The fourth aspect of this application: A computer-readable storage medium is provided, on which a computer program is stored, characterized in that, when the program is executed by a processor, it implements the route-aware quantization heterogeneous reasoning method of the hybrid expert large language model described in the first aspect of this application.

[0022] Compared with existing technologies, the beneficial effects of adopting the above solution are as follows: Firstly, after completing attention calculation and routing calculation, the first processor quantizes the hidden state to be transmitted into a low-bit format before transmitting it. Compared with directly transmitting the half-precision floating-point hidden state, the amount of data transmitted by the main hidden state can be reduced by about half when using the 8-bit integer format, and can be further reduced when using the 4-bit integer format, thereby alleviating the bandwidth pressure of the cross-device interconnection link and reducing the transmission delay. Secondly, the first processor generates low-bit activation data and scale factors that are compatible with the expert computation format of the second processor, so that the second processor can directly perform low-bit expert matrix multiplication without requantization, reducing the overhead of repeated quantization on the second processor. Third, by utilizing the expert identifier, gating weight, and expert quantization sensitivity of the routing output, different quantization formats and granularities are selected for different tokens or branches, thereby further reducing transmission volume and computational overhead while ensuring accuracy. Fourth, the ratio of high-precision format to low-bit format is adaptively adjusted based on runtime bandwidth status and quantization risk; Fifth, by pre-bucketing based on the non-uniform memory access nodes where experts reside, the overhead of cross-node access and data rearrangement is reduced, thereby improving the end-to-end efficiency of heterogeneous hybrid expert large language model inference; Sixth, this invention does not limit specific low-bit formats, processor types, interconnection methods or expert structures, and has good scalability and compatibility. Attached Figure Description

[0023] To more clearly illustrate the embodiments of this application or the existing technical solutions, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 The flowchart shows the route-aware quantization heterogeneous inference method of the hybrid expert big language model in this application. Figure 2 A schematic diagram of the heterogeneous inference device provided in this application; Figure 3 The internal structure diagram of the electronic device provided in this application; Figure 4 A schematic diagram of a computer-readable storage medium provided in this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] This invention provides a route-aware quantization heterogeneous inference method based on a hybrid expert large language model, applicable to heterogeneous computing systems including a first processor and a second processor.

[0027] The first processor can be a graphics processing unit (GPU), a neural network processor, a tensor processor, or other accelerated processor suitable for performing attention calculations, routing calculations, and parallel quantization calculations; the second processor can be a central processing unit (CPU), another GPU, a data processor, or other processor suitable for performing expert calculations. In a preferred embodiment, the first processor is a GPU, the second processor is a CPU, and the CPU can include one or more non-uniform memory access nodes, storing the weights of different experts on different non-uniform memory access nodes.

[0028] The core of this invention is that after the first processor completes attention calculation and hybrid expert routing, the floating-point format hidden state is no longer directly transmitted to the second processor. Instead, based on the top-k expert identifier, gating weight, expert quantization sensitivity, the processor or node where the expert is located, the activation amplitude characteristics of the current token, and the runtime bandwidth status, the hidden state is adaptively quantized with low bit and expert bucketing is performed. The quantized low-bit activation data, scaling factor, and metadata are then asynchronously transmitted to the second processor. The second processor directly uses the low-bit activation data and scaling factor to perform low-bit expert matrix multiplication calculations without re-activating and quantizing the hidden state.

[0029] The specific embodiments of the present invention will be described below in conjunction with each step.

[0030] Step S1: The first processor performs attention calculation on the input hidden state of the current hybrid expert layer to obtain attention output, and performs hybrid expert routing on the attention output to obtain the hidden state of the expert module to be input, the top-k expert identifiers and gating weights corresponding to each token.

[0031] The first processor receives the input hidden state of the current hybrid expert layer, and sequentially performs normalization, QKV projection, position encoding, KVcache read / write, attention score calculation, normalization exponent operation, and output projection to obtain the attention output, which is then used as the hidden state of the expert module to be input.

[0032] The hidden state is denoted as: in, Indicates the layer number, Indicates the number of tokens currently being processed. This represents the hidden layer dimension, meaning the hidden state is a real matrix with rows equal to the number of tokens and columns equal to the hidden layer dimension.

[0033] Subsequently, the first processor performs hybrid expert routing on the hidden state, selecting an active expert for each token and outputting the corresponding gating weight, resulting in: in, This represents the number of experts activated by each token, i.e., the top-K, for example, 2 or 8; Represents the expert identifier matrix, its first... Line 1 Column elements represent the first The first token The activation expert's number; This represents the gating weight matrix, where each element represents the gating weight or routing weight of the corresponding expert branch.

[0034] The expert identifier is used for bucketing in step S4 and expert selection in step S6, and the gating weight is used for quantitative decision-making in step S3 and weighted aggregation in step S7, thereby ensuring that the routing information generated in the previous step is applied in subsequent steps.

[0035] Step S2: The first processor performs statistical activation amplitude features on the hidden state according to the token dimension.

[0036] The first processor calculates activation amplitude features for the hidden state obtained in step S1 according to the token dimension or token-block dimension. These activation amplitude features include, but are not limited to, maximum absolute value, root mean square value, outlier score, and outlier proportion. For the t-th token, its maximum absolute value is defined as the maximum absolute value of all elements of that token in the hidden dimension, calculated as follows: The root mean square value of this token is defined as the arithmetic square root of the mean of the squares of all elements in the hidden dimension of this token, and is calculated as follows: The outlier score of a token is defined as the ratio of its maximum absolute value to its root mean square value. It is used to characterize the relative strength of outliers in the token's activation distribution. The calculation method is as follows: in, To prevent small positive numbers with a denominator of zero, the outlier ratio of this token is defined as the proportion of the number of elements in the token whose absolute value exceeds a preset outlier threshold to the size of the hidden dimension. The calculation method is as follows: in, The preset outlier threshold is determined by the following criteria: during the offline calibration phase, the distribution of the absolute values ​​of the hidden state elements of each token on the calibration set can be statistically analyzed, and the high quantile can be taken as the outlier threshold, or a multiple of the root mean square value can be taken as the outlier threshold, so that the proportion of elements judged as outliers is maintained at the expected level. This invention does not limit the threshold to a fixed value; its specific value is determined by calibration.

[0037] The statistical process for activating the amplitude feature described above can be integrated with the calculation process of the scale factor in the quantization kernel of the first processor, thus eliminating the need to start the statistical kernel separately.

[0038] The activation amplitude feature will be used as one of the inputs for the quantization decision in step S3.

[0039] Step S3: Based on the top-k expert identifier, the gating weight, the activation amplitude feature, and the quantization sensitivity of each expert obtained from offline calibration, determine the quantization format and quantization granularity for each token or each token-expert branch.

[0040] The routing-aware quantization decision module determines the quantization format and granularity for each token or each token-expert branch based on the top-k expert identifiers and gating weights obtained in step S1, the quantization sensitivity of each expert, the quantization sensitivity of the current layer, the activation amplitude characteristics obtained in step S2, the location of the expert's processor or non-uniform memory access node, and the cross-device transmission bandwidth pressure during runtime.

[0041] Among them, the expert's quantization sensitivity is used to measure the expert's sensitivity to input activation quantization error, which can be obtained through offline calibration. The present invention provides the following three calculation methods.

[0042] The first type is local output error sensitivity.

[0043] For a given expert, we collect the set of input activations assigned to that expert from a batch of calibration samples. For each input activation, we calculate the reference output using half-precision floating-point and the low-bit output using low-bit format. Taking 8-bit integer as an example, the local output error sensitivity of this expert is defined as follows: in, For the first The reference output is activated by each input. This corresponds to an 8-bit integer output. This represents the number of samples in the input activation set corresponding to that expert. To prevent small positive numbers with a denominator of zero; Let be the hidden state of the i-th token assigned to expert e. The numerator is the square of the L2 norm of the difference between the low-bit output and the reference output, and the denominator is the square of the L2 norm of the reference output. That is, the sensitivity of the expert to quantization error is characterized by the mean of the relative error. The sensitivity of 4-bit integers is defined in the same way.

[0044] The second type is weighted local output error sensitivity, which further considers the gating weights corresponding to each input activation as weights based on the local output error. Its definition is: in, The gating weights assigned to the input activation for routing. The normalized gating weights ensure that samples with higher gating weights contribute more to sensitivity, thus more closely reflecting the importance of the expert during actual reasoning.

[0045] The third type is end-to-end loss sensitivity.

[0046] Local output error only considers the differences in expert outputs and is not necessarily equivalent to the final language model quality. Therefore, end-to-end loss sensitivity can also be defined: Given a calibration dataset, first calculate the baseline loss of the half-precision floating-point model. Then, quantize only the e-th expert to an 8-bit or 4-bit integer, while keeping the other experts at their default precision. Calculate the loss of the quantized model. The end-to-end loss sensitivity of that expert is then defined as: in, For half-precision floating-point baseline loss, The difference between the model loss after quantifying only the expert and the loss after quantification reflects the impact of quantifying the expert on the final model quality. All three sensitivities are calculated and stored in the expert profile during the offline calibration phase for querying during online inference.

[0047] In addition to quantization sensitivity, this invention maintains an expert profile for each expert. This profile includes fields such as expert identifier, expert quantization sensitivity, call frequency, average gating weight, recommended quantization format, weighted quantization format, scale granularity, non-uniform memory access node identifier, device identifier, and fallback format. The expert call frequency represents the frequency with which the expert is selected in the calibration set or online request routing. Its value is determined as follows: Assuming there are N experts in layer l, and each token activates K experts, for all tokens passing through layer l, the total number of tokens is denoted as... Then the total number of expert selections at this level is The product with K, for the th The number of times a certain expert in a layer is selected by the route is recorded as follows. Its calling frequency is defined as: The average gating weight of an expert represents the average gating weight assigned to that expert by the route when the expert is selected. Its value is determined by summing the gating weights corresponding to the expert's selected and activated status. Divide by the number of times it has been selected, i.e.: The aforementioned call frequency and average gating weight were statistically obtained during the offline calibration phase and written into the expert profile; the recommended quantization format indicates the recommended transmission quantization format for input activation routed to the expert; the weight quantization format indicates the storage and calculation format of the expert weight on the second processor; the scale granularity indicates the number of elements covered by the quantization scale, including by token, by channel, by group, or by block; the non-uniform memory access node identifier indicates the node where the expert weight is mainly stored, used for pre-bucketing on the first processor to reduce cross-node access; the fallback format indicates the format to which to fall back when the current quantization format is high-risk or an anomaly is detected, to ensure that low-bit transmission does not significantly impair accuracy on difficult samples.

[0048] After obtaining the above statistics, the quantitative decision-making of the present invention can be carried out at the token level, token-expert branch level, expert level, or layer level.

[0049] Regarding the token level, for the t-th token, calculate its quantitative risk score: in, , , For configurable weights, This is the root mean square value of the token. The outlier score for this token. The quantization sensitivity of the current layer can be obtained through the same offline calibration method as the expert quantization sensitivity. The tuning method of the above weights is as follows: during the offline calibration stage, with the goal of minimizing the loss of the model after quantization, the above weights are fine-tuned by grid search or coordinate descent on the validation set to maximize the correlation between the quantization risk score and the actual quantization error. This invention does not limit the fixed value of these weights.

[0050] Furthermore, regarding the expert level, for the top-k experts to whom the t-th token is routed, the expert-weighted risk is calculated by combining their gating weights: in, This represents the quantitative sensitivity of the corresponding experts. The overall quantitative risk is then obtained by combining the expert-weighted risk with the token quantitative risk score. in, , The weights are configurable and are tuned in the same way as the token-level weights mentioned above. The overall quantification risk is then compared with preset first and second thresholds to determine the quantification format. When the overall quantization risk is not less than the first threshold, it is judged as high risk, and a high-precision format is maintained or 8-bit integer quantization is adopted by group or block. When the overall quantification risk is between the second threshold and the first threshold, it is judged as medium risk, and 8-bit integer quantization is used by channel or by token; When the overall quantization risk is less than the second threshold, it is judged as low risk and 4-bit integer quantization is used.

[0051] The values ​​of the first threshold and the second threshold are determined based on the following: During the offline calibration phase, with the trade-off between transmission volume and model accuracy as a constraint, the distribution of quantified risk scores on the calibration set is statistically analyzed. Two quantiles that enable the proportions of high-risk, medium-risk, and low-risk tokens to reach the expected transmission compression ratio are selected as the first threshold and the second threshold. This invention does not limit these values ​​to fixed values.

[0052] In addition, the system can maintain runtime metrics such as cross-device transmission queue length, average transmission time, interconnect bandwidth utilization, or second processor expert wait time as bandwidth pressure. When bandwidth pressure is high, the usage ratio of low-bit formats is increased; when bandwidth pressure is low or accuracy risk is high, the usage ratio of high-precision formats is increased.

[0053] The quantization format and quantization granularity will serve as the basis for quantization encoding in step S4.

[0054] Step S4: The first processor quantizes the hidden state into low-bit activation data according to the quantization format and quantization granularity, generates a scale factor and metadata corresponding to the low-bit activation data, and buckets the low-bit activation data according to the top-k expert identifier.

[0055] The first processor, based on the quantization format and quantization granularity determined in step S3, quantizes the hidden state obtained in step S1 into low-bit active data, generates a scaling factor and metadata, and buckets the data according to the expert identifier obtained in step S1. The quantization method can be obtained by combining the data format (8-bit integer, 4-bit integer, 8-bit floating-point, 4-bit floating-point, or other low-bit formats) and the quantization granularity (by token, by channel, by group, or by block, etc.).

[0056] Taking an 8-bit integer as an example, when quantizing by token, a scaling factor is calculated for each token: in, As the upper bound for quantization, 8-bit integer symmetric quantization Let's take 127. The corresponding quantization formula is: in, This indicates rounding to the nearest integer. This means limiting the result to a range, thus obtaining the quantized low-bit activation data. During channel quantization, a scaling factor is calculated for each dimension of the hidden state: When quantizing by group or by block, the hidden dimension of each token is divided according to the group or block size (usually 64, 128, or 256), and a scaling factor is calculated for each group or block. in, This indicates the groups or blocks obtained from the segmentation; the remaining quantization process is the same as that for token-based quantization. The low-bit activation data, scaling factor, and metadata generated by the above quantization constitute the data to be transmitted, with the metadata used by the second processor for parsing and expert calculations.

[0057] The value of the group or block size is determined based on the following: the smaller the block, the lower the quantization error, but the greater the storage and computational overhead of the scale factor. A compromise between accuracy and overhead can be made during the offline calibration stage. This invention does not limit its value to a fixed value.

[0058] In terms of bucketing, the first processor writes the low-bit activation data of each token to the bucket corresponding to the expert it is routed to, based on the expert identifier. For top-k hybrid experts, each token can correspond to multiple token-expert branches, and the first processor can generate a branch record for each branch, including the token identifier, expert identifier, gating weight, data offset, and scale offset. In another implementation, the hidden state of the same token is quantized only once, and multiple expert branches share the same low-bit activation data and scaling factor. The metadata records the reference relationships of multiple expert branches to this data, thereby reducing redundant transmissions. In scenarios with multiple non-uniform memory access nodes, expert bucketing can be further bucketed according to the node where the expert is located. The first processor writes the low-bit activation data belonging to different nodes into different output buffers, thereby reducing the cross-node access and reordering overhead on the second processor.

[0059] Step S5: Asynchronously transmit the bucketed low-bit activation data, the scaling factor, and the metadata to the second processor.

[0060] The first processor asynchronously transmits the low-bit activation data, scaling factor, and metadata generated in step S4 to the second processor. The transmitted content includes, but is not limited to, the low-bit activation data buffer, scaling factor buffer, metadata buffer, expert offset, token identifier, expert identifier, and gating weight.

[0061] In the context of graphics processing units (GPUs) and central processing units (CPUs), data transfer can be achieved through paged memory, zero-copy memory, unified virtual addresses, or other asynchronous direct memory access mechanisms.

[0062] The first processor can perform quantization and transmission in a dedicated stream and synchronize with attention and routing computations through event dependencies. For example, the first stream performs attention and routing, the second stream performs quantization and expert bucketing after routing is completed, the third stream asynchronously transmits the quantization results to the second processor's page buffer, the second processor's thread pool performs expert computations after the corresponding buffers are ready, and the fourth stream asynchronously transmits the computation results back.

[0063] Therefore, computation of subsequent layers or subsequent tokens, cross-device transmission, and expert computation can be performed concurrently, thereby reducing end-to-end inference latency. If the second processor contains multiple non-uniform memory access nodes, data from different buckets is transmitted to the memory regions bound to the corresponding nodes.

[0064] Step S6: The second processor directly uses the low-bit activation data and the scaling factor to perform low-bit expert matrix multiplication calculation based on the metadata, and obtains the expert output of each expert.

[0065] The second processor parses the token list, quantization data position, scaling factor position, and gating weight of each expert bucket based on the metadata transmitted in step S5, and directly uses the low-bit activation data and scaling factor as the input activation and scaling factor of the expert matrix multiplication, without having to re-quantize the hidden state from floating point to low bit.

[0066] The second processor pre-stores the gate projection weights, up projection weights, and down projection weights of each expert, along with their corresponding weight scaling factors, in low-bit format. For a hybrid expert feedforward network, the expert computation includes, in sequence: in, This represents a low-bit quantized matrix multiplication, whose inputs are low-bit activation data, low-bit weights, activation scaling factor, and weight scaling factor. The low-bit activation data generated in step S4, For the corresponding activation scaling factor, , , These are the scale factors for the gate projection, top projection, and bottom projection weights, respectively. First, calculate the gate projection and top projection separately. Then, apply an activation function to the gate projection result and multiply it element-wise with the top projection result to obtain the intermediate activation. Finally, calculate the bottom projection to obtain the expert's output.

[0067] Since the low-bit activation data and activation scale factor received by the second processor are already compatible with its low-bit expert weight format, the above calculations can be performed directly. This is an important feature that distinguishes this invention from existing heterogeneous inference methods.

[0068] Step S7: The second processor performs weighted aggregation on the expert output according to the gating weight to obtain a hybrid expert output, and sends the hybrid expert output back to the first processor as the input of the next hybrid expert layer.

[0069] When each token is routed to multiple experts, the second processor performs a weighted aggregation of the multiple expert outputs corresponding to that token based on the gating weights obtained in step S1, to obtain a hybrid expert output: in, For the first The first token Gating weights for each expert branch, The corresponding expert output obtained in step S6 is the weighted sum of each expert output with the gating weight as the coefficient.

[0070] The aggregated hybrid expert output can be returned to the first processor in two ways: The first is floating-point format return, where the aggregated output is converted to half-precision floating-point, brain-level floating-point, or single-precision floating-point format before return, offering good compatibility and facilitating connection with subsequent layers; the second is quantized format return, where the aggregated output is quantized again to a low-bit format and an output scaling factor is generated before return. The first processor then performs inverse quantization in the normalization, input projection, or other operators of the next layer, further reducing the return bandwidth and the overhead of independent inverse quantization operations. After receiving the hybrid expert output, the first processor uses it as input to the next hybrid expert layer to continue inference.

[0071] Example 1: 8-bit integer cross-device quantization transmission and 8-bit integer expert calculation by a second processor In this embodiment, the first processor is a graphics processor and the second processor is a central processing unit.

[0072] After performing attention and routing calculations, the graphics processor performs 8-bit integer quantization on the hidden state, either by group or by block, with a block size of 64, 128, or 256. Based on this, the graphics processor generates 8-bit integer low-bit activation data, a scaling factor in half-precision floating-point representation with the shape of the number of tokens multiplied by the number of groups, and metadata.

[0073] The central processing unit (CPU) pre-stores the 8-bit integer gate projection weights, up projection weights, down projection weights, and their weight scaling factors for each expert. Upon receiving the low-bit activation data and scaling factors, the CPU directly performs gate projection, up projection, activation function operations, and down projection without needing to re-quantize the hidden state from floating-point to 8-bit integers at the CPU level.

[0074] Therefore, this embodiment can reduce the amount of hidden state transmission from the graphics processor to the central processing unit from two bytes per dimension per token in half-precision floating point to about one byte per dimension per token, and avoid redundant quantization at the central processing unit.

[0075] Example 2: Expert-perceived mixed quantization transmission of 8-bit and 4-bit integers In this embodiment, the system maintains a quantitative sensitivity for each expert and stores an expert profile.

[0076] For tokens or branches routed to highly sensitive experts, use 8-bit integer or half-precision floating-point transmission; for tokens or branches routed to low-sensitive experts with low gating weights, use 4-bit integer transmission.

[0077] For example, when an expert's quantization sensitivity is greater than a preset high sensitivity threshold, the corresponding branch is treated as an 8-bit block integer; when an expert's quantization sensitivity is not greater than a preset low sensitivity threshold and the gating weight is less than a preset gating threshold, the corresponding branch is treated as a 4-bit block integer.

[0078] The values ​​of the high sensitivity threshold, low sensitivity threshold, and gate threshold are determined by: statistically analyzing the distribution of expert quantification sensitivity and gate weight on the calibration set during the offline calibration phase, and selecting corresponding quantiles according to the target transmission compression ratio. This invention does not limit them to fixed values.

[0079] This embodiment can further reduce cross-device transmission volume while maintaining inference accuracy based on the sensitivity of different experts to quantization errors.

[0080] Example 3: Expert Bucketing Transmission Aware of Non-Unified Memory Access Nodes In this embodiment, the second processor is a central processing unit containing multiple non-uniform memory access nodes. The expert weights are distributed on different nodes according to the expert identifiers. For example, some consecutively numbered experts are stored on the first node, and another part is stored on the second node.

[0081] After obtaining the routing results, the graphics processor queries the node identifier in the expert profile based on the expert identifier, and activates the quantized token on the graphics processor side and distributes it into buckets of different nodes. Then, the different buckets are transmitted to the page buffer bound to the corresponding node, and the thread pool on each node processes the local expert.

[0082] This reduces data dispatching at the central processing unit, cross-node data access, and additional memory copying.

[0083] Example 4: Quantization feedback from the second processor is fused with the first processor for inverse quantization. In this embodiment, after the central processing unit completes the aggregation of expert outputs, it does not directly send the output back to the graphics processor as half-precision floating point or single-precision floating point. Instead, it performs 8-bit integer or 4-bit integer quantization on the aggregated output and generates an output scaling factor. Then, it sends the quantized output and the output scaling factor back to the graphics processor.

[0084] The graphics processor integrates the quantized output and output scale factor in the next layer's normalization, QKV projection, KV cache read / write, or other operators to complete dequantization and subsequent calculations.

[0085] This embodiment can further reduce the backhaul bandwidth from the central processing unit to the graphics processing unit and reduce the overhead of independent inverse quantization operations.

[0086] Hardware Feasibility Specification The method described in this invention can be implemented by a heterogeneous computing system including a first processor and a second processor.

[0087] The system may further include a memory, a communication interface, and a bus. The first processor, the second processor, the memory, and the communication interface communicate with each other through the bus. The memory is used to store computer programs that can run on the processor. When the first processor executes the program, it performs attention calculation, routing calculation, activation amplitude feature statistics, quantization decision, quantization encoding and binning, and asynchronous transmission control. When the second processor executes the program, it performs low-bit expert matrix multiplication calculation and weighted aggregation.

[0088] The first processor and the second processor may be a graphics processor, a central processing unit, a neural network processor, a tensor processor, a field-programmable gate array, an application-specific integrated circuit, or a combination thereof; the communication interface and bus may be a high-speed peripheral component interconnect, a fast computing link, a high-speed interconnect bus, or other inter-chip interconnect.

[0089] Therefore, this invention is applicable not only to current heterogeneous hybrid expert reasoning systems using central processing units and graphics processors, but also to multi-graphics processor, multi-central processing unit, multi-accelerator and other future heterogeneous large language model reasoning systems.

[0090] Each module in the device of the present invention can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in the processor in hardware form or independent of the processor, or stored in the memory in software form, so that the processor can call and execute the operation corresponding to each module.

[0091] All or part of the process of the method described in this invention can be implemented by hardware related to computer program instructions. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, the program can implement the processes of the above-described method embodiments. The computer-readable storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, or other non-volatile media.

[0092] It should be noted that the above detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0093] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0094] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0095] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0096] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A route-aware quantization heterogeneous reasoning method based on a hybrid expert large language model, applied to a heterogeneous computing system including a first processor and a second processor, characterized in that, include: The first processor performs attention calculation on the input hidden state of the current hybrid expert layer to obtain attention output, and performs hybrid expert routing on the attention output to obtain the hidden state of the expert module to be input, the top-k expert identifiers and gating weights corresponding to each token; The first processor statistically analyzes the activation amplitude characteristics of the hidden state according to the token dimension; Based on the top-k expert identifiers, the gating weights, the activation amplitude features, and the quantization sensitivity of each expert obtained from offline calibration, determine the quantization format and quantization granularity for each token or each token-expert branch; The first processor quantizes the hidden state into low-bit activation data according to the quantization format and quantization granularity, generates a scale factor and metadata corresponding to the low-bit activation data, and buckets the low-bit activation data according to the top-k expert identifier. The bucketed low-bit activation data, the scaling factor, and the metadata are asynchronously transmitted to the second processor; The second processor, based on the metadata, directly uses the low-bit activation data and the scaling factor to perform low-bit expert matrix multiplication calculations to obtain the expert outputs of each expert; The second processor performs weighted aggregation of the expert output according to the gating weights to obtain a hybrid expert output, and sends the hybrid expert output back to the first processor as the input of the next hybrid expert layer.

2. The route-aware quantization heterogeneous reasoning method based on a hybrid expert large language model according to claim 1, characterized in that, The attention calculation includes: performing normalization, QKV projection, position encoding, KVcache read / write, attention score calculation, normalization exponent operation, and output projection sequentially on the input hidden state to obtain the attention output; the hybrid expert routing includes: performing routing calculation on the attention output, selecting the top-k active experts for each token, and outputting the gating weights corresponding to the top-k experts.

3. The route-aware quantization heterogeneous reasoning method based on a hybrid expert large language model according to claim 1, characterized in that, The activation amplitude feature includes at least one of the following: maximum absolute value, root mean square value, outlier score, and outlier proportion; wherein, the maximum absolute value is the maximum absolute value of each element in the hidden dimension of the corresponding token, the root mean square value is the arithmetic square root of the mean of the squares of each element in the hidden dimension of the corresponding token, the outlier score is the ratio of the maximum absolute value to the root mean square value, and the outlier proportion is the proportion of the number of elements in the corresponding token whose absolute value exceeds a preset outlier threshold to the size of the hidden dimension; the statistical process of the activation amplitude feature is integrated with the calculation process of the quantization scaling factor.

4. The route-aware quantization heterogeneous reasoning method based on a hybrid expert large language model according to claim 1, characterized in that, Calculate the token quantization risk score based on the activation amplitude characteristics and the quantization sensitivity of the current layer; calculate the expert weighted sensitivity based on the expert quantization sensitivity corresponding to the top-k expert identifier and the gating weight, and combine it with the token quantization risk score to obtain the overall quantization risk; compare the overall quantization risk with a preset first threshold and a second threshold, select a higher precision quantization format or finer quantization granularity for tokens or token-expert branches with high quantization risk, and select a lower bit count quantization format for tokens or token-expert branches with low quantization risk.

5. The route-aware quantization heterogeneous reasoning method based on a hybrid expert large language model according to claim 1, characterized in that, The quantization granularity includes one of token quantization, channel quantization, and group or block quantization; the bucketing of the low-bit activation data according to the top-k expert identifier includes: writing the low-bit activation data of each token into the bucket corresponding to the expert to which it is routed; when the second processor contains multiple non-unified memory access nodes, the low-bit activation data is further bucketed according to the non-unified memory access node where each expert is located; when the same token is routed to multiple experts, the hidden state is quantized once and the same low-bit activation data and scale factor are shared by multiple expert branches, and the metadata records the reference relationship of each expert branch to the low-bit activation data.

6. The route-aware quantization heterogeneous reasoning method based on a hybrid expert large language model according to claim 5, characterized in that, The asynchronous transmission is implemented through page-locked memory, zero-copy memory, unified virtual address, or asynchronous direct memory access, and low-bit activation data belonging to different non-unified memory access nodes are transmitted to the memory regions bound to the corresponding nodes respectively; the second processor pre-stores the weights of each expert and the weight scaling factors corresponding to the weights in low-bit format, and the second processor directly uses the low-bit activation data and the scaling factors, combined with the pre-stored weights and the weight scaling factors, to sequentially perform gate projection, up projection, activation function operation and down projection calculation to obtain the expert output.

7. The route-aware quantization heterogeneous reasoning method based on a hybrid expert large language model according to claim 1, characterized in that, The backhaul includes: converting the hybrid expert output into a floating-point format and then backhauling it to the first processor; or, quantizing the hybrid expert output into a low-bit format and generating an output scaling factor and then backhauling it to the first processor, whereby the first processor performs inverse quantization in the next layer operator.

8. A route-aware quantization heterogeneous inference device based on a hybrid expert large language model, applied to a heterogeneous computing system including a first processor and a second processor, characterized in that, include: The first processor computing module is configured to control the first processor to perform attention calculation on the input hidden state of the current hybrid expert layer to obtain attention output, and to perform hybrid expert routing on the attention output to obtain the hidden state of the expert module to be input, the top-k expert identifiers corresponding to each token, and the gating weights. The route-aware quantization decision module is configured to control the first processor to statistically analyze the activation amplitude features of the hidden state according to the token dimension; and is configured to determine the quantization format and quantization granularity for each token or each token-expert branch based on the top-k expert identifier, the gating weight, the activation amplitude features, and the quantization sensitivity of each expert obtained through offline calibration. The quantization activation generation and expert bucketing module is configured to control the first processor to quantize the hidden state into low-bit activation data according to the quantization format and quantization granularity, generate the scale factor and metadata corresponding to the low-bit activation data, and bucket the low-bit activation data according to the top-k expert identifier. An asynchronous transmission module is configured to asynchronously transmit the low-bit activation data, the scaling factor, and the metadata to the second processor; The second processor expert calculation module is configured to control the second processor to directly use the low-bit activation data and the scaling factor to perform low-bit expert matrix multiplication calculations based on the metadata, so as to obtain the expert outputs of each expert. The output aggregation and feedback module is configured to control the second processor to perform weighted aggregation of the expert output according to the gating weights to obtain a hybrid expert output, and to feed the hybrid expert output back to the first processor as the input of the next hybrid expert layer.

9. An electronic device, characterized in that, The electronic device includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the route-aware quantization heterogeneous reasoning method of the hybrid expert big language model as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the route-aware quantization heterogeneous inference method of the hybrid expert large language model as described in any one of claims 1 to 7.

Images