Model quantification method, model reasoning method and electronic equipment

Abstract

The invention discloses a quantification method of a model, a reasoning method of the model and electronic equipment. In the method, the first model is quantized based on the quantization parameter of the weight, the quantization parameter of the activation value, the first bias vector and the second bias vector, on one hand, the channel-by-channel quantization of the weight and the lexical element-by-lexical quantization of the activation value can be realized, and the quantization precision is relatively high; and on the other hand, the original bias vector is decomposed into two low-rank bias vectors, and the two low-rank bias vectors are spliced with the weight and the activation value respectively during quantization, so that the parameter quantity can be reduced, the memory occupation is reduced, and the acceleration effect of the quantization model is realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method for quantifying a model, a method for reasoning about a model, and an electronic device. Background Technology

[0002] Neural network models have a wide range of applications. By quantizing the operators in a neural network model, that is, quantizing the parameters of the operators and quantizing the input data, floating-point operations can be converted into fixed-point operations, resulting in benefits in terms of model size, inference speed, and power consumption.

[0003] Bidirectional encoder representations from transformers (BERT), a key model in natural language processing, boasts core advantages in its bidirectional context modeling capabilities, powerful transfer learning abilities, and capacity to handle complex linguistic phenomena. It has profoundly changed language processing methods and propelled academic research. Currently, improving the inference efficiency of BERT models on hardware while maintaining accuracy is of great significance, but existing technologies have significant shortcomings. On one hand, accuracy loss in quantization algorithms is a core issue. For example, post-training quantization (PTQ) schemes have limited calibration capabilities and insufficient generalization on models with large activation distribution differences, leading to a severe drop in accuracy. Quantization-aware training (QAT) is needed to recover accuracy. On the other hand, the inference performance of hardware operators has not met expectations. For instance, on the 310P hardware, the computational efficiency of quantization operators offers no speedup benefit compared to floating-point operators, making it difficult for quantization to achieve the goal of improving performance. Summary of the Invention

[0004] This application provides a model quantization method, a model inference method, and an electronic device, which can obtain quantized model parameters with higher inference accuracy during the model training stage, and connect a quantized matrix multiplication operator during the inference stage to perform efficient inference based on the quantized model parameters obtained during the training stage.

[0005] Firstly, a method for quantizing a model is provided. This method can be executed by a first device, or by a processor in the first device, or by a processing unit, chip, or circuit in the processor. For ease of understanding, the first device will be used as an example of the execution subject below.

[0006] The first device includes a first model. For example, when performing at least some steps of the quantization method in the first aspect and any implementation thereof, all or some of the parameters of the first model are loaded into the storage module of the first device, causing the first model to run on the first device.

[0007] The method includes: determining a first parameter set, the first parameter set including a first quantization parameter, a second quantization parameter, a first bias vector, and a second bias vector, wherein the first quantization parameter is used to quantize the activation values ​​input to a first model, the second quantization parameter is used to quantize the weights of the first model, and the first bias vector and the second bias vector are used to perform data fitting processing on the first model; and performing a quantization operation on the first model based on the first parameter set, wherein the activation values ​​and the first bias vector of the first model are quantized using the first quantization parameter, and the weights and the second bias vector of the first model are quantized using the second quantization parameter.

[0008] As an example, the first model is a neural network model, such as the BERT model, but this application does not limit this embodiment.

[0009] As an example, the second quantization parameter can also be called the quantization parameter of the weight. As an example, it can also be replaced with the quantization scale of the weight, the quantization coefficient of the weight, and similar terms. This application does not limit this.

[0010] As an example, the first quantization parameter can also be called the quantization parameter of the activation value. As an example, it can also be replaced with the quantization scale of the activation value, the quantization coefficient of the activation value, or similar terms. This application does not limit this.

[0011] Based on the above method, the first model is quantized based on the quantization parameters of the weights, the quantization parameters of the activation values, the first bias vector, and the second bias vector. By decomposing the original bias vector into two low-rank bias vectors, which are then concatenated with the weights and activation values ​​respectively during quantization, the number of parameters and memory usage can be reduced, thereby achieving an acceleration effect for the quantization model.

[0012] In traditional quantization, taking a floating-point bias vector b_int32 (a 32-bit integer type) as an example, the bias vector b_int32 is a tensor of shape (s×m) (where s is the length of the input sequence and m is the dimension of the output features). The bias tensor itself has a large size (s×m), resulting in a large memory access overhead. In this application, the bias b_int32 is decomposed into two low-rank matrices A (shape (s×k)) and B (shape (k×m)), where s is the length of the input sequence, m is the output feature dimension, and k is the bias compression dimension, where k is a small value (e.g., 32). This reduces the number of parameters: the original bias parameters are s×m, which are decomposed into s×k + k×m. When k When min(s,m), the number of parameters is significantly reduced; memory access optimization: the sizes of A and B are much smaller than the original bias, reducing memory usage. The original scheme required independent access to the bias value for each output element, resulting in s×m memory reads. In this application's scheme, the calculation is completed in one step through matrix multiplication, requiring only one read each of A and B, reducing memory access to s×k + k×m. This significantly reduces memory access and computational redundancy, ultimately achieving an acceleration effect for the quantized model. This design maximizes the utilization of the hardware's parallel computing capabilities while maintaining model accuracy.

[0013] In some implementations of the first aspect, the method further includes: training the quantized first model based on a first set of learning parameters to obtain first parameter information, the first set of learning parameters including the first quantization parameter, the second quantization parameter, the quantized weight, the quantized first bias vector, and the quantized second bias vector; and sending the first parameter information, which is used to determine a second model, and the second model is used for inference.

[0014] In some implementations of the first aspect, the first parameter information includes the training results of the weights of the first model, the training results of the first quantization parameter, the training results of the second quantization parameter, the training results of the first bias vector of the first model, and the training results of the second bias vector of the first model.

[0015] As an example, the first parameter information can be understood as the model parameters trained based on the quantized first model, or in other words, the model parameter information that ensures the model outputs accurate values. The quantized first model is trained using QAT with the original data. The parameters in the first set of learned parameters can be used as learning parameters and updated during training. After training, a set of model parameters that ensures accurate model output is obtained, i.e., the first parameter information. This first parameter information can be used for the inference model.

[0016] In some implementations of the first aspect, the training result of the first bias vector of the first model is quantized based on the training result of the first quantization parameter to obtain a first quantization result; the training result of the weights of the first model is quantized based on the training result of the second quantization parameter to obtain a second quantization result; the training result of the second bias vector of the first model is quantized based on the training result of the second quantization parameter to obtain a third quantization result; and the first quantization result, the second quantization result, and the third quantization result are then transmitted. Alternatively, the training result of the first bias vector of the first model is determined to be quantized based on the training result of the first quantization parameter, the training result of the weights of the first model is quantized based on the training result of the second quantization parameter, and the training result of the second bias vector of the first model is quantized based on the training result of the second quantization parameter; and the training result of the first bias vector is quantized based on the training result of the first quantization parameter, the training result of the weights of the first model is quantized based on the training result of the second quantization parameter, and the training result of the second bias vector of the first model is quantized based on the training result of the second quantization parameter.

[0017] In some implementations of the first aspect, the quantization parameter of the activation value (the first quantization parameter) is a static parameter, which is used to perform word-by-word meta-quantization on the activation value input in the first model.

[0018] In this technical solution, static quantization parameters refer to quantization configurations that do not dynamically change with the input data during neural network inference. Specifically, for activation value quantization, the quantization parameters (such as the scaling factor and zero point) are determined before inference and remain unchanged throughout the inference process. This contrasts with dynamic quantization (where parameters are calculated in real time with the input). It avoids real-time calculations, reduces computational overhead, and improves model inference efficiency. Furthermore, it enables channel-by-channel quantization of weights and word-by-word quantization of activation values ​​with high quantization accuracy.

[0019] In some implementations of the first aspect, the first bias vector and the second bias vector are low-rank vectors.

[0020] In this technical solution, the size of the two bias vectors is much smaller than the original bias, which reduces memory usage and is beneficial for accelerating model quantization.

[0021] In some implementations of the first aspect, the quantized weights are data precision ranges of 8-bit integers, and the quantized activation values ​​are data precision ranges of 8-bit integers.

[0022] In this technical solution, quantization reduces the number of model parameters, further reducing memory usage and computational load. Meanwhile, integer operations can run better on modern hardware, thus improving the computational efficiency of the model.

[0023] In some implementations of the first aspect, the quantized weights, the quantized first bias vector, and the quantized second bias vector are pre-stored in the quantized result.

[0024] In this technical solution, the quantized weights, the quantized first bias vector, and the quantized second bias vector are pre-stored and can be directly loaded and used during the inference stage, avoiding the execution of additional computational tasks and improving the model's computational efficiency.

[0025] In some implementations of the first aspect, the linear operations in the first operator of the quantized first model satisfy the following condition:

[0026] in, Indicates the tensor result obtained from the linear operation. This indicates the concatenation result of the quantized activation value and the quantized first bias vector. The concatenation result of the quantized weights and the quantized second bias vector indicates that T represents... The transpose of matrix multiplication.

[0027] In some implementations of the first aspect, the quantized activation value and the quantized first bias vector respectively satisfy the following conditions:

[0028]

[0029] The quantized weights and the quantized second bias vector satisfy the following conditions respectively:

[0030]

[0031] in, Indicates the quantized activation value, Indicates the quantized first bias vector, Indicates the quantized weights, Indicates the quantized second bias vector. Indicates the activation value, Indicates the first bias vector. Indicates the weight, Indicates the second bias vector, The quantization parameter indicating the activation value The quantization parameters indicating the weights, Indicates rounding. The instruction will truncate the rounded integer. This instruction truncates rounded integers to the range of 8-bit integer precision.

[0032] In this technical solution, the first bias vector A and the activation value X are concatenated along the feature dimension to form a new matrix X' = [X; A] (shape (s×(n+k))); the second bias vector B and the weight W are concatenated along the input dimension to form a new matrix W' = [W; B] (shape ((n+k)×m)), where s is the length of the input sequence, m is the output feature dimension, n is the hidden layer dimension, and k is the bias compression dimension. Matrix multiplication is then performed, meaning that a single matrix multiplication X' × W' directly yields the output (shape (s×m)) containing the bias effect. In this case, bias addition can be implicitly performed; that is, the original bias addition operation is integrated into the matrix multiplication without explicit broadcasting. Furthermore, computational pipeline optimization can be achieved; matrix multiplication is a highly optimized hardware operator, and deep pipeline design can hide memory access latency, further improving throughput.

[0033] Secondly, a reasoning method for a model is provided. This method can be executed by a second device, or by a processor in the second device, or by a processing unit, chip, or circuit in the processor. For ease of understanding, the second device will be used as an example of the execution subject below.

[0034] The second device includes an initial inference model. For example, when executing at least some steps of the inference method in the second aspect and any of its implementations, all or some of the parameters of the initial inference model are loaded into the storage module of the second device, causing the initial inference model to run on the second device.

[0035] The method includes: obtaining the activation values ​​of the second model; quantizing the activation values ​​of the second model based on the training results of the first quantization parameters to obtain a fourth quantization result, wherein the first bias vector of the second model is quantized based on the training results of the first quantization parameters, the weights of the second model and the second bias vector of the second model are quantized based on the training results of the second quantization parameters, and the first bias vector and the second bias vector are used to perform data fitting processing on the second model; controlling the second model to perform inference based on the fourth quantization result and a first operator, wherein the first operator performs linear operations based on the fourth quantization result.

[0036] In this technical solution, during the inference process, the training results of the first quantization parameter are used to quantize the input activation value. The weights, the first bias vector, and the second bias vector can be directly loaded with the quantized results, thereby avoiding repeated calculations and reducing the amount of computation. Furthermore, the original bias vector is decomposed into two low-rank bias vectors, which are concatenated with the weights and activation values ​​respectively during quantization, thereby reducing the number of parameters, reducing memory usage, and improving the inference speed.

[0037] In some implementations of the second aspect, the method further includes: receiving first parameter information, the first parameter information being obtained by training a quantized first model; updating the parameters of an initial inference model based on the first parameter information to obtain a second model, the initial inference model including the first operator.

[0038] In some implementations of the second aspect, the first parameter information includes the training results of the weights of the first model, the training results of the first quantization parameter, the training results of the second quantization parameter, the training results of the first bias vector of the first model, and the training results of the second bias vector of the first model.

[0039] In some implementations of the second aspect, the method further includes: receiving a first quantization result, a second quantization result, and a third quantization result, wherein the first quantization result corresponds to a first bias vector of the second model, the second quantization result corresponds to the weights of the second model, and the third quantization result corresponds to a second bias vector of the second model, wherein the first quantization result is the result of quantizing the training result of the first bias vector of the first model based on the training result of the first quantization parameter, the second quantization result is the result of quantizing the training result of the weights of the first model based on the training result of the second quantization parameter, and the third quantization result is the result of quantizing the training result of the second bias vector of the first model based on the training result of the second quantization parameter.

[0040] In this technical solution, the second device can determine the weights, the first bias vector, and the second bias vector of the second model based on the received first quantization result, second quantization result, and third quantization result. That is, it can directly load the quantized result, avoid repeated calculations, and improve the inference speed.

[0041] In some implementations of the second aspect, the linear operations in the first operator satisfy the following condition:

[0042] in, Indicates the tensor result obtained from the linear operation. The concatenation result of the activation value of the second model after quantization and the first bias vector of the second model. The concatenation result of the weights of the second model and the second bias vector of the second model, where T represents... Transpose of matrix multiplication.

[0043] In some implementations of the second aspect, the quantized activation value of the second model and the first bias vector of the second model respectively satisfy the following conditions:

[0044]

[0045] The weights of the second model and the second bias vector of the second model satisfy the following conditions:

[0046]

[0047] in, The activation value of the second model after quantization is indicated. The first bias vector of the second model after quantization. Indicates the weights of the second model after quantization. The second bias vector of the second model after quantization. Indicates the activation value of the second model. Indicates the first bias vector of the second model. Indicates the weights of the second model, Indicates the second bias vector of the second model. Indicates the training result of the first quantization parameter. The training result indicating the weights of the first model. Indicates rounding. The instruction will truncate the rounded integer. This instruction truncates rounded integers to the range of 8-bit integer precision.

[0048] In this technical solution, during the inference phase, the activation values ​​of the second model and the first bias vector are concatenated along the feature dimension to form a new matrix. The weights of the second model and the second bias vector are concatenated along the input dimension to form a new matrix. Fusion matrix multiplication, that is, through a single matrix multiplication. This directly yields an output that includes the bias effect. In this case, bias addition can be implicitly performed; that is, the original bias addition operation is integrated into matrix multiplication without explicit broadcasting. Furthermore, computational pipeline optimization is possible. Matrix multiplication is a highly optimized hardware operator, and deep pipeline design can hide memory access latency, further improving throughput.

[0049] In some implementations of the first aspect, the method further includes: simultaneously performing a first computation and a second computation based on the first operator, wherein the first computation is a matrix multiplication operation and the second computation is other vector operations.

[0050] As an example, the first computation is a matrix multiplication operation performed by the multidimensional computation unit cube, and the second computation is the remaining vector operations performed by the computation unit vector.

[0051] In this computational scheme, the inputs / outputs of two operations are independent (e.g., the output of operation A is not used as the input of operation B); and the operations use different computational resources or bandwidths (e.g., global memory and shared memory). During parallel execution, the computational latency of one operation can be masked by the memory access latency of another operation, thus reducing the overall latency. For example, quantization operations are mainly performed by vector units (element-by-element operations), while matrix multiplication is performed by cube units; both can be parallelized, thus allowing the memory access latency of quantization operations to be masked by the computational latency of matrix multiplication, reducing the overall latency.

[0052] In some implementations of the first or second aspect, the neural network model is a model that includes linear layers.

[0053] In some implementations of the first or second aspect, the neural network model is a bidirectional encoding representation transformation model.

[0054] Thirdly, an electronic device is provided, comprising: one or more processors; one or more memories; wherein the one or more memories store one or more programs that, when executed by the one or more processors, cause the methods described in the first or second aspect and any possible implementation thereof to be performed.

[0055] Fourthly, a server device is provided, comprising: one or more processors; one or more memories; wherein the one or more memories store one or more programs that, when executed by the one or more processors, cause the method described in the first or second aspect and any possible implementation thereof to be performed.

[0056] Fifthly, a chip is provided, the chip including a processor and a communication interface for receiving a signal and transmitting the signal to the processor, the processor processing the signal such that the method described in the first or second aspect and any possible implementation thereof is performed.

[0057] A sixth aspect provides a readable storage medium storing instructions that, when executed on an electronic device, cause the methods described in the first or second aspect and any possible implementation thereof to be performed.

[0058] In a seventh aspect, a program product is provided, the program product including program code that, when run on an electronic device, causes the methods described in the first or second aspect and any possible implementation thereof to be executed. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the system architecture 100 to which the embodiments of this application apply.

[0060] Figure 2 This is a schematic diagram of the system architecture 200 provided in the embodiments of this application.

[0061] Figure 3 This is a schematic diagram of a model quantization method 300 provided in an embodiment of this application.

[0062] Figure 4 This is a schematic diagram of a model reasoning method 400 provided in an embodiment of this application.

[0063] Figure 5 This is a schematic diagram illustrating an optimized parallel computing process for computer vision (CV) provided in an embodiment of this application.

[0064] Figure 6 This is a flowchart illustrating a model quantization method and inference method 600 provided in an embodiment of this application.

[0065] Figure 7 This is a schematic diagram of an electronic device 700 provided in an embodiment of this application. Detailed Implementation

[0066] To facilitate understanding of the embodiments of this application, the following points will be explained first.

[0067] 1. Unless otherwise stated, "multiple" means two or more. "At least one" means "one or more".

[0068] 2. Unless otherwise specified or in case of logical conflict, the terms and / or descriptions in different embodiments of this application are consistent and can be referenced in each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0069] III. The various numerical designations used in this application are merely for descriptive convenience and are not intended to limit the scope of protection of this application. The magnitude of the serial numbers used in this application does not imply the order of execution; the execution order of each process should be determined by its function and internal logic. For example, the terms "first," "second," and other various terminology (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0070] Furthermore, any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0071] IV. The terms “comprising” and “having” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may include other steps or units that are not expressly listed or that are inherent to such process, method, product or device.

[0072] V. In this application, "for indicating" can be understood as "enabling", and "enabling" includes direct enabling and indirect enabling. When describing information for enabling A, it may include whether the information directly enables A or indirectly enables A, but it does not mean that the information necessarily carries A.

[0073] The information that enables the information is called the information to be enabled. In the specific implementation process, there are many ways to enable the information to be enabled, such as, but not limited to, directly enabling the information to be enabled, such as the information to be enabled itself or its index. It can also be indirectly enabled by enabling other information, where there is a relationship between the other information and the information to be enabled. It can also enable only a part of the information to be enabled, while the other parts are known or pre-agreed upon. For example, enabling specific information can be achieved by using a pre-agreed (e.g., protocol-defined) arrangement of various pieces of information, thereby reducing enabling overhead to some extent. Simultaneously, common parts of various pieces of information can be identified and enabled uniformly to reduce the enabling overhead caused by individually enabling the same information.

[0074] In addition, "instruction" can include direct instruction, indirect instruction, explicit instruction, and implicit instruction. When describing a certain instruction information to indicate A, it can be understood that the instruction information carries A, directly indicates A, or indirectly indicates A.

[0075] In this application, the information indicated by the instruction information is called the information to be instructed. In specific implementations, there are many ways to indicate the information to be instructed, such as, but not limited to, directly indicating the information to be instructed, such as the information to be instructed itself or its index. It can also indirectly indicate the information to be instructed by indicating other information, where there is a relationship between the other information and the information to be instructed. It can also indicate only a part of the information to be instructed, while the other parts are known or pre-agreed upon. For example, the instruction of specific information can be achieved by using a pre-agreed (e.g., protocol-defined) arrangement of various pieces of information, thereby reducing instruction overhead to some extent. Furthermore, the information to be instructed can be sent as a whole or divided into multiple sub-information pieces, and the sending period and / or timing of these sub-information pieces can be the same or different.

[0076] VI. The term "storage" or "preservation" in this application can refer to storage in one or more memory devices. These memory devices can be separately configured or integrated into an encoder, decoder, processor, or communication device. Alternatively, some memory devices can be separately configured, while others can be integrated into a decoder, processor, or communication device. The type of memory can be any form of storage medium, and this is not limited.

[0077] VII. The arrows or boxes indicated by dashed lines in the schematic diagrams in the accompanying drawings of this application represent optional steps or optional modules.

[0078] 8. In the embodiments of this application, "send" and "receive" represent the direction of signal transmission. For example, "send information to XX" can be understood as the destination of the information being XX, which may include direct transmission via the air interface or indirect transmission by other units or modules via the air interface. "Receive information from YY" can be understood as the source of the information being YY, which may include direct reception from YY via the air interface or indirect reception from YY by other units or modules via the air interface. "Send" can also be understood as the "output" of the chip interface, and "receive" can also be understood as the "input" of the chip interface. In other words, sending and receiving can occur between devices, such as between network devices and terminal devices, or within a device, such as between components, modules, chips, software modules, or hardware modules within the device via a bus, wiring, or interface.

[0079] To facilitate understanding of the embodiments of this application, the terms used in this application will be briefly explained.

[0080] To facilitate a better understanding of the technical solution of this application, some related technologies involved in the technical solution of this application are introduced.

[0081] 1. Neural Networks Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can output the following: (1-1) Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce non-linear characteristics into the neural network to convert the input signal into the output signal. The output signal of this activation function can be used as the input to the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple individual neural units, meaning the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0082] 2. Bidirectional encoding representation transformation model: Bidirectional encoder representations from transformers (BERT) are representative of bidirectional encoder representations from transformers. Their core lies in using the bidirectional encoding mechanism of the transformer architecture, combined with pre-training tasks such as masked language model (MLM) and next sentence prediction (NSP), to achieve context-sensitive deep language representations. The transformer model is a deep learning architecture based on self-attention.

[0083] The following section uses the BERT model as an example to introduce the structure of the bidirectional encoding representation transformation model.

[0084] The BERT network structure includes an embedding layer, a self-attention layer, a pooling layer, and a downstream task classification head. The self-attention layer contains a wide range of linear layers.

[0085] The embedding layer is the input interface of BERT, which is responsible for converting the original text into numerical vectors that the model can process. Its core design includes a triple embedding mechanism: token embeddings, position embeddings, and segment embeddings.

[0086] Among them, word embedding includes word piece: splitting words into sub-words (such as "playing" → "play" + "#ing"), which alleviates the problem of out-of-vocabulary words and reduces the size of the vocabulary.

[0087] Lexical embedding also includes special markers: a [CLS] marker is added at the beginning of the input sequence (for classification tasks) and a [SEP] marker is added at the end (to separate sentences or paragraphs).

[0088] The positional embeddings include learnable parameters: Unlike the original version of the transformer, BERT's positional embeddings are trainable vectors (with the same dimensions as the word embeddings, such as 768 dimensions), which are directly optimized through backpropagation and better adapt to sequences of different lengths.

[0089] Position embedding can provide the model with sequence order information, making up for the lack of natural order awareness in the self-attention mechanism itself.

[0090] Among them, fragment embedding includes sentence pair differentiation: different sentences in the input are marked with 0 and 1 (such as question and answer paragraphs in a question-answering task) to help the model understand the relationship between sentences.

[0091] Fragment embedding also includes single-sentence processing: if the input is a single sentence, all tokens share the same fragment embedding (all 0s or all 1s).

[0092] The self-attention layer is the core module of BERT. It achieves bidirectional context awareness through a multi-head self-attention mechanism and a feed-forward neural network. Its design includes the following key components: multi-head self-attention, feed-forward neural network (FFN), residual connections, and layer normalization.

[0093] Multi-head self-attention is used to execute parallel attention heads and attention computation. The parallel attention heads are used to divide the input vector into multiple subspaces, and each head independently calculates attention weights to capture the associations of different semantic dimensions. Attention computation is used by each head to generate query (Q), key (K), and value (V) vectors through a linear layer (fully connected layer) and then calculate the attention score.

[0094] The role of the linear layer in multi-head self-attention: Q, K, and V of each head are generated through an independent linear layer. For example, a vector with a dimension of 768 as input may become a vector with a dimension of 64 (12 heads × 64 = 768) after passing through a linear layer.

[0095] As an example, a linear layer of a self-attention layer holds two parameters: a weight matrix and a bias vector.

[0096] Wherein, the weight matrix W has a size of Where n is the input feature dimension (aligned with the output dimension of the embedding layer or preceding layer), and m is the output feature dimension (determining the input dimension of subsequent layers or the final output dimension). For example, in the BERT model, if the hidden layer dimension n=768, and the output feature dimension m=3072 (such as the first layer of a feedforward network), then... .

[0097] Bias vector b: size is The output feature dimension m is consistent with the output feature dimension m, and is initialized to zero or by a strategy.

[0098] Storage precision: Typically FP32 (32-bit floating-point number). In mixed-precision training, FP16 (16-bit floating-point number) can be used to improve computational efficiency, but gradient scaling must be ensured to avoid numerical overflow.

[0099] The input X of a linear layer, generally called the activation value, is a three-dimensional tensor with size . R is the real number field, B is the batch size, s is the length of the input sequence (e.g., the maximum sequence length of BERT model is 512), and n is the dimension of the input features.

[0100] The computation function of the linear layer defines the computational logic from the linear layer input X to the output Y. It is responsible for performing linear operations on the activation values ​​X of the input linear layer using weights W and biases b to obtain a tensor result Y. The computation process includes matrix multiplication. ,in, , The result is (Automatically expand batch and sequence dimensions via broadcast mechanism); bias addition: ,in, via broadcast to (Each sequence position is independently assigned the same bias). Output Y: Dimensions B×s×m, used as input to the next layer (e.g., self-attention or activation function). Since the parameters of the linear layer are generally of floating-point precision, the multiplication and addition operations in its linear operations are also of floating-point precision, and its calculations are performed on the floating-point computing unit of the hardware.

[0101] The feedforward neural network consists of two fully connected layers: each layer is followed by a rectified linear unit (ReLU) activation function, which performs a non-linear transformation on the features output by the self-attention mechanism. For example, an input vector of dimension 768 is first expanded to 3072 dimensions, and then compressed back to 768 dimensions.

[0102] The role of linear layers in a feedforward neural network: The two fully connected layers in an FFN are typical linear layers, responsible for feature transformation and dimension adjustment.

[0103] Among them, residual connection and layer normalization are used to perform residual connection and layer normalization. Residual connection adds the output of each layer to the input to alleviate the gradient vanishing problem in deep networks. Layer normalization normalizes the output after residual connection to stabilize the training process.

[0104] Pooling layers are used for global feature extraction. The design of BERT's pooling layers varies depending on the task, with two common forms: [CLS] label pooling and average / max pooling.

[0105] [CLS] Label Pooling: Directly uses the output vector labeled [CLS] (e.g., 768-dimensional) as the global representation of the entire sequence, suitable for classification tasks (e.g., sentiment analysis, text matching).

[0106] Associated with linear layers: If the dimension needs to be adjusted (e.g., reduced from 768 dimensions to 2 dimensions for binary classification), a linear layer will be added.

[0107] Average / Max Pooling: Takes the average or maximum value of the output vector of all words to generate a global representation, which is suitable as an auxiliary feature for sequence labeling tasks (such as NER).

[0108] Related to linear layers: After pooling, the linear layers may undergo dimensional transformation.

[0109] The downstream task-specific header is used for task adaptation.

[0110] BERT adapts to different downstream tasks through fine-tuning. The classification head design varies depending on the task, with three common task types: text classification tasks, question answering tasks, and sequence labeling tasks.

[0111] The structure of the text classification task is as follows: [CLS] Output → Linear Layer → Activation.

[0112] As an example, in sentiment analysis, a 768-dimensional vector is reduced to 2 dimensions through a linear layer, outputting positive / negative probabilities.

[0113] The structure of the question-answering task is as follows: output of the [CLS] and all lexical units of two sentences → linear layer → predict the start / end position of the answer.

[0114] As an example, the model outputs two scores (corresponding to the word positions at the beginning and end of the answer).

[0115] The structure of the sequence labeling task is as follows: output of each word → linear layer → activation (number of label categories).

[0116] As an example, each lexical character outputs the probability of corresponding tags such as person names and place names.

[0117] The role of the linear layer: The linear layer in the classification head is responsible for mapping the general representation of BERT to the task-specific space, which is the key to task adaptation.

[0118] BERT, through the collaborative design of embedding layers, self-attention layers, pooling layers, and classification heads, combined with the flexible feature transformation capabilities of linear layers, has achieved powerful bidirectional context modeling and task adaptation capabilities, becoming a cornerstone model in the field of NLP.

[0119] 3. Quantification: Quantization is a key technology in model compression and acceleration. Its core goal is to reduce the model's storage requirements, computational complexity, and memory bandwidth usage by converting numerical values ​​(such as weights and activation values) in the model from high precision (e.g., 32-bit floating-point numbers, FP32) to low precision (e.g., 8-bit integers, INT8), thereby ultimately improving the model's inference efficiency on hardware. The following section provides a detailed introduction to quantization principles, types, and the challenges of applying quantization in the BERT model.

[0120] 1) Basic principles of quantification Numerical range compression: Quantization maps continuous floating-point values ​​to a discrete integer space. For example, it maps the weights of FP32 from the range [-1.0, 1.0] to the range of INT8 [-128, 127].

[0121] Improved computational efficiency: INT8 occupies 1 / 4 of the storage space of FP32 (8-bit vs 32-bit), achieving storage savings; hardware support for integer operations (such as INT8 multiplication) is more efficient and consumes less power, achieving computational acceleration; low-precision data transmission reduces memory access latency, improves overall throughput, and achieves memory bandwidth optimization.

[0122] 2) Main types of quantification Post-training quantization Process: Apply quantization directly to the pre-trained model without retraining.

[0123] Advantages: Simple to implement, suitable for rapid deployment.

[0124] Disadvantages: It is sensitive to the distribution of model activation. If the distribution is very different (such as the attention layer of the BERT model), the quantization error will accumulate significantly, leading to a decrease in accuracy.

[0125] Quantitative perception training Process: Simulate quantization effects during training (e.g., insert pseudo-quantization operators), and optimize quantization parameters through backpropagation.

[0126] Advantages: It can learn quantization error and significantly improve accuracy (e.g., BERT's QAT scheme can control the accuracy loss to within 1%).

[0127] Disadvantages: High training cost, requiring adjustment of hyperparameters and training process.

[0128] Mixed precision quantization Strategy: Use different levels of precision for different layers or operations to balance precision and efficiency.

[0129] Applications: Suitable for precision-sensitive critical layers (such as the attention head of BERT).

[0130] Quantization is the core means to improve the hardware inference efficiency of BERT models, but it is necessary to solve two major problems: accuracy loss and operator efficiency.

[0131] Currently, improving the inference efficiency of BERT models on hardware while maintaining accuracy is of great significance, but existing technologies have significant shortcomings. On the one hand, accuracy loss in quantization algorithms is a core issue. For example, the PTQ scheme has limited calibration capabilities and insufficient generalization on models with large differences in activation distribution, leading to a severe drop in accuracy, necessitating the use of QAT to recover accuracy. On the other hand, inference performance in terms of hardware operators has not met expectations. Taking the 310P hardware as an example, its quantization operator computational efficiency offers no speedup benefit compared to floating-point operators, making it difficult for quantization to achieve the goal of improving performance.

[0132] In view of this, this application proposes a model quantization method, a model inference method, and an electronic device, which can obtain quantized model parameters with higher inference accuracy during the model training stage, and connect a quantized matrix multiplication operator during the inference stage to perform efficient inference based on the quantized model parameters obtained during the training stage.

[0133] See Figure 1 ,like Figure 1 As shown, this application embodiment provides a system architecture 100. In Figure 1 In the middle, data acquisition device 160 is used to collect training data.

[0134] After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0135] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input training data and compares the output result with the training label corresponding to the training input data until the difference between the output result of the training device 120 and the training label is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0136] The target model / rule 101 described above can be used to implement the method of the embodiments of this application. Specifically, the target model / rule 101 in the embodiments of this application can be a neural network. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0137] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 1The execution device 110 shown can be a terminal, such as a mobile phone, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud service. Figure 1 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include data to be processed input by the client device.

[0138] Preprocessing module 113 and preprocessing module 114 are used to preprocess the input data (such as data to be processed) received by I / O interface 112. In this embodiment, preprocessing module 113 and preprocessing module 114 may be omitted (or only one of them may be used), and the calculation module 111 may be used directly to process the input data.

[0139] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0140] Finally, I / O interface 112 returns the processing result, such as the data processing result obtained above, to client device 140, thereby providing it to the user.

[0141] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0142] exist Figure 1In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0143] It is worth noting that, Figure 1 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 1 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0144] like Figure 1 As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 can be the neural network in this application embodiment. Specifically, the neural network used in this application embodiment can be CNN, deep convolutional neural network (DCNN), recurrent neural network (RNN), etc.

[0145] See Figure 2 As an example, Figure 2 A schematic diagram of the system architecture 200 applicable to the embodiments of this application is provided.

[0146] System architecture 200 includes a training service device 210 and an inference service device 220. The training service device 210 quantizes data and trains a model based on the quantized data to obtain trained model parameters. The training service device 210 can be, for example, a server, a distributed training cluster, a supercomputer, or an edge training device. The model to be trained is, for example, a transformer model or an AI model based on a transformer model architecture. The training data is domain-adapted structured sequence data. For example, data used to train a transformer model in the field of natural language processing (NLP) may include text data. Data used to train a transformer model in the field of computer vision (CV) may include image or video data. Data used to train a transformer model in the field of speech may include audio signal data. This application does not limit the content, format, or acquisition method of the training data.

[0147] The target model can be used to implement the model training method or model inference method of the embodiments of this application. Specifically, the target model in the embodiments of this application can be a transformer model or an AI model based on the transformer model architecture.

[0148] The model parameters trained by the training service device 210 can be used by the inference service device 220 to determine the inference model, thereby enabling the execution of model inference calculations.

[0149] System architecture 200 also includes an edge device 230. Users can input data to the inference service device 220 through the edge device 230. The inference service device 220 can process the input data and return the processing results to the edge device 230, thereby providing them to the user.

[0150] In one scenario, the user can input data to the inference service device 220 via the edge device 230. In another scenario, the edge device 230 can actively acquire input data and input it into the inference service device 220. That is, the edge device 230 can act as a data acquisition terminal, collecting input data.

[0151] The edge device 230 can present the output results to the user, in the form of sound, image, or text data, etc.

[0152] Understandable, Figure 2 The system architecture 200 shown is merely an example of a system architecture provided in this application embodiment, and the positional relationships between the devices, components, modules, etc., shown in the figure do not constitute any limitation. For example, in Figure 2 In this context, training service device 210 and inference service device 220 can be the same device or separately deployed devices; for example, in... Figure 2 In this case, the endpoint device 230 and the inference service device 220 can be the same device, in which case the inference calculation process of the target model can be executed by the endpoint device 230. Alternatively, the inference calculation process of the target model can be partially executed by the inference service device 220 and partially by the endpoint device 230. Furthermore, the endpoint device 230 used for input data and the endpoint device 230 used for output results can be different endpoint devices.

[0153] The solutions described in this application are applicable to various hardware platforms and business scenarios. Hardware platforms include, but are not limited to, AI accelerators and inference chips, and edge computing devices. AI accelerators and inference chips can be designed with operators that integrate quantization-matrix multiplication-inverse quantization to reduce intermediate result storage and data transfer overhead, taking into account the parallel computing characteristics of tensor computing cores (such as tensor processing units (TPUs) and neural processing units (NPUs)). Edge computing devices include, for example, smartphones, autonomous vehicle computing units, and IoT terminals—devices sensitive to power consumption and latency. The above hardware platforms and business scenarios are merely illustrative examples, and this application does not limit their scope.

[0154] The solutions described in this application are applicable to various deployment and inference environments, such as cloud-based Model-as-a-Service, edge offline inference, and edge AI applications. Specifically, cloud-based Model-as-a-Service provides high-concurrency, low-latency API services for large language models or large-scale visual models. Edge offline inference enables real-time inference of complex models on resource-constrained edge devices without continuous network connectivity. Edge AI applications primarily integrate intelligent features into mobile applications or operating systems, such as real-time image processing and voice assistants. The above deployment and inference environments are merely illustrative examples, and this application does not limit their scope.

[0155] The solutions described in this application are applicable to various model types, such as large-scale language models, specifically various models in the Transformer series, where matrix multiplication is the core computational load and can significantly benefit from the operator fusion and bias optimization of this application. Another example is other models with linear layers at their core, whose computational graphs contain a large number of matrix multiplications and bias addition operations.

[0156] The solutions in this application are applicable to various industries and business scenarios, such as natural language processing, machine translation, intelligent dialogue, content generation, code completion, etc., and this application does not limit them.

[0157] See Figure 3 , Figure 3 This is a flowchart illustrating a model quantization method 300 provided in an embodiment of this application. Method 300 can be executed by a first device, or by a processor in the first device, or by a processing unit, chip, or circuit in the processor. As an example, the first device can be a training service device 210. For ease of understanding, the following description uses the first device as the execution subject.

[0158] like Figure 3 As shown, the method 300 includes: S310, The first device determines a set of parameters.

[0159] The first parameter set includes a first quantization parameter, a second quantization parameter, a first bias vector, and a second bias vector.

[0160] Wherein, the first quantization parameter is the quantization parameter of the activation value, and the second quantization parameter is the quantization parameter of the weight.

[0161] In this application, the first parameter set includes the parameters of the linear layer in the self-attention layer of the first model.

[0162] As an example, the first model is a neural network model, such as the BERT model.

[0163] As an example, the linear layer in the self-attention layer of the first training model includes weights, quantization parameters of the weights, quantization parameters of the activation values, a first bias vector, and a second bias vector.

[0164] In this application, the parameters in the first parameter set are used for the calculation of the linear layer.

[0165] The quantization parameter of the weight is used to quantize the weight. This quantization parameter of the weight can also be called the quantization scale of the weight, the quantization coefficient of the weight, etc. The weight can be represented by W (weights), but this application embodiment does not limit it.

[0166] As an example, the quantization parameters of the weights are a set of parameters used to convert the weights in the first model from their original high-precision floating-point representation to a low-precision fixed-point (or integer) representation. These parameters typically include the quantization bit width (e.g., 8-bit, 16-bit, etc.), the quantization scale factor, and the zero point.

[0167] As an example, the quantization parameters of the weights are used to perform per-channel quantization on the weights, and the quantization parameters of the weights are denoted as follows: , , where m is the output feature dimension, that is, each output channel (dimension m) is independently assigned quantization parameters.

[0168] For example, in the BERT model, the hidden layer dimension n=768 and the output feature dimension m=3072 (such as the first layer of the feedforward network). Each output channel uses an independent scale value to avoid quantization errors caused by distribution differences between channels.

[0169] The quantization parameter of the activation value is used to quantize the activation value. This quantization parameter of the activation value can also be called the quantization scale of the activation value, the quantization coefficient of the activation value, etc. The activation value can be represented by X (activations), but this application embodiment does not limit it.

[0170] As an example, the quantization parameters for activation values ​​are a set of parameters used to convert the activation values ​​of the first model input from their original high-precision floating-point representation to a low-precision fixed-point (or integer) representation. Similar to the quantization parameters for weights, it also includes the quantization bit width, quantization scaling factor, and zeros.

[0171] As an example, the quantization parameter of the activation value is used to perform per-token quantization on the activation value, and the quantization parameter of the activation value is denoted as... , , where s is the length of the input sequence, that is, each sequence position (token, sequence length s) is independently assigned a quantization parameter.

[0172] For example, when the maximum sequence length s=512, the activation value of each token uses an independent scale value to adapt to the distribution characteristics of different tokens.

[0173] As an example, the quantization parameter of the activation value is a static parameter.

[0174] One possible interpretation is that the quantization parameter of this static activation value can be understood as a fixed quantization parameter that does not require dynamic calculation based on different real-time inputs. That is, it does not need to be dynamically calculated based on the real-time input of the network's forward inference.

[0175] As an example, the quantization parameters of the static activation value can be pre-programmed or learned, for example, through a training process, and this application embodiment does not limit this.

[0176] The first bias vector and the second bias vector are used to provide the first model with the ability to fit data, or in other words, the first bias vector and the second bias vector are used to perform data fitting processing on the first model.

[0177] The first bias vector and the second bias vector can be understood as two matrices A and B. The first bias vector can be represented by matrix A and the second bias vector by matrix B.

[0178] As an example, a bias vector (e.g., b) can be decomposed into two low-rank matrices A and B. Where R is the real number field, s is the length of the input sequence, m is the output feature dimension, and k is the bias compression dimension. min(s,m), meaning k is much smaller than the minimum of s and m. For example, when k=16, the dimensions of A and B are s×16 and m×16 respectively, which can be calculated using low-rank matrix multiplication (A·B). T The approximate original bias vector b, where T represents the transpose of the A·B matrix multiplication.

[0179] S320, the first device quantizes the first model based on the first set of parameters.

[0180] In this application, the quantization of the first model includes quantizing the activation values ​​of the input of the first model and quantizing the weights of the first model.

[0181] As an example, the quantization of the weights in the first model is to quantize the weights W from a 32-bit floating-point format to an 8-bit integer format (int8) (this quantization operation can be denoted as w8).

[0182] As an example, the quantization of the activation value of the input to the first model is to quantize the activation value X from floating-point format to 8-bit integer format (this quantization operation can be denoted as a8).

[0183] As an example, the quantized weights are data precision ranges of 8-bit integers, and the quantized activation values ​​are data precision ranges of 8-bit integers.

[0184] In one possible implementation, the activation values ​​and the first bias vector of the first model are quantized using a first quantization parameter.

[0185] In one possible implementation, the weights of the first model and the second bias vector are quantized using a second quantization parameter.

[0186] As an example, the activation values ​​and the first bias vector of the first model are quantized using the following formula, or in other words, the quantized activation values ​​and the quantized first bias vector satisfy the following conditions respectively:

[0187]

[0188] As an example, the weights and second bias vector of the first model are quantized using the following formula, or in other words, the quantized weights and the quantized second bias vector satisfy the following conditions respectively:

[0189]

[0190] in, Indicates the quantized activation value, Indicates the quantized first bias vector, Indicates the quantized weights, Indicates the quantized second bias vector. Indicates the activation value, Indicates the first bias vector. Indicates the weight, Indicates the second bias vector, Indicates the first quantization parameter, Indicates the second quantization parameter, Indicates rounding. The instruction will truncate the rounded integer. This instruction truncates rounded integers to the range of 8-bit integer precision.

[0191] Alternatively, the activation value X and the first bias vector A are each divided by the first quantization parameter. We get a floating-point result, then use round() to round it to integer precision, and finally use clamp to truncate the integer after round() rounding to [ ]. Within the precision range of the int8 integer type, the quantized activation value is obtained. and the first bias vector after quantization .

[0192] An alternative interpretation is that the weight W and the second bias vector B are each divided by the second quantization parameter. We get a floating-point result, then use round() to round it to integer precision, and finally use clamp to truncate the integer after round() rounding to [ ]. Within the precision range of the int8 integer type, the quantized weights are obtained. and the quantized second bias vector .

[0193] As an example, [ The range is [-127, 127]. This application does not limit this aspect in its embodiments.

[0194] It should be noted that the linear operation method in the linear layer of the self-attention layer of the first model after quantization changes, that is, the operator changes.

[0195] As an example, the linear operations in the operators of the first model satisfy the following conditions:

[0196] in, Indicates the tensor result obtained from the linear operation. This indicates the concatenation result of the quantized activation value and the quantized first bias vector. The concatenation result of the quantized weights and the quantized second bias vector is indicated by T, where T represents... Transpose of matrix multiplication.

[0197] As an example, the quantized first bias vector The broadcast is [B, s, k], and the quantized activation values ​​are... splice as new input , Quantized weights and the quantized second bias vector spliced ​​as Where R is the real number field, s is the length of the input sequence, k is the bias compression dimension, n is the hidden layer dimension, m is the output feature dimension, and B is the second bias vector. This is the second bias vector after quantization.

[0198] As an example, based on and Perform int8 matrix multiplication: The result is within the int32 range (the data precision range of a 32-bit integer).

[0199] It should be noted that the linear operation of the quantized linear layer simulates the w8a8 quantization operation. However, since the quantization algorithm is required for training afterward, and the gradient of the real integer operation cannot be propagated, the quantization algorithm simulates the quantized integer operation. That is, the size of the data is the size of the int8 value (such as 127), but the precision is still a floating-point value (such as 127.0).

[0200] In one possible implementation, the output of the quantized linear layer can be restored to floating-point precision through dequantization, so as to represent a wider and more accurate range of data in subsequent inference processes.

[0201] As an example, element-wise sum Multiplying the matrix products together yields the inverse quantization result. .Right now, The following conditions must be met: .

[0202] In the above technical solutions, w8a8 quantization reduces the number of model parameters through per-channel / per-token quantization, low-rank decomposition, and dequantization operations, significantly reducing memory usage and computational overhead while maintaining model accuracy. Simulated quantization training utilizes pseudo-quantization nodes and gradient propagation, while integer operations run better on hardware, promoting hardware acceleration and enabling efficient inference deployment.

[0203] In this application, a quantization scheme during training, such as QAT (Quadratic Quantization), is used to obtain model parameters that enable accurate inference during training, which are then used to further determine the inference model.

[0204] Optionally, the method 300 further includes step S330.

[0205] S330, the first device trains based on the quantized first model and the first set of learning parameters to obtain first parameter information, the first parameter information is used to determine the second model, and the second model is used for inference.

[0206] In one possible implementation, the first device trains the quantized first model based on the first set of learning parameters to obtain the first parameter information.

[0207] The first set of learning parameters includes quantized parameters and quantized parameter information. For example, the first set of learning parameters includes quantized weights, quantized first bias vectors and quantized second bias vectors, first quantization parameters and second quantization parameters.

[0208] In one possible implementation, the first device uses the raw data to train the quantized first model to obtain the first parameter information.

[0209] As an example, the first quantized model is trained using QAT with the original data. The parameters in the first set of learning parameters can be used as learning parameters and updated during training. After training, a set of model parameters that ensures accurate model output is obtained, i.e., the first parameter information.

[0210] The first parameter information can be understood as the model parameters trained based on the quantized first model, or in other words, the first parameter information is the model parameters that can ensure the model output is accurate.

[0211] As an example, the first parameter information may include the training results of the weights of the first model, the training results of the first quantization parameter, the training results of the second quantization parameter, the training results of the first bias vector of the first model, and the training results of the second bias vector of the first model.

[0212] As an example, the training result of the weights of the first model mentioned above indicates the trained weights, which can also be called the model parameters of the weights, the parameter information of the weights, etc. The training results of other parameters can also be replaced with similar terms, and this application embodiment does not limit this.

[0213] As an example, the parameters in the first parameter information above can be transformed from model parameters.

[0214] As an example, model parameter transformation, also known as model conversion, refers to converting the original trained model parameters (first parameter information) saved by the training framework into the format of the inference framework. This includes removing training-related components (such as training mode markers, optimizer states, etc.) and converting the parameter format during training to a format compatible with the inference engine. This helps to further improve inference speed.

[0215] In one possible implementation, the first parameter information is used to determine the second model, and the second model is used for inference.

[0216] As an example, the first device can determine the weights of the first model, the first bias vector, and the quantization result of the second bias vector based on the first parameter information.

[0217] For example, the first device determines that the training result of the first bias vector of the first model is a quantized result based on the training result of the first quantization parameter. That is, based on the training result of the first quantization parameter, the training result of the first bias vector of the first model is quantized to obtain the first quantized result, which can be used as the first bias vector of the second model. It can be understood that the first bias vector of the second model is quantized.

[0218] For example, the first device determines the training result of the weights of the first model based on the training result of the second quantization parameter. That is, the training result of the weights of the first model is quantized based on the training result of the second quantization parameter to obtain the second quantization result, which can be used as the weights of the second model. It can be understood that the weights of the second model are quantized.

[0219] For example, the first device determines that the training result of the second bias vector of the first model is the result of quantization based on the training result of the second quantization parameter. That is, the training result of the second bias vector of the first model is quantized based on the training result of the second quantization parameter to obtain the third quantization result, which can be used as the second bias vector of the second model. It can be understood that the second bias vector of the second model is quantized.

[0220] In one possible implementation, the parameters of the second model of the above transformation (the first quantization result, the second quantization result, and the third quantization result) can be pre-stored.

[0221] In another possible implementation, the parameters of the second model of the above transformation (the first quantization result, the second quantization result, and the third quantization result) can be sent to the inference service device.

[0222] It is understood that the quantization parameters determined by the first device based on the first parameter information will not actually change during the inference stage. Therefore, they can be pre-stored or transmitted and directly loaded and used in the linear layer operations of the inference model, avoiding the execution of repeated calculation processes, reducing latency, and improving inference efficiency.

[0223] See Figure 4 , Figure 4 This is a flowchart illustrating a model inference method 400 provided in an embodiment of this application. Method 400 can be executed by a second device, or by a processor in the second device, or by a processing unit, chip, or circuit in the processor. For ease of understanding, the following description uses a second device as the execution subject. The second device, for example, is the inference service device 220 in system architecture 200.

[0224] like Figure 4 As shown, the method 400 includes: S410, the second device obtains the activation value of the second model.

[0225] The activation values ​​of the second model are used as the input of the second model.

[0226] As an example, the second model is a neural network model, such as the BERT model.

[0227] S420, the second device quantizes the activation value of the second model based on the training result of the first quantization parameter.

[0228] In one possible implementation, the activation values ​​of the second model input are quantized using the training result of the first quantization parameter to obtain a fourth quantization result, thereby ensuring the data precision range of the activation values ​​for linear computation as 8-bit integers.

[0229] S430, the second device controls the second model to perform inference based on the fourth quantization result and the first operator.

[0230] The first operator performs a linear operation based on the fourth quantization result.

[0231] When the second model performs inference based on the fourth quantization result and the first operator, the first bias vector of the second model is quantized based on the training result of the first quantization parameter, and the weights of the second model and the second bias vector of the second model are quantized based on the training result of the second quantization parameter.

[0232] In one optional implementation, the second device can receive first parameter information and determine the weights of the second model, the first bias vector of the second model, and the second bias vector of the second model based on the first parameter information.

[0233] As an example, the second device can acquire pre-stored first quantization result, second quantization result, and third quantization result.

[0234] As another example, the second device can receive the aforementioned model parameters sent by the first device, namely, the first quantization result, the second quantization result, and the third quantization result.

[0235] In one possible implementation, the second model is obtained based on the quantized first model.

[0236] As an example, an initial inference model is determined based on a quantized first model, which includes a first operator that is obtained based on quantization.

[0237] As an example, the second model is determined based on the initial inference model and the first parameter information.

[0238] As an example, the initial inference model is initialized based on the first parameter information, or in other words, the parameters of the initial inference model are updated according to the first parameter information to obtain the second model.

[0239] For example, the model parameters are transformed based on the first parameter information to obtain the transformed model parameters, and the initialization is completed using the transformed model parameters.

[0240] Specifically, the parameters of the initial inference model are updated based on the first parameter information, and model parameter transformation is performed. For example, the first quantization result corresponds to the first bias vector of the second model, that is, the first bias vector of the second model is the result of quantizing the training result of the first bias vector of the first model based on the training result of the first quantization parameters; the second quantization result corresponds to the weights of the second model, that is, the weights of the second model are the result of quantizing the training result of the weights of the first model based on the training result of the second quantization parameters; the third quantization result corresponds to the second bias vector of the second model, that is, the second bias vector of the second model is the result of quantizing the training result of the second bias vector of the first model based on the training result of the second quantization parameters.

[0241] The first quantization result corresponds to the first bias vector of the second model, which can be understood as the first quantization result serving as the first bias vector of the second model; the second quantization result corresponds to the weights of the second model, which can be understood as the second quantization result serving as the weights of the second model; and the third quantization result corresponds to the second bias vector of the second model, which can be understood as the third quantization result serving as the second bias vector of the second model.

[0242] It can be understood that the parameters of the second model (first bias vector, second bias vector, weights) are quantized parameters. Therefore, in the inference result, there is no need to recalculate them. Only the activation values ​​of the input to the second model need to be quantized.

[0243] As an example, model parameter transformation, also known as model conversion, refers to converting the original parameters (first parameter information) saved by the training framework into the format of the inference framework. This includes removing training-related components (such as training mode markers, optimizer states, etc.) and converting the parameter format during training to a format compatible with the inference engine. This helps to further improve inference speed.

[0244] In one possible implementation, the first result is obtained based on the reasoning of the first operator.

[0245] As an example, the linear operations in the first operator satisfy the following conditions:

[0246] in, Indicates the tensor result obtained from the linear operation. The concatenation result of the activation value of the second model after quantization and the first bias vector of the second model. The concatenation result of the weights of the second model and the second bias vector of the second model, where T represents... Transpose of matrix multiplication.

[0247] The quantized activation values ​​of the second model and the first bias vector of the second model satisfy the following conditions:

[0248]

[0249] The weights of the second model and the second bias vector of the second model satisfy the following conditions:

[0250]

[0251] in, The activation value of the second model after quantization is indicated. The first bias vector of the second model after quantization. Indicates the weights of the second model after quantization. The second bias vector of the second model after quantization. Indicates the activation value of the second model. Indicates the first bias vector of the second model. Indicates the weights of the second model, Indicates the second bias vector of the second model. Indicates the training result of the first quantization parameter. The training result indicating the weights of the first model. Indicates rounding. The instruction will truncate the rounded integer. This instruction truncates rounded integers to the range of 8-bit integer precision.

[0252] In one possible implementation, the output of the quantized linear layer can be restored to floating-point precision, which requires performing a dequantization operation, i.e. element-wise sum Multiplying the matrix products together yields the inverse quantization result. :

[0253] It should be noted that during the inference process, floating-point values ​​can represent a more precise range of data, so the quantized values ​​need to be dequantized.

[0254] As an example, during the second model inference, the first calculation and the second calculation are performed simultaneously based on the first operator. The first calculation is a matrix multiplication calculation, and the second calculation is the remaining vector operations.

[0255] As an example, the first computation is the matrix multiplication operation performed by the computation unit cube (matrix multiplication computation), such as convolution, fully connected layers, etc.; the second computation is the remaining vector operation performed by the computation unit vector (other vector computation), such as low-dimensional vector operations, specifically such as element-wise addition, scalar multiplication, dequantization scaling, etc.

[0256] As an example, when two computation or memory access operations have no data dependency and use different computing power or bandwidth, they can be executed in parallel to achieve mutual latency masking. For example, the parallel computation of cube and vector can be simply referred to as CV parallelism.

[0257] As an example, for deep fusion and computational memory access masking, the overall execution logic of the first operator is a fusion of Quant (activation quantization) + Int8MatMul (integer matrix multiplication) + Dequant (computation result dequantization).

[0258] This approach separates computationally intensive and memory-intensive tasks, avoiding resource idleness, maximizing hardware utilization, and improving model inference efficiency.

[0259] As an example, in the computation of a linear layer, operations that can achieve CV parallelism include: low-rank biased matrix multiplication and input activation quantization, Int8 core matrix multiplication and dequantization scaling factor calculation (weight scaling factor * activation scaling factor). Simultaneously, by partitioning the matrix multiplication along the three dimensions [m, k, n], the original matrix is ​​divided into smaller data blocks, eliminating data dependencies between different data blocks and achieving staggered computation / memory access parallelism.

[0260] See Figure 5 , Figure 5 A schematic diagram of a parallel computational process optimization for computer vision is shown. Figure 5 This involves parallel processing of computation and data loading / unloading operations. Specifically, computation is used to execute computational operations; data loading is used to move input data from memory into the computation unit; and data loading is used to move the computation results from the computation unit back into memory.

[0261] from Figure 5 As can be seen from the data, before optimization, the process was executed sequentially: data import; calculation; data export.

[0262] This sequential execution method means that at any given time, the state of a computing unit is unique; for example, it may be waiting for data to be loaded, performing calculations, or waiting for data to be loaded out, resulting in a significant amount of idle time.

[0263] After optimization, some operations can be parallelized.

[0264] For example, in the first stage: data import, the calculation unit and the data import operation have not yet started; in the second stage: data import: new data is imported again; calculation: while the new data is imported, the previously imported data is being calculated; data import: after the calculation is completed, the data import begins immediately, while the new data has been partially imported, and the calculation unit can continue to perform calculations.

[0265] This parallel operation greatly reduces the idle time of the computing unit, and the data loading, computation, and unloading operations partially overlap in time, thereby improving overall efficiency. Figure 4 The data shows that the optimized process saves about one-third of the time.

[0266] Therefore, by parallelizing computation and data loading / unloading operations, the idle time of computing units is reduced, thereby improving computational efficiency.

[0267] The above technical solution can decompose the original bias vector b into two matrices A and B. A and B are concatenated with the activation value X and weight W, and incorporated into the matrix multiplication of activation X and weight W, thereby improving inference efficiency. In addition, the designed algorithm adapts to the operator and combines hardware-friendly partitioning design and parallel optimization to achieve optimal performance.

[0268] The following is passed Figure 6 A flowchart detailing a model quantization method and inference method 600 of this application is provided.

[0269] See Figure 6 As an example, Figure 6 This is a brief introduction to a model quantization and inference method 600 provided in an embodiment of this application. Method 600 can be seen as a specific implementation of the above-described methods 300 and 400. Method 600 includes steps S610 to S650.

[0270] S610, the training service device performs quantization of the first model.

[0271] In the first model, the linear layer in the original model structure is replaced with a quantized linear layer. After quantization, the linear layer has new parameters and the calculation of the forward propagation function (forward method) changes.

[0272] The quantization algorithm replaces the linear layers in BERT with quantized linear layers. The quantization of the linear layers adds quantization parameters to the weights. (That is, the second quantization parameter), the quantization parameter of the input activation value. (First quantization parameter) Two quantization parameters, due to activation performing word-by-word meta-quantization, Because the weights are quantized channel by channel, Furthermore, the original bias vector It is altered and decomposed into two matrices A and B (i.e., the first bias vector and the second bias vector). Where R is the real number field, s is the length of the input sequence, m is the output feature dimension, and k is the bias compression dimension. Since min(s,m) and k are much smaller than the minimum values ​​of s and M, A and B have the characteristic of low rank.

[0273] The quantization algorithm uses w8a8 quantization. w8 refers to weight quantization: quantization of the weights in the model... Quantize from 32-bit floating-point format to 8-bit integer format. a8 indicates active quantization: the active value is also quantized... Quantization is performed from floating-point format to 8-bit integer format. This reduces the number of parameters in the model, further reducing memory usage and computational load. At the same time, integer operations can run better on modern hardware, thus improving the computational efficiency of the model.

[0274] For details on the specific quantization operations of the linear layer, please refer to the detailed description in S310 of Method 300 above.

[0275] S620, the training service device trains the inference model parameters based on the quantized first model.

[0276] The quantized first model is trained using the original data to obtain a set of model parameters that ensure accurate model output, i.e., the first parameter information. For details of the first parameter information, please refer to S330 in Method 300, which will not be elaborated upon here.

[0277] After quantization, the model is trained using the original data with QAT, and the linear layer weights... Bias matrices A and B and quantization parameters , These parameters are learned and updated during training. After training, a set of model parameters (including linear layer weights) is obtained that ensures accurate model output. Bias matrices A and B and quantization parameters , ).

[0278] S630 converts the trained model parameters into inference model parameters and sends them to the initial inference model.

[0279] The parameters of the inference model are obtained through parameter transformation.

[0280] For details on model parameter transformation, please refer to the specific description of S330 in Method 300, which will not be elaborated here.

[0281] It can also determine the quantization results of the weights, first bias vector, and second bias vector of the first model based on the trained model parameters, and pre-store them or send them to the initial inference model. The quantized linear layer in the inference model can then directly load and use them, thereby improving inference latency.

[0282] For details, please refer to the specific description of S330 in method 300, which will not be repeated here.

[0283] S640, the inference service device initializes the initial inference model based on the model parameters to obtain an accurate inference model (second model).

[0284] The inference service device can determine an initial inference model based on the quantized first model. This initial inference model includes a first operator, which is obtained based on quantization.

[0285] The inference service device initializes the initial inference model based on the parameters of the inference model converted by S530, and obtains the second model.

[0286] The embodiments of this application do not limit the initialization process.

[0287] S650, the second model performs inference based on real data.

[0288] In this process, real data serves as the input to the second model. The input activation values ​​of the second model are quantized based on the training results of the first quantization parameters included in the first parameter information. The weights, first bias vector, and second bias vector of the second model can be directly loaded from pre-stored quantized parameters, avoiding redundant calculations. The prediction result is obtained after inference based on the first operator. This application does not limit the inference process in its embodiments.

[0289] Based on the above solution, a quantization acceleration method is provided for the entire AI training and inference process, specifically covering quantization model generation tools in the training phase and a library of efficient quantization matrix multiplication operators in the inference phase. In the training phase, dedicated algorithm tools generate quantization weights and related quantization parameters with high inference accuracy; in the inference phase, the trained weights and quantization parameters are loaded, combined with optimized quantization matrix multiplication operators, to achieve accurate inference with low latency and high throughput.

[0290] The above, combined with Figures 2 to 6 The quantization and inference methods provided in the embodiments of this application are described in detail below. The following will combine... Figure 7 The electronic device provided in the embodiments of this application will be described. It should be understood that the description of the electronic device corresponds to the description of the method embodiments. Therefore, for content not described in detail, please refer to the method embodiments above. For the sake of brevity, it will not be repeated here.

[0291] Figure 7 This is a schematic structural diagram of an electronic device 700 provided in an embodiment of this application. The electronic device 700 can be used to execute the methods 300, 400, or 600 described above.

[0292] like Figure 7 As shown, the electronic device 700 includes one or more processors 710; one or more memories 720; the one or more memories 720 store one or more instructions that, when executed by one or more processors 710, cause the method as described in any of the possible implementations above to be executed.

[0293] For example, electronic device 700 can be the first device mentioned above, the second device mentioned above, or other server devices or electronic devices. For instance, electronic device 700 can be a server group or computer cluster consisting of one or more servers. Furthermore, electronic device 700 can be a mobile phone, tablet computer, wearable device, in-vehicle device, AR / VR device, laptop computer, UMPC, netbook, personal digital assistant, etc.

[0294] This application also provides an electronic device including one or more processors and one or more memories, wherein the memories store one or more programs, and the processor executes the one or more programs to cause the methods described in any of the possible implementations above to be executed. For example, the electronic device may be used to execute method 300, method 400, or method 600 described above.

[0295] In some examples, the electronic device also includes a communication interface for receiving signals, a memory for storing signals, and a communication receiver for transmitting signals to a processor, which processes the signals such that methods as described in any of the possible implementations above are executed.

[0296] This application also provides a server device, including one or more processors and one or more memories, wherein the memories store one or more programs, and the processors execute the one or more programs to cause the methods described in any of the possible implementations above to be performed. For example, the electronic device can be used to execute method 300, method 400, or method 600 described above.

[0297] In some examples, the server device also includes a communication interface for receiving signals, a memory for storing signals, and a communication receiver for transmitting signals to a processor, which processes the signals such that methods as described in any of the possible implementations above are executed.

[0298] This application also provides a chip including a processor and a communication interface. The communication interface is used to receive signals and transmit the signals to the processor. The processor processes the signals so that the method in any of the possible implementations described above is executed.

[0299] For example, the chip can be a chip system or a standalone chip, etc.

[0300] This application also provides a readable storage medium (also known as a computer-readable storage medium) that stores instructions that, when executed on an electronic device, cause the electronic device to perform the aforementioned method steps to implement the methods described in the above embodiments.

[0301] This application also provides a program product (also known as a computer program product) that, when run on an electronic device, causes the electronic device to perform the aforementioned steps to implement the methods described in the above embodiments.

[0302] Embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store instructions, and when the apparatus is running, the processor may execute the instructions stored in the memory to cause the apparatus to perform the methods in the above-described method embodiments.

[0303] One or more modules or units described herein can be implemented in software, hardware, or a combination of both. When any of the above modules or units are implemented in software, the software exists as computer program instructions and is stored in memory. The processor can be used to execute the program instructions and implement the above method flow. The processor can include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be built into a SoC (System-on-a-Chip) or an application-specific integrated circuit (ASIC), or it can be a separate semiconductor chip. In addition to the cores within the processor for executing software instructions to perform calculations or processing, it may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0304] When the modules or units described herein are implemented in hardware, the hardware may be any one or any combination of a CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, application-specific digital circuit, hardware accelerator, or non-integrated discrete device, which may run the necessary software or perform the above method flow independently of software.

[0305] When the modules or units described herein are implemented using software, they may be implemented, in whole or in part, in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, they produce, in whole or in part, the flow or function according to the embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0306] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0307] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0308] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0309] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0310] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0311] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0312] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of quantizing a model, the method comprising: The method applied to a first device comprises: determining a first parameter set, the first parameter set comprising a first quantization parameter, a second quantization parameter, a first bias vector and a second bias vector, the first quantization parameter being used for quantizing activation values of input in a first model, the second quantization parameter being used for quantizing weights of the first model, the first bias vector and the second bias vector being used for data fitting processing of the first model; performing a quantization operation on the first model based on the first parameter set, wherein the activation values of the first model and the first bias vector are quantized using the first quantization parameter, and the weights of the first model and the second bias vector are quantized using the second quantization parameter.

2. The method of claim 1, wherein, The method further comprises: training the quantized first model based on a first learning parameter set to obtain first parameter information, the first parameter information comprising a training result of the weights of the first model, a training result of the first quantization parameter, a training result of the second quantization parameter, a training result of the first bias vector of the first model and a training result of the second bias vector of the first model, the first learning parameter set comprising the first quantization parameter, the second quantization parameter, the quantized weights, the quantized first bias vector and the quantized second bias vector; sending the first parameter information, the first parameter information being used for determining a second model, the second model being used for inference.

3. The method of claim 2, wherein, The method further comprises: quantizing the training result of the first bias vector of the first model according to the training result of the first quantization parameter to obtain a first quantization result; quantizing the training result of the weights of the first model according to the training result of the second quantization parameter to obtain a second quantization result; quantizing the training result of the second bias vector of the first model according to the training result of the second quantization parameter to obtain a third quantization result; sending the first quantization result, the second quantization result and the third quantization result.

4. The method according to any one of claims 1 to 3, characterized in that, The first quantization parameter is a static parameter, and the first quantization parameter is used for word-by-word quantization of the activation values of input in the first model.

5. The method according to any one of claims 1-3, characterized in that, The first bias vector and the second bias vector are low-rank vectors.

6. The method according to any one of claims 1-3, characterized in that, The quantized weights are in a data precision interval of 8-bit integers, and the quantized activation values are in a data precision interval of 8-bit integers.

7. The method of any one of claims 1-3, wherein: The quantized activation values and the quantized first bias vector satisfy the following conditions respectively: The quantized weights and the quantized second bias vector satisfy the following conditions respectively: wherein, indicating the quantized activation value, indicating the quantized first bias vector, indicating the quantized weight, indicating the quantized second bias vector, indicating the activation value, indicating the first bias vector, indicating the weight, indicating the second bias vector, indicating a quantization parameter of the activation value, indicating a quantization parameter of the weight, indicating rounding, indicating truncating the rounded integer, indicating truncating the rounded integer to a data precision interval of 8-bit integer.

8. The method of any one of claims 1-3, wherein, The neural network model is a model comprising a linear layer.

9. The method of claim 8, wherein, The neural network model is a bidirectional encoding representation transformation model.

10. A model inference method, comprising: The method applied to a second device comprises: obtaining activation values of a second model; quantize, based on the training result of the first quantization parameter, the activation value of the second model to obtain a fourth quantization result, the first bias vector of the second model being quantized based on the training result of the first quantization parameter, the weight of the second model and the second bias vector of the second model being quantized based on the training result of the second quantization parameter, the first bias vector and the second bias vector being used for data fitting processing of the second model; control the second model to perform inference according to the fourth quantization result and a first operator, wherein the first operator performs linear operation based on the fourth quantization result.

11. The method of claim 10, wherein, The method further includes: receiving first parameter information, the first parameter information being obtained by training the quantized first model; updating parameters of an initial inference model according to the first parameter information to obtain the second model, the initial inference model including the first operator.

12. The method of claim 11, wherein, The first parameter information includes a training result of a weight of the first model, a training result of the first quantization parameter, a training result of the second quantization parameter, a training result of a first bias vector of the first model, and a training result of a second bias vector of the first model.

13. The method of claim 12, wherein, The method further includes: receiving a first quantization result, a second quantization result, and a third quantization result, the first quantization result corresponding to the first bias vector of the second model, the second quantization result corresponding to the weight of the second model, and the third quantization result corresponding to the second bias vector of the second model, wherein the first quantization result is a result of quantizing the training result of the first bias vector of the first model according to the training result of the first quantization parameter, the second quantization result is a result of quantizing the training result of the weight of the first model according to the training result of the second quantization parameter, and the third quantization result is a result of quantizing the training result of the second bias vector of the first model according to the training result of the second quantization parameter.

14. The method according to any one of claims 10-12, characterized in that, The linear operation in the first operator satisfies the following conditions: wherein, an indication of a tensor result of the linear operation, an indication of a concatenation result of quantized activation values of the second model and a first bias vector of the second model, an indication of a concatenation result of weights of the second model and a second bias vector of the second model.

15. The method of claim 14, wherein, the quantized activation value of the second model and the first bias vector of the second model satisfy the following conditions, respectively: the weight of the second model and the second bias vector of the second model satisfy the following conditions, respectively: wherein, an activation value of the second model after quantization, a first bias vector of the second model after quantization, a weight of the second model after quantization, a second bias vector of the second model after quantization, an activation value of the second model, a first bias vector of the second model, a weight of the second model, a second bias vector of the second model, a training result of the first quantization parameter, a training result of the weight of the first model, indicating rounding, indicating truncating the integer after rounding, indicating truncating the integer after rounding to a data precision interval of 8-bit integer.

16. The method of any one of claims 10-12, wherein, The neural network model is a model including a linear layer.

17. The method of claim 16, wherein, The neural network model is a bidirectional encoder representation transformation model.

18. An electronic device, comprising: The computer readable storage medium stores computer program code which, when executed on an electronic device, causes the electronic device to perform the method of any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program code which, when executed on an electronic device, causes the electronic device to perform the method of any one of claims 1 to 17.

20. A computer program product, characterised in that, The computer program product includes computer program code which, when executed on an electronic device, causes the electronic device to perform the method of any one of claims 1 to 17.

Images