Mixed-precision model quantization method and system

Abstract

A mixed-precision model quantization method includes loading a trained model and quantizing the trained model using mixed-precision settings to generate a quantized model for inference. The trained model includes multiple residual connections. In each residual connection, a first activation bypasses at least one operand and is added to a second activation to generate a fourth activation. The second activation is the output of the first activation after processing by the at least one operand. The mixed-precision settings include: (a) in at least one residual connection of the multiple residual connections, the first activation, the second activation, and the fourth activation are assigned a first precision; and (b) a third activation in all operands bypassed by the at least one residual connection is assigned a second precision.

Description

[Technical Field]

[0001] This invention generally relates to a model quantization method, and more particularly, to a mixed-precision model quantization method and system. [Previous Technology]

[0002] In machine learning and deep learning, neural network models are commonly used to perform tasks such as image recognition, natural language processing, and speech recognition. These models consist of multiple interconnected layers of nodes that process input data to generate predictions or classification results. During model inference, the model uses weights to compute inputs and generate activations, which can be intermediate states produced by each layer in the model. Model quantization is often used to improve the efficiency of model inference. Model quantization involves using lower-precision numerical representations to represent the model's weights and / or activations. Model quantization reduces the model's size and computational requirements, thereby reducing latency.

[0003] However, reducing precision may also lead to a decrease in model accuracy, as lower precision may not represent the full range of numerical values, especially outliers. Existing solutions involve quantizing only the weights while keeping all activations at full precision. This approach maintains satisfactory accuracy, but because the activations are not quantized, existing solutions do not optimize latency well. Another solution is to perform low-precision full quantization on both weights and activations to achieve good latency. However, if the activations contain outliers, low precision cannot represent the data range of outliers, leading to a decrease in accuracy. [Summary of the Invention]

[0004] The following summary is illustrative only and is not intended to be limiting in any way. That is, the following overview is provided to introduce the concepts, key points, benefits, and advantages of the novel and non-obvious techniques described herein. Selected embodiments are further described in the detailed description below. Therefore, the following summary is neither intended to identify the essential features of the claimed subject matter nor to define the scope of the claimed subject matter.

[0005] In one embodiment, a mixed-precision model quantization method is disclosed. The mixed-precision model quantization method includes loading a trained model and quantizing the trained model using mixed-precision settings to generate a quantized model for inference. The trained model includes multiple residual connections. In each residual connection, a first activation bypasses at least one operand and is added to a second activation to generate a fourth activation. The second activation is the output of the first activation after processing by the at least one operand. The mixed-precision settings include: (a) in at least one residual connection of the multiple residual connections, the first activation, the second activation, and the fourth activation are assigned a first precision; and (b) a third activation in all operands bypassed by the at least one residual connection is assigned a second precision. The third activation is generated by any bypassed operand and processed within the bypassed operand (e.g., the third activation is generated by one bypassed operand and processed by another bypassed operand). The first precision is higher than the second precision.

[0006] In some embodiments, the trained model is a neural network model.

[0007] In some embodiments, the first precision is determined based on the precision configuration of the trained model, and the second precision is determined based on the latency configuration of the trained model.

[0008] In some embodiments, the full-precision format of the trained model is a 32-bit floating-point data format, and the first precision is represented by the full-precision format or is a precision lower than that of the full-precision format.

[0009] In some embodiments, the full-precision format of the trained model is a 32-bit floating-point data format, the first precision is represented by the full-precision format or a 16-bit floating-point data format, and the second precision is represented by a 16-bit integer data format.

[0010] In some embodiments, the method further includes: quantizing all weights in all operands bypassed by the at least one residual connection.

[0011] In some embodiments, the quantized weights have a 4-bit integer data format.

[0012] In some embodiments, the method further includes: generating inference output through the quantized model after the trained model has been quantized.

[0013] In some embodiments, the trained model is a large language model (LLM), and the plurality of residual connections are located within the transformation layer of the LLM.

[0014] In some embodiments, the at least one operand includes a normalization operand and a multi-head attention operand.

[0015] In another embodiment, a mixed-precision model quantization system is disclosed. The mixed-precision model quantization system includes a processor and a memory coupled to the processor, for example, the memory for storing program code. The processor is configured to perform operations including: loading a trained model stored in the memory, wherein the trained model includes a plurality of residual connections, in each of the plurality of residual connections, a first activation bypasses at least one operand and is added to a second activation to generate a fourth activation, wherein the second activation is the output of the first activation after processing by the at least one operand; and quantizing the trained model using a mixed-precision setting to generate a quantized model for inference. The mixed-precision setting includes: in at least one of the plurality of residual connections, the first activation, the second activation, and the fourth activation are assigned a first precision; and a third activation in all operands bypassed by the at least one residual connection is assigned a second precision, wherein the third activation is generated by any bypassed operand and processed within the bypassed operand; and the first precision is higher than the second precision.

[0016] In some embodiments, the trained model is a neural network model.

[0017] In some embodiments, the first precision is determined based on the precision configuration of the trained model, and the second precision is determined based on the latency configuration of the trained model.

[0018] In some embodiments, the full-precision format of the trained model is a 32-bit floating-point data format, and the first precision is represented by the full-precision format or is a precision lower than that of the full-precision format.

[0019] In some embodiments, the full-precision format of the trained model is a 32-bit floating-point data format, the first precision is represented by the full-precision format or a 16-bit floating-point data format, and the second precision is represented by a 16-bit integer data format.

[0020] In some embodiments, the operation performed by the processor further includes: quantizing all weights in all operands bypassed by the at least one residual connection.

[0021] In some embodiments, the quantized weights have a 4-bit integer data format.

[0022] In some embodiments, the operation performed by the processor further includes: generating inference output through the quantized model after the trained model has been quantized.

[0023] In some embodiments, the trained model is a large language model (LLM), and the plurality of residual connections are located within the transformation layer of the LLM.

[0024] In some embodiments, the at least one operand includes a normalization operand and a multi-head attention operand.

[0025] The present invention is provided by way of example and is not intended to be limiting. Other embodiments and advantages are described in the following detailed description. The invention is defined by the claims.

Implementation Method

[0027] The following description illustrates preferred embodiments of the present invention. These embodiments are merely illustrative of the technical features of the invention and are not intended to limit the scope of the invention. Certain terms are used throughout this specification and the claims to refer to specific components. Those skilled in the art will understand that manufacturers may use different terms to refer to the same components. This specification and the claims do not distinguish components by differences in name, but by differences in function. The scope of the invention should be determined with reference to the appended claims. The terms "comprising" and "including" used in the following description and claims are open-ended terms and should be interpreted as "comprising, but not limited to...". Furthermore, the term "coupled" means an indirect or direct electrical connection. Therefore, if a device is described as coupled to another device, it means that the device can be directly electrically connected to the other device, or indirectly electrically connected to the other device through other devices or connection means.

[0028] The mixed-precision model quantization method provided by this invention is applied to a trained model (also described as a "trained model" or "model after training"), such as a machine learning / deep learning model. Understandably, after model quantization, a trained model will generate a quantized model. Model quantization technology can reduce the storage requirements of the model on the device (the size of the quantized model is smaller than the size of the original trained model), improve the inference speed of the model, and reduce power consumption, etc. For ease of explanation, a neural network model is used as an example of a trained model, but this invention is not limited to this. For example, the trained model can be modified to be any model with multiple residual connections. In each residual connection, a first activation bypasses at least one operator and is added to a second activation to generate a fourth activation, where the second activation is the output of the first activation after processing by the at least one operator. In this embodiment, the operator can be a series of operations. In the following embodiments, for ease of explanation, the bypassed at least one operand is described as a series of N operands, but the present invention is not limited thereto.

[0029] Figure 1 is a schematic diagram of a mixed-precision model quantization system 100 according to an embodiment of the present invention. The mixed-precision model quantization system 100 addresses challenges associated with model quantization, a technique that uses lower-precision numerical representations for weights and activations to reduce the computational requirements and size of trained models (e.g., neural network models). Conventional quantization methods can result in lower latency but can lead to decreased model accuracy, especially when activations contain outliers. Outliers or extreme values ​​can negatively impact accuracy because lower precision may not represent the full range of data. The mixed-precision model quantization system 100 mitigates the problems of conventional quantization methods by employing a strategy of using different levels of precision within the same model. The key idea in the model quantization process provided by this invention is to maintain higher precision for specific activations that are more likely to contain outliers, while using lower precision for other activations to improve latency. The mixed-precision model quantization system 100 is effective in trained models with residual connections, where multiple activations are added together and outliers may accumulate. By assigning activations in residual connections to full or high precision, the mixed-precision model quantization system 100 effectively prevents the loss of anomalous information and maintains accuracy. Simultaneously, most computationally intensive operations are still quantized, ensuring that the quantized model improves latency. In the following embodiments, the mixed-precision model quantization system 100 aims to reduce latency and improve accuracy in (neural network) models with residual connections.

[0030] In Figure 1, the mixed-precision model quantization system 100 includes a processor 120 and a memory 11. The processor 120 is coupled to the memory 11. The memory 11 is used to store program code, which may include, for example, a trained model 110, on which the system is designed to process the trained model 110. Understandably, the trained model 110 (e.g., a neural network model like LLM) has completed its training phase and therefore has fixed weights. By storing the trained model 110 in the memory 11, the processor 120 can load the trained model 110 from the memory 11 and access all activations of the trained model 110 in subsequent quantization phases. In particular, the processor 120 retrieves the trained model 110 from the memory 11 to apply mixed-precision model quantization methods, thereby ultimately generating a quantized model that optimizes inference speed and reduces latency.

[0031] The processor 120 serves as a quantization tool or function applied to the trained model 110 stored in the memory 11. The processor 120 performs a mixed-precision model quantization process on the trained model 110, selectively quantizing different activations within the model structure. Specifically, the processor 120 can quantize the input activations (e.g., ACT1) of residual connection units, the output activations (e.g., ACT2 / ACT4) of residual connection units, and the intermediate activations (e.g., ACT3) generated by multiple operands within the residual connection units and processed within those units. The purpose of the processor 120 in quantizing these activations is to generate a quantized model that effectively balances inference accuracy and latency by assigning appropriate precision levels to these different activations.

[0032] In short, the mixed-precision model quantization process is performed on the trained model 110. The processor 120 loads the trained model 110 from the memory 11. The trained model 110 includes multiple residual connections. In each residual connection, a first activation (ACT1 as shown in Figure 3) bypasses at least one operand and is added to a second activation (ACT2 as shown in Figure 3) to generate a fourth activation (ACT4 as shown in Figure 4). The second activation is the output of the first activation after being processed by the at least one operand. The processor 120 uses a mixed-precision setting to quantize the trained model 110 to generate a quantized model for inference. The mixed-precision setting includes the following configuration: In at least one residual connection of the multiple residual connections, the first activation, the second activation, and the fourth activation are assigned a first precision. A third activation in / within all operands bypassed by the at least one residual connection is assigned a second precision. The third activation is generated by any bypassed operand and processed within the bypassed operand. The first precision is higher than the second precision.

[0033] Figure 2 is a schematic diagram of a trained model 110 stored in the memory 11 of the mixed-precision model quantization system 100. The trained model 110 may include residual connection units 1 to M, for example, the operands in different residual connection units may be different. Multiple residual connection units 1 to M may be coupled in series. M represents a positive integer indicating the total number of residual connection units within the trained model 110. For simplicity, residual connection unit 20 is described in the exemplary embodiment. Residual connection unit 20 may be a design used in machine learning and deep learning architectures. Residual connection unit 20 includes N operands 20a and an adder 20b, where N represents a positive integer. The N operands 20a may include, but are not limited to, various types of layers or functions common in the neural network of the trained model 110, such as fully-connected layers, multi-head attention layers, normalization layers, convolution layers, or other mathematical operations. In residual connection unit 20, the inputs and outputs of N operands 20a are combined by adder 20b to generate new activations. The residual connection unit 20 is designed to address issues such as outlier accumulation and improve the accuracy of quantization model inference. In one embodiment, the neural network model is a large language model (LLM). Residual connection units 1 to M can be transformer layers of the LLM, meaning multiple residual connections are located within the LLM's transformer layer. The mixed-precision model quantization system 100 can be applied to quantize the trained model 110. For example, the processor 120 of the mixed-precision model quantization system 100 can assign different precision levels to "at least one" residual connection unit. The precision allocation strategy in this embodiment balances the trade-off between model accuracy and computational efficiency (latency) during model inference. Details of the mixed-precision model quantization method are shown below.

[0034] Figure 3 is a schematic diagram of the residual connection unit 20 of the trained model 110 of the mixed-precision model quantization system 100. As previously described, the residual connection unit 20 may include N operands 20a and an adder 20b. Specifically, the N operands 20a include operands 20a1 to 20aN connected in series. N is a positive integer greater than zero. In Figure 3, the N operands 20a include an input for receiving a first activation ACT1 and an output for outputting a second activation ACT2. Understandably, "activation" refers to the intermediate data / state generated by each layer or operand within the trained model 110. For example, during model inference, inputs are computed using fixed weights, and these computations produce activations. Therefore, activations can be considered as the outputs computed by each layer or operand, serving as inputs for subsequent processes. In this embodiment, as shown in Figures 1 and 2, "first activation ACT1" can be defined as the output generated by the previous residual connection unit (or equivalently defined as the input of the residual connection unit 20). The second activation ACT2 is defined as the output generated by N operands 20a. Adder 20b is coupled to the input and output of the N operands 20a to output the fourth activation ACT4. Furthermore, in Figure 2, the residual connection unit 20 may include a residual connection RC and at least one operand (N operands 20a1 to 20aN as shown in Figure 2) that are skipped / bypassed by the residual connection RC. In the residual connection RC, the first activation ACT1, output from the previous layer / operation (such as the previous residual connection unit), bypasses at least one operand (20a1 to 20aN) and is added to the second activation ACT2, where the second activation ACT2 is the output of the first activation after being processed by at least one operand (20a1 to 20aN) bypassed by the corresponding residual connection. For example, in the residual connection RC shown in Figure 3, the first activation ACT1 and the second activation ACT2 are added to generate the fourth activation ACT4. The first activation ACT1 may refer to the output of the previous operand. At least one operand can refer to the portion of the residual connection unit that does not involve residual connection RC. For example, the third activation ACT3 is generated by any operand among the N bypassed operands 20a (e.g., operands 20a1 to 20aN) and processed within those N operands 20a (e.g., operands 20a1 to 20aN). In this embodiment, as shown in Figure 3, the at least one operand follows the sequential flow of operands 20a1 to 20aN. However, the present invention is not limited thereto.

[0035] In one embodiment, after the trained model 110 is quantized, the processor 120 can generate a quantized model in the memory 11. This quantized model is configured to generate inference output. To reduce latency and improve accuracy during the inference process of the quantized model, during model quantization, activations (ACT1, ACT2, and ACT4) in at least one residual connection RC are configured to meet a first precision, while activations (ACT3) in at least one residual connection unit associated with but not in the residual connection are configured to meet a second precision. For example, in this embodiment, the first activation ACT1, the second activation ACT2, and the fourth activation ACT4 in at least one residual connection RC are assigned the first precision. The third activation ACT3, among all operands bypassed by the at least one residual connection RC, is assigned the second precision, wherein the third activation ACT3 is generated by the bypassed operands (e.g., operands 20a1-20aN) and processed within the bypassed operands (as shown in Figure 3, the third activation is located inside the bypassed operands). The first precision is higher than the second precision. Here, "precision" refers to the level of detail used to represent a numerical value. In machine learning (ML) computation, precision determines the number of bits used to store the value. More bits allow for finer granularity and a wider range of representable values. In other words, precision refers to the accuracy with which a value is represented. Common numerical representations include floating-point numbers (e.g., fp32, fp16) and integers (e.g., int32, int16, int8, int4). For example, it can be understood that fp32 has higher precision than fp16, fp16 has higher precision than int16, int16 has higher precision than int4, and so on. The first precision is determined based on the precision configuration of the trained model 110. The second precision is determined based on the latency configuration of the trained model 110. In one embodiment, the weights in the trained model 110 are fixed. The trained model 110 is then run using a mixed precision setting, using lower precision for activations not in the residual connection RC (third activation ACT3) and higher precision for activations in the residual connection RC (first activation ACT1, second activation ACT2, and fourth activation ACT4). In one embodiment, the full-precision format of the trained model may be a 32-bit floating-point data format. In particular, the first activation ACT1, the second activation ACT2, and the fourth activation ACT4 in at least one residual connection RC may have a full-precision format or a higher precision than full-precision format, such as a 16-bit floating-point data format (fp16).The third activation ACT3, which is not in at least one residual connection, can have low precision in 16-bit integer data format (int16).

[0036] Understandably, in the mixed-precision model quantization system 100, the "mixed-precision model quantization" mechanism can be applied to "all" residual connection units or to "at least one" residual connection unit. For ease of explanation, Figure 4 shows an example of performing mixed-precision model quantization in "all" residual connection units. Figure 5 shows an example of performing mixed-precision model quantization in "one" residual connection unit. Details are as follows. For ease of explanation, in Figures 4 and 5, "thick arrows" indicate residual connection paths where mixed-precision model quantization is applied. The first type of activations in the residual connection (i.e., first activation ACT1, second activation ACT2, and fourth activation ACT4) are assigned higher precision. The second type of activations not in the residual connection (i.e., activation ACT3) are assigned lower precision. The reason for balancing the trade-off between model accuracy and computational efficiency (latency) of the mixed-precision model quantization system 100 based on this mixed-precision configuration is also explained below.

[0037] In the mixed-precision model quantization system 100, "outliers" refer to certain values ​​in activations that may cause accuracy problems. As previously mentioned, activations are intermediate states generated by each layer or operand in the model. Activations may include some outliers. In particular, outliers with low-precision data types cannot accurately represent large data ranges. For example, "int16" can only represent integers between -32768 and +32767. "fp16" can represent values ​​between -65504 and 66504. If an activation has an outlier of, for example, 50000, quantizing some activations to "int16" may result in information loss and negatively impact accuracy. In other words, outliers are values ​​that far exceed the typical range of the data, and these outliers may cause problems when attempting to represent the data in a low-precision format after model quantization. To address this issue, the mixed-precision model quantization system 100 assigns different precision levels to different activations. Activations in residual connections (ACT1, ACT2, and ACT4) are assigned higher precision formats. Assigning higher precision to activations in residual connections allows these activations to accurately represent and propagate outliers, preserving the model's accuracy. For example, if outliers exist in the first activation ACT1 and / or the second activation ACT2, the fourth activation ACT4 will maintain the information integrity of the outliers. Furthermore, in residual connection units, activations not in the residual connection (ACT3) are assigned a lower precision format. Since skipped / bypassed operands do not accumulate outliers in the same way, using lower precision for these activations reduces computational load and latency.

[0038] Furthermore, in the mixed-precision model quantization system 100, at least one weight of the N operands 20a in the trained model 110 can be quantized. At least one weight of the N operands 20a is fixed. The main reason for quantizing the weights of the N operands 20a is to achieve faster model inference and lower latency. Raw weights represented in a higher precision format, such as floating-point 32 (fp32), increase the complexity and computational requirements of the model during inference. By quantizing the weights, their numerical representation is reduced (e.g., from fp32 to int4 or int8), which reduces the model size and computational load. Therefore, since at least one quantized weight may have a 4-bit integer (int4) or 8-bit integer (int8) data format, the model requires less time to perform computation, thus reducing latency. In one embodiment, the processor 120 quantizes all weights in all operands bypassed by at least one residual connection. For example, the quantized weights may have a 4-bit integer (int4) data format. After configuring the precision of all activations and weights of the trained model 110, the mixed-precision model quantization system 100 can use the "quantized model" to generate inference output during the inference phase, providing a combination of high precision and low latency.

[0039] Figure 4 is a schematic diagram of the first precision configuration of multiple residual connection units 10 to 22 of the mixed-precision model quantization system 100. Figure 4 shows multiple residual connection units 10 to 22 coupled in series to illustrate the application of the mixed-precision model quantization method. In each residual connection unit, signal flow and activation processing are depicted. The first activation ACT1 is considered as the input to each residual connection unit. The first activation ACT1 is processed by a series of operands of each residual connection unit. In Figure 4, these operands may include normalization operands and multi-head attention operands. Each residual connection unit can be used as a transformation layer of LLM. The normalization operand receives the first activation ACT1 and performs a normalization function. The normalization operand can be any type of normalization operand. Normalization operands include, but are not limited to, root mean square (RMS) normalization operands, layer normalization operands, and group normalization factors. The output of the normalization operand is then passed to the multi-head attention operand.

[0040] For example, for the residual connection unit 10, a series of operands of the residual connection unit 10 includes a normalization operand 10a1 and a multi-head attention operand 10a2 coupled to the normalization operand 10a1. The multi-head attention operand 10a2 further processes the third activation ACT3. It is worth noting that the weights within the multi-head attention operand 10a2 can be quantized in a 4-bit integer data format (“int4”). However, the invention is not limited thereto. For example, all weights in the trained model 110 (e.g., the weights including the weights of the normalization operand 10a1 and the weights of the multi-head attention operand 10a2) can be quantized in a low-precision format (e.g., a 4-bit integer data format).

[0041] For the residual connection unit 10, the third activation ACT3 is generated and processed in a series of operands, specifically generated by the normalization operand 10a1 and processed within the multi-head attention operand 10a2. The third activation ACT3 can be specified to have a 16-bit integer data format (“int16”). The second activation ACT2 is output from the multi-head attention operand 10a2. In the residual connection unit 10, the second activation ACT2 can be specified to have a 16-bit floating-point data format (fp16) or a full-precision data format (e.g., a 32-bit floating-point data format (fp32)). Each residual connection unit has an adder. The adder is used to add the first activation ACT1 to the second activation ACT2 to generate a fourth activation ACT4, wherein the fourth activation ACT4 has the same precision as the first activation ACT1 and the second activation ACT2. In Figure 4, the fourth activation ACT4 in the residual connection unit 10 can have a 16-bit floating-point data format (fp16) or a full-precision data format.

[0042] In one embodiment, all residual connection units 10 to 22 can be configured with different precision levels to provide an optimal balance between model accuracy and computational efficiency (latency) during model inference. Since the mixed-precision mechanism and connection structure of residual connection unit 22 are similar to those of residual connection unit 10, details are omitted here. Taking two residual connection units 10 and 22 as examples, if mixed-precision model quantization is applied only to residual connection unit 10, then the third activation ACT3 processed with low precision corresponds to the activation ACT3 within the operands in residual connection unit 10. Similarly, the weights processed with low precision correspond to all weights within the operands in residual connection unit 10. Third activations unrelated to residual connection unit 10 may not be configured with low precision. In other words, the mixed-precision method of this embodiment can be performed at the granularity of a single residual connection (unit).

[0043] In one embodiment, "at least one" residual connection unit can be configured with different precision levels. Understandably, since the possible occurrence points of outliers can be predicted in advance, in order to optimize latency while maintaining accuracy, the mixed-precision model quantization system 100 can allocate only higher precision to residual connection units where outliers are more likely to occur, so that the information about outliers is not distorted. Furthermore, the mixed-precision model quantization system 100 can allocate lower precision to residual connection units where outliers are less likely to occur, in order to further optimize latency. Detailed explanation follows.

[0044] Figure 5 is a schematic diagram of the second precision configuration of multiple residual connection units 10 to 22 of the mixed-precision model quantization system 100. Figure 5 shows multiple residual connection units 10 to 22 coupled in series to illustrate the application of the mixed-precision model quantization method. However, the signal stream, activation definition, and structure of the mixed-precision model quantization system in Figure 5 are similar to those in Figure 4. Therefore, details are omitted here. As mentioned earlier, since the occurrence point of outliers can be predicted in advance, the mixed-precision model quantization system 100 can allocate lower precision in residual connection units where outliers are less likely to occur to further optimize latency. For example, in residual connection unit 10, the activations (ACT1, ACT2, ACT4) in the residual connection of residual connection unit 10 can be specified as 16-bit integer data format ("int16", lower precision), the same as the activation (ACT3) in the non-residual connection. Therefore, latency can be further optimized in residual connection unit 10 where outliers are less likely to occur. For example, in residual connection unit 22, the activations (ACT1, ACT2, ACT4) in the residual connections can be specified as 16-bit floating-point data format ("fp16", higher precision). By doing so, outlier information in residual connection unit 22 is not distorted. Therefore, accuracy can be maintained in residual connection unit 22.

[0045] In some embodiments, to achieve latency improvements in the trained model 110, the mixed-precision model quantization system 100 can quantize the weights of operands. The underlying principle is that raw weights are typically represented in a high-precision format (e.g., fp32), which increases the complexity and computational demands of the model during inference. By quantizing the weights, their numerical representation is reduced, for example, from fp32 to int4 or int8, which reduces the model size and computational load, thereby reducing the time required to perform computations and thus reducing latency. However, in other embodiments, the mixed-precision model quantization system 100 may quantize an operand even if it has no weights, as long as quantizing the operand contributes to overall latency optimization. Any technical modifications are within the scope of these embodiments.

[0046] Figure 6 is a flowchart illustrating the mixed-precision model quantization method executed by the mixed-precision model quantization system 100. The mixed-precision model quantization method includes steps S601 to S602. Steps S601 to S602 are shown below.

[0047] Step S601: Load the trained model 110, wherein the trained model 110 includes multiple residual connections, in each residual connection, the first activation ACT1 bypasses at least one operand and is added to the second activation ACT2 to generate a fourth activation ACT4, wherein the second activation ACT2 is the output of the first activation ACT1 after being processed by the at least one operand (bypassed by the residual connection);

[0048] Step S602: Quantize the trained model 110 using mixed precision settings to generate a quantized model for inference, wherein the mixed precision settings include: (a) in at least one residual connection of a plurality of residual connections, a first activation ACT1, a second activation ACT2 and a fourth activation ACT4 are assigned a first precision; (b) in all operands bypassed by the at least one residual connection, a third activation ACT3 is assigned a second precision, wherein the third activation is generated by any bypassed operand and processed within the bypassed operand.

[0049] The details of steps S601 to S602 have been described previously. Therefore, they are omitted here. The mixed-precision model quantization system 100 provides a solution to the trade-off between accuracy and latency in model quantization. As mentioned above, model quantization uses lower-precision numerical representations to allow for faster model inference speeds and shorter latency. However, lower precision may lead to a decrease in model accuracy because the numerical range is limited. The mixed-precision model quantization system 100 addresses this problem by using mixed-precision settings, using different precision levels in the same model to balance latency and accuracy. Its core idea is to specifically maintain high precision for activations in residual connections. Maintaining high precision in residual connections is important because activations in residual connections tend to accumulate outliers, which negatively impacts accuracy after quantization because low precision cannot represent the data range of outliers. By maintaining full precision or high precision (slightly lower than full precision) in residual connections, the mixed-precision model quantization system 100 can maintain the information integrity of outliers because the numerical values ​​of outliers can be represented using full precision or high precision. Furthermore, the mixed-precision model quantization system 100 retains the benefits of model inference latency introduced by quantization, since most computationally intensive operations are still quantized. Therefore, the mixed-precision model quantization system 100 improves accuracy while introducing acceptable latency.

[0050] In summary, embodiments of the present invention illustrate a mixed-precision model quantization system and a mixed-precision model quantization method. By setting the activations in residual connections to high or full precision, significant accuracy improvements can be achieved with only a minimal increase in latency. Embodiments of the present invention utilize the value accumulation property of activations in residual connections. When outliers occur in activations, they tend to propagate and accumulate through subsequent layers due to the design of residual connections. By using high or full precision for these activations, embodiments of the present invention ensure that outliers do not compromise accuracy during the inference process of the quantized model. Compared to conventional methods that suffer from decreased accuracy due to the limited numerical range of uniform low-precision representations, the present invention effectively mitigates the negative impact of outliers on accuracy while maintaining the latency advantage of model quantization. Therefore, the mixed-precision model quantization system can be applied to various models containing residual connections, such as large language models (LLMs) with transformer architectures.

[0051] The use of ordinal terms such as “first,” “second,” “third” to modify the elements of a patent application in the scope of the patent application does not in itself indicate any priority, priority, or order of one element of a patent application relative to another element of a patent application, or the temporal order of performing method actions, but is only used as a marker to distinguish one element of a patent application with the same name from another element element with the same name.

[0052] Although the embodiments and advantages of the present invention have been described in detail, it should be understood that various changes, substitutions, and modifications can be made to the present invention without departing from the spirit and scope defined by the claims. For example, new embodiments can be derived by combining several parts of different embodiments. The described embodiments are for illustrative purposes only and are not intended to limit the present invention. The scope of protection of the present invention shall be determined by the appended claims. Those skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. The above description is only a preferred embodiment of the present invention, and all equivalent changes and modifications made in accordance with the claims of the present invention should be included within the scope of the present invention. [Simplified Explanation of the Diagram]

[0026] The accompanying drawings (in which the same numerals denote the same components) illustrate embodiments of the present invention. The drawings are included to provide a further understanding of embodiments of the present disclosure, and are incorporated in and constitute a part of the present disclosure. The drawings illustrate implementations of embodiments of the present disclosure and, together with the description, serve to explain the principles of the embodiments of the present disclosure. It is understood that the drawings are not necessarily drawn to scale, as some components may be shown out of proportion to actual implementations to clearly illustrate the concepts of embodiments of the present disclosure. Figure 1 is a schematic diagram of a mixed-precision model quantization system according to an embodiment of the present invention. Figure 2 is a schematic diagram of a trained model stored in the memory of the mixed-precision model quantization system of Figure 1. Figure 3 is a schematic diagram of the residual connection units of the trained model of the mixed-precision model quantization system of Figure 1. Figure 4 is a schematic diagram of a first precision configuration of a plurality of residual connection units of the mixed-precision model quantization system of Figure 1. Figure 5 is a schematic diagram of a second precision configuration of a plurality of residual connection units of the mixed-precision model quantization system of Figure 1. Figure 6 is a flowchart illustrating a mixed-precision model quantization method performed by the mixed-precision model quantization system of Figure 1. In the following detailed description, numerous specific details are set forth for illustrative purposes so that those skilled in the art may more thoroughly understand the embodiments of the invention. However, it will be apparent that one or more embodiments may be practiced without these specific details, and different embodiments or different features disclosed in different embodiments may be combined as needed, and should not be limited to the embodiments illustrated in the accompanying drawings.

Claims

1. A mixed-precision model quantization method, comprising: A trained model is loaded, the trained model comprising multiple residual connections, wherein in each of the multiple residual connections, a first activation bypasses at least one operand and is added to a second activation to generate a fourth activation, wherein the second activation is the output of the first activation after processing by the at least one operand; and the trained model is quantized using a mixed precision setting, and all weights in all operands bypassed by the at least one residual connection are quantized to generate a quantized model for inference; wherein the mixed precision setting comprises: in at least one residual connection of the multiple residual connections, the first activation, the second activation, and the fourth activation are assigned a first precision; and a third activation in all operands bypassed by the at least one residual connection is assigned a second precision, wherein the third activation is generated by any bypassed operand and processed within the bypassed operand; and the first precision is higher than the second precision.

2. The method as described in request item 1, wherein, The trained model is a neural network model.

3. The method as described in request item 1, wherein, The first precision is determined based on the precision configuration of the trained model, and the second precision is determined based on the latency configuration of the trained model.

4. The method as described in request item 1, wherein, The full-precision format of the trained model is a 32-bit floating-point data format, and the first precision is represented by the full-precision format or a precision lower than that full-precision format.

5. The method as described in request item 1, wherein, The full-precision format of the trained model is a 32-bit floating-point data format, the first precision is represented by the full-precision format or a 16-bit floating-point data format, and the second precision is represented by a 16-bit integer data format.

6. The method as described in request item 1, wherein, The quantized weights are in 4-digit integer format.

7. The method as described in request item 1, wherein, The method also includes: generating inference output through the quantized model after the trained model has been quantized.

8. The method as described in request item 1, wherein, The trained model is a large language model (LLM), and the multiple residual connections are located within the transformation layer of the LLM.

9. The method as described in request item 1, wherein, The at least one operand includes a normalization operand and a multi-head attention operand.

10. A mixed-precision model quantization system, comprising: processor; and a memory coupled to the processor; wherein the processor is configured to perform operations including: loading a trained model stored in the memory, wherein the trained model includes a plurality of residual connections, in each of the plurality of residual connections, a first activation bypasses at least one operand and is added to a second activation to generate a fourth activation, wherein the second activation is the output of the first activation after processing by the at least one operand; and quantizing the trained model using a mixed precision setting, and quantizing all weights in all operands bypassed by the at least one residual connection to generate a quantized model for inference; wherein the mixed precision setting includes: in at least one of the plurality of residual connections, the first activation, the second activation, and the fourth activation are assigned a first precision; and a third activation in all operands bypassed by the at least one residual connection is assigned a second precision, wherein the third activation is generated by any bypassed operand and processed within the bypassed operand; and the first precision is higher than the second precision.

11. The system as described in claim 10, wherein, The trained model is a neural network model.

12. The system as described in claim 10, wherein, The first precision is determined based on the precision configuration of the trained model, and the second precision is determined based on the latency configuration of the trained model.

13. The system as described in claim 10, wherein, The full-precision format of the trained model is a 32-bit floating-point data format, and the first precision is represented by the full-precision format or a precision lower than that full-precision format.

14. The system as described in claim 10, wherein, The full-precision format of the trained model is a 32-bit floating-point data format, the first precision is represented by the full-precision format or a 16-bit floating-point data format, and the second precision is represented by a 16-bit integer data format.

15. The system as described in claim 10, wherein, The quantized weights are in 4-digit integer format.

16. The system as described in claim 10, wherein, The processor also performs the following operations: after the trained model is quantized, it generates inference output using the quantized model.

17. The system as described in claim 10, wherein, The trained model is a large language model (LLM), and the multiple residual connections are located within the transformation layer of the LLM.

18. The system as described in claim 10, wherein, The at least one operand includes a normalization operand and a multi-head attention operand.

Images