Model quantization methods and computing equipment

Abstract

This application provides a model quantization method and computing device. The method involves: determining first model data to be quantized; determining a corresponding target exponent representation based on the numerical value of the first model data; and quantizing the first model data from a first precision to a second precision according to the target exponent representation to generate second model data. Both the first and second precisions correspond to floating-point types. Furthermore, exponent portions with the same actual value correspond to different exponent encoding values ​​for different exponent representations. The technical solution provided by this application improves the flexibility of model quantization operations.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and in particular, to a model quantization method and a computing device. BACKGROUND

[0002] Neural network models are usually calculated using floating-point numbers. Model quantization refers to the operation process of quantizing model data from high precision to low precision, to achieve the purposes of model compression, reducing memory occupation, and improving calculation speed.

[0003] However, the data expression range of low-precision floating-point numbers is limited, resulting in poor flexibility of model quantization operations. SUMMARY

[0004] Embodiments of the present application provide a model quantization method and a computing device to solve the technical problem of poor flexibility of model quantization operations in the prior art.

[0005] In a first aspect, embodiments of the present application provide a model quantization method, comprising:

[0006] determining first model data to be quantized in a model;

[0007] determining a target exponential representation mode corresponding to the expression value of the first model data;

[0008] quantizing the first model data from a first precision to a second precision according to the target exponential representation mode to generate second model data; wherein the first precision and the second precision both correspond to floating-point numbers; and wherein the exponential encoding values of the same exponential part corresponding to different exponential representation modes are different.

[0009] In a second aspect, embodiments of the present application provide a computing device, comprising a processing component and a storage component, the storage component storing one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the model quantization method of the first aspect described above.

[0010] In a third aspect, embodiments of the present application provide a computer storage medium storing a computer program, which is executed by a computing device to implement the model quantization method of the first aspect described above.

[0011] In this embodiment, for the first model data to be quantized, a corresponding target exponent representation is determined based on the expressed value of the first model data. The first model data is then quantized from a first precision to a second precision according to the target exponent representation to generate second model data. Both the first precision and the second precision correspond to floating-point types. Furthermore, exponents with the same actual value correspond to different exponent encoding values ​​for different exponent representations. Since the exponent portion of a floating-point type determines the data expression range and is limited by the number of bits, this embodiment selects the corresponding target exponent representation for quantization based on the expressed value of the first model data, enabling the first model data to be effectively quantized to the second precision and improving the flexibility of the model quantization operation.

[0012] These or other aspects of this application will become more apparent from the description of the following embodiments. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart of an embodiment of a model quantization method provided in this application is shown;

[0015] Figure 2 This illustration shows a scenario interaction diagram of an embodiment of this application in a practical application;

[0016] Figure 3 A flowchart of one embodiment of a data quantization method provided in this application is shown;

[0017] Figure 4 This invention provides a schematic diagram of the structure of one embodiment of a data quantization device.

[0018] Figure 5 A schematic diagram of one embodiment of a computing device provided in this application is shown. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0020] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0021] The technical solutions of this application can be applied to data precision quantization scenarios, that is, quantizing from one precision to another, mainly for floating-point data. In practical applications, in order to ensure high precision, most scientific calculations are performed using floating-point data, especially the related calculations involved in artificial intelligence (AI) models.

[0022] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and technology, and fundamental theory.

[0023] Artificial intelligence models can be implemented using neural network models, which are deep learning models obtained by training artificial neural networks. With technological advancements, neural network models have made significant progress in computer vision tasks such as image classification, object detection, image segmentation, natural language processing, and video classification. However, neural network models often contain large amounts of model data, requiring substantial device resources and making them difficult to run efficiently on terminal devices. Therefore, it is necessary to compress neural network models to reduce the device resources they consume.

[0024] Model quantization is an effective method for compressing neural networks. Specifically, it involves quantizing the model data of a neural network model from high precision (high bit count) to low precision (low bit count) for representation. Model quantization operations can include quantization during the training phase or quantization during the inference phase. The technical solution of this application embodiment can be applied to quantization scenarios during the training phase or quantization scenarios during the inference phase. In particular, quantization during the inference phase is particularly important, as the inference demand of neural network models on public clouds is increasing, creating a need to optimize the performance of inference tasks. However, inference does not involve backpropagation, and there are many unimportant parameters in the network or parameters that do not require very fine precision to represent. Therefore, model quantization can reduce the memory footprint of neural network models and improve computation speed.

[0025] To facilitate understanding of the technical solution of this application, the technical terms that may be involved in this application will be explained below:

[0026] Bit (Binary digit): A computer science term, it is a unit of information, derived from the English word BIT. It is also the digit in binary numbers, a unit of measurement for information. In the binary number system, each 0 or 1 represents a bit; therefore, the number of bits is the number of bits in a binary number.

[0027] Floating-point numbers: Floating-point data is numbers where the exponent is not fixed; it has both a fractional part and an integer part. In computers, floating-point numbers are typically divided into two parts: an exponent and a mantissa. The exponent part is represented by a binary fixed-point integer, and the mantissa part is represented by a binary fixed-point fractional part. The length of the exponent part determines the range of data, and the length of the mantissa part determines the precision. In addition, floating-point numbers also have a sign bit and a radix. The sign bit is located in the highest bit of the floating-point number; 0 represents a positive number, and 1 represents a negative number.

[0028] Exponent bias: In floating-point representation, the exponent part is encoded as an index minus an offset from the actual value of the exponent part. The IEEE 754 standard (a binary floating-point arithmetic standard) specifies an exponent bias of 2. e-1 -1, where e refers to the number of bits in the exponent part.

[0029] FP8: 8-bit floating-point number, where 1 bit is the sign bit and the remaining 7 bits are the exponent and mantissa.

[0030] FP16: 16-bit floating-point number.

[0031] FP32: 32-bit floating-point number.

[0032] Currently, neural network models mostly represent their parameters and computations using 32-bit single-precision floating-point numbers or 64-bit double-precision floating-point numbers. Model quantization can be used to quantize model data from 32 bits to 16 bits or 8 bits, for example. However, due to the limited data representation range of low-precision floating-point numbers, issues such as overflow can occur, which can affect model performance, computational processing, and reduce the flexibility of model quantization operations.

[0033] To improve the flexibility of model quantization operations, the inventors, after a series of studies, proposed the technical solution of this application embodiment. In this application embodiment, for the first model data to be quantized, the corresponding target exponent representation is determined according to the expressed value of the first model data; according to the target exponent representation, the first model data is quantized from the first precision to the second precision to generate the second model data; wherein, both the first precision and the second precision correspond to floating-point type; wherein, the exponent part with the same actual value corresponds to different exponent encoding values ​​for different exponent representations, that is, the data expression range corresponding to different exponent representations is different. Since the exponent part of the floating-point type determines the data expression range and is limited by the number of bits, this application embodiment selects the corresponding target exponent representation for quantization according to the expressed value of the first model data, so that the data expression range corresponding to the second precision can be dynamically adjusted, which can enable the first model data to be effectively quantized to the second precision, thereby improving the flexibility of model quantization operations.

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] Figure 1 A flowchart of one embodiment of a model quantization method provided in this application is shown. The method may include the following steps:

[0036] 101: Determine the first model data to be quantized in the model.

[0037] The first model data refers to the object to be quantized in the model, which can be model parameters or input data, etc.

[0038] This model can be used to handle processing tasks such as image classification, object detection, image segmentation, natural language processing (speech recognition, text recognition, etc.), or video classification. Specifically, this model can refer to a multimedia data processing model, where the input data can be multimedia data, including images, text, and video.

[0039] In one implementation, the model can be a specific image processing model, the input data can be image data, and the image processing model can be used for image classification, object detection, or image classification processing.

[0040] In another implementation, the model can be a speech processing model, the input data can be speech data, and the speech processing model can be used for speech recognition or speech conversion, etc.

[0041] In another implementation, the model can be a text processing model, and the input data can be text data. This text processing model can be used for text matching, intent recognition, text conversion, text classification, etc.

[0042] In another implementation, the model can be a video processing model, the input data can be video data, and the video processing model can be used for video classification, etc.

[0043] Of course, the model can also refer to any model that performs other types of processing tasks, and this application does not specifically limit it.

[0044] In a practical application, the model in the embodiments of this application can be a trained model, that is, the technical solution of the embodiments of this application can be applied to the model inference stage to perform quantization processing on the trained model, etc.

[0045] Optionally, in response to a model quantization request, the model to be quantized can be determined, and the first model data to be quantized within that model can be identified. This model quantization request can be triggered by a user, or it can be generated upon detecting a model invocation event. This model invocation event can be generated when a request is made to perform corresponding computational processing using the model, such as image classification, image segmentation, or natural language processing. It can be generated and sent by a user client with processing needs.

[0046] The technical solutions of this application embodiment can be executed by the server. The above-mentioned model quantization request can be triggered by the user based on the corresponding interface provided by the server, or the above-mentioned model quantization request or model call event can be triggered by the user. Of course, the technical solutions of this application embodiment can also be executed by the user. After the model training is completed, it can be deployed on the user. For example, when the user detects a model processing request, it can generate a model call event. Before executing the model call event, it first generates a model quantization request, thereby triggering the technical solutions of this application embodiment to perform model quantization.

[0047] This can be achieved by traversing the network layers contained in the model to determine the first model data to be quantized. Optionally, it can be to determine the first model data to be quantized that meets the quantization conditions. The quantization conditions can be set according to actual conditions, such as specific types of model parameters, or model parameters or input data of specific network layers. For example, model data related to matrix multiplication operations, such as convolution operations, can be used as the first model data to be quantized.

[0048] 102: Determine the corresponding target index representation method based on the numerical expression of the first model data.

[0049] In this paper, the first model data is a floating-point number. The expressed value of the first model data refers to its corresponding actual value. For ease of understanding, the expressed value in this paper can be represented in decimal form. The floating-point number can be represented as: (sign bit) * exponent (exponent part) * fraction (mantissa part). Its corresponding expressed value can be: z = (-1) n *x*2 y Where n is the sign bit, x is the mantissa value, 2 is the base, and y is the exponent code value. The mantissa value x = 1 + 1 / 2 m-1 +1 / 2 m—2 +……+1 / 2 m-m The actual value of the exponent is 2. e-1 +2 e—2 +……+2 e-e The exponential encoded value is equal to the actual value of the exponent portion minus the exponential offset. Here, 'e' represents the number of bits in the exponent portion, and 'm' represents the number of bits in the mantissa portion.

[0050] Among them, the exponent parts with the same actual value correspond to different exponent codes for different exponent representations, that is, the range of exponent codes corresponding to different exponent representations is different, and the range of data representation corresponding to floating-point numbers is different for different exponent representations.

[0051] Since the exponent determines the range of data representation, different exponent representations can correspond to different data representation ranges while keeping the number of bits in the exponent constant. The exponent representation indicates how the exponent code value corresponding to the exponent is calculated, so that exponents with the same actual value correspond to different exponent code values ​​for different exponent representations. The specific implementation of the target exponent representation will be described in detail in the following embodiments.

[0052] Quantization can be performed on the network layers of the model. This can be achieved by combining multiple model data points involved in the network layer to determine the corresponding target index representation. For example, the target index representation can be determined based on the maximum representation value among multiple model data points in the network layer. Alternatively, the target index representation can be determined by combining all model data points involved in the model, for example, by determining the maximum representation value among multiple model data points in the model.

[0053] 103: According to the target index representation, the first model data is quantized from the first precision to the second precision to generate the second model data.

[0054] In this application, both the first precision and the second precision correspond to floating-point numbers. The first precision is the initial precision of the first model data, and the second precision can be the target quantization precision. In a practical application, the second precision can be 8 bits, and the first precision can be 64 bits, 32 bits, or 16 bits, etc. This application does not limit this.

[0055] Optionally, the second precision may be included in the model quantization request.

[0056] The quantization from the first precision to the second precision can be achieved using traditional methods. Of course, in order to reduce the complexity of quantization, other methods can be used, which will be described in detail in the following examples.

[0057] In this embodiment, the corresponding target index expression method is selected for quantization based on the expression value of the first model data, so that the data expression range corresponding to the second precision can be dynamically adjusted, and the first model data can be effectively quantized to the second precision, thereby improving the flexibility of model quantization operation.

[0058] As an optional approach, determining the corresponding target index representation based on the numerical representation of the first model data may include:

[0059] Based on the numerical representation of the first model data, determine the target index representation method for adjusting the original index offset value to the target index offset value.

[0060] The original exponent offset value can be 2 as specified in the IEEE 754 standard. e-1-1. For example, for FP8, assuming the E4M3 encoding format (4 bits for the exponent and 3 bits for the mantissa) is used, the original exponent offset is 7, and the exponent encoding range is [-6, 8]. In this example, the exponent can start from 0001, corresponding to the decimal data 1. Subtracting the exponent offset 7 gives the minimum value of the exponent encoding range, -6. For example, assuming the maximum value of FP8 is 0 1111111, according to the original exponent offset, the value of 0 1111 111 is: 2. 8 *(1+7 / 8)=480. It should be noted that this application mainly introduces FP8 using the E4M3 encoding format, but this application is not limited to this. For example, it is also applicable to other encoding formats such as E5M2 (the exponent part occupies 5 bits and the mantissa part occupies 2 bits).

[0061] In this embodiment, the original exponential offset value can be adjusted based on the expression value of the first model data to determine the target exponential offset value. For example, to expand the data expression range of FP8, the target exponential offset value can be adjusted to 0, then the exponential encoding range corresponding to the exponential part is [1, 15]. At this time, the expression value of 0 1111 111 is: 2 15 *(1+7 / 8)=61440, which increases the maximum expression value that FP8 can express by 128 times.

[0062] Optionally, in order to expand the range of numerical values ​​expressed in the second precision, the target exponent offset value can be smaller than the original exponent offset value. Of course, this application is not limited to this. The target exponent offset value can be selected from the range of values ​​formed by the maximum and minimum actual values ​​of the exponent part, etc. Of course, this application is not limited to this.

[0063] In some embodiments, multiple candidate exponent offset values ​​can be preset, with different candidate exponent offset values ​​corresponding to different exponent representations. Then, based on the numerical representation of the first model data, determining the target exponent representation for adjusting the original exponent offset value to the target exponent offset value may include:

[0064] Multiple index representations are determined to adjust the original index offset values ​​to candidate index offset values; different index representations correspond to different candidate index offset values.

[0065] Determine the numerical range corresponding to each of the multiple index representation methods;

[0066] Based on the expression values ​​of the first model data, determine the corresponding target expression value range;

[0067] Use the exponential representation of the target numerical range as the target exponential representation.

[0068] The target expression value range includes the expression value of the first model data. Optionally, if the expression value of the first model data hits multiple expression value ranges, the expression value range with the largest boundary value can be selected as the target expression value range.

[0069] In some embodiments, determining the target exponential representation for adjusting the original exponential offset value to the target exponential offset value based on the numerical representation of the first model data may include:

[0070] Multiple index representations are determined to adjust the original index offset value to the candidate index offset value; different index representations correspond to different candidate index offset values.

[0071] The data from the first model are quantized into second precision according to multiple index representation methods;

[0072] Calculate the precision loss corresponding to each of the multiple exponential representations;

[0073] Choose the exponential representation with the least loss of accuracy as the target exponential representation.

[0074] Alternatively, the target exponential offset value can be selected as the exponential representation that expresses the numerical range of the first model data and has the least precision loss.

[0075] Specifically, by quantizing the first model data into second precision using multiple exponential representations, multiple quantized second-precision data can be obtained. The precision loss corresponding to each of the multiple exponential representations can then be calculated as follows:

[0076] Determine the second-precision data corresponding to each of the multiple exponential representations;

[0077] Determine the numerical value of the second-precision data corresponding to each of the multiple index representation methods;

[0078] Calculate the differences between the expression values ​​of the first model data and the expression values ​​of data at different second precision levels;

[0079] The calculated difference information is used as the accuracy loss.

[0080] Therefore, the index representation with the smallest corresponding difference information can be selected as the target index representation.

[0081] As an optional approach, the difference information can specifically refer to the error. Optionally, the error calculation can be implemented in various ways, such as calculating MAE (Mean Absolute Error), MSE (Mean Square Error), or RMSE (Root Mean Square Error), etc. This application does not limit the specific implementation of this method.

[0082] In addition, as another alternative, the difference information can also be the ratio or difference between the expression values ​​of the first model data and the expression values ​​of different second precision data.

[0083] The numerical value of the second-precision data can be expressed as z = (-1) as described above. n *x*2 y The formula is obtained by conversion.

[0084] Different index representations, that is, different candidate index offset values, may result in different values ​​for the second precision data. By calculating the difference information, the index representation with the smallest difference information can be selected as the target index to ensure the accuracy of data quantification.

[0085] As an alternative approach, determining the corresponding target index representation based on the numerical values ​​of the first model data may include:

[0086] Based on the numerical values ​​expressed in the first model data, the target exponent representation method for representing the effective values ​​is determined by the maximum actual value of the exponent portion corresponding to the second precision.

[0087] According to the IEEE 754 standard, when the exponent is the maximum actual value, it does not represent a valid value and is usually used to represent outlier data, such as infinity. However, for model data, there are usually no outliers that require special representation. Therefore, the maximum actual value of the exponent can be used to represent a valid value. For example, for FP8, according to the IEEE 754 standard, 0 1111 111 represents the meaningless value NAN, and the maximum actual value is 0 1110 111 = 240. However, according to the technical solution of this application embodiment, 0 1111 111 can represent a valid value: 480, making the range of values ​​that can be expressed for FP8 twice as large as before.

[0088] In some embodiments, the target exponential representation method for determining the maximum actual value of the exponential portion corresponding to the second precision, based on the numerical representation of the first model data, to represent the effective numerical value may include:

[0089] If the model data meets the valid conditions, determine the target exponent representation method that uses the maximum actual value of the exponent part corresponding to the second precision to represent the valid values.

[0090] This valid condition can be set according to the actual situation to determine whether the model data is valid. At this time, the maximum actual value of the second precision corresponding to the exponent part can be determined to represent the target exponent representation of the valid value.

[0091] Optionally, the target exponent representation method for determining the maximum actual value of the exponent portion corresponding to the second precision, based on the numerical representation of the first model data, to represent the effective value may include:

[0092] Determine the target expression value corresponding to the maximum actual value when the exponent part corresponding to the second precision is the highest actual value;

[0093] When the numerical value of the first model data is less than the target numerical value, the maximum actual value of the second precision corresponding to the exponent part is determined to be the target exponent representation method for representing the effective value.

[0094] One way to achieve the valid condition is that the expression value of the first model data is less than the target expression value corresponding to the maximum actual value of the exponent part corresponding to the second precision.

[0095] Of course, as another optional approach, determining the corresponding target index representation based on the numerical representation of the model data can include:

[0096] The maximum actual value of the second precision corresponding to the exponent portion is determined to represent the effective value, and the target exponent representation method is used to adjust the original exponent offset value to the target exponent offset value.

[0097] In some embodiments, the method of determining the maximum actual value of the exponent portion corresponding to the second precision to represent the effective value and the target exponent representation of adjusting the original exponent offset value to the target exponent offset value may include:

[0098] The maximum actual value of the exponent portion corresponding to the second precision is determined to represent the effective value, and multiple exponent representation methods are used to adjust the original exponent offset value to the candidate exponent offset value; different exponent representation methods correspond to different candidate exponent offset values;

[0099] Determine the numerical range corresponding to each of the multiple index representation methods;

[0100] Based on the expression values ​​of the first model data, determine the corresponding target expression value range;

[0101] Use the exponential representation of the target numerical range as the target exponential representation.

[0102] Multiple index representation methods can be preset. Each index representation method can include: the maximum actual value of the index part is used to represent the effective value, and the original index offset value is adjusted to the corresponding candidate index offset value.

[0103] In some embodiments, the method of determining the maximum actual value of the exponent portion corresponding to the second precision to represent the effective value and the target exponent representation of adjusting the original exponent offset value to the target exponent offset value may include:

[0104] The maximum actual value of the exponent portion corresponding to the second precision is determined to represent the effective value, and multiple exponent representation methods are used to adjust the original exponent offset value to the candidate exponent offset value respectively;

[0105] Determine the numerical range corresponding to each of the multiple index representation methods;

[0106] The data from the first model are quantized into second precision according to multiple index representation methods;

[0107] Calculate the precision loss corresponding to each of the multiple exponential representations;

[0108] The target exponential representation is selected as the one with the least loss of accuracy and whose numerical range includes the numerical values ​​of the first model data.

[0109] The calculation of the precision loss corresponding to the multiple exponential representations can be found in the above embodiments, and will not be repeated here.

[0110] In some embodiments, quantizing the first model data from a first precision to a second precision may include:

[0111] Determine the initial precision of the first model data;

[0112] Determine the first exponent portion and the first mantissa portion corresponding to the first precision of the first model data;

[0113] Based on the second precision, determine the exponent encoding threshold of the second exponent portion corresponding to the second precision;

[0114] If the index code value corresponding to the first index part is less than the index code threshold, the index code value corresponding to the first index part is used as the index code value of the second index part, and the data of the first mantissa part is used as the data corresponding to the second mantissa part.

[0115] The exponent encoding threshold can be determined based on the exponent offset value of the second precision. When the target exponent representation includes the target exponent offset value, the exponent offset value of the second precision is the target exponent offset value. For example, assuming the target exponent offset value is 7, for FP8, the second precision is 8 bits, and the exponent encoding range corresponding to the exponent part is [-6, 8]. The exponent encoding threshold is 8.

[0116] If the index code value corresponding to the first index part is less than the index code threshold, it means that the index code value of the first index part does not exceed the index code range of the second index part corresponding to the second precision. In this case, the index code value corresponding to the first index part is used as the index code value of the second index part, and the data of the first mantissa part is used as the data of the second mantissa part.

[0117] Alternatively, the data in the first part of the decimal portion can be used as the data in the second part of the decimal portion by rounding (rounding to the nearest integer).

[0118] For example, assuming the first precision is 32 bits (i.e., the first model data is FP32) and the second precision is 8 bits (i.e., the second model data is FP8), since the range of the first mantissa value in FP32 is different from that in FP8 (FP32 has 23 bits in the first mantissa and FP8 has 3 bits in the second mantissa), optionally, the first mantissa can be rescaled by rounding, reducing it from 23 bits to 3 bits, and the scaled result can be used as the data corresponding to the second mantissa. For example, if the model data is 100, converting it to a 32-bit number results in z1 = (-1). 0 *1.100100*2 6 The mantissa in z1 contains 6 digits (6 decimal places). However, if it is converted to an 8-bit number, the mantissa will only have 3 digits. In this case, 1.100100 can be scaled by rounding to reduce it from 6 bits to 3 bits. Since the fourth bit in 100100 is 1, when scaling it to 3 bits, the third bit is incremented by 1. Therefore, the scaling result for 100100 is 101, and the scaling result for 1.100100 is 1.101. The resulting conversion is z1 = (-1). 2 *1.101*2 6 If the transformation result is z1 = (-1) 2 *1.101000*2 6If the mantissa is 101000 and the fourth bit is 0, then when scaling 101000 to 3 bits, the fourth to sixth bits can be discarded, and the resulting scaling is 101. The scaling result corresponding to 1.101000 is 1.101. The final 8-bit conversion result is z1 = (-1). 0 *1.101*2 6 .

[0119] In some embodiments, the method may further include: if the exponent coding value corresponding to the first exponent portion is greater than the exponent coding threshold, using the exponent coding threshold as the exponent coding value of the second exponent portion of the second model data;

[0120] Based on the first exponent part and the exponent encoding threshold, the first mantissa part is quantized to obtain the mantissa value corresponding to the second mantissa part.

[0121] If the index code value corresponding to the first index part is greater than the index code threshold, it means that the index code value of the first index part exceeds the index code range of the second index part. In this case, the first index part can be truncated. Specifically, the index code threshold is directly used as the index code value of the second index part. The part of the index part that exceeds the threshold can be padded with data from the mantissa part. Therefore, in some embodiments, the first mantissa part is quantized according to the first index part and the index code threshold to obtain the mantissa value corresponding to the second mantissa part.

[0122] The difference between the exponent code value corresponding to the first exponent portion and the exponent code threshold is determined, and the first mantissa portion is quantized based on the difference and the base, wherein the base corresponding to the first precision is equal to the base corresponding to the second precision.

[0123] Optionally, the quantization process of the first tail portion based on the difference and the base may include: using the base as the base and the difference as the exponent, performing a power operation on the base and the difference to generate the first operation result;

[0124] The data corresponding to the first mantissa is multiplied with the first operation result to generate the second operation result, and the second operation result is used as the data corresponding to the second mantissa.

[0125] For example, suppose the model data could be z1 = (-1) n1 *x1*2 y1 The quantized model data can be z3 = (-1) n3 *x3*2 y3In z1 and z3, n1 and n3 are equal. Therefore, quantizing z1 to z3 is essentially equivalent to quantizing x1 to x3 and y1 to y3. Assume z1 is FP32, z2 is FP8, the exponent encoding range of y1 is [-127, 128], the exponent encoding range of y3 is [-6, 8], and the exponent encoding threshold is 8. Therefore, if the exponent encoding value of y1 is greater than 8, the exponent encoding value of y3 can be determined to be 8. The portion of y1 exceeding 8 is then padded by adjusting x1. For example, if the exponent encoding value of y1 is 10, which is greater than 8, then after determining the encoding value of y3 to be 8, z3 = (-1). n3 *x3*2 8 To ensure that the quantized z3 is equal to or nearly equal to the size of z1, x3 can be adjusted to x1*2. 2 That is, z3 = (-1) n3 *(x1*2 2 )*2 8 In z3, the last part is x1*2. 2 The exponent is 8.

[0126] If the result of the second operation is correct, the corresponding data for the second decimal part can be rounded off, for example, x1*2. 2 The number of bits was reduced from 23 to 3, and the scaling result was used as the data corresponding to the mantissa in z3.

[0127] In some embodiments, after quantizing the first model data from a first precision to a second precision according to the target exponent representation to generate the second model data, the method further includes:

[0128] Replace the first model data in the model with the second model data;

[0129] The floating-point processing unit is invoked to perform calculations on the model.

[0130] The first model data includes model parameters and / or input data.

[0131] A floating-point processing unit (Floating-point processing unit) is a structure used to perform floating-point operations. It is typically implemented using circuitry and used in computer chips. After determining the target representation, the floating-point processing unit is designed accordingly. Therefore, in some embodiments, the method may further include:

[0132] Based on the target exponent representation, generate the hardware description program corresponding to the floating-point processing unit; the hardware description program is used to design the floating-point processing unit.

[0133] In a practical application, the technical solution of this application embodiment can be applied to the model quantization of neural network models on public clouds. The inference requirements of neural network models on public clouds are increasing, creating a need to optimize inference task performance. Currently, most neural network models on public clouds use FP16 for computation. The technical solution of this application embodiment can quantize the model data of neural network models to FP8. FP8 has lower precision and can effectively compress neural network models. However, since the numerical range of FP8 is limited, it may ultimately affect the quantization effect and model performance. The technical solution of this application embodiment improves the flexibility of model quantization operations. The target exponent representation can be selected based on the first model data to be quantized, allowing selection of the numerical range of FP8, ultimately achieving quantization to FP8.

[0134] Furthermore, the neural network model described in this application can be used to process processing tasks such as image classification, object detection, image segmentation, natural language processing (speech recognition, text recognition, etc.) or video classification. The technology in the embodiments of this application can quantize the trained model to meet the purpose of optimizing the performance of inference tasks.

[0135] As described above, the neural network model can be a multimedia data processing model, thereby improving data processing efficiency through the technical solutions of this application embodiment.

[0136] For example, the neural network model can specifically be an image processing model, and the input data in the first model data can refer to image data. This image processing model can be used for image classification, object detection, or image classification processing. Through the technical solution of this application embodiment, the image processing model can be effectively quantized, achieving compression of the image processing model to improve image processing efficiency.

[0137] In another implementation, the neural network model can specifically be a speech processing model, and the input data can be speech data. This speech processing model can be used for speech recognition or speech conversion, etc. Through the technical solution of this application embodiment, the speech processing model can be effectively quantized, achieving compression of the speech processing model to improve speech processing efficiency.

[0138] In another implementation, the neural network model can specifically be a text processing model, and the input data can be text data. This text processing model can be used for text matching, intent recognition, text conversion, text classification, etc. Through the technical solution of this application embodiment, the text processing model can be effectively quantized, achieving compression of the text processing model to improve text processing efficiency.

[0139] In another implementation, the neural network model can be a video processing model, and the input data can be video data. This video processing model can be used for video classification, etc. Through the technical solutions of this application, the video processing model can be effectively quantized, achieving compression of the video processing model to improve video processing efficiency.

[0140] like Figure 2 As shown below, taking the system architecture applicable to an implementation scenario of the technical solution of this application as an example, the technical solution of this application will be introduced. The server can determine the first model data to be quantized in the model based on the model quantization request triggered by the user, and determine the corresponding target index representation method according to the expression value of the first model data. The server 20 can be, for example, a cloud server provided by a cloud computing platform.

[0141] The target exponent method can generate a hardware description language, which can then be used to design the floating-point processing unit 21. In practical applications, this floating-point processing unit 21 can be deployed as an external device on the server.

[0142] The server 20 can quantize the first model data from the first precision to the second precision according to the target exponent representation method to generate the second model data. The server can then use the second model data to replace the first model data in the model. After that, the server can call the floating-point processing unit 21, which is designed and generated based on the target exponent method, to perform calculations on the model.

[0143] Furthermore, besides model quantization scenarios, the technical solutions of this application embodiment can also be applied to other data precision quantization scenarios, such as... Figure 3 As shown in the embodiments of this application, a data quantification method is also provided, which may include the following steps:

[0144] 301: Determine the first data to be quantized in the model.

[0145] 302: Determine the corresponding target index representation method based on the numerical value of the first data;

[0146] 303: According to the target exponent representation, the first data is quantized from the first precision to the second precision to generate the second data.

[0147] Both the first precision and the second precision correspond to floating-point types; and the exponent parts with the same actual value correspond to different exponent encoding values ​​for different exponent representation methods.

[0148] Figure 3 The illustrated embodiments and Figure 1 The difference in the illustrated embodiment is that the first data is specifically the first model data; other identical or similar steps can be found in [link to documentation].Figure 1 The embodiments described herein will not be repeated here.

[0149] The technical solution of this application embodiment can realize the flexibility of data quantization operation. According to the expression value of the first data, the corresponding target index expression method is selected for quantization, so that the data expression range corresponding to the second precision can be dynamically adjusted, and the first data can be effectively quantized to the second precision.

[0150] Figure 4 This application provides a schematic diagram of the structure of a data quantization device according to one embodiment. The device may include:

[0151] The first determining module 401 is used to determine the first data to be quantized in the model;

[0152] The second determining module 402 is used to determine the corresponding target index representation method based on the numerical value of the first data;

[0153] The quantization module 403 is used to quantize the first data from the first precision to the second precision according to the target exponent representation, so as to generate the second data; wherein, both the first precision and the second precision correspond to floating point type; wherein, the exponent part with the same actual value corresponds to different exponent encoding values ​​for different exponent representations.

[0154] In a practical application, the technical solution of this application embodiment can be applied to a model quantization scenario, where the first data can be the first model data to be quantized in the model, and finally quantized to generate the second model data.

[0155] In some embodiments, the second determining module may specifically determine the target exponent representation method for adjusting the original exponent offset value to the target exponent offset value based on the numerical expression of the first model data;

[0156] Specifically, the quantization module can quantize the first model data from the first precision to the second precision according to the target exponential offset value, so as to generate the second model data.

[0157] In some embodiments, the second determining module may specifically determine multiple index representations to adjust the original index offset value to the candidate index offset value; different index representations correspond to different candidate index offset values; according to the multiple index representations, the first model data is quantized to a second precision respectively; the precision loss corresponding to the multiple index representations is calculated respectively; and the index representation with the smallest precision loss is selected as the target index representation.

[0158] In some embodiments, the second determining module may specifically determine the target exponential representation method for representing the effective value by determining the maximum actual value of the exponential portion corresponding to the second precision based on the numerical expression of the first model data.

[0159] In some embodiments, the second determining module may specifically determine the target exponent representation method for the maximum actual value of the exponent portion corresponding to the second precision, based on the numerical expression of the first model data, to represent the effective numerical value. This includes:

[0160] In some embodiments, the second determining module may specifically determine the target exponential representation method for the maximum actual value of the exponential portion corresponding to the second precision, which is used to represent the effective value, when the first model data meets the valid conditions.

[0161] In some embodiments, the second determining module may specifically determine the target expression value corresponding to the maximum actual value when the exponent portion corresponding to the second precision is the maximum actual value; when the expression value of the first model data is less than the target expression value, the target exponent representation method that uses the maximum actual value of the exponent portion corresponding to the second precision to represent the effective value is determined.

[0162] In some embodiments, the second determining model may specifically be to determine the maximum actual value of the exponent portion corresponding to the second precision to represent the effective value and to adjust the original exponent offset value to the corresponding candidate exponent offset value in multiple exponent representation methods; to determine the expression value range corresponding to each of the multiple exponent representation methods; to determine the corresponding target expression value range based on the expression value of the first model data; and to use the exponent representation method corresponding to the target expression value range as the target exponent representation method.

[0163] In some embodiments, the quantization module quantizes the first model data from a first precision to a second precision by: determining the first precision of the first model data; determining the first exponent portion and the first mantissa portion of the first model data corresponding to the first precision; determining the exponent encoding threshold of the second exponent portion corresponding to the second precision based on the second precision; and if the exponent encoding value corresponding to the first exponent portion is less than the exponent encoding threshold, using the exponent encoding value of the first exponent portion as the exponent encoding value of the second exponent portion, and using the data of the first mantissa portion as the data of the second mantissa portion.

[0164] In some embodiments, the quantization module is further configured to, when the exponent encoding value corresponding to the first exponent portion is greater than the exponent encoding threshold, use the exponent encoding threshold as the exponent encoding value of the second exponent portion of the second model data; and to quantize the first mantissa portion according to the exponent encoding value of the first exponent portion and the exponent encoding threshold to obtain the mantissa value corresponding to the second mantissa portion.

[0165] In some embodiments, the quantization module quantizes the first mantissa portion based on the exponent code value of the first exponent portion and the exponent code threshold to obtain the mantissa value corresponding to the second mantissa portion, which may include:

[0166] The difference between the first exponent part's exponent code value and the exponent code threshold is determined, and the first mantissa part is quantized based on the difference and the base, wherein the base corresponding to the first precision is equal to the base corresponding to the second precision.

[0167] In some embodiments, the quantization module may perform quantization processing on the first mantissa based on the difference and the base, which may include: using the base as the base and the difference as the exponent, performing a power operation on the base and the difference to generate a first operation result; multiplying the data of the first mantissa with the first operation result to generate a second operation result, and using the second operation result as the data of the second mantissa.

[0168] In some embodiments, the device may further include:

[0169] The processing module is used to replace the first model data in the model with the second model data; and to call the floating-point processing unit to perform calculations on the model.

[0170] In some embodiments, the processing model is also used to generate a hardware description program corresponding to the floating-point processing unit according to the target exponent representation; the hardware description program is used to design the floating-point processing unit.

[0171] Figure 4 The data quantization device can perform Figure 1 The implementation principle and technical effects of the model quantization method described in the illustrated embodiments will not be repeated here. The specific methods by which each module and unit of the data quantization device in the above embodiments performs its operations have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0172] This application also provides a computing device, such as... Figure 5 As shown, the device may include a storage component 501 and a processing component 502;

[0173] Storage component 501 stores one or more computer instructions, wherein one or more computer instructions are invoked and executed by processing component 502 to achieve, for example, Figure 1 The model quantization method shown or such Figure 3 The data quantization method shown.

[0174] Of course, computing devices may also include other components, such as input / output interfaces, display components, and communication components. Input / output interfaces provide an interface between processing components and peripheral interface modules, which can be output devices, input devices, etc. Communication components are configured to facilitate wired or wireless communication between the computing device and other devices.

[0175] In addition, the computing device may also include a floating-point processing unit, which can be invoked to perform calculations on the quantized model.

[0176] The processing component may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0177] Storage components are configured to store various types of data to support operations on the terminal. Storage components can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0178] The display component can be an electroluminescent (EL) element, a liquid crystal display or a microdisplay with a similar structure, or a retina-direct display or a similar laser scanning display.

[0179] It should be noted that the aforementioned computing devices can be physical devices or elastic computing hosts provided by cloud computing platforms. They can be implemented as a distributed cluster of multiple servers or terminal devices, or as a single server or a single terminal device.

[0180] The aforementioned computing device can also be specifically implemented as an electronic device. An electronic device can refer to a device used by a user that has the computing, internet access, and communication functions required by the user, such as a mobile phone, tablet computer, personal computer, wearable device, etc.

[0181] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a computing device, can perform the above-described functions. Figure 1 The model quantization method of the embodiment shown or the above Figure 5 The data quantization method of the illustrated embodiment. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device.

[0182] This application also provides a computer program product, which includes a computer program carried on a computer-readable storage medium. When the computer program is executed by a computer, it can perform the functions described above. Figure 1The model quantization method of the embodiment shown or the above Figure 5 The illustrated embodiment describes a data quantization method. In such an embodiment, the computer program may be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, it performs the various functions defined in the system of this application.

[0183] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0184] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0185] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A model quantization method, characterized in that, The method comprises the following steps: determining first model data to be quantized in a model; the model comprises a multimedia data processing model, input data processed by the multimedia data processing model comprises multimedia data, and the multimedia data comprises images, texts or videos; determining a target exponential representation mode according to an expression value of the first model data; quantizing the first model data from a first precision to a second precision according to the target exponential representation mode to generate second model data; wherein the first precision and the second precision both correspond to floating point types; and wherein exponential encoding values corresponding to exponential parts of different exponential representation modes are different when the actual values of the exponential parts are the same; the step of quantizing the first model data from the first precision to the second precision comprises: determining the first precision of the first model data; determining a first exponential part and a first mantissa part of the first model data corresponding to the first precision; determining an exponential encoding threshold value of a second exponential part corresponding to the second precision according to the second precision; quantizing the first exponential part to obtain an exponential encoding value of the second exponential part and quantizing the first mantissa part to obtain a mantissa value corresponding to a second mantissa part according to a size relationship between an exponential encoding value corresponding to the first exponential part and the exponential encoding threshold value.

2. The method of claim 1, wherein, the step of determining a target exponential representation mode according to an expression value of the first model data comprises: determining a target exponential representation mode in which an original exponential offset value is adjusted to a target exponential offset value according to the expression value of the first model data; the step of quantizing the first model data from the first precision to the second precision according to the target exponential representation mode to generate second model data comprises: quantizing the first model data from the first precision to the second precision according to the target exponential offset value to generate second model data.

3. The method of claim 2, wherein, the step of determining an exponential representation mode in which an original exponential offset value is adjusted to a target exponential offset value according to an expression value of the first model data comprises: determining a plurality of exponential representation modes in which an original exponential offset value is adjusted to a candidate exponential offset value; the candidate exponential offset values corresponding to different exponential representation modes are different; quantizing the first model data to the second precision according to the plurality of exponential representation modes respectively; calculating precision losses respectively corresponding to the plurality of exponential representation modes; selecting an exponential representation mode with the minimum precision loss as a target exponential representation mode.

4. The method of claim 1, wherein, the step of determining a target exponential representation mode according to an expression value of the first model data comprises: determining a target exponential representation mode in which a maximum actual value of an exponential part corresponding to the second precision is used to represent an effective value according to the expression value of the first model data.

5. The method of claim 4, wherein, the step of determining a target exponential representation mode in which a maximum actual value of an exponential part corresponding to the second precision is used to represent an effective value according to an expression value of the first model data comprises: determining a target exponential representation mode in which a maximum actual value of an exponential part corresponding to the second precision is used to represent an effective value in a case where the first model data meets an effective condition.

6. The method of claim 1, wherein, The determining of the corresponding target index representation mode according to the expression value of the model data comprises: determining a maximum actual value of the second precision corresponding index part to represent the effective value, and adjusting the original index offset value to the corresponding candidate index offset value respectively to obtain a plurality of index representation modes; determining the expression value range corresponding to the plurality of index representation modes respectively; determining the target expression value range corresponding to the first model data; taking the index representation mode corresponding to the target expression value range as the target index representation mode.

7. The method of claim 1, wherein, The quantization processing of the first index part according to the size relationship between the index encoding value corresponding to the first index part and the index encoding threshold value, the obtaining of the index encoding value of the second index part, and the quantization processing of the first mantissa part to obtain the mantissa value corresponding to the second mantissa part comprise: in the case that the index encoding value corresponding to the first index part is less than the index encoding threshold value, taking the index encoding value of the first index part as the index encoding value of the second index part, and taking the data of the first mantissa part as the data of the second mantissa part.

8. The method of claim 1, wherein, The quantization processing of the first index part according to the size relationship between the index encoding value corresponding to the first index part and the index encoding threshold value, the obtaining of the index encoding value of the second index part, and the quantization processing of the first mantissa part to obtain the mantissa value corresponding to the second mantissa part comprise: in the case that the index encoding value corresponding to the first index part is greater than the index encoding threshold value, taking the index encoding threshold value as the index encoding value of the second index part of the second model data; quantizing the first mantissa part according to the index encoding value of the first index part and the index encoding threshold value to obtain the mantissa value corresponding to the second mantissa part.

9. The method of claim 8, wherein, The quantization processing of the first mantissa part according to the index encoding value of the first index part and the index encoding threshold value to obtain the mantissa value corresponding to the second mantissa part comprises: determining the difference between the index encoding value of the first index part and the index encoding threshold value, and quantizing the first mantissa part based on the difference and the base number, wherein the base number corresponding to the first precision is equal to the base number corresponding to the second precision.

10. The method of claim 9, wherein, The quantization processing of the first mantissa part based on the difference and the base number comprises: taking the base number as the base and the difference as the index, performing power operation on the base number and the difference to generate a first operation result; performing product operation on the data of the first mantissa part and the first operation result to generate a second operation result, and taking the second operation result as the data of the second mantissa part.

11. The method of claim 1, wherein, After quantizing the first model data from the first precision to the second precision according to the target index representation mode to generate the second model data, the method further comprises: replacing the first model data in the model with the second model data; calling the floating point processing unit to perform operation processing on the model.

12. The method of claim 11, wherein, Further comprising: According to the target index expression mode, a hardware description program corresponding to the floating point processing unit is generated; the hardware description program is used to design the floating point processing unit.

13. A computing device, comprising: The processing component and the storage component are included. The storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the model quantization method in any one of claims 1-12.

14. A computer storage medium, characterized in that The computer program is stored in the computer device and is executed to implement the model quantization method in any one of claims 1-12.

Images