Data processing method and device, storage medium, electronic equipment and chip

Abstract

The invention discloses a data processing method and device, a storage medium, electronic equipment and a chip, and relates to the technical field of data processing.The method comprises the steps that firstly, the search range generated in the PTQ stage of a model is obtained, and the preset number of iterations of a to-be-optimized layer of the model is obtained; dividing the search range based on the preset number of iterations, and determining a plurality of cut-off values according to division points; calculating measurement indexes respectively corresponding to the plurality of truncation values, and determining a target truncation value according to the measurement indexes; and finally, quantizing the activation value of the to-be-optimized layer by using the target cutoff value to obtain the activated and optimized model. According to the method, the search range generated in the PTQ stage can be divided according to the preset number of iterations, and the optimal target truncation value optimization model is automatically selected from the search range according to the measurement index, so that the time length of model optimization is shortened, the efficiency of model optimization is effectively improved, and the labor cost is reduced.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data processing method, apparatus, storage medium, electronic device, and chip. Background Art

[0002] After a deep learning model is trained, Post-Training Quantization (PTQ) can be used to convert the activation values of the deep learning model from high-precision floating-point numbers (such as 32-bit floating-point numbers, FP32) to low-precision data types (such as 8-bit integers, INT8), so as to reduce the storage space and computational burden of the model, making the model more suitable for running on hardware with limited resources (such as mobile devices and embedded devices).

[0003] Currently, usually according to the performance of the model after the PTQ stage, the quantization range of the activation values is manually adjusted for model optimization, such as relaxing or tightening the quantization range, to balance the contradiction among model size, computational efficiency, and prediction accuracy.

[0004] However, when the number of network layers of the model is too large, it is difficult to manually adjust the quantization range of the activation values, which takes a long time, resulting in low model optimization efficiency and increasing a lot of labor costs. Summary of the Invention

[0005] In view of this, this application provides a data processing method, apparatus, storage medium, electronic device, and chip, mainly aiming to improve the technical problem that when the number of network layers of the model is too large, it is difficult to manually adjust the quantization range of the activation values, which takes a long time, resulting in low model optimization efficiency and increasing a lot of labor costs.

[0006] In a first aspect, this application provides a data processing method, including:

[0007] Obtaining the search range generated by the model in the PTQ stage, and obtaining the preset number of iterations of the layer to be optimized of the model;

[0008] Dividing the search range based on the preset number of iterations, and determining a plurality of truncation values according to the division points;

[0009] Calculating the metric corresponding to each of the plurality of truncation values, and determining the target truncation value according to the metric;

[0010] Quantizing the activation values of the layer to be optimized using the target truncation value to obtain the model with optimized activation.

[0011] Optionally, the calculating the metric corresponding to each of the plurality of truncation values includes:

[0012] Quantize the activation values of the layer to be optimized using the multiple truncation values in sequence, and perform forward inference on the model to obtain the model inference results corresponding to the multiple truncation values respectively;

[0013] Calculate the metric between the model inference result and the first floating-point model inference result as the metric corresponding to the truncation value, where the first floating-point model inference result is the model result of forward inference with the activation values as floating-point types.

[0014] Optionally, the metric includes cosine similarity; determining the target truncation value based on the metric includes:

[0015] Generate a truncation value interval based on the cosine similarity;

[0016] Use binary search to search the truncation value interval to determine the target truncation value.

[0017] Optionally, generating the truncation value interval based on the cosine similarity includes:

[0018] Sort the cosine similarities to obtain the first truncation value corresponding to the maximum cosine similarity and the second truncation value corresponding to the second largest cosine similarity;

[0019] Generate the truncation value interval according to the first truncation value and the second truncation value.

[0020] Optionally, sorting the cosine similarities includes:

[0021] Obtain the PTQ truncation values generated in the PTQ stage;

[0022] Calculate the PTQ cosine similarities corresponding to the PTQ truncation values;

[0023] Sort the PTQ cosine similarities and the cosine similarities corresponding to the multiple truncation values.

[0024] Optionally, using binary search to search the truncation value interval to determine the target truncation value includes:

[0025] Use the binary search to narrow the truncation value interval until the stop search condition is met, and obtain the narrowed truncation value interval;

[0026] Determine the target truncation value according to the narrowed truncation value interval.

[0027] Optionally, the interval endpoints of the narrowed truncation value interval include a third truncation value and a fourth truncation value, and meeting the stop search condition includes at least one of the following:

[0028] The maximum cosine similarity among the cosine similarities corresponding to the third truncation value and the cosine similarities corresponding to the fourth truncation value is greater than or equal to a preset cosine similarity threshold;

[0029] The cosine similarity corresponding to the third truncation value is equal to the cosine similarity corresponding to the fourth truncation value;

[0030] The number of times of using the binary search to narrow the truncation value range is greater than or equal to a preset search number.

[0031] Optionally, before calculating the metric corresponding to the multiple truncation values and determining the target truncation value according to the metric, the method further includes:

[0032] Setting the activation bit width of the layer to be optimized to the target activation bit width, and setting the activation bit widths of other network layers in the model except the layer to be optimized to the bit widths corresponding to the floating point type.

[0033] Optionally, the method further includes:

[0034] Obtaining the preset weight bit width of the layer to be optimized;

[0035] Calculating the metric corresponding to the preset weight bit width, and determining the target weight bit width according to the metric;

[0036] Using the target weight bit width to optimize the weights of the layer to be optimized, and obtaining the model with optimized weights.

[0037] Optionally, the calculating the metric corresponding to the preset weight bit width includes:

[0038] Processing the weight values of the layer to be optimized using the preset weight bit width, and performing forward inference of the model to obtain the model inference result corresponding to the preset weight bit width;

[0039] Calculating the metric between the model inference result and the second floating-point model inference result as the metric corresponding to the preset weight bit width, where the second floating-point model inference result is the model result of forward inference with the weight values as the floating point type.

[0040] Optionally, the determining the target weight bit width according to the metric includes:

[0041] Judging whether the metric corresponding to the preset weight bit width is greater than or equal to a metric threshold;

[0042] If the metric corresponding to the preset weight bit width is greater than or equal to the metric threshold, determining the preset weight bit width as the target weight bit width.

[0043] Second aspect, the present application provides a data processing device, including:

[0044] An acquisition module, configured to acquire the search range generated by the model in the PTQ phase, and acquire the preset number of iterations corresponding to the layer to be optimized of the model;

[0045] A determination module, configured to divide the search range based on the preset number of iterations, and determine a plurality of truncation values according to the division points;

[0046] A determination module, configured to calculate the metric indexes corresponding to the plurality of truncation values respectively, and determine the target truncation value according to the metric indexes;

[0047] An acquisition module, configured to perform quantization processing on the activation values of the layer to be optimized using the target truncation value, and obtain the model with optimized activation.

[0048] Third aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the data processing method described in the first aspect is implemented.

[0049] Fourth aspect, the present application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, and when the processor executes the computer program, the data processing method described in the first aspect is implemented.

[0050] Fifth aspect, the present application provides a chip, including one or more interface circuits and one or more processors; the interface circuit is used to receive signals from the memory of the electronic device and send the signals to the processor, and the signals include computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device is enabled to execute the data processing method described in the first aspect.

[0051] With the above technical solution, the present application provides a data processing method, apparatus, storage medium, electronic device and chip. Specifically, first, obtain the search range generated by the model in the PTQ stage, and obtain the preset number of iterations of the layer to be optimized of the model; then divide the search range based on the preset number of iterations, and determine a plurality of cut-off values according to the division points; then calculate the metric indexes corresponding to the plurality of cut-off values respectively, and determine the target cut-off value according to the metric indexes; finally, use the target cut-off value to perform quantization processing on the activation values of the layer to be optimized, and obtain the model with optimized activation. The present application can divide the search range generated in the PTQ stage according to the preset number of iterations, and then automatically select a better target cut-off value from the search range according to the metric indexes to optimize the model. Compared with the method of manually adjusting the quantization range of the activation values in the current existing technology, the present application reduces the duration of model optimization, effectively improves the efficiency of model optimization, and reduces the labor cost.

[0052] The above description is only an overview of the technical solution of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present application more obvious and understandable, the following specifically describes the specific embodiments of the present application. Brief Description of the Drawings

[0053] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0055] Figure 1 Shows a schematic flowchart of a data processing method provided by an embodiment of the present application;

[0056] Figure 2 Shows a schematic flowchart of another data processing method provided by an embodiment of the present application;

[0057] Figure 3 Shows a schematic flowchart of an example provided by an embodiment of the present application;

[0058] Figure 4 Shows a schematic flowchart of an example provided by an embodiment of the present application;

[0059] Figure 5 Shows a schematic flowchart of an example provided by an embodiment of the present application;

[0060] Figure 6 The structural schematic diagram of a data processing device provided by an embodiment of the present application is shown. Specific embodiments

[0061] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other.

[0062] In order to improve the current technical problems that when the number of network layers of a model is too large, it is difficult to manually adjust the quantization range of activation values, which takes a long time, resulting in low model optimization efficiency and increasing a large amount of labor costs. This embodiment provides a data processing method, as Figure 1 shown, the method includes:

[0063] Step 101, obtain the search range generated by the model in the PTQ stage, and obtain the preset number of iterations of the layer to be optimized of the model.

[0064] For the execution subject of this embodiment, it can be a data processing device or equipment, which can be configured on the terminal side. The terminal can include but is not limited to a mobile terminal, a cloud, etc. Exemplarily, when configured on a smart watch, due to the limited computing power provided by the smart watch, the PTQ method can be used to compress the model. According to the preset number of iterations of the layer to be optimized, the search range generated in the PTQ stage is further optimized, and a more optimal activation truncation value is selected, so as to automatically obtain a model that meets the model accuracy requirements and occupies less memory space, and improve the model calculation speed. Among them, the model can include but is not limited to machine learning models, deep learning models, etc., for example: large language models (such as: LLaMA), detection models, recognition models, classification models, segmentation models, etc.; the PTQ method can include but is not limited to linear quantization, perceptual quantization, etc.; the search range can be used to represent the maximum value of the set truncation value search range, which can be obtained by moderately amplifying the maximum truncation value calculated layer by layer in the PTQ stage; the preset number of iterations can be used to divide the search range. The more the preset number of iterations, the smaller the interval between truncation values, the finer the data, and the easier it is to approach the optimal truncation value; the layer to be optimized can be a network layer that needs to optimize activation values or weight values in the model, and can be one or more layers.

[0065] Step 102, divide the search range based on the preset number of iterations, and determine multiple truncation values according to the division points.

[0066] For example, when the search range is 1024 and the preset number of iterations is 128, 1024 is divided into 128 parts, 128 division points are obtained, the division point interval is 8, adjacent division points can form an optimization interval, and then the truncation values are automatically determined according to the division points, such as 8, 16, etc., to improve the model optimization efficiency.

[0067] Step 103: Calculate the metric indicators corresponding to multiple truncation values, and determine the target truncation value based on the metric indicators.

[0068] In this embodiment, the optimal truncation value can be determined based on the metric indicators. Multiple truncation values can be used in sequence to truncate the activation values of the layer to be optimized, and the truncated activation values can be used for forward inference of the model to obtain the inference result. Then, calculate the metric indicator between the inference result and the inference result without model optimization. Finally, determine the target truncation value based on the metric indicator to automatically select a better target truncation value to optimize the model and improve the model accuracy. Among them, the metric indicator can be used to measure the model accuracy after optimizing the layer to be optimized using the current truncation value, and can include but are not limited to indicators for calculating similarity, such as cosine similarity, mean absolute error (L1 Loss), mean square error (L2 Loss), etc.; the target truncation value can be the truncation value corresponding to the layer to be optimized that is finally determined to optimize the model.

[0069] Step 104: Use the target truncation value to perform quantization processing on the activation values of the layer to be optimized to obtain an optimized model with activation.

[0070] Exemplarily, when there are multiple layers to be optimized, the method of this embodiment can be automatically used for layer-by-layer optimization to determine the target truncation value corresponding to each layer to be optimized, and use the target truncation value to perform quantization processing on the activation values of each layer to be optimized. The quantization processing process can include truncating the activation values using the target truncation value, and quantizing the activation values according to the data type (for example: signed integer int, unsigned integer uint) and data bit width (for example: 8bit, 16bit) to obtain a model that meets the accuracy requirements and improves the calculation rate at the same time. In this way, the efficiency of model optimization is effectively improved and the labor cost is reduced.

[0071] This embodiment first obtains the search range generated during the PTQ stage of the model and the preset number of iterations of the layer to be optimized of the model; then divides the search range based on the preset number of iterations and determines multiple truncation values according to the division points; then calculates the metric indicators corresponding to the multiple truncation values and determines the target truncation value based on the metric indicators; finally, uses the target truncation value to perform quantization processing on the activation values of the layer to be optimized to obtain an optimized model with activation. This embodiment can divide the search range generated during the PTQ stage according to the preset number of iterations, and then automatically select a better target truncation value from the search range to optimize the model according to the metric indicator. Compared with the current method of manually adjusting the quantization range of activation values in the prior art, this embodiment reduces the duration of model optimization, effectively improves the efficiency of model optimization, and reduces the labor cost.

[0072] Based on the technical implementation content shown in the above embodiments, in order to further illustrate the specific implementation process of the method in this embodiment, this embodiment provides a specific method as shown in Figure 2 which includes:

[0073] Step 201, obtain the search range generated by the model in the PTQ stage, and obtain the preset number of iterations of the layer to be optimized of the model.

[0074] Exemplarily, as shown in Figure 3 , a schematic diagram of a network environment in which this technical solution can be implemented is shown. When receiving an instruction to perform wireless transmission using a cellular circuit, stop the wifi connection, and when receiving an instruction to stop performing wireless transmission using a cellular circuit, resume the wifi connection. The network environment may include a notebook, a smart watch, a WIFI access point, a cellular base station (base station), and a network. When deploying a large language model for mobile devices in the network environment, it is necessary to reduce the computing power requirements of the model.

[0075] For example, taking a smart watch as an example, since the test data used in the PTQ stage is limited and cannot guarantee the accuracy when inferring with other test data, it is usually necessary to appropriately enlarge the search range. The search range abs_max can be set to 1024 according to experience and the maximum value generated by the model in the PTQ stage.

[0076] Step 202, divide the search range based on the preset number of iterations, and determine multiple truncation values according to the division points.

[0077] Among them, the preset number of iterations iter_num can be 128. The iteration number i can be set to increase from 1 to 128. Using abs_max / iter_num as the division point and i*(abs_max / iter_num) as a truncation value amax, multiple truncation values can be obtained. The interval between adjacent truncation values can be called an optimization interval, that is, iter_num optimization intervals are generated, and the optimal truncation value will be generated within a certain optimization interval.

[0078] Step 203, sequentially use multiple truncation values to perform quantization processing on the activation values of the layer to be optimized, and perform model forward inference to obtain the model inference results corresponding to the multiple truncation values respectively.

[0079] Optionally, before calculating the metric indicators corresponding to the multiple truncation values and determining the target truncation value according to the metric indicators, this embodiment can also set the activation bit width of the layer to be optimized to the target activation bit width, and set the activation bit widths of other network layers in the model except the layer to be optimized to the bit widths corresponding to the floating-point type.

[0080] In this embodiment, the activation value bit width of one or more layers in the model can be adjusted according to the actual project requirements, the target activation bit width and the activation data type can be set. The target activation bit width can be any bit width, and the activation data type can be any data type. Then, the model is initialized according to the target activation bit width. When it is necessary to obtain a model with activation mixed precision, different target activation bit widths can be set for different layers to be optimized, so as to realize the optimization of the model with activation mixed precision, meet the project requirements and improve the model calculation efficiency. Exemplarily, for sensitive convolutional layers or fully connected layers, a higher activation bit width (such as 8 bits) can be retained to maintain accuracy, while for other layers or certain channels, a lower activation bit width (such as 4 bits or lower) can be used for quantization processing to achieve mixed precision optimization.

[0081] In this embodiment, when optimizing a certain layer to be optimized, only the quantization processing of this layer can be turned on, and the quantization processing of the remaining layers in the network (the remaining layers are of floating-point type) can be turned off. After quantization processing, a forward inference is performed once to obtain the model inference result. Among them, the model inference result can be the inference result obtained when only the layer to be optimized is turned on for quantization processing and the remaining layers in the model are of floating-point type during forward inference.

[0082] Exemplarily, the truncation value of this layer can be updated first according to the truncation value corresponding to the number of iterations (Update amax(amax = i * abs_max / iter_num)), and the activation value input to the current layer to be optimized is truncated using the updated truncation value amax, that is, the activation value greater than amax is updated to amax. Then, based on the target activation bit width and the activation data type, the quantization ratio is calculated. The calculation formula of the quantization ratio can be expressed as:

[0083] scale = bit_max_value / amax

[0084] In the formula, scale is the quantization ratio, and bit_max_value can be the maximum value of the value range corresponding to the target data type. For example, when the target activation bit width is 8 bits and the activation data type is signed integer int, it can be expressed as int8, and the maximum value of the value range corresponding to int8 is 127; when the target activation bit width is 8 bits and the activation data type is unsigned integer uint, it can be expressed as uint8, and the maximum value of the value range corresponding to uint8 is 255.

[0085] Correspondingly, according to the activation value calculation formula, the current activation value is converted into the activation value corresponding to the target activation bit width and the activation data type. The activation value calculation formula can be expressed as:

[0086] y = round(x * scale)

[0087] Where x can be the activation value of the current layer to be optimized, round is the rounding function, and y is the activation value obtained after quantization processing.

[0088] Step 204: Calculate the metric between the model inference result and the first floating-point model inference result as the metric corresponding to the truncation value. The first floating-point model inference result is the model result of forward inference with the activation value as the floating-point type. The metric includes cosine similarity.

[0089] In this embodiment, after forward inference using each truncation value, the model inference result can be compared with the first floating-point model inference result, and the metric corresponding to each truncation value can be calculated. The accuracy of the model inference result can be measured according to the metric. The first floating-point model inference result can be the forward inference model result when the activation values in the model are all of the floating-point type and no quantization processing is performed.

[0090] Exemplarily, the cosine similarity between the model inference result and the first floating-point model inference result (floating-point - quantization processing cosine similarity cos-sim) can be calculated, and the model accuracy can be measured according to the cosine similarity.

[0091] It should be noted that in the following steps, the cosine similarity is taken as an example for explanation, and other metrics that can calculate the model similarity can also be used.

[0092] Step 205: Generate a truncation value interval based on the cosine similarity.

[0093] Optionally, step 205 may specifically include: sorting the cosine similarities, obtaining the first truncation value corresponding to the maximum cosine similarity and the second truncation value corresponding to the second largest cosine similarity; generating a truncation value interval according to the first truncation value and the second truncation value.

[0094] In this embodiment, the higher the cosine similarity, the higher the accuracy of the model truncated using the truncation value corresponding to the cosine similarity. Generating a truncation value interval based on the first truncation value corresponding to the maximum cosine similarity and the second truncation value corresponding to the second largest cosine similarity helps to narrow the search range and improve the model optimization efficiency.

[0095] Further optionally, sorting the cosine similarities includes: obtaining the PTQ truncation values generated in the PTQ stage; calculating the PTQ cosine similarities corresponding to the PTQ truncation values; sorting the PTQ cosine similarities and the cosine similarities corresponding to multiple truncation values.

[0096] Among them, the PTQ truncation value can be the PTQ truncation value obtained by the model in the PTQ stage, and can be determined according to the PTQ method applied in the PTQ stage. For example, if two PTQ methods are adopted, 2 PTQ truncation values will be obtained. In this embodiment, since the PTQ truncation value may be a local optimal solution, the PTQ cosine similarity corresponding to the PTQ truncation value can be calculated, and the PTQ cosine similarity is sorted with the cosine similarities corresponding to multiple truncation values in step 204, so as to obtain a better truncation value, thereby improving the model optimization effect.

[0097] Step 206: Use binary search to search the truncation value interval to determine the target truncation value.

[0098] Optionally, step 206 may specifically include: using binary search to narrow the truncation value interval until the stop search condition is met, and obtaining the narrowed truncation value interval; determining the target truncation value according to the narrowed truncation value interval.

[0099] Optionally, the interval endpoints of the narrowed truncation value interval may include a third truncation value and a fourth truncation value. Meeting the stop search condition may include at least one of the following: the maximum cosine similarity among the cosine similarities corresponding to the third truncation value and the fourth truncation value is greater than or equal to the preset cosine similarity threshold; the cosine similarity corresponding to the third truncation value is equal to the cosine similarity corresponding to the fourth truncation value; the number of times of using binary search to narrow the truncation value interval is greater than or equal to the preset search times.

[0100] Among them, the third truncation value and the fourth truncation value can be the interval endpoints for narrowing the truncation value interval by binary search; the preset cosine similarity threshold can be the cosine similarity threshold set to meet the accuracy requirement, such as 0.9999; the preset search times can be the search times for binary search, which can be adjusted according to different quantization bit numbers, and usually can be set to 12 - 16. The accuracy reaches the decimal millionth place at the 16th iteration. Calculating the truncation value with this accuracy usually has no further impact on the 8-bit and 16-bit quantization accuracies, that is, further increasing the decimal places of the truncation value will not bring about an improvement in accuracy.

[0101] Exemplarily, the first truncation value and the second truncation value of the truncation value interval can be used as the optimization boundaries, the first truncation value as the upper optimization boundary, the second truncation value as the lower optimization boundary, the median of the first truncation value and the second truncation value as the truncation value and calculate the cosine similarity, select the larger value of the first truncation value and the second truncation value as the updated upper optimization boundary (the third truncation value), and update the lower optimization boundary (the fourth truncation value) corresponding to the median of the first truncation value and the second truncation value to form a new optimization boundary, and perform the next iteration until the stop search condition is met, and so on, layer by layer to optimize the activation layer.

[0102] Among them, meeting the stop search condition may include: if the maximum cosine similarity among the cosine similarities corresponding to the third truncation value and the fourth truncation value is greater than or equal to the preset cosine similarity threshold (max(cos-sim) >= thres), the truncation value corresponding to the maximum cosine similarity can be used as the target truncation value; and / or, if the cosine similarities corresponding to the third truncation value and the fourth truncation value are equal (cos_top == cos_bottom, and in further calculations, the cosine similarity of the median value as the truncation value is equal to the upper and lower boundaries, and there is no need for further calculation), the larger truncation value among the third truncation value and the fourth truncation value can be returned as the target truncation value; and / or, the number of times of using binary search to narrow the truncation value interval is greater than or equal to the preset search times (j >= iter_opt).

[0103] Exemplarily, as Figure 4 shown, the process of determining the truncation value of a layer in the model is shown. First, the model can be initialized according to the target activation bit width of the layer to be optimized. Then, the search range is divided according to the preset number of iterations, an optimization interval is generated, and the PTQ truncation values are compared together according to the cosine similarity to determine the truncation value interval and narrow the search range. Then, binary search is used to finely search the truncation value interval and update the upper and lower boundaries of the truncation value interval.

[0104] When the layer to be optimized includes multiple layers, the truncation values can be generated layer by layer according to the Figure 4 process flow. The activation values are quantized respectively according to the truncation values of each layer to obtain the optimized model. For example, the target activation bit width of the first layer in the layer to be optimized can be set to 8bit, and the optimization is carried out according to the Figure 4 process flow. After the optimization is completed, the second layer in the layer to be optimized is optimized. The target activation bit width of the second layer in the layer to be optimized can be set to 16bit, and the optimization is carried out according to the Figure 4 process flow, so as to realize the mixed-precision optimization of the model.

[0105] When the stop search condition is met, it is further determined whether the maximum cosine similarity corresponding to the upper and lower boundaries of the current truncation value interval is greater than or equal to the preset cosine similarity threshold. If the maximum cosine similarity returned by the above search fails to meet the threshold requirement, the cosine similarities corresponding to the upper and lower boundaries of the truncation value interval are equal, or the activation bit width is 16bit, then this layer can be updated with 16bit, the PTQ truncation value is restored, and the above process is repeated for optimization; if the maximum cosine similarity returned by the above search meets the threshold requirement, the optimization of this layer can be ended.

[0106] Step 207: Quantize the activation values of the layer to be optimized using the target truncation value to obtain an optimized model of the activation.

[0107] In this embodiment, the activation values of multiple layers to be optimized can be quantized according to multiple target truncation values to obtain activation values with different activation bit widths, thereby achieving the optimization of the model with mixed precision.

[0108] Optionally, the method of this embodiment may further include: obtaining the preset weight bit width of the layer to be optimized; calculating the metric corresponding to the preset weight bit width, and determining the target weight bit width according to the metric; using the target weight bit width to optimize the weight of the layer to be optimized to obtain a model with optimized weights.

[0109] In this embodiment, the weight bit widths of one or more layers in the model can be adjusted according to the actual project requirements, the preset weight bit width can be set, and the model can be initialized according to the preset weight bit width. When a model with weight mixed precision is required, different preset weight bit widths can be set for different layers to be optimized, so as to achieve the optimization of the model with weight mixed precision, meet the project requirements, and improve the model calculation efficiency. Among them, the preset weight bit width and the target weight bit width can be any bit for mixed precision optimization.

[0110] Optionally, calculating the metric corresponding to the preset weight bit width includes: processing the weight values of the layer to be optimized using the preset weight bit width and performing forward inference on the model to obtain the model inference result corresponding to the preset weight bit width; calculating the metric between the model inference result and the second floating-point model inference result as the metric corresponding to the preset weight bit width, where the second floating-point model inference result is the model result obtained by performing forward inference with the weight values as floating-point types.

[0111] Optionally, determining the target weight bit width according to the metric includes: determining whether the metric corresponding to the preset weight bit width is greater than or equal to the metric threshold; if the metric corresponding to the preset weight bit width is greater than or equal to the metric threshold, then determining the preset weight bit width as the target weight bit width.

[0112] Among them, the preset weight bit width can be the weight bit width initialized for the layer to be optimized, the target weight bit width can be the weight bit width finally determined for the layer to be optimized, and the metric threshold can be the threshold that meets the model accuracy requirements. It should be noted that when determining the target weight bit width, the minmax quantization method can be used based on the weights, and there is no need to search for truncation values. When other quantization methods (such as quantization by channel, quantization by layer) are used for weight quantization and truncation values need to be calculated, the method in steps 201 to 206 can also be used to determine the target truncation value, and then the weights can be quantized according to the target truncation value.

[0113] Exemplarily, such as Figure 5As shown, the process of determining the weights of a layer in a model is presented. When the weights of the model are quantized using 8-bit minmax quantization, the cosine similarity of 99.56% of the layers is greater than 0.9999, and only one layer has a cosine similarity of 0.99989. Based on this premise, mixed-precision optimization of the weights can be performed. The weight optimization can be initialized with 4 bits. If the cosine similarity corresponding to the 4 bits does not meet the cosine similarity threshold requirement, the weights can be updated to 8 bits. If the cosine similarity corresponding to the 4 bits meets the cosine similarity threshold requirement, the weights can be updated to 4 bits.

[0114] Exemplarily, as shown in Table 1, the test results obtained by quantizing a large language model are presented. In the table, Origin is the test result of the large language model based on the truncation value calculated in the PTQ stage, which may include Perplexity (ppl) and cos-sim. Optimize is the test result of the large language model after optimizing the truncation value generated in the PTQ stage using the method of this embodiment, which may include ppl, cos-sim, and average bits (avg bits). For example, when the model is optimized with mixed precision, the average bits of the calculated weights can be 4 (W4), and the average bits of the activations can be 12.99 (A12.99). It should be noted that the test results can be calculated according to the actual model and project requirements, and other metrics in the metrics can be used to measure the model accuracy. If the tested model is not a large language model, then ppl does not need to be calculated.

[0115] Table 1

[0116]

[0117] By applying the method of this embodiment, it is possible to adjust the activation value bitwidth and weight bitwidth of one or more layers in the model according to the actual project requirements, obtain a model that meets the accuracy requirements and reduces the hardware resource requirements, achieve mixed-precision model optimization, save the occupied memory size, and improve the model calculation efficiency.

[0118] Further, as a Figures 1 to 2 specific implementation of the method shown, this embodiment provides a data processing device, as Figure 6 shown, the device includes: an acquisition module 31, a determination module 32, and a judgment module 33.

[0119] The acquisition module 31 is configured to acquire the search range generated by the model in the PTQ stage and acquire the preset number of iterations of the layer to be optimized in the model;

[0120] The determination module 32 is configured to divide the search range based on the preset number of iterations and determine multiple truncation values according to the division points;

[0121] A determination module 32, configured to calculate metrics corresponding to multiple truncation values respectively, and determine a target truncation value based on the metrics.

[0122] An acquisition module 31, which uses the target truncation value to perform quantization processing on the activation values of the layer to be optimized, and obtains an optimized activation model.

[0123] In some examples of this embodiment, the determination module 32 is specifically configured to sequentially use multiple truncation values to perform quantization processing on the activation values of the layer to be optimized, and perform forward inference of the model to obtain model inference results corresponding to the multiple truncation values respectively; calculate the metric between the model inference result and the first floating-point model inference result as the metric corresponding to the truncation value, where the first floating-point model inference result is the model result of forward inference with the activation value as the floating-point type.

[0124] In some examples of this embodiment, the metric includes cosine similarity; the determination module 32 is specifically configured to generate a truncation value interval based on the cosine similarity; use binary search to search the truncation value interval to determine the target truncation value.

[0125] In some examples of this embodiment, the determination module 32 is specifically configured to sort the cosine similarities, and obtain a first truncation value corresponding to the maximum cosine similarity and a second truncation value corresponding to the second largest cosine similarity; generate a truncation value interval according to the first truncation value and the second truncation value.

[0126] In some examples of this embodiment, the determination module 32 is specifically configured to obtain the PTQ truncation value generated in the PTQ stage; calculate the PTQ cosine similarity corresponding to the PTQ truncation value; sort the PTQ cosine similarity and the cosine similarities corresponding to the multiple truncation values.

[0127] In some examples of this embodiment, the determination module 32 is specifically configured to use binary search to narrow the truncation value interval until the stop search condition is met, and obtain the narrowed truncation value interval; determine the target truncation value according to the narrowed truncation value interval.

[0128] In some examples of this embodiment, the interval endpoints of the narrowed truncation value interval include a third truncation value and a fourth truncation value, and the stop search condition includes at least one of the following: the maximum cosine similarity between the cosine similarity corresponding to the third truncation value and the cosine similarity corresponding to the fourth truncation value is greater than or equal to a preset cosine similarity threshold; the cosine similarity corresponding to the third truncation value is equal to the cosine similarity corresponding to the fourth truncation value; the number of times of using binary search to narrow the truncation value interval is greater than or equal to a preset search number.

[0129] In some examples of this embodiment, the determining module 32 is specifically configured to set the activation bitwidth of the layer to be optimized to the target activation bitwidth, and set the activation bitwidths of other network layers in the model except the layer to be optimized to the bitwidth corresponding to the floating-point type.

[0130] In some examples of this embodiment, the determining module 32 is specifically configured to obtain the preset weight bitwidth of the layer to be optimized; calculate the metric corresponding to the preset weight bitwidth, and determine the target weight bitwidth based on the metric; use the target weight bitwidth to optimize the weights of the layer to be optimized, and obtain the model with optimized weights.

[0131] In some examples of this embodiment, the determining module 32 is specifically configured to process the weight values of the layer to be optimized using the preset weight bitwidth, and perform forward inference of the model to obtain the model inference result corresponding to the preset weight bitwidth; calculate the metric between the model inference result and the second floating-point model inference result as the metric corresponding to the preset weight bitwidth, where the second floating-point model inference result is the model result obtained by performing forward inference with the weight values as the floating-point type.

[0132] In some examples of this embodiment, the determining module 32 is specifically configured to determine whether the metric corresponding to the preset weight bitwidth is greater than or equal to the metric threshold; if the metric corresponding to the preset weight bitwidth is greater than or equal to the metric threshold, then determine the preset weight bitwidth as the target weight bitwidth.

[0133] It should be noted that for other corresponding descriptions of each functional unit involved in the data processing device provided in this embodiment, reference can be made to Figures 1 to 2 the corresponding description in, which will not be elaborated here.

[0134] Based on the method as described above Figures 1 to 2 shown, correspondingly, this embodiment also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method as described above Figures 1 to 2 shown is implemented.

[0135] Based on such an understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods in various implementation scenarios of this application.

[0136] Based on the method as described above Figures 1 to 2 shown, and Figure 6In the virtual device embodiment shown, to achieve the above object, an embodiment of the present application further provides an electronic device, such as intelligent terminals like smart phones, smart watches, smart bracelets, tablet computers, drones, smart robots, etc. The device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the above as Figures 1 to 2 the method shown.

[0137] Optionally, the above entity device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, sensors, an audio circuit, a WI-FI module, etc. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), etc. Optionally, the user interface may further include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0138] Those skilled in the art can understand that the above structure of the entity device provided in this embodiment does not constitute a limitation on the entity device, and may include more or fewer components, or combine certain components, or arrange different components.

[0139] The storage medium may further include an operating system and a network communication module. The operating system is a program for managing the hardware and software resources of the above entity device, supporting the information processing program and the operation of other software and / or programs. The network communication module is used to implement communication between components inside the storage medium, and communication between other hardware and software in the information processing entity device.

[0140] Based on the above as Figures 1 to 2 the method shown, and Figure 6 the virtual device embodiment shown, this embodiment further provides a chip, including one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from the memory of the electronic device and send the signal to the processor, and the signal includes computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device is enabled to execute the above as Figures 1 to 2 the method shown.

[0141] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or can also be implemented by hardware. By applying the solution of this embodiment, compared with the current prior art, this embodiment can divide the search range generated in the PTQ stage according to the preset number of iterations, and then automatically select a better target truncation value from the search range to optimize the model, reducing the duration of model optimization, effectively improving the efficiency of model optimization, and reducing the labor cost.

[0142] It should be noted that in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0143] The above are only specific embodiments of the present application, enabling those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments described herein, but rather to the broadest scope consistent with the principles and novel features claimed herein.

Claims

1. A data processing method, characterized in that, Including: Obtaining the search range generated during the post-training quantization PTQ stage of the model, and obtaining the preset number of iterations for the layer to be optimized of the model; Dividing the search range based on the preset number of iterations, and determining multiple truncation values according to the division points; Calculating the metric indicators corresponding to the multiple truncation values respectively, and determining the target truncation value according to the metric indicators; Quantizing the activation values of the layer to be optimized using the target truncation value to obtain the model with optimized activations.

2. The method according to claim 1, characterized in that The calculating the metric indicators corresponding to the multiple truncation values respectively includes: Successively quantizing the activation values of the layer to be optimized using the multiple truncation values, and performing forward inference of the model to obtain the model inference results corresponding to the multiple truncation values respectively; Calculating the metric indicator between the model inference result and the first floating-point model inference result as the metric indicator corresponding to the truncation value, where the first floating-point model inference result is the model result of forward inference with the activation values as floating-point types.

3. The method according to claim 1, characterized in that The metric indicator includes cosine similarity; The determining the target truncation value according to the metric indicators includes: Generating a truncation value interval according to the cosine similarity; Using binary search to search the truncation value interval to determine the target truncation value.

4. The method according to claim 3, wherein The generating a truncation value interval according to the cosine similarity includes: Sorting the cosine similarities, and obtaining the first truncation value corresponding to the maximum cosine similarity and the second truncation value corresponding to the second largest cosine similarity; Generating the truncation value interval according to the first truncation value and the second truncation value.

5. The method according to claim 4, characterized in that, The sorting the cosine similarities includes: Obtaining the PTQ truncation value generated during the PTQ stage; Calculating the PTQ cosine similarity corresponding to the PTQ truncation value; Sorting the PTQ cosine similarity and the cosine similarities corresponding to the multiple truncation values.

6. The method according to claim 3, wherein The using binary search to search the truncation value interval to determine the target truncation value includes: Using the binary search to narrow the truncation value interval until the stop search condition is met, and obtaining the narrowed truncation value interval; Determining the target truncation value according to the narrowed truncation value interval.

7. The method according to claim 6, wherein The interval endpoints of the narrowed truncation value interval include a third truncation value and a fourth truncation value, and the meeting the stop search condition includes at least one of the following: The maximum cosine similarity among the cosine similarities corresponding to the third truncation value and the cosine similarities corresponding to the fourth truncation value is greater than or equal to a preset cosine similarity threshold; The cosine similarity corresponding to the third truncation value is equal to the cosine similarity corresponding to the fourth truncation value; The number of times of using the binary search to narrow the truncation value interval is greater than or equal to a preset search number.

8. The method according to claim 1, wherein Before the calculating the metric indicators corresponding to the multiple truncation values and determining the target truncation value according to the metric indicators, the method further includes: Setting the activation bit width of the layer to be optimized to the target activation bit width, and setting the activation bit widths of other network layers in the model except the layer to be optimized to the bit widths corresponding to the floating-point type.

9. The method according to any one of claims 1 to 11, characterized in that, The method further includes: Obtaining a preset weight bit width of the layer to be optimized; Calculating a metric corresponding to the preset weight bit width, and determining a target weight bit width based on the metric; Using the target weight bit width to optimize the weights of the layer to be optimized, and obtaining the model with optimized weights.

10. The method according to claim 9, wherein The calculating the metric corresponding to the preset weight bit width includes: Processing the weight values of the layer to be optimized using the preset weight bit width, and performing forward inference of the model to obtain a model inference result corresponding to the preset weight bit width; Calculating a metric between the model inference result and a second floating-point model inference result as the metric corresponding to the preset weight bit width, where the second floating-point model inference result is the model result obtained by performing forward inference with the weight values as floating-point types.

11. The method according to claim 9, wherein The determining the target weight bit width based on the metric includes: Judging whether the metric corresponding to the preset weight bit width is greater than or equal to a metric threshold; If the metric corresponding to the preset weight bit width is greater than or equal to the metric threshold, determining the preset weight bit width as the target weight bit width.

12. A data processing device, characterized in that, including: An obtaining module configured to obtain a search range generated by the model in the PTQ stage, and obtain a preset number of iterations corresponding to the layer to be optimized of the model; A determining module configured to divide the search range based on the preset number of iterations, and determine a plurality of truncation values according to the division points; A determining module configured to calculate metrics corresponding to the plurality of truncation values respectively, and determine a target truncation value based on the metrics; An obtaining module configured to perform quantization processing on the activation values of the layer to be optimized using the target truncation value, and obtain the model with optimized activations.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 11.

14. An electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 11.

15. A chip, characterized in that, including one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processors, and the signals include computer instructions stored in the memory; when the processors execute the computer instructions, the electronic device is caused to execute the method according to any one of claims 1 to 11.

Images