Quantization method of machine learning model, data processing method and related device

Abstract

The invention discloses a quantification method of a machine learning model, a data processing method and a related device, and the method comprises the steps: carrying out the simulation reasoning of first data through a machine learning model, so as to execute N simulation iterations; dividing the N rounds of iteration into M iteration groups based on the N rounds of simulation iteration, and determining a corresponding first quantization parameter for each iteration group in the M iteration groups; wherein the first quantization parameters corresponding to the M iteration groups are different; m is greater than 1 and less than or equal to N; the first quantization parameter corresponding to the M iteration groups is used for performing quantization processing by using the first quantization parameter corresponding to the first iteration group in the M iteration groups when the ith round of iteration is executed in the process of reasoning the second data by the target model; the target model is obtained based on a machine learning model; the first iteration group comprises the ith round of iteration. According to the method, the quantification precision of the model in multi-round iteration can be ensured, and the precision loss of model reasoning is reduced.

Description

Technical Field

[0001] This application relates to the fields of computer and deep learning technology, and in particular to a quantization method, data processing method and related apparatus for a machine learning model. Background Technology

[0002] As artificial intelligence transitions from the era of small models in traditional deep learning to the era of large models, deploying billions of parameters on resource-constrained edge devices presents significant challenges, particularly in terms of memory usage and inference speed. To address these challenges, model quantization has been extensively explored and has become a commonly used deep learning model optimization technique. Static quantization is one of the mainstream quantization approaches, where the quantization parameters are determined before execution on the edge and can be directly used to map the corresponding data to a lower-bit representation. However, because the quantization parameters are fixed, static quantization cannot adjust them in real time based on the input. For scenarios where the model input changes, quantization accuracy cannot be guaranteed, leading to a significant loss of model inference accuracy and a deterioration in the quality of the model output. Summary of the Invention

[0003] This application provides a quantization method, data processing method, and related apparatus for machine learning models. It can determine multiple sets of different quantization parameters for input multi-round iterative models. When the model is inferring, it can select the quantization parameters corresponding to its iteration group based on the current iteration for quantization processing. This helps to ensure the quantization accuracy of the model in multi-round iterations and reduce the accuracy loss of model inference.

[0004] In a first aspect, embodiments of this application provide a quantization method for a machine learning model. This machine learning model requires N iterations for inference, where the output of the i-th iteration is the input of the (i+1)-th iteration; where N is greater than 1; and i is greater than or equal to 1 and less than N. This method can be applied to a first electronic device or a chip within the first electronic device. Taking the application of this method to a first electronic device as an example, the method includes:

[0005] The machine learning model is used to perform simulation inference on the first data to execute N rounds of simulation iterations;

[0006] Based on N rounds of simulation iterations, the N rounds of iterations are divided into M iteration groups, and a corresponding first quantization parameter is determined for each of the M iteration groups; wherein, the first quantization parameters corresponding to the M iteration groups are different; M is greater than 1 and less than or equal to N; the first quantization parameters corresponding to the M iteration groups are used in the process of the target model inferring the second data, and the first quantization parameters corresponding to the first iteration group in the M iteration groups are used for quantization processing when the i-th iteration is executed; the target model is obtained based on the machine learning model; the first iteration group includes the i-th iteration.

[0007] As can be seen in this embodiment, the first electronic device can use a machine learning model to perform simulated inference on the first data. Since the input of each iteration in the real inference scenario of the machine learning model is different, the input of the N simulation iterations in the simulated inference is also different. Based on the N simulation iterations with different inputs, the N iterations to be executed in the real inference scenario of the machine learning model are divided into M iteration groups, and then a corresponding quantization parameter (i.e., the first quantization parameter) is determined for each iteration group. The quantization parameters corresponding to the M iteration groups are different, and the quantization parameter corresponding to each iteration group is more in line with the distribution of the input data of that iteration group. When the hardware platform uses the machine learning model or the quantization model of the machine learning model to perform inference on the second data, it can select the quantization parameter corresponding to the first iteration group to which the i-th iteration belongs for quantization processing. Since the quantization parameter corresponding to the first iteration group to which the i-th iteration belongs is more in line with the distribution of the input data of the i-th iteration, using it for quantization processing can reduce the accuracy loss of the model output, thereby helping to improve the model inference accuracy and ensure the quality of the model output.

[0008] Furthermore, sharing the same set of quantization parameters across at least one iteration within the same iteration group reduces redundancy and saves storage space, enabling the achievement of business objectives with minimal quantization parameters. Compared to dynamic quantization techniques, this application performs data statistics and quantization parameter determination during the simulation iteration phase, avoiding the overhead of data statistics and quantization parameter calculation during actual inference. This results in lower computational complexity, reduces operational overhead on hardware platforms, improves hardware adaptability, and facilitates model deployment.

[0009] In one possible implementation, the first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter corresponding to each iteration group; the machine learning model has a fixed first weight quantization parameter;

[0010] After determining the corresponding first quantization parameter for each of the M iteration groups, the method further includes:

[0011] The weights of the machine learning model are quantized using the first weight quantization parameter to obtain the quantized model; the target model includes the quantized model and the first activation value quantization parameter corresponding to M iteration groups.

[0012] In this implementation, when the weight quantization parameters of the machine learning model are fixed, the first electronic device can use the first weight quantization parameter to quantize the weights of the machine learning model, thereby converting the machine learning model into a quantized model. This allows the hardware platform to perform real inference using the quantized model by simply selecting the first activation value quantization parameter corresponding to the current iteration round to quantize the activation value, without having to perform weight quantization operations again, which helps reduce the overhead of the hardware platform.

[0013] In one possible implementation, the first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter and the second weight quantization parameter corresponding to each iteration group; the target model includes a machine learning model, the first activation value quantization parameter and the second weight quantization parameter corresponding to M iteration groups.

[0014] In this implementation, each iteration group has a corresponding first activation value quantization parameter and a second weight quantization parameter. That is, the activation value quantization parameter and weight quantization parameter are different between groups. When the hardware platform executes each iteration, it can select different weight quantization parameters to quantize the weights of the machine learning model and different activation value quantization parameters to quantize the output activation values ​​of the machine learning model. Since the first activation value quantization parameter corresponding to each iteration group is closer to the distribution of the input data for that iteration group, it can reduce the accuracy loss of activation value quantization. Furthermore, the second weight quantization parameter corresponding to each group is learned with the aim of making the first output activation value and the corresponding second output activation value of each network layer in each simulation iteration group closer. Therefore, using it to quantize the weights of the machine learning model can further reduce the quantization accuracy loss of the model output, thereby improving the accuracy of model inference and prediction.

[0015] In one possible implementation, a machine learning model is used to perform simulation inference on the first data to execute N rounds of simulation iterations, including:

[0016] Using a machine learning model, perform the m-th simulation iteration in N simulation iterations through the following steps until m equals N:

[0017] The second activation value quantization parameter of the j-th network layer is determined based on the unprocessed activation value of the j-th network layer; where m is greater than or equal to 1 and less than or equal to N; j is greater than or equal to 1 and less than or equal to L; and L is the number of network layers in the machine learning model.

[0018] Obtain the third weight quantization parameter of the j-th network layer;

[0019] The activation value to be processed in the j-th network layer is quantized using the second activation value quantization parameter, and then the quantized value is dequantized to obtain the input activation value of the j-th network layer.

[0020] The first weight of the j-th network layer is quantized using the third weight quantization parameter, and then the quantized value is dequantized to obtain the second weight of the j-th network layer.

[0021] The input activation value of the j-th network layer is processed based on the second weight of the j-th network layer to obtain the first output activation value of the j-th network layer in the m-th simulation iteration; wherein, when j equals L, the m-th simulation iteration is completed.

[0022] Specifically, when m equals 1 and j equals 1, the activation value to be processed for the j-th network layer is the first data; when m does not equal 1 and j equals 1, the activation value to be processed for the j-th network layer is the first output activation value of the L-th network layer of the machine learning model in the (m-1)th simulation iteration; when m does not equal 1 and j does not equal 1, the activation value to be processed for the j-th network layer is the first output activation value of the (j-1)-th network layer of the machine learning model in the m-th simulation iteration.

[0023] In this implementation, the activation values ​​to be processed in each network layer of the machine learning model are quantized and dequantized in each simulation iteration to introduce quantization error. As an additional piece of information, quantization error helps the machine learning model learn how to make better predictions in the presence of quantization error during the simulation inference stage, thereby compensating for the error caused by quantization. This helps reduce the accuracy loss of the machine learning model or the quantized model of the machine learning model in real inference and improves the prediction accuracy of the model.

[0024] In one possible implementation, the N rounds of simulation iterations are divided into M iteration groups, including:

[0025] Based on the input activation values ​​of each network layer in N simulation iterations of the machine learning model, the distribution range of the input activation values ​​of each network layer in each simulation iteration is determined.

[0026] Based on the distribution range of the input activation values ​​of each network layer in each simulation iteration, the distribution distance of the input activation values ​​in N simulation iterations is determined;

[0027] The simulation iteration group is defined as the simulation iteration group in which the distribution distance of the input activation values ​​in N simulation iterations is less than or equal to a preset threshold.

[0028] The at least one iteration in N iterations whose iteration number is the same as the iteration number of at least one simulation iteration is identified as an iteration group, so that M iteration groups are obtained.

[0029] In this implementation, when the first electronic device performs N rounds of simulation inference, it can determine the distribution range of the input activation values ​​of each network layer in each simulation iteration based on the input activation values ​​of each network layer. This allows it to further determine at least one simulation iteration in the N rounds where the distribution of input activation values ​​is relatively close, and to group this at least one simulation iteration into the same simulation iteration group. In the real inference scenario, at least one iteration with the same number of iterations also constitutes an iteration group. Therefore, at least one iteration in the N rounds where the input data distribution is relatively close is grouped into an iteration group, resulting in M ​​iteration groups. For at least one iteration belonging to the same iteration group, since the distribution of their input data is relatively close, the same quantization parameter can be determined for the at least one iteration in the same group. However, since the distribution of input data differs significantly among the M iteration groups, different quantization parameters can be determined for the M iteration groups.

[0030] In one possible implementation, the M iteration groups correspond one-to-one with the M simulation iteration groups in terms of the number of iterations included. The M simulation iteration groups are obtained by dividing at least one of the N simulation iterations into one simulation iteration group.

[0031] For each of the M iteration groups, determine the corresponding first quantization parameter, including:

[0032] For any one of the M simulation iteration groups, based on the distribution range of the input activation values ​​of each network layer in each round of simulation iteration in any simulation iteration group, determine the maximum and minimum input activation values ​​of each network layer in any simulation iteration group.

[0033] Based on the maximum and minimum input activation values ​​and the preset data bit width of each network layer in any simulation iteration group, the activation value quantization parameter corresponding to each network layer in any simulation iteration group is determined; the activation value quantization parameter is the first activation value quantization parameter corresponding to the second iteration group; the second iteration group is the iteration group corresponding to any simulation iteration group among M iteration groups.

[0034] In this implementation, for at least one round of simulation iteration in the same simulation iteration group, the maximum and minimum input activation values ​​of each network layer in that at least one round of simulation iteration (i.e., the maximum and minimum input activation values ​​within the group) are selected to calculate the activation value quantization parameter corresponding to that simulation iteration group. This activation value quantization parameter is then determined as the first activation value quantization parameter corresponding to at least one round of iteration with the same iteration number in the real inference scenario. This achieves the determination of different activation value quantization parameters for M iteration groups based on the different distributions of input data between iteration groups.

[0035] In one possible implementation, each network layer has a corresponding second output activation value in each simulation iteration of N simulation iterations; the second output activation value is obtained by performing simulation inference on the first data using the machine learning model in a way that does not involve any quantization before performing simulation inference on the first data using the machine learning model to execute N simulation iterations.

[0036] The method also includes:

[0037] Obtain the second output activation value of each network layer in each round of simulation iteration in any simulation iteration group;

[0038] The target loss is determined based on the first output activation value of each network layer in each simulation iteration in any simulation iteration group and the corresponding second output activation value of each network layer in each simulation iteration in any simulation iteration group.

[0039] Based on the target loss, the third weight quantization parameter of each network layer is updated through the backpropagation algorithm to obtain the weight quantization parameter of each network layer in any simulation iteration group; the weight quantization parameter is the second weight quantization parameter corresponding to the second iteration group.

[0040] In this implementation, the target loss is the value of the constructed loss function. By constructing the loss function, machine learning is used to make the first output activation value of each network layer in each simulation iteration of any simulation iteration group closer to the corresponding second output activation value. The first output activation value of each network layer in each simulation iteration of any simulation iteration group is related to the weight quantization parameter to be optimized. By optimizing the target loss, the weight quantization parameter that can reduce the output loss can be determined. This weight quantization parameter is then used as the second weight quantization parameter corresponding to the second iteration group in the real inference scenario. Based on the first activation value quantization parameter, the accuracy loss of the output can be further reduced, thereby improving the inference accuracy of the model (such as the quantization model of a machine learning model).

[0041] In one possible implementation, the quantization model is an intermediate representation of the graph structure, the IR model; each network layer of the machine learning model is represented as a node in the IR model, and the quantization parameter of the first activation value corresponding to each network layer in each of the N iterations is used as the attribute information of the node corresponding to each network layer.

[0042] In this implementation, the machine learning model is converted into a graph-structured IR model. This preserves the structure and functionality of the machine learning model while simplifying direct interaction with the hardware platform. Furthermore, the IR model can generate target code specific to the hardware platform, facilitating deployment on various hardware platforms. When the hardware platform performs inference based on the graph structure, it can obtain the quantization parameter of the first activation value corresponding to the current iteration based on the node's attribute information, thereby quantizing the activation value output by each node.

[0043] Secondly, embodiments of this application provide a data processing method, which can be applied to a second electronic device or a chip in the second electronic device. Taking the application of this method to a second electronic device as an example, the method includes:

[0044] Obtain the second data;

[0045] The target model is used to infer the second data to perform N rounds of iteration, and the processing result of the second data is obtained; the output of the i-th iteration in the N rounds of iteration is the input of the (i+1)-th iteration; where N is greater than 1; i is greater than or equal to 1 and less than N;

[0046] The N iterations are divided into M iteration groups. Each iteration group has a corresponding first quantization parameter. The first quantization parameters of the M iteration groups are different. M is greater than 1 and less than or equal to N. When the target model performs the i-th iteration, it uses the first quantization parameter corresponding to the first iteration group in the M iteration groups for quantization. The first iteration group includes the i-th iteration.

[0047] The target model is obtained based on the machine learning model described in the first aspect above; the first quantization parameters corresponding to the M iteration groups are obtained based on the method described in the first aspect above.

[0048] As can be seen from the embodiments of this application, during the inference process of the second electronic device (such as a hardware platform) on the second data, the first quantization parameter corresponding to the iteration group to which the i-th iteration belongs can be selected for quantization processing based on the round number of the i-th iteration. Since the quantization parameters corresponding to the M iteration groups are different, and the quantization parameter corresponding to each iteration group is more consistent with the distribution of the input data of that iteration group, the hardware platform can select different first quantization parameters for quantization processing based on the distribution differences of the input data of the M iteration groups. When the distribution of the quantization parameter is closer to that of the input data, using it for quantization processing can reduce the accuracy loss of the model output, thereby helping to improve the model inference accuracy and ensure the quality of the model output.

[0049] In one possible implementation, the first quantization parameter corresponding to the M iteration groups includes the first activation value quantization parameter corresponding to each iteration group; the machine learning model has a fixed first weight quantization parameter; the target model includes the quantization model and the first activation value quantization parameter corresponding to the M iteration groups; the quantization model is obtained by quantizing the weights of the machine learning model using the first weight quantization parameter.

[0050] In this implementation, since the quantization model is obtained by quantizing the weights of the machine learning model through the first weight quantization parameter, when the hardware platform uses the quantization model for real inference, it only needs to select the first activation value quantization parameter corresponding to the current iteration round to quantize the activation value, without having to perform the weight quantization operation again, which helps to reduce the overhead of the hardware platform.

[0051] In one possible implementation, the first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter and the second weight quantization parameter corresponding to each iteration group; the target model includes a machine learning model, the first activation value quantization parameter and the second weight quantization parameter corresponding to M iteration groups.

[0052] In this implementation, each iteration group has a corresponding first activation value quantization parameter and a second weight quantization parameter. That is, the activation value quantization parameter and weight quantization parameter are different between groups. When the hardware platform executes each iteration, it can select different weight quantization parameters to quantize the weights of the machine learning model and different activation value quantization parameters to quantize the output activation values ​​of the machine learning model. Since the first activation value quantization parameter corresponding to each iteration group is closer to the distribution of the input data for that iteration group, it can reduce the accuracy loss of activation value quantization. Furthermore, the second weight quantization parameter corresponding to each group is learned with the aim of making the first output activation value and the corresponding second output activation value of each network layer in each simulation iteration group closer. Therefore, using it to quantize the weights of the machine learning model can further reduce the quantization accuracy loss of the model output, thereby improving the accuracy of model inference and prediction.

[0053] In one possible implementation, the quantization model is an intermediate representation of the graph structure, the IR model; each network layer of the machine learning model is represented as a node in the IR model, and the quantization parameter of the first activation value corresponding to each network layer in each of the N iterations is used as the attribute information of the node corresponding to each network layer.

[0054] In this implementation, the machine learning model is converted into a graph-structured IR model. This preserves the structure and functionality of the machine learning model while simplifying direct interaction with the hardware platform. Furthermore, the IR model can generate target code specific to the hardware platform, facilitating deployment on various hardware platforms. When the hardware platform performs inference based on the graph structure, it can obtain the quantization parameter of the first activation value corresponding to the current iteration based on the node's attribute information, thereby quantizing the activation value output by each node.

[0055] In one possible implementation, N rounds of iteration are performed by using a machine learning model to perform simulation inference on the first data to execute N rounds of simulation iterations, which are divided into M iteration groups.

[0056] In this implementation, during N rounds of simulation inference, the distribution range of the input activation values ​​of each network layer in each simulation iteration can be determined based on the input activation values ​​of each network layer. This further determines at least one simulation iteration in which the distribution of input activation values ​​is relatively close in the N rounds of simulation iteration, and divides this at least one simulation iteration into the same simulation iteration group. In the real inference scenario, at least one iteration with the same number of iterations also constitutes an iteration group. Thus, at least one iteration in which the distribution of input data is relatively close in the N rounds of iteration is divided into an iteration group to obtain M iteration groups.

[0057] Thirdly, embodiments of this application provide a quantization device for a machine learning model. This machine learning model requires N iterations for inference, where the output of the i-th iteration is the input of the (i+1)-th iteration; wherein N is greater than 1; i is greater than or equal to 1 and less than N; this device can be applied to a first electronic device or a chip within the first electronic device; the device includes a simulation inference unit, a grouping unit, and a quantization parameter determination unit; wherein:

[0058] The simulation inference unit is used to perform simulation inference on the first data using a machine learning model to execute N rounds of simulation iterations;

[0059] Grouping unit, used to divide N rounds of simulation iterations into M iteration groups;

[0060] A quantization parameter determination unit is used to determine the corresponding first quantization parameter for each of the M iteration groups; wherein the first quantization parameters corresponding to the M iteration groups are different; M is greater than 1 and less than or equal to N; the first quantization parameters corresponding to the M iteration groups are used for quantization processing when the target model performs inference on the second data and executes the i-th iteration; the target model is obtained based on a machine learning model; the first iteration group includes the i-th iteration.

[0061] In one possible implementation, the first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter corresponding to each iteration group; the machine learning model has a fixed first weight quantization parameter;

[0062] The device also includes a conversion unit, which is used for:

[0063] The weights of the machine learning model are quantized using the first weight quantization parameter to obtain the quantized model; the target model includes the quantized model and the first activation value quantization parameter corresponding to M iteration groups.

[0064] In one possible implementation, the first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter and the second weight quantization parameter corresponding to each iteration group; the target model includes a machine learning model, the first activation value quantization parameter and the second weight quantization parameter corresponding to M iteration groups.

[0065] In one possible implementation, in terms of using a machine learning model to perform simulation inference on the first data to execute N rounds of simulation iterations, the simulation inference unit is specifically used for:

[0066] Using a machine learning model, perform the m-th simulation iteration in N simulation iterations through the following steps until m equals N:

[0067] The second activation value quantization parameter of the j-th network layer is determined based on the unprocessed activation value of the j-th network layer; where m is greater than or equal to 1 and less than or equal to N; j is greater than or equal to 1 and less than or equal to L; and L is the number of network layers in the machine learning model.

[0068] Obtain the third weight quantization parameter of the j-th network layer;

[0069] The activation value to be processed in the j-th network layer is quantized using the second activation value quantization parameter, and then the quantized value is dequantized to obtain the input activation value of the j-th network layer.

[0070] The first weight of the j-th network layer is quantized using the third weight quantization parameter, and then the quantized value is dequantized to obtain the second weight of the j-th network layer.

[0071] The input activation value of the j-th network layer is processed based on the second weight of the j-th network layer to obtain the first output activation value of the j-th network layer in the m-th simulation iteration; wherein, when j equals L, the m-th simulation iteration is completed.

[0072] Specifically, when m equals 1 and j equals 1, the activation value to be processed for the j-th network layer is the first data; when m does not equal 1 and j equals 1, the activation value to be processed for the j-th network layer is the first output activation value of the L-th network layer of the machine learning model in the (m-1)th simulation iteration; when m does not equal 1 and j does not equal 1, the activation value to be processed for the j-th network layer is the first output activation value of the (j-1)-th network layer of the machine learning model in the m-th simulation iteration.

[0073] In one possible implementation, regarding dividing the N iterations into M iteration groups based on N simulation iterations, the grouping unit is specifically used for:

[0074] Based on the input activation values ​​of each network layer in N simulation iterations of the machine learning model, the distribution range of the input activation values ​​of each network layer in each simulation iteration is determined.

[0075] Based on the distribution range of the input activation values ​​of each network layer in each simulation iteration, the distribution distance of the input activation values ​​in N simulation iterations is determined;

[0076] The simulation iteration group is defined as the simulation iteration group in which the distribution distance of the input activation values ​​in N simulation iterations is less than or equal to a preset threshold.

[0077] The at least one iteration in N iterations whose iteration number is the same as the iteration number of at least one simulation iteration is identified as an iteration group, so that M iteration groups are obtained.

[0078] In one possible implementation, the M iteration groups correspond one-to-one with the M simulation iteration groups in terms of the number of iterations they include. The M simulation iteration groups are obtained by dividing at least one of the N simulation iterations into one simulation iteration group. Regarding determining the corresponding first quantization parameter for each of the M iteration groups, the quantization parameter determination unit is specifically used for:

[0079] For any one of the M simulation iteration groups, based on the distribution range of the input activation values ​​of each network layer in each round of simulation iteration in any simulation iteration group, determine the maximum and minimum input activation values ​​of each network layer in any simulation iteration group.

[0080] Based on the maximum and minimum input activation values ​​and the preset data bit width of each network layer in any simulation iteration group, the activation value quantization parameter corresponding to each network layer in any simulation iteration group is determined; the activation value quantization parameter is the first activation value quantization parameter corresponding to the second iteration group; the second iteration group is the iteration group corresponding to any simulation iteration group among M iteration groups.

[0081] In one possible implementation, each network layer has a corresponding second output activation value in each of the N simulation iterations; the second output activation value is obtained by performing simulation inference on the first data using the machine learning model without any quantization before performing simulation inference on the first data using the machine learning model to execute the N simulation iterations; the quantization parameter determination unit is also used for:

[0082] Obtain the second output activation value of each network layer in each round of simulation iteration in any simulation iteration group;

[0083] The target loss is determined based on the first output activation value of each network layer in each simulation iteration in any simulation iteration group and the corresponding second output activation value of each network layer in each simulation iteration in any simulation iteration group.

[0084] Based on the target loss, the third weight quantization parameter of each network layer is updated through the backpropagation algorithm to obtain the weight quantization parameter of each network layer in any simulation iteration group; the weight quantization parameter is the second weight quantization parameter corresponding to the second iteration group.

[0085] In one possible implementation, the quantization model is an intermediate representation of the graph structure, the IR model; each network layer of the machine learning model is represented as a node in the IR model, and the quantization parameter of the first activation value corresponding to each network layer in each of the N iterations is used as the attribute information of the node corresponding to each network layer.

[0086] It should be understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the first aspect of the embodiments of this application should be adapted to the third aspect of the embodiments of this application simultaneously, and can achieve the same or similar beneficial effects, which will not be repeated here.

[0087] Fourthly, embodiments of this application provide a data processing apparatus, which can be applied to a second electronic device or a chip in a second electronic device; the apparatus includes an acquisition unit and a processing unit; wherein:

[0088] The acquisition unit is used to acquire the second data;

[0089] The processing unit is used to perform inference on the second data using the target model to execute N rounds of iterations and obtain the processing result of the second data; the output of the i-th iteration in the N rounds of iterations is the input of the (i+1)-th iteration; where N is greater than 1; i is greater than or equal to 1 and less than N;

[0090] The N iterations are divided into M iteration groups. Each iteration group has a corresponding first quantization parameter. The first quantization parameters of the M iteration groups are different. M is greater than 1 and less than or equal to N. When the target model performs the i-th iteration, it uses the first quantization parameter corresponding to the first iteration group in the M iteration groups for quantization. The first iteration group includes the i-th iteration.

[0091] The target model is obtained based on the machine learning model described in the first aspect above; the first quantization parameters corresponding to the M iteration groups are obtained based on the method described in the first aspect above.

[0092] In one possible implementation, the first quantization parameter corresponding to the M iteration groups includes the first activation value quantization parameter corresponding to each iteration group; the machine learning model has a fixed first weight quantization parameter; the target model includes the quantization model and the first activation value quantization parameter corresponding to the M iteration groups; the quantization model is obtained by quantizing the weights of the machine learning model using the first weight quantization parameter.

[0093] In one possible implementation, the first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter and the second weight quantization parameter corresponding to each iteration group; the target model includes a machine learning model, the first activation value quantization parameter and the second weight quantization parameter corresponding to M iteration groups.

[0094] In one possible implementation, the quantization model is an intermediate representation of the graph structure, the IR model; each network layer of the machine learning model is represented as a node in the IR model, and the quantization parameter of the first activation value corresponding to each network layer in each of the N iterations is used as the attribute information of the node corresponding to each network layer.

[0095] In one possible implementation, N rounds of iteration are performed by using a machine learning model to perform simulation inference on the first data to execute N rounds of simulation iterations, which are divided into M iteration groups.

[0096] It should be understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the second aspect of the embodiments of this application should be adapted to the fourth aspect of the embodiments of this application simultaneously, and can achieve the same or similar beneficial effects, which will not be repeated here.

[0097] Fifthly, embodiments of this application provide an electronic device including a processor, a memory, a communication interface, and one or more programs, the one or more programs being stored in the memory and configured to, when executed by the processor, cooperate with the communication interface to implement the method in any of the embodiments of the first or second aspect described above.

[0098] In a sixth aspect, embodiments of this application provide a chip, including: a processor, configured to call and run a computer program from a memory, causing a device on which the chip is installed to perform the method as described in any of the embodiments of the first or second aspect above.

[0099] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program for execution by a device, wherein the computer program, when executed, implements the method as described in any of the embodiments of the first or second aspect above.

[0100] Eighthly, embodiments of this application provide a computer program product that, when run by a device, causes the device to perform the method as described in any of the embodiments of the first or second aspect above. Attached Figure Description

[0101] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0102] Figure 1 A schematic diagram of a diffusion model provided in an embodiment of this application;

[0103] Figure 2 A schematic diagram of quantization parameters in static quantization provided for an embodiment of this application;

[0104] Figure 3 A schematic diagram of a scenario architecture provided in an embodiment of this application;

[0105] Figure 4 A flowchart illustrating a quantization method for a machine learning model provided in an embodiment of this application;

[0106] Figure 5 A schematic diagram illustrating a simulation iteration method provided in an embodiment of this application;

[0107] Figure 6 A schematic diagram illustrating another simulation iteration method provided in the embodiments of this application;

[0108] Figure 7A A schematic diagram of an iteration group and corresponding quantization parameters provided for an embodiment of this application;

[0109] Figure 7B A schematic diagram of another iteration group and corresponding quantization parameters provided for an embodiment of this application;

[0110] Figure 8 A schematic diagram illustrating a model conversion provided in an embodiment of this application;

[0111] Figure 9 A schematic diagram of an IR model provided in an embodiment of this application;

[0112] Figure 10A A schematic diagram of another iteration group and corresponding quantization parameters provided for an embodiment of this application;

[0113] Figure 10B A schematic diagram of another iteration group and corresponding quantization parameters provided for an embodiment of this application;

[0114] Figure 11 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0115] Figure 12A A schematic diagram illustrating a deployment method of a target model provided in an embodiment of this application;

[0116] Figure 12B A schematic diagram illustrating another deployment method of the target model provided in an embodiment of this application;

[0117] Figure 13 A schematic diagram of the structure of a quantization device for a machine learning model provided in an embodiment of this application;

[0118] Figure 14 A schematic diagram of the structure of a quantization device for another machine learning model provided in an embodiment of this application;

[0119] Figure 15 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0120] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0121] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0122] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0123] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on electronic devices and electronic devices themselves can be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0124] First, a brief introduction to the relevant terms and technical background used in this application will be provided to facilitate understanding by those skilled in the art.

[0125] The main method of model quantization is to map the model from floating-point format to a low-bit representation, such as 8-bit integer (INT8) / 4-bit integer (INT4). This allows the model to run on edge devices, gaining storage benefits while reducing energy consumption and latency, thus achieving performance and energy efficiency gains. Multi-round iterative models play an important role in image and audio generation; the diffusion model, consisting of a variational autoencoder (VAE) and a U-Net, is a typical example. Figure 1As shown, a VAE includes an encoder and a decoder. Assuming an image is taken as input, the VAE's encoder encodes the image to obtain a noisy image. During inference, U-Net runs multiple rounds to denoise the noisy image. The U-Net's output is then decoded by the VAE's decoder to obtain the final output. There are roughly two main ideas behind quantization techniques: static quantization and dynamic quantization. For multi-round iterative models, static quantization uses the same quantization parameters for each iteration, such as... Figure 2 As shown, s w s represents the weight quantization parameter. a This represents the activation value quantization parameter. In a static quantization scheme, the quantization parameter remains unchanged for each iteration of a multi-round iterative model. Dynamic quantization, on the other hand, dynamically calculates and adjusts the model's quantization parameter based on the input of each iteration. For each iteration, the calculated quantization parameter better reflects the current distribution range of the input data.

[0126] Therefore, static quantization cannot adjust quantization parameters in real time based on the input. For activation values, inference can only be performed using quantization parameters pre-calculated from calibration data, resulting in poor flexibility. Since each iteration uses fixed quantization parameters, which differ from the distribution range of the input data, the model's quantization accuracy is low, potentially leading to decreased inference accuracy and significant precision loss. The calculation of activation value quantization parameters in static quantization is directly related to the selection of calibration data; the higher the generalization of the calibration data distribution, the higher the generalization of the calculated quantization parameters, thus exhibiting high data dependence. Dynamic quantization requires dynamically calculating activation value quantization parameters during model inference, increasing computational complexity and runtime overhead, impacting model inference performance. Furthermore, the hardware needs to perform data range statistics during calculation, incurring additional hardware overhead, making deployment impossible on hardware platforms that do not support real-time statistical data range analysis.

[0127] To overcome the shortcomings of existing technologies, embodiments of this application provide a quantization method for machine learning models, which can be applied to... Figure 3 The scene architecture is shown below. Figure 3As shown, this application scenario includes a model to be deployed and the edge devices on which the model can be deployed. The edge devices can be smartwatches, smartphones, tablets, wearable devices, smart home devices, personal computers (PCs), etc. The model to be deployed is a model with multiple network layers that requires N rounds of inference, such as U-Net, or a model with multiple identical blocks where the output of the previous block is the input of the next block. For this model to be deployed, electronic devices (such as servers, computers, cloud servers, chips, high-performance edge devices, etc.) can group its N rounds of iteration into multiple iteration groups, and then determine the corresponding quantization parameters for each iteration group. Based on these quantization parameters, a new model can be obtained. For example, the model to be deployed can be converted into a fixed-point model. The converted model can include the determined quantization parameters and can be deployed on edge devices. The model deployed on the edge device can perform N rounds of iteration for inference based on user input (such as images, speech, text, etc.), and in each round of iteration, the corresponding quantization parameters can be used to quantize the weights and / or activation values. In this approach, the same quantization parameters are used for at least one iteration within the same iteration group. Compared to static quantization, this scheme uses different quantization parameters for each iteration group, and these parameters closely approximate the distribution of the input data for at least one iteration within the group, ensuring the model's quantization accuracy and reducing inference accuracy loss. Furthermore, determining specific quantization parameters for each iteration group eliminates the need for calibration data, decoupling the quantization parameters from the calibration data and reducing data dependency. Compared to dynamic quantization, the quantization parameters do not need to be recalculated during model inference, significantly reducing computational complexity. Additionally, the hardware does not need to perform data statistics, reducing hardware overhead and improving generalization across hardware platforms.

[0128] The technical solution provided in this application will be described in detail below with reference to specific implementation methods.

[0129] Please see Figure 4 , Figure 4 This application provides a flowchart illustrating a quantization method for a machine learning model, which can be based on... Figure 3 The scenario architecture shown can be implemented by the first electronic device. For example... Figure 4 As shown, the method includes steps 401-402:

[0130] 401: Use a machine learning model to perform simulation inference on the first data to execute N rounds of simulation iterations.

[0131] In this embodiment, the machine learning model needs to perform N iterations for inference, where the output of the i-th iteration is the input of the (i+1)-th iteration; where N is greater than 1, and i is greater than or equal to 1 and less than N. That is, in simulated inference scenarios, the machine learning model needs to perform N simulation iterations; in real-world inference, the machine learning model or its quantized form also needs to perform N iterations, with the output of the previous iteration serving as the input of the next iteration.

[0132] The first data can be activation values ​​obtained from sample data. For example, in image generation, the sample data can be images; in text generation, it can be corpora; and in speech generation, it can be audio files. For instance, in a diffusion model composed of a machine learning model (e.g., U-Net) and a VAE, the first data can be the activation values ​​output by the VAE's encoder when encoding the sample data. For instance, in a scenario where the machine learning model is used alone, the first data can be the activation values ​​output by the machine learning model's own encoder when encoding the sample data.

[0133] (a) Perform N rounds of simulation iterations

[0134] (1) Simulation iteration without introducing quantization

[0135] For example, in performing simulation inference on the first data using a machine learning model to execute N rounds of simulation iterations, the simulation inference on the first data can be performed without any quantization. Taking the m-th simulation iteration of the N rounds of simulation iterations by the machine learning model as an example, the input activation value of the j-th network layer is processed based on the weights of the j-th network layer of the machine learning model to obtain the output activation value of the j-th network layer. The output activation value of the j-th network layer is used as the input activation value of the (j+1)-th network layer. The input activation value of the (j+1)-th network layer is processed based on the weights of the (j+1)-th network layer to obtain the output activation value of the (j+1)-th network layer, and so on, until the output activation value of the L-th network layer of the machine learning model in the m-th simulation iteration is obtained. Here, m is greater than or equal to 1 and less than or equal to N; j is greater than or equal to 1 and less than or equal to L; L is the number of network layers in the machine learning model. If m equals 1 and j equals 1, then the input activation value of the j-th network layer is the first data; if m does not equal 1 and j equals 1, then the input activation value of the j-th network layer is the output activation value of the L-th network layer in the (m-1)th simulation iteration; if m does not equal 1 and j does not equal 1, then the input activation value of the j-th network layer is the output activation value of the (j-1)-th network layer of the machine learning model in the m-th simulation iteration.

[0136] For example, such as Figure 5As shown, assuming the machine learning model includes three network layers, the input to the first network layer in the first simulation iteration can be the first data. After processing the first data, the first network layer obtains the output activation value. This output activation value is used as the input activation value of the second network layer. The second network layer processes the input activation value and obtains the output activation value. This output activation value is then used as the input activation value of the third network layer. The first simulation iteration is completed. The output activation value of the third network layer is used as the input activation value of the first network layer in the second simulation iteration, thus enabling the machine learning model to execute the second simulation iteration, until the output activation value of the Lth network layer in the Nth simulation iteration is obtained.

[0137] (2) Introducing quantization for simulation iteration

[0138] For example, in performing simulation inference on the first data using a machine learning model to execute N rounds of simulation iterations, quantization can be introduced to perform simulation inference on the first data. Taking the m-th simulation iteration of the N rounds of simulation iterations performed by the machine learning model as an example, for the j-th network layer of the machine learning model:

[0139] A: The first electronic device can determine the activation value quantization parameter of the j-th network layer based on the unprocessed activation value of the machine learning model. This activation value quantization parameter is denoted as the second activation value quantization parameter. The second activation value quantization parameter is not used as the final determined activation value quantization parameter, but is used to perform pseudo-quantization (i.e., quantization during simulation iteration) on the unprocessed activation value of the j-th network layer. For example, the first electronic device can count the maximum and minimum activation values ​​among the unprocessed activation values ​​of the j-th network layer, and calculate the second activation value quantization parameter of the j-th network layer using the following formula:

[0140]

[0141]

[0142] Among them, scale a Offset represents the scaling factor in the quantization parameters of the second activation value. a This represents the offset in the quantization parameters of the second activation value. a Min represents the maximum input activation value among the activation values ​​to be processed. a This represents the minimum input activation value among the activation values ​​to be processed. B represents the preset data bit width, for example, the value range of B can be (0, 255). round indicates rounding.

[0143] b: Based on the quantization parameters of the second activation value of the j-th network layer, the first electronic device quantizes the activation value to be processed in the j-th network layer using the following formula:

[0144]

[0145] Among them, quantized a This represents the quantized value, and activation represents the value to be quantized, which here is the activation value to be processed for the j-th network layer.

[0146] c: After quantizing the activation value to be processed in the j-th network layer, the quantized value is dequantized using the following formula to convert the fixed-point value back to a floating-point value:

[0147] dequantized a =scale a ×(quantized a -offset a (4)

[0148] The dequantized value a As the input activation value of the j-th network layer.

[0149] d: The first electronic device acquires the third weight quantization parameter of the j-th network layer. The first weights of the j-th network layer can be pre-statistically calculated. Based on these first weights, the weight quantization parameter of the j-th network layer can be determined and saved. This weight quantization parameter is denoted as the third weight quantization parameter. The third weight quantization parameter is not used as the final weight quantization parameter but is used for pseudo-quantization of the first weights of the j-th network layer (i.e., quantization is performed during simulation iteration). Here, the first weight refers to the weights trained by the machine learning model. For example, the first electronic device can statistically analyze the maximum and minimum values ​​of the weights in the j-th network layer and calculate the third weight quantization parameter of the j-th network layer using the following formula:

[0150]

[0151]

[0152] Where, max w Min represents the maximum value among the weights of the j-th network layer. w This represents the minimum weight among the weights of the j-th network layer. w The offset represents the scaling factor in the third-weighted quantization parameters. w This represents the offset in the third weight quantization parameter.

[0153] e: Based on the quantization parameters of the third weight of the j-th network layer, the first electronic device quantizes the first weight of the j-th network layer using the following formula:

[0154]

[0155] Among them, quantized w This represents the value after quantization of the first weight, where weight represents the first weight.

[0156] f: After quantizing the first weight of the j-th network layer, the quantized value is dequantized using the following formula to convert the fixed-point value back to a floating-point value:

[0157] dequantized w =scale w ×(quantized w -offset w (8)

[0158] The dequantized value w As a new weight for the j-th network layer, it is denoted as the second weight.

[0159] g: dequantized a Input the j-th network layer, and process the input activation value of the j-th network layer based on the second weight of the j-th network layer (such as multiplication in convolution) to obtain the output activation value of the j-th network layer in the m-th simulation iteration. The output activation value in this stage is denoted as the first output activation value.

[0160] In the m-th simulation iteration, the first output activation value of the previous network layer is used as the activation value to be processed in the next network layer. During the simulation iteration, steps a to g are performed for each network layer until the first output activation value of the L-th network layer is obtained. That is, when j equals L, the m-th simulation iteration is completed.

[0161] Specifically, when m equals 1 and j equals 1, the unprocessed activation value of the j-th network layer is the first data; when m does not equal 1 and j equals 1, the unprocessed activation value of the j-th network layer is the first output activation value of the L-th network layer in the (m-1)th simulation iteration of the machine learning model, that is, the first output activation value of the L-th network layer in the previous simulation iteration is the unprocessed activation value of the 1-th network layer in the next simulation iteration; when m does not equal 1 and j does not equal 1, the unprocessed activation value of the j-th network layer is the first output activation value of the (j-1)-th network layer in the m-th simulation iteration of the machine learning model. By using the first output activation value of the L-th network layer in the previous simulation iteration as the unprocessed activation value of the 1-th network layer in the next simulation iteration, the above steps ag are performed for each network layer until m equals N, which means that the first output activation value of the L-th network layer in the N-th simulation iteration is obtained, indicating that the N-th simulation iteration has been completed.

[0162] For example, such as Figure 6 As shown, assuming the machine learning model includes three network layers, for the first network layer, its activation value to be processed is the first data. Step ac is performed on the first data to obtain the input activation value of the first network layer. Step df is performed on the weights of the first network layer to obtain the second weights of the first network layer. Based on these second weights, step g is performed to obtain the first output activation value of the first network layer. The first output activation value of the first network layer is used as the activation value to be processed for the second network layer. Step ac is performed on this activation value to obtain the input activation value of the second network layer. Step df is performed on the weights of the second network layer to obtain the second weights of the second network layer. Based on these second weights, step g is performed to obtain the first output activation value of the second network layer. The first output activation value of the second network layer is used as the activation value to be processed in the third network layer. Step ac is performed on this activation value to obtain the input activation value of the third network layer. Step df is then performed on the weights of the third network layer to obtain the second weights. Based on these second weights, step g is performed to obtain the first output activation value of the third network layer. At this point, the first round of simulation iteration is complete. The first output activation value of the third network layer in the first round of simulation iteration is used as the activation value to be processed in the second round of simulation iteration for the first network layer. Step ag is then performed on each network layer to complete the second round of simulation iteration, until N rounds of simulation iteration are completed.

[0163] In this implementation, the activation values ​​to be processed in each network layer of the machine learning model are quantized and dequantized in each simulation iteration to introduce quantization error. As an additional piece of information, the quantization error helps the machine learning model learn how to make better predictions in the presence of quantization error during the simulation inference stage, thereby compensating for the error caused by quantization. This helps reduce the accuracy loss of the machine learning model or the quantized model of the machine learning model in real inference (such as after deployment on the edge) and improve the prediction accuracy of the model.

[0164] 402: Based on N rounds of simulation iterations, the N rounds of iterations are divided into M iteration groups, and a corresponding first quantization parameter is determined for each of the M iteration groups; wherein, the first quantization parameters corresponding to the M iteration groups are different; M is greater than 1 and less than or equal to N; the first quantization parameters corresponding to the M iteration groups are used in the process of the target model inferring the second data, and the first quantization parameters corresponding to the first iteration group in the M iteration groups are used for quantization processing when the i-th iteration is executed; the target model is obtained based on the machine learning model; the first iteration group includes the i-th iteration.

[0165] In this embodiment, the first iteration group is any one of the M iteration groups, and the first quantization parameter is the quantization parameter required for quantization processing in the real inference scenario. Regardless of whether N rounds of simulation iterations are performed with or without quantization, each network layer of the machine learning model has an input activation value in each round of simulation iteration. Based on the input activation values ​​of each network layer in each round of simulation iteration, the first electronic device can divide the N rounds of simulation iterations into M rounds. Therefore, the N rounds of iterations in the real inference scenario are also divided into M iteration groups according to the iteration rounds.

[0166] (ii) Grouping the N iterations

[0167] 1) Based on the input activation values ​​of each network layer in N simulation iterations of the machine learning model, the first electronic device can count the maximum and minimum input activation values ​​of each network layer in each simulation iteration, thereby determining the distribution range of the input activation values ​​of each network layer in each simulation iteration.

[0168] For example, such as Figure 5As shown, the machine learning model includes three network layers. For the first network layer, its input in the first simulation iteration can be the first data. The first electronic device can count the maximum and minimum values ​​in the first data, using the maximum value as the upper boundary and the minimum value as the lower boundary, thus obtaining the distribution range of the input activation values ​​of the first network layer in the first simulation iteration, for example, [-0.001, 3.14]. After processing the first data, the first network layer obtains the output activation value, which can be used as the input activation value of the second network layer. The first electronic device can also determine the maximum and minimum values ​​in this input activation value, obtaining the distribution range of the input activation values ​​of the second network layer. After processing the input activation values, the second network layer obtains the output activation value, which can be used as the input activation value of the third network layer. The first electronic device can also determine the maximum and minimum values ​​in this input activation value, obtaining the distribution range of the input activation values ​​of the third network layer. After processing the input activation values, the third network layer obtains the output activation value, which serves as the input activation value for the next simulation iteration of the machine learning model. Based on the above description, the first electronic device can statistically analyze the input activation values ​​of each network layer in each simulation iteration to obtain the distribution range of the input activation values ​​of each network layer in each simulation iteration.

[0169] For example, such as Figure 6 As shown, for Figure 6 The three network layers have input activation values ​​that are the values ​​of the activation values ​​to be processed after quantization and dequantization. The first electronic device can count the maximum and minimum input activation values ​​of each network layer in each simulation iteration to obtain the distribution range of the input activation values ​​of each network layer in each simulation iteration.

[0170] 2) Based on the distribution range of the input activation values ​​of each network layer in each simulation iteration, the first electronic device can obtain the distribution distance of the overall input activation values ​​in N simulation iterations.

[0171] 3) Divide at least one simulation iteration in which the distribution distance of the input activation values ​​in N simulation iterations is less than or equal to a preset threshold into a simulation iteration group.

[0172] by Figure 5 or Figure 6Taking a three-layer network as an example, suppose that in the first simulation iteration: the distribution interval of the input activation values ​​of the first network layer is [a1, a2], the distribution interval of the input activation values ​​of the second network layer is [a3, a4], and the distribution interval of the input activation values ​​of the third network layer is [a5, a6]; in the second simulation iteration: the distribution interval of the input activation values ​​of the first network layer is [b1, b2], the distribution interval of the input activation values ​​of the second network layer is [b3, b4], and the distribution interval of the input activation values ​​of the third network layer is [b5, b6]. In one possible implementation, a statistic can be calculated for the distribution interval of the input activation values ​​of each network layer in each simulation iteration. This statistic can be the difference between the maximum and minimum input activation values, such as a2-a1 for the first network layer; or the statistic can be the mean of the maximum and minimum activation values, and so on. Thus, L statistics of the machine learning model in each simulation iteration can be obtained, where L represents the number of network layers in the machine learning model, as in the example above, which is 3. Based on L statistics from the machine learning model in each simulation iteration, the first electronic device can determine the distribution distance of the input activation values ​​across N simulation iterations. This distribution distance characterizes the overall distribution distance of the input activation values ​​across N simulation iterations. For example, it can be calculated by measuring the mean, standard deviation, variance, or Gaussian distribution of the L statistics from the machine learning model in each simulation iteration. Taking the mean of the L statistics as an example, for every two simulation iterations, the difference between the means of the L statistics for each iteration is used as the distribution distance of their input activation values. If the difference between the means of the L statistics for each iteration is less than or equal to a preset threshold, it indicates that the distribution of the overall input activation values ​​in the two simulation iterations is relatively close, and they can be grouped into the same simulation iteration group.

[0173] In another possible implementation, based on obtaining L statistics of the machine learning model in each simulation iteration, for each network layer of the machine learning model, the difference in statistics between every two simulation iterations is calculated. For example, for the first network layer mentioned above, its statistics in the first simulation iteration are L11, and its statistics in the second simulation iteration are L12. Then, L12 - L11 represents the distribution distance of the input activation values ​​of the first network layer in the first and second simulation iterations. This yields L distribution distances between the first and second simulation iterations. If all L distribution distances are less than or equal to a preset threshold, it indicates that the overall distribution of input activation values ​​in the two simulation iterations is relatively close, and they can be grouped into the same simulation iteration group. Alternatively, the mean of the L distribution distances between the first and second simulation iterations can also be used. If this mean is less than or equal to a preset threshold, it also indicates that the overall distribution of input activation values ​​in the two simulation iterations is relatively close, and they can be grouped into the same simulation iteration group.

[0174] 4) Determine at least one iteration in N iterations whose iteration number is the same as the iteration number of at least one simulation iteration as an iteration group, so as to obtain M iteration groups.

[0175] For at least one simulation iteration in N rounds where the distribution of the overall input activation values ​​is relatively similar, it can be divided into a simulation iteration group. Therefore, N rounds of simulation iterations can be divided into M simulation iteration groups. Correspondingly, for the machine learning model or the model transformed from the machine learning model, at least one iteration in the N rounds of iterations required for actual inference that has the same number of iterations as at least one simulation iteration is also determined as an iteration group, thus obtaining M iteration groups. For example, if the first to third simulation iterations in N rounds of simulation iterations belong to the same simulation iteration group, then the first to third iterations in actual inference belong to the same iteration group; if the fourth and sixth simulation iterations in N rounds of simulation iterations belong to the same simulation iteration group, then the fourth and sixth iterations in actual inference belong to the same iteration group; if the fifth simulation iteration in N rounds of simulation iterations is a separate simulation iteration group, then the fifth iteration in actual inference is a separate iteration group.

[0176] In this implementation, when the first electronic device performs N rounds of simulation inference, it can determine the distribution range of the input activation values ​​of each network layer in each simulation iteration based on the input activation values ​​of each network layer. This allows it to further determine at least one simulation iteration in the N rounds where the distribution of input activation values ​​is relatively close, and to group this at least one simulation iteration into the same simulation iteration group. In the real inference scenario, at least one iteration with the same number of iterations also constitutes an iteration group. Therefore, at least one iteration in the N rounds where the input data distribution is relatively close is grouped into an iteration group, resulting in M ​​iteration groups. For at least one iteration belonging to the same iteration group, since the distribution of their input data is relatively close, the same quantization parameter can be determined for the at least one iteration in the same group. However, since the distribution of input data differs significantly among the M iteration groups, different quantization parameters can be determined for the M iteration groups.

[0177] (III) Determine the quantization parameters for each iteration group

[0178] For example, since the first electronic device can statistically determine the maximum and minimum input activation values ​​(the upper and lower bounds of the distribution range of input activation values) of each network layer in each simulation iteration of the machine learning model, it can determine the maximum and minimum input activation values ​​of each network layer in at least one simulation iteration within the same simulation iteration group. For instance, in N simulation iterations, iterations 1 to 3 belong to simulation iteration group A. Figure 5The first network layer in the simulation has an input activation value distribution range of [-0.001, 3.14] in the first simulation iteration, [-0.02, 1.34] in the second simulation iteration, and [0.2, 4.13] in the third simulation iteration. Therefore, the maximum input activation value of the first network layer in each simulation iteration group A is 4.13 in the third simulation iteration, and the minimum input activation value in each simulation iteration group A is -0.02 in the second simulation iteration. Based on determining the maximum and minimum input activation values ​​of each network layer in any simulation iteration group, the first electronic device can use the above formulas (1) and (2) to calculate the activation value quantization parameters corresponding to each network layer in any simulation iteration group, thus obtaining the first activation value quantization parameters to be used in the real inference scenario for the corresponding iteration group (second iteration group) in M ​​iteration groups. Based on determining the maximum and minimum input activation values ​​of the first network layer in simulation iteration group A, the first electronic device can use the above formulas (1) and (2) to calculate the activation value quantization parameters corresponding to the first network layer in simulation iteration group A. Based on this, the activation value quantization parameters corresponding to each network layer in any simulation iteration group can be determined. Assuming that simulation iteration group A corresponds to iteration group A in the real inference scenario, the activation value quantization parameters corresponding to each network layer in simulation iteration group A can be used as the first activation value quantization parameters corresponding to iteration group A. For the i-th iteration in the real inference scenario, if the i-th iteration belongs to iteration group A, the target model uses the first activation value quantization parameters corresponding to iteration group A to quantize the output activation values ​​of each network layer in the i-th iteration. If the (i+1)-th iteration also belongs to iteration group A, the target model also uses the first activation value quantization parameters corresponding to iteration group A to quantize the output activation values ​​of each network layer in the (i+1)-th iteration. The target model is determined based on the machine learning model, and may include the machine learning model itself or the quantized model of the machine learning model.

[0179] It should be noted that the first activation value quantization parameter is the quantization parameter used to quantize the output activation value in a real inference scenario. That is, one of the purposes of the current simulation inference stage is to determine this first activation value quantization parameter.

[0180] In this implementation, for at least one round of simulation iteration in the same simulation iteration group, the maximum and minimum input activation values ​​of each network layer in the at least one round of simulation iteration are selected to calculate the activation value quantization parameter corresponding to the simulation iteration group. The activation value quantization parameter is then determined as the first activation value quantization parameter corresponding to at least one round of iteration with the same iteration number in the real inference scenario. This achieves the determination of different activation value quantization parameters for M iteration groups based on the different distributions of input data between iteration groups.

[0181] (1) Scenarios with fixed weight quantization parameters

[0182] For example, in one possible implementation, the machine learning model has fixed weight quantization parameters, denoted as the first weight quantization parameter. This first weight quantization parameter can be calculated using the formulas (5) and (6) above to obtain the first weight of each network layer of the machine learning model (in this case, the first weight quantization parameter of each network layer is equivalent to the third weight quantization parameter), or it can be pre-set. That is, the weight quantization parameter of each network layer of the machine learning model is fixed and will not differ due to different iteration rounds. For example, the first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter corresponding to each iteration group. When M equals N (i.e., each iteration is a separate group), the quantization parameters corresponding to N iterations in a real inference scenario can be found in [reference needed]. Figure 7A .like Figure 7A As shown, N iterations correspond to different activation value quantization parameters s a1 ,s a2 ,s a3 ,…,s aN The weight quantization parameter corresponding to each iteration is S. w When M is greater than 1 and less than N, at least one iteration belonging to the same iteration group corresponds to the same activation value quantization parameter. For the quantization parameters corresponding to M iteration groups in a real inference scenario, please refer to [reference needed]. Figure 7B .like Figure 7B As shown, the first and second iterations form the first iteration group, the third iteration forms another iteration group, and the Nth iteration forms yet another iteration group. Therefore, the M iteration groups correspond to different activation value quantization parameters s. a1 ,s a2 ,…,s aM The weight quantization parameter corresponding to each iteration group is S. w .

[0183] For example, in a scenario where the weight quantization parameters of each network layer in a machine learning model are fixed, the first electronic device uses the first weight quantization parameter of each network layer to quantize the weights of each network layer, such as using the above formula (7) for weight quantization. The model after weight quantization is the quantized model, which is a fixed-point model. The first activation value quantization parameters corresponding to the M iteration groups and the quantized model constitute the transformed target model, that is, the target model includes the quantized model and the first activation value quantization parameters corresponding to the M iteration groups. When the hardware platform needs to use the target model for inference, it does not need to perform weight quantization operations again, but only needs to use the first activation value quantization parameter corresponding to the iteration group to which the current iteration belongs to quantize the output activation value of each network layer. Wherein, when the target model is local, the hardware platform is the first electronic device; when the target model is deployed on the edge device, the hardware platform is the edge device.

[0184] In this implementation, when the weight quantization parameters of the machine learning model are fixed, the first electronic device can use the first weight quantization parameter to quantize the weights of the machine learning model, thereby converting the machine learning model into a quantized model. This allows the hardware platform to perform real inference using the quantized model by simply selecting the first activation value quantization parameter corresponding to the current iteration round to quantize the activation value, without having to perform weight quantization operations again, which helps reduce the overhead of the hardware platform.

[0185] For example, such as Figure 8 As shown, the first weight quantization parameter of the machine learning model and the first activation value quantization parameter corresponding to M iteration groups can be stored in a quantization parameter file. The first electronic device can quantize the weights of the machine learning model by reading the first weight quantization parameter in the quantization parameter file. Furthermore, the first activation value quantization parameter corresponding to the M iteration groups can be used as the attribute information of the quantized model, thus the target model is constructed by quantizing the model and the first activation value quantization parameter corresponding to the M iteration groups. For example, the quantization parameter information corresponding to iteration group A (iteration 1-3) is 1, where 1 is the index of the first activation value quantization parameter corresponding to iteration group A; the quantization parameter information corresponding to iteration group B (iteration 4-7) is 2, where 2 is the index of the first activation value quantization parameter corresponding to iteration group B, and so on. Based on this index, the hardware platform can obtain the first activation value quantization parameter corresponding to the index when executing the corresponding iteration round.

[0186] For example, the quantization model is an intermediate representation (IR) model with a graph structure; each network layer of the machine learning model is represented as a node in the IR model, and the quantization parameter of the first activation value corresponding to each network layer in each of the N iterations serves as the attribute information of the node corresponding to each network layer. Assume... Figure 5The operators corresponding to the three network layers in the model are all convolutions. Therefore, the IR model formed by these three network layers as nodes can be found in [reference needed]. Figure 9 ,like Figure 9 As shown, for each node, if N=7, iterations 1-3 belong to iteration group A, and iterations 4-7 belong to iteration group B, then the attribute information of each node can include: the quantization parameter information corresponding to iterations 1-3 is 1, where 1 is the index of the first activation value quantization parameter corresponding to iteration group A; the quantization parameter information corresponding to iterations 4-7 is 2, where 2 is the index of the first activation value quantization parameter corresponding to iteration group B.

[0187] In this implementation, the machine learning model is converted into a graph-structured IR model. This preserves the structure and functionality of the machine learning model while simplifying direct interaction with the hardware platform. Furthermore, the IR model can generate target code specific to the hardware platform, facilitating deployment on various hardware platforms. When the hardware platform performs inference based on the graph structure, it can obtain the quantization parameter of the first activation value corresponding to the current iteration based on the node's attribute information, thereby quantizing the activation value output by each node.

[0188] (2) Scenario of dynamic adjustment of weight quantization parameters

[0189] For example, based on dividing N iterations into M iteration groups, the first electronic device can also determine a corresponding second weight quantization parameter for each iteration group, that is, the weight quantization parameter corresponding to each iteration group is no longer fixed.

[0190] It should be noted that the simulation inference mentioned in the previous embodiments can be understood as the second-stage simulation inference (using quantization-introduced simulation inference). Before performing simulation inference on the first data using the machine learning model to execute N rounds of simulation iteration, the machine learning model also performs a first-stage simulation inference on the first data. The simulation inference process in this stage can be referred to the description in the above-mentioned scenario of "simulation iteration without quantization". That is, the first-stage simulation inference on the first data is performed without any quantization. Each network layer of the machine learning model in this stage also has an output activation value in each round of simulation iteration, denoted as the second output activation value. This second output activation value is the raw floating-point inference value obtained without any quantization. The first-stage simulation inference can be achieved through... Figure 4The method shown can be performed by the same executing entity, or by other executing entities. After obtaining the second output activation value of each network layer in each simulation iteration, it can be stored so that during the second-stage simulation inference, the second output activation value can be used to obtain the weight quantization parameters corresponding to each iteration group. In other words, for the second-stage simulation inference, each network layer has a corresponding second output activation value in each of the N simulation iterations.

[0191] For any one of the M predefined simulation iteration groups, the first electronic device can obtain the second output activation value of each network layer in each round of simulation iteration within that group. For example, in simulation iteration group A, the first network layer has a first output activation value in the first round of simulation iteration, and it also generates a second output activation value in the first round of simulation iteration in the first stage of the simulation iteration. The first electronic device can read the second output activation value of the first network layer in the first round of simulation iteration based on pre-stored data.

[0192] For example, embodiments of this application can determine different weight quantization parameters for each iteration group using machine learning. The pre-built loss function LOSS is expressed as:

[0193]

[0194] Where x represents the x-th simulation iteration in any simulation iteration group, and K represents the number of simulation iteration rounds included in any simulation iteration group. x,a,j The dequantized value represents the second output activation value (i.e., the original floating-point output) of the j-th network layer in the x-th simulation iteration of any simulation iteration group. x,a,j depquantized represents the input activation value of the j-th network layer in the x-th simulation iteration within any simulation iteration group. w,j This represents the second weight of the j-th network layer, where bias represents the bias term, and dequantized x,a,j ·dequantized w,j +bias represents the first output activation value of the j-th network layer in each simulation iteration of any simulation iteration group.

[0195] Based on formula (9), the first electronic device can determine the target loss, i.e. the LOSS value, between all first output activation values ​​and all second output activation values ​​in any simulation iteration group based on the first output activation value and the corresponding second output activation value of each network layer in each simulation iteration group.

[0196] It should be understood that, as can be seen from formula (8), dequantized w,j With weight quantization parameter scale w,j and offset w,j The purpose of LOSS is to minimize the difference between the first and second output activation values, that is, to make the floating-point output obtained by quantization closer to the original floating-point output. Therefore, the quantization parameter can be scaled. w,j and offset w,j As learnable parameters.

[0197] Based on the target loss, the third weight quantization parameter of each network layer is updated using the backpropagation algorithm to obtain the weight quantization parameter corresponding to each network layer in any simulation iteration group. This weight quantization parameter is the second weight quantization parameter corresponding to the second iteration group. It should be understood that the first data is usually input in batches. For each batch, the first electronic device calculates the LOSS once. If the LOSS value does not converge, the third weight quantization parameter of each network layer is updated using the backpropagation algorithm, and the process iterates through the next batch of first data. After multiple batches of first data, for any iteration group, when the obtained LOSS value converges (e.g., the LOSS value is less than or equal to a preset threshold), the corresponding weight quantization parameter can be used as the second weight quantization parameter for the second iteration group. This second weight quantization parameter and the first activation value quantization parameter are the quantization parameters required for the actual inference of the machine learning model. Therefore, the purpose of the current simulation inference stage is to determine the first activation value quantization parameter and the second weight quantization parameter corresponding to each iteration group.

[0198] In this implementation, the target loss is the value of the constructed loss function. By constructing the loss function, machine learning is used to make the first output activation value of each network layer in each simulation iteration of any simulation iteration group closer to the corresponding second output activation value. The first output activation value of each network layer in each simulation iteration of any simulation iteration group is related to the weight quantization parameter to be optimized. By optimizing the target loss, the weight quantization parameter that can reduce the output loss can be determined. This weight quantization parameter is then used as the second weight quantization parameter for the second iteration group in the real inference scenario. Based on the first activation value quantization parameter, the accuracy loss of the output can be further reduced, thereby improving the inference accuracy of the model.

[0199] For example, since the first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter and the second weight quantization parameter corresponding to each iteration group (i.e., the weight quantization parameter also varies depending on the iteration group), there is no need to pre-quantize the machine learning model in this scenario. After deployment on the hardware platform, when using the machine learning model for real inference, the hardware platform uses the second weight quantization parameter corresponding to the iteration group to which the current iteration belongs to quantize the weights and uses the first activation value quantization parameter corresponding to the iteration group to which the current iteration belongs to quantize the output activation value. That is, the machine learning model, the first activation value quantization parameter corresponding to M iteration groups, and the second weight quantization parameter constitute the target model. For example, the first activation value quantization parameter and the second weight quantization parameter corresponding to M iteration groups can be stored in a quantization parameter file, and the target model includes the machine learning model and the quantization parameter file.

[0200] In this implementation, each iteration group has a corresponding first activation value quantization parameter and a second weight quantization parameter. That is, the activation value quantization parameter and weight quantization parameter are different between groups. When the hardware platform executes each iteration, it can select different weight quantization parameters to quantize the weights of the machine learning model and different activation value quantization parameters to quantize the output activation values ​​of the machine learning model. Since the first activation value quantization parameter corresponding to each iteration group is closer to the distribution of the input data for that iteration group, it can reduce the accuracy loss of activation value quantization. Furthermore, the second weight quantization parameter corresponding to each group is learned with the aim of making the first output activation value and the corresponding second output activation value of each network layer in each simulation iteration group closer. Therefore, using it to quantize the weights of the machine learning model can further reduce the quantization accuracy loss of the model output, thereby improving the accuracy of model inference and prediction.

[0201] For example, when M equals N (i.e., each iteration is a separate group), the quantization parameters corresponding to N iterations in a real inference scenario can be found in [reference needed]. Figure 10A .like Figure 10A As shown, N iterations correspond to different activation value quantization parameters s a1 ,s a2 ,s a3 ,…,s aN N iterations correspond to different weight quantization parameters s w1 ,s w2 ,s w3 ,…,s wN When M is greater than 1 and less than N, at least one iteration belonging to the same iteration group corresponds to the same activation value quantization parameter. For the quantization parameters corresponding to M iteration groups in a real inference scenario, please refer to [reference needed]. Figure 10B .like Figure 10BAs shown, the first and second iterations form the first iteration group, the third iteration forms another iteration group, and the Nth iteration forms yet another iteration group. Therefore, the M iteration groups correspond to different activation value quantization parameters s. a1 ,s a2 ,…,s aM M iteration groups correspond to different weight quantization parameters s w1 ,s w2 ,…,s wM .

[0202] As can be seen in this embodiment, the first electronic device can use a machine learning model to perform simulated inference on the first data. Since the input of each iteration in the real inference scenario of the machine learning model is different, the input of the N simulation iterations in the simulated inference is also different. Based on the N simulation iterations with different inputs, the N iterations to be executed in the real inference scenario of the machine learning model are divided into M iteration groups, and then a corresponding quantization parameter (i.e., the first quantization parameter) is determined for each iteration group. The quantization parameters corresponding to the M iteration groups are different, and the quantization parameter corresponding to each iteration group is more in line with the distribution of the input data of that iteration group. When the hardware platform uses the machine learning model or the quantization model of the machine learning model to perform inference on the second data, it can select the quantization parameter corresponding to the first iteration group to which the i-th iteration belongs for quantization processing. Since the quantization parameter corresponding to the first iteration group to which the i-th iteration belongs is more in line with the distribution of the input data of the i-th iteration, using it for quantization processing can reduce the accuracy loss of the model output, thereby helping to improve the model inference accuracy and ensure the quality of the model output.

[0203] Please see Figure 11 , Figure 11 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method can be applied to a second electronic device or a chip within that device. The second electronic device can be used to execute... Figure 4 The same electronic device in the illustrated method embodiment can also be a different electronic device, such as an edge device deployed with a target model obtained based on a machine learning model. Figure 11 As shown, steps 1101-1102 are included:

[0204] 1101: Retrieve the second data.

[0205] The second data can be activation values ​​obtained based on user input, such as images, speech, or text to be processed. In one possible implementation, if the target model has an encoder, the second data can be the output activation values ​​of the target model's encoder after processing the user input. In another possible implementation, the second data can be the output activation values ​​of the VAE's encoder after processing the user input, such as in a diffusion model composed of the VAE and the target model.

[0206] For example, the target model is obtained based on the machine learning model in the foregoing embodiments. Figures 4-10B The illustrated embodiment describes a scenario where the weight quantization parameters are fixed. The weights of the machine learning model can be quantized using the first quantization parameter to obtain a quantized model. This quantized model, along with the first activation value quantization parameters corresponding to M iteration groups, constitutes the target model. The target model can be deployed locally on a first electronic device with determined quantization parameters, or it can be deployed on other hardware platforms, such as edge devices. Figure 12A The corresponding deployment methods are given.

[0207] For example, the quantization model is an intermediate representation of the graph structure, the IR model; each network layer of the machine learning model is represented as a node in the IR model, and the quantization parameter of the first activation value corresponding to each network layer in each of the N iterations serves as the attribute information of the node corresponding to each network layer. For a detailed description, please refer to... Figure 9 The embodiments shown are not described in detail here.

[0208] In this implementation, since the quantization model is obtained by quantizing the weights of the machine learning model through the first weight quantization parameter, when the hardware platform uses the quantization model for real inference, it only needs to select the first activation value quantization parameter corresponding to the current iteration round to quantize the activation value, without having to perform the weight quantization operation again, which helps to reduce the overhead of the hardware platform.

[0209] For example, based on Figures 4-10B The illustrated embodiment describes a scenario where the weight quantization parameters are not fixed. A corresponding second weight quantization parameter is also determined for each iteration group. Therefore, the first quantization parameter for each iteration group includes the first activation value quantization parameter and the second weight quantization parameter for each iteration group. The target model includes a machine learning model, the first activation value quantization parameter, and the second weight quantization parameter for M iteration groups. The target model can be deployed locally on the first electronic device whose quantization parameters are determined, or it can be deployed on other hardware platforms, such as edge devices. Figure 12B The corresponding deployment methods are given.

[0210] In this implementation, each iteration group has a corresponding first activation value quantization parameter and a second weight quantization parameter. That is, the activation value quantization parameter and weight quantization parameter are different between groups. When the hardware platform executes each iteration, it can select different weight quantization parameters to quantize the weights of the machine learning model and different activation value quantization parameters to quantize the output activation values ​​of the machine learning model. Since the first activation value quantization parameter corresponding to each iteration group is closer to the distribution of the input data for that iteration group, it can reduce the accuracy loss of activation value quantization. Furthermore, the second weight quantization parameter corresponding to each group is learned with the aim of making the first output activation value and the corresponding second output activation value of each network layer in each simulation iteration group closer. Therefore, using it to quantize the weights of the machine learning model can further reduce the quantization accuracy loss of the model output, thereby improving the accuracy of model inference and prediction.

[0211] 1102: The target model is used to infer the second data to perform N rounds of iterations to obtain the processing result of the second data; the output of the i-th iteration in the N rounds of iterations is the input of the (i+1)-th iteration; where N is greater than 1; i is greater than or equal to 1 and less than N; where the N rounds of iterations are divided into M iteration groups, each of the M iteration groups has a corresponding first quantization parameter, and the first quantization parameters corresponding to the M iteration groups are different; M is greater than 1 and less than or equal to N; when the target model performs the i-th iteration, it uses the first quantization parameter corresponding to the first iteration group in the M iteration groups for quantization processing; the first iteration group includes the i-th iteration.

[0212] The target model is based on the above. Figures 4-10B The first quantization parameter corresponding to the M iteration groups is obtained from the machine learning model described above. Figures 4-10B The method described above is used to obtain the results.

[0213] In the embodiments of this application, the target model is... Figure 12A When deployed as shown, Figure 5Taking the 3-layer network architecture shown as an example, the second data is used as the input of the first network layer of the quantization model. The first network layer processes the second data using the quantized weights to obtain the output activation value. The first activation value quantization parameter corresponding to the iteration group to which the l-th iteration belongs (the first activation value quantization parameter corresponding to the first network layer) is selected to quantize the output activation value of the first network layer. The quantized value is used as the input of the second network layer of the quantization model. The second network layer processes the input using the quantized weights to obtain the output activation value. The first activation value quantization parameter corresponding to the iteration group to which the l-th iteration belongs (the first activation value quantization parameter corresponding to the second network layer) is selected to quantize the output activation value of the second network layer. The quantized value is used as the input of the third network layer of the quantization model, and so on. After obtaining the quantized value of the output activation value of the third network layer, it is used as the input of the second iteration. If the second iteration belongs to the same iteration group as the first iteration, then the first activation value quantization parameter used is the same. If the second iteration belongs to a different iteration group than the first iteration, then the second iteration selects the first activation value quantization parameter corresponding to its iteration group for activation value quantization. This process is repeated N times to obtain the final processing result. For example, in a scenario where an image is generated using a diffusion model, the processing result of the second data can be a denoised image. Based on the iterative process, it can be seen that if the first iteration group includes multiple iterations, then its corresponding first activation value quantization parameter will be called multiple times.

[0214] In the embodiments of this application, the target model is... Figure 12B When deploying in a demonstrative manner, Figure 5 Taking the three-layer network architecture shown as an example, during the first iteration, the second weight quantization parameter corresponding to the iteration group to which the first iteration belongs is selected to quantize the weights of the machine learning model, resulting in a quantized model. This quantized model is then used to execute the first iteration. During execution, the first activation value quantization parameter corresponding to the iteration group to which the first iteration belongs is selected to quantize the output activation values ​​of each network layer. During the second iteration, if the second iteration belongs to the same iteration group as the first, the quantized model is used again for the second iteration. If the second iteration belongs to a different iteration group, the quantized model is dequantized, and the second weight quantization parameter corresponding to the second iteration is used to quantize the weights of the dequantized model, resulting in a new quantized model. This new quantized model is then used to execute the second iteration, and the first activation value quantization parameter corresponding to the iteration group to which the second iteration belongs is selected to quantize the output activation values ​​of each network layer. It can be seen that, unlike the scenario where the weight quantization parameter is fixed, when each iteration group has a corresponding second weight quantization parameter, online quantization of the machine learning model is required. If the first iteration group includes multiple iterations, its corresponding first activation value quantization parameter and second weight quantization parameter will also be called multiple times.

[0215] For example, since each iteration group has a corresponding index for its first quantization parameter, the second electronic device (such as a hardware platform) can call the first quantization parameter corresponding to its iteration group based on the index of the first quantization parameter corresponding to the current iteration round. The first quantization parameters called between the M iteration groups are different, while the first quantization parameters called within the same group are the same.

[0216] As can be seen in this embodiment, during the inference process of the second data, the hardware platform can select the first quantization parameter corresponding to the iteration group to which the i-th iteration belongs for quantization processing based on the round number of the i-th iteration. Since the quantization parameters corresponding to the M iteration groups are different, and the quantization parameter corresponding to each iteration group is more consistent with the distribution of the input data of that iteration group, the hardware platform can select the corresponding first quantization parameter for quantization processing based on the distribution differences of the input data of the M iteration groups. When the distribution of the quantization parameter is closer to that of the input data, using it for quantization processing can reduce the accuracy loss of the model output, thereby improving the model inference accuracy and ensuring the quality of the model output.

[0217] To verify the effectiveness of the data processing method provided in this application, researchers conducted image reconstruction experiments using a target model deployed on a device. Image reconstruction typically refers to filling in damaged or missing parts of an image with appropriate pixels to restore its integrity. Generally, it utilizes information from surrounding undamaged areas to infer and reconstruct the missing regions. The evaluation metrics primarily used were Learned Perceptual Image Patch Similarity (LPIPS) and Fréchet Inception Distance (FID). LPIPS is a metric for measuring perceptual differences between two images. It uses a deep learning model to evaluate the similarity of image features, thus better aligning with human visual perception. LPIPS values ​​are typically non-negative; lower values ​​indicate greater perceptual similarity between the two images, while higher values ​​indicate greater differences. FID is a metric for evaluating the performance of generative models, particularly in image generation tasks. Lower FID values ​​indicate a more similar distribution of the generated image to the real image in the feature space; conversely, higher FID values ​​indicate greater differences.

[0218] In one approach, the model inference rounds are defined as N=20, and each round is divided into an iteration group. Therefore, the quantization parameters for all 20 iterations are different. The performance data for static quantization techniques using fixed quantization parameters are shown in Table 1.

[0219] Table 1

[0220]

[0221] As shown in Table 1, compared with using fixed quantization parameters, LPIPS and FID both decreased significantly after 20 iterations using different quantization parameters. The lower the index value, the more similar the two images are perceptually, that is, the reconstructed image is closer to the original image, and the higher the quality of the model output.

[0222] In another approach, the model inference rounds N = 20 are defined, and these 20 iterations are divided into 2 groups, 4 groups, and 10 groups, with different quantization parameters for each group. The performance data compared to static quantization techniques using fixed quantization parameters are shown in Table 2.

[0223] Table 2

[0224]

[0225] As shown in Table 2, when the 20 iterations are divided into 2 groups, 4 groups, and 10 groups, and each group uses different quantization parameters, although the LPIPS and FID values ​​are not significantly different when divided into 20 groups, the values ​​are still lower than when divided into 1 group, meaning that the reconstructed image is relatively closer to the original image.

[0226] Please see Figure 13 , Figure 13 This is a schematic diagram of the structure of a quantization device for a machine learning model provided in an embodiment of this application. Figure 13 As shown, the device may include a simulation inference unit 1301, a grouping unit 1302, and a quantization parameter determination unit 1303; wherein:

[0227] The simulation inference unit 1301 is used to perform simulation inference on the first data using a machine learning model to execute N rounds of simulation iterations;

[0228] Grouping unit 1302 is used to divide N rounds of simulation iterations into M iteration groups;

[0229] The quantization parameter determination unit 1303 is used to determine the corresponding first quantization parameter for each of the M iteration groups; wherein, the first quantization parameters corresponding to the M iteration groups are different; M is greater than 1 and less than or equal to N; the first quantization parameter corresponding to the M iteration groups is used for quantization processing when the target model performs inference on the second data and executes the i-th iteration; the target model is obtained based on the machine learning model; the first iteration group includes the i-th iteration.

[0230] In one possible implementation, the first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter corresponding to each iteration group; the machine learning model has a fixed first weight quantization parameter;

[0231] like Figure 14 As shown, the device also includes a conversion unit 1304, which is used for:

[0232] The weights of the machine learning model are quantized using the first weight quantization parameter to obtain the quantized model; the target model includes the quantized model and the first activation value quantization parameter corresponding to M iteration groups.

[0233] In one possible implementation, the first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter and the second weight quantization parameter corresponding to each iteration group; the target model includes a machine learning model, the first activation value quantization parameter and the second weight quantization parameter corresponding to M iteration groups.

[0234] In one possible implementation, in terms of using a machine learning model to perform simulation inference on the first data to execute N rounds of simulation iterations, the simulation inference unit 1301 is specifically used for:

[0235] Using a machine learning model, perform the m-th simulation iteration in N simulation iterations through the following steps until m equals N:

[0236] The second activation value quantization parameter of the j-th network layer is determined based on the unprocessed activation value of the j-th network layer; where m is greater than or equal to 1 and less than or equal to N; j is greater than or equal to 1 and less than or equal to L; and L is the number of network layers in the machine learning model.

[0237] Obtain the third weight quantization parameter of the j-th network layer;

[0238] The activation value to be processed in the j-th network layer is quantized using the second activation value quantization parameter, and then the quantized value is dequantized to obtain the input activation value of the j-th network layer.

[0239] The first weight of the j-th network layer is quantized using the third weight quantization parameter, and then the quantized value is dequantized to obtain the second weight of the j-th network layer.

[0240] The input activation value of the j-th network layer is processed based on the second weight of the j-th network layer to obtain the first output activation value of the j-th network layer in the m-th simulation iteration; wherein, when j equals L, the m-th simulation iteration is completed.

[0241] Specifically, when m equals 1 and j equals 1, the activation value to be processed for the j-th network layer is the first data; when m does not equal 1 and j equals 1, the activation value to be processed for the j-th network layer is the first output activation value of the L-th network layer of the machine learning model in the (m-1)th simulation iteration; when m does not equal 1 and j does not equal 1, the activation value to be processed for the j-th network layer is the first output activation value of the (j-1)-th network layer of the machine learning model in the m-th simulation iteration.

[0242] In one possible implementation, regarding the division of N iterations into M iteration groups based on N simulation iterations, the grouping unit 1302 is specifically used for:

[0243] Based on the input activation values ​​of each network layer in N simulation iterations of the machine learning model, the distribution range of the input activation values ​​of each network layer in each simulation iteration is determined.

[0244] Based on the distribution range of the input activation values ​​of each network layer in each simulation iteration, the distribution distance of the input activation values ​​in N simulation iterations is determined;

[0245] The simulation iteration group is defined as the simulation iteration group in which the distribution distance of the input activation values ​​in N simulation iterations is less than or equal to a preset threshold.

[0246] The at least one iteration in N iterations whose iteration number is the same as the iteration number of at least one simulation iteration is identified as an iteration group, so that M iteration groups are obtained.

[0247] In one possible implementation, the M iteration groups correspond one-to-one with the M simulation iteration groups in terms of the number of iterations they include. The M simulation iteration groups are obtained by dividing at least one of the N simulation iterations into one simulation iteration group. Regarding determining the corresponding first quantization parameter for each of the M iteration groups, the quantization parameter determination unit 1303 is specifically used for:

[0248] For any one of the M simulation iteration groups, based on the distribution range of the input activation values ​​of each network layer in each round of simulation iteration in any simulation iteration group, determine the maximum and minimum input activation values ​​of each network layer in any simulation iteration group.

[0249] Based on the maximum and minimum input activation values ​​and the preset data bit width of each network layer in any simulation iteration group, the activation value quantization parameter corresponding to each network layer in any simulation iteration group is determined; the activation value quantization parameter is the first activation value quantization parameter corresponding to the second iteration group; the second iteration group is the iteration group corresponding to any simulation iteration group among M iteration groups.

[0250] In one possible implementation, each network layer has a corresponding second output activation value in each of the N simulation iterations; the second output activation value is obtained by performing simulation inference on the first data using the machine learning model without any quantization before performing simulation inference on the first data using the machine learning model to execute the N simulation iterations; the quantization parameter determination unit 1303 is further used for:

[0251] Obtain the second output activation value of each network layer in each round of simulation iteration in any simulation iteration group;

[0252] The target loss is determined based on the first output activation value of each network layer in each simulation iteration in any simulation iteration group and the corresponding second output activation value of each network layer in each simulation iteration in any simulation iteration group.

[0253] Based on the target loss, the third weight quantization parameter of each network layer is updated through the backpropagation algorithm to obtain the weight quantization parameter of each network layer in any simulation iteration group; the weight quantization parameter is the second weight quantization parameter corresponding to the second iteration group.

[0254] In one possible implementation, the quantization model is an intermediate representation of the graph structure, the IR model; each network layer of the machine learning model is represented as a node in the IR model, and the quantization parameter of the first activation value corresponding to each network layer in each of the N iterations is used as the attribute information of the node corresponding to each network layer.

[0255] It should be noted that, Figure 13 or Figure 14 The implementation of each unit described can also be referenced accordingly. Figures 4 to 10B The corresponding description of the illustrated embodiments. Furthermore, Figure 13 or Figure 14 The beneficial effects of the described device can be referred to Figures 4 to 10B The corresponding descriptions of the embodiments shown will not be repeated here.

[0256] Please see Figure 15 , Figure 15 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 15 As shown, the device may include an acquisition unit 1501 and a processing unit 1502; wherein:

[0257] Acquisition unit 1501 is used to acquire the second data;

[0258] Processing unit 1502 is used to perform inference on the second data using the target model to execute N rounds of iteration and obtain the processing result of the second data; the output of the i-th iteration in the N rounds of iteration is the input of the (i+1)-th iteration; where N is greater than 1; i is greater than or equal to 1 and less than N;

[0259] The N iterations are divided into M iteration groups. Each iteration group has a corresponding first quantization parameter. The first quantization parameters of the M iteration groups are different. M is greater than 1 and less than or equal to N. When the target model performs the i-th iteration, it uses the first quantization parameter corresponding to the first iteration group in the M iteration groups for quantization. The first iteration group includes the i-th iteration.

[0260] The target model is obtained based on the machine learning model described in the first aspect above; the first quantization parameters corresponding to the M iteration groups are obtained based on the method described in the first aspect above.

[0261] In one possible implementation, the first quantization parameter corresponding to the M iteration groups includes the first activation value quantization parameter corresponding to each iteration group; the machine learning model has a fixed first weight quantization parameter; the target model includes the quantization model and the first activation value quantization parameter corresponding to the M iteration groups; the quantization model is obtained by quantizing the weights of the machine learning model using the first weight quantization parameter.

[0262] In one possible implementation, the first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter and the second weight quantization parameter corresponding to each iteration group; the target model includes a machine learning model, the first activation value quantization parameter and the second weight quantization parameter corresponding to M iteration groups.

[0263] In one possible implementation, the quantization model is an intermediate representation of the graph structure, the IR model; each network layer of the machine learning model is represented as a node in the IR model, and the quantization parameter of the first activation value corresponding to each network layer in each of the N iterations is used as the attribute information of the node corresponding to each network layer.

[0264] In one possible implementation, N rounds of iteration are performed by using a machine learning model to perform simulation inference on the first data to execute N rounds of simulation iterations, which are divided into M iteration groups.

[0265] It should be noted that, Figure 15 The implementation of each unit described can also be referenced accordingly. Figures 11 to 12B The corresponding description of the illustrated embodiments. Furthermore, Figure 15 The beneficial effects of the described device can be referred to Figures 11 to 12B The corresponding descriptions of the embodiments shown will not be repeated here.

[0266] Based on the description of the method and device embodiments above, this application also provides an electronic device. Please refer to... Figure 16 , Figure 16 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device includes at least a processor 1601, a memory 1602, and a communication interface 1603, which are interconnected via a bus 1604. This electronic device can be used to execute steps related to quantization methods or data processing methods for machine learning models. The electronic device can be a desktop computer, laptop computer, server, cloud server, or other computer equipment, or it can be a chip or module from such computer equipment. The electronic device can also be an edge device such as a tablet, smartphone, wearable device, or in-vehicle device. The processor 1601 in this electronic device is used to read the computer program code stored in the memory 1602 and execute it. Figures 4 to 12B The method of any of the embodiments shown.

[0267] The memory 1602 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and is used to store related computer programs and data.

[0268] Processor 1601 can be one or more central processing units (CPUs). If processor 1601 is a CPU, the CPU can be a single-core CPU or a multi-core CPU.

[0269] For example, when the electronic device is the first electronic device described above, the processor 1601 in the electronic device can be used to read one or more programs stored in the memory 1602 and perform the following operations:

[0270] The machine learning model is used to perform simulation inference on the first data to execute N rounds of simulation iterations;

[0271] Based on N rounds of simulation iterations, the N rounds of iterations are divided into M iteration groups, and a corresponding first quantization parameter is determined for each of the M iteration groups; wherein, the first quantization parameters corresponding to the M iteration groups are different; M is greater than 1 and less than or equal to N; the first quantization parameters corresponding to the M iteration groups are used in the process of the target model inferring the second data, and the first quantization parameters corresponding to the first iteration group in the M iteration groups are used for quantization processing when the i-th iteration is executed; the target model is obtained based on the machine learning model; the first iteration group includes the i-th iteration.

[0272] It should be noted that the implementation of each operation can also be referenced accordingly. Figures 4 to 10B A corresponding description of the method of any of the embodiments shown.

[0273] For example, when the electronic device is the second electronic device described above, the processor 1601 in the electronic device can be used to read one or more programs stored in the memory 1602 and perform the following operations:

[0274] Obtain the second data;

[0275] The target model is used to infer the second data to perform N rounds of iteration, and the processing result of the second data is obtained; the output of the i-th iteration in the N rounds of iteration is the input of the (i+1)-th iteration; where N is greater than 1; i is greater than or equal to 1 and less than N;

[0276] The N iterations are divided into M iteration groups. Each iteration group has a corresponding first quantization parameter. The first quantization parameters of the M iteration groups are different. M is greater than 1 and less than or equal to N. When the target model performs the i-th iteration, it uses the first quantization parameter corresponding to the first iteration group in the M iteration groups for quantization. The first iteration group includes the i-th iteration.

[0277] The target model is based on the above. Figures 4 to 10B The first quantization parameter corresponding to the M iteration groups is obtained from the machine learning model described above. Figures 4 to 10B The method described above is used to obtain the results.

[0278] It should be noted that, although Figure 16 The illustrated electronic device only shows the processor 1601, memory 1602, communication interface 1603, and bus 1604. However, in specific implementations, those skilled in the art should understand that the electronic device may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the electronic device may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the electronic device may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 16All the devices shown.

[0279] This application also provides a chip, including: a processor, configured to retrieve and run a computer program from memory, causing a device with the chip installed to perform the above-described actions. Figures 4 to 10B or Figures 11 to 12B The method described in any of the embodiments. The chip may be a chip in an electronic device.

[0280] This application also provides a computer-readable storage medium (memory) storing a computer program, which, when run, performs the functions described above. Figures 4 to 10B or Figures 11 to 12B The method described in any of the embodiments is described above. It is understood that the computer-readable storage medium herein may include both built-in storage media within the device and extended storage media supported by the device. The computer-readable storage medium provides storage space containing the operating system of the device. Furthermore, one or more computer programs suitable for loading and execution by the processor of the device are also stored in this storage space. It should be noted that the computer-readable storage medium herein may be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it may also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0281] This application also provides a computer program product, which includes computer program code. When the computer program code is run by an electronic device,... Figures 4 to 10B or Figures 11 to 12B The method flow described in any one of the embodiments can be implemented.

[0282] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0283] It should be understood that the processor mentioned in the embodiments of this application can be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0284] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be ROM, Programmable Read-Only Memory (PROM), EPROM, Electrically Erasable Programmable Read-Only Memory (EEPROM), or flash memory. Volatile memory can be RAM, which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchlink Dynamic Random Access Memory (SLDRAM), and Direct Rambus RAM (DR RAM).

[0285] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) is integrated into the processor.

[0286] It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0287] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0288] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely exemplary. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0289] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0290] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0291] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0292] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0293] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0294] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A quantization method for a machine learning model, characterized in that, The machine learning model needs to perform N rounds of iteration for inference, and the output of the i-th iteration is the input of the (i+1)-th iteration; where N is greater than 1; i is greater than or equal to 1 and less than N; The method includes: The machine learning model is used to perform simulation inference on the first data to execute N rounds of simulation iteration; Based on the N rounds of simulation iterations, the N rounds of iterations are divided into M iteration groups, and a corresponding first quantization parameter is determined for each of the M iteration groups; wherein, the first quantization parameters corresponding to the M iteration groups are different; M is greater than 1 and less than or equal to N; the first quantization parameters corresponding to the M iteration groups are used for quantization processing when the target model infers the second data and the first iteration group in the M iteration groups is executed during the i-th iteration; the target model is obtained based on the machine learning model; the first iteration group includes the i-th iteration.

2. The method according to claim 1, characterized in that, The first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter corresponding to each iteration group; the machine learning model has a fixed first weight quantization parameter. After determining the corresponding first quantization parameter for each of the M iteration groups, the method further includes: The weights of the machine learning model are quantized using the first weight quantization parameter to obtain a quantized model; The target model includes the quantization model and the first activation value quantization parameters corresponding to the M iteration groups.

3. The method according to claim 1, characterized in that, The first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter and the second weight quantization parameter corresponding to each iteration group; the target model includes the machine learning model, the first activation value quantization parameter and the second weight quantization parameter corresponding to the M iteration groups.

4. The method according to any one of claims 1-3, characterized in that, The step of using the machine learning model to perform simulation inference on the first data to execute N rounds of simulation iterations includes: Using the machine learning model, the m-th simulation iteration in the N rounds of simulation iterations is performed through the following steps until m equals N: The second activation value quantization parameter of the j-th network layer is determined based on the unprocessed activation value of the j-th network layer of the machine learning model; where m is greater than or equal to 1 and less than or equal to N; j is greater than or equal to 1 and less than or equal to L; and L is the number of network layers in the machine learning model. Obtain the third weight quantization parameter of the j-th network layer; The activation value to be processed in the j-th network layer is quantized using the second activation value quantization parameter, and then the quantized value is dequantized to obtain the input activation value of the j-th network layer. The first weight of the j-th network layer is quantized using the third weight quantization parameter, and then the quantized value is dequantized to obtain the second weight of the j-th network layer. The input activation value of the j-th network layer is processed based on the second weight of the j-th network layer to obtain the first output activation value of the j-th network layer in the m-th simulation iteration; wherein, when j equals L, the m-th simulation iteration is completed; Wherein, when m equals 1 and j equals 1, the activation value to be processed of the j-th network layer is the first data; when m does not equal 1 and j equals 1, the activation value to be processed of the j-th network layer is the first output activation value of the L-th network layer of the machine learning model in the (m-1)-th simulation iteration; when m does not equal 1 and j does not equal 1, the activation value to be processed of the j-th network layer is the first output activation value of the (j-1)-th network layer of the machine learning model in the m-th simulation iteration.

5. The method according to any one of claims 2-4, characterized in that, The process of dividing the N rounds of simulation iterations into M iteration groups includes: Based on the input activation values ​​of each network layer in the N simulation iterations of the machine learning model, the distribution range of the input activation values ​​of each network layer in each simulation iteration is determined. Based on the distribution range of the input activation values ​​of each network layer in each simulation iteration, the distribution distance of the input activation values ​​in the N simulation iterations is determined; The simulation iteration group is defined as at least one simulation iteration in which the distribution distance of the input activation values ​​in the N simulation iterations is less than or equal to a preset threshold. The at least one iteration in the N iterations whose iteration number is the same as the iteration number of the at least one simulation iteration is determined as an iteration group to obtain the M iteration groups.

6. The method according to claim 5, characterized in that, The M iteration groups correspond one-to-one with the M simulation iteration groups in terms of the number of iterations included. The M simulation iteration groups are obtained by dividing at least one simulation iteration in the N simulation iterations into one simulation iteration group. Determining the corresponding first quantization parameter for each of the M iteration groups includes: For any one of the M simulation iteration groups, based on the distribution range of the input activation values ​​of each network layer in each round of simulation iteration in the simulation iteration group, determine the maximum and minimum input activation values ​​of each network layer in the simulation iteration group. Based on the maximum and minimum input activation values ​​and the preset data bit width of each network layer in any simulation iteration group, the activation value quantization parameter corresponding to each network layer in any simulation iteration group is determined; the activation value quantization parameter is the first activation value quantization parameter corresponding to the second iteration group; the second iteration group is the iteration group among the M iteration groups that corresponds to any simulation iteration group.

7. The method according to claim 6, characterized in that, Each network layer has a corresponding second output activation value in each of the N simulation iterations; The second output activation value is obtained by performing simulation inference on the first data using the machine learning model without any quantization before performing simulation inference on the first data using the machine learning model to execute N rounds of simulation iterations; The method further includes: Obtain the second output activation value of each network layer in each round of simulation iteration in any simulation iteration group; The target loss is determined based on the first output activation value of each network layer in each round of simulation iteration in any simulation iteration group and the corresponding second output activation value of each network layer in each round of simulation iteration in any simulation iteration group; Based on the target loss, the third weight quantization parameter of each network layer is updated using the backpropagation algorithm to obtain the weight quantization parameter of each network layer in any simulation iteration group; the weight quantization parameter is the second weight quantization parameter corresponding to the second iteration group.

8. The method according to claim 2, 4, 5 or 7, characterized in that, The quantization model is an intermediate representation (IR) model of a graph structure; each network layer of the machine learning model is represented as a node in the IR model, and the first activation value quantization parameter corresponding to each iteration of each network layer in the N iterations is used as the attribute information of the node corresponding to each network layer.

9. A data processing method, characterized in that, The method includes: Obtain the second data; The target model is used to infer the second data to perform N rounds of iteration, and the processing result of the second data is obtained; the output of the i-th iteration in the N rounds of iteration is the input of the (i+1)-th iteration; where N is greater than 1; i is greater than or equal to 1 and less than N; The N iterations are divided into M iteration groups, each of the M iteration groups has a corresponding first quantization parameter, and the first quantization parameters corresponding to the M iteration groups are different; M is greater than 1 and less than or equal to N; when the target model performs the i-th iteration, it uses the first quantization parameter corresponding to the first iteration group in the M iteration groups for quantization processing; the first iteration group includes the i-th iteration; The target model is obtained based on the machine learning model of any one of claims 1-8; the first quantization parameter corresponding to the M iteration groups is obtained based on the method of any one of claims 1-8.

10. The method according to claim 9, characterized in that, The first quantization parameter corresponding to the M iteration groups includes the first activation value quantization parameter corresponding to each iteration group; the machine learning model has a fixed first weight quantization parameter; the target model includes a quantization model and the first activation value quantization parameter corresponding to the M iteration groups; the quantization model is obtained by quantizing the weights of the machine learning model using the first weight quantization parameter.

11. The method according to claim 9, characterized in that, The first quantization parameter corresponding to each iteration group includes the first activation value quantization parameter and the second weight quantization parameter corresponding to each iteration group; the target model includes the machine learning model and the first activation value quantization parameter and the second weight quantization parameter corresponding to the M iteration groups.

12. The method according to claim 10, characterized in that, The quantization model is an intermediate representation (IR) model of a graph structure; each network layer of the machine learning model is represented as a node in the IR model, and the first activation value quantization parameter corresponding to each iteration of each network layer in the N iterations is used as the attribute information of the node corresponding to each network layer.

13. The method according to any one of claims 9-12, characterized in that, The N rounds of iteration are performed by using the machine learning model to simulate and infer the first data, and the N rounds of simulation iteration are divided into the M iteration groups.

14. A quantization device for a machine learning model, characterized in that, The machine learning model requires N iterations for inference, where the output of the i-th iteration is the input of the (i+1)-th iteration; where N is greater than 1; i is greater than or equal to 1 and less than N; the device includes a simulation inference unit, a grouping unit, and a quantization parameter determination unit; wherein: The simulation inference unit is used to perform simulation inference on the first data using the machine learning model to execute N rounds of simulation iteration; The grouping unit is used to divide the N rounds of simulation iterations into M iteration groups based on the N rounds of simulation iterations; The quantization parameter determination unit is used to determine a corresponding first quantization parameter for each of the M iteration groups; wherein the first quantization parameters corresponding to the M iteration groups are different; M is greater than 1 and less than or equal to N; the first quantization parameter corresponding to the M iteration groups is used for quantization processing when the target model performs inference on the second data and executes the i-th iteration; the target model is obtained based on the machine learning model; the first iteration group includes the i-th iteration.

15. A data processing apparatus, characterized in that, The device includes an acquisition unit and a processing unit; wherein: The acquisition unit is used to acquire the second data; The processing unit is used to perform inference on the second data using the target model to execute N rounds of iterations to obtain the processing result of the second data; the output of the i-th iteration in the N rounds of iterations is the input of the (i+1)-th iteration; where N is greater than 1; i is greater than or equal to 1 and less than N; The N iterations are divided into M iteration groups, each of the M iteration groups has a corresponding first quantization parameter, and the first quantization parameters corresponding to the M iteration groups are different; M is greater than 1 and less than or equal to N; when the target model performs the i-th iteration, it uses the first quantization parameter corresponding to the first iteration group in the M iteration groups for quantization processing; the first iteration group includes the i-th iteration; The target model is obtained based on the machine learning model of any one of claims 1-8; the first quantization parameter corresponding to the M iteration groups is obtained based on the method of any one of claims 1-8.

16. An electronic device, characterized in that, The device includes a processor, a memory, a communication interface, and one or more programs stored in the memory and configured to, when executed by the processor, cooperate with the communication interface to implement the method as described in any one of claims 1-8 or 9-13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for execution by the device, which, when executed, implements the method of any one of claims 1-8 or 9-13.

18. A computer program product, characterized in that, When the computer program product is run by the device, the device performs the method as claimed in any one of claims 1-8 or 9-13.

Images