Quantization-based data processing method and apparatus, and device and medium

Abstract

Disclosed in the embodiments of the present application are a quantization-based data processing method and apparatus, and a device and a medium. The method comprises: using a plurality of bit quantization modes to respectively perform quantization processing on an original model parameter corresponding to a model function network, so as to obtain a quantized model parameter corresponding to each bit quantization mode; then, acquiring a first output result of the model function network under the original model parameter, and acquiring a second output result of the model function network under each quantized model parameter; and next, on the basis of the first output result and a plurality of second output results, selecting a target bit quantization mode from among the plurality of bit quantization modes, wherein the target bit quantization mode is used for performing quantization processing on the original model parameter during an inference phase of the model function network. The technical solution of the present application improves the accuracy of quantization processing, and achieves good model performance.

Description

Quantization-based data processing method and device, equipment and medium

[0001] The present application claims priority to the Chinese patent application No. 202411010046.1, filed on July 26, 2024, and entitled "Quantization-based data processing method, device, equipment and medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the technical field of computer, in particular to a quantization-based data processing method, a quantization-based data processing device, an electronic device, and a computer readable medium. BACKGROUND

[0003] In related technologies, two schemes are usually used to determine the quantization bit mode corresponding to different function networks of a mixture of experts (MoE). In scheme one, a user selects a quantization bit mode for each function network according to artificial experience. In scheme two, a function network is selected a quantization bit mode according to its task type, for example, a function network with a task type requiring high precision is selected a quantization bit mode with high bit (such as fp16 or int8), and a function network with a task type requiring low precision is selected a quantization bit mode with low bit (such as int4 or int2).

[0004] In scheme one, the quantization bit mode of a function network may be selected incorrectly due to insufficient artificial experience, which reduces the accuracy of quantization processing and further affects the model performance. In scheme two, the sensitivity of a function network changes even for the same task type due to different models or different training stages of a model, so selecting a quantization bit mode for a function network according to its task type is relatively fixed, which also has the problem of incorrect selection of the quantization bit mode, reduces the accuracy of quantization processing, and further affects the model performance.

[0005] Therefore, how to improve the accuracy of quantization processing to improve the model performance is a problem to be solved. SUMMARY

[0006] Embodiments of the present application provide a quantization-based data processing method, device, equipment and medium, which improves the accuracy of quantization processing and has good model performance.

[0007] In a first aspect, an embodiment of the present application provides a data processing method based on quantization, the method comprising: performing quantization processing on original model parameters corresponding to a model function network through a plurality of quantization bit modes respectively to obtain quantized model parameters corresponding to each quantization bit mode; obtaining a first output result of the model function network under the original model parameters, and obtaining second output results of the model function network under each quantized model parameter; and selecting a target quantization bit mode from the plurality of quantization bit modes based on the first output result and the plurality of second output results, the target quantization bit mode being used for performing quantization processing on the original model parameters in an inference stage of the model function network.

[0008] In a second aspect, an embodiment of the present application provides a data processing apparatus based on quantization, the apparatus comprising: a quantization processing module configured to perform quantization processing on original model parameters corresponding to a model function network through a plurality of quantization bit modes respectively to obtain quantized model parameters corresponding to each quantization bit mode; an obtaining module configured to obtain a first output result of the model function network under the original model parameters, and obtain second output results of the model function network under each quantized model parameter; and a selecting module configured to select a target quantization bit mode from the plurality of quantization bit modes based on the first output result and the plurality of second output results, the target quantization bit mode being used for performing quantization processing on the original model parameters in an inference stage of the model function network.

[0009] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors; and a memory configured to store one or more computer programs, when the one or more computer programs are executed by the one or more processors, the electronic device is caused to implement the data processing method based on quantization as described above.

[0010] In a fourth aspect, an embodiment of the present application provides a computer readable medium having stored thereon a computer program, when the computer program is executed by a processor, the data processing method based on quantization as described above is implemented.

[0011] In a fifth aspect, an embodiment of the present application provides a computer program product comprising computer instructions, when the computer instructions are executed by a processor, the data processing method based on quantization as described above is implemented.

[0012] In the technical solutions provided by the embodiments of the present application, the original model parameters corresponding to the model are quantized in multiple quantization bit manners to obtain quantized model parameters corresponding to each quantization bit manner, and the first output result of the model function network under the original model parameters and the second output result of the model function network under each quantized model parameter are used to select a target quantization bit manner from the multiple quantization bit manners. In this way, the accurate selection of the quantization bit manner of the model function network is realized, the accuracy of the quantization processing is high, and the high accuracy of the quantization processing helps to improve the generalization ability of the model beyond the training data, better adapts to new data distribution and scenarios, enhances the stability and robustness of the model, maintains good performance under complex conditions, further enables faster inference, reduces unnecessary resource consumption, improves resource utilization, greatly improves the performance of the model, and enables tasks in various application scenarios to be well performed by the model.

[0013] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application.

[0014] BRIEF DESCRIPTION OF DRAWINGS

[0015] FIG. 1 is a schematic diagram of an exemplary implementation environment to which the technical solutions of the present application can be applied.

[0016] FIG. 2 is a flowchart of a data processing method based on quantization according to an exemplary embodiment of the present application.

[0017] FIG. 3a is a schematic diagram of a data channel including channel values according to an exemplary embodiment of the present application.

[0018] FIG. 3b is a schematic diagram of a data channel including channel values according to another exemplary embodiment of the present application.

[0019] FIG. 4 is a flowchart of a data processing method based on quantization according to another exemplary embodiment of the present application.

[0020] FIG. 5 is a schematic diagram of generating a target quantization scaling factor according to an exemplary embodiment of the present application.

[0021] FIG. 6 is a flowchart of a data processing method based on quantization according to another exemplary embodiment of the present application.

[0022] FIG. 7 is a schematic diagram of generating a target quantization bit manner according to an exemplary embodiment of the present application.

[0023] FIG. 8 is a block diagram of a data processing apparatus based on quantization according to an exemplary embodiment of the present application.

[0024] FIG. 9 is a structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0025] The exemplary embodiments will be described in detail herein below with reference to the drawings. In the following description, the same drawings refer to the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments in accordance with the present application. Rather, they are merely examples of apparatuses and methods in accordance with some aspects of the present application, as detailed in the appended claims.

[0026] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program having a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented in whole or in part by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0027] The block diagrams shown in the drawings are merely functional entities, and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the drawings are merely exemplary illustrations, and do not necessarily include all contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to the actual situation.

[0029] It should be noted that "multiple" as mentioned in the embodiments of the present application refers to two or more. The association relationship of "and / or" describes the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0030] Before introducing the technical solutions of the embodiments of the present application, the terms and nomenclature involved in the embodiments of the present application are explained, which are applicable to the following explanations.

[0031] A mixture of experts (MoE) is a neural network architecture used to solve complex tasks by dividing the artificial intelligence (AI) model into separate sub-networks (or expert networks), each of which is responsible for processing a subset of input data to collectively perform the task. The mixture of experts includes multiple expert networks and a gating network, each of which is responsible for handling different aspects or sub-tasks of input data, such as edge detection in image processing tasks, and another expert network for texture recognition. The gating network is responsible for dynamically selecting one of the multiple expert networks to process the current input data. The mixture of experts effectively improves the model's ability to handle complex tasks and performance by integrating the gating network and multiple expert networks.

[0032] Hybrid bit quantization refers to the mixed processing of analog and digital signals, which compresses each segment using different quantization methods by segmenting the input signal into multiple levels, and finally obtains a digital signal composed of multiple binary codes with different bit numbers. Hybrid bit quantization can effectively improve data compression rate and signal quality. Hybrid bit quantization is a method for optimizing model inference, which can be applied to the mixture of experts, allowing different quantization precisions to be used in the same model to balance the computational efficiency and accuracy requirements of the model. Specifically, during model inference, different expert networks (also known as functional networks) can use different quantization precisions, such as int8 (an 8-bit integer representation method) for some functional networks, int4 (a 4-bit integer representation method) for some functional networks, int2 (a 2-bit integer representation method) for some functional networks, and fp16 (Fixed Point 16-bit, a 16-bit floating-point representation method) for some functional networks.

[0033] The prior art uses the two schemes described above to select the quantization bit method for functional networks, but the accuracy of the quantization process is not high, which affects the model performance. Therefore, in order to improve the accuracy of the quantization process and improve the model performance, the embodiment of the present application provides a data processing scheme based on quantization. Please refer to FIG. 1, which is a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes a terminal device 101 and a server 102.

[0034] The terminal device 101 includes but is not limited to a smartphone, a computer (a tablet computer, a notebook computer, a desktop computer, etc.), a smart home device (a television, a refrigerator, an air conditioner, a washing machine, a sweeping and mopping robot, etc.), a smart wearable device (a bracelet, a watch, etc.), etc.

[0035] The server 102 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, wherein the server cluster or distributed system includes cloud servers for providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0036] In an embodiment, the terminal device 101 and the server 102 establish a communication connection through a wired or wireless network. Exemplarily, the wireless network or the wired network uses standard communication technologies and / or protocols. The network is usually the Internet, but can also be any other network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network, or any combination thereof.

[0037] In an embodiment of the present application, the data processing method based on quantization can be executed by the server 102. Specifically, the server 102 quantizes the original model parameters corresponding to the model function network through a plurality of quantization bit modes respectively, to obtain quantized model parameters corresponding to each quantization bit mode; then obtains a first output result of the model function network under the original model parameters, and obtains a second output result of the model function network under each quantized model parameter; and then selects a target quantization bit mode from the plurality of quantization bit modes based on the first output result and the plurality of second output results, the target quantization bit mode being used for quantizing the original model parameters in the inference stage of the model function network.

[0038] In an embodiment of the present application, the data processing method based on quantization can also be executed by the terminal device 101 alone or by the terminal device 101 and the server 102 jointly. In actual application, the execution subject of the data processing method based on quantization can be flexibly adjusted according to specific application scenarios.

[0039] The number of terminal devices 101 and servers 102 shown in FIG. 1 is only illustrative, and any number of terminal devices 101 and servers 102 can be provided according to actual needs.

[0040] It should be noted that in the specific embodiments of the present application, data related to the user is involved, and when the embodiments of the present application are applied to specific products or technologies, the user's permission or consent needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region.

[0041] The various implementation details of the technical solutions of the embodiments of the present application are described in detail below.

[0042] Please refer to FIG. 2, which is a flowchart of the data processing method based on quantization according to an embodiment of the present application. As shown in FIG. 2, the data processing method based on quantization at least includes steps S201 to S203, which are described in detail as follows.

[0043] S201, quantize the original model parameters corresponding to the model function network respectively by using multiple quantization bit modes to obtain quantized model parameters corresponding to each quantization bit mode.

[0044] In the embodiments of the present application, the quantization bit mode includes at least one of, but is not limited to, integer representation of any bit, floating point number representation of any bit, etc., wherein the any bit can be 2, 3, 4, 6, 8, 16, 32, 64, 128, etc.

[0045] In the embodiments of the present application, the original model parameter refers to the model parameter of the functional network (referred to as model function network) of the hybrid expert model before the model parameter is quantized by using the quantization bit mode, and correspondingly, the quantized model parameter refers to the model parameter of the model function network after the model parameter is quantized by using the quantization bit mode. It can be understood that the model parameter includes at least one of, but is not limited to, weight value (weight), activation input value (activation), cache value (KV-Cache), etc. In the embodiments of the present application, the model parameter is taken as the weight value for example, the original model parameter is the original weight value, and the quantized model parameter is the quantized weight value, and other examples can be similarly deduced.

[0046] In the embodiments of the present application, the original model parameter is quantized by using multiple quantization bit modes respectively, so as to obtain quantized model parameters corresponding to the multiple quantization bit modes respectively.

[0047] For example, assuming that the multiple quantization bit modes are int2, int4 and int8, and w0 represents the original weight value, and winti represents the quantized weight value corresponding to the quantization bit mode inti; then the quantized weight value corresponding to int2 is wint2=int2(w0), the quantized weight value corresponding to int4 is wint4=int4(w0), and the quantized weight value corresponding to int8 is wint8=int8(w0).

[0048] S202, obtaining a first output result of the model function network under the original model parameter, and obtaining a second output result of the model function network under each quantized model parameter.

[0049] In the embodiments of the present application, after obtaining the quantized model parameter corresponding to each quantization bit mode, the first output result of the model function network under the original model parameter can be obtained, and the second output result of the model function network under each quantized model parameter can be obtained.

[0050] In the embodiments of the present application, the first output result refers to the result output by the model function network under the original model parameter, and the second output result refers to the result output by the model function network under the quantized model parameter.

[0051] In the embodiments of the present application, the original model parameter includes an original weight value, and the quantized model parameter includes a quantized weight value; accordingly, the process of obtaining the first output result of the model function network under the original model parameter and obtaining the second output result of the model function network under each quantized model parameter in S202 can include:

[0052] Obtaining an activation input value of the model function network;

[0053] Performing a multiplication operation on the activation input value and the original weight value to obtain the first output result of the model function network, and performing a multiplication operation on the activation input value and each quantized weight value to obtain a plurality of second output results of the model function network.

[0054] For example, taking the foregoing example, let a represent the activation input value, let out0 represent the first output result, and let outinti represent the second output result corresponding to the quantization bit mode inti; then the first output result is out0=a×w0, the second output result corresponding to int2 is outint2=a×wint2, the second output result corresponding to int4 is outint4=a×wint4, and the second output result corresponding to int8 is outint8=a×wint8.

[0055] In this way, the first output result and the plurality of second output results can be obtained simply and accurately through the embodiments.

[0056] In the embodiments of the present application, the model function network corresponds to a plurality of data channels, and each data channel corresponds to one or more channel values. In specific embodiments, each data channel corresponds to a plurality of channel values, for example, at least three channel values. The channel value refers to a key parameter used to distinguish different types of channel data when reading the channel data of the data channel.

[0057] In an embodiment, the process of obtaining the activation input value of the model function network can include:

[0058] detecting whether there is an outlier channel value in each of the plurality of data channels; wherein the outlier channel value refers to at least one channel value in a plurality of channel values corresponding to a data channel, which is significantly different from other channel values, for example, the difference between the at least one channel value and other channel values is greater than a preset threshold;

[0059] If it is detected that there is an outlier channel value, compare all detected outlier channel values, take the data channel corresponding to the maximum outlier channel value as the target data channel, take the channel value corresponding to the target data channel as the activation input value of the model function network, and obtain the original weight value and the quantized weight value corresponding to the target data channel. For example, as described in the foregoing example, let a represent the activation input value, a is a vector matrix [N, M], where N is a parameter related to the input of the model function network, and M is a pre-set size of the hidden layer of the model function network. In the embodiments of the present application, N and M are integers greater than 1. N can be the length of the text sequence input into the model function network determined according to the channel value corresponding to the target data channel, for example, if the length of the text sequence is 4096 characters, then N is equal to 4096, and M can be a pre-set value, for example, 2048, 4096, etc.

[0060] Correspondingly, the process of performing multiplication operation on the activation input value and the original weight value to obtain the first output result of the model function network, and performing multiplication operation on the activation input value and each quantized weight value to obtain a plurality of second output results of the model function network can include:

[0061] performing multiplication operation on the channel value corresponding to the target data channel and the original weight value to obtain the first output result of the model function network, and performing multiplication operation on the channel value corresponding to the target data channel and each quantized weight value to obtain a plurality of second output results of the model function network.

[0062] In the embodiments, the outlier channel value is for the data channel. For example, referring to FIG. 3a, the data channel c1 includes a plurality of channel values, as shown in FIG. 3a, a-c1-301 and a-c1-302 are outlier channel values. Referring to FIG. 3b, the data channel c2 includes a plurality of channel values, as shown in FIG. 3b, a-c2-301, a-c2-302, and a-c3-303 are outlier channel values.

[0063] In the embodiment, the maximum channel value is selected from the multiple channel values of the outliers, and the data channel corresponding to the maximum channel value is taken as the target data channel. For example, continuing the foregoing example, assuming that a-c1-301 is the maximum among a-c1-301, a-c1-302, a-c2-301, a-c2-302, and a-c3-303, the data channel c1 is the target data channel.

[0064] In the embodiment, there can be only one outlier channel value, and the data channel corresponding to the outlier channel value is the target data channel.

[0065] In the embodiment, the channel value corresponding to the target data channel is taken as the activation input value of the model function network. For example, continuing the foregoing example, as shown in FIG. 3a, all channel values included in the data channel c1 are the activation input value of the model function network.

[0066] In the embodiment, the product operation is performed on the channel value corresponding to the target data channel and the original weight value corresponding to the target data channel to obtain the first output result of the target data channel, and the product operation is performed on the channel value corresponding to the target data channel and each quantized weight value corresponding to the target data channel to obtain multiple second output results of the target data channel.

[0067] For example, continuing the foregoing example in which the data channel c1 is taken as the target data channel, a-c1 represents the multiple channel values (i.e., multiple points in FIG. 3a) corresponding to the target data channel c1. If the size of the data channel c1 is 2048, the data channel c1 has 2048 channel values, and a-c1 is a vector matrix [N, M] determined according to the size of the data channel c1, where N is 2048 and M is the pre-set size of the hidden layer of the model function network. Then, the first output result out0-c1 of the target data channel c1 is (a-c1)×w0, the second output result of int2 of the target data channel c1 is outint2-c1=(a-c1)×wint2, the second output result of int4 of the target data channel c1 is outint4-c1=(a-c1)×wint4, and the second output result of int8 of the target data channel c1 is outint8-c1=(a-c1)×wint8.

[0068] Thus, by the embodiments, since the maximum outlying channel value can reflect the sensitivity of the model functional network to a certain extent, when the data channel corresponding to the maximum outlying channel value in the model functional network is taken as the target time data channel, the accuracy of the first output result and the plurality of second output results calculated is higher.

[0069] In the embodiments of the present application, the process of taking the channel value corresponding to the target data channel as the activation input value of the model functional network can include:

[0070] calculating the similarity between the target data channel and the adjacent data channel corresponding to the target data channel;

[0071] performing reduction processing on the outlying channel value in the target data channel based on the similarity, to obtain a target channel value;

[0072] taking the target channel value and the channel value other than the outlying channel value in the target data channel as the activation input value of the model functional network.

[0073] That is, in the embodiments, the similarity between the target data channel and the adjacent data channel corresponding to the target data channel is used to perform reduction processing on the outlying channel value in the target data channel, so as to obtain the target channel value, and at this time, the target channel value and the channel value other than the outlying channel value in the target data channel (i.e., other channel values) are the activation input value of the model functional network.

[0074] For example, taking the aforementioned data channel c1 as the target data channel, and using c1' to represent the adjacent data channel of the target data channel c1; first, the similarity s between the target data channel c1 and the adjacent data channel c1' is calculated, and then the outlying channel values a-c1-301 and a-c1-302 in the target data channel c1 are reduced using the similarity s, to obtain the target channel value a-c1-301' corresponding to the outlying channel value a-c1-301 and the target channel value a-c1-302' corresponding to the outlying channel value a-c1-302, and at this time, the target channel value a-c1-301', the target channel value a-c1-302', and the channel value other than the outlying channel values a-c1-301 and a-c1-302 in the target data channel c1 are the activation input value of the model functional network.

[0075] Thus, by the embodiments, the outlying channel value in the target data channel is reduced to represent the target data channel with a smaller value, so as to obtain the activation input value of the model functional network, which is helpful for subsequently selecting a lower-precision quantization bit mode for the model functional network.

[0076] In the embodiments of the present application, the process of calculating the similarity between the target data channel and the adjacent data channel corresponding to the target data channel can comprise:

[0077] performing distance operation on the channel value of the target data channel and the channel value of the adjacent data channel corresponding to the target data channel to obtain a distance value, the distance value being greater than 0 and less than 1, and taking the distance value as the similarity between the target data channel and the adjacent data channel;

[0078] Correspondingly, the outlier channel value in the target data channel is reduced based on the similarity to obtain a target channel value, comprising:

[0079] performing product operation on the distance value and the outlier channel value in the target data channel to obtain the target channel value.

[0080] That is, in the embodiments, the channel value of the adjacent data channel corresponding to the target data channel is first determined, the channel value of the adjacent data channel is usually multiple, then the distance operation is performed on each channel value of the target data channel and the channel value of the adjacent data channel to obtain a distance value, and then the average distance value is obtained by performing average operation on all the distance values, which is the similarity between the target data channel and the adjacent data channel, and then the outlier channel value in the target data channel is reduced by using the distance value to obtain the target channel value.

[0081] In the embodiments, the distance algorithm is used to calculate the distance value between the target data channel and the adjacent data channel, and the distance algorithm comprises at least one of cosine similarity, Euclidean distance, Manhattan distance, Chebyshev distance, and Minkowski distance.

[0082] For example, taking the foregoing example, assuming that the distance algorithm is cosine similarity, then the distance value between the target data channel c1 and the adjacent data channel c1' is where c1 and c1' are vector matrices.

[0083] In the embodiments, the distance value between the target data channel and the adjacent data channel is obtained, and then the product operation is performed on the distance value and the outlier channel value in the target data channel to obtain the target channel value.

[0084] For example, taking the foregoing example, the outlier channel values are a-c1-301 and a-c1-302; then the target channel value a-c1-301' = cosθ×(a-c1-301), and the target channel value a-c1-302' = cosθ×(a-c1-302). It can be understood that, since 0<cosθ<1, the target channel value a-c1-301' is less than a-c1-301, and the target channel value a-c1-302' is less than a-c1-302.

[0085] Thus, the target channel value can be obtained simply and accurately through the example.

[0086] S203, selecting a target quantization bit mode from the plurality of quantization bit modes based on the first output result and the plurality of second output results, the target quantization bit mode being used for quantization processing of the original model parameter in the inference stage of the model function network.

[0087] In the embodiments of the present application, after obtaining the first output result and the plurality of second output results, the first output result and the plurality of second output results can be used to select a target quantization bit mode from the plurality of quantization bit modes, wherein the target quantization bit mode is used for quantization processing of the original model parameter in the inference stage of the model function network.

[0088] In the embodiments of the present application, the process of selecting a target quantization bit mode from the plurality of quantization bit modes based on the first output result and the plurality of second output results in S203 can include:

[0089] calculating a difference value between the first output result and each second output result;

[0090] selecting a quantization bit mode corresponding to the smallest difference value from the plurality of quantization bit modes, and taking the selected quantization bit mode as the target quantization bit mode.

[0091] That is, in the embodiments, a difference value between the first output result and each second output result is calculated first, and then a quantization bit mode corresponding to the smallest difference value is selected from the plurality of quantization bit modes, and the selected quantization bit mode is the target quantization bit mode. It can be understood that the smaller the difference value, the smaller the influence on the output result before and after the quantization bit mode is adopted; therefore, in the embodiments, the quantization bit mode corresponding to the smallest difference value is taken as the target quantization bit mode.

[0092] For example, taking the foregoing example, assuming that the difference value between the first output result out0 and the second output result outint2 is Lout0-outint2, the difference value between the first output result out0 and the second output result outint4 is Lout0-outint4, the difference value between the first output result out0 and the second output result outint8 is Lout0-outint8, and Lout0-outint2 is the smallest among Lout0-outint2, Lout0-outint4 and Lout0-outint8, then int2 corresponding to Lout0-outint2 is selected from the plurality of quantization bit modes int2, int4 and int8 as the target quantization bit mode.

[0093] Thus, by means of the embodiments, the difference value can be calculated to select the target quantization bit mode from the plurality of quantization bit modes conveniently and accurately.

[0094] In the embodiments of the present application, the process of calculating the difference value between the first output result and each second output result can include:

[0095] For each second output result, the difference between the first output result and the second output result is calculated to obtain a difference value, and the difference value is calculated to the specified power to obtain the difference value between the first output result and the second output result.

[0096] In the embodiments, the specified power can be the square power, and the difference value between the first output result and each second output result is calculated by using the Mean Squared Error (MSE) algorithm.

[0097] For example, taking the foregoing example, Lout0-outint2=(out0-outint2) 2 , Lout0-outint4=(out0-outint4) 2 , Lout0-outint8=(out0-outint8) 2 .

[0098] Thus, by means of the embodiments, the difference value between the first output result and each second output result can be obtained conveniently and accurately.

[0099] In the embodiments of the present application, the process of calculating the difference value between the first output result and each second output result can include:

[0100] For each second output result, the signal power value is calculated based on the second output result, the noise power value is calculated based on the first output result and the second output result, the signal-to-noise ratio is calculated based on the signal power value and the noise power value, and the difference value between the first output result and the second output result is calculated based on the signal-to-noise ratio.

[0101] That is, in the embodiments, the difference value between the first output result and each second output result is calculated by using the Signal-to-Noise Ratio (SNR) algorithm.

[0102] For example, taking the foregoing example, to calculate the difference value Lout0-outint2 between the first output result out0 and the second output result outint2, the signal power value is represented by Psignal, then the noise power value is represented by Pnoise, then The signal-to-noise ratio is represented by SNR, and then Correspondingly, Lout0-outint2=-SNR, and other difference values are deduced in the same way, which is not described herein.

[0103] In this way, the difference value between the first output result and each second output result can be obtained simply and accurately through the embodiments.

[0104] In the embodiments of the present application, the process of selecting the quantization bit mode corresponding to the smallest difference value from the plurality of quantization bit modes can include:

[0105] selecting a difference value smaller than a preset difference threshold from the plurality of difference values;

[0106] If the selected difference value is multiple, the quantization bit mode corresponding to the smallest difference value is selected from the plurality of quantization bit modes.

[0107] That is, in the embodiments, a difference value smaller than a preset difference threshold is first selected from the plurality of difference values as a candidate difference value, and when the candidate difference value is multiple, the quantization bit mode corresponding to the smallest candidate difference value is selected from the plurality of quantization bit modes.

[0108] For example, taking the foregoing example, assuming that the preset difference threshold is 0.001, and Lout0-outint2 and Lout0-outint4 are both smaller than 0.001 in Lout0-outint2, Lout0-outint4, and Lout0-outint8, Lout0-outint2 and Lout0-outint4 are candidate difference values at this time, and assuming that Lout0-outint2 is smaller than Lout0-outint4, then int2 corresponding to Lout0-outint2 is selected as the target quantization bit mode from the plurality of quantization bit modes int2, int4, and int8. In actual applications, the preset difference threshold can be flexibly set according to specific application scenarios.

[0109] In this way, through the embodiments, the target quantization bit mode can be accurately obtained by first performing primary screening using the preset difference threshold and then performing secondary screening through comparison of the difference values.

[0110] In the embodiments of the present application, the process of selecting the quantization bit mode corresponding to the smallest difference value from the plurality of quantization bit modes can include:

[0111] selecting the smallest difference value from the plurality of difference values;

[0112] If the selected difference value is less than the preset difference threshold, a quantization bit mode corresponding to the selected difference value is selected from the plurality of quantization bit modes.

[0113] That is, in the embodiment, the smallest difference value is selected from the plurality of difference values as a candidate difference value, and when the candidate difference value is less than the preset difference threshold, a quantization bit mode corresponding to the candidate difference value is selected from the plurality of quantization bit modes.

[0114] For example, taking the foregoing example, assuming that Lout0-outint2, Lout0-outint4, and Lout0-outint8 are Lout0-outint2, Lout0-outint2 is the candidate difference value, assuming that the preset difference threshold is 0.001, and Lout0-outint2 is less than 0.001, then int2 corresponding to Lout0-outint2 is selected from the plurality of quantization bit modes int2, int4, and int8 as the target quantization bit mode. In actual applications, the preset difference threshold can be flexibly set according to specific application scenarios.

[0115] In this way, through the embodiment, the initial screening is performed through the comparison of the difference values, and the secondary screening is performed by using the preset difference threshold, so that the target quantization bit mode can be accurately obtained.

[0116] In the embodiment, the number of model function networks of S201 to S203 shown in FIG. 2 can be one or more. When the number of model function networks is one, the corresponding target quantization bit mode is finally obtained. When the number of model function networks is more than one, the target quantization bit mode corresponding to each model function network is finally obtained. The target quantization bit modes corresponding to different model function networks can be the same or different.

[0117] Referring to Table 1, Table 1 is an example table of the target quantization bit modes corresponding to the plurality of model function networks.

[0118] Table 1

[0119] In the embodiments of the present application, the original model parameters corresponding to the model are quantized by multiple quantization bit modes to obtain quantized model parameters corresponding to each quantization bit mode, and the first output result of the model function network under the original model parameters and the second output result of the model function network under each quantized model parameter are used to select a target quantization bit mode from the multiple quantization bit modes. In this way, the accurate selection of the quantization bit mode of the model function network is realized, the accuracy of the quantization processing is high, which helps to improve the generalization ability of the model outside the training data, can better adapt to new data distribution and scene, at the same time, also enhances the stability and robustness of the model, can maintain good performance under complex conditions, in addition, can also faster reasoning, reduces unnecessary resource consumption, improves the resource utilization, so greatly improves the model performance, makes the tasks in various application scenarios can be well executed by the model.

[0120] In the embodiments of the present application, another quantization-based data processing method is provided. As shown in FIG. 4, the quantization-based data processing method can further include steps S401-S403 after S203.

[0121] S401-S403 are described in detail as follows:

[0122] S401, generating a target quantization scaling factor based on the target quantization bit mode.

[0123] In the embodiments, the process of quantizing the function network by using the quantization bit mode involves the use of the quantization scaling factor, which is mainly used to map data of a higher precision type (such as int8) to a representation range corresponding to a lower precision type (such as int2). Therefore, after obtaining the target quantization bit mode in the embodiments, the target quantization scaling factor corresponding to the model function network can be generated by using the target quantization bit mode.

[0124] In the embodiments of the present application, the model function network includes multiple; for example, as shown in Table 1, the multiple model function networks are E1, E2, and E3.

[0125] Correspondingly, the process of generating a target quantization scaling factor based on the target quantization bit mode in S401 can include at least two cases:

[0126] Case 1: If there is a first model function network in the multiple model function networks whose quantization precision of the target quantization bit mode is greater than a preset precision threshold, then the target quantization scaling factor corresponding to each data channel in the first model function network is generated.

[0127] The first model function network in the embodiment refers to a model function network whose quantization precision of the target quantization bit mode is greater than the preset precision threshold in the plurality of model function networks.

[0128] For example, taking the foregoing example, assuming that the preset precision threshold is int4, it can be known from Table 1 that the quantization precision int8 of the model function network E3 is greater than the preset precision threshold int4, and at this time, the model function network E3 is the first model function network.

[0129] In the embodiment, after obtaining the first model function network, a target quantization scaling factor corresponding to each data channel in the first model function network can be generated.

[0130] For example, taking the foregoing example in which the model function network E3 is the first model function network, assuming that the model function network E3 includes 64 data channels, and correspondingly, 64 target quantization scaling factors corresponding to the 64 data channels are generated.

[0131] Case 2: If there is a second model function network whose quantization precision of the target quantization bit mode is less than or equal to the preset precision threshold in the plurality of model function networks, the channel values of each data channel in the second model function network are grouped, and a target quantization scaling factor corresponding to each combination is generated.

[0132] The second model function network in the embodiment refers to a model function network whose quantization precision of the target quantization bit mode is less than or equal to the preset precision threshold in the plurality of model function networks.

[0133] For example, taking the foregoing example, assuming that the preset precision threshold is int4, it can be known from Table 1 that the quantization precision int2 of the model function network E1 is less than the preset precision threshold int4, and the quantization precision int4 of the model function network E2 is equal to the preset precision threshold int4, and at this time, the model function networks E1 and E2 are the second model function networks.

[0134] In the embodiment, after obtaining the second model function network, the channel values of each data channel in the second model function network can be grouped, and a target quantization scaling factor corresponding to each combination is generated.

[0135] For example, taking the foregoing example in which the model function networks E1 and E2 are the second model function networks.

[0136] For the model function network E2, assuming that the model function network E2 includes 64 data channels, each data channel has 128 channel values, and each 64 channel values is taken as a combination, that is, the channel values of each data channel are divided into 2 combinations, and there are 128 combinations in total. Correspondingly, 128 target quantization scaling factors corresponding to the 128 combinations are generated.

[0137] For the model function network E1, it is assumed that the model function network E1 includes 128 data channels, each data channel has 128 channel values, and each data channel can be grouped into 2 combinations at every 64 channel values, that is, the channel values of each data channel are divided into 2 combinations, and there are 256 combinations in total. Correspondingly, 256 combinations are generated to correspond to the target quantization scaling factors.

[0138] In an embodiment, the number of data channels included in different model function networks can be the same or different, and the channel values included in each combination when grouping can be flexibly adjusted according to specific application scenarios.

[0139] In this way, through the embodiments, when the quantization precision of the target quantization bit mode corresponding to the model function network is high, the target quantization scaling factor is calculated based on the data channel with a relatively coarse granularity, thereby saving computing resources and time, etc.; when the quantization precision of the target quantization bit mode corresponding to the model function network is low, the target quantization scaling factor is calculated based on the combination with a relatively fine granularity, thereby improving the quantization precision as much as possible and ensuring the accuracy of the quantization processing.

[0140] For ease of understanding, please refer to FIG. 5. For a first model function network whose quantization precision of the target quantization bit mode is greater than a preset precision threshold, the target quantization scaling factor corresponding to each data channel in the first model function network is generated (i.e., channel-by-channel quantization); for a second model function network whose quantization precision of the target quantization bit mode is less than or equal to the preset precision threshold, the channel values of each data channel in the second model function network are grouped, and the target quantization scaling factor corresponding to each combination is generated (i.e., grouped quantization).

[0141] In an embodiment of the present application, the process of generating the target quantization scaling factor corresponding to each data channel in the first model function network can include:

[0142] Obtaining the target quantization bit mode corresponding to the first model function network;

[0143] For each data channel in the first model function network, based on the obtained target quantization bit mode, the activation input value corresponding to the data channel, and each quantization scaling factor, an operation is performed to obtain an operation result corresponding to each quantization scaling factor, and the quantization scaling factor corresponding to the operation result with the largest absolute value in the multiple operation results is taken as the target quantization scaling factor of the data channel.

[0144] That is, in the embodiment, for each data channel in the first model function network, the corresponding target quantization bit mode of the first model function network, the corresponding activation input value of the data channel, and each quantization scaling factor are used to perform operations, so as to obtain an operation result corresponding to each quantization scaling factor. At this time, the quantization scaling factor corresponding to the operation result with the largest absolute value in the plurality of operation results is the target quantization scaling factor of the data channel. It can be understood that the larger the absolute value of the operation result is, the more it can reflect the significant change of the data channel, so as to better focus on the sensitivity of the model function network.

[0145] In this way, through the embodiment, the target quantization scaling factor of each data channel in the first model function network can be obtained simply and accurately.

[0146] In the embodiment of the present application, the process of performing operations based on the obtained target quantization bit mode, the corresponding activation input value of the data channel, and each quantization scaling factor to obtain an operation result corresponding to each quantization scaling factor can include:

[0147] The first quantization scaling factor in the first scaling factor list is traversed, and operations are performed based on the obtained target quantization bit mode, the corresponding activation input value of the data channel, and the traversed quantization scaling factor to obtain an operation result corresponding to the traversed quantization scaling factor.

[0148] The activation input value corresponding to the data channel is updated, the next adjacent quantization scaling factor of the first quantization scaling factor is traversed, and operations are performed based on the obtained target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor to obtain an operation result corresponding to the traversed quantization scaling factor, until the first scaling factor list is traversed to obtain an operation result corresponding to each quantization scaling factor in the first scaling factor list.

[0149] That is, in the embodiment, the operation result corresponding to each quantization scaling factor is obtained by traversal calculation. Specifically, first, the first quantization scaling factor in the first scaling factor list is traversed, and operations are performed using the corresponding target quantization bit mode of the first model function network, the corresponding activation input value of the data channel, and the first quantization scaling factor to obtain an operation result corresponding to the first quantization scaling factor. Then, the activation input value of the data channel is updated, the next adjacent quantization scaling factor (i.e., the second quantization scaling factor in the first scaling factor list) of the first quantization scaling factor is traversed, and operations are performed using the corresponding target quantization bit mode of the first model function network, the updated activation input value, and the second quantization scaling factor to obtain an operation result corresponding to the second quantization scaling factor. This cycle is repeated until the first scaling factor list is traversed to obtain an operation result corresponding to each quantization scaling factor in the first scaling factor list.

[0150] In the embodiment, the updating of the activation input value of the data channel in each iteration process can be updating the activation input value of the data channel sampled in the current iteration process and the activation input value of the data channel sampled in the last iteration process to obtain an absolute maximum value, if the absolute value of the activation input value of the data channel sampled in the current iteration process is less than the absolute value of the activation input value of the data channel sampled in the last iteration process, updating the activation input value of the data channel sampled in the current iteration process to the activation input value of the data channel sampled in the last iteration process, and if the absolute value of the activation input value of the data channel sampled in the current iteration process is greater than or equal to the absolute value of the activation input value of the data channel sampled in the last iteration process, keeping the activation input value of the data channel sampled in the current iteration process. The activation input value of the data channel sampled refers to the input data of the model being predicted by the model to obtain the activation input value.

[0151] In this way, by the embodiment, the activation input value of the data channel is updated in the iteration process, and the accuracy of the operation result corresponding to the quantization scaling factor is improved.

[0152] In the embodiment of the present application, the process of generating the target quantization scaling factor corresponding to each combination can include:

[0153] obtaining a target quantization bit mode corresponding to the second model function network;

[0154] For each combination corresponding to each data channel in the second model function network, based on the obtained target quantization bit mode, the activation input value corresponding to the combination, and each quantization scaling factor, an operation is performed to obtain an operation result corresponding to each quantization scaling factor, and the quantization scaling factor corresponding to the operation result with the largest absolute value in the multiple operation results is taken as the target quantization scaling factor of the combination.

[0155] That is, in the embodiment, for each combination corresponding to each data channel in the first model function network, the target quantization bit mode corresponding to the second model function network, the activation input value corresponding to the combination, and each quantization scaling factor are used to perform an operation to obtain an operation result corresponding to each quantization scaling factor, and the quantization scaling factor corresponding to the operation result with the largest absolute value in the multiple operation results is the target quantization scaling factor of the combination. It can be understood that the larger the absolute value of the operation result is, the more significant changes of the combination can be reflected, so that the sensitivity of the model function network can be better focused.

[0156] In this way, by the embodiment, the target quantization scaling factor of each combination of each data channel in the second model function network can be obtained simply and accurately.

[0157] In the embodiments of the present application, the process of obtaining the operation result corresponding to each quantization scaling factor based on the obtained target quantization bit mode, the combined corresponding activation input value, and each quantization scaling factor can include:

[0158] traversing a first quantization scaling factor in the second scaling factor list, and obtaining an operation result corresponding to the traversed quantization scaling factor based on the obtained target quantization bit mode, the combined corresponding activation input value, and the traversed quantization scaling factor;

[0159] updating the combined corresponding activation input value, traversing a next adjacent quantization scaling factor of the first quantization scaling factor, and obtaining an operation result corresponding to the traversed quantization scaling factor based on the obtained target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor, until the second scaling factor list is traversed to obtain the operation result corresponding to each quantization scaling factor in the second scaling factor list.

[0160] That is, in the embodiments, the operation result corresponding to each quantization scaling factor is obtained by traversal calculation. Specifically, first, a first quantization scaling factor in the second scaling factor list is traversed, and the target quantization bit mode corresponding to the second model function network, the combined corresponding activation input value, and the first quantization scaling factor are operated to obtain the operation result corresponding to the first quantization scaling factor; then the combined activation input value is updated, a next adjacent quantization scaling factor of the first quantization scaling factor (i.e., a second quantization scaling factor in the second scaling factor list) is traversed, and the target quantization bit mode corresponding to the second model function network, the updated activation input value, and the second quantization scaling factor are operated to obtain the operation result corresponding to the second quantization scaling factor, and the cycle is repeated until the second scaling factor list is traversed to obtain the operation result corresponding to each quantization scaling factor in the second scaling factor list.

[0161] In the embodiments, the combined activation input value is updated in each iteration process, which can be to obtain the absolute value maximum of the combined activation input value sampled in the current iteration process and the combined activation input value sampled in the last iteration process, if the absolute value of the combined activation input value sampled in the current iteration process is less than the absolute value of the combined activation input value sampled in the last iteration process, the combined activation input value sampled in the current iteration process is updated to the combined activation input value sampled in the last iteration process, if the absolute value of the combined activation input value sampled in the current iteration process is greater than or equal to the absolute value of the combined activation input value sampled in the last iteration process, the combined activation input value sampled in the current iteration process is maintained.

[0162] Thus, by the embodiments, the activation input values of the combination are updated in the iteration process, and the accuracy of the operation result corresponding to the quantization scaling factor is improved.

[0163] In the embodiments of the present application, the process of generating the target quantization scaling factor based on the target quantization bit mode in S401 can include:

[0164] The target quantization scaling factor corresponding to the target quantization bit mode is obtained by the plurality of devices respectively deployed with the hybrid bit quantization model.

[0165] The maximum quantization scaling factor is selected from the plurality of quantization scaling factors as the target quantization scaling factor.

[0166] That is, in the embodiments, the target quantization scaling factor corresponding to the target quantization bit mode is obtained by the plurality of devices respectively deployed with the hybrid bit quantization model, and the maximum quantization scaling factor is selected from the plurality of quantization scaling factors as the target quantization scaling factor, considering the limited device resources.

[0167] The hybrid bit quantization model refers to a plurality of model function networks contained therein using different quantization bit modes, wherein the quantization bit modes include int2, int4, int8, and bf16, etc. For example, the three devices can use the same quantization bit mode, for example, by deploying the hybrid bit quantization model on three devices. Taking the model function network E1 as an example, the three devices will obtain the quantization scaling factor corresponding to the model function network E1 (i.e. three quantization scaling factors), and then the maximum quantization scaling factor is selected from the three quantization scaling factors as the target quantization scaling factor corresponding to the model function network E1.

[0168] Thus, by the embodiments, the target quantization scaling factor of the same model function network is obtained by deploying the hybrid bit model on the plurality of devices respectively, ensuring that each device can normally perform tasks, and improving the accuracy of the target quantization scaling factor.

[0169] S402, the target quantization bit mode, the target quantization scaling factor, and the identification information corresponding to the model function network are stored in the specified area.

[0170] In the embodiments of the present application, after the target quantization scaling factor is generated by the target quantization bit mode, the target quantization bit mode, the target quantization scaling factor, and the identification information (used to uniquely identify the model function network, including but not limited to name, number, etc.) corresponding to the model function network can be stored in the specified area.

[0171] For example, for example, please refer to Table 2 for an example of associated storage.

[0172] Table 2

[0173] S403, if it is determined that the model function network is in the inference stage, obtaining a target quantization bit mode and a target quantization scaling factor from the specified area to perform quantization processing on the original model parameters.

[0174] In the embodiment of the application, when it is determined that the model function network is in the inference stage, the target quantization bit mode and the target quantization scaling factor corresponding to the model function network can be obtained from the specified storage area to perform quantization processing on the original model parameters corresponding to the model function network. In the embodiment, whether the model function network is an inference model, i.e., whether it is in the inference stage, can be determined according to the file format of the model function network.

[0175] The detailed introduction of S201 to S203 shown in FIG. 4 can refer to S201 to S203 shown in FIG. 2, which will not be repeated here.

[0176] In the embodiment of the application, the target quantization bit mode, the target quantization scaling factor, and the identification information corresponding to the model function network are associatedly stored, which can be directly used in model inference, thereby improving the efficiency of quantization processing and the reliability of quantization processing, and further improving the model performance.

[0177] The specific scenarios of the embodiment of the application are described in detail as follows.

[0178] Referring to FIG. 6, FIG. 6 is a flowchart of a data processing method based on quantization according to an embodiment of the application. As shown in FIG. 6, the data processing method based on quantization at least includes steps S601 to S607, which are described in detail as follows.

[0179] S601, obtaining a verification set and inputting the verification set into a hybrid expert model.

[0180] In the embodiment of the application, the hybrid expert model includes but is not limited to a model used for text processing, video processing, speech processing, etc. The verification set is a set obtained by sampling a small part from a data set used for model training, which is used for inputting into the model to calibrate sampling and obtain the quantization scaling factor of each layer.

[0181] S602, detecting whether the verification round reaches a preset number threshold, if not, performing S603, and if yes, performing S604.

[0182] S603, recording the activation input value corresponding to each model function network and returning to S602.

[0183] S604, performing sensitivity analysis on each model function network to generate the target quantization bit mode and the target quantization scaling factor of each model function network.

[0184] In the embodiments of the present application, the sensitivity analysis can be realized by calculating the cosine similarity as described above to generate the target quantization bit mode of each model function network. For example, the target quantization bit mode of each model function network can be generated as shown in FIG. 7. As shown in FIG. 7, the hybrid expert model includes four expert networks (i.e., function networks), namely Expert1, Expert2, Expert3, and Expert4. The sensitivity analysis of Expert1 obtains the target quantization bit mode INT8, the sensitivity analysis of Expert2 obtains the target quantization bit mode INT4, the sensitivity analysis of Expert3 obtains the target quantization bit mode INT4, and the sensitivity analysis of Expert4 obtains the target quantization bit mode INT2.

[0185] For example, generating the target quantization scaling factor of each model function network can include: if there is a first model function network in the plurality of model function networks whose quantization precision of the target quantization bit mode is greater than the preset precision threshold, generating the target quantization scaling factor corresponding to each data channel in the first model function network; and if there is a second model function network in the plurality of model function networks whose quantization precision of the target quantization bit mode is less than or equal to the preset precision threshold, grouping the channel values of each data channel in the second model function network and generating the target quantization scaling factor corresponding to each combination. That is, when the quantization precision of the target quantization bit mode corresponding to the model function network is high, the target quantization scaling factor is calculated based on the data channel with a coarse granularity, and when the quantization precision of the target quantization bit mode corresponding to the model function network is low, the target quantization scaling factor is calculated based on the combination with a fine granularity.

[0186] S605, the target quantization bit mode and the target quantization scaling factor corresponding to each model function network, and the identification information of each model function network are stored in the specified area.

[0187] In the embodiments of the present application, the target quantization bit mode and the target quantization scaling factor corresponding to each model function network, and the identification information of each model function network can be stored in the specified area in the local or cloud in the form of a vector through a storage interface (such as torch.save).

[0188] S606, if it is detected that the hybrid expert model is in the inference stage, the target quantization bit mode and the target quantization scaling factor corresponding to each model function network are obtained from the specified area.

[0189] S607, the original weight values of each model function network are quantized by using the target quantization bit mode and the target quantization scaling factor corresponding to each model function network.

[0190] In the embodiments of the present application, the mapping relationship table of the preset bit mode and the kernel can be searched to obtain the target computing kernel corresponding to each model function network, and then the corresponding target quantization scaling factor is input into the process of calculating each target computing kernel to quantize the original weight value to the corresponding target quantization bit number, thereby realizing the mixed bit quantization reasoning of the mixed expert model and obtaining the corresponding task processing result.

[0191] In embodiments, when the mixed expert model is applied to a text processing application scenario, including but not limited to:

[0192] (1) Text classification (such as email classification, news classification, etc.), at this time the mixed expert model can be a text classification model. Specifically, the text to be classified is input into the text classification model, the target quantization bit mode and the target quantization scaling factor corresponding to each function network in the text classification model are obtained in the text classification model by using the foregoing manner, and the target quantization bit mode and the target quantization scaling factor corresponding to each function network are used for quantization processing, and then the data after quantization processing is used for text classification, and text classification information is output. The performance of the text classification model is good, thereby improving the accuracy and efficiency of text classification, etc.

[0193] (2) Context understanding and automatic reply (such as automatic customer service, chat robot, etc.), at this time the mixed expert model can be a reply model. Specifically, the text to be replied is input into the reply model, the target quantization bit mode and the target quantization scaling factor corresponding to each function network in the reply model are obtained in the reply model by using the foregoing manner, and the target quantization bit mode and the target quantization scaling factor corresponding to each function network are used for quantization processing, and then the data after quantization processing is used to understand the user's intention, and reply information is output. The performance of the reply model is good, thereby improving the accuracy and efficiency of context understanding and automatic reply, etc.

[0194] (3) Multilingual text processing (such as text translation, language recognition, etc.), at this time the mixed expert model can be a multilingual processing model. Specifically, the text to be processed is input into the multilingual processing model, the target quantization bit mode and the target quantization scaling factor corresponding to each function network in the multilingual processing model are obtained in the multilingual processing model by using the foregoing manner, and the target quantization bit mode and the target quantization scaling factor corresponding to each function network are used for quantization processing, and then the data after quantization processing is used for text analysis, and text analysis information is output. The performance of the multilingual processing model is good, thereby improving the accuracy and efficiency of multilingual text processing, etc.

[0195] In embodiments, when the mixed expert model is applied to a video processing application scenario, including but not limited to:

[0196] (1) Video detection (e.g., recognizing different actions, scenes, objects, etc. in a video), in which case the hybrid expert model can be a video detection model. Specifically, the video to be detected is input into the video detection model, in which the aforementioned method is used to obtain the target quantization bit mode and the target quantization scaling factor corresponding to each functional network in the video detection model, and the target quantization bit mode and the target quantization scaling factor corresponding to each functional network are used for quantization processing, and then the data after quantization processing is used for video detection, and video detection information is output. The performance of the video detection model is good, thereby improving the accuracy and efficiency of video detection, etc.

[0197] (2) Video compression and transmission (e.g., online video playing, video conferencing, etc.), in which case the hybrid expert model can be a video transmission model. Specifically, the video to be transmitted is input into the video transmission model, in which the aforementioned method is used to obtain the target quantization bit mode and the target quantization scaling factor corresponding to each functional network in the video transmission model, and the target quantization bit mode and the target quantization scaling factor corresponding to each functional network are used for quantization processing, and then the data after quantization processing is used for video compression, realizing efficient video transmission and playing experience. The performance of the video transmission model is good, thereby improving the accuracy and efficiency of video compression and transmission, etc.

[0198] (3) Video special effects and enhancement (e.g., film production, virtual reality, augmented reality, etc.), in which case the hybrid expert model can be a video processing model. Specifically, the video to be processed is input into the video processing model, in which the aforementioned method is used to obtain the target quantization bit mode and the target quantization scaling factor corresponding to each functional network in the video processing model, and the target quantization bit mode and the target quantization scaling factor corresponding to each functional network are used for quantization processing, and then the data after quantization processing is used for special effect and enhancement processing, realizing efficient video rendering. The performance of the video processing model is good, thereby improving the accuracy and efficiency of video special effects and enhancement, etc.

[0199] In embodiments, when the hybrid expert model is applied to a speech processing application scenario, it includes but is not limited to:

[0200] (1) Speech recognition and transcription (e.g., voice assistants, voice search, caption generation, etc.), at which time the hybrid expert model can be a speech recognition model. Specifically, input the speech to be recognized into the speech recognition model, obtain the target quantization bit mode and the target quantization scaling factor corresponding to each functional network in the speech recognition model in the speech recognition model in the foregoing manner, and perform quantization processing using the target quantization bit mode and the target quantization scaling factor corresponding to each functional network, and then perform speech recognition and transcription using the quantized data, and output speech recognition information. The performance of the speech recognition model is good, thereby improving the accuracy and efficiency of speech recognition and transcription, etc.

[0201] (2) Voice activity detection and keyword recognition (e.g., wake-up word detection, voice control, etc.), at which time the hybrid expert model can be a voice detection model. Specifically, input the voice to be detected into the voice detection model, obtain the target quantization bit mode and the target quantization scaling factor corresponding to each functional network in the voice detection model in the voice detection model in the foregoing manner, and perform quantization processing using the target quantization bit mode and the target quantization scaling factor corresponding to each functional network, and then perform voice activity detection and keyword recognition using the quantized data, and output voice detection information. The performance of the voice detection model is good, thereby improving the accuracy and efficiency of voice activity detection and keyword recognition, etc.

[0202] (3) Speech quality enhancement and noise reduction (e.g., teleconferencing, voice calls, recording applications, etc.), at which time the hybrid expert model can be a speech enhancement model. Specifically, input the speech to be enhanced into the speech enhancement model, obtain the target quantization bit mode and the target quantization scaling factor corresponding to each functional network in the speech enhancement model in the speech enhancement model in the foregoing manner, and perform quantization processing using the target quantization bit mode and the target quantization scaling factor corresponding to each functional network, and then perform speech quality enhancement and noise reduction using the quantized data, and output speech enhancement information. The performance of the speech enhancement model is good, thereby improving the accuracy and efficiency of speech quality enhancement and noise reduction, etc.

[0203] It should be noted that the determination of the target quantization bit mode and the target quantization scaling factor corresponding to each functional network in the corresponding model involved in the foregoing application scenarios, and the quantization processing using the target quantization bit mode and the target quantization scaling factor corresponding to each functional network are described in detail in the foregoing embodiments, and will not be described here.

[0204] In the embodiments of the present application, different size models of 7B-MoE and 70B-MoE are used to test on the ptb-en validation set, and the test results are shown in Table 3.

[0205] Table 3

[0206] As shown in Table 3, the obtained values are all perplexity (PPL) values, wherein the smaller the PPL value is, the smaller the loss is, and the PPL values are arranged in ascending order as 12.209 < 12.323 < 12.350 for 7B-MoE, and the PPL values are arranged in ascending order as 8.858 < 8.903 < 8.925 for 70B-MoE, wherein the mixed bit quantization and bf16 and int8 are not much different. It can be seen that the precision of the quantization of the embodiments of the present application is close to lossless, and the model performance is good.

[0207] FIG. 8 is a block diagram of a data processing apparatus based on quantization according to an embodiment of the present application. As shown in FIG. 8, the apparatus includes:

[0208] The quantization processing module 801 is configured to quantize the original model parameters corresponding to the model function network through a plurality of quantization bit modes respectively, to obtain quantized model parameters corresponding to each quantization bit mode.

[0209] The acquisition module 802 is configured to acquire a first output result of the model function network under the original model parameters, and acquire a second output result of the model function network under each quantized model parameter.

[0210] The selection module 803 is configured to select a target quantization bit mode from the plurality of quantization bit modes based on the first output result and the plurality of second output results, and the target quantization bit mode is used for quantizing the original model parameters in an inference stage corresponding to the model function network.

[0211] In the embodiments of the present application, based on the foregoing scheme, the selection module 803 is specifically configured to:

[0212] calculate a difference value between the first output result and each second output result;

[0213] select a quantization bit mode corresponding to the smallest difference value from the plurality of quantization bit modes, and take the selected quantization bit mode as the target quantization bit mode.

[0214] In the embodiments of the present application, based on the foregoing scheme, the selection module 803 is further specifically configured to:

[0215] for each second output result, perform a difference operation on the first output result and the second output result to obtain a difference value, and perform a specified power operation on the obtained difference value to obtain a difference value between the first output result and the second output result.

[0216] In the embodiments of the present application, based on the foregoing scheme, the selection module 803 is further specifically configured to:

[0217] For each second output result, a signal power value is calculated based on the second output result, a noise power value is calculated based on the first output result and the second output result, a signal-to-noise ratio is calculated based on the signal power value and the noise power value, and a difference value between the first output result and the second output result is calculated based on the signal-to-noise ratio.

[0218] In embodiments of the present application, based on the foregoing scheme, the original model parameters include original weight values, and the quantized model parameters include quantized weight values; the obtaining module 802 is specifically configured to:

[0219] obtain an activation input value of the model function network;

[0220] perform a multiplication operation on the activation input value and the original weight values to obtain a first output result of the model function network, and perform a multiplication operation on the activation input value and each quantized weight value to obtain a plurality of second output results of the model function network.

[0221] In embodiments of the present application, based on the foregoing scheme, the model function network corresponds to a plurality of data channels, and each data channel corresponds to a channel value; the obtaining module 802 is further specifically configured to:

[0222] if it is detected that there is an outlying channel value, taking a data channel corresponding to a largest outlying channel value as a target data channel, and taking a channel value corresponding to the target data channel as an activation input value of the model function network, and obtaining original weight values and quantized weight values corresponding to the target data channel;

[0223] performing a multiplication operation on the channel value corresponding to the target data channel and the original weight values to obtain a first output result of the model function network, and performing a multiplication operation on the channel value corresponding to the target data channel and each quantized weight value to obtain a plurality of second output results of the model function network.

[0224] In embodiments of the present application, based on the foregoing scheme, the obtaining module 802 is further specifically configured to:

[0225] calculating a similarity between the target data channel and a neighboring data channel corresponding to the target data channel;

[0226] performing a reduction processing on the outlying channel value in the target data channel based on the similarity to obtain a target channel value;

[0227] taking the target channel value and channel values other than the outlying channel value in the target data channel as an activation input value of the model function network.

[0228] In an embodiment of the present application, based on the foregoing scheme, the acquisition module 802 is further specifically configured to:

[0229] perform distance operation on the channel value of the target data channel and the channel value of the adjacent data channel corresponding to the target data channel to obtain a distance value, the distance value being greater than 0 and less than 1, and take the distance value as the similarity between the target data channel and the adjacent data channel;

[0230] perform product operation on the distance value and the outlier channel value in the target data channel to obtain a target channel value.

[0231] In an embodiment of the present application, based on the foregoing scheme, the device further comprises:

[0232] The generation module is configured to generate a target quantization scaling factor based on the target quantization bit mode;

[0233] The storage module is configured to store the target quantization bit mode, the target quantization scaling factor, and the identification information corresponding to the model function network in a specified area;

[0234] The inference module is configured to, if it is detected that the inference stage of the model function network, acquire the target quantization bit mode and the target quantization scaling factor from the specified area to perform quantization processing on the original model parameter.

[0235] In an embodiment of the present application, based on the foregoing scheme, the model function network comprises a plurality of; the generation module is specifically configured to:

[0236] If there is a first model function network in the plurality of model function networks whose quantization precision of the target quantization bit mode is greater than a preset precision threshold, the generation module generates a target quantization scaling factor corresponding to each data channel in the first model function network;

[0237] If there is a second model function network in the plurality of model function networks whose quantization precision of the target quantization bit mode is less than or equal to the preset precision threshold, the generation module groups the channel values of each data channel in the second model function network and generates a target quantization scaling factor corresponding to each combination.

[0238] In an embodiment of the present application, based on the foregoing scheme, the generation module is further specifically configured to:

[0239] Acquire a target quantization bit mode corresponding to the first model function network;

[0240] For each data channel in the first model function network, based on the obtained target quantization bit mode, the activation input value corresponding to the data channel, and each quantization scaling factor, an operation is performed to obtain an operation result corresponding to each quantization scaling factor, and a quantization scaling factor corresponding to an operation result with the largest absolute value in the multiple operation results is taken as a target quantization scaling factor of the data channel.

[0241] In an embodiment of the present application, based on the foregoing scheme, the generating module is further specifically configured to:

[0242] The first quantization scaling factor in the first scaling factor list is traversed, and based on the obtained target quantization bit mode, the activation input value corresponding to the data channel, and the traversed quantization scaling factor, an operation is performed to obtain an operation result corresponding to the traversed quantization scaling factor;

[0243] The activation input value corresponding to the data channel is updated, a next adjacent quantization scaling factor of the first quantization scaling factor is traversed, and based on the obtained target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor, an operation is performed to obtain an operation result corresponding to the traversed quantization scaling factor, until the first scaling factor list is traversed to obtain an operation result corresponding to each quantization scaling factor in the first scaling factor list.

[0244] In an embodiment of the present application, based on the foregoing scheme, the generating module is further specifically configured to:

[0245] The target quantization bit mode corresponding to the second model function network is obtained;

[0246] For each combination corresponding to each data channel in the second model function network, based on the obtained target quantization bit mode, the activation input value corresponding to the combination, and each quantization scaling factor, an operation is performed to obtain an operation result corresponding to each quantization scaling factor, and a quantization scaling factor corresponding to an operation result with the largest absolute value in the multiple operation results is taken as a target quantization scaling factor of the combination.

[0247] In an embodiment of the present application, based on the foregoing scheme, the generating module is further specifically configured to:

[0248] The first quantization scaling factor in the second scaling factor list is traversed, and based on the obtained target quantization bit mode, the activation input value corresponding to the combination, and the traversed quantization scaling factor, an operation is performed to obtain an operation result corresponding to the traversed quantization scaling factor;

[0249] update the activation input value corresponding to the combination, traverse a next adjacent quantization scaling factor of the first quantization scaling factor, and perform operation based on the obtained target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor to obtain an operation result corresponding to the traversed quantization scaling factor, until the second scaling factor list is traversed to obtain an operation result corresponding to each quantization scaling factor in the second scaling factor list.

[0250] In the embodiments of the present application, based on the foregoing scheme, the generating module is further specifically configured to:

[0251] obtain the quantization scaling factor corresponding to the target quantization bit mode through a plurality of devices respectively deployed with the hybrid bit quantization model;

[0252] select the maximum quantization scaling factor from the plurality of quantization scaling factors as the target quantization scaling factor.

[0253] It should be noted that the apparatus provided in the foregoing embodiments and the method provided in the foregoing embodiments belong to the same concept, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiments.

[0254] The embodiments of the present application further provide an electronic device, including: one or more processors; and a memory for storing one or more computer programs, when the one or more computer programs are executed by the one or more processors, the electronic device implements the quantization-based data processing method as described in the foregoing embodiments.

[0255] FIG. 9 is a structural schematic diagram of a computer system of an electronic device (such as the terminal device or the server shown in FIG. 1) suitable for use to implement the embodiments of the present application.

[0256] It should be noted that the computer system 900 of the electronic device shown in FIG. 9 is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0257] As shown in FIG. 9, the computer system 900 includes a central processing unit (CPU) 901 which can perform various appropriate actions and processes, such as executing the methods in the above-described embodiments, according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage section 908 into a random access memory (RAM) 903. In the RAM 903, various computer programs and data required for system operation are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0258] Connected to the I / O interface 905 are an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as necessary. A removable recording medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 910 as necessary, so that a computer program read therefrom is installed into the storage section 908 as necessary.

[0259] In particular, according to the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer program. For example, the embodiments of the present application include a computer program product including a computer program carried on a computer-readable medium, the computer program containing computer instructions for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 909, and / or installed from the removable recording medium 911. When the computer program is executed by the central processing unit (CPU) 901, various functions defined in the system of the embodiments of the present application are performed.

[0260] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable medium may, for example, be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer-readable medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present application, the computer-readable medium can be any tangible medium containing or storing a computer program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the embodiments of the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable computer programs. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, transmit, propagate, or transport a computer program for use by or in conjunction with an instruction execution system, device or apparatus. The computer program contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination of the above.

[0261] The flowcharts and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In each flowchart or block diagram, each block can represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by special-purpose hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0262] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware, and the units described can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0263] Embodiments of the present application also provide a computer readable medium having a computer program stored thereon, the computer program being executed by a processor to implement the foregoing data processing method based on quantization. The computer readable medium can be included in the electronic device described in the foregoing embodiments, or can exist separately and not be assembled into the electronic device.

[0264] Embodiments of the present application also provide a computer program product or computer program, which includes computer instructions stored in a computer readable medium. The processor of the electronic device reads the computer instructions from the computer readable medium, and the processor executes the computer instructions to enable the electronic device to perform the data processing method based on quantization provided in the foregoing embodiments.

[0265] The foregoing is merely exemplary embodiments of the present application, and is not intended to limit the implementation of the present application. Those skilled in the art can easily make corresponding modifications or variations based on the main concept and spirit of the embodiments of the present application, and therefore the protection scope of the present application should be subject to the protection scope required by the claims.

Claims

1. A quantization-based data processing method, comprising: quantizing original model parameters corresponding to a model function network through a plurality of quantization bit modes respectively to obtain quantized model parameters corresponding to each quantization bit mode; obtaining a first output result of the model function network under the original model parameters, and obtaining a second output result of the model function network under each quantized model parameter; selecting a target quantization bit mode from the plurality of quantization bit modes based on the first output result and a plurality of second output results, the target quantization bit mode being used for quantizing the original model parameters in an inference stage of the model function network.

2. The method of claim 1, wherein, The selecting of the target quantization bit mode from the plurality of quantization bit modes based on the first output result and the plurality of second output results comprises: calculating a difference value between the first output result and each second output result; selecting a quantization bit mode corresponding to a smallest difference value from the plurality of quantization bit modes, and taking the selected quantization bit mode as the target quantization bit mode.

3. The method of claim 2, wherein, The calculating of the difference value between the first output result and each second output result comprises: for each second output result, performing a difference operation on the first output result and the second output result to obtain a difference value, and performing a specified power operation on the obtained difference value to obtain the difference value between the first output result and the second output result.

4. The method of claim 2, wherein, The calculating of the difference value between the first output result and each second output result comprises: for each second output result, calculating a signal power value based on the second output result, calculating a noise power value based on the first output result and the second output result, calculating a signal-to-noise ratio based on the signal power value and the noise power value, and calculating the difference value between the first output result and the second output result based on the signal-to-noise ratio.

5. The method according to any one of claims 1 to 4, wherein, The original model parameters comprise original weight values, and the quantized model parameters comprise quantized weight values; the obtaining of the first output result of the model function network under the original model parameters and the obtaining of the second output result of the model function network under each quantized model parameter comprise: obtaining an activation input value of the model function network; performing a product operation on the activation input value and the original weight value to obtain the first output result of the model function network, and performing a product operation on the activation input value and each quantized weight value to obtain a plurality of second output results of the model function network.

6. The method of claim 5, wherein, The model function network corresponds to a plurality of data channels, and each data channel corresponds to a channel value; the obtaining of the activation input value of the model function network comprises: detecting whether there is an outlying channel value in each data channel of the plurality of data channels; If it is detected that there are outlier channel values, compare all detected outlier channel values, take the data channel corresponding to the maximum outlier channel value as a target data channel, take the channel value corresponding to the target data channel as the activation input value of the model function network, and obtain the original weight value and the quantized weight value corresponding to the target data channel; The product operation is performed on the activation input value and the original weight value to obtain a first output result of the model function network, and the product operation is performed on the activation input value and each quantized weight value to obtain a plurality of second output results of the model function network, including: The product operation is performed on the channel value corresponding to the target data channel and the original weight value to obtain a first output result of the model function network, and the product operation is performed on the channel value corresponding to the target data channel and each quantized weight value to obtain a plurality of second output results of the model function network.

7. The method of claim 6, wherein, The channel value corresponding to the target data channel is taken as the activation input value of the model function network, including: The similarity between the target data channel and the adjacent data channel corresponding to the target data channel is calculated; The outlier channel value in the target data channel is reduced based on the similarity to obtain a target channel value; The target channel value and the channel value in the target data channel except the outlier channel value are taken as the activation input value of the model function network.

8. The method of claim 7, wherein, The similarity between the target data channel and the adjacent data channel corresponding to the target data channel is calculated, including: The distance operation is performed on the channel value of the target data channel and the channel value of the adjacent data channel corresponding to the target data channel to obtain a distance value, the distance value is greater than 0 and less than 1, and the distance value is taken as the similarity between the target data channel and the adjacent data channel; The outlier channel value in the target data channel is reduced based on the similarity to obtain a target channel value, including: The product operation is performed on the distance value and the outlier channel value in the target data channel to obtain a target channel value.

9. The method of any one of claims 1 to 8, wherein, After the target quantization bit mode is selected from the plurality of quantization bit modes based on the first output result and the plurality of second output results, the method further includes: A target quantization scaling factor is generated based on the target quantization bit mode; The target quantization bit mode, the target quantization scaling factor, and the identification information corresponding to the model function network are associated and stored in a specified area; If it is detected that the model function network is in an inference stage, the original model parameters are quantized by obtaining the target quantization bit mode and the target quantization scaling factor from the specified area.

10. The method of claim 9, wherein, The model function network includes a plurality of; the target quantization scaling factor is generated based on the target quantization bit mode, including: If there is a first model function network in which the quantization precision of the target quantization bit mode is greater than a preset precision threshold in the plurality of model function networks, a target quantization scaling factor corresponding to each data channel in the first model function network is generated; If there is a second model function network in the plurality of model function networks, quantization precision of a target quantization bit mode of which is less than or equal to the preset precision threshold, the channel values of each data channel in the second model function network are grouped, and a target quantization scaling factor corresponding to each combination is generated.

11. The method of claim 10, wherein, The generating of the target quantization scaling factor corresponding to each data channel in the first model function network comprises: obtaining a target quantization bit mode corresponding to the first model function network; for each data channel in the first model function network, performing operation based on the obtained target quantization bit mode, the activation input value corresponding to the data channel, and each quantization scaling factor, obtaining an operation result corresponding to each quantization scaling factor, and taking the quantization scaling factor corresponding to the operation result with the largest absolute value in the plurality of operation results as the target quantization scaling factor of the data channel.

12. The method of claim 11, wherein, The operation based on the obtained target quantization bit mode, the activation input value corresponding to the data channel, and each quantization scaling factor to obtain an operation result corresponding to each quantization scaling factor comprises: traversing a first scaling factor list to obtain a first quantization scaling factor, and performing operation based on the obtained target quantization bit mode, the activation input value corresponding to the data channel, and the traversed quantization scaling factor to obtain an operation result corresponding to the traversed quantization scaling factor; updating the activation input value corresponding to the data channel, traversing a next adjacent quantization scaling factor of the first quantization scaling factor, and performing operation based on the obtained target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor to obtain an operation result corresponding to the traversed quantization scaling factor, until the first scaling factor list is traversed to obtain an operation result corresponding to each quantization scaling factor in the first scaling factor list.

13. The method according to any one of claims 10-12, wherein, The generating of the target quantization scaling factor corresponding to each combination comprises: obtaining a target quantization bit mode corresponding to the second model function network; for each combination corresponding to each data channel in the second model function network, performing operation based on the obtained target quantization bit mode, the activation input value corresponding to the combination, and each quantization scaling factor to obtain an operation result corresponding to each quantization scaling factor, and taking the quantization scaling factor corresponding to the operation result with the largest absolute value in the plurality of operation results as the target quantization scaling factor of the combination.

14. The method of claim 13, wherein, The operation based on the obtained target quantization bit mode, the activation input value corresponding to the combination, and each quantization scaling factor to obtain an operation result corresponding to each quantization scaling factor comprises: traversing a second scaling factor list to obtain a first quantization scaling factor, and performing operation based on the obtained target quantization bit mode, the activation input value corresponding to the combination, and the traversed quantization scaling factor to obtain an operation result corresponding to the traversed quantization scaling factor; The activation input value corresponding to the combination is updated, and the next adjacent quantization scaling factor of the first quantization scaling factor is traversed, and an operation is performed based on the obtained target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor to obtain an operation result corresponding to the traversed quantization scaling factor, until the second scaling factor list is traversed to obtain an operation result corresponding to each quantization scaling factor in the second scaling factor list.

15. The method of claim 9, wherein, The target quantization scaling factor is generated based on the target quantization bit mode, including: A plurality of devices deployed with a hybrid bit quantization model are used to obtain quantization scaling factors corresponding to the target quantization bit mode respectively; The maximum quantization scaling factor is selected from the plurality of quantization scaling factors as the target quantization scaling factor.

16. A quantization-based data processing apparatus, comprising: a quantization processing module configured to quantize original model parameters corresponding to a model function network through a plurality of quantization bit modes to obtain quantization model parameters corresponding to each quantization bit mode; an acquisition module configured to acquire a first output result of the model function network under the original model parameters, and acquire second output results of the model function network under each quantization model parameter; a selection module configured to select a target quantization bit mode from the plurality of quantization bit modes based on the first output result and the plurality of second output results, the target quantization bit mode being used to quantize the original model parameters in an inference stage corresponding to the model function network.

17. An electronic device, comprising: one or more processors; and a memory for storing one or more programs, when the one or more programs are executed by the electronic device, causing the electronic device to implement the quantization-based data processing method according to any one of claims 1 to 15.

18. A computer readable medium having stored thereon a computer program, wherein, The computer program is executed by the processor to implement the quantization-based data processing method according to any one of claims 1 to 15.

19. A computer program product comprising computer instructions, wherein, The computer program is executed by the processor to implement the quantization-based data processing method according to any one of claims 1 to 15. The computer program is executed by the processor to implement the quantization-based data processing method according to any one of claims 1 to 15.

Images