Data processing method, device, equipment and medium based on quantization

By evaluating and selecting multiple quantized bit methods of the model functional network and selecting the quantized bit method corresponding to the minimum difference value, the problem of low accuracy of quantization processing in the prior art is solved, and the model performance and robustness are improved.

CN118673290BActive Publication Date: 2024-11-05TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411010046.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-05
Estimated Expiration
2044-07-26

AI Technical Summary

Technical Problem

In the prior art, there are errors in the selection of quantization bit modes for different functional networks of the model, resulting in low accuracy of quantization processing and affecting model performance.

Method used

Through multiple quantization bit methods, the original model parameters of the model functional network are quantized, the output results under each quantization bit method are obtained, the difference value is calculated and the quantization bit method corresponding to the minimum difference value is selected as the target quantization bit method, and used for the model The reasoning stage of functional networks.

Benefits of technology

It improves the accuracy of quantitative processing, improves the generalization ability and robustness of the model outside the training data, reduces resource consumption, and improves model performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118673290B_ABST
    Figure CN118673290B_ABST
Patent Text Reader

Abstract

The embodiments of the present application disclose a data processing method, device, equipment and medium based on quantization. The method includes: quantizing the original model parameters corresponding to the model function network respectively through multiple quantization bit modes to obtain the quantization model parameters corresponding to each quantization bit mode; then obtaining the first output result of the model function network under the original model parameters, and obtaining the second output result of the model function network under each quantization model parameter; then based on the first output result and multiple second output results, selecting a target quantization bit mode from multiple quantization bit modes, and the target quantization bit mode is used to quantize the original model parameters in the inference stage of the model function network. The technical solution of the present application improves the accuracy of the quantization processing and has good model performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular, to a quantization-based data processing method, a quantization-based data processing device, an electronic device, and a computer-readable medium. Background Art

[0002] Two schemes are usually used in the related art to determine the quantization bit modes corresponding to different functional networks of the model. In scheme one, the user selects the quantization bit mode for each functional network based on manual experience. In scheme two, the quantization bit mode is selected for each functional network according to the task type of the functional network. For example, for a functional network that requires higher precision for the task type, a quantization bit mode with higher bits (such as fp16 or int8) is selected for it, and for a functional network that requires lower precision for the task type, a quantization bit mode with lower bits (such as int4 or int2) is selected for it.

[0003] Among them, in scheme one, due to insufficient manual experience, the quantization bit mode of the functional network may be incorrectly selected, which reduces the accuracy of the quantization processing and further affects the model performance; in scheme two, due to different models or different training stages of the models, the sensitivity of the functional network will change even for the same task type. Therefore, the quantization bit mode selected for the functional network according to the task type is relatively fixed, and there is also the phenomenon of incorrect selection of the quantization bit mode, which reduces the accuracy of the quantization processing and further affects the model performance.

[0004] Therefore, how to improve the accuracy of quantization processing to improve model performance is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a quantization-based data processing method, device, equipment and medium, which improve the accuracy of quantization processing and have good model performance.

[0006] In a first aspect, an embodiment of the present application provides a quantization-based data processing method, the method comprising: quantizing the original model parameters corresponding to a model function network through multiple quantization bit modes, respectively, to obtain the quantized model parameters corresponding to each quantization bit mode; obtaining the first output result of the model function network under the original model parameters, and obtaining the second output result of the model function network under each quantization model parameter; based on the first output result and multiple second output results, selecting a target quantization bit mode from the multiple quantization bit modes, the target quantization bit mode being used to quantize the original model parameters in the inference stage of the model function network.

[0007] In the second aspect, an embodiment of the present application provides a quantization-based data processing device, the device comprising: a quantization processing module, configured to perform quantization processing on the original model parameters corresponding to the model function network through multiple quantization bit modes, respectively, to obtain the quantization model parameters corresponding to each quantization bit mode; an acquisition module, configured to obtain the first output result of the model function network under the original model parameters, and to obtain the second output result of the model function network under each quantization model parameter; a selection module, configured to select a target quantization bit mode from the multiple quantization bit modes based on the first output result and multiple second output results, and the target quantization bit mode is used to perform quantization processing on the original model parameters in the inference stage corresponding to the model function network.

[0008] In one embodiment of the present application, based on the aforementioned scheme, the selection module is specifically configured as follows: calculating the difference value between the first output result and each second output result; selecting the quantization bit mode corresponding to the smallest difference value from the multiple quantization bit modes, and using the selected quantization bit mode as the target quantization bit mode.

[0009] In one embodiment of the present application, based on the aforementioned scheme, the selection module is further specifically configured as: for each second output result, performing a difference operation on the first output result and the second output result to obtain a difference value, and raising the obtained difference value to a specified power to obtain a difference value between the first output result and the second output result.

[0010] In one embodiment of the present application, based on the aforementioned scheme, the selection module is further specifically configured as: for each second output result, a signal power value is calculated based on the second output result, a noise power value is calculated based on the first output result and the second output result, a signal-to-noise ratio is calculated based on the signal power value and the noise power value, and a difference value between the first output result and the second output result is calculated based on the signal-to-noise ratio.

[0011] In one embodiment of the present application, based on the aforementioned scheme, the original model parameters include original weight values, and the quantized model parameters include quantized weight values; the acquisition module is specifically configured to: obtain the activation input value of the model function network; perform a product operation on the activation input value and the original weight value to obtain a first output result of the model function network, and perform a product operation on the activation input value and each quantized weight value to obtain multiple second output results of the model function network.

[0012] In one embodiment of the present application, based on the aforementioned scheme, the model function network corresponds to multiple data channels, and each data channel corresponds to a channel value; the acquisition module is also specifically configured as: if an outlier channel value is detected, the data channel corresponding to the largest channel value in the outlier is used as the target data channel, and the channel value corresponding to the target data channel is used as the activation input value of the model function network, and the original weight value and quantized weight value corresponding to the target data channel are obtained; the channel value corresponding to the target data channel and the original weight value are multiplied to obtain the first output result of the model function network, and the channel value corresponding to the target data channel and each quantized weight value are multiplied to obtain multiple second output results of the model function network.

[0013] In one embodiment of the present application, based on the aforementioned scheme, the acquisition module is further specifically configured to: calculate the similarity between the target data channel and the adjacent data channel corresponding to the target data channel; based on the similarity, reduce the outlier channel value in the target data channel to obtain the target channel value; use the target channel value and the channel value in the target data channel except the outlier channel value as the activation input value of the model functional network.

[0014] In one embodiment of the present application, based on the aforementioned scheme, the acquisition module is further specifically configured as follows: performing a distance operation based on the channel value of the target data channel and the channel value of the adjacent data channel corresponding to the target data channel to obtain a distance value, wherein the distance value is greater than 0 and less than 1, and the distance value is used as the similarity between the target data channel and the adjacent data channel; performing a product operation on the distance value and the outlier channel value in the target data channel to obtain the target channel value.

[0015] In one embodiment of the present application, based on the aforementioned scheme, the device also includes: a generation module, configured to generate a target quantization scaling factor based on the target quantization bit mode; a storage module, configured to associate and store the target quantization bit mode, the target quantization scaling factor, and the identification information corresponding to the model function network in a designated area; an inference module, configured to obtain the target quantization bit mode and the target quantization scaling factor from the designated area to quantize the original model parameters if it is detected that it is in the inference stage of the model function network.

[0016] In one embodiment of the present application, based on the aforementioned scheme, the model function network includes multiple; the generation module is specifically configured as follows: if there is a first model function network among the multiple model function networks whose quantization accuracy in the target quantization bit mode is greater than a preset accuracy threshold, then a target quantization scaling factor corresponding to each data channel in the first model function network is generated; if there is a second model function network among the multiple model function networks whose quantization accuracy in the target quantization bit mode is less than or equal to the preset accuracy threshold, then the channel values ​​of each data channel in the second model function network are grouped, and a target quantization scaling factor corresponding to each combination is generated.

[0017] In one embodiment of the present application, based on the aforementioned scheme, the generation module is further specifically configured to: obtain a target quantization bit mode corresponding to the first model functional network; for each data channel in the first model functional network, perform operations based on the obtained target quantization bit mode, the activation input value corresponding to the data channel, and each quantization scaling factor to obtain an operation result corresponding to each quantization scaling factor, and use the quantization scaling factor corresponding to the operation result with the largest absolute value among the multiple operation results as the target quantization scaling factor of the data channel.

[0018] In one embodiment of the present application, based on the aforementioned scheme, the generation module is further specifically configured as follows: traversing the first quantization scaling factor in the first scaling factor list, and performing operations based on the acquired target quantization bit mode, the activation input value corresponding to the data channel, and the traversed quantization scaling factor to obtain the operation result corresponding to the traversed quantization scaling factor; updating the activation input value corresponding to the data channel, and traversing the next adjacent quantization scaling factor of the first quantization scaling factor, and performing operations based on the acquired target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor to obtain the operation result corresponding to the traversed quantization scaling factor, until the first scaling factor list is traversed and the operation result corresponding to each quantization scaling factor in the first scaling factor list is obtained.

[0019] In one embodiment of the present application, based on the aforementioned scheme, the generation module is further specifically configured to: obtain the target quantization bit mode corresponding to the second model functional network; for each combination corresponding to each data channel in the second model functional network, perform operations based on the obtained target quantization bit mode, the activation input value corresponding to the combination, and each quantization scaling factor to obtain the operation result corresponding to each quantization scaling factor, and use the quantization scaling factor corresponding to the operation result with the largest absolute value among the multiple operation results as the target quantization scaling factor of the combination.

[0020] In one embodiment of the present application, based on the aforementioned scheme, the generation module is further specifically configured to: traverse the first quantization scaling factor in the second scaling factor list, and perform operations based on the acquired target quantization bit mode, the activation input value corresponding to the combination, and the traversed quantization scaling factor to obtain the operation result corresponding to the traversed quantization scaling factor; update the activation input value corresponding to the combination, and traverse the next adjacent quantization scaling factor of the first quantization scaling factor, and perform operations based on the acquired target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor to obtain the operation result corresponding to the traversed quantization scaling factor, until the second scaling factor list is traversed and the operation result corresponding to each quantization scaling factor in the second scaling factor list is obtained.

[0021] In one embodiment of the present application, based on the aforementioned scheme, the generation module is further specifically configured as: obtaining the quantization scaling factors corresponding to the target quantization bit mode respectively through multiple devices deployed with a mixed bit model; and selecting the largest quantization scaling factor from the multiple quantization scaling factors as the target quantization scaling factor.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors; and a memory for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the quantization-based data processing method as described above.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the quantization-based data processing method as described above.

[0024] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer instructions, which, when executed by a processor, implement the quantization-based data processing method as described above.

[0025] In the technical solution provided in the embodiments of the present application:

[0026] The original model parameters corresponding to the model are quantized through multiple quantization bit modes to obtain the quantization model parameters corresponding to each quantization bit mode, and the first output result of the model function network under the original model parameters and the second output result of the model function network under each quantization model parameter are used to select the target quantization bit mode from the multiple quantization bit modes; thereby, the accurate selection of the quantization bit mode of the model function network is achieved, the accuracy of the quantization processing is high, and the high accuracy of the quantization processing helps to improve the generalization ability of the model beyond the training data, and can better adapt to new data distributions and scenarios. At the same time, it also enhances the stability and robustness of the model, and can maintain good performance under complex conditions. In addition, it can perform reasoning faster, reduce unnecessary resource consumption, and improve resource utilization, so the model performance is greatly improved, so that tasks in various application scenarios can be well executed through the model.

[0027] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of an exemplary implementation environment to which the technical solution of the embodiments of the present application can be applied.

[0029] Figure 2 It is a flowchart of a data processing method based on quantization shown in an exemplary embodiment of the present application.

[0030] Figure 3a is a schematic diagram of a data channel including a channel value shown in an exemplary embodiment of the present application.

[0031] Figure 3b is a schematic diagram of a data channel including a channel value shown in another exemplary embodiment of the present application.

[0032] Figure 4 is a flow chart of a data processing method based on quantization shown in another exemplary embodiment of the present application.

[0033] Figure 5 It is a schematic diagram of generating a target quantization scaling factor according to an exemplary embodiment of the present application.

[0034] Figure 6 is a flow chart of a data processing method based on quantization shown in another exemplary embodiment of the present application.

[0035] Figure 7 It is a schematic diagram of a method for generating target quantization bits shown in an exemplary embodiment of the present application.

[0036] Figure 8It is a block diagram of a quantization-based data processing device shown as an exemplary embodiment of the present application.

[0037] Fig. 9 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0038] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments that are the same as the present application. Instead, they are only examples of devices and methods that are the same as some aspects of the present application as detailed in the attached claims.

[0039] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0040] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0041] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0042] It should be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0043] Before introducing the technical solutions of the embodiments of the present application, the nouns and terms involved in the embodiments of the present application are explained first. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0044] The Mixture of Experts (MoE) is a neural network architecture used to solve complex tasks. It includes an expert network and a gating network. Each expert network is responsible for processing different aspects or subtasks of the input data. For example, in image processing tasks, one expert network is used to process edge detection, and another expert network is used for texture recognition. The gating network is responsible for dynamically selecting which expert network to process the current input data. It can be understood that the Mixture of Experts model effectively improves the model's complex task processing capabilities and performance by integrating the gating network and multiple expert networks.

[0045] Hybrid bit quantization is a method for optimizing model reasoning, which can be applied to hybrid expert models. It allows different quantization precisions to be used in the same model to balance the computational efficiency and precision requirements of the model. Specifically, in the model reasoning process, different expert networks (also called functional networks) can use different quantization precisions. For example, some functional networks use int8 (an 8-bit integer representation) quantization, some functional networks use int4 (a 4-bit integer representation) quantization, some functional networks use int2 (a 2-bit integer representation) quantization, and some functional networks use fp16 (Fixed Point 16-bit, a 16-bit floating point representation) quantization, etc.

[0046] Two schemes are usually used in the related art to determine the quantization bit modes corresponding to different functional networks of the model. In scheme one, the user selects the quantization bit mode for each functional network based on manual experience. In scheme two, the quantization bit mode is selected for each functional network according to the task type of the functional network. For example, for a functional network that requires higher precision for the task type, a quantization bit mode with higher bits (such as fp16 or int8) is selected for it, and for a functional network that requires lower precision for the task type, a quantization bit mode with lower bits (such as int4 or int2) is selected for it.

[0047] Among them, in scheme one, due to insufficient manual experience, the quantization bit mode of the functional network may be incorrectly selected, which reduces the accuracy of the quantization processing and further affects the model performance; in scheme two, due to different models or different training stages of the models, the sensitivity of the functional network will change even for the same task type. Therefore, the quantization bit mode selected for the functional network according to the task type is relatively fixed, and there is also the phenomenon of incorrect selection of the quantization bit mode, which reduces the accuracy of the quantization processing and further affects the model performance.

[0048] Therefore, in order to improve the accuracy of quantization processing and improve model performance, this application provides a data processing solution based on quantization. Figure 1 , Figure 1 1 is a schematic diagram of an implementation environment involved in the present application. The implementation environment mainly includes a terminal device 101 and a server 102; wherein:

[0049] The terminal device 101 includes but is not limited to smart phones, computers (tablet computers, laptops, desktop computers, etc.), smart home devices (televisions, refrigerators, air conditioners, washing machines, sweeping and mopping robots, etc.), smart wearable devices (bracelets, watches, etc.), etc.

[0050] Server 102 can be an independent physical server or a server cluster or distributed system composed of multiple physical servers, wherein the server cluster or distributed system includes cloud servers for providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0051] It is understood that the terminal device 101 and the server 102 establish a communication connection through a wired or wireless network. Exemplarily, the wireless network or wired network uses standard communication technology and / or protocols. The network is usually the Internet, but it can also be any other network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network, etc.

[0052] In one embodiment of the present application, quantization-based data processing can be performed by server 102. Specifically, server 102 quantizes the original model parameters corresponding to the model function network through multiple quantization bit modes, and obtains the quantized model parameters corresponding to each quantization bit mode; then obtains the first output result of the model function network under the original model parameters, and obtains the second output result of the model function network under each quantization model parameter; then, based on the first output result and multiple second output results, selects a target quantization bit mode from multiple quantization bit modes, and the target quantization bit mode is used to quantize the original model parameters in the inference stage of the model function network.

[0053] In other embodiments of the present application, the quantization-based data processing method can also be executed by the terminal device 101 alone or jointly by the terminal device 101 and the server 102. In actual applications, the execution entity of the quantization-based data processing method can be flexibly adjusted according to the specific application scenario.

[0054] It should be made clear that Figure 1 The number of terminal devices 101 and servers 102 in the figure is only for illustration, and any number of terminal devices 101 and servers 102 may be provided according to actual needs.

[0055] It should be noted that in the specific implementation of this application, user-related data is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0056] The following is a detailed description of various implementation details of the technical solution of the embodiment of the present application:

[0057] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of a data processing method based on quantization according to an embodiment of the present application. Figure 2 As shown, the quantization-based data processing method at least includes S201 to S203, which are described in detail as follows:

[0058] S201, quantizing the original model parameters corresponding to the model function network respectively through multiple quantization bit modes to obtain quantized model parameters corresponding to each quantization bit mode.

[0059] In the embodiments of the present application, the quantization bit method includes but is not limited to at least one of an integer representation of any bit position, a floating point representation of any bit position, etc.; wherein the arbitrary bit position can be 2, 3, 4, 6, 8, 16, 32, 64, 128, etc.

[0060] In the embodiment of the present application, the original model parameters refer to the model parameters corresponding to the functional network of the model (all referred to as the model functional network) that are not quantized by the quantization bit method. Correspondingly, the quantized model parameters refer to the model parameters corresponding to the model functional network that are quantized by the quantization bit method. It can be understood that the model parameters include but are not limited to at least one of the weight value (weight), activation input value (activation), cache value (KV-Cache), etc. In the embodiment of the present application, the model parameter is taken as the weight value as an example. At this time, the original model parameter is the original weight value, and the quantized model parameter is the quantized weight value, and the others can be deduced by analogy.

[0061] In the embodiment of the present application, multiple quantization bit modes are used to quantize the original model parameters respectively, so as to obtain quantization model parameters corresponding to the multiple quantization bit modes respectively.

[0062] For example, assuming that multiple quantization bit modes are int2, int4, and int8, and w0 is used to represent the original weight value, and winti is used to represent the quantization weight value corresponding to the quantization bit mode inti; then the quantization weight value corresponding to int2 is wint2=int2(w0), the quantization weight value corresponding to int4 is wint4=int4(w0), and the quantization weight value corresponding to int8 is wint8=int8(w0).

[0063] S202, obtaining a first output result of the model function network under the original model parameters, and obtaining a second output result of the model function network under each quantized model parameter.

[0064] In the embodiment of the present application, the quantization model parameters corresponding to each quantization bit mode are obtained, and then the first output result of the model function network under the original model parameters can be obtained, and the second output result of the model function network under each quantization model parameter can be obtained.

[0065] In the embodiment of the present application, the first output result refers to the result output by the model function network under the original model parameters, and the second output result refers to the result output by the model function network under the quantized model parameters.

[0066] In one embodiment of the present application, the original model parameters include original weight values, and the quantized model parameters include quantized weight values; accordingly, the process of obtaining the first output result of the model function network under the original model parameters and obtaining the second output result of the model function network under each quantized model parameter in S202 may include:

[0067] Get the activation input value of the model function network;

[0068] The activation input value and the original weight value are multiplied to obtain a first output result of the model function network, and the activation input value and each quantized weight value are multiplied to obtain multiple second output results of the model function network.

[0069] That is, in an optional embodiment, the activation input value of the model function network is first obtained, and then the activation input value and the original weight value are multiplied to obtain the first output result of the model function network, and the activation input value and each quantized weight value are multiplied to obtain multiple second output results of the model function network.

[0070] For example, continuing with the previous example, use a to represent the activation value, out0 to represent the first output result, and outinti to represent the second output result corresponding to the quantization bit mode inti; then out0=a×w0, the second output result corresponding to int2 is outint2=a×wint2, the second output result corresponding to int4 is outint4=a×wint4, and the second output result corresponding to int8 is outint8=a×wint8.

[0071] In this way, by implementing the optional embodiment, the first output result and multiple second output results can be obtained simply and accurately.

[0072] In one embodiment of the present application, the model function network corresponds to multiple data channels, each data channel corresponds to a channel value, and each data channel usually corresponds to multiple channel values.

[0073] Optionally, the process of obtaining the activation input value of the model function network may include:

[0074] If an outlier channel value is detected, the data channel corresponding to the largest channel value in the outlier is used as the target data channel, and the channel value corresponding to the target data channel is used as the activation input value of the model function network, and the original weight value and quantized weight value corresponding to the target data channel are obtained;

[0075] Accordingly, the process of performing a product operation on the activation input value and the original weight value to obtain a first output result of the model function network, and performing a product operation on the activation input value and each quantized weight value to obtain multiple second output results of the model function network may include:

[0076] The channel value corresponding to the target data channel and the original weight value are multiplied to obtain a first output result of the model function network, and the channel value corresponding to the target data channel and each quantized weight value are multiplied to obtain multiple second output results of the model function network.

[0077] In the optional embodiment, the outlier channel value is for the data channel; for example, see Figure 3a , is a data channel c1 including multiple channel values, such as Figure 3a As shown, a-c1-301 and a-c1-302 are both outlier channel values; see Figure 3b , is a data channel c2 including multiple channel values, such as Figure 3b As shown, a-c2-301, a-c2-302, and a-c3-303 are all outlier channel values.

[0078] Among them, an optional embodiment is to select the largest channel value from multiple outlier channel values, and use the data channel corresponding to the largest channel value as the target data channel; for example, following the previous example, assuming that a-c1-301 is the largest among a-c1-301, a-c1-302, a-c2-301, a-c2-302, and a-c3-303, then data channel c1 is the target data channel.

[0079] It is understandable that in other embodiments, there may be only one outlier channel value, and the data channel corresponding to the outlier channel value is the target data channel.

[0080] In an optional embodiment, the channel value corresponding to the target data channel is used as the activation input value of the model function network; for example, following the above example, Figure 3a As shown, all channel values ​​included in the data channel c1 are the activation input values ​​of the model function network.

[0081] Among them, in an optional embodiment, the channel value corresponding to the target data channel and the original weight value corresponding to the target data channel are multiplied to obtain the first output result of the target data channel. At this time, the first output result of the target data channel is the first output result of the model function network, and the channel value corresponding to the target data channel and each quantized weight value corresponding to the target data channel are multiplied to obtain multiple second output results of the target data channel. At this time, the multiple second output results of the target data channel are the multiple second output results of the model function network.

[0082] For example, taking the above data channel c1 as the target data channel, a-c1 is used to represent multiple channel values ​​corresponding to the target data channel c1 (i.e. Figure 3a multiple points in the target data channel); then the first output result of the target data channel c1 is out0-c1=(a-c1)×w0, the second output result of the target data channel c1 using int2 is outint2-c1=(a-c1)×wint2, the second output result of the target data channel c1 using int4 is outint4-c1=(a-c1)×wint4, and the second output result of the target data channel c1 using int8 is outint8-c1=(a-c1)×wint8.

[0083] In this way, by implementing the optional embodiment, since the maximum channel value in the outlier can reflect the sensitivity of the model functional network to a certain extent, the data channel corresponding to the maximum channel value of the outlier in the model functional network is used as the basis, and the accuracy of calculating the first output result and multiple second output results is higher.

[0084] In one embodiment of the present application, the process of using the channel value corresponding to the target data channel as the activation input value of the model function network may include:

[0085] Calculating the similarity between the target data channel and the adjacent data channels corresponding to the target data channel;

[0086] Based on the similarity, the outlier channel values ​​in the target data channel are reduced to obtain the target channel value;

[0087] The target channel value and the channel values ​​in the target data channel except the outlier channel value are used as activation input values ​​of the model function network.

[0088] That is, in an optional embodiment, the similarity between the target data channel and the adjacent data channels corresponding to the target data channel is used to reduce the outlier channel values ​​in the target data channel to obtain the target channel value. At this time, the target channel value and the channel values ​​in the target data channel except the outlier channel values ​​(i.e., other channel values) are the activation input values ​​of the model functional network.

[0089] For example, taking the aforementioned data channel c1 as the target data channel, c1' is used to represent the adjacent data channel of the target data channel c1; then the similarity s between the target data channel c1 and the adjacent data channel c1' is calculated, and then the similarity s is used to reduce the outlier channel values ​​a-c1-301 and a-c1-302 in the target data channel c1 to obtain the target channel value a-c1-301' corresponding to the outlier channel value a-c1-301, and the target channel value a-c1-302' corresponding to the outlier channel value a-c1-302. At this time, the target channel value a-c1-301', the target channel value a-c1-302', and the channel values ​​in a-c1 corresponding to the target data channel c1 except the outlier channel values ​​a-c1-301 and a-c1-302 are the activation input values ​​of the model functional network.

[0090] In this way, by implementing the optional embodiment, the outlier channel values ​​in the target data channel are reduced and represented by smaller numerical values, thereby obtaining the activation input value of the model function network, which helps to subsequently select a lower precision quantization bit method for the model function network.

[0091] In one embodiment of the present application, the process of calculating the similarity between the target data channel and the adjacent data channel corresponding to the target data channel may include:

[0092] A distance operation is performed based on the channel value of the target data channel and the channel value of the adjacent data channel corresponding to the target data channel to obtain a distance value, the distance value is greater than 0 and less than 1, and the distance value is used as the similarity between the target data channel and the adjacent data channel;

[0093] Accordingly, the outlier channel values ​​in the target data channel are reduced based on the similarity to obtain the target channel value, including:

[0094] The distance value and the outlier channel value in the target data channel are multiplied to obtain the target channel value.

[0095] That is, in an optional embodiment, the channel value of the adjacent data channel corresponding to the target data channel is first determined, and the channel value of the adjacent data channel is usually multiple. Then, a distance operation is performed on the channel value of the target data channel and the channel value of the adjacent data channel to obtain a distance value. At this time, the distance value is the similarity between the target data channel and the adjacent data channel. Thereafter, the distance value is used to reduce the outlier channel value in the target data channel to obtain the target channel value.

[0096] Among them, in an optional embodiment, a distance algorithm is used to calculate the distance value between the target data channel and the adjacent data channel. The distance algorithm includes but is not limited to at least one of cosine similarity, Euclidean distance, Manhattan distance, Chebyshev distance, and Minkowski distance.

[0097] For example, continuing the previous example, assuming that the distance algorithm is cosine similarity; then the distance value between the target data channel c1 and the adjacent data channel c1' = .

[0098] In the optional embodiment, after obtaining the distance value between the target data channel and the adjacent data channel, the distance value and the outlier channel value in the target channel may be multiplied to obtain the target channel value.

[0099] For example, continuing the previous example, the outlier channel values ​​are a-c1-301 and a-c1-302; then the target channel value a-c1-301' = (a-c1-301), target channel value a-c1-302'= (a-c1-302). It is understandable that, because 0< <1, so the target channel value a-c1-301' is less than a-c1-301, and the target channel value a-c1-302' is less than a-c1-302.

[0100] In this way, by implementing the optional embodiment, the target channel value can be obtained simply and accurately.

[0101] S203, based on the first output result and the multiple second output results, select a target quantization bit mode from multiple quantization bit modes, and the target quantization bit mode is used to quantize the original model parameters in the inference stage of the model function network.

[0102] In the embodiment of the present application, a first output result and multiple second output results are obtained, and then the first output result and the multiple second output results can be used to select a target quantization bit mode from multiple quantization bit modes, wherein the target quantization bit mode is used to quantize the original model parameters in the inference stage of the model function network.

[0103] In one embodiment of the present application, the process of selecting a target quantization bit mode from multiple quantization bit modes based on the first output result and the multiple second output results in S203 may include:

[0104] Calculating a difference value between the first output result and each second output result;

[0105] A quantization bit mode corresponding to the smallest difference value is selected from multiple quantization bit modes, and the selected quantization bit mode is used as a target quantization bit mode.

[0106] That is, in the optional embodiment, the difference between the first output result and each second output result is first calculated, and then the quantization bit mode corresponding to the smallest difference value is selected from multiple quantization bit modes, and the selected quantization bit mode is the target quantization bit mode. It can be understood that the smaller the difference value, the smaller the impact on the output result before and after the quantization bit mode is adopted; therefore, in the optional embodiment, the quantization bit mode corresponding to the smallest difference value is used as the target quantization bit mode.

[0107] For example, following the previous example, assuming that the calculated difference between the first output result out0 and the second output result outint2 is Lout0-outint2, the difference between the first output result out0 and the second output result outint4 is Lout0-outint4, and the difference between the first output result out0 and the second output result outint8 is Lout0-outint8, and Lout0-outint2 is the smallest among Lout0-outint2, Lout0-outint4, and Lout0-outint8; then int2 corresponding to Lout0-outint2 is selected from multiple quantization bit modes int2, int4, and int8 as the target quantization bit mode.

[0108] In this way, by implementing the optional embodiment, the target quantization bit mode can be selected from multiple quantization bit modes simply and accurately by calculating the difference value.

[0109] In one embodiment of the present application, the process of calculating the difference value between the first output result and each second output result may include:

[0110] For each second output result, a difference operation is performed on the first output result and the second output result to obtain a difference value, and the obtained difference value is raised to a specified power to obtain a difference value between the first output result and the second output result.

[0111] In an optional embodiment, the specified power may be quadratic, in which case the difference between the first output result and each second output result is calculated using a mean square error (MSE) algorithm.

[0112] For example, continuing the previous example, Lout0-outint2= 、Lout0-outint4= 、Lout0-outint8= .

[0113] In this way, by implementing the optional embodiment, the difference value between the first output result and each second output result can be obtained simply and accurately.

[0114] In one embodiment of the present application, the process of calculating the difference value between the first output result and each second output result may include:

[0115] For each second output result, a signal power value is calculated based on the second output result, a noise power value is calculated based on the first output result and the second output result, a signal-to-noise ratio is calculated based on the signal power value and the noise power value, and a difference value between the first output result and the second output result is calculated based on the signal-to-noise ratio.

[0116] That is, in an optional embodiment, a signal-to-noise ratio (SNR) algorithm is used to calculate the difference between the first output result and each second output result.

[0117] For example, following the above example, taking the calculation of the difference value Lout0-outint2 between the first output result out0 and the second output result outint2 as an example, using Psignal to represent the signal power value, then , use Pnoise to represent the noise power value, then , use SNR to represent the signal-to-noise ratio, then SNR= , accordingly, Lout0-outint2=-SNR, and the other difference values ​​are similar, which will not be repeated here.

[0118] In this way, by implementing the optional embodiment, the difference value between the first output result and each second output result can be obtained simply and accurately.

[0119] In one embodiment of the present application, the process of selecting a quantization bit mode corresponding to a minimum difference value from multiple quantization bit modes may include:

[0120] Selecting a difference value that is smaller than a preset difference threshold from the multiple difference values;

[0121] If there are multiple selected difference values, a quantization bit mode corresponding to the smallest difference value is selected from the multiple quantization bit modes.

[0122] That is, in an optional embodiment, a difference value less than a preset difference threshold is first selected from multiple difference values ​​as a candidate difference value, and when there are multiple candidate difference values, a quantization bit mode corresponding to the smallest candidate difference value is selected from multiple quantization bit modes.

[0123] For example, following the above example, assuming that the preset difference threshold is 0.001, and Lout0-outint2, Lout0-outint4, and Lout0-outint8, Lout0-outint2 and Lout0-outint4 are all less than 0.001, then Lout0-outint2 and Lout0-outint4 are candidate difference values, and assuming that Lout0-outint2 is smaller than Lout0-outint2 and Lout0-outint4, then int2 corresponding to Lout0-outint2 is selected from multiple quantization bit modes int2, int4, int8 as the target quantization bit mode. It can be understood that in actual applications, the preset difference threshold can be flexibly set according to the specific application scenario.

[0124] In this way, by implementing the optional embodiment, the preset difference threshold is first used for primary screening, and then the minimum difference value is compared for secondary screening, so that the target quantization bit mode can be accurately obtained.

[0125] In one embodiment of the present application, the process of selecting a quantization bit mode corresponding to a minimum difference value from multiple quantization bit modes may include:

[0126] Select the smallest difference value from multiple difference values;

[0127] If the selected difference value is smaller than the preset difference threshold, a quantization bit mode corresponding to the selected difference value is selected from a plurality of quantization bit modes.

[0128] That is, in an optional embodiment, the smallest difference value is first selected from multiple difference values ​​as a candidate difference value, and when the candidate difference value is less than a preset difference threshold, a quantization bit mode corresponding to the candidate difference value is selected from multiple quantization bit modes.

[0129] For example, following the above example, assuming that Lout0-outint2 is the smallest among Lout0-outint2, Lout0-outint4, and Lout0-outint8, Lout0-outint2 is the candidate difference value, and assuming that the preset difference threshold is 0.001, and Lout0-outint2 is less than 0.001, then int2 corresponding to Lout0-outint2 is selected from multiple quantization bit modes int2, int4, and int8 as the target quantization bit mode. It can be understood that in actual applications, the preset difference threshold can be flexibly set according to the specific application scenario.

[0130] In this way, by implementing the optional embodiment, the target quantization bit mode can be accurately obtained by first performing a primary screening by comparing the minimum difference value and then performing a secondary screening by using a preset difference threshold.

[0131] It should be noted that Figure 2 The number of model function networks S201 to S203 shown in the figure can be one or more. When there is one model function network, the final result is the corresponding target quantization bit mode. When there are multiple model function networks, the final result is the target quantization bit mode corresponding to each model function network, wherein the target quantization bit modes corresponding to different model function networks can be the same or different.

[0132] Please refer to Table 1, which is an example table of target quantization bit modes corresponding to multiple model function networks.

[0133]

[0134] In the embodiment of the present application, the original model parameters corresponding to the model are quantized using multiple quantization bit modes to obtain quantization model parameters corresponding to each quantization bit mode, and the first output result of the model function network under the original model parameters and the second output result of the model function network under each quantization model parameter are used to select the target quantization bit mode from the multiple quantization bit modes; thereby, the accurate selection of the quantization bit mode of the model function network is achieved, the accuracy of the quantization processing is high, and the high accuracy of the quantization processing helps to improve the generalization ability of the model beyond the training data, and can better adapt to new data distributions and scenarios. At the same time, it also enhances the stability and robustness of the model, and can maintain good performance under complex conditions. In addition, it can perform reasoning faster, reduce unnecessary resource consumption, and improve resource utilization, so the model performance is greatly improved, so that tasks in various application scenarios can be well executed through the model.

[0135] In one embodiment of the present application, another data processing method based on quantization is provided. Figure 4 As shown, the quantization-based data processing method may further include S401 to S403 after S203.

[0136] S401 to S403 are described in detail as follows:

[0137] S401, generating a target quantization scaling factor based on a target quantization bit mode.

[0138] It can be understood that the quantization process using the quantization bit mode in the functional network involves the use of a quantization scaling factor, wherein the quantization scaling factor is mainly used to map data of a higher precision type (such as int8) to a representation range corresponding to a lower precision type (such as int2); therefore, in an optional embodiment, after obtaining the target quantization bit mode, the target quantization bit mode can be used to generate a target quantization scaling factor corresponding to the model functional network.

[0139] In one embodiment of the present application, the model function network includes multiple model function networks; for example, as shown in Table 1, the multiple model function networks are E1, E2, and E3.

[0140] Accordingly, the process of generating the target quantization scaling factor based on the target quantization bit mode in S401 may include at least two cases:

[0141] Case 1: If there is a first model function network among multiple model function networks whose quantization accuracy of the target quantization bit mode is greater than a preset accuracy threshold, a target quantization scaling factor corresponding to each data channel in the first model function network is generated.

[0142] Among them, in the optional embodiment, the first model function network refers to a model function network in which the quantization accuracy of the target quantization bit mode is greater than a preset accuracy threshold among multiple model function networks.

[0143] For example, following the above example, assuming that the preset accuracy threshold is int4, referring to Table 1, it can be seen that the quantization accuracy int8 of the model function network E3 is greater than the preset accuracy threshold int4. At this time, the model function network E3 is the first model function network.

[0144] Among them, in an optional embodiment, a first model function network is obtained, and then a target quantization scaling factor corresponding to each data channel in the first model function network can be generated.

[0145] For example, following the example in which the aforementioned model function network E3 is the first model function network, it is assumed that the model function network E3 includes 64 data channels. Accordingly, target quantization scaling factors corresponding to the 64 data channels are generated respectively.

[0146] Case 2: If there is a second model function network among multiple model function networks whose quantization accuracy of the target quantization bit mode is less than or equal to a preset accuracy threshold, the channel values ​​of each data channel in the second model function network are grouped, and a target quantization scaling factor corresponding to each combination is generated.

[0147] Among them, in the optional embodiment, the second model function network refers to a model function network in which the quantization accuracy of the target quantization bit mode is less than or equal to a preset accuracy threshold among multiple model function networks.

[0148] For example, following the previous example, assuming that the preset accuracy threshold is int4, referring to Table 1, it can be seen that the quantization accuracy int2 of the model function network E1 is less than the preset accuracy threshold int4, and the quantization accuracy int4 of the model function network E2 is equal to the preset accuracy threshold int4. At this time, the model function networks E1 and E2 are both the second model function networks.

[0149] Among them, in an optional embodiment, a second model function network is obtained, and then the channel values ​​of each data channel in the second model function network can be grouped, and a target quantization scaling factor corresponding to each combination is generated.

[0150] For example, the aforementioned model function networks E1 and E2 are taken as an example of the second model function network.

[0151] For the model functional network E2, assuming that the model functional network E2 includes 64 data channels, and the 64 data channels have 128 channel values ​​respectively, each 64 channel values ​​can be used as a combination, that is, the channel value of each data channel is divided into 2 combinations, then there are a total of 128 combinations, and accordingly, the target quantization scaling factors corresponding to the 128 combinations are generated.

[0152] For the model functional network E1, assuming that the model functional network E1 includes 128 data channels, and the 128 data channels have 128 channel values ​​respectively, each 64 channel values ​​can be used as a combination, that is, the channel value of each data channel is divided into 2 combinations, then there are a total of 256 combinations, and accordingly, the target quantization scaling factors corresponding to the 256 combinations are generated.

[0153] It is understandable that the number of data channels included in different model function networks can be the same or different, and the channel values ​​included in each combination can be flexibly adjusted according to the specific application scenario when grouping.

[0154] In this way, by implementing the optional embodiment, when the quantization accuracy of the target quantization bit mode corresponding to the model function network is high, the target quantization scaling factor is calculated based on the coarser-grained data channel, thereby saving computing resources and time, etc.; when the quantization accuracy of the target quantization bit mode corresponding to the model function network is low, the target quantization scaling factor is calculated based on the finer-grained combination, thereby improving the quantization accuracy as much as possible and ensuring the accuracy of the quantization processing.

[0155] For easier understanding, see Figure 5 , for the first model function network whose quantization accuracy of the target quantization bit mode is greater than the preset accuracy threshold, generate the target quantization scaling factor corresponding to each data channel in the first model function network (i.e., channel-by-channel quantization); for the second model function network whose quantization accuracy of the target quantization bit mode is less than or equal to the preset accuracy threshold, group the channel values ​​of each data channel in the second model function network, and generate the target quantization scaling factor corresponding to each combination (i.e., grouped quantization).

[0156] In one embodiment of the present application, the process of generating a target quantization scaling factor corresponding to each data channel in the first model function network may include:

[0157] Obtaining a target quantization bit mode corresponding to the first model functional network;

[0158] For each data channel in the first model functional network, operations are performed based on the acquired target quantization bit mode, the activation input value corresponding to the data channel, and each quantization scaling factor to obtain an operation result corresponding to each quantization scaling factor, and the quantization scaling factor corresponding to the operation result with the largest absolute value among the multiple operation results is used as the target quantization scaling factor of the data channel.

[0159] That is, in the optional embodiment, for each data channel in the first model function network, the target quantization bit mode corresponding to the first model function network, the activation input value corresponding to the data channel, and each quantization scaling factor are used to perform operations, so as to obtain the operation result corresponding to each quantization scaling factor. At this time, the quantization scaling factor corresponding to the operation result with the largest absolute value among the multiple operation results is the target quantization scaling factor of the data channel. It can be understood that the larger the absolute value of the operation result, the more significant changes in the data channel can be reflected, thereby better focusing on the sensitivity of the model function network.

[0160] In this way, by implementing the optional embodiment, the target quantization scaling factor of each data channel in the first model function network can be obtained simply and accurately.

[0161] In one embodiment of the present application, a process of performing calculations based on the acquired target quantization bit mode, the activation input value corresponding to the data channel, and each quantization scaling factor to obtain a calculation result corresponding to each quantization scaling factor may include:

[0162] Traversing the first quantization scaling factor in the first scaling factor list, and performing calculations based on the acquired target quantization bit mode, the activation input value corresponding to the data channel, and the traversed quantization scaling factor to obtain a calculation result corresponding to the traversed quantization scaling factor;

[0163] The activation input value corresponding to the data channel is updated, and the next adjacent quantization scaling factor of the first quantization scaling factor is traversed, and operations are performed based on the acquired target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor to obtain the operation result corresponding to the traversed quantization scaling factor, until the first scaling factor list is traversed and the operation result corresponding to each quantization scaling factor in the first scaling factor list is obtained.

[0164] That is, in an optional embodiment, the operation result corresponding to each quantization scaling factor is obtained by traversal calculation, specifically: first, the first quantization scaling factor in the first scaling factor list is traversed, and the target quantization bit mode corresponding to the first model function network, the activation input value corresponding to the data channel, and the first quantization scaling factor are used to perform operations to obtain the operation result corresponding to the first quantization scaling factor; then, the activation input value of the data channel is updated, and the next adjacent quantization scaling factor of the first quantization scaling factor (that is, the second quantization scaling factor in the first scaling factor list) is traversed, and the target quantization bit mode corresponding to the first model function network, the updated activation input value, and the second quantization scaling factor are used to perform operations to obtain the operation result corresponding to the second quantization scaling factor, and this cycle is repeated until the first scaling factor list is traversed to obtain the operation result corresponding to each quantization scaling factor in the first scaling factor list.

[0165] Among them, in an optional embodiment, the activation input value of the data channel is updated during each iteration, and the absolute maximum value of the activation input value of the data channel sampled during the current iteration and the activation input value of the data channel sampled during the previous iteration is calculated. If the absolute value of the activation input value of the data channel sampled during the current iteration is less than the absolute value of the activation input value of the data channel sampled during the previous iteration, the activation input value of the data channel sampled during the current iteration is updated to the activation input value of the data channel sampled during the previous iteration. If the absolute value of the activation input value of the data channel sampled during the current iteration is greater than or equal to the absolute value of the activation input value of the data channel sampled during the previous iteration, the activation input value of the data channel sampled during the current iteration is maintained.

[0166] In this way, by implementing the optional embodiment, the activation input value of the data channel is updated during the iteration process, thereby improving the accuracy of the calculation result corresponding to the quantization scaling factor.

[0167] In one embodiment of the present application, the process of generating the target quantization scaling factor corresponding to each combination may include:

[0168] Obtaining a target quantization bit mode corresponding to the second model function network;

[0169] For each combination corresponding to each data channel in the second model functional network, operations are performed based on the acquired target quantization bit mode, the activation input value corresponding to the combination, and each quantization scaling factor to obtain the operation result corresponding to each quantization scaling factor, and the quantization scaling factor corresponding to the operation result with the largest absolute value among the multiple operation results is used as the target quantization scaling factor of the combination.

[0170] That is, in the optional embodiment, for the combination corresponding to each data channel in the first model function network, the target quantization bit mode corresponding to the second model function network, the activation input value corresponding to the combination, and each quantization scaling factor are used for calculation, so as to obtain the calculation result corresponding to each quantization scaling factor. At this time, the quantization scaling factor corresponding to the calculation result with the largest absolute value among the multiple calculation results is the target quantization scaling factor of the combination. It can be understood that the larger the absolute value of the calculation result, the more significant changes in the combination can be reflected, so as to better pay attention to the sensitivity of the model function network.

[0171] In this way, by implementing the optional embodiment, the target quantization scaling factor for each combination of each data channel in the second model function network can be obtained simply and accurately.

[0172] In one embodiment of the present application, a process of performing operations based on the acquired target quantization bit mode, the activation input value corresponding to the combination, and each quantization scaling factor to obtain the operation result corresponding to each quantization scaling factor may include:

[0173] Traversing the first quantization scaling factor in the second scaling factor list, and performing calculations based on the acquired target quantization bit mode, the activation input value corresponding to the combination, and the traversed quantization scaling factor to obtain a calculation result corresponding to the traversed quantization scaling factor;

[0174] The activation input value corresponding to the combination is updated, and the next adjacent quantization scaling factor of the first quantization scaling factor is traversed, and operations are performed based on the acquired target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor to obtain the operation result corresponding to the traversed quantization scaling factor, until the second scaling factor list is traversed to obtain the operation result corresponding to each quantization scaling factor in the second scaling factor list.

[0175] That is, in an optional embodiment, the operation result corresponding to each quantization scaling factor is obtained by traversal calculation, specifically: first, the first quantization scaling factor in the second scaling factor list is traversed, and the target quantization bit mode corresponding to the second model function network, the corresponding activation input value of the combination, and the first quantization scaling factor are used to perform operations to obtain the operation result corresponding to the first quantization scaling factor; then, the activation input value of the combination is updated, and the next adjacent quantization scaling factor of the first quantization scaling factor (that is, the second quantization scaling factor in the second scaling factor list) is traversed, and the target quantization bit mode corresponding to the second model function network, the updated activation input value, and the second quantization scaling factor are used to perform operations to obtain the operation result corresponding to the second quantization scaling factor, and this cycle is repeated until the second scaling factor list is traversed to obtain the operation result corresponding to each quantization scaling factor in the second scaling factor list.

[0176] Among them, in an optional embodiment, the activation input value of the combination is updated during each iteration, and the absolute maximum value of the activation input value of the combination sampled during the current iteration and the activation input value of the combination sampled during the previous iteration is calculated. If the absolute value of the activation input value of the combination sampled during the current iteration is less than the absolute value of the activation input value of the combination sampled during the previous iteration, the activation input value of the combination sampled during the current iteration is updated to the activation input value of the combination sampled during the previous iteration. If the absolute value of the activation input value of the combination sampled during the current iteration is greater than or equal to the absolute value of the activation input value of the combination sampled during the previous iteration, the activation input value of the combination sampled during the current iteration is maintained.

[0177] In this way, by implementing the optional embodiment, the combined activation input value is updated during the iteration process, thereby improving the accuracy of the calculation result corresponding to the quantization scaling factor.

[0178] In one embodiment of the present application, the process of generating a target quantization scaling factor based on a target quantization bit mode in S401 may include:

[0179] A quantization scaling factor corresponding to a target quantization bit mode is obtained respectively by deploying a plurality of devices with a hybrid bit model;

[0180] A maximum quantization scaling factor is selected from multiple quantization scaling factors as a target quantization scaling factor.

[0181] That is, in an optional embodiment, taking into account the situation of limited device resources, multiple devices with mixed bit models are deployed to obtain the quantization scaling factors corresponding to the target quantization bit mode, and the largest quantization scaling factor is selected from the multiple quantization scaling factors as the target quantization scaling factor.

[0182] For example, a mixed bit model is deployed on three devices respectively. Taking the model function network E1 as an example, the three devices will obtain the quantization scaling factor corresponding to the model function network E1 (that is, three quantization scaling factors), and then select the largest quantization scaling factor from the three quantization scaling factors as the target quantization scaling factor corresponding to the model function network E1.

[0183] In this way, by implementing the optional embodiment, the target quantization scaling factor of the same model function network is obtained by deploying each device, which ensures that each device can perform tasks normally and improves the accuracy of the target quantization scaling factor.

[0184] S402, associatively storing the target quantization bit mode, the target quantization scaling factor, and identification information corresponding to the model function network in a designated area.

[0185] In the embodiment of the present application, a target quantization bit mode is used to generate a target quantization scaling factor, and then the target quantization bit mode, the target quantization scaling factor, and the identification information corresponding to the model function network (used to uniquely identify the model function network, including but not limited to the name, number, etc.) can be associated and stored in a designated area.

[0186] For example, continuing with the above example, please refer to Table 2, which is an example table of associative storage.

[0187]

[0188] S403: If it is detected that the model function network is in the inference stage, a target quantization bit mode and a target quantization scaling factor are obtained from a specified area to quantize the original model parameters.

[0189] In the embodiment of the present application, when it is detected that the model function network is in the inference stage, the target quantization bit mode and target quantization scaling factor corresponding to the model function network can be obtained from the designated storage area, and the original model parameters corresponding to the model function network can be quantized.

[0190] It should be noted that Figure 4 For detailed description of S201 to S203, please refer to Figure 2 S201 to S203 shown are not described in detail here.

[0191] In the embodiment of the present application, the target quantization bit mode, the target quantization scaling factor, and the identification information corresponding to the model function network are associated and stored, and can be directly used during model inference, which improves the efficiency of the quantization processing and the reliability of the quantization processing, thereby further improving the model performance.

[0192] The specific scenarios of the embodiments of the present application are described in detail below:

[0193] See also Figure 6 , Figure 6 FIG. 1 is a flow chart of a data processing method based on quantization according to an embodiment of the present application. Figure 6 As shown, the quantization-based data processing method at least includes S601 to S607, which are described in detail as follows:

[0194] S601, obtaining a verification set, and inputting the verification set into the hybrid expert model.

[0195] In the embodiments of the present application, the hybrid expert model includes but is not limited to being used for text processing, video processing, speech processing, etc.

[0196] S602, check whether the verification rounds have reached a preset number threshold, if not, execute S603, if yes, execute S604.

[0197] S603, record the activation input value corresponding to each model functional network, and return to S602.

[0198] S604: Perform sensitivity analysis on each model function network to generate a target quantization bit mode and a target quantization scaling factor for each model function network.

[0199] For example, the target quantization bit method for generating each model function network can be referred to Figure 7 ,like Figure 7 As shown, the hybrid expert model includes four expert networks (i.e., functional networks), namely Expert1, Expert2, Expert3, and Expert4, wherein a sensitivity analysis is performed on Expert1 to obtain the target quantization bit mode INT8, a sensitivity analysis is performed on Expert2 to obtain the target quantization bit mode INT4, a sensitivity analysis is performed on Expert3 to obtain the target quantization bit mode INT4, and a sensitivity analysis is performed on Expert4 to obtain the target quantization bit mode INT2.

[0200] Exemplarily, generating a target quantization scaling factor for each model function network may include: if there is a first model function network in the plurality of model function networks whose quantization accuracy of the target quantization bit mode is greater than a preset accuracy threshold, then generating a target quantization scaling factor corresponding to each data channel in the first model function network; if there is a second model function network in the plurality of model function networks whose quantization accuracy of the target quantization bit mode is less than or equal to the preset accuracy threshold, then grouping the channel values ​​of each data channel in the second model function network, and generating a target quantization scaling factor corresponding to each combination. That is, when the quantization accuracy of the target quantization bit mode corresponding to the model function network is high, the target quantization scaling factor is calculated based on a coarser-grained data channel, and when the quantization accuracy of the target quantization bit mode corresponding to the model function network is low, the target quantization scaling factor is calculated based on a finer-grained combination.

[0201] S605 , associatively storing the target quantization bit mode and the target quantization scaling factor corresponding to each model function network and the identification information of each model function network in a designated area.

[0202] In an embodiment of the present application, the target quantization bit mode and target quantization scaling factor corresponding to each model function network, as well as the identification information of each model function network can be stored in a designated area locally or in the cloud in vector form through a storage interface such as torch.save.

[0203] S606: If it is detected that the hybrid expert model is in the inference stage, a target quantization bit mode and a target quantization scaling factor corresponding to each model functional network are obtained from the specified area.

[0204] S607, using the target quantization bit mode and the target quantization scaling factor corresponding to each model function network, quantize the original weight value of each model function network.

[0205] In the embodiment of the present application, the mapping relationship table between the preset bit mode and the computing kernel can be searched to obtain the target computing kernel corresponding to each model functional network, and then the corresponding target quantization scaling factor is passed in the calculation process of each target computing kernel to quantize the original weight value to the corresponding target quantization bit number, thereby realizing the hybrid bit quantization reasoning of the hybrid expert model and obtaining the corresponding task processing result.

[0206] Optionally, when the hybrid expert model is applied to text processing application scenarios, it includes but is not limited to:

[0207] (1) Text classification (e.g., e-mail classification, news classification, etc.). In this case, the hybrid expert model can be a text classification model. Specifically, the text to be classified is input into the text classification model. The target quantization bit mode and target quantization scaling factor corresponding to each functional network in the text classification model are obtained by the aforementioned method in the text classification model. The target quantization bit mode and target quantization scaling factor corresponding to each functional network are used for quantization processing. The quantized data are then used for text classification, and the text classification information is output. The performance of the text classification model is good, thereby improving the accuracy and efficiency of text classification.

[0208] (2) Situational understanding and automatic response (such as automatic customer service, chatbots, etc.). In this case, the hybrid expert model can be a response model. Specifically, the text to be responded to is input into the response model, and the above-mentioned method is used in the response model to obtain the target quantization bit mode and target quantization scaling factor corresponding to each functional network in the response model, and the target quantization bit mode and target quantization scaling factor corresponding to each functional network are used for quantization processing, and then the quantized data is used to understand the user's intention and output the response information. The performance of the response model is good, thereby improving the accuracy and efficiency of situational understanding and automatic response.

[0209] (3) Multilingual text processing (such as text translation, language recognition, etc.). In this case, the hybrid expert model can be a multilingual processing model. Specifically, the text to be processed is input into the multilingual processing model, and the target quantization bit mode and target quantization scaling factor corresponding to each functional network in the multilingual processing model are obtained by the aforementioned method in the multilingual processing model. The target quantization bit mode and target quantization scaling factor corresponding to each functional network are used for quantization processing, and then the quantized data are used for text parsing, and text parsing information is output. The performance of the multilingual processing model is good, thereby improving the accuracy and efficiency of multilingual text processing.

[0210] Optionally, when the hybrid expert model is applied to a video processing application scenario, it includes but is not limited to:

[0211] (1) Video detection (for example, identifying different actions, scenes, objects, etc. in a video). In this case, the hybrid expert model can be a video detection model. Specifically, the video to be detected is input into the video detection model, and the target quantization bit mode and target quantization scaling factor corresponding to each functional network in the video detection model are obtained by the aforementioned method in the video detection model. The target quantization bit mode and target quantization scaling factor corresponding to each functional network are used for quantization processing, and then the quantized data are used for video detection, and video detection information is output. The performance of the video detection model is good, thereby improving the accuracy and efficiency of video detection.

[0212] (2) Video compression and transmission (such as online video playback, video conferencing, etc.). In this case, the hybrid expert model can be a video transmission model. Specifically, the video to be transmitted is input into the video transmission model, and the above-mentioned method is used in the video transmission model to obtain the target quantization bit mode and target quantization scaling factor corresponding to each functional network in the video transmission model, and the target quantization bit mode and target quantization scaling factor corresponding to each functional network are used for quantization processing, and then the quantized data is used to perform video compression to achieve efficient video transmission and playback experience. The performance of the video transmission model is good, thereby improving the accuracy and efficiency of video compression and transmission.

[0213] (3) Video special effects and enhancements (such as film and television production, virtual reality, augmented reality, etc.). In this case, the hybrid expert model can be a video processing model. Specifically, the video to be processed is input into the video processing model, and the above-mentioned method is used in the video processing model to obtain the target quantization bit mode and target quantization scaling factor corresponding to each functional network in the video processing model, and the target quantization bit mode and target quantization scaling factor corresponding to each functional network are used for quantization processing, and then the quantized data is used for special effects and enhancement processing to achieve efficient video rendering. The performance of the video processing model is good, thereby improving the accuracy and efficiency of video special effects and enhancement.

[0214] Optionally, when the hybrid expert model is applied to speech processing application scenarios, it includes but is not limited to:

[0215] (1) Speech recognition and transcription (such as voice assistant, voice search, subtitle generation, etc.). In this case, the hybrid expert model can be a speech recognition model. Specifically, the speech to be recognized is input into the speech recognition model, and the target quantization bit mode and target quantization scaling factor corresponding to each functional network in the speech recognition model are obtained in the speech recognition model using the above-mentioned method, and the target quantization bit mode and target quantization scaling factor corresponding to each functional network are used for quantization processing, and then the quantized data are used for speech recognition and transcription, and speech recognition information is output. The performance of the speech recognition model is good, thereby improving the accuracy and efficiency of speech recognition and transcription.

[0216] (2) Voice activity detection and keyword recognition (such as wake-up word detection, voice control, etc.). In this case, the hybrid expert model can be a voice detection model. Specifically, the voice to be detected is input into the voice detection model, and the target quantization bit mode and target quantization scaling factor corresponding to each functional network in the voice detection model are obtained in the voice detection model using the aforementioned method. The target quantization bit mode and target quantization scaling factor corresponding to each functional network are used for quantization processing, and then the quantized data are used for voice activity detection and keyword recognition, and voice detection information is output. The performance of the voice detection model is good, thereby improving the accuracy and efficiency of voice activity detection and keyword recognition.

[0217] (3) Speech quality enhancement and noise reduction (such as telephone conferencing, voice calls, recording applications, etc.). In this case, the hybrid expert model can be a speech enhancement model. Specifically, the speech to be enhanced is input into the speech enhancement model, and the target quantization bit mode and target quantization scaling factor corresponding to each functional network in the speech enhancement model are obtained in the speech enhancement model using the aforementioned method. The target quantization bit mode and target quantization scaling factor corresponding to each functional network are used for quantization processing, and then the quantized data are used to perform speech quality enhancement and noise reduction, and the speech enhancement information is output. The performance of the speech enhancement model is good, thereby improving the accuracy and efficiency of speech quality enhancement and noise reduction.

[0218] It should be noted that, for the determination of the target quantization bit mode and the target quantization scaling factor corresponding to each functional network in the corresponding model involved in the aforementioned application scenario, and the detailed introduction of the quantization processing using the target quantization bit mode and the target quantization scaling factor corresponding to each functional network, please refer to the aforementioned embodiment, which will not be repeated here.

[0219] Among them, the present application scheme uses models of different sizes, 7B-MoE and 70B-MoE, to be tested on the ptb-en verification set, and the test results are shown in Table 3.

[0220]

[0221] As shown in Table 3, the values ​​obtained are all perplexity (PPL) values, where the smaller the PPL value, the smaller the loss. For 7B-MoE, the PPL values ​​are arranged from small to large as 12.209<12.323<12.350, and for 70B-MoE, the PPL values ​​are arranged from small to large as 8.858<8.903<8.925, where the mixed bit quantization and bf16 and int8 are not much different. It can be seen that the quantization accuracy of the present application scheme is close to lossless, and the model performance is good.

[0222] Figure 8 FIG. 1 is a block diagram of a data processing device based on quantization according to an embodiment of the present application. Figure 8 As shown, the device comprises:

[0223] The quantization processing module 801 is configured to quantize the original model parameters corresponding to the model function network respectively through multiple quantization bit modes to obtain the quantized model parameters corresponding to each quantization bit mode;

[0224] An acquisition module 802 is configured to acquire a first output result of the model function network under the original model parameters, and acquire a second output result of the model function network under each quantized model parameter;

[0225] The selection module 803 is configured to select a target quantization bit mode from the multiple quantization bit modes based on the first output result and multiple second output results, and the target quantization bit mode is used to quantize the original model parameters in the inference stage corresponding to the model function network.

[0226] In one embodiment of the present application, based on the above solution, the module 803 is selected, and the specific configuration is as follows:

[0227] Calculating a difference value between the first output result and each second output result;

[0228] A quantization bit mode corresponding to the smallest difference value is selected from the multiple quantization bit modes, and the selected quantization bit mode is used as the target quantization bit mode.

[0229] In one embodiment of the present application, based on the above solution, the selection module 803 is further specifically configured as follows:

[0230] For each second output result, a difference operation is performed on the first output result and the second output result to obtain a difference value, and the obtained difference value is raised to a specified power to obtain a difference value between the first output result and the second output result.

[0231] In one embodiment of the present application, based on the above solution, the selection module 803 is further specifically configured as follows:

[0232] For each second output result, a signal power value is calculated based on the second output result, a noise power value is calculated based on the first output result and the second output result, a signal-to-noise ratio is calculated based on the signal power value and the noise power value, and a difference value between the first output result and the second output result is calculated based on the signal-to-noise ratio.

[0233] In one embodiment of the present application, based on the above solution, the original model parameters include original weight values, and the quantized model parameters include quantized weight values; the acquisition module 802 is specifically configured as follows:

[0234] Obtaining activation input values ​​of the model functional network;

[0235] The activation input value and the original weight value are multiplied to obtain a first output result of the model function network, and the activation input value and each quantized weight value are multiplied to obtain multiple second output results of the model function network.

[0236] In one embodiment of the present application, based on the above solution, the model function network corresponds to multiple data channels, and each data channel corresponds to a channel value; the acquisition module 802 is further specifically configured as follows:

[0237] If an outlier channel value is detected, the data channel corresponding to the largest channel value in the outlier is used as the target data channel, and the channel value corresponding to the target data channel is used as the activation input value of the model function network, and the original weight value and the quantized weight value corresponding to the target data channel are obtained;

[0238] The channel value corresponding to the target data channel and the original weight value are multiplied to obtain a first output result of the model function network, and the channel value corresponding to the target data channel and each quantized weight value are multiplied to obtain multiple second output results of the model function network.

[0239] In one embodiment of the present application, based on the above solution, the acquisition module 802 is further specifically configured as follows:

[0240] Calculating the similarity between the target data channel and an adjacent data channel corresponding to the target data channel;

[0241] Based on the similarity, outlier channel values ​​in the target data channel are reduced to obtain a target channel value;

[0242] The target channel value and the channel values ​​in the target data channel except the outlier channel value are used as activation input values ​​of the model function network.

[0243] In one embodiment of the present application, based on the above solution, the acquisition module 802 is further specifically configured as follows:

[0244] Performing a distance operation based on the channel value of the target data channel and the channel value of the adjacent data channel corresponding to the target data channel to obtain a distance value, wherein the distance value is greater than 0 and less than 1, and using the distance value as the similarity between the target data channel and the adjacent data channel;

[0245] A product operation is performed on the distance value and the outlier channel value in the target data channel to obtain a target channel value.

[0246] In one embodiment of the present application, based on the above solution, the device further includes:

[0247] A generating module configured to generate a target quantization scaling factor based on the target quantization bit mode;

[0248] A storage module configured to associate and store the target quantization bit mode, the target quantization scaling factor, and identification information corresponding to the model function network in a designated area;

[0249] The inference module is configured to obtain the target quantization bit mode and the target quantization scaling factor from the specified area to quantize the original model parameters if it is detected that the model function network is in the inference stage.

[0250] In one embodiment of the present application, based on the above scheme, the model function network includes multiple generation modules, which are specifically configured as follows:

[0251] If there is a first model function network among the multiple model function networks whose quantization accuracy of the target quantization bit mode is greater than a preset accuracy threshold, generating a target quantization scaling factor corresponding to each data channel in the first model function network;

[0252] If there is a second model function network among the multiple model function networks whose quantization accuracy of the target quantization bit mode is less than or equal to the preset accuracy threshold, the channel values ​​of each data channel in the second model function network are grouped, and a target quantization scaling factor corresponding to each combination is generated.

[0253] In one embodiment of the present application, based on the above solution, the generation module is further specifically configured as follows:

[0254] Obtaining a target quantization bit mode corresponding to the first model function network;

[0255] For each data channel in the first model functional network, operations are performed based on the acquired target quantization bit mode, the activation input value corresponding to the data channel, and each quantization scaling factor to obtain the operation result corresponding to each quantization scaling factor, and the quantization scaling factor corresponding to the operation result with the largest absolute value among the multiple operation results is used as the target quantization scaling factor of the data channel.

[0256] In one embodiment of the present application, based on the above solution, the generation module is further specifically configured as follows:

[0257] Traversing the first quantization scaling factor in the first scaling factor list, and performing calculations based on the acquired target quantization bit mode, the activation input value corresponding to the data channel, and the traversed quantization scaling factor to obtain a calculation result corresponding to the traversed quantization scaling factor;

[0258] The activation input value corresponding to the data channel is updated, and the next adjacent quantization scaling factor of the first quantization scaling factor is traversed, and operations are performed based on the acquired target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor to obtain the operation result corresponding to the traversed quantization scaling factor, until the first scaling factor list is traversed to obtain the operation result corresponding to each quantization scaling factor in the first scaling factor list.

[0259] In one embodiment of the present application, based on the above solution, the generation module is further specifically configured as follows:

[0260] Obtaining a target quantization bit mode corresponding to the second model function network;

[0261] For each combination corresponding to each data channel in the second model function network, operations are performed based on the acquired target quantization bit mode, the activation input value corresponding to the combination, and each quantization scaling factor to obtain the operation result corresponding to each quantization scaling factor, and the quantization scaling factor corresponding to the operation result with the largest absolute value among the multiple operation results is used as the target quantization scaling factor of the combination.

[0262] In one embodiment of the present application, based on the above solution, the generation module is further specifically configured as follows:

[0263] Traversing the first quantization scaling factor in the second scaling factor list, and performing calculations based on the acquired target quantization bit mode, the activation input value corresponding to the combination, and the traversed quantization scaling factor to obtain a calculation result corresponding to the traversed quantization scaling factor;

[0264] The activation input value corresponding to the combination is updated, and the next adjacent quantization scaling factor of the first quantization scaling factor is traversed, and operations are performed based on the acquired target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor to obtain the operation result corresponding to the traversed quantization scaling factor, until the second scaling factor list is traversed to obtain the operation result corresponding to each quantization scaling factor in the second scaling factor list.

[0265] In one embodiment of the present application, based on the above solution, the generation module is further specifically configured as follows:

[0266] Obtaining quantization scaling factors corresponding to the target quantization bit mode respectively through a plurality of devices deployed with a hybrid bit model;

[0267] A maximum quantization scaling factor is selected from multiple quantization scaling factors as a target quantization scaling factor.

[0268] It should be noted that the apparatus provided in the aforementioned embodiment and the method provided in the aforementioned embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment.

[0269] An embodiment of the present application also provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the aforementioned quantization-based data processing method.

[0270] Fig. 9 It is an electronic device suitable for implementing the embodiments of the present application (for example Figure 1 A schematic diagram of the structure of a computer system of a terminal device or server as shown.

[0271] It should be noted that Fig. 9 The computer system 900 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0272] like Fig. 9As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage part 908 to a random access memory (RAM) 903, such as executing the method in the above embodiment. In RAM 903, various computer programs and data required for system operation are also stored. CPU 901, ROM 902 and RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0273] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed so that a computer program read therefrom is installed into the storage section 908 as needed.

[0274] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes computer instructions for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 909, and / or installed from a removable medium 911. When the computer program is executed by a central processing unit (CPU) 901, various functions defined in the system of the present application are executed.

[0275] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable media can be, for example, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium can be any tangible medium containing or storing a computer program, which can be used by an instruction execution system, a device or a device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit computer programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0276] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0277] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0278] Another aspect of the present application further provides a computer-readable medium having a computer program stored thereon, which implements the above-mentioned quantization-based data processing method when executed by a processor. The computer-readable medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0279] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable medium. A processor of an electronic device reads the computer instruction from the computer-readable medium, and the processor executes the computer instruction, so that the electronic device executes the quantization-based data processing method provided in each of the above embodiments.

[0280] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A data processing method based on quantization, characterized in that: include: Quantizing the original model parameters corresponding to the model function network respectively through multiple quantization bit modes to obtain quantized model parameters corresponding to each quantization bit mode, wherein the quantization bit mode refers to the bit type required for quantization processing and the number of bits corresponding to the bit type; Obtaining a first output result of the model function network under the original model parameters, and obtaining a second output result of the model function network under each quantized model parameter; Based on the first output result and multiple second output results, a target quantization bit mode is selected from the multiple quantization bit modes, and the target quantization bit mode is used to quantize the original model parameters in the inference stage of the model function network.

2. The method according to claim 1, characterized in that: The selecting a target quantization bit mode from the multiple quantization bit modes based on the first output result and the multiple second output results includes: Calculating a difference value between the first output result and each second output result; A quantization bit mode corresponding to the smallest difference value is selected from the multiple quantization bit modes, and the selected quantization bit mode is used as the target quantization bit mode.

3. The method according to claim 2, characterized in that The calculating the difference value between the first output result and each second output result includes: For each second output result, a difference operation is performed on the first output result and the second output result to obtain a difference value, and the obtained difference value is raised to a specified power to obtain a difference value between the first output result and the second output result.

4. The method according to claim 2, characterized in that: The calculating the difference value between the first output result and each second output result includes: For each second output result, a signal power value is calculated based on the second output result, a noise power value is calculated based on the first output result and the second output result, a signal-to-noise ratio is calculated based on the signal power value and the noise power value, and a difference value between the first output result and the second output result is calculated based on the signal-to-noise ratio.

5. The method according to claim 1, characterized in that The original model parameters include original weight values, and the quantized model parameters include quantized weight values; obtaining a first output result of the model function network under the original model parameters, and obtaining a second output result of the model function network under each quantized model parameter, comprises: Obtaining activation input values ​​of the model functional network; The activation input value and the original weight value are multiplied to obtain a first output result of the model function network, and the activation input value and each quantized weight value are multiplied to obtain multiple second output results of the model function network.

6. The method according to claim 5, characterized in that The model function network corresponds to a plurality of data channels, each of which corresponds to a channel value; and obtaining the activation input value of the model function network includes: If an outlier channel value is detected, the data channel corresponding to the largest channel value in the outlier is used as the target data channel, and the channel value corresponding to the target data channel is used as the activation input value of the model function network, and the original weight value and the quantized weight value corresponding to the target data channel are obtained; The step of performing a product operation on the activation input value and the original weight value to obtain a first output result of the model function network, and performing a product operation on the activation input value and each quantized weight value to obtain multiple second output results of the model function network includes: The channel value corresponding to the target data channel and the original weight value are multiplied to obtain a first output result of the model function network, and the channel value corresponding to the target data channel and each quantized weight value are multiplied to obtain multiple second output results of the model function network.

7. The method according to claim 6, characterized in that The step of using the channel value corresponding to the target data channel as the activation input value of the model function network includes: Calculating the similarity between the target data channel and an adjacent data channel corresponding to the target data channel; Based on the similarity, outlier channel values ​​in the target data channel are reduced to obtain a target channel value; The target channel value and the channel values ​​in the target data channel except the outlier channel value are used as activation input values ​​of the model function network.

8. The method according to claim 7, characterized in that The calculating the similarity between the target data channel and an adjacent data channel corresponding to the target data channel includes: Performing a distance operation based on the channel value of the target data channel and the channel value of the adjacent data channel corresponding to the target data channel to obtain a distance value, the distance value being greater than 0 and less than 1, and using the distance value as the similarity between the target data channel and the adjacent data channel; The step of performing a reduction process on the outlier channel value in the target data channel based on the similarity to obtain a target channel value includes: A product operation is performed on the distance value and the outlier channel value in the target data channel to obtain a target channel value.

9. The method according to any one of claims 1 to 8, characterized in that After selecting a target quantization bit mode from the multiple quantization bit modes based on the first output result and the multiple second output results, the method further includes: Generating a target quantization scaling factor based on the target quantization bit mode; The target quantization bit mode, the target quantization scaling factor, and identification information corresponding to the model function network are stored in a designated area in association with each other; If it is detected that the model function network is in the inference stage, the target quantization bit mode and the target quantization scaling factor are obtained from the specified area to quantize the original model parameters.

10. The method according to claim 9, characterized in that The model function network includes a plurality of networks; the generating a target quantization scaling factor based on the target quantization bit mode includes: If there is a first model function network among the multiple model function networks whose quantization accuracy of the target quantization bit mode is greater than a preset accuracy threshold, generating a target quantization scaling factor corresponding to each data channel in the first model function network; If there is a second model function network among the multiple model function networks whose quantization accuracy of the target quantization bit mode is less than or equal to the preset accuracy threshold, the channel values ​​of each data channel in the second model function network are grouped, and a target quantization scaling factor corresponding to each combination is generated.

11. The method according to claim 10, characterized in that The generating a target quantization scaling factor corresponding to each data channel in the first model function network includes: Obtaining a target quantization bit mode corresponding to the first model function network; For each data channel in the first model functional network, operations are performed based on the acquired target quantization bit mode, the activation input value corresponding to the data channel, and each quantization scaling factor to obtain the operation result corresponding to each quantization scaling factor, and the quantization scaling factor corresponding to the operation result with the largest absolute value among the multiple operation results is used as the target quantization scaling factor of the data channel.

12. The method according to claim 11, characterized in that The performing operation based on the acquired target quantization bit mode, the activation input value corresponding to the data channel, and each quantization scaling factor to obtain the operation result corresponding to each quantization scaling factor includes: Traversing the first quantization scaling factor in the first scaling factor list, and performing calculations based on the acquired target quantization bit mode, the activation input value corresponding to the data channel, and the traversed quantization scaling factor to obtain a calculation result corresponding to the traversed quantization scaling factor; The activation input value corresponding to the data channel is updated, and the next adjacent quantization scaling factor of the first quantization scaling factor is traversed, and operations are performed based on the acquired target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor to obtain the operation result corresponding to the traversed quantization scaling factor, until the first scaling factor list is traversed to obtain the operation result corresponding to each quantization scaling factor in the first scaling factor list.

13. The method according to claim 10, characterized in that The generating a target quantization scaling factor corresponding to each combination includes: Obtaining a target quantization bit mode corresponding to the second model function network; For each combination corresponding to each data channel in the second model function network, operations are performed based on the acquired target quantization bit mode, the activation input value corresponding to the combination, and each quantization scaling factor to obtain the operation result corresponding to each quantization scaling factor, and the quantization scaling factor corresponding to the operation result with the largest absolute value among the multiple operation results is used as the target quantization scaling factor of the combination.

14. The method according to claim 13, characterized in that The operation based on the acquired target quantization bit mode, the activation input value corresponding to the combination, and each quantization scaling factor is performed to obtain the operation result corresponding to each quantization scaling factor, including: Traversing the first quantization scaling factor in the second scaling factor list, and performing calculations based on the acquired target quantization bit mode, the activation input value corresponding to the combination, and the traversed quantization scaling factor to obtain a calculation result corresponding to the traversed quantization scaling factor; The activation input value corresponding to the combination is updated, and the next adjacent quantization scaling factor of the first quantization scaling factor is traversed, and operations are performed based on the acquired target quantization bit mode, the updated activation input value, and the traversed quantization scaling factor to obtain the operation result corresponding to the traversed quantization scaling factor, until the second scaling factor list is traversed to obtain the operation result corresponding to each quantization scaling factor in the second scaling factor list.

15. The method according to claim 9, characterized in that The generating a target quantization scaling factor based on the target quantization bit mode includes: Obtaining quantization scaling factors corresponding to the target quantization bit mode respectively through a plurality of devices deployed with a hybrid bit model; A maximum quantization scaling factor is selected from multiple quantization scaling factors as a target quantization scaling factor.

16. A data processing device based on quantization, characterized in that: include: A quantization processing module is configured to quantize the original model parameters corresponding to the model function network respectively through multiple quantization bit modes to obtain quantized model parameters corresponding to each quantization bit mode, wherein the quantization bit mode refers to the bit type required for quantization processing and the number of bits corresponding to the bit type; An acquisition module, configured to acquire a first output result of the model function network under the original model parameters, and to acquire a second output result of the model function network under each quantized model parameter; A selection module is configured to select a target quantization bit mode from the multiple quantization bit modes based on the first output result and multiple second output results, and the target quantization bit mode is used to quantize the original model parameters in the inference stage corresponding to the model function network.

17. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs, which, when executed by the electronic device, enables the electronic device to implement the quantization-based data processing method as described in any one of claims 1 to 15.

18. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the quantization-based data processing method according to any one of claims 1 to 15 is implemented.

19. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the quantization-based data processing method according to any one of claims 1 to 15 is implemented.