Neural network lightweight method, device, equipment, medium and product

By presetting the transfer difficulty coefficient, quantization cropping range and quantization bit width interval, a search space is built, and the optimal parameter combination of neural networks is determined for quantization, which solves the problem of reduced accuracy due to lightweight neural networks, and realizes efficient inference and resources Saving effects.

CN119089962APending Publication Date: 2024-12-06CHINA MOBILEHANGZHOUINFORMATION TECH CO LTD +1
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411094400.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing lightweight neural network methods will lead to reduced model accuracy, making it difficult to maintain efficient inference performance while reducing computing resource consumption.

Method used

By presetting the transfer difficulty coefficient, quantized cropping range and quantized bit width value interval, a search space is built, the optimal parameter combination of each network layer is determined, and the quantization is layer by layer based on these combinations to ensure that the quantized model has no accuracy. Reduce computing resource consumption under the premise of reduction.

Benefits of technology

It realizes the reduction of computing resource consumption, speed up processing speed, and improves the accuracy and performance of the model while maintaining model accuracy. It is suitable for different types of network layer structures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089962A_ABST
    Figure CN119089962A_ABST
Patent Text Reader

Abstract

The invention provides a neural network lightweight method, apparatus and device, a medium and a product. The method comprises the steps of determining a first preset value interval of a transfer difficulty coefficient, a second preset value interval of a quantization cutting range and a third preset value interval of a quantization bit width; constructing a search space based on the first preset value interval, the second preset value interval and the third preset value interval; in the search space, determining an optimal parameter combination of each network layer; and based on the optimal parameter combination of each network layer, carrying out layer-by-layer quantization on the to-be-quantized model to obtain a quantized model. According to the neural network lightweight method provided by the invention, the search space is constructed by comprehensively considering various factors, so that the lightweight method is more universal and suitable for different types of network structures, the precision of the quantized model can be ensured to be kept, and the optimal parameter combination of each network layer is further selected to realize model lightweight; the computing resource consumption can be reduced and the processing speed can be accelerated on the premise of keeping the precision.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a neural network lightweight method, device, equipment, medium and product. Background Art

[0002] With the development of artificial intelligence technology, it is widely used in the fields of machine learning, natural language processing, computer vision, control decision-making, etc. Processing data based on neural network models to realize application functions such as recognition is a key technology for artificial intelligence applications. In order to improve the accuracy of data processing by neural network models, the number of parameters of neural network models is increasing, which leads to higher and higher computing resources such as computing power and storage required for neural network models to process data, and correspondingly, the time it takes to process data is also increasing. Especially after the outbreak of large model technology, although large language models perform well in performing natural language processing and natural language generation tasks, their training and deployment are quite complex, and the number of parameters usually reaches tens of billions or even trillions. Therefore, using a lightweight model method to slim down the model is crucial to achieving efficient reasoning.

[0003] The current lightweight method generally selects a quantization bit width and unifies the quantization parameters according to the quantization bit width. However, this lightweight processing method usually leads to a decrease in the accuracy of the lightweight model. Summary of the invention

[0004] The present invention provides a neural network lightweight method, device, equipment, medium and product, which are used to solve the problem that the existing neural network lightweight method will cause the accuracy of the lightweight model to be reduced.

[0005] In a first aspect, the present invention provides a method for lightweighting a neural network, comprising: Determine a first preset value interval of a transfer difficulty coefficient, a second preset value interval of a quantization clipping range, and a third preset value interval of a quantization bit width; the transfer difficulty coefficient is used to adjust the coefficient of the quantization difficulty of multiple parameters in the network layer; the quantization clipping range refers to the range to which the parameter to be quantized belongs among the multiple parameters in the network layer; Constructing a search space based on the first preset value interval, the second preset value interval, and the third preset value interval; In the search space, determining an optimal parameter combination of each network layer in the model to be quantized; the optimal parameter combination includes a target transfer difficulty coefficient, a target quantization clipping range, and a target quantization bit width; Based on the optimal parameter combination of each network layer, the model to be quantized is quantized layer by layer to obtain a quantized model.

[0006] In one embodiment, in the search space, when determining the optimal parameter combination of each network layer in the model to be quantized, the following steps are performed for each network layer: Acquire a calibration data set of the model to be quantified; Performing model inference based on the calibration data set to obtain multiple initial activation values ​​of the network layer; Determine the initial weight corresponding to each initial activation value; Based on multiple initial activation values ​​and multiple initial weights, a target transfer difficulty coefficient of the network layer in the model to be quantized is selected within the first preset value interval.

[0007] In one embodiment, after selecting the target transfer difficulty coefficient of the network layer in the model to be quantized within the first preset value interval based on the multiple initial activation values ​​and the multiple initial weights, the method further includes: Determining a scaling factor based on the target transfer difficulty coefficient; the scaling factor is used to quantize the parameter to be quantized; Based on the scaling factor, each initial activation value and each initial weight are converted to obtain multiple target activation values ​​and multiple target weights.

[0008] In one embodiment, in the search space, when determining the optimal parameter combination of each network layer in the model to be quantized, the following steps are further performed for each network layer: selecting a plurality of outliers among the plurality of target activation values ​​and the plurality of target weights; Constructing a number axis based on a plurality of target parameters; the plurality of target parameters comprising the plurality of target activation values ​​and the plurality of target weights; Determine the remaining target parameters among the plurality of target parameters except the plurality of outliers as normal values; Based on the number axis, determining a first parameter distribution of a plurality of normal values ​​and a second parameter distribution of the plurality of outliers; Based on the first parameter distribution and the second parameter distribution, a target quantization clipping range of the network layer in the model to be quantized is selected within the second preset value interval.

[0009] In one embodiment, selecting a plurality of outliers from the plurality of target activation values ​​and the plurality of target weights comprises: determining the magnitude of each target activation value; Determine the target activation values ​​corresponding to the first N largest amplitudes among the multiple amplitudes as abnormal activation values; Determine the target weight corresponding to each abnormal activation value as the abnormal weight; Multiple abnormal activation values ​​and multiple abnormal weights are respectively determined as outliers.

[0010] In one embodiment, in the search space, when determining the optimal parameter combination of each network layer in the model to be quantized, the following steps are further performed for each network layer: Determine the target hardware for the deployment of the model to be quantized; According to the target hardware, a target quantization bit width of the network layer in the model to be quantized is selected within the third preset value interval.

[0011] In one embodiment, when the model to be quantized is quantized layer by layer based on the optimal parameter combination of each network layer, the following steps are performed for each network layer: Determine the target parameters that meet the target quantization clipping range as parameters to be quantized respectively; Determine the target parameters that do not meet the target quantization clipping range as parameters to be mapped; quantizing a plurality of parameters to be quantized based on the scaling factor and the target quantization bit width; If any parameter to be mapped is greater than the maximum quantization value of the target quantization bit width, mapping the parameter to be mapped to the maximum quantization value; If any parameter to be mapped is smaller than the minimum quantization value of the target quantization bit width, the any parameter to be mapped is mapped to the minimum quantization value.

[0012] In one embodiment, after quantizing the model to be quantized layer by layer based on the optimal parameter combination of each network layer to obtain the quantized model, the method further includes: Obtaining a first output of the model to be quantized, a first inference time consumption of the model to be quantized, a second output of the quantized model, and a second inference time consumption of the quantized model; Determine a loss value based on the first output, the second output, the first reasoning time, and the second reasoning time; Based on the loss value, a target parameter combination that meets the optimal quantization loss condition is selected in the search space.

[0013] In a second aspect, the present invention further provides a neural network lightweight device, comprising: A first determination module is used to determine a first preset value interval of a transfer difficulty coefficient, a second preset value interval of a quantization clipping range, and a third preset value interval of a quantization bit width; the transfer difficulty coefficient is used to adjust the coefficient of the quantization difficulty of multiple parameters in the network layer; the quantization clipping range refers to the range to which the parameter to be quantized belongs among the multiple parameters in the network layer; A construction module, configured to construct a search space based on the first preset value interval, the second preset value interval, and the third preset value interval; A second determination module is used to determine the optimal parameter combination of each network layer in the model to be quantized in the search space; the optimal parameter combination includes a target transfer difficulty coefficient, a target quantization clipping range and a target quantization bit width; The quantization module is used to quantize the model to be quantized layer by layer based on the optimal parameter combination of each network layer to obtain a quantized model.

[0014] In a third aspect, the present invention provides a device, comprising an electronic device, wherein the electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the above-mentioned neural network lightweighting methods are implemented.

[0015] In a fourth aspect, the present invention further provides a medium, comprising a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above-mentioned neural network lightweight methods are implemented.

[0016] In a fifth aspect, the present invention also provides a product, comprising a computer program product, wherein the computer program product comprises a computer program, wherein the computer program can be stored on a non-transitory computer-readable storage medium, and wherein when the computer program is executed by the processor, the steps of any of the above-mentioned neural network lightweighting methods are implemented.

[0017] The neural network lightweighting method, device, equipment, medium and product provided by the present invention comprehensively consider various factors affecting lightweighting, such as transfer difficulty coefficient, quantization clipping range and quantization bit width, and therefore pre-set the value range corresponding to each factor affecting lightweighting, so as to construct a search space, so that the lightweight method can be more universal and applicable to different types of network layer structures, and can also ensure that the subsequent quantized model can maintain accuracy, and further select the optimal parameter combination of each network layer in the model to be quantized in the search space, and based on the optimal parameter combination of each network layer, quantize the model to be quantized, so that the quantized model can reduce the consumption of computing resources and speed up the processing speed while maintaining accuracy, thereby ensuring the accuracy and performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1It is a flow chart of the neural network lightweight method provided by the present invention.

[0020] Figure 2 It is a schematic diagram of the difficulty of quantifying the transfer of model activation values ​​provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the model quantization and cropping provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the neural network lightweight device provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein.

[0026] Combine the following Figure 1-Figure 5 The invention describes the neural network lightweight method, device, equipment, medium and product provided by the invention.

[0027] Figure 1 It is a flow chart of the neural network lightweight method provided by the present invention.

[0028] like Figure 1 As shown, the method includes the following: Step 101, determining a first preset value interval of a transfer difficulty coefficient, a second preset value interval of a quantization clipping range, and a third preset value interval of a quantization bit width; Step 102: construct a search space based on the first preset value interval, the second preset value interval and the third preset value interval; Step 103: Determine the optimal parameter combination of each network layer in the model to be quantized in the search space; Step 104: Based on the optimal parameter combination of each network layer, the model to be quantized is quantized layer by layer to obtain a quantized model.

[0029] It should be noted that the neural network lightweight method provided in the embodiment of the present invention is implemented based on the neural network lightweight device. It compresses the size of the neural network model by using data types with smaller memory overhead, accelerates the inference speed of the model on hardware, and ensures the accuracy and performance of the model. The embodiment of the present invention describes the neural network lightweight method by taking the neural network lightweight device as the execution subject as an example.

[0030] Specifically, before performing neural network lightweighting, various factors related to lightweighting are comprehensively considered, and a first preset value range of the transfer difficulty coefficient, a second preset value range of the quantization clipping range, and a third preset value range of the quantization bit width are set in advance in the neural network lightweighting.

[0031] It should be noted that the transfer difficulty coefficient can be controlled by the hyperparameter α in the SmoothQuant method. Specifically, the core idea of ​​the SmoothQuant method is to control how much quantization difficulty is transferred from the activation value to the weight value by introducing a hyperparameter α, aiming to balance the quantization difficulty of the weights and activation values ​​in the model by adjusting the quantization difficulty.

[0032] The quantization clipping range refers to the range of parameters to be quantized among multiple parameters in the network layer to be quantized, while taking into account the quantization of activation values ​​and weights. Therefore, it can be understood that the quantization clipping range refers to the range of parameters to be quantized among multiple activation values ​​and weights in the network layer to be quantized.

[0033] Quantization bit width refers to the number of bits used to represent weights and activation values ​​in the process of neural network lightweighting. Specifically, the quantization bit width determines the number of bits occupied by weights and activation values ​​when they are stored and processed in the computer. Generally speaking, the smaller the quantization bit width, the lower the storage requirements and computing speed of the model, but at the same time it may cause loss of model accuracy. Therefore, in actual application, it is necessary to select an appropriate quantization bit width based on specific performance requirements.

[0034] In the embodiment of the present invention, the parameters to be quantized within the quantization clipping range are quantized according to the appropriate quantization bit width, while the parameters within the non-quantization clipping range are directly mapped to the quantization minimum value or the quantization maximum value of the quantization bit width.

[0035] Furthermore, the neural network lightweight device determines a first preset value interval of the transfer difficulty coefficient, a second preset value interval of the quantization clipping range, and a third preset value interval of the quantization bit width.

[0036] In one embodiment, the first preset value interval of the transfer difficulty coefficient can be [90%, 95%], the second preset value interval of the quantization clipping range can be [0.3, 0.7], and the third preset value interval of the quantization bit width can be {int4, int8, fp4, pf8}. Since the distribution of activation values ​​is different between different layers of the same model, the transfer difficulty coefficient and quantization clipping range between different layers may be different, and a value range is required for flexible selection. The choice of quantization bit width is determined according to the hardware conditions of the model. Generally speaking, the hardware conditions of the model will not change for each layer, so the quantization bit widths of different layers can be the same.

[0037] Furthermore, the neural network lightweight device sets a first step length of a first preset value interval and a second step length of a second preset value interval.

[0038] Furthermore, the neural network lightweight device constructs a search space based on the first preset value interval and its first step length, the second preset value interval and its second step length, and the third preset value interval.

[0039] It should be noted that each preset value interval has multiple values, and multiple parameter combinations can be constructed between different preset value intervals. Selecting a value from the first preset value interval, selecting a value from the second preset value interval, and selecting a value from the third preset value interval, and three values ​​selected from three different preset value intervals can construct a parameter combination.

[0040] Furthermore, the neural network lightweight device uses optimization algorithms, such as grid search, Bayesian optimization and other methods, in the search space to determine the optimal parameter combination of each network layer in the model to be quantized. The optimal parameter combination of each network layer includes the target transfer difficulty coefficient, the target quantization clipping range and the target quantization bit width.

[0041] Furthermore, the neural network lightweight device quantizes the model to be quantized layer by layer based on the optimal parameter combination of each network layer to obtain a quantized model.

[0042] The neural network lightweight method provided by the present invention comprehensively considers various factors affecting lightweight, such as transfer difficulty coefficient, quantization clipping range and quantization bit width, and therefore pre-sets the value range corresponding to each factor affecting lightweight, so as to construct a search space, so that the lightweight method can be more universal and applicable to different types of network layer structures, and can also ensure that the subsequent quantized model can maintain accuracy, and further select the optimal parameter combination of each network layer in the model to be quantized in the search space, and based on the optimal parameter combination of each network layer, the model to be quantized is quantized, so that the quantized model reduces the consumption of computing resources and speeds up the processing speed while maintaining accuracy, thereby ensuring the accuracy and performance of the model.

[0043] Further, based on step 103, when determining the optimal parameter combination of each network layer in the model to be quantized in the search space, the following steps are performed for each network layer: Acquire a calibration data set of the model to be quantified; Performing model inference based on the calibration data set to obtain multiple initial activation values ​​of the network layer; Determine the initial weight corresponding to each initial activation value; Based on multiple initial activation values ​​and multiple initial weights, a target transfer difficulty coefficient of the network layer in the model to be quantized is selected within the first preset value interval.

[0044] Specifically, the neural network lightweight device obtains a calibration data set of the model to be quantized.

[0045] Furthermore, the neural network lightweight device performs model inference based on the calibration data set to obtain multiple initial activation values ​​of the current network layer.

[0046] Furthermore, the neural network lightweight device determines an initial weight corresponding to each initial activation value.

[0047] It should be noted that each neuron receives the input signal from the previous layer, performs linear combination through weights, and adds bias to obtain the linear combination result, which is further converted into the activation value of the neuron through the activation function. Therefore, each activation value will have a corresponding weight, and there may be multiple weights, which are not limited here.

[0048] Furthermore, the neural network lightweight device uses an optimization algorithm based on multiple initial activation values ​​and multiple initial weights to select a target transfer difficulty coefficient of the current network layer in the model to be quantized within a first preset value range.

[0049] The embodiment of the present invention selects a target transfer difficulty coefficient of a network layer within a first preset value range based on an initial activation value and an initial weight, thereby controlling the difficulty of migrating from an activation value to a weight to balance the accuracy and efficiency in the quantization process, thereby improving the reasoning speed and efficiency of the model while maintaining the accuracy and performance of the model.

[0050] Further, after selecting the target transfer difficulty coefficient of the network layer in the model to be quantized within the first preset value interval based on the multiple initial activation values ​​and the multiple initial weights, the method further includes: Determining a scaling factor based on the target transfer difficulty coefficient; the scaling factor is used to quantize the parameter to be quantized; Based on the scaling factor, each initial activation value and each initial weight are converted to obtain multiple target activation values ​​and multiple target weights.

[0051] Specifically, the neural network lightweight device determines a scaling factor for quantizing the parameter to be quantized based on the target transfer difficulty coefficient.

[0052] In one embodiment, based on the target transfer difficulty coefficient, a specific calculation formula for determining the scaling factor is as follows:

[0053] Among them, s represents the scaling factor; Characterizes the target transfer difficulty coefficient; X is an activation value matrix composed of multiple initial activation values; W is a weight matrix composed of multiple initial weights; Characterization i The maximum absolute value of the initial activation value of the input channels; Characterization i The maximum absolute value of the initial weights of the input channels.

[0054] It should be noted that the reference Figure 2 , Figure 2 Schematic diagram of the difficulty of quantifying the transfer of model activation values ​​provided by the present invention. Figure 2 As shown in the figure, X is an activation matrix composed of multiple initial activation values, with a size of 2×4, and W is a weight matrix composed of multiple initial weights, with a size of 4×3. The maximum activation value in the activation matrix X is 16 times worse than the minimum activation value, while the maximum weight in the weight matrix W is only 2 times worse than the minimum weight. Therefore, the activation matrix X is more difficult to quantize than the weight matrix W. It is possible to consider transferring part of the difficulty of quantizing the activation value to the quantizing weight through the matrix equivalent substitution method.

[0055] Furthermore, the neural network lightweight device converts each initial activation value and each initial weight based on the scaling factor to obtain multiple target activation values ​​and multiple target weights.

[0056] It can also be understood that the neural network lightweight device converts each initial activation value and each initial weight by equivalent replacement based on the scaling factor to obtain multiple target activation values ​​and multiple target weights after conversion.

[0057] Among them, we can consider the equivalent replacement method of matrix multiplication. The specific formula is as follows:

[0058] Among them, X is an activation value matrix composed of multiple initial activation values; W is a weight matrix composed of multiple initial weights; is the activation value matrix after equivalent replacement; is the weight matrix after equivalent replacement.

[0059] like Figure 2 As shown, through the converted It is equivalent to the original XW. After the conversion, the maximum activation value in the activation value matrix X is reduced to 4 times the minimum activation value, while the maximum activation value in the converted weight matrix W is increased to 4 times the minimum activation value, so as to adjust the difficulty of quantizing the activation value and the quantization weight.

[0060] The embodiment of the present invention first determines the scaling factor based on the target transfer difficulty coefficient, and further based on the scaling factor, performs equivalent replacement of each initial activation value and each initial weight to achieve the difficulty of controlling the migration from the activation value to the weight. Compared with the traditional separate quantization, the value range of the activation value is narrowed, the difficulty of quantizing the activation value is reduced, and a more effective quantization process can be achieved, thereby ensuring the performance and accuracy of the incremental model.

[0061] Further, based on step 103, when determining the optimal parameter combination of each network layer in the model to be quantized in the search space, the following steps are further performed for each network layer: selecting a plurality of outliers among the plurality of target activation values ​​and the plurality of target weights; Constructing a number axis based on a plurality of target parameters; the plurality of target parameters comprising the plurality of target activation values ​​and the plurality of target weights; Determine the remaining target parameters among the plurality of target parameters except the plurality of outliers as normal values; Based on the number axis, determining a first parameter distribution of a plurality of normal values ​​and a second parameter distribution of the plurality of outliers; Based on the first parameter distribution and the second parameter distribution, a target quantization clipping range of the network layer in the model to be quantized is selected within the second preset value interval.

[0062] Specifically, the neural network lightweight device determines multiple target activation values ​​and multiple target weights as multiple parameters. Because both the activation values ​​and the weights are quantized during the quantization process, the following process is described using multiple parameters.

[0063] Furthermore, the neural network lightweight device selects multiple outliers from multiple parameters.

[0064] Furthermore, the neural network lightweight device constructs a number axis based on multiple target parameters, which can be specifically: select a coordinate origin on a straight line, the right side of the coordinate origin is the positive direction, and the left side of the coordinate origin is the negative direction. According to the values ​​of multiple target parameters, the position of each target parameter is marked on the straight line to obtain the corresponding number axis. The number axis can be used to display the distribution of multiple target parameters, and the concentration trend or dispersion degree of the parameters can be intuitively understood.

[0065] Furthermore, the neural network lightweight device determines the remaining target parameters among the multiple target parameters except for the multiple outliers as normal values.

[0066] Furthermore, the neural network lightweight device determines a first parameter distribution of multiple normal values ​​and a second parameter distribution of multiple outliers based on a number axis.

[0067] Furthermore, the neural network lightweight device selects the target quantization clipping range of the current network layer in the model to be quantized within the second preset value interval based on the first parameter distribution and the second parameter distribution through an optimization algorithm, which can include as many target parameters as possible, including both outliers and normal values.

[0068] It should be noted that if the target parameter is mainly distributed on the positive semi-axis and the maximum value of the parameter, the maximum value of the parameter is taken as the starting point, and the cropping ratio is determined. The target quantization cropping range can be obtained by dividing from the maximum value of the parameter on the numerical axis until the cropping ratio is met. If the target parameter is mainly distributed on the negative semi-axis and the minimum value of the parameter, the minimum value of the parameter is taken as the starting point, and the cropping ratio is determined. The target quantization cropping range can be obtained by dividing from the minimum value of the parameter on the numerical axis until the cropping ratio is met. If the target parameter is mainly distributed near the origin, rather than near the maximum value of the parameter, the maximum value of the parameter is not taken as the starting point. The most densely populated area within the range of [minimum value of parameter, maximum value of parameter] is selected and divided according to the cropping ratio to obtain the target quantization cropping range.

[0069] In one embodiment, referring to Figure 3 , Figure 3 Schematic diagram of model quantization and cropping provided by the present invention. Figure 3As shown in the figure, taking the quantization bit width quantized to int8 as an example, the left figure shows the quantization cropping of the full mapping of the maximum and minimum values. There is a loss of precision in the dense area after quantization on the negative semi-axis. The right figure shows the quantization cropping of the maximum value mapping of the cropped part. The unimportant maximum value can be directly mapped to the maximum value after quantization to better maintain the model inference accuracy.

[0070] The embodiment of the present invention takes into account the distribution of multiple normal values ​​and the distribution of multiple outliers, and can divide these relatively dense parameters as much as possible in the target quantization clipping range, taking into account the non-uniform distribution of activation values ​​and weights, thereby realizing adaptive scaling quantization based on the distribution characteristics of weights and activation values, thereby ensuring the accuracy and performance of the quantized model.

[0071] Further, based on step 103, the selecting a plurality of outliers from the plurality of target activation values ​​and the plurality of target weights includes: determining the magnitude of each target activation value; Determine the target activation values ​​corresponding to the first N largest amplitudes among the multiple amplitudes as abnormal activation values; Determine the target weight corresponding to each abnormal activation value as the abnormal weight; Multiple abnormal activation values ​​and multiple abnormal weights are respectively determined as outliers.

[0072] Specifically, the neural network lightweight device determines the magnitude of each target activation value.

[0073] Furthermore, the neural network lightweight device compares the numerical sizes of multiple amplitudes to obtain a numerical comparison result.

[0074] Furthermore, the neural network lightweight device determines the target activation values ​​corresponding to the top N maximum amplitudes as abnormal activation values ​​based on the numerical comparison results, where N is set according to actual conditions, and preferably, the top 0.5% or 1% of the amplitudes sorted by the maximum value can be selected, and the corresponding target activation values ​​are the abnormal activation values.

[0075] It should be noted that the abnormal activation value can also be determined by the L2 norm.

[0076] Furthermore, the neural network lightweight device determines the target weight corresponding to each abnormal activation value as the abnormal weight.

[0077] Furthermore, the neural network lightweight device determines multiple abnormal activation values ​​and multiple abnormal weights as outliers respectively.

[0078] The embodiment of the present invention determines the outliers of the activation value through the amplitude or L2 norm in the model activation value, and determines the outliers of the weight correspondingly through the outliers of the activation value, so as to better preserve the weights that affect the activation value, and can also better protect the weight parameters that can produce outliers of the activation value when selecting the quantization range, so as to better maintain the accuracy of the quantized model.

[0079] Further, based on step 103, when determining the optimal parameter combination of each network layer in the model to be quantized in the search space, the following steps are further performed for each network layer: Determine the target hardware for the deployment of the model to be quantized; According to the target hardware, a target quantization bit width of the network layer in the model to be quantized is selected within the third preset value interval.

[0080] Specifically, the neural network lightweight device determines the target hardware on which the model to be quantized is deployed.

[0081] It should be noted that different quantization bit widths are determined according to the target hardware for model deployment, such as int4, int8, fp4, fp8 and other data types. Some low-power devices only have integer data types, such as int4 and int8, while fp4 and fp8 on the server side have the advantage of inference speed.

[0082] Furthermore, the neural network lightweight device selects a target quantization bit width of the current network layer in the model to be quantized within a third preset value interval according to the target hardware.

[0083] The embodiments of the present invention take into account that the target hardware for model deployment will directly affect the storage requirements, execution efficiency and inference speed of the model. Therefore, in order to adapt to different hardware conditions, data types with different quantization bit widths can be considered to improve the adaptability of quantization.

[0084] Further, based on step 103, when the model to be quantized is quantized layer by layer based on the optimal parameter combination of each network layer, the following steps are performed for each network layer: Determine the target parameters that meet the target quantization clipping range as parameters to be quantized respectively; Determine the target parameters that do not meet the target quantization clipping range as parameters to be mapped; quantizing a plurality of parameters to be quantized based on the scaling factor and the target quantization bit width; If any parameter to be mapped is greater than the maximum quantization value of the target quantization bit width, mapping the parameter to be mapped to the maximum quantization value; If any parameter to be mapped is smaller than the minimum quantization value of the target quantization bit width, the any parameter to be mapped is mapped to the minimum quantization value.

[0085] Specifically, the neural network lightweight device determines the target parameters that meet the target quantization clipping range as parameters to be quantized.

[0086] Furthermore, the neural network lightweight device determines the target parameters that do not meet the target quantization clipping range as parameters to be mapped.

[0087] Furthermore, the neural network lightweight device quantizes multiple parameters to be quantized based on the scaling factor and the target quantization bit width.

[0088] It should be noted that the parameters to be mapped that do not meet the target quantization clipping range are relatively discrete and will have certain quantization difficulties. However, in order to avoid the loss of precision after quantization, these discrete values ​​can be directly mapped to the maximum value after quantization.

[0089] Furthermore, if any parameter to be mapped is greater than the maximum quantization value of the target quantization bit width, the neural network lightweight quantization device maps the arbitrary parameter to be mapped to the maximum quantization value.

[0090] Furthermore, if any parameter to be mapped is smaller than the minimum quantization value of the target quantization bit width, the neural network lightweight quantization device maps the arbitrary parameter to be mapped to the minimum quantization value.

[0091] It should be noted that the bit width of the network layer parameters is usually much larger than the bit width after quantization, and the parameters to be mapped that do not meet the target quantization clipping range are usually distributed at the extreme value. Therefore, the parameters to be mapped are either smaller than the minimum quantization value of the target quantization bit width, or larger than the maximum quantization value of the target quantization bit width.

[0092] The embodiment of the present invention quantizes the parameters to be quantized within the target quantization clipping range according to the scaling factor, and directly maps the discrete parameters to be mapped to the quantized bit width, thereby achieving a more effective quantization effect, thereby ensuring the accuracy and performance of the quantized model.

[0093] Furthermore, after quantizing the model to be quantized layer by layer based on the optimal parameter combination of each network layer to obtain the quantized model, the method further includes: Obtaining a first output of the model to be quantized, a first inference time consumption of the model to be quantized, a second output of the quantized model, and a second inference time consumption of the quantized model; Determine a loss value based on the first output, the second output, the first reasoning time, and the second reasoning time; Based on the loss value, a target parameter combination that meets the optimal quantization loss condition is selected in the search space.

[0094] Specifically, the neural network lightweight device obtains the first output of the model to be quantized, the first inference time of the model to be quantized, the second output of the quantized model, and the second inference time of the quantized model.

[0095] Furthermore, the neural network lightweight device determines a loss value based on the first output, the second output, the first inference time, and the second inference time.

[0096] In one embodiment, the calculation formula of the loss value is as follows:

[0097] in, l Characterize the loss value; represents the second output of the quantized model; Y represents the first output of the model to be quantized; Characterizes the second inference time of the quantized model; T characterizes the first inference time of the model to be quantized; MSE is the mean square error function.

[0098] Furthermore, the neural network lightweight device selects a target parameter combination that meets the optimal quantization loss condition in the search space based on the loss value. The target parameter combination can be used to adjust specific parameters in the quantization process, thereby guiding subsequent quantization operations.

[0099] It should be noted that the above-mentioned neural network lightweight method can effectively reduce the deployment cost of large models and improve the efficiency of large model reasoning. On this basis, large models can be applied to smart communities, mobile home security, real-time applications and reasoning, and other businesses to promote the intelligent development of businesses.

[0100] The embodiment of the present invention obtains a loss value by calculating the model after quantization and the model before quantization, and further selects a target parameter combination of the optimal quantization loss condition through the loss value, aiming to minimize the performance loss in the quantization process, thereby ensuring that the quantized model achieves the best lightweight effect while maintaining performance.

[0101] The neural network lightweight device provided by the present invention is described below. The neural network lightweight device described below and the neural network lightweight method described above can be referenced to each other.

[0102] Reference Figure 4 , Figure 4 It is a structural schematic diagram of the neural network lightweight device provided by the present invention.

[0103] The neural network lightweight device comprises: The first determination module 410 is used to determine a first preset value interval of a transfer difficulty coefficient, a second preset value interval of a quantization clipping range, and a third preset value interval of a quantization bit width; the transfer difficulty coefficient is used to adjust the coefficient of the quantization difficulty of multiple parameters in the network layer; the quantization clipping range refers to the range to which the parameters to be quantized belong among the multiple parameters in the network layer.

[0104] The construction module 420 is used to construct a search space based on the first preset value interval, the second preset value interval and the third preset value interval.

[0105] The second determination module 430 is used to determine the optimal parameter combination of each network layer in the model to be quantized in the search space; the optimal parameter combination includes a target transfer difficulty coefficient, a target quantization clipping range and a target quantization bit width.

[0106] The quantization module 440 is used to quantize the model to be quantized layer by layer based on the optimal parameter combination of each network layer to obtain a quantized model.

[0107] The neural network lightweight device provided by the present invention comprehensively considers various factors affecting lightweighting, such as transfer difficulty coefficient, quantization clipping range and quantization bit width, and therefore pre-sets the value range corresponding to each factor affecting lightweighting, so as to construct a search space, so that the lightweight method can be more universal and applicable to different types of network layer structures, and can also ensure that the subsequent quantized model can maintain accuracy, and further select the optimal parameter combination of each network layer in the model to be quantized in the search space, and based on the optimal parameter combination of each network layer, quantize the model to be quantized, so that the quantized model can reduce the consumption of computing resources and speed up the processing speed while maintaining accuracy, thereby ensuring the accuracy and performance of the model.

[0108] Furthermore, the second determining module 430 is further configured to: Acquire a calibration data set of the model to be quantified; Performing model inference based on the calibration data set to obtain multiple initial activation values ​​of the network layer; Determine the initial weight corresponding to each initial activation value; Based on multiple initial activation values ​​and multiple initial weights, a target transfer difficulty coefficient of the network layer in the model to be quantized is selected within the first preset value interval.

[0109] Furthermore, the neural network lightweight device is also used for: Determining a scaling factor based on the target transfer difficulty coefficient; the scaling factor is used to quantize the parameter to be quantized; Based on the scaling factor, each initial activation value and each initial weight are converted to obtain multiple target activation values ​​and multiple target weights.

[0110] Furthermore, the second determining module 430 is further configured to: selecting a plurality of outliers among the plurality of target activation values ​​and the plurality of target weights; Constructing a number axis based on a plurality of target parameters; the plurality of target parameters comprising the plurality of target activation values ​​and the plurality of target weights; Determine the remaining target parameters among the plurality of target parameters except the plurality of outliers as normal values; Based on the number axis, determining a first parameter distribution of a plurality of normal values ​​and a second parameter distribution of the plurality of outliers; Based on the first parameter distribution and the second parameter distribution, a target quantization clipping range of the network layer in the model to be quantized is selected within the second preset value interval.

[0111] Furthermore, the second determining module 430 is further configured to: determining the magnitude of each target activation value; Determine the target activation values ​​corresponding to the first N largest amplitudes among the multiple amplitudes as abnormal activation values; Determine the target weight corresponding to each abnormal activation value as the abnormal weight; Multiple abnormal activation values ​​and multiple abnormal weights are respectively determined as outliers.

[0112] Furthermore, the second determining module 430 is further configured to: Determine the target hardware for the deployment of the model to be quantized; According to the target hardware, a target quantization bit width of the network layer in the model to be quantized is selected within the third preset value interval.

[0113] Furthermore, the quantization module 440 is also used for: Determine the target parameters that meet the target quantization clipping range as parameters to be quantized respectively; Determine the target parameters that do not meet the target quantization clipping range as parameters to be mapped; quantizing a plurality of parameters to be quantized based on the scaling factor and the target quantization bit width; If any parameter to be mapped is greater than the maximum quantization value of the target quantization bit width, mapping the parameter to be mapped to the maximum quantization value; If any parameter to be mapped is smaller than the minimum quantization value of the target quantization bit width, the any parameter to be mapped is mapped to the minimum quantization value.

[0114] Furthermore, the neural network lightweight device is also used for: Obtaining a first output of the model to be quantized, a first inference time consumption of the model to be quantized, a second output of the quantized model, and a second inference time consumption of the quantized model; Determine a loss value based on the first output, the second output, the first reasoning time, and the second reasoning time; Based on the loss value, a target parameter combination that meets the optimal quantization loss condition is selected in the search space.

[0115] It should be noted that the neural network lightweight device provided by the present invention can execute the neural network lightweight method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0116] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the neural network lightweight method, which includes: determining a first preset value interval of a transfer difficulty coefficient, a second preset value interval of a quantization clipping range, and a third preset value interval of a quantization bit width; the transfer difficulty coefficient is used to adjust the coefficient of the quantization difficulty of multiple parameters in the network layer; the quantization clipping range refers to the range to which the parameters to be quantized belong among the multiple parameters in the network layer; based on the first preset value interval, the second preset value interval, and the third preset value interval, a search space is constructed; in the search space, the optimal parameter combination of each network layer in the model to be quantized is determined; the optimal parameter combination includes a target transfer difficulty coefficient, a target quantization clipping range, and a target quantization bit width; based on the optimal parameter combination of each network layer, the model to be quantized is quantized layer by layer to obtain a quantized model.

[0117] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0118] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the neural network lightweight method provided by the above-mentioned embodiments, and the method includes: determining a first preset value interval of a transfer difficulty coefficient, a second preset value interval of a quantization clipping range, and a third preset value interval of a quantization bit width; the transfer difficulty coefficient is used to adjust the coefficient of the quantization difficulty of multiple parameters in the network layer; the quantization clipping range refers to the range to which the parameters to be quantized belong among the multiple parameters in the network layer; based on the first preset value interval, the second preset value interval, and the third preset value interval, a search space is constructed; in the search space, the optimal parameter combination of each network layer in the model to be quantized is determined; the optimal parameter combination includes a target transfer difficulty coefficient, a target quantization clipping range, and a target quantization bit width; based on the optimal parameter combination of each network layer, the model to be quantized is quantized layer by layer to obtain a quantized model.

[0119] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the neural network lightweight quantization method provided in the above-mentioned embodiments, the method comprising: determining a first preset value interval of a transfer difficulty coefficient, a second preset value interval of a quantization clipping range, and a third preset value interval of a quantization bit width; the transfer difficulty coefficient is used to adjust the coefficient of the quantization difficulty of multiple parameters in the network layer; the quantization clipping range refers to the range to which the parameters to be quantized belong among the multiple parameters in the network layer; based on the first preset value interval, the second preset value interval, and the third preset value interval, a search space is constructed; in the search space, the optimal parameter combination of each network layer in the model to be quantized is determined; the optimal parameter combination includes a target transfer difficulty coefficient, a target quantization clipping range, and a target quantization bit width; based on the optimal parameter combination of each network layer, the model to be quantized is quantized layer by layer to obtain a quantized model.

[0120] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A neural network lightweight method, characterized in that: include: Determine a first preset value interval of the transfer difficulty coefficient, a second preset value interval of the quantization clipping range, and a third preset value interval of the quantization bit width; The transfer difficulty coefficient is used to adjust the coefficient of the difficulty of quantizing multiple parameters in the network layer; The quantization clipping range refers to the range to which the parameters to be quantized belong among multiple parameters in the network layer; Constructing a search space based on the first preset value interval, the second preset value interval, and the third preset value interval; In the search space, determining an optimal parameter combination of each network layer in the model to be quantized; the optimal parameter combination includes a target transfer difficulty coefficient, a target quantization clipping range, and a target quantization bit width; Based on the optimal parameter combination of each network layer, the model to be quantized is quantized layer by layer to obtain a quantized model.

2. The neural network lightweight method according to claim 1, characterized in that: In the search space, when determining the optimal parameter combination of each network layer in the model to be quantized, the following steps are performed for each network layer: Acquire a calibration data set of the model to be quantified; Performing model inference based on the calibration data set to obtain multiple initial activation values ​​of the network layer; Determine the initial weight corresponding to each initial activation value; Based on multiple initial activation values ​​and multiple initial weights, a target transfer difficulty coefficient of the network layer in the model to be quantized is selected within the first preset value interval.

3. The neural network lightweight method according to claim 2, characterized in that: After selecting the target transfer difficulty coefficient of the network layer in the model to be quantized within the first preset value interval based on the multiple initial activation values ​​and the multiple initial weights, the method further includes: Based on the target transfer difficulty coefficient, determining a scaling factor; the scaling factor is used to quantize the parameter to be quantized; Based on the scaling factor, each initial activation value and each initial weight are converted to obtain multiple target activation values ​​and multiple target weights.

4. The neural network lightweight method according to claim 3, characterized in that: In the search space, when determining the optimal parameter combination of each network layer in the model to be quantized, the following steps are further performed for each network layer: selecting a plurality of outliers among the plurality of target activation values ​​and the plurality of target weights; Constructing a number axis based on a plurality of target parameters; the plurality of target parameters comprising the plurality of target activation values ​​and the plurality of target weights; Determine the remaining target parameters among the plurality of target parameters except the plurality of outliers as normal values; Based on the number axis, determining a first parameter distribution of a plurality of normal values ​​and a second parameter distribution of the plurality of outliers; Based on the first parameter distribution and the second parameter distribution, a target quantization clipping range of the network layer in the model to be quantized is selected within the second preset value interval.

5. The neural network lightweight method according to claim 4, characterized in that: The selecting a plurality of outliers from the plurality of target activation values ​​and the plurality of target weights comprises: determining the magnitude of each target activation value; Determine the target activation values ​​corresponding to the first N largest amplitudes among the multiple amplitudes as abnormal activation values; Determine the target weight corresponding to each abnormal activation value as the abnormal weight; Multiple abnormal activation values ​​and multiple abnormal weights are respectively determined as outliers.

6. The neural network lightweight method according to claim 1, characterized in that: In the search space, when determining the optimal parameter combination of each network layer in the model to be quantized, the following steps are further performed for each network layer: Determine the target hardware for the deployment of the model to be quantized; According to the target hardware, a target quantization bit width of the network layer in the model to be quantized is selected within the third preset value interval.

7. The neural network lightweight method according to claim 5, characterized in that: When the model to be quantized is quantized layer by layer based on the optimal parameter combination of each network layer, the following steps are performed for each network layer: Determine the target parameters that meet the target quantization clipping range as parameters to be quantized respectively; Determine the target parameters that do not meet the target quantization clipping range as parameters to be mapped; quantizing a plurality of parameters to be quantized based on the scaling factor and the target quantization bit width; If any parameter to be mapped is greater than the maximum quantization value of the target quantization bit width, mapping the parameter to be mapped to the maximum quantization value; If any parameter to be mapped is smaller than the minimum quantization value of the target quantization bit width, the any parameter to be mapped is mapped to the minimum quantization value.

8. The method for lightweighting a neural network according to any one of claims 1 to 7, characterized in that: After quantizing the model to be quantized layer by layer based on the optimal parameter combination of each network layer to obtain the quantized model, the method further includes: Obtaining a first output of the model to be quantized, a first inference time consumption of the model to be quantized, a second output of the quantized model, and a second inference time consumption of the quantized model; Determine a loss value based on the first output, the second output, the first reasoning time, and the second reasoning time; Based on the loss value, a target parameter combination that meets the optimal quantization loss condition is selected in the search space.

9. A neural network lightweight device, characterized in that: include: A first determination module, used to determine a first preset value interval of a transfer difficulty coefficient, a second preset value interval of a quantization clipping range, and a third preset value interval of a quantization bit width; The transfer difficulty coefficient is used to adjust the coefficient of the difficulty of quantizing multiple parameters in the network layer; The quantization clipping range refers to the range to which the parameters to be quantized belong among multiple parameters in the network layer; A construction module, configured to construct a search space based on the first preset value interval, the second preset value interval, and the third preset value interval; A second determination module is used to determine the optimal parameter combination of each network layer in the model to be quantized in the search space; the optimal parameter combination includes a target transfer difficulty coefficient, a target quantization clipping range and a target quantization bit width; The quantization module is used to quantize the model to be quantized layer by layer based on the optimal parameter combination of each network layer to obtain a quantized model.

10. A device, comprising an electronic device, the electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the neural network lightweight method according to any one of claims 1 to 8 are implemented.

11. A medium, comprising a non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the neural network lightweight method as claimed in any one of claims 1 to 8 are implemented.

12. A product, comprising a computer program product, wherein the computer program product comprises a computer program, characterized in that: When the computer program is executed by a processor, the steps of the neural network lightweight method as claimed in any one of claims 1 to 8 are implemented.

Citation Information

Cited By

  • Neural network lightweight and efficient FPGA deployment method based on Bayesian optimization

    CN119849561A

  • A lightweight and efficient FPGA deployment method for neural networks based on Bayesian optimization

    CN119849561B

  • Method for selecting activation value processing mode in ai model, and related device

    WO2026214430A1