Weight quantization method, device and server based on hybrid segment coding

Abstract

The application provides a weight quantization method and device based on hybrid segmentation coding and a server, relates to the technical field of artificial intelligence, and comprises the following steps: obtaining floating-point weights and activation values to be processed; performing hybrid segmentation coding processing on the floating-point weights based on the Laplace distribution of the floating-point weights to obtain a set of coded weights in a quantized weight coding format; determining a multiplication coefficient and a shift value of the coded weights based on an identification bit, and performing multiplication and addition calculation processing on the activation values and weight information in data bits based on the multiplication coefficient and the shift value to obtain output activation values corresponding to the coded weights. The application can significantly improve the quantization precision of a neural network, and reduce quantization errors while keeping the compression rate unchanged.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a weight quantization method and device based on hybrid segmentation coding and a server. BACKGROUND

[0002] With the rapid popularization of artificial intelligence in mobile terminals and edge computing scenarios, weight quantization has become a key technology to ensure the efficiency and deployment cost of deep neural network inference. Currently, related technologies propose that tensor-by-tensor quantization, channel-by-channel quantization, and group quantization can be used. However, tensor-by-tensor quantization is prone to introduce significant quantization errors when the numerical distribution between channels is significantly different. Channel-by-channel quantization is prone to cause the weights within a channel to have a large span when significant numerical contrast is introduced in the output channel, thereby reducing the quantization precision. Group quantization has low accuracy when the channel characteristics within a group are significantly different. Therefore, the above-mentioned prior art solutions have the problems of low quantization precision and large quantization error. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a weight quantization method and device based on hybrid segmentation coding and a server, which can significantly improve the quantization precision of neural networks and reduce quantization error.

[0004] In a first aspect, an embodiment of the present application provides a weight quantization method based on hybrid segmentation coding. The method includes obtaining floating-point weights and activation values to be processed, performing hybrid segmentation coding processing on the floating-point weights based on the Laplace distribution of the floating-point weights to obtain a set of quantized weight coding formats, wherein each item of the set of coding weights includes data bits and identification bits, determining the multiplication coefficient and shift value of the coding weight based on the identification bits, and performing multiplication and addition calculation processing on the activation value and the weight information in the data bits based on the multiplication coefficient and shift value to obtain the output activation value corresponding to the coding weight.

[0005] In one embodiment, after the step of obtaining floating-point weights and activation values to be processed, the method includes obtaining a data distribution histogram corresponding to the floating-point weights by statistically analyzing the floating-point weights, and obtaining a probability density function after data fitting of the floating-point weights based on a Laplace model; and combining the data distribution histogram and the probability density function to obtain the Laplace distribution of the floating-point weights.

[0006] In an implementation, the step of mixing and segmenting the floating-point weight based on the Laplace distribution of the floating-point weight to obtain a set of quantized weight coding format coding weights comprises: segmenting and quantizing the floating-point weight based on the Laplace distribution of the floating-point weight to divide the weight value of the floating-point weight into a set of sub-intervals; mapping the weight value in each sub-interval to a fixed-point representation value of data bits, and constructing weight grouping information of identification bits using the index corresponding to the sub-interval; combining the fixed-point representation value and the corresponding weight grouping information to obtain a set of coding weights.

[0007] In an implementation, the step of segmenting and quantizing the floating-point weight to divide the weight value of the floating-point weight into a set of sub-intervals comprises: using the weight value as a threshold by the percentile method to divide the weight value of the floating-point weight into a main peak region, a moderately sparse region, and a positive value region, and determining the main peak region, the moderately sparse region, and the positive value region as the set of sub-intervals.

[0008] In an implementation, the step of combining the fixed-point representation value and the corresponding weight grouping information to obtain a set of coding weights comprises: determining the corresponding weight grouping information according to the sub-interval to which the weight value before mapping belongs for each fixed-point representation value, and inserting the weight grouping information into the end of the corresponding fixed-point representation value to obtain a set of coding weights, wherein the weight grouping information corresponding to the weight values in the same sub-interval is the same.

[0009] In an implementation, the step of determining the multiplication coefficient and the shift value of the coding weight based on the identification bits comprises: determining the multiplication coefficient and the shift value of the coding weight by table lookup processing of the weight grouping information corresponding to any coding weight.

[0010] In an implementation, the step of performing multiplication and addition calculation processing on the activation value and the weight information in the data bits to obtain an output activation value corresponding to the coding weight comprises: substituting the activation value and the weight information into a multiplication and addition calculation model, and performing fixed-point processing on a floating-point proportion item in the multiplication and addition calculation model to convert the floating-point proportion item into a fixed-point form, so as to solve the multiplication and addition calculation model to obtain an output activation value.

[0011] In a second aspect, the embodiment of the present application further provides a weight quantization device based on hybrid segment coding, which comprises: an information acquisition module, configured to acquire floating-point weights and activation values to be processed; a hybrid segment coding module, configured to perform hybrid segment coding processing on the floating-point weights based on the Laplace distribution of the floating-point weights, to obtain a set of coded weights in a quantized weight coding format, wherein each coded weight in the set of coded weights comprises data bits and an identification bit; and a weight quantization module, configured to determine a multiplication coefficient and a shift value of the coded weight based on the identification bit, and perform multiplication and addition calculation processing on the activation value and weight information in the data bits based on the multiplication coefficient and the shift value, to obtain an output activation value corresponding to the coded weight.

[0012] In a third aspect, the embodiment of the present application further provides a server, comprising a processor and a memory, wherein the memory stores computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the method of any one of the first aspect.

[0013] In a fourth aspect, the embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions, when invoked and executed by a processor, cause the processor to implement the method of any one of the first aspect.

[0014] The embodiment of the present application brings the following beneficial effects:

[0015] The weight quantization method, device and server based on hybrid segment coding provided by the embodiment of the present application, after acquiring floating-point weights and activation values to be processed, perform hybrid segment coding processing on the floating-point weights based on the Laplace distribution of the floating-point weights, to obtain a set of coded weights in a quantized weight coding format, then determine a multiplication coefficient and a shift value of the coded weight based on the identification bit, and perform multiplication and addition calculation processing on the activation value and weight information in the data bits based on the multiplication coefficient and the shift value, to obtain an output activation value corresponding to the coded weight, which can improve the quantization precision of the neural network and enhance the adaptability to the non-uniform weight distribution within the channel.

[0016] Other features and advantages of the present application will be further described in the following description, and part of them will become apparent from the description, or will be understood by those skilled in the art. The purposes and other advantages of the present application will be achieved and obtained by the structure specifically pointed out in the specification, claims and drawings.

[0017] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are referred to for detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application or the prior art, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0019] Figure 1 A flowchart of a weight quantization method based on hybrid segmentation coding provided for an embodiment of the present application is shown in the figure.

[0020] Figure 2 A specific flowchart of a weight quantization method based on hybrid segmentation coding provided for an embodiment of the present application is shown in the figure.

[0021] Figure 3 A schematic diagram of a neural network weight distribution provided for an embodiment of the present application is shown in the figure.

[0022] Figure 4 A schematic diagram of 10-bit quantization weight bit partitioning provided for an embodiment of the present application is shown in the figure.

[0023] Figure 5 A structural schematic diagram of a weight quantization device based on hybrid segmentation coding provided for an embodiment of the present application is shown in the figure.

[0024] Figure 6 A structural schematic diagram of a server provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the present application will be described clearly and completely in combination with embodiments. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0026] Currently, with the popularity of artificial intelligence in mobile and edge computing scenarios, weight quantization has become a key technology to ensure the inference efficiency and deployment cost of deep neural networks. In the model deployment process, after the floating-point weights are loaded into the buffer of the compiler, the quantization module first converts them into integer reference values, and then writes the encoded weights into the device storage. Subsequently, these integer weights are sent to the multiplication unit for operation, and after accumulation and scaling, they are restored to approximate floating-point results by the dequantization module and passed to the subsequent layer. The entire process must balance the dynamic range and resolution under strict bandwidth and bit width budget. Currently, the main quantization methods include perTensor (per-tensor) quantization, perChannel (per-channel) quantization, and Group (fixed grouping) quantization strategies.

[0027] The related art proposes that perTensor quantization uniformly allocates scaling factors and zero points to the entire tensor. Although it is simple to implement and has low overhead, it is easy to introduce significant quantization errors in cases where the numerical distribution between channels is significantly different. To alleviate this problem, perChannel quantization allows each channel to use independent quantization parameters, improving the adaptability to distribution differences. However, during the training process of neural networks, in order to enhance the model expression ability and improve the separability between features, significant numerical contrasts are often introduced in each output channel, resulting in a distribution with a large span of weight values even within a single channel. This high dynamic range feature compresses the effective precision interval, making it difficult for perChannel quantization to accurately approximate within that channel. Compared to perTensor, perChannel has more flexibility, but its precision improvement depends on the density and concentration of data within the channel. If the original distribution itself is already extensive and uneven, the quantization effect will still be limited. In addition, Group quantization, as a compromise, attempts to divide multiple channels into fixed groups to share quantization parameters, reducing complexity while improving accuracy. However, the grouping strategy is usually based on static structure settings and lacks dynamic adaptation to the distribution differences within the group. If the channel characteristics within the group are significantly different, it will still affect the final inference effect, limiting its universality and stability.

[0028] Based on this, the weight quantization method, device and server based on hybrid segment coding provided by the embodiment of the application can significantly improve the quantization accuracy of neural networks while reducing quantization errors at the same compression rate.

[0029] Referring to Figure 1 The method mainly includes the following steps S102 to S106:

[0030] Step S102, obtaining floating-point weights and activation values to be processed.

[0031] In an embodiment, the floating-point weight is an original model parameter obtained in a model training process of a neural network, and the activation value is a value input into the model, for example, in a punch card, there is a trained model, and information input into the model by each punch card person when punching the card is the activation value.

[0032] In step S104, the floating-point weight is mixed segment coding processing based on a Laplace distribution of the floating-point weight, to obtain an encoded weight set in a quantized weight coding format, wherein each encoded weight in the encoded weight set includes a data bit and an identification bit.

[0033] In an embodiment, by using a segment structure fixed-point quantization coding mode, each neural network weight can be represented as a combination of an 8-bit quantized value and a 2-bit segment identification, that is, the data bit is 8 bits and the identification bit is 2 bits. First, according to the histogram distribution characteristics of the overall numerical value of the weight, the weights in the same channel are divided into a plurality of subintervals (such as a main distribution area, an extended distribution area, and an outlier distribution area), and a quantization parameter is independently configured for each subinterval to realize segmented linear quantization. Subsequently, all weights are uniformly coded to obtain a 10-bit coded fixed-point weight, wherein 8 bits are used to represent the quantized value of the weight in the segment to which it belongs, and 2 bits are used to identify the quantization segment identification to which the weight belongs.

[0034] In addition, the above embodiment uses 8 bits to represent the fixed-point value and 2 bits to represent the segment index, and the overall structure is 10 bits. In actual application, it can also be extended to other bit width structures such as 6+2 bits and 7+1 bits, to adjust the weight coding precision and compression rate according to the target hardware resource.

[0035] In step S106, the multiplication coefficient and the shift value of the encoded weight are determined based on the identification bit, and the multiplication coefficient and the shift value are used to perform multiplication and addition calculation processing on the activation value and the weight information in the data bit, to obtain the output activation value corresponding to the encoded weight.

[0036] In an embodiment, the multiplication coefficient and the shift value of the encoded weight can be determined by looking up the weight grouping information corresponding to any encoded weight, that is, by reading the 2-bit identification to select the corresponding decoding parameter, so as to realize the restoration of each weight.

[0037] The above weight quantization method based on mixed segment coding provided by the embodiment of the present application can significantly improve the quantization precision of the neural network, while reducing the quantization error without changing the compression rate.

[0038] The embodiment of the present application studies the histogram distribution of the weights of the YOLO series network and the weights of the Transformer series network, and finds that the weight distribution of most neural networks has the following characteristics:

[0039] 1. Highly concentrated around zero: The vast majority of weight values are distributed in a narrow interval centered at 0, forming a clear dense main peak, consistent with the central concentration characteristic of the Laplace distribution.

[0040] 2. Long-tail distribution structure: Although the main peak area accounts for a very high proportion, there are still a small number of large positive and negative extreme weight values extending to both ends, forming the tail of the distribution, which is a typical long-tail feature.

[0041] 3. Obvious consistency across layers: In different network structures (YOLO and Transformer) and different layers (including convolutional layers and fully connected layers), similar weight distribution patterns can be observed, indicating that this statistical characteristic is universal.

[0042] 4. Large dynamic range difference within segments: Even within a single channel or block, there are often small values and outlying large values coexisting, making it difficult to balance the main dense area and tail features with a uniform quantization precision.

[0043] 5. Insufficient sparsity but approximate sparsity: Although most weights are not zero, the number of low-amplitude weights far exceeds that of high-amplitude values, with the potential for sparse compression and segmented quantization optimization.

[0044] Based on this, referring to the specific flowchart of a weight quantization method based on mixed segmented coding as shown in Figure 2 , the embodiments of the present application also provide an implementation for determining a set of coded weights, specifically referring to (1) to (3) as follows:

[0045] (1) Through statistical analysis of floating-point weights, a data distribution histogram corresponding to the floating-point weights is obtained, and the floating-point weights are data-fitted based on a Laplace model to obtain a fitted probability density function. Then, the data distribution histogram and the probability density function are combined to obtain the Laplace distribution of the floating-point weights.

[0046] In one embodiment, referring to the schematic diagram of the neural network weight distribution as shown in Figure 3 , it can be observed that the weights overall present a symmetric distribution with a sharp peak, with the main dense area concentrated around zero, and a certain proportion of outlying weight values, forming a long-tail distribution. The probability density function fitted using the Laplace model shows that the weight distribution can be well represented as a Laplace distribution with , as parameters. This distribution characteristic indicates that most weights are concentrated in the main area near 0, but there are still non-negligible high-amplitude outliers, making it difficult to balance precision and range using a uniform scaling parameter.

[0047] (2) Based on the Laplace distribution of the floating-point weight, the floating-point weight is segmented and quantized, and the weight value of the floating-point weight is divided into a sub-interval set, that is, by using the percentile method, the weight value of the floating-point weight is divided into a main peak region, a medium sparse region and a positive interval, and the main peak region, the medium sparse region and the positive interval are determined as the sub-interval set.

[0048] In an embodiment, a segmented quantization method can be used to divide the weight into multiple sub-intervals according to the numerical interval, and independent quantization parameters are used for linear coding, so as to realize fine expression of different distribution segments, reduce quantization error and improve model accuracy. Based on the Laplace distribution, the fitting distribution function of the weight is assumed to be:

[0049]

[0050] wherein, is the distribution center (symmetry axis, usually 0); is the scale parameter, which can be obtained by maximum likelihood estimation:

[0051]

[0052] According to the cumulative distribution property of the distribution function, the percentile method is used for segmentation, specifically (a) to (c):

[0053] (a) Main peak region: contains about 99% of the weight value (in the measured YOLO, Transformer weight, more than 99% of the values are concentrated in this range). For Laplace distribution, since:

[0054]

[0055] wherein, for the Laplace distribution with scale parameter b (probability density function ), by integrating the probability density on the interval (simplified calculation by using the symmetry of the distribution), can be obtained . Since , therefore is about 99.3%.

[0056] Therefore, the threshold is selected as:

[0057]

[0058] The value density in this interval is the highest, and the weight precision has a significant impact on the model output. The minimum quantization step is used to ensure accuracy. Since segmented quantization is used, the numerical range of each segment is different, so each segment has its own independent step value. The smaller the step, the higher the accuracy, but the narrower the range of representation. The larger the step, the wider the range, but the lower the accuracy. That is, the step is the floating-point difference represented by each integer unit during quantization. The smaller the step, the more accurate it is, and the larger the step, the more rough it is. The quantization parameter is calculated as:

[0059]

[0060] (b) Medium sparse area: This segment contains about 0.6% of the weights, with an amplitude between the main area and the tail, and the threshold is selected as:

[0061]

[0062] The corresponding cumulative probability is about 99.3% to 99.96%. The data in this area is sparse, but it is not yet an abnormal outlier. Separate segmentation can prevent tail compression. The quantization parameter is calculated as:

[0063]

[0064] (c) Positive value interval: This segment contains a small amount of outlier values (about 0.04%) of weights, with the largest numerical range, and the threshold is selected as:

[0065]

[0066] Although the probability of these values is extremely low, they have a significant impact on widening the unified quantization scale. If not separated, the accuracy of the main area will be greatly diluted. The quantization parameter is calculated as:

[0067]

[0068] wherein, represents the maximum absolute value of all values in the group of weights.

[0069] (3) Map the weight values in each sub-interval to the fixed-point representation value of the data bits, and use the index corresponding to the sub-interval to construct the weight grouping information of the identification bits. Combine the fixed-point representation value with the corresponding weight grouping information to obtain the set of encoded weights.

[0070] In an embodiment, refer to Figure 4An example of 10-bit quantized weight bit partitioning is shown, according to the sub-interval to which the weight value belongs before mapping, the corresponding weight grouping information is determined, and the weight grouping information is inserted into the end of the corresponding fixed-point representation value to obtain the encoded weight set, wherein the weight values in the same sub-interval correspond to the same weight grouping information.

[0071] That is, according to the quantization parameters of the above three segments , , , the weights in the respective intervals are mapped to 8-bit fixed-point representation values, and then the 8-bit fixed-point values are left shifted by 2 bits and 2-bit segment index identifiers are inserted at the end, thereby forming the 10-bit quantized weight encoding format as shown in Figure 4 . In actual application, the number of segments can be extended to 2, 4 or more, and the segment index bit width can be adjusted to 1 bit, 3 bits, etc.

[0072] By using the above segment-based weight quantization method to encode the weights as 8-bit quantized values and 2-bit segment identifiers, the precision is improved while the storage is compact, and the following advantages are achieved:

[0073] 1. Reduce quantization error under the premise of constant compression rate: by dividing the weights into up to four typical value distribution regions, each region uses independent quantization intervals and step settings to achieve high-precision segmented quantization. The weight is identified by the 2-bit identifier at the end, which avoids the precision loss caused by uniform parameters, and is especially suitable for scenarios where the dynamic range of values in the channel is large.

[0074] 2. Each weight independently carries quantization information: compared to perTensor and perChannel quantization, which can only use one quantization parameter, this scheme allows each weight to use the corresponding quantization parameter according to its distribution segment, achieving adaptive quantization at the weight granularity and significantly improving the feature representation capability and model expressiveness in the channel.

[0075] 3. Good flexibility and adaptability: the segmentation strategy of this scheme supports configurable modeling, which can flexibly divide different quantization regions according to specific models, layer types or training statistical characteristics, adapt to different structures, balance precision, resources and versatility, and is suitable for efficient deployment of convolutional layers, fully connected layers and other network modules.

[0076] In addition, the embodiment of the present application also provides a specific calculation method of output activation value, specifically referring to (1) to (3) as follows:

[0077] (1) Fixed-point calculation of segment quantization: since the weights use a segment quantization structure, that is, each quantized weight is composed of an 8-bit main quantized value with 2bit segment index The activation value is quantized asymmetrically with 8bit, and the weight is quantized symmetrically with 8bit. The multiplication and addition calculation process of the activation value and the weight can be represented as:

[0078]

[0079] wherein, , are the quantization scales of the activation value and the output respectively; is the scale selected according to the weight segment index; , are the zero points of the activation value and the output respectively; are the quantization scales of the activation value and the output respectively, represents the activation value.

[0080] (2) Fixed-point multiplication of floating-point: the activation value and the weight information are substituted into the multiplication and addition calculation model, and the floating-point ratio item in the multiplication and addition calculation model is processed by fixed-point, so that the floating-point ratio item is converted into fixed-point form, so as to solve the multiplication and addition calculation model to obtain the output activation value. Specifically, in order to realize efficient fixed-point reasoning, the floating-point ratio item needs to be converted into fixed-point form:

[0081]

[0082] The fixed-point coefficient and the corresponding shift amount are pre-calculated for each segment, which are approximately:

[0083]

[0084] Thus, the entire reasoning calculation is transformed into the following form:

[0085]

[0086] wherein, is a fixed-point integer (usually 16-32bit), is a bit shift scaling item (which can be a constant or a lookup table value), which avoids runtime floating-point division.

[0087] (3) Hardware implementation optimization: in order to adapt to the pipeline structure, the multiplication coefficient and the shift amount of each segment can be written into the hardware lookup table (LUT) in advance in actual calculation, and the segment index Fast mapping is performed. Considering the hardware resource limit, up to 4 32-bit accumulators can be designed, and the maximum four-segment segmentation quantization is adapted, and the multiplication and addition results in each segment are independently accumulated, that is:

[0088] 1. For each weight , a group of is selected according to the 2-bit segment index thereof.

[0089] 2. Perform fixed-point multiplication: .

[0090] 3. Perform right shift operation , and accumulate the result in the corresponding segment register.

[0091] 4. When , the results of the four segment registers are added to obtain:

[0092]

[0093] wherein represents the index set belonging to the segment .

[0094] In summary, the above scheme of the present application has the following characteristics:

[0095] 1. The weight quantization coding mode based on the 8+2 bit structure represents each neural network weight in the form of 10 bits binary, wherein 8 bits represent the quantization value in the belonging segment, and 2 bits are used to identify the segment number to which the weight belongs.

[0096] 2. The quantization method based on three-segment division of weight statistical distribution, by fitting the Laplace distribution of weight data, using a fixed threshold (such as ±5b), the weight value range is divided into three subintervals of negative tail region, main dense region and positive tail region, and different quantization steps and scales are configured, in addition, the fixed ±5b threshold is also used as the segment boundary, and the boundary can also be adaptively selected according to the actual network layer statistical distribution, using methods such as percentile, histogram density threshold or mean ± standard deviation.

[0097] 3. Through the independent linear quantization parameter setting method in the segment, according to the dynamic range of each segment interval, the scale and zero-point used in the segment are independently calculated, which improves the accuracy and adaptability of each segment quantization.

[0098] 4. Through the fixed-point multiplication and addition decoding structure combined with segment index lookup table, in the inference stage, according to the 2-bit segment identifier carried by each weight, the multiplication coefficient and shift parameter of the corresponding segment are obtained through the lookup table (LUT), instead of floating-point scaling, to realize efficient fixed-point multiplication and addition calculation.

[0099] 5. The hardware optimization implementation of the segmented accumulator structure limits the maximum supported segment number to 4 and configures 4 independent accumulators for independent accumulation of the multiplication and addition results of each segment, and finally unified merging output.

[0100] 6. Based on the weight restoration mechanism matching the encoding and decoding, according to the quantization parameter set and the corresponding 8-bit effective value mapped by the 2-bit segment index, the approximate floating-point weight value is restored through fixed-point multiplication and shift calculation to participate in inference calculation.

[0101] 7. The universal segmented quantization method suitable for convolutional layer, fully connected layer and other modules in neural network supports weight encoding generation and decoding execution in the process of static quantization and quantization-aware training (QAT).

[0102] Therefore, in order to improve the quantization precision of the neural network and enhance the adaptability to the non-uniform weight distribution within the channel, each weight is encoded into a combination of 8-bit data value and 2-bit segment identifier, wherein the 8-bit is used to represent the value after local linear quantization, and the 2-bit is used to identify the data distribution segment to which the weight belongs. By modeling the actual distribution of the weights within the channel in three segments, different quantization intervals and step sizes can be used to effectively cover the main distribution area and the outlier area, taking into account the precision and range. This method breaks through the limitation of perTensor and perChannel quantization that only uniform quantization parameters can be used within the channel, so that each weight can adaptively select the most suitable quantization strategy according to its local distribution characteristics. Without significantly increasing the storage overhead, the quantization error is greatly reduced, and the robustness and expressiveness of the model under low-bit deployment are improved.

[0103] For the weight quantization method based on hybrid segmented encoding provided by the foregoing embodiments, an embodiment of the present application provides a weight quantization device based on hybrid segmented encoding, which refers to a structure schematic diagram of a weight quantization device based on hybrid segmented encoding as shown in Figure 5 The device comprises the following parts:

[0104] An information acquisition module 502 acquires floating-point weights and activation values to be processed;

[0105] A hybrid segmented encoding module 504 performs hybrid segmented encoding processing on the floating-point weights based on the Laplace distribution of the floating-point weights to obtain a set of encoded weights in a quantized weight encoding format, wherein each encoded weight in the set of encoded weights comprises data bits and identification bits;

[0106] The weight quantization module 506 determines the multiplication coefficient and the shift value of the encoded weight based on the identification bit, and performs multiplication and addition calculation processing on the activation value and the weight information in the data bit based on the multiplication coefficient and the shift value, to obtain the output activation value corresponding to the encoded weight.

[0107] The weight quantization device based on the hybrid segment coding provided by the embodiments of the present application can significantly improve the quantization precision of the neural network, while reducing the quantization error without changing the compression rate.

[0108] In one embodiment, after the step of obtaining the floating-point weight and the activation value to be processed, the information acquisition module 502 is further configured to: obtain a data distribution histogram corresponding to the floating-point weight by statistically analyzing the floating-point weight, and obtain a fitted probability density function by fitting the floating-point weight based on a Laplace model; and combine the data distribution histogram and the probability density function to obtain the Laplace distribution of the floating-point weight.

[0109] In one embodiment, when the step of performing hybrid segment coding processing on the floating-point weight based on the Laplace distribution of the floating-point weight to obtain a set of encoded weights in the quantized weight coding format is performed, the hybrid segment coding module 504 is further configured to: perform segment quantization processing on the floating-point weight based on the Laplace distribution of the floating-point weight, divide the weight values of the floating-point weight into a set of sub-intervals; map the weight values in each sub-interval to a fixed-point representation value of the data bit, and construct weight grouping information of the identification bit using the index corresponding to the sub-interval; and combine the fixed-point representation value and the corresponding weight grouping information to obtain the set of encoded weights.

[0110] In one embodiment, when the step of performing segment quantization processing on the floating-point weight to divide the weight values of the floating-point weight into a set of sub-intervals is performed, the hybrid segment coding module 504 is further configured to: divide the weight values of the floating-point weight into a main peak region, a moderately sparse region, and a positive interval by using the weight value as a threshold through the percentile method, and determine the main peak region, the moderately sparse region, and the positive interval as the set of sub-intervals.

[0111] In one embodiment, when the step of combining the fixed-point representation value and the corresponding weight grouping information to obtain the set of encoded weights is performed, the hybrid segment coding module 504 is further configured to: determine the corresponding weight grouping information according to the sub-interval to which the weight value before mapping belongs for each fixed-point representation value, and insert the weight grouping information into the end of the corresponding fixed-point representation value to obtain the set of encoded weights, wherein the weight grouping information corresponding to the weight values in the same sub-interval is the same.

[0112] In one implementation, when determining the multiplication coefficient and the shift value of the encoding weight based on the identification bit, the weight quantization module 506 is further configured to determine the multiplication coefficient and the shift value of the encoding weight by performing a table lookup on the weight grouping information corresponding to any encoding weight.

[0113] In one implementation, when performing the multiplication and addition calculation between the activation value and the weight information in the data bit to obtain the output activation value corresponding to the encoding weight, the weight quantization module 506 is further configured to: substitute the activation value and the weight information into a multiplication and addition calculation model, and perform a fixed-point processing on a floating-point scaling term in the multiplication and addition calculation model to convert the floating-point scaling term into a fixed-point form, so as to solve the multiplication and addition calculation model to obtain the output activation value.

[0114] The device provided in the embodiments of the present application has the same implementation principle and technical effects as the foregoing method embodiments. For brevity, the part not mentioned in the device embodiment part can be referred to the corresponding content in the foregoing method embodiments.

[0115] The embodiments of the present application provide a server, specifically, the server includes a processor and a storage device; the storage device stores a computer program, and the computer program performs the method of any one of the foregoing embodiments when executed by the processor.

[0116] Figure 6 A structural diagram of a server provided by the embodiments of the present application is shown in FIG. 6. The server 100 includes a processor 60, a memory 61, a bus 62 and a communication interface 63, and the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62. The processor 60 is configured to execute executable modules stored in the memory 61, such as a computer program.

[0117] The memory 61 can include a high-speed random access memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. The communication between the system network element and at least one other network element is realized through at least one communication interface 63 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0118] The bus 62 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 6 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0119] The memory 61 is configured to store a program, and the processor 60 executes the program after receiving an execution instruction. The method performed by the device for defining a flow process according to any of the foregoing embodiments of the application can be applied to the processor 60 or implemented by the processor 60.

[0120] The processor 60 can be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the foregoing method can be completed by an integrated logic circuit of hardware in the processor 60 or an instruction in the form of software. The foregoing processor 60 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), and the like; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage medium in the art. The storage medium is located in the memory 61, and the processor 60 reads information in the memory 61 and combines the hardware to complete the steps of the foregoing method.

[0121] The computer program product of the readable storage medium provided by the embodiments of the application includes a computer readable storage medium storing a program code, and the program code includes instructions for executing the method described in the foregoing method embodiments. For specific implementation, reference can be made to the foregoing method embodiments, which will not be described here.

[0122] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0123] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited to this. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments within the technical range disclosed by the present application, or make equivalent replacements to some technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for weight quantization based on hybrid segment coding, characterized in that, The method is applied to an edge computing device, and the method comprises: obtaining a floating-point weight to be processed and an activation value, wherein the floating-point weight is an original model parameter obtained in a model training process of a neural network, and the activation value is a value of an input model; based on a Laplace distribution of the floating-point weight, performing mixed segment coding processing on the floating-point weight to obtain a set of quantized weight coding formats of coded weights, wherein each coded weight in the set of coded weights comprises a data bit and an identification bit, wherein each neural network weight is represented as a combination of an 8-bit quantized value and a 2-bit segment identification through a segment structure fixed-point quantization coding mode, the data bit is 8 bits, the identification bit is 2 bits, according to the histogram distribution characteristics of the overall numerical value of the weight, the weights in the same channel are divided into a main distribution area, an extended distribution area and an outlier distribution area, a quantization parameter is independently configured for each subarea to perform segmented linear quantization, and all weights are uniformly coded to obtain 10-bit coded fixed-point weights, wherein 8 bits are used to represent the quantized value of the weight in the segment to which the weight belongs, and 2 bits are used to identify the quantization segment identification to which the weight belongs; based on the identification bit, determining a multiplication coefficient and a shift value of the coded weight, and based on the multiplication coefficient and the shift value, performing multiplication and addition calculation processing on the activation value and the weight information in the data bit to obtain an output activation value corresponding to the coded weight; wherein the step of determining the multiplication coefficient and the shift value of the coded weight based on the identification bit comprises: determining the multiplication coefficient and the shift value of the coded weight by performing table lookup processing on the weight grouping information corresponding to any coded weight; wherein the step of performing multiplication and addition calculation processing on the activation value and the weight information in the data bit to obtain an output activation value corresponding to the coded weight comprises: substituting the activation value and the weight information into a multiplication and addition calculation model, and performing fixed-point processing on a floating-point proportion item in the multiplication and addition calculation model to convert the floating-point proportion item into a fixed-point form, so as to solve the multiplication and addition calculation model to obtain the output activation value; wherein hardware optimization is performed through a segmented accumulator structure, the maximum supported segment number is limited to 4, and 4 independent accumulators are configured for independent accumulation of multiplication and addition results of each segment and unified merging output; wherein when performing hardware implementation optimization, the multiplication coefficient and the bit displacement of each segment are written into a hardware lookup table, quick mapping is performed through a segment index, and 4 32-bit accumulators are set to adapt to the segmented quantization of a maximum of four segments for adapting to the segmented quantization of a maximum of four segments, and the multiplication and addition results in the corresponding segment are independently accumulated for each segment; wherein when the bit width structure is expanded to 6+2 bits or 7+1 bits, the weight coding precision and compression rate are adjusted according to the target hardware resources.

2. The method of claim 1, wherein the weight quantization is based on hybrid segment coding. After the step of obtaining the floating-point weight to be processed and the activation value, comprising: obtaining a data distribution histogram corresponding to the floating-point weight through statistical analysis on the floating-point weight, and performing data fitting on the floating-point weight based on a Laplace model to obtain a fitted probability density function; The data distribution histogram is combined with the probability density function to obtain a Laplace distribution of the floating-point weight.

3. The method of claim 1, wherein the weight quantization is based on hybrid segment coding. The step of performing hybrid segment coding on the floating-point weight based on the Laplace distribution of the floating-point weight to obtain an encoded weight set in a quantized weight coding format, comprises: Performing segment quantization processing on the floating-point weight based on the Laplace distribution of the floating-point weight, and dividing the weight values of the floating-point weight into a sub-interval set; Mapping the weight values in each sub-interval to fixed-point representation values of the data bits, and constructing weight grouping information of the identification bits using the index identification corresponding to the sub-interval; Combining the fixed-point representation values with the corresponding weight grouping information to obtain the encoded weight set.

4. The method of claim 3, wherein the weight quantization is based on a hybrid segment coding. The step of performing segment quantization processing on the floating-point weight to divide the weight values of the floating-point weight into a sub-interval set, comprises: By using the percentile cutting method, the weight values of the floating-point weight are divided into a main peak region, a moderately sparse region and a positive value region by taking the weight numerical value as a threshold, and the main peak region, the moderately sparse region and the positive value region are determined as the sub-interval set.

5. The method of claim 3, wherein the weight quantization is based on a hybrid segment coding. The step of combining the fixed-point representation values with the corresponding weight grouping information to obtain the encoded weight set, comprises: According to the sub-interval to which the weight value of each fixed-point representation value belongs before mapping, the corresponding weight grouping information is determined, and the weight grouping information is inserted into the end of the corresponding fixed-point representation value to obtain the encoded weight set, wherein the weight grouping information corresponding to the weight values in the same sub-interval is the same.

6. A weight quantization apparatus based on hybrid segment coding, characterized by, The device is applied to an edge computing device, and the device comprises: An information acquisition module acquires floating-point weights to be processed and activation values, wherein the floating-point weights are original model parameters obtained in a model training process of a neural network, and the activation values are values input to the model; A hybrid segment coding module performs hybrid segment coding processing on the floating-point weight based on the Laplace distribution of the floating-point weight to obtain an encoded weight set in a quantized weight coding format, wherein each encoded weight in the encoded weight set comprises data bits and identification bits, wherein each neural network weight is represented as a combination of an 8-bit quantized value and a 2-bit segment identification by using a segment structure fixed-point quantization coding mode, the data bits are 8 bits, the identification bits are 2 bits, the weights in the same channel are divided into a main distribution region, an extended distribution region and an outlier distribution region according to the histogram distribution characteristics of the overall numerical values of the weights, quantization parameters are independently configured for each sub-interval to perform segment linear quantization, and all weights are uniformly coded to obtain 10-bit coded fixed-point weights, wherein the 8 bits are used to represent the quantized value of the weight in the segment to which the weight belongs, and the 2 bits are used to identify the quantization segment identification to which the weight belongs; A weight quantization module determines a multiplication coefficient and a shift value of the encoded weight based on the identification bits, and performs multiplication and addition calculation processing on the activation value and the weight information in the data bits based on the multiplication coefficient and the shift value to obtain an output activation value corresponding to the encoded weight. The step of determining the multiplication coefficient and the shift value of the encoding weight based on the identification bit comprises: determining the multiplication coefficient and the shift value of the encoding weight by performing table lookup processing on the weight grouping information corresponding to any encoding weight. The step of performing multiplication and addition calculation processing on the activation value and the weight information in the data bit to obtain the output activation value corresponding to the encoding weight comprises: substituting the activation value and the weight information into a multiplication and addition calculation model, performing fixed-point processing on a floating-point proportion item in the multiplication and addition calculation model, converting the floating-point proportion item into a fixed-point form, and performing solving processing on the multiplication and addition calculation model to obtain the output activation value. The hardware optimization is performed through a segmented accumulator structure, the maximum supported segment number is limited to 4, and four independent accumulators are configured to independently accumulate multiplication and addition results of each segment and uniformly merge and output; when the hardware implementation optimization is performed, the multiplication coefficient and the bit displacement of each segment are written into a hardware lookup table, fast mapping is performed through a segment index, four 32-bit accumulators are set in combination with hardware resource limitation to adapt to the maximum segmented quantization of four segments, multiplication and addition results in each segment are independently accumulated, when the bit width structure is expanded to 6+2 bits or 7+1 bits, the weight encoding precision and the compression rate are adjusted according to target hardware resources.

7. A server, characterized by The processor executes the computer executable instructions to implement the method in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and the computer executable instructions, when called and executed by the processor, cause the processor to implement the method in any one of claims 1 to 5.

Images