Model compression method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a model compression method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: segmenting a network weight of a target model along a predetermined axis based on a predetermined segmentation parameter, compressing at least one weight group formed after segmentation, and achieving the compression of the target model. And the model size is reduced. Meanwhile, the weight difference value in the segmented weight group is smaller than the weight difference value on the corresponding axis, so that the influence on the change of the model precision is smaller, and the precision loss of the model can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a model compression method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology and large language models (i.e. LLM, Large Language Model), model data is constantly expanding, and model parameters have reached the level of hundreds of billions or even more, which is a great challenge to the hardware of model deployment. Therefore, it is necessary to compress the model to reduce the size of the model.

[0003] However, although the existing model compression method can reduce the size of the model to a certain extent, the precision loss after compression is difficult to control. SUMMARY

[0004] Therefore, the purpose of the embodiments of the present application is to provide a model compression method, device, electronic equipment and computer readable storage medium to reduce the precision loss of the model while realizing model compression.

[0005] In a first aspect, the embodiments of the present application aim to provide a model compression method, which comprises:

[0006] obtaining a target model;

[0007] segmenting network weights of the target model along a predetermined axis based on a predetermined segmentation parameter corresponding to the predetermined axis, to determine at least one weight group;

[0008] determining a compression scale factor corresponding to each weight group according to the weights in the weight group;

[0009] compressing the corresponding weight group based on the compression scale factor, to obtain a compressed model, wherein the compressed model comprises a decompression operator corresponding to the compression.

[0010] Further, the segmentation of the network weights of the target model along the predetermined axis based on the predetermined segmentation parameter to determine at least one weight group comprises:

[0011] rearranging the network weights of the target model along the predetermined axis;

[0012] segmenting the rearranged network weights along the predetermined axis based on the predetermined segmentation parameter, to determine at least one weight group.

[0013] Further, the segmentation of the network weights of the target model along the predetermined axis based on the predetermined segmentation parameter to determine at least one weight group comprises:

[0014] grouping the network weights along the predetermined axis based on the split parameter to determine at least one weight group.

[0015] Further, the grouping the network weights along the predetermined axis based on the split parameter to determine at least one weight group comprises:

[0016] in response to the number of the remaining ungrouped weights being less than the split parameter, dividing the ungrouped weights into one weight group.

[0017] Further, the predetermined axis is determined according to a splittable axis of a model operator in the target model.

[0018] Further, the split parameter comprises a first dimension parameter and a second dimension parameter, and the grouping the network weights of the target model along the predetermined axis based on the predetermined split parameter to determine at least one weight group comprises:

[0019] grouping the network weights of the target model along a first dimension axis based on the first dimension parameter to determine at least one first dimension data;

[0020] grouping each of the first dimension data along a second dimension axis based on the second dimension parameter to determine at least one corresponding weight group.

[0021] Further, the determining the corresponding compression scale factor of each weight group based on the weights in the weight group comprises: determining a ratio of an absolute value of a weight with the largest absolute value in the weight group to a predetermined constant as the compression scale factor corresponding to the weight group, the predetermined constant being determined according to a data bit number of the compressed weight.

[0022] Further, the compressing the corresponding weight group based on the compression scale factor comprises:

[0023] performing data type conversion on the data in the weight group based on the compression scale factor.

[0024] Further, the method further comprises:

[0025] running the compressed model according to the decompression operator to determine a corresponding running result.

[0026] In a second aspect, an embodiment of the present application aims to provide a model compression device, the device comprising:

[0027] an obtaining unit configured to obtain a target model;

[0028] a splitting unit configured to split network weights of the target model along a predetermined axis based on predetermined splitting parameters corresponding to the predetermined axis, to determine at least one weight group;

[0029] a compressing unit configured to determine a compression scale factor corresponding to each weight group according to weights in the weight group, and to compress the corresponding weight group based on the compression scale factor, to obtain a compressed model, the compressed model comprising a decompression operator corresponding to the compression.

[0030] In a third aspect, an embodiment of the present application aims to provide a computer program product comprising computer programs / instructions, which, when executed by a processor, implement the method according to any one of the above.

[0031] In a fourth aspect, an embodiment of the present application aims to provide an electronic device comprising a memory and a processor, the memory being configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of the above.

[0032] In a fifth aspect, an embodiment of the present application aims to provide a computer-readable storage medium, which stores a computer program, the computer program being executed by a processor to implement the method according to any one of the above.

[0033] The technical solution of the embodiment of the present application splits network weights of a target model along a predetermined axis based on predetermined splitting parameters, and compresses at least one weight group formed after the splitting, to realize compression of the target model, thereby reducing the size of the model. At the same time, since the weight difference in the weight group after the splitting is smaller than the weight difference on the corresponding axis, the change in model precision is less affected, thereby reducing the precision loss of the model. BRIEF DESCRIPTION OF DRAWINGS

[0034] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application, taken in conjunction with the accompanying drawings, in which:

[0035] Figure 1 is a flowchart of the model compression method of the embodiment of the present application;

[0036] Figure 2 is a splitting diagram of network weights of the embodiment of the present application;

[0037] Figure 3 is a diagram of generating a weight group of the embodiment of the present application;

[0038] Figure 4 is another diagram of generating a weight group of the embodiment of the present application;

[0039] Figure 5 is a rearrangement diagram of network weights of an embodiment of the present application;

[0040] Figure 6 is a diagram of a compression process of an embodiment of the present application;

[0041] Figure 7 is a diagram of a decompression process of an embodiment of the present application;

[0042] Figure 8 is a diagram of a model compression apparatus of an embodiment of the present application;

[0043] Figure 9 is a diagram of an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0044] The present application is described below based on embodiments, but the present application is not limited to only these embodiments. In the following detailed description of the present application, some specific details are described in detail. The present application can also be fully understood without the description of these details by those skilled in the art. In order to avoid confusion of the essence of the present application, well-known methods, processes, procedures, elements and circuits are not described in detail.

[0045] In addition, it should be understood by those of ordinary skill in the art that the drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0046] Unless the context clearly requires otherwise, throughout the description, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to".

[0047] In the description of the present application, it should be understood that the terms "first", "second", and the like are used only for the purpose of description and should not be construed as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0048] Model compression is of great significance to the deployment and use of a model, and existing model compression methods include parameter pruning and quantization. In parameter pruning, a threshold is usually set, and the size of the model is reduced by pruning the weights whose absolute values are less than the threshold. This method is simple and easy to implement, but the accuracy loss of the model is large. The quantization method usually converts the parameters in the model from high precision (such as 32-bit floating point numbers) to low precision (such as 8-bit integers), thereby reducing the memory occupation and computational demand of the model, but the accuracy loss after compression is difficult to control. Therefore, the embodiment of the present application aims to provide a model compression method to reduce the accuracy loss of the model while achieving model compression.

[0049] Figure 1 is a flowchart of the model compression method of the embodiment of the present application. As shown in Figure 1 , the model compression is achieved by the following steps in the embodiment.

[0050] In step S110, the target model is obtained.

[0051] In the embodiment, the target model is a model to be compressed, and the model type can be a large language model (LLM, Large Language Model) or other types of neural network models. The target model usually includes network weights and model operators, etc. The network weights refer to the numerical values between the connections of the neurons in the model network, which are used to adjust and learn the behavior of the model, so as to effectively map and extract useful values from the input data of the model. The model operator refers to the mathematical function or operation used in the model calculation process, which defines how to transform and calculate the input data to produce the output of the model. Therefore, the network weights and the model operator are important information in the use and running process of the target model, which determines the output performance of the target model.

[0052] In step S120, the network weights of the target model are segmented along a predetermined axis based on a predetermined segmentation parameter, and at least one weight group is determined.

[0053] Since the network weights of the target model are usually represented by vector data including multiple dimensions (such as channel dimension, width dimension, length dimension, depth dimension, etc.), and different dimensions have corresponding axes, in this embodiment, the network weights of the target model are segmented along the predetermined axes based on predetermined segmentation parameters to determine at least one weight group, so as to realize model compression and reduce the accuracy loss in the model compression process based on the at least one weight group corresponding to the network weights. The segmentation parameters correspond to the predetermined axes, the predetermined axes determine the segmentation direction of the network weight segmentation data, and the segmentation parameters determine the segmentation data size of the network weights. That is, the segmentation parameters are used to represent the data granularity of the segmented weight groups.

[0054] Further, in this embodiment, the predetermined axes for network weight segmentation can be determined according to the splittable axes of the model operators in the target model. The splittable axes of a model operator refer to the dimensions (such as width, height, and number of channels of an image) that can be independently calculated in the calculation process of a model operator (such as convolution, matrix multiplication, etc.), and there is no data dependency or calculation dependency between these dimension data, so they can be split for processing, which is usually used in parallel computing, distributed computing, or operator optimization processing scenarios, aiming to improve computing efficiency or reduce the demand for computing resources.

[0055] Generally, the splittable axes of a model operator depend on the calculation logic and data structure of the model operator, and are also related to the implementation of the hardware platform and the computing framework. For example, in a convolution operator, the width and height dimensions of the input feature map are usually splittable for parallel computing, because the convolution operation is independent in these dimensions, and the axes corresponding to these dimensions can be used as splittable axes. For another example, since different hardware platforms and computing frameworks may have different parallel computing capabilities and optimization strategies, the axes suitable for the current platform and framework can be selected as the splittable axes according to the calculation needs.

[0056] It should be noted that the splittable axes of a model operator are not fixed, and they change with the changes of the input, output, and calculation logic of the model operator. Therefore, a suitable model operator segmentation strategy can be selected according to the calculation logic and data structure of the model operator or the running scenario of the target model, and the splittable axes of the model operator and the predetermined axes for network weight segmentation can be determined. Meanwhile, the number of predetermined axes can be one or multiple.

[0057] Further, after determining the predetermined axes for network weight segmentation, in this embodiment, the segmentation parameters corresponding to each axis in the predetermined axes can be selected according to the actual use scenario.

[0058] Optionally, the segmentation parameter in this embodiment is related to the hardware characteristics (such as processor performance, memory resources, etc.) of the running platform where the target model is located, that is, the segmentation parameter can be set according to the hardware characteristics of the running platform where the target model is located, so that the size of the segmentation parameter is adapted to the running platform, which is beneficial to further optimize the model compression effect.

[0059] Generally speaking, the smaller the value of the segmentation parameter is, the smaller the data granularity of the weight group is, and the higher the precision of the compressed model is, but the larger the compression calculation amount is, therefore, a suitable segmentation parameter is selected in this embodiment to segment the network weight, so as to ensure the precision of the compressed model while improving the overall compression performance of the model. Specifically, before compressing the target model, a corresponding relationship between different hardware characteristics and the segmentation parameter of the target model is established based on testing or developer experience in this embodiment. Then, when determining the segmentation parameter corresponding to the target model, the segmentation parameter corresponding to the hardware characteristics of the running platform where the target model is located is directly used as the segmentation parameter used when compressing the target model, wherein the segmentation parameter corresponding to different hardware characteristics is the segmentation parameter that makes the overall performance of the model compression good, for example, the segmentation parameter that can balance the compression precision and compression efficiency requirements at the same time.

[0060] Moreover, when the number of axes in the predetermined axis is greater than 1, the segmentation parameters of different axes in this embodiment can be the same or different. Therefore, by determining the segmentation parameter corresponding to the predetermined axis in the above manner, the model compression can be implemented subsequently while balancing the model size change and the precision loss of the model, thereby improving the use performance of the model compression method.

[0061] After determining the predetermined axis and the segmentation parameter of the network weight of the segmentation target model, the network weight is grouped according to the preset segmentation manner along the predetermined axis in this embodiment, for example, starting from the start position, end position, center position, or a randomly selected position of the network weight, the network weight is grouped based on the pre-determined segmentation parameter.

[0062] Optionally, the network weight is grouped along the predetermined axis using the segmentation parameter starting from the start position of the network weight in this embodiment to determine at least one weight group.

[0063] Furthermore, when the predetermined axes of the network weights include a first dimension axis and a second dimension axis, and the segmentation parameters include a first dimension parameter and a second dimension parameter, the first dimension parameter is used to characterize the data granularity segmented along the first dimension axis, and the second dimension parameter is used to characterize the data granularity segmented along the second dimension axis. In this embodiment, when segmenting the network weights, the network weights of the target model are first segmented along the first dimension axis based on the first dimension parameter to determine at least one first dimension data; then, each first dimension data is segmented along the second dimension axis based on the second dimension parameter to determine at least one corresponding weight reassembly.

[0064] For ease of understanding, this embodiment combines Figure 2 The segmentation process shown is explained below. Figure 2 As shown, assuming the predetermined axes of network weights A include two dimensions, the N-axis and the K-axis, with m weights on the N-axis and n weights on the K-axis, and segmentation parameters including a first-dimensional parameter and b, in this embodiment, the network weights of the target model are first segmented along the N-axis based on the first-dimensional parameter a, resulting in (m / a) first-dimensional data A1. At this point, each first-dimensional data A1 has a weight on the N-axis and n weights on the K-axis. Then, each first-dimensional data A1 is segmented along the K-axis based on the second-dimensional parameter b, resulting in (n / b) weighted reassemblies A2 corresponding to each first-dimensional data A1. Therefore, network weights A correspond to ((m / a) × (n / b)) weighted reassemblies, and each weighted reassembly A2 has a weight on the N-axis and b weights on the K-axis. Thus, by segmenting network weights A using the above method, all weighted reassemblies corresponding to the network weights are determined.

[0065] Specifically, such as Figure 2 As shown, when m=5, n=6, a=1, b=2, that is, when the network weight A has 5 columns of weights on the N-axis and 6 rows of weights on the K-axis, in this embodiment, the data is first divided along the N-axis in units of one column to obtain 5 first-dimensional data A1. Each first-dimensional data A1 has 1 weight along the N-axis and 6 weights along the K-axis. Then, each first-dimensional data A1 is divided along the K-axis in units of 2 rows, and each first-dimensional data A1 is divided into 3 weighted sets. Finally, the network weight A is divided into 15 weighted sets.

[0066] Therefore, in the embodiment, the network weights of the target model are segmented by the above method to determine at least one weight group, and then the model compression can be realized by compressing the weights in each weight group. Meanwhile, compared with the standard deviation or variance of the difference between the absolute values of the weights in the network weights when the existing quantization method directly compresses all the weights without in-axis segmentation, the standard deviation or variance of the difference between the absolute values of the weights in each segmented weight group in the embodiment is smaller, which can make the change of the model accuracy smaller when the model compression is realized based on the weight group, and thus the accuracy loss in the subsequent model compression can be reduced.

[0067] Meanwhile, considering that the number of all the weights in the network weights can be divided by the value corresponding to the segmentation parameter or cannot be divided by the value corresponding to the segmentation parameter, the weight groups generated by the segmentation of the network weights with different numbers of weights are different.

[0068] Specifically, in the case that the number of all the weights in the network weights can be divided by the value corresponding to the segmentation parameter, for example, as shown in the segmentation axis K-axis in FIG. 6, the number of all the weights in each column is 6, and when the segmentation parameter of the K-axis is 2, three weight groups are formed for each column of weights, and the number of weights in each weight group is the same as the segmentation parameter, that is, 3. Figure 2

[0069] In the case that the number of all the weights in the network weights cannot be divided by the value corresponding to the segmentation parameter, for example, as shown in the segmentation axis K-axis in FIG. 7 and FIG. 8, the number of all the weights in each column is 7, and when the segmentation parameter of the K-axis is 2, the number of all the weights in the network weights cannot be divided by the value corresponding to the segmentation parameter, that is, 2. Figure 3 Figure 4

[0070] At this time, in an optional implementation, the embodiment can choose not to segment the weights on this axis to ensure the accuracy of the model. Meanwhile, for the weights on the axis that are not segmented, the embodiment can choose not to compress the weights on the axis, or compress all the weights on the axis as a weight group, and the compression processing method is the same as that of the other segmented weight groups.

[0071] In another optional implementation, optionally, the embodiment can first group the network weights along a predetermined axis based on the segmentation parameter from the starting position of the network weights to determine at least one weight group, and then divide the ungrouped weights into a weight group in response to the number of the ungrouped weights being less than the segmentation parameter. For example, as shown in FIG. 9, the network weights are segmented along the K-axis from the starting position of the network weights, and the number of the ungrouped weights is less than the segmentation parameter, so the ungrouped weights are divided into a weight group. Figure 3 ​​​For example, as shown in the K-axis of a column of weights, after grouping the weights starting from the weight at the start position of the column according to the division parameter 2 to sequentially form the weight group 1, the weight group 2 and the weight group 3, one remaining weight is added to the weight group 3 to form the final weight group 1, the weight group 2 and the weight group 3. Thus, in the embodiment, by the above method of dividing the network weights that cannot be evenly divided by the division parameter, the data loss during weight division can be avoided, and the model compression effect is affected.

[0072] Further, in the embodiment, for the weight group with the number of weights less than the division parameter, when the number of weights in the weight group is 1, the weight can not be compressed, and the original weight is directly used for subsequent processing; or the weight group including one weight can be compressed separately.

[0073] Alternatively, in the embodiment, the ungrouped weights can be added to the determined weight group in response to the number of ungrouped weights remaining after the network weight division being less than the division parameter. At this time, the at least one weight group corresponding to the network weight division includes the weight group with the same number of weights as the division parameter and the weight group with the number of weights greater than the division parameter, and the number of weight groups with the same number of weights as the division parameter is more than the number of weight groups with the number of weights greater than the division parameter. For example, as shown in the K-axis of a column of weights, after grouping the weights starting from the weight at the start position of the column according to the division parameter 2 to sequentially form the weight group 1, the weight group 2 and the weight group 3, one remaining weight is added to the weight group 3 to form the final weight group 1, the weight group 2 and the weight group 3. Figure 4 For example, as shown in the K-axis of a column of weights, after grouping the weights starting from the weight at the start position of the column according to the division parameter 2 to sequentially form the weight group 1, the weight group 2 and the weight group 3, one remaining weight is added to the weight group 3 to form the final weight group 1, the weight group 2 and the weight group 3. Thus, in the embodiment, by the above method of dividing the network weights that cannot be evenly divided by the division parameter, the data loss during weight division can be avoided, and the model compression effect is affected.

[0074] Alternatively, in the embodiment, the ungrouped weights can be added to the determined weight group in response to the number of ungrouped weights remaining after the network weight division being less than the division parameter. At this time, the at least one weight group corresponding to the network weight division includes the weight group with the same number of weights as the division parameter and the weight group with the number of weights greater than the division parameter, and the number of weight groups with the same number of weights as the division parameter is more than the number of weight groups with the number of weights greater than the division parameter. For example, as shown in the K-axis of a column of weights, after grouping the weights starting from the weight at the start position of the column according to the division parameter 2 to sequentially form the weight group 1, the weight group 2 and the weight group 3, one remaining weight is added to the weight group 3 to form the final weight group 1, the weight group 2 and the weight group 3.

[0075] Figure 5 is a rearrangement diagram of the network weight of the embodiment of the application. As shown in Figure 5As shown, it is assumed that the network weights before and after rearrangement are A and A' respectively. The network weights A and A' before and after rearrangement both include 6 rows of weights along the K axis and 5 columns of weights along the N axis. When rearrangement is performed, in this embodiment, rearrangement can be achieved by adjusting the positions of the row weights in the network weight A. Specifically, in this embodiment, a column (such as the first column) in the network weight A can be selected as a reference for rearrangement, and the weights in the column are arranged in order of absolute value from small to large, while ensuring that the weights in each column do not change. For example, for the weight 1.1 in the first row of the first column in the network weight A, the weight 1.1 is arranged in the first column of the third row in the network weight A', and the other weights in the row of the weight 1.1 in the network weight A are arranged in the corresponding positions in the third row of the network weight A'. Thus, in this embodiment, the weights in the entire row are rearranged based on the weight size in a certain column to achieve rearrangement of the network weights, which can make the variance of the absolute value difference of the weights in each weight group in the rearranged network weights smaller, the accuracy loss of the model compression smaller, and the accuracy of the compressed model higher.

[0076] Alternatively, in this embodiment, the row weights in the network weights can also be rearranged multiple times in different ways, and the variance of the absolute value difference of the weights in each weight group in the rearranged network weights is determined after each rearrangement, and finally the network weight with the smallest variance is selected as the final rearranged network weight. Thus, in this embodiment, by providing different rearrangement methods, the rearrangement of the network weights can be more convenient.

[0077] Meanwhile, when rearranging the weights in the network weights, in this embodiment, the weights before and after rearrangement can be identified according to the attribute information of each weight, so as to ensure that each weight can accurately participate in the calculation of the corresponding model operator and ensure the accuracy of the model output.

[0078] Alternatively, in this embodiment, the weights before and after rearrangement can be identified according to the position identifier of each weight, wherein the position identifier is used to represent the position of the weight in the network weights. For example, the position identifier 23 of the weight number A23 represents that the position of the weight number in the network weights before rearrangement is the second row and the third column, and when an operator needs to call the weight number A23, the weight number to be used is determined according to the position identifier 23.

[0079] Further, since the number of weights and the weight distribution in different dimensions of the network weights before and after rearrangement do not change, i.e., the total number of weights and the number of weights in each dimension of the network weights do not change, the positions of each weight before and after rearrangement can also be numbered in the embodiment, and the correspondence between the position numbers before and after rearrangement can be saved. When the weights need to be called for calculation, the weights before rearrangement can be queried and used according to the correspondence between the position numbers; or the position numbers of the weights after rearrangement can be passed to the model operator (such as the previous model operator) related to the weights, so that the model operator calls the correct weights, thereby ensuring the accuracy of model calculation while improving the model precision of the compressed model.

[0080] Meanwhile, when the network weights after rearrangement are segmented, the network weight segmentation method described above can be used for segmentation in the embodiment, which will not be described here.

[0081] In step S130, the compression scale factor corresponding to each weight group is determined according to the weights in the weight group.

[0082] In the embodiment, after the network weights are segmented and at least one weight group is determined, the compression of the network weights is realized by compressing each weight group, and the compression of the target model is realized. The compression degree of each weight group is represented by the corresponding compression scale factor.

[0083] Optionally, since the weights in the network weights are usually represented by floating-point data, and floating-point data requires more memory space than integer data, the compression of the network weights is realized by converting the data type of the weights in each weight group in the embodiment. Meanwhile, the compression scale factor of each weight group can be determined according to the weights in the weight group; or it can be determined based on a pre-set weight threshold, which can be set according to the weights in the weight group and the actual compression requirement. Therefore, by providing different methods to determine the compression scale factor corresponding to each weight group, the determination of the compression scale factor of each weight group is more convenient in the embodiment.

[0084] Further, to reduce the change of weights before and after compression and further improve the precision of the compressed model, the compression scale factor in the embodiment is determined according to the weight values in the weight group, for example, the minimum-maximum of the corresponding absolute values of the weights in the weight group is determined in a quantization manner.

[0085] Specifically, in this embodiment, the ratio of the absolute value of the weight with the largest absolute value in the weight reassembly to a predetermined constant is determined as the scaling factor corresponding to the weight reassembly. The predetermined constant is determined based on the number of bits in the compressed weight data. For example, when converting high-order floating-point weights (such as float32 or float16) in the network weights to 8-bit integer data (i.e., int8), the highest bit in the 8 bits is the sign bit indicating whether the weight value is positive or negative, and the remaining 7 bits are used to represent the weight value, corresponding to an integer value range of 0 to 127 (i.e., 2^32). 7 -1), at this point, the corresponding predetermined constant is set to 127. Furthermore, when determining the compression ratio factor for the weight group, it is determined based on the following formula:

[0086] scale = max(|(Group) min )|,|(Group max )|) / 127

[0087] Where scale is the compression ratio factor of the weight group, |(Group min | represents the absolute value of the smallest weight in the weight reorganization; | (Group max | is the absolute value of the largest weight in the weight reorganization.

[0088] For example, for a weighted group including weights 1.1 and 3.2, the corresponding compression factor is 3.2 / 127; for a weighted group including weights -2.3 and 1.0, the corresponding compression factor is 2.3 / 127; and for a weighted group including weights -3.8 and 7.0, the corresponding compression factor is 7 / 127. It should be understood that when the maximum absolute value of the weights corresponding to different weighted groups is different, the compression factor corresponding to different weighted groups will also be different. Therefore, in this embodiment, determining the compression factor corresponding to each weighted group using the above method can reduce the number of weights compressed within a group while making the compression factor of each weight group more closely match the weight distribution within the corresponding weighted group, which is beneficial for further improving the accuracy of the compression model.

[0089] It should be noted that when determining the compression ratio factor based on a pre-set weight threshold, the compression ratio factor in this embodiment is the ratio of the preset threshold to a predetermined constant, and the weight threshold is greater than the absolute value of the weight with the largest absolute value in the weighted reassembly, to ensure that all weights in the compressed weighted reassembly fall within the predetermined numerical range. Simultaneously, to facilitate subsequent decompression of the target model, the weight threshold and the compressed network weights need to be encapsulated together to successfully complete the decompression.

[0090] In step S140, the corresponding weighted recombination is compressed based on each compression ratio factor to obtain a compression model, which includes the corresponding decompression operator.

[0091] In this embodiment, after determining the compression scale factor corresponding to each weight group, the corresponding weight group is compressed based on the compression scale factor. The compression process is the data type conversion process. Compressing the corresponding weight group based on the compression scale factor includes converting the data in the weight group to a data type based on the compression scale factor, and determining the weight group after data type conversion as the compressed weight group.

[0092] For example, assuming that the weight group before compression includes weight x and weight y, both weight x and weight y are floating-point data, and |x|<|y|, the compressed weight is int8 type data, then the compression scale factor corresponding to the weight group is |y| / 127, and the compressed weight x becomes |x| / (|y| / 127), and the compressed weight y becomes 127. Thus, in this embodiment, the corresponding weight group is compressed based on the compression scale factor corresponding to each weight group, which can make the weight distribution in the weight group before and after compression more close, which is beneficial to reduce the error before and after weight compression, and further improve the precision of the compressed model.

[0093] Alternatively, considering that the ratio of the absolute value of the weight in the weight group to the preset constant may not be an integer, in this embodiment, after determining the ratio of the absolute value of the weight to the preset constant, the ratio can be processed based on rounding, rounding or other data processing methods, so that the data type of the compressed weight is consistent with the target data type, and the target data type is the data type of the weight after compression which is set in advance.

[0094] Specifically, in this embodiment, the ratio of the absolute value of the weight to the preset constant is processed by rounding, for example, assuming that the weight group before compression includes weight 1.1 and 3.2, based on the above method, the compression scale factor corresponding to the weight group can be determined as 3.2 / 127, and when the weight group is compressed based on the compression scale factor, the weight 1.1 becomes 1.1 / (3.2 / 127)=43.65625, and the weight 3.2 becomes 127. At this time, the value of the compressed weight 1.1 needs to be rounded to obtain the value 44, so that the corresponding weight of the compressed weight group includes 44 and 127, that is, the data compression process of converting the data in the weight group from floating-point data to integer data is realized.

[0095] Further, after compressing each weight group based on the corresponding compression scale factor of the weight group, the embodiment will arrange the compressed weight groups based on the position of the weight groups in the network weight and the relative position relationship between the weight groups, and then obtain the compressed network weight and the compression model corresponding to the compressed network weight. Thus, by compressing each weight group based on the corresponding compression scale factor, the embodiment can compress the network weight of the target model, and then compress the target model and obtain the compression model formed after the compression of the target model. Moreover, since the embodiment compresses the weight groups determined after the segmentation of the network weight based on the compression scale factor corresponding to the weight group, the embodiment can further reduce the influence of compression on the change of model accuracy and the loss of model accuracy, and improve the accuracy of the compressed model while realizing model compression.

[0096] Optionally, to facilitate the compression of the target model and the subsequent use of the compressed compression model, the model compression method in the embodiment can be packaged into a corresponding compression tool, and the compression tool can be developed based on an open source framework, such as PPQ. PPQ is a relatively mature network quantization tool, and its target hardware includes all common back-end hardware today. Users can select a suitable back-end target according to their own target deployment hardware, or select a suitable compression strategy (including segmentation parameters of network weight) according to the characteristics of the back-end hardware to be deployed. In this way, the model can be automatically compressed to reduce the loss of model accuracy, and the model compression method can be compatible with different back-end platforms to improve the usability and generalization of the model method.

[0097] Meanwhile, since the model compression method in the embodiment is for network weight, which is usually stored in a constant form and does not differ due to different models or scenes. Therefore, the model compression method and the model compression tool in the embodiment can compress various models, including traditional neural network models and large models (including large language models), to optimize the size of the model, and the model compression method has higher adaptability to models and a higher application range.

[0098] Meanwhile, the compression tool in the embodiment supports the weight separation mode of the large language model, which can facilitate the conversion of the model between various frameworks and prevent the situation that some model frameworks do not support the storage type of the model.

[0099] Further, the model compression tool in the embodiment can adopt a one-key operation scheme, and the user only needs to provide the network model and the model input to realize model compression without performing too many steps and mastering complicated compression or quantization knowledge, thereby reducing the learning cost of the user. Moreover, the compression tool in the embodiment can provide an interface for comparing the model running results, so that the user can optimize the model according to the comparison result of the model running results before and after model compression, thereby improving the convenience of model optimization.

[0100] Further, after determining the corresponding compressed model by compressing the target model, the embodiment encapsulates the corresponding decompression operator and the compressed network weight in the compressed model, so as to subsequently decompress the compressed network weight according to the decompression operator, and then perform corresponding calculation by using the network weight obtained after decompression.

[0101] Correspondingly, the model compression method in the embodiment further includes: running the compressed model according to the decompression operator, and determining the corresponding running result. Thus, in the embodiment, the compressed network weight is decompressed by the decompression operator, so that the compressed model can run the model operator by using the network weight obtained after decompression, and process the received input data to determine the corresponding calculation result.

[0102] Optionally, to further improve the accuracy of the compressed model, the embodiment compares and analyzes the calculation results determined by processing the same input based on the target model before compression and the compressed model after compression, so that the user can adjust the granularity of the segmentation parameter of the network weight of the target model according to the comparison and analysis result, and then optimize the model compression process. Thus, in the embodiment, the model input to output can be realized in all aspects through a complete process, and the model compression process is optimized by the running results of the same input data before and after compression of the model, the interaction logic between the user and the model layer data is hidden, the user does not need to perform too many operations and master complicated compression knowledge, thereby reducing the additional cost of model compression, improving the accuracy of the compressed model, and making the model optimization process convenient and efficient.

[0103] The technical scheme of the embodiment segments the network weight of the target model along the predetermined axis based on the predetermined segmentation parameter, and compresses at least one weight group formed after segmentation, thereby realizing compression of the target model and reducing the model size. Meanwhile, compared with the segmentation of all weights in the existing quantization method, the weight difference in the segmented weight group is smaller than the weight difference on the corresponding axis, and the influence on the model accuracy is smaller, thereby reducing the accuracy loss of the model.

[0104] In this embodiment, in order to more carefully understand the model compression method in this embodiment, the network weight compression process of a specific model operator will be described.

[0105] Figure 6 is a schematic diagram of the compression process of an embodiment of the application. As shown in Figure 6 , the network weight compression process of the matmul operator in this embodiment is described, and of course the compression method in this embodiment can also be applied to the network weight compression of the gemm, dense, etc. model operator.

[0106] As shown in Figure 6 , in the matmul operator, the right input of the operator is generally the network weight, and the left input is the activation value, which is used to trigger the compression operation. When the network weight includes two dimensions of N axis and K axis, and the K axis corresponds to the channel axis (i.e. channel axis), the corresponding network weight is represented as [K, N].

[0107] When the network weight is segmented, the N axis is already segmented into multiple column vectors (the column vector corresponds to the first dimension data described above), at this time each channel is grouped along the K axis direction according to the predetermined segmentation parameter, and multiple weight groups are obtained. It should be understood that when the axis number of the network weight is greater than 2, the last but one axis (i.e. the predetermined divisible axis) can be operated in this embodiment. When grouping, first determine whether the number of weights on the K axis can be divided by the predetermined segmentation parameter group, and when it cannot be divided, the operation on the weight constant on the K axis is not performed. When it can be divided, the K axis is split into [K / group, group] two axes (the order of the two axes can be exchanged), where K / group represents the number of weight groups corresponding to a column vector, and group represents the number of weights in a weight group, and then the shape of the network weight is split from [K, N] to [K / group, group, N]. Then, the maximum value and the minimum value are queried along the group axis respectively, so that the absolute value of the minimum weight in each weight group (Group min ) and the absolute value of the maximum weight (Group max ) are obtained.

[0108] Finally, the maximum value of the two weight absolute values is taken, and the maximum value is divided by a predetermined constant to obtain a compression scale factor scale. It should be noted that the division calculation here is the division of a floating point number by a predetermined constant, and the shape of the final compression scale factor is [K / group, 1, N].

[0109] After determining the compression scale factors corresponding to each weight group, the weight numbers in the corresponding weight group are compressed based on the compression scale factors, so as to obtain the compressed weight group corresponding to each weight group. After the compression of each weight group is completed and the compressed weight group corresponding to the network weight is determined, the compressed weight group is integrated, so as to obtain the network weight with the compressed weight as integer data. At this time, the shape of the compressed network weight is [K, N].

[0110] Further, after the compression of the network weight is realized based on the foregoing method and the compressed network weight corresponding to the target model is determined, the compressed network weight and the decompression operator corresponding to the compression are packaged at the same time in the embodiment, so as to obtain the compressed compression model.

[0111] Figure 7 is a schematic diagram of the decompression process of the embodiment of the application. As shown in Figure 7 , the decompression operator in the embodiment includes a reshape operator, a data conversion operator (i.e., a cast operator) and a multiplication operator (i.e., a mul operator). When the compressed model is decompressed, the decompression operation is activated by the activation value, and the reshape operator is inserted before the network weight input, so that the shape of the network weight is changed from [K, N] to [K / group, group, N]. Then, the cast operator is inserted to convert the data type of the weight in the network weight from int8 to float16. Finally, the mul operator is inserted to multiply the weight of [K / group, group, N] by the compression scale factor of [K / group, 1, N], and the decompression is completed.

[0112] It should be understood that in the embodiment, after the input data is input into the compressed model, the network weight required to be used is decompressed during the processing of the input data by the compressed model, so as to reduce the waiting time of the model running for the decompression of the network weight, and thus improve the running efficiency of the model.

[0113] Figure 8 is a schematic diagram of the model compression device of the embodiment of the application. As shown in Figure 8 , the model compression device in the embodiment includes an acquisition unit 1, a segmentation unit 2 and a compression unit 3. The acquisition unit 1 is used to acquire a target model. The segmentation unit 2 is used to segment the network weight of the target model along a predetermined axis based on a pre-determined segmentation parameter, so as to determine at least one weight group. The compression unit 3 is used to determine the compression scale factor of each weight group, and to compress the corresponding weight group based on each compression scale factor, so as to obtain a compressed model. At the same time, the segmentation parameter is related to the hardware characteristics of the running platform where the target model is located, the segmentation parameter corresponds to the predetermined axis, and the compressed model includes a decompression operator corresponding to the compression.

[0114] Optionally, the predetermined axis in this embodiment is determined according to a separable axis of a model operator in the target model. The compression unit 3 is further configured to determine a ratio of an absolute value of a weight pair corresponding to a weight with the largest absolute value in the weight group to a predetermined constant as a compression scale factor corresponding to the weight group, and the predetermined constant is determined according to a data bit number of the compressed weight.

[0115] Optionally, the segmentation unit 2 in this embodiment is further configured to rearrange the network weights of the target model along the predetermined axis, and segment the rearranged network weights along the predetermined axis based on the predetermined segmentation parameter to determine the at least one weight group.

[0116] Further, when the network weights of the target model are segmented along the predetermined axis based on the predetermined segmentation parameter to determine the at least one weight group, the segmentation unit 2 is configured to group the network weights along the predetermined axis using the segmentation parameter from a starting position of the network weights to determine the at least one weight group. Optionally, the segmentation unit 2 is further configured to divide the ungrouped weights into one weight group in response to a number of the ungrouped weights remaining in the network weights being less than the segmentation parameter.

[0117] Specifically, when the segmentation parameter includes a first dimension parameter and a second dimension parameter, the segmentation unit 2 in this embodiment is specifically configured to segment the network weights of the target model along a first dimension axis based on the first dimension parameter to determine at least one first dimension data, and segment each first dimension data along a second dimension axis based on the second dimension parameter to determine at least one corresponding weight group.

[0118] Optionally, when the compression unit 3 compresses the corresponding weight group based on the compression scale factor to obtain the compressed model, the compression unit 3 in this embodiment is specifically configured to perform data type conversion on the data in the weight group based on the compression scale factor.

[0119] Optionally, as shown in Figure 8 the model compression apparatus in this embodiment further includes a running unit 4 configured to run the compressed model according to the decompression operator to determine a corresponding running result.

[0120] The technical solution of this embodiment realizes compression of the target model by segmenting the network weights of the target model along a predetermined axis based on a predetermined segmentation parameter through the segmentation unit and compressing at least one weight group formed after segmentation through the compression unit after the obtaining unit obtains the target model, thereby reducing the model size. Meanwhile, compared with the segmentation of all weights in the existing quantization method, since the weight difference in the segmented weight group is smaller than the weight difference on the corresponding axis, the change in model accuracy is smaller, thereby reducing the accuracy loss of the model.

[0121] Figure 9is a schematic diagram of an electronic device according to an embodiment of the present application. As shown in Figure 9 Figure 9 The electronic device shown is a general address query device, which comprises a general computer hardware structure, which at least comprises a processor 91 and a memory 92. The processor 91 and the memory 92 are connected through a bus 93. The memory 92 is adapted to store instructions or programs executable by the processor 91. The processor 91 can be a stand-alone microprocessor, or can be a set of one or more microprocessors. Thus, the processor 91 performs the processing of data and the control of other devices by executing the instructions stored in the memory 92, thereby implementing the method flow of the embodiments of the present application as described above. The bus 93 connects the above-mentioned components together, and connects the above-mentioned components to a display controller 94 and a display device, and to input / output (I / O) devices 95. The input / output (I / O) devices 95 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a body sense input device, a printer, and other devices known in the art. Typically, the input / output devices 95 are connected with the system through an input / output (I / O) controller 99.

[0122] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device (apparatus) or computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer- readable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) containing computer usable program code.

[0123] The present application is described with reference to flowcharts according to the methods, devices (apparatuses) and computer program products of the embodiments of the present application. It should be understood that each flow in the flowcharts can be implemented by computer program instructions.

[0124] These computer program instructions can be stored in a computer readable memory capable of directing a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction apparatus, which implements the flow Figure 1 specified in one flow or multiple flows.

[0125] These computer program instructions can also be provided to a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device for implementing the functions specified in one flow or multiple flows. Figure 1

[0126] ​​Another embodiment of the present application relates to a non-volatile storage medium for storing a computer readable program for causing a computer to execute some or all of the above-mentioned method embodiments.

[0127] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned method embodiments can be completed by a program stored in a storage medium, including a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage media that can store program codes.

[0128] The above-mentioned is only the preferred embodiment of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A model compression method, characterized in that, The method comprises: obtaining a target model; segmenting network weights of the target model along a predetermined axis based on a predetermined segmentation parameter, the segmentation parameter corresponding to the predetermined axis, to determine at least one weight group; determining a compression scale factor corresponding to each of the weight groups according to weights in the weight group; compressing the weight group corresponding to each of the compression scale factors to obtain a compressed model, the compressed model comprising a decompression operator corresponding to the compression.

2. The method of claim 1, wherein, The segmentation of the network weights of the target model along the predetermined axis based on the predetermined segmentation parameter to determine at least one weight group comprises: rearranging the network weights of the target model along the predetermined axis; segmenting the rearranged network weights along the predetermined axis based on the predetermined segmentation parameter to determine at least one weight group.

3. The method of claim 1, wherein, The segmentation of the network weights of the target model along the predetermined axis based on the predetermined segmentation parameter to determine at least one weight group comprises: grouping the network weights along the predetermined axis using the segmentation parameter from a starting position of the network weights to determine at least one weight group.

4. The method of claim 3, wherein, The grouping of the network weights along the predetermined axis based on the segmentation parameter to determine at least one weight group comprises: in response to the number of ungrouped weights remaining in the network weights being less than the segmentation parameter, dividing the ungrouped weights into one weight group.

5. The method according to any one of claims 1 to 4, characterized in that, The predetermined axis is determined according to a splittable axis of a model operator in the target model.

6. The method according to any one of claims 1 to 5, characterized in that, The segmentation parameter comprises a first dimension parameter and a second dimension parameter, and the segmentation of the network weights of the target model along the predetermined axis based on the predetermined segmentation parameter to determine at least one weight group comprises: segmenting the network weights of the target model along a first dimension axis based on the first dimension parameter to determine at least one first dimension data; segmenting each of the first dimension data along a second dimension axis based on the second dimension parameter to determine at least one weight group corresponding to the first dimension data.

7. The method according to any one of claims 1 to 6, characterized in that, The determination of the compression scale factor corresponding to each of the weight groups according to the weights in the weight group comprises: determining a ratio of an absolute value of a weight with the largest absolute value in the weight group to a predetermined constant as the compression scale factor corresponding to the weight group, the predetermined constant being determined according to a data bit number of the compressed weight.

8. The method according to any one of claims 1 to 7, characterized in that, The compression of the weight group corresponding to each of the compression scale factors comprises: performing data type conversion on data in the weight group based on the compression scale factor.

9. The method according to any one of claims 1 to 8, characterized in that, The method further comprises: running the compressed model according to the decompression operator to determine a corresponding running result.

10. A model compression apparatus, comprising: The apparatus comprises: an obtaining unit configured to obtain a target model; a segmentation unit configured to segment network weights of the target model along a predetermined axis based on a predetermined segmentation parameter, the segmentation parameter corresponding to the predetermined axis, to determine at least one weight group; a compression unit configured to determine a compression scale factor corresponding to each of the weight groups, and compress the weight group corresponding to each of the compression scale factors to obtain a compressed model, the compressed model comprising a decompression operator corresponding to the compression.

11. A computer program product, characterised in that, The computer program product comprises computer programs / instructions which, when executed by a processor, implement the method of any one of claims 1-9.

12. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any one of claims 1-9.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium has stored therein a computer program which, when executed by a processor, implements the method of any one of claims 1-9.

Images