Model compression method and related device

Abstract

The present application relates to the field of AI, and particularly relates to a model compression method and a related device. The method comprises: during the training of a target model, a model compression apparatus acquiring model checkpoints of the target model at a plurality of moments, wherein the model checkpoint at a first moment comprises a weight parameter and an optimizer momentum parameter of the target model at the first moment, and the plurality of moments comprise the first moment; the model compression apparatus determining a residual between each weight parameter at the plurality of moments and a corresponding reference weight parameter, so as to obtain residuals of the weight parameters at the plurality of moments; and the model compression apparatus performing joint compression on the residual of the weight parameter at each moment among the weight parameters at the plurality of moments and an optimizer momentum parameter at each moment, so as to obtain a compression result for the plurality of moments. The solution of the present application facilitates an increase in the overall compression ratio.

Description

Model compression method and related device

[0001] This application claims priority to the Chinese Patent Application No. 202410708565.9, filed on May 31, 2024, and entitled "Model Compression Method and Related Device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI), and in particular, to a model compression method and related device. BACKGROUND

[0003] Large model training needs to occupy a large storage space to store the model training state for retraining when the training is interrupted or the model deteriorates. The parameter storage space of a single checkpoint of the current mainstream large model reaches several TB. Therefore, it is crucial to reduce the storage space occupation and transmission bandwidth.

[0004] The current main compression method is to compress the weight file of the model, which aims to reduce the storage space size of the model weight file. Specifically, pruning, quantization, encoding and other compression operations are performed on the weight value, and the final trained model weight is compressed. Since the weight of the model is compressed in the compression process, the overall compression ratio is low. SUMMARY

[0005] The embodiments of the present application provide a model compression method and related device, which is beneficial to improve the overall compression ratio of the model.

[0006] In a first aspect, the embodiments of the present application provide a model compression method. The method is applied to a model compression device.

[0007] In the process of training the target model, the model compression device obtains the model checkpoints of the target model at multiple time points. The model checkpoint at the first time point includes the weight parameter and the optimizer momentum parameter of the target model at the first time point. The multiple time points include the first time point. The model compression device determines the residual error between the weight parameter at each time point and the corresponding reference weight parameter to obtain the residual error of the weight parameter at each time point. The model compression device jointly compresses the residual error of the weight parameter at each time point and the optimizer momentum parameter at each time point to obtain the compression result at each time point.

[0008] It can be seen that, during model compression, not only the weight values but also the optimizer momentum parameters are considered, improving the overall compression ratio; and each time point of the model checkpoint during training (including the weight values and the optimizer momentum at each time point) is compressed, that is, when the weight values are compressed, the training parameters such as the optimizer momentum are also compressed synchronously, thereby ensuring the recovery of training from any checkpoint. The residual of the weight value is compressed instead of the weight value itself, which is beneficial to further improve the compression ratio.

[0009] In combination with the first aspect, in a possible implementation, the model compression apparatus jointly compresses the residual of the weight value parameter at each time point and the optimizer momentum parameter at each time point to obtain the compression results at the plurality of time points, including:

[0010] The model compression apparatus determines the first pruning threshold of each layer of the target model at the first time point and the second pruning threshold of the optimizer momentum parameter at the first time point based on the optimizer momentum parameter at the first time point; the model compression apparatus compresses the residual of the weight value parameter at the first time point based on the first pruning threshold of each layer at the first time point to obtain the compression result of the weight value parameter at the first time point; and the model compression apparatus compresses the compression result of the weight value parameter at the first time point and the second pruning threshold to obtain the compression result of the optimizer momentum parameter at the first time point; wherein the compression results at the plurality of time points include the compression result at the first time point, and the compression result at the first time point includes the compression result of the weight value parameter at the first time point and the compression result of the optimizer momentum parameter at the first time point.

[0011] The joint compression of the weight value parameter and the optimizer momentum parameter is realized in the above manner, which is beneficial to improve the overall compression ratio.

[0012] In combination with the first aspect, in a possible implementation, the optimizer momentum parameter includes a plurality of second elements, the compression result of the weight value parameter at the first time point includes a plurality of third elements, and the compression result of the optimizer momentum parameter at the first time point includes a plurality of fourth elements.

[0013] When the ith first element is greater than the first target pruning threshold, the ith third element is the same as the ith first element; when the ith first element is not greater than the first target pruning threshold, the ith third element is 0, and the ith first element corresponds to the ith third element; the first target pruning threshold is determined based on the second moment of the corresponding optimizer momentum parameter of the layer to which the ith first element belongs, a hyperparameter, and the median of the residual of the layer to which the ith first element belongs; and the first target pruning threshold is the first pruning threshold of the layer to which the ith first element belongs.

[0014] The fourth element is the same as the second element when the i th second element is greater than the second target pruning threshold and the i th third element is the same as the i th first element; the fourth element is 0 when the i th second element is not greater than the second target pruning threshold or the i th third element is 0; the i th second element corresponds to the i th fourth element, and the i th fourth element corresponds to the i th third element; the second target pruning threshold is determined based on the first moment of the corresponding optimizer momentum parameter of the layer to which the i th first element belongs at the first moment and a hyperparameter; and the second target pruning threshold is a second pruning threshold of the optimizer momentum parameter corresponding to the layer to which the i th first element belongs.

[0015] The joint compression of the weight parameter and the optimizer momentum parameter is implemented in the above manner, which is beneficial to improving the overall compression ratio; and the pruning manner adopted for the weight and the optimizer during compression is a joint pruning manner, that is, the optimizer momentum parameter is considered when pruning the weight, and the weight is considered when pruning the optimizer momentum parameter. Compared with a separate pruning manner, the joint pruning manner has higher result accuracy, which is beneficial to further improving the overall compression ratio.

[0016] With reference to the first aspect, in a possible implementation manner, the reference weight parameter corresponding to the weight parameter at the first moment is a weight parameter at a second moment, and the multiple moments include the first moment and the second moment, and the second moment is before the first moment.

[0017] With reference to the first aspect, in a possible implementation manner, the reference weight parameter corresponding to the weight parameter at the first moment is determined according to a first residual and a reference weight parameter corresponding to a weight parameter at a second moment, the first residual is obtained by decompressing a compression result at the second moment, the multiple moments include the first moment and the second moment, and the second moment is before the first moment.

[0018] It can be seen that, in the above manner, the model compression device always saves a latest reference weight parameter during the training process, the information amount of the residual determined by using the reference weight parameter is small, and thus it is beneficial to improve the compression ratio.

[0019] With reference to the first aspect, in a possible implementation manner, the method of the embodiment further includes:

[0020] The model compression device performs quantization and / or encoding processing on the compression result at each moment of the multiple moments.

[0021] By performing quantization and / or encoding processing on the compression result, it is beneficial to further improve the compression ratio.

[0022] In a second aspect, an embodiment of the present application provides a model compression device. The model compression device comprises an obtaining unit, a determining unit, a compression unit and a post-processing unit.

[0023] The obtaining unit is configured to obtain model checkpoints of a target model at multiple time points in a process of training the target model, wherein the model checkpoint at a first time point comprises weight parameters and optimizer momentum parameters of the target model at the first time point; and the multiple time points comprise the first time point.

[0024] The determining unit is configured to determine residuals between the weight parameters at the multiple time points and corresponding reference weight parameters, to obtain the residuals of the weight parameters at the multiple time points.

[0025] The compression unit is configured to jointly compress the residual of the weight parameters at each time point among the weight parameters at the multiple time points and the optimizer momentum parameters at each time point, to obtain compression results at the multiple time points.

[0026] In combination with the first aspect, in a possible implementation, the compression unit is specifically configured to:

[0027] determine a first pruning threshold of each layer of the target model at the first time point and a second pruning threshold of the optimizer momentum parameters at the first time point based on the optimizer momentum parameters at the first time point; compress the residual of the weight parameters at the first time point based on the first pruning threshold of each layer at the first time point, to obtain a compression result of the weight parameters at the first time point; and compress the compression result of the weight parameters at the first time point and the second pruning threshold, to obtain a compression result of the optimizer momentum parameters at the first time point; wherein the compression results at the multiple time points comprise the compression result at the first time point, and the compression result at the first time point comprises the compression result of the weight parameters at the first time point and the compression result of the optimizer momentum parameters at the first time point.

[0028] In combination with the first aspect, in a possible implementation, the optimizer momentum parameters comprise a plurality of second elements, the compression result of the weight parameters at the first time point comprises a plurality of third elements, and the compression result of the optimizer momentum parameters at the first time point comprises a plurality of fourth elements.

[0029] when the ith first element is greater than a first target pruning threshold, the ith third element is the same as the ith first element; when the ith first element is not greater than the first target pruning threshold, the ith third element is 0, and the ith first element corresponds to the ith third element; the first target pruning threshold is determined based on a second moment of the corresponding optimizer momentum parameter of the layer to which the ith first element belongs, a hyperparameter and a median of residuals of the layer to which the ith first element belongs; and the first target pruning threshold is the first pruning threshold of the layer to which the ith first element belongs.

[0030] The fourth element is the same as the second element when the i th second element is greater than the second target pruning threshold and the i th third element is the same as the i th first element; the fourth element is 0 when the i th second element is not greater than the second target pruning threshold or the i th third element is 0; the i th second element corresponds to the i th fourth element, and the i th fourth element corresponds to the i th third element; the second target pruning threshold is determined based on the first moment of the corresponding optimizer momentum parameter of the layer to which the i th first element belongs at the first moment and a hyperparameter; and the second target pruning threshold is a second pruning threshold of the optimizer momentum parameter corresponding to the layer to which the i th first element belongs.

[0031] With reference to the first aspect, in a possible implementation, the reference weight parameter corresponding to the weight parameter at the first moment is a weight parameter at a second moment, the multiple moments include the first moment and the second moment, and the second moment is before the first moment.

[0032] With reference to the first aspect, in a possible implementation, the reference weight parameter corresponding to the weight parameter at the first moment is determined based on a first residual and a reference weight parameter corresponding to a weight parameter at a second moment, the first residual is obtained by decompressing a compression result at the second moment, the multiple moments include the first moment and the second moment, and the second moment is before the first moment.

[0033] With reference to the first aspect, in a possible implementation, the model compression apparatus further includes:

[0034] The post-processing unit is configured to quantize and / or encode each compression result in the multiple compression results.

[0035] In a third aspect, an embodiment of the present application provides a model compression apparatus, including a processor and a memory. The memory is configured to store program code. The processor is configured to invoke the program code stored in the memory to execute the method provided in the first aspect or any possible implementation of the first aspect.

[0036] In a fourth aspect, an embodiment of the present application provides a computer storage medium, including computer instructions. When the computer instructions run on an electronic device, the electronic device executes the method provided in any possible implementation of the first aspect.

[0037] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, the computer executes the method provided in any possible implementation of the first aspect.

[0038] It can be understood that the model compression apparatus provided in the second aspect or the third aspect is used to execute the method provided in any of the first aspects, and the computer storage medium provided in the fourth aspect and the computer program product provided in the fifth aspect are used to implement the method provided in any of the first aspects. Therefore, the beneficial effects achieved thereby can refer to the beneficial effects in the corresponding method, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0039] FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0040] FIG. 2 is a schematic diagram of a model compression method provided by an embodiment of the present application;

[0041] FIG. 3 is a schematic diagram of a specific process of model compression provided by an embodiment of the present application;

[0042] FIG. 4 is a schematic diagram of a structure of a model compression apparatus provided by an embodiment of the present application;

[0043] FIG. 5 is a schematic diagram of another model compression apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION

[0044] The terms “first”, “second”, “third”, and “fourth” and the like in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order.

[0045] “Multiple” refers to two or more. “And / or” describes the association relationship of the associated objects, indicating that there are three relationships, for example, A and / or B means that there are three cases of A alone, A and B together, and B alone. The character “ / ” generally represents that the associated objects before and after it are in an “or” relationship.

[0046] Embodiments of the present application will be described below with reference to the accompanying drawings.

[0047] Referring to FIG. 1, FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application. As shown in FIG. 1, the system architecture includes a first device 101 and a plurality of second devices 102.

[0048] The first device 101 can be a server, such as a cloud server, a distributed server, a rack server, a blade server, a tower server, etc. The second device 102 can be a server, a terminal device, etc., and the terminal device can be a smart phone, a smart watch, a smart bracelet, a desktop computer, a notebook computer, a tablet, etc.

[0049] In one scenario, the first device 101 is configured to train a model for the second device 102. During the training of the model, the first device 101 obtains model checkpoints of the target model at multiple time points, the model checkpoint at the first time point including the weight parameters and the optimizer momentum parameters of the target model at the first time point; the multiple time points include the first time point; the model compression apparatus determines the residuals between the weight parameters at the multiple time points and the corresponding reference weight parameters, to obtain the residuals of the weight parameters at the multiple time points; the model compression apparatus jointly compresses the residuals of the weight parameters at each time point and the optimizer momentum parameters at each time point, to obtain compression results at the multiple time points; and the compression results at the multiple time points are sent to the second device 102. Based on the compression results at the multiple time points, the second device 102 can obtain the weight parameters of the trained model. In this way, the amount of data transmission can be reduced, and the transmission overhead is reduced.

[0050] It should be understood that the first device 101 can train a model for multiple second devices 102 respectively. For the target model of each second device 102, the first device 101 can process in the above manner.

[0051] In the distributed training scenario, during the training process, the second device 102 obtains model checkpoints of the target model at multiple time points, the model checkpoint at the first time point including the weight parameters and the optimizer momentum parameters of the target model at the first time point; the multiple time points include the first time point; the model compression apparatus determines the residuals between the weight parameters at the multiple time points and the corresponding reference weight parameters, to obtain the residuals of the weight parameters at the multiple time points; the model compression apparatus jointly compresses the residuals of the weight parameters at each time point and the optimizer momentum parameters at each time point, to obtain compression results at the multiple time points; and the compression results at the multiple time points are sent to the first device 101. Based on the compression results at the multiple time points, the first device 101 can obtain the weight parameters of the current training; and the first device 101 adjusts the parameters of the model based on the weight parameters of the current training of each second device 102. In this way, the amount of data transmission can be reduced, and the transmission overhead is reduced.

[0052] As can be seen, during the model compression, not only the weight values are considered, but also the optimizer momentum parameters are considered, the overall compression ratio is improved, and the transmission resource overhead is reduced when the model is transmitted; the residuals of the weight values are compressed instead of the weight values themselves, which is beneficial to improve the compression ratio, and the transmission resource overhead is further reduced when the model is transmitted.

[0053] The implementation process of the present application will be described in detail below.

[0054] Referring to FIG. 2, FIG. 2 is a flowchart of a model compression method provided by an embodiment of the present application. As shown in FIG. 2, the method includes:

[0055] S201, in the process of training the target model, the model compression device obtains model checkpoints of the target model at multiple time points, wherein the model checkpoint at the first time point includes the weight parameters and the optimizer momentum parameters of the target model at the first time point.

[0056] It should be noted that the scheme of the present application is applied to the training process of the model. In the training process of the model, the weight parameters and / or the optimizer momentum parameters of the model are changing. The optimizer momentum parameters are used to represent the update amount and update direction of the weight parameters.

[0057] Here, the target model can be implemented based on a neural network, such as a convolutional neural network, a recurrent neural network, etc. Of course, the target model here can also be implemented in other ways, which are not limited here.

[0058] S202, the model compression device determines the residual error between the weight parameters at multiple time points and the corresponding reference weight parameters to obtain the residual error of the weight parameters at multiple time points.

[0059] In one possible implementation, the reference weight parameters corresponding to the weight parameters at the first time point are the weight parameters at the second time point, the multiple time points include the first time point and the second time point, and the second time point is before the first time point.

[0060] In one example, assuming that the training time required to obtain the target model by training the initial model is 5h, the reference weight parameters corresponding to the weight parameters at the 1st hour can be the weight parameters at the 0th hour, i.e. the weight parameters of the initial model, the reference weight parameters corresponding to the weight parameters at the 2nd hour are the weight parameters at the 0th hour, or the weight parameters at the 1st hour; the reference weight parameters corresponding to the weight parameters at the 3rd hour are the weight parameters at the 1st hour, or the weight parameters at the 1st hour, or the weight parameters at the 2nd hour; similarly, the reference weight parameters corresponding to the weight parameters at other time points can be determined in the above manner.

[0061] In another example, the reference weight parameters corresponding to the weight parameters at the first time point are determined by the first residual error and the reference weight parameters corresponding to the weight parameters at the second time point, the first residual error is obtained by decompressing the compression result at the second time point, the multiple time points include the first time point and the second time point, and the second time point is before the first time point.

[0062] It should be noted that when the first time point is the first time point in the multiple time points, the weight parameters at the second time point are the weight parameters of the initial model.

[0063] Assuming that the training time required for training the target model from the initial model is 5s, the reference weight parameter corresponding to the weight parameter of the 1th hour can be the weight parameter of the 0th hour, i.e., the weight parameter of the initial model; the reference weight parameter corresponding to the weight parameter of the 2th hour is determined based on the first residual and the reference weight parameter of the 1th hour, the first residual being obtained by decompressing the compression result of the residual of the 1th hour; the reference weight parameter corresponding to the weight parameter of the 3th hour is determined based on the second residual and the reference weight parameter of the 2th hour, the second residual being obtained by decompressing the compression result of the residual of the 2th hour; similarly, the reference weight parameter corresponding to the weight parameter of other time can be determined in the above manner.

[0064] Further, after obtaining the compression results of the residuals of multiple time points and the compression results of the optimizer momentum parameters of multiple time points, the model compression device deletes the reference weight parameters corresponding to the weight parameters of multiple time points.

[0065] In this way, the model compression device saves a latest reference weight parameter corresponding to a weight parameter, and further reduces the information amount of the residual determined based on the latest reference weight parameter corresponding to the weight parameter.

[0066] S203, the model compression device jointly compresses the residual of each weight parameter of multiple time points and the optimizer momentum parameter of each time point to obtain the compression result of multiple time points.

[0067] Specifically, the model compression device determines the first pruning threshold of each layer of the target model and the second pruning threshold of the optimizer momentum parameter of the first time point based on the optimizer momentum parameter of the first time point; the model compression device compresses the residual of the weight parameter of the first time point based on the first pruning threshold of each layer to obtain the compression result of the weight parameter of the first time point; the model compression device compresses based on the compression result of the weight parameter of the first time point and the second pruning threshold to obtain the compression result of the optimizer momentum parameter of the first time point; wherein the compression result of multiple time points includes the compression result of the first time point, and the compression result of the first time point includes the compression result of the weight parameter of the first time point and the compression result of the optimizer momentum parameter of the first time point.

[0068] It should be noted that the target model can include one or more layers, such as convolutional layers, fully connected layers, pooling layers, etc., the weight parameters of the target model include the weight parameters of each layer of the target model, and the weight parameters of the target model correspond to the residual.

[0069] For the model checkpoint at the first time step, the model compression device determines a first pruning threshold and a second pruning threshold for each layer of the target model. The first pruning threshold is used to prune the residuals, and the second pruning threshold is used to prune the optimizer momentum parameters. The first pruning threshold for the j-th layer of the target model is determined based on the median residual of the j-th layer, the second moment of the optimizer momentum parameters at the first time step, and the hyperparameters. The second pruning threshold is determined based on the first moment of the optimizer momentum parameters at the first time step and the hyperparameters. The second target pruning threshold is the second pruning threshold for the optimizer momentum parameters corresponding to the layer to which the i-th second element belongs.

[0070] In one example, the first pruning threshold of the j-th layer can be expressed as:

[0071] in, Let median(W) be the first pruning threshold of the j-th layer. j ) represents the median value of the residuals in the j-th layer, m t Let be the second moment of the optimizer momentum parameter, and ∝ be the hyperparameter.

[0072] In one example, the second pruning threshold of the optimizer momentum parameter corresponding to the j-th layer can be expressed as:

[0073] in, The second pruning threshold is the optimizer momentum parameter corresponding to the j-th layer, β is the second hyperparameter, and mean(v t ) represents the average value of the first moment of the optimizer momentum parameter corresponding to the j-th layer.

[0074] The model compression device prunes the residuals of the j-th layer based on the first pruning threshold of the j-th layer. Specifically, the residuals of the j-th layer include multiple first elements, and the compression result of the j-th layer residuals includes multiple third elements. When the ith first element is greater than the first pruning threshold of the j-th layer, the ith third element is the same as the ith first element; when the ith first element is not greater than the first pruning threshold of the j-th layer, the ith third element is 0. The ith third element corresponds to the ith first element.

[0075] The model compression apparatus prunes the optimizer momentum parameter corresponding to the jth layer based on the second pruning threshold of the optimizer momentum parameter corresponding to the jth layer. Specifically, the optimizer momentum parameter corresponding to the jth layer includes a plurality of second elements, and the compression result of the optimizer momentum parameter corresponding to the jth layer includes a plurality of fourth elements. When the ith second element is greater than the second pruning threshold of the optimizer momentum parameter corresponding to the jth layer, and the ith third element is the same as the ith first element, the ith second element is the same as the ith fourth element. When the ith second element is not greater than the second pruning threshold of the optimizer momentum parameter corresponding to the jth layer, or the ith third element is 0, the ith fourth element is 0. The ith second element corresponds to the ith third element, and the ith third element corresponds to the ith fourth element.

[0076] It should be pointed out here that the size of the jth layer residual is the same as the size of the optimizer momentum parameter corresponding to the jth layer.

[0077] In an optional implementation, the model compression apparatus quantizes and / or encodes the compression result of each time to obtain a quantized result / encoded result of each time. By quantizing and / or encoding the compression result of each time, the compression rate can be further improved.

[0078] Here, the quantization can be uniform quantization based on clustering, non-uniform quantization based on clustering, or other quantization manners, which are not limited here.

[0079] It should be understood that if the model compression apparatus quantizes and / or encodes the compression result, the first residual can be obtained by the model compression apparatus performing dequantization and / or decoding on the quantized result / encoded result of the second time, and then performing decompression processing.

[0080] It is pointed out here that after the training is completed, the model compression device deletes the reference weight parameters corresponding to each time point weight parameter, only includes the compression results, quantization results or encoding results of the residual of each time point weight parameter and the optimizer momentum parameter, and the weight initialization random seed, and the model compression device can obtain the initial model of the target model based on the weight initialization random seed. The model compression device obtains the residual of the first weight parameter based on the compression results, quantization results or encoding results of the residual of the first weight parameter, and obtains the first weight parameter based on the residual of the first weight parameter and the weight parameter of the initial model. The model compression device obtains the first optimizer momentum parameter based on the compression results, quantization results or encoding results of the first optimizer momentum parameter. The model compression device obtains the residual of the second weight parameter based on the compression results, quantization results or encoding results of the residual of the second weight parameter, and obtains the second weight parameter based on the residual of the second weight parameter and the first weight parameter. The model compression device obtains the second optimizer momentum parameter based on the compression results, quantization results or encoding results of the second optimizer momentum parameter. In this way, the model compression device can obtain the weight parameter and the optimizer momentum parameter at each time point, that is, the model compression device can restore the weight parameter and the optimizer momentum parameter at each time point, and further ensure that the training can be restored from any checkpoint.

[0081] In a specific embodiment, the compression process of the model checkpoint at time t is specifically described in combination with FIG. 3. The model checkpoint at time t includes the weight parameter at time t and the optimizer momentum parameter at time t. As shown in FIG. 3, the model compression device obtains the residual at time t based on the weight parameter at time t-1 and the weight parameter at time t. The model compression device jointly compresses and encodes the residual at time t and the optimizer momentum parameter at time t to obtain the compression results of the weight parameter at time t and the compression results of the optimizer momentum parameter at time t. As shown in FIG. 3, for the elements in the residual at time t, the model compression device sets the elements lower than 1.8 to 0, and keeps the elements not lower than 1.8 unchanged; wherein 1.8 can be regarded as a first pruning threshold, which is determined in combination with the second moment m2 of the optimizer momentum parameter at time t, the hyperparameter, etc.; the size of the residual at time t is the same as the size of the optimizer momentum parameter at time t, the elements in the residual at time t correspond to the elements in the optimizer momentum parameter at time t, and for the elements in the optimizer momentum parameter at time t, the model compression device keeps the elements not lower than 0.8 and corresponding to the elements not equal to 0 in the compression results of the weight parameter at time t unchanged, and sets the elements lower than 0.8 and corresponding to the elements equal to 0 in the compression results of the weight parameter at time t to 0. It can be seen that the pruning of the optimizer momentum parameter at time t takes into account the pruning results of the residual at time t, that is, W t t . ​

[0082] Optionally, the model compression device obtains the compression result of the weight parameter at the time t and the compression result of the optimizer momentum parameter at the time t, and then performs clustering-based non-uniform quantization on the compression result of the weight parameter at the time t and the compression result of the optimizer momentum parameter at the time t, and then encodes the quantization result.

[0083] It can be seen that, in the model compression, not only the weight value is considered, but also the optimizer momentum parameter is considered, so that the overall compression ratio is improved; and each time point of the model checkpoint (including the weight value and the optimizer momentum at each time point) in the training process is compressed, that is, when the weight value is compressed, the training parameters such as the optimizer momentum are synchronously compressed, so that the training can be restored from any checkpoint. The residual of the weight value is compressed instead of the weight value itself, which is beneficial to further improve the compression ratio. By quantizing and / or encoding the compression result, the compression ratio can be further improved.

[0084] It should be noted that the scheme of the present application can be extended to the joint compression of the weight and the training parameter of the model, and is not limited to the joint compression of the weight and the optimizer momentum parameter of the model.

[0085] Referring to FIG. 4, a structural schematic diagram of a model compression device provided by an embodiment of the present application is shown. As shown in FIG. 4, the model compression device 400 includes:

[0086] The acquisition unit 401 is configured to acquire the model checkpoints of the target model at a plurality of time points in the process of training the target model, and the model checkpoint at the first time point includes the weight parameter and the optimizer momentum parameter of the target model at the first time point; the plurality of time points include the first time point;

[0087] The determination unit 402 is configured to determine the residual between the weight parameter at each time point and the corresponding reference weight parameter, to obtain the residual of the weight parameter at each time point;

[0088] The compression unit 403 is configured to jointly compress the residual of the weight parameter at each time point and the optimizer momentum parameter at each time point, to obtain the compression result at the plurality of time points.

[0089] In one possible implementation, the compression unit 403 is specifically configured to:

[0090] determining a first pruning threshold of each layer of the target model at the first time and a second pruning threshold of the optimizer momentum parameter at the first time based on the optimizer momentum parameter at the first time; compressing a residual of the weight parameter at the first time based on the first pruning threshold of each layer at the first time to obtain a compression result of the weight parameter at the first time; and compressing based on the compression result of the weight parameter at the first time and the second pruning threshold to obtain a compression result of the optimizer momentum parameter at the first time; wherein the compression results of the plurality of times include the compression result of the first time, and the compression result of the first time includes the compression result of the weight parameter at the first time and the compression result of the optimizer momentum parameter at the first time.

[0091] In one possible implementation, the optimizer momentum parameter includes a plurality of second elements, the compression result of the weight parameter at the first time includes a plurality of third elements, and the compression result of the optimizer momentum parameter at the first time includes a plurality of fourth elements.

[0092] When the ith first element is greater than the first target pruning threshold, the ith third element is the same as the ith first element; when the ith first element is not greater than the first target pruning threshold, the ith third element is 0, and the ith first element corresponds to the ith third element; the first target pruning threshold is determined based on the second moment of the corresponding optimizer momentum parameter of the layer to which the ith first element belongs, the hyperparameter, and the median of the residual of the layer to which the ith first element belongs at the first time; and the first target pruning threshold is the first pruning threshold of the layer to which the ith first element belongs.

[0093] When the ith second element is greater than the second target pruning threshold and the ith third element is the same as the ith first element, the ith fourth element is the same as the ith second element; when the ith second element is not greater than the second target pruning threshold or the ith third element is 0, the ith fourth element is 0; the ith second element corresponds to the ith fourth element, and the ith fourth element corresponds to the ith third element; and the second target pruning threshold is determined based on the first moment of the corresponding optimizer momentum parameter of the layer to which the ith first element belongs and the hyperparameter at the first time; and the second target pruning threshold is the second pruning threshold of the corresponding optimizer momentum parameter of the layer to which the ith first element belongs.

[0094] In one possible implementation, the reference weight parameter corresponding to the weight parameter at the first time is a weight parameter at a second time, the plurality of times include the first time and the second time, and the second time is before the first time.

[0095] In a possible implementation, the reference weight parameter corresponding to the weight parameter at the first time point is determined according to a first residual and the reference weight parameter corresponding to the weight parameter at the second time point, the first residual is obtained by decompressing the compression result at the second time point, the plurality of time points include the first time point and the second time point, and the second time point is before the first time point.

[0096] In a possible implementation, the model compression apparatus 400 further includes:

[0097] The post-processing unit 404 is configured to perform quantization and / or encoding processing on the compression result at each time point in the plurality of time points.

[0098] It should be noted that, the specific function implementation of the model compression apparatus 400 can refer to the related description of the above model compression method, for example, the obtaining unit 401 is configured to perform the related content of S201, the determining unit 402 is configured to perform the related content of S202, and the compression unit 403 and the post-processing unit 404 are configured to perform the related content of S203. The units or modules in the model compression apparatus 400 can be combined into one or several other units or modules respectively or totally, or some of the units or modules can be further split into a plurality of units or modules with smaller functions to constitute, which can realize the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The above units or modules are divided based on logical functions, and in actual application, the function of one unit (or module) is realized by a plurality of units (or modules), or the functions of a plurality of units (or modules) are realized by one unit (or module).

[0099] Based on the description of the above method embodiments and related device embodiments, referring to FIG. 5, the present embodiment further provides a structural schematic diagram of a model compression apparatus 500. The model compression apparatus 500 shown in FIG. 5 includes a memory 501, a processor 502, a communication interface 503 and a bus 504. The memory 501, the processor 502 and the communication interface 503 are communicatively connected with each other through the bus 504.

[0100] Optionally, the memory 501 is a ROM, a static storage device, a dynamic storage device or a RAM.

[0101] The memory 501 can store a program, and when the program stored in the memory 501 is executed by the processor 502, the processor 502 and the communication interface 503 are configured to perform each step of the model compression method of the embodiment shown in FIG. 2.

[0102] The processor 502 can be a general purpose CPU, a microprocessor, an application specific integrated circuit (ASIC), a GPU, or one or more integrated circuits that are configured to perform the methods described in the embodiments of FIG. 2.

[0103] The processor 502 can also be an integrated circuit chip that has the ability to process signals. In the implementation process, the various steps of the traffic scheduling of the present application can be completed through the integrated logic circuit of hardware in the processor 502 or the instructions in the form of software. Alternatively, the processor 502 is a general purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general purpose processor is a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor. The optional software modules are located in the random access memory, the flash memory, the read only memory, the programmable read only memory or the electrically erasable programmable memory, the register or other mature storage mediums in the art. The storage medium is located in the memory 501, and the processor 502 reads the information in the memory 501, and combines the hardware to complete the functions required by the units included in the model compression device 500 in the embodiments of the present application, or executes the model compression method of the method embodiments of the present application.

[0104] The communication interface 503 uses a transceiver related device such as but not limited to a transceiver to realize the communication between the model compression device 500 and other devices or communication networks.

[0105] The bus 504 can include a path for transmitting information between the various components (e.g., the memory 501, the processor 502, the communication interface 503) of the model compression device 500.

[0106] It should be noted that although the model compression device 500 shown in FIG. 5 only shows the memory, the processor, the communication interface, in the specific implementation process, those skilled in the art should understand that the model compression device 500 also includes other devices necessary for normal operation. At the same time, according to the specific needs, those skilled in the art should understand that the model compression device 500 can also include hardware devices that realize other additional functions. In addition, those skilled in the art should understand that the model compression device 500 can also only include the devices necessary for the embodiments of the present application, and does not have to include all the devices shown in FIG. 5.

[0107] The embodiment of the present application further provides a chip, which comprises a processor and a data interface, and the processor reads instructions stored on a memory through the data interface to implement the model compression method of the embodiment of the present application.

[0108] Optionally, as an implementation manner, the chip further comprises a memory, and the memory stores instructions, and the processor is used for executing the instructions stored on the memory, and when the instructions are executed, the processor is used for executing the model compression method.

[0109] The embodiment of the present application further provides a computer readable storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in any one of the methods.

[0110] The embodiment of the present application further provides a computer program product comprising instructions, and when the computer program product is executed on a computer or a processor, the computer or the processor executes one or more steps in any one of the methods.

[0111] Those skilled in the art will appreciate that the functions described with respect to the various illustrative logical blocks, modules and algorithm steps described in this specification can be implemented by hardware, software, firmware or any combination thereof. If implemented in software, the functions described with respect to the various illustrative logical blocks, modules and steps described in this specification can be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., based on a communication protocol. In this manner, computer-readable media generally can correspond to (1) tangible computer- readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this specification. A computer program product can include a computer-readable medium.

[0112] By way of example, and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code means in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0113] Instructions can be executed by one or more processors, such as one or more DSPs, general purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor," as used herein can refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functions described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0114] In several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, and method can be implemented in other ways. For example, the division of the units is only a logical function division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. Alternatively, the coupling or direct coupling or communication connection between the units shown or discussed is indirect coupling or communication connection through some interfaces, devices, or units, such as electrical, mechanical, or other forms.

[0115] Optionally, the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., located in one place, or distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0116] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part can be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, the computer program instructions produce all or part of the processes or functions according to the embodiments of the present application.

[0117] The above is only a specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto, any change or replacement within the technical scope disclosed by the embodiments of the present application should be covered within the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

1. A model compression method, characterized in that, The method includes: During the training of the target model, model checkpoints of the target model at multiple time points are obtained. The model checkpoint at the first time point includes the weight parameters and optimizer momentum parameters of the target model at the first time point; the multiple time points include the first time point. The residuals between the weight parameters at the plurality of time points and the corresponding reference weight parameters are determined to obtain the residuals of the weight parameters at the plurality of time points. The residuals of the weight parameters at each time step and the optimizer momentum parameters at each time step are jointly compressed to obtain the compression result for the multiple time steps.

2. The method according to claim 1, characterized in that, The step of jointly compressing the residuals of the weight parameters at each time step and the optimizer momentum parameters at each time step to obtain the compression result at the multiple time steps includes: Based on the optimizer momentum parameters at the first time step, determine the first pruning threshold and the second pruning threshold of the optimizer momentum parameters at the first time step for each layer of the target model at the first time step. The residuals of the weight parameters at the first time are compressed based on the first pruning threshold of each layer at the first time to obtain the compressed result of the weight parameters at the first time. Compression is performed based on the compression result of the weight parameters at the first time step and the second pruning threshold to obtain the compression result of the optimizer momentum parameters at the first time step. The compression results at the multiple time points include the compression results at the first time point, which in turn include the compression results of the weight parameters and the optimization momentum parameters at the first time point.

3. The method according to claim 2, characterized in that, The residual includes multiple first elements, the optimizer momentum parameter includes multiple second elements, the compression result of the weight parameter at the first time step includes multiple third elements, and the compression result of the optimizer momentum parameter at the first time step includes multiple fourth elements. When the i-th first element is greater than the first target pruning threshold, the i-th third element is the same as the i-th first element; when the i-th first element is not greater than the first target pruning threshold, the i-th third element is 0, and the i-th first element corresponds to the i-th third element; the first target pruning threshold is determined based on the median of the second moment, hyperparameter, and residual of the layer to which the i-th first element belongs at the first time; the first target pruning threshold is the first pruning threshold of the layer to which the i-th first element belongs; When the i-th second element is greater than the second target pruning threshold, and the i-th third element is the same as the i-th first element, the i-th fourth element is the same as the i-th second element; when the i-th second element is not greater than the second target pruning threshold, or when the i-th third element is 0, the i-th fourth element is 0; the i-th second element corresponds to the i-th fourth element, and the i-th fourth element corresponds to the i-th third element; the second target pruning threshold is determined based on the first moment of the optimizer momentum parameter of the layer to which the i-th first element belongs at the first time and the hyperparameter; the second target pruning threshold is the second pruning threshold of the optimizer momentum parameter of the layer to which the i-th first element belongs.

4. The method according to any one of claims 1-3, characterized in that, The reference weight parameter corresponding to the weight parameter at the first moment is the weight parameter at the second moment. The plurality of moments includes the first moment and the second moment, and the second moment is before the first moment.

5. The method according to any one of claims 1-3, characterized in that, The reference weight parameter corresponding to the weight parameter at the first time step is determined by the reference weight parameter corresponding to the weight parameter at the second time step. The first residual is obtained by decompressing the compression result at the second time step. The plurality of times step includes the first time step and the second time step, and the second time step is before the first time step.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The compression result at each of the multiple time points is quantized and / or encoded.

7. A model compression device, characterized in that, The method includes: The acquisition unit is used to acquire model checkpoints of the target model at multiple time points during the training process of the target model. The model checkpoint at the first time point includes the weight parameters and optimizer momentum parameters of the target model at the first time point. The multiple time points include the first time point. A determining unit is used to determine the residuals between the weight parameters at the plurality of times and the corresponding reference weight parameters, so as to obtain the residuals of the weight parameters at the plurality of times; A compression unit is used to jointly compress the residuals of the weight parameters at each time step and the optimizer momentum parameters at each time step among the weight parameters at the multiple time steps, so as to obtain the compression result at the multiple time steps.

8. The apparatus according to claim 7, characterized in that, The compression unit is specifically used for: Based on the optimizer momentum parameters at the first time step, determine the first pruning threshold and the second pruning threshold of the optimizer momentum parameters at the first time step for each layer of the target model at the first time step. The residuals of the weight parameters at the first time are compressed based on the first pruning threshold of each layer at the first time to obtain the compressed result of the weight parameters at the first time. Compression is performed based on the compression result of the weight parameters at the first time step and the second pruning threshold to obtain the compression result of the optimizer momentum parameters at the first time step. The compression results at the multiple time points include the compression results at the first time point, which in turn include the compression results of the weight parameters and the optimization momentum parameters at the first time point.

9. The apparatus according to claim 8, characterized in that, The residual includes multiple first elements, the optimizer momentum parameter includes multiple second elements, the compression result of the weight parameter at the first time step includes multiple third elements, and the compression result of the optimizer momentum parameter at the first time step includes multiple fourth elements. When the i-th first element is greater than the first target pruning threshold, the i-th third element is the same as the i-th first element; when the i-th first element is not greater than the first target pruning threshold, the i-th third element is 0, and the i-th first element corresponds to the i-th third element; the first target pruning threshold is determined based on the median of the second moment, hyperparameter, and residual of the layer to which the i-th first element belongs at the first time; the first target pruning threshold is the first pruning threshold of the layer to which the i-th first element belongs; When the i-th second element is greater than the second target pruning threshold, and the i-th third element is the same as the i-th first element, the i-th fourth element is the same as the i-th second element; when the i-th second element is not greater than the second target pruning threshold, or when the i-th third element is 0, the i-th fourth element is 0; the i-th second element corresponds to the i-th fourth element, and the i-th fourth element corresponds to the i-th third element; the second target pruning threshold is determined based on the first moment of the optimizer momentum parameter of the layer to which the i-th first element belongs at the first time and the hyperparameter; The second target pruning threshold is the second pruning threshold of the optimizer momentum parameter corresponding to the layer to which the i-th first element belongs.

10. The apparatus according to any one of claims 7-9, characterized in that, The reference weight parameter corresponding to the weight parameter at the first moment is the weight parameter at the second moment. The plurality of moments includes the first moment and the second moment, and the second moment is before the first moment.

11. The apparatus according to any one of claims 7-10, characterized in that, The reference weight parameter corresponding to the weight parameter at the first time step is determined by the reference weight parameter corresponding to the weight parameter at the second time step. The first residual is obtained by decompressing the compression result at the second time step. The plurality of times step includes the first time step and the second time step, and the second time step is before the first time step.

12. The apparatus according to any one of claims 7-11, characterized in that, The device further includes: The post-processing unit is used to quantize and / or encode the compression result at each of the multiple compression times.

13. A model compression device, characterized in that, The method includes a processor and a memory, wherein the memory is used to store program code, and the processor is used to execute the program code to implement the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

15. A computer program product that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-6.

Images