Model compression method and apparatus, data processing method and apparatus, and storage medium and chip

Abstract

A model compression method and apparatus, a data processing method and apparatus, and a storage medium and a chip, which relate to the technical field of computers. The model compression method comprises: acquiring a first model; performing pruning processing on the first model according to historical output information and first label information of the first model, so as to obtain a second model, wherein the first label information is ground-truth label information of the first model; and performing self-distillation processing on the second model according to historical output information and second label information of the second model, so as to obtain a third model, wherein the second label information is ground-truth label information of the second model. By means of the method, the number of model parameters of a network model can be reduced on the premise of ensuring that the network model has good performance, thereby realizing the compression of the network model.

Description

Model compression method, data processing method, device, storage medium and chip

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 27, 2023, with application number "202310927731.X" and application name "Model Compression Method, Data Processing Method, Device, Storage Medium and Chip", all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a model compression method, a data processing method, a device, a storage medium, and a chip. Background Art

[0003] Deep neural network models usually contain a large number of parameters to ensure performance, but as the scale of network models continues to grow, the number of parameters of the network models gradually increases, which increases the computational and storage costs of the network models and increases the cost of using the network models.

[0004] Summary of the Invention

[0005] This application aims to solve at least one of the technical problems existing in the prior art or related art.

[0006] To this end, the first aspect of the present application is to propose a model compression method, which includes: obtaining a first model; pruning the first model based on the historical output information and first annotation information of the first model to obtain a second model, and the first annotation information is the true annotation information of the first model; self-distilling the second model based on the historical output information and second annotation information of the second model to obtain a third model, and the second annotation information is the true annotation information of the second model.

[0007] Through the above-mentioned model compression method, during the network model compression process, parameter pruning and self-distillation are combined to compress the first model, reducing the number of model parameters, thereby reducing the model's computational and storage costs, and reducing the model's usage cost. At the same time, the third model obtained after compression can still have good performance, improving the model compression effect. In addition, by combining the model's hard label information and historical output information, pruning and self-distilling the compressed model alleviate the degradation problem of the pruned model's self-distillation on the complete dataset, and solve the problem that the pruned model has difficulty in quickly learning knowledge from "hard labels", reducing the model's training time and improving the speed and efficiency of model compression.

[0008] The second aspect of the present application is to propose a data processing method, which includes: obtaining a data processing model; compressing the data processing model according to the model compression method of the first aspect; inputting the target data to be processed into the compressed data processing model; processing the target data through the compressed data processing model to obtain the processed target data.

[0009] The third aspect of the present application is to propose a model compression device, which includes: an acquisition unit for acquiring a first model; a processing unit for pruning the first model based on the historical output information of the first model and the first annotation information to obtain a second model, wherein the first annotation information is the true annotation information of the first model; the processing unit is also used to self-distill the second model based on the historical output information of the second model and the second annotation information to obtain a third model, wherein the second annotation information is the true annotation information of the second model.

[0010] The fourth aspect of the present application is to propose a data processing device, which includes: an acquisition unit for acquiring a data processing model; a processing unit for compressing the data processing model according to the model compression method of the first aspect; the processing unit is also used to input the target data to be processed into the compressed data processing model; the processing unit is also used to process the target data through the compressed data processing model to obtain the processed target data.

[0011] The fifth aspect of the present application is to propose a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the model compression method of the first aspect are implemented, or when the program or instruction is executed by a processor, the steps of the data processing method of the second aspect are implemented.

[0012] The sixth aspect of the present application is to propose a computer program product, which implements the steps of the model compression method of the first aspect when the computer program is executed by a processor, or implements the steps of the data processing method of the second aspect when the computer program is executed by a processor.

[0013] The seventh aspect of the present application is to propose a chip, which includes a program or instruction, and when the chip is running, it is used to implement the steps of the model compression method of the first aspect, or, when the chip is running, it is used to implement the steps of the data processing method of the second aspect.

[0014] Additional aspects and advantages of the present application will become apparent in the following description or may be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0016] FIG1 is a schematic diagram showing a flow chart of a model compression method according to an embodiment of the present application;

[0017] FIG2 shows one of the principle diagrams of the model compression method according to an embodiment of the present application;

[0018] FIG3 shows a second schematic diagram of the model compression method according to an embodiment of the present application;

[0019] FIG4 is a schematic diagram showing a flow chart of a data processing method according to an embodiment of the present application;

[0020] FIG5 shows one structural block diagram of a model compression device according to an embodiment of the present application;

[0021] FIG6 shows a second structural block diagram of the model compression device according to an embodiment of the present application;

[0022] FIG7 shows a structural block diagram of an electronic device according to an embodiment of the present application;

[0023] FIG8 shows a structural block diagram of a data processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to more clearly understand the above-mentioned objects, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features therein can be combined with each other in the absence of conflict.

[0025] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present application is not limited to the specific embodiments disclosed below.

[0026] 1 to 8 , the model compression method, data processing method, device, storage medium, and chip provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0027] In one embodiment of the present application, as shown in FIG1 , the model compression method may specifically include the following steps 102 to 106:

[0028] Step 102, obtaining a first model;

[0029] Step 104: Pruning the first model based on the historical output information of the first model and the first annotation information to obtain a second model;

[0030] Step 106: performing self-distillation on the second model based on the historical output information of the second model and the second annotation information to obtain a third model;

[0031] The first annotation information is the real annotation information of the first model, and the second annotation information is the real annotation information of the second model.

[0032] The technical solution for the model compression method provided in this application can be implemented by an electronic device or a model compression device, and can be determined based on actual usage requirements, and is not specifically limited here. To more clearly describe the model compression method provided in this application, the following description uses the model compression device as the implementation entity of the model compression method.

[0033] It is understandable that deep neural network models typically contain a large number of parameters to ensure performance. However, as the scale of network models continues to grow, the computational and storage costs of the network models also increase. Therefore, it is necessary to reduce the number of network model parameters without seriously affecting the performance of the network model in order to compress the network model. Currently, common model compression methods include network pruning and knowledge distillation. However, both of these model compression methods have difficulty in achieving a good balance between reducing the number of parameters and ensuring model performance, resulting in poor model compression results.

[0034] Therefore, the present application provides a model compression method for reducing the number of model parameters of a network model to compress the network model. When compressing a network model using the model compression method provided by the present application, it is possible to better balance the two goals of reducing the number of parameters and ensuring model performance. In other words, the model compression method provided by the present application can reduce the number of model parameters of the network model while ensuring that the network model has good performance, thereby achieving compression of the network model.

[0035] Specifically, in the model compression method provided herein, during the process of compressing a network model, the model compression device first obtains a first model to be compressed. Then, based on the first annotation information and historical output information of the first model to be compressed, the model compression device performs a pruning process on the first model to prune the model parameters of the first model, thereby reducing the number of model parameters of the first model and obtaining a second model with reduced model parameters. The first annotation information is the actual annotation information of the first model, that is, the first annotation information is the hard label information of the first model. On this basis, the model compression device further performs a self-distillation process on the obtained second model based on the second annotation information and historical output information of the second model obtained after the pruning process to train the second model. This self-distillation training process does not change the number of model parameters of the second model, but improves the model performance of the second model obtained after the pruning process. After the self-distillation process is performed on the second model obtained after the pruning process, a third model with reduced model parameters and better model performance can be obtained. The second annotation information is the actual annotation information of the second model, that is, the second annotation information is the hard label information of the second model.

[0036] Thus, in the process of compressing the first model to be compressed, the first model is first pruned based on the hard label information and historical output information of the first model, combined with the principle of self-distillation, to obtain a second model with reduced model parameters. Then, based on the hard label information and historical output information of the second model, the second model is trained based on the principle of self-distillation to obtain a third model with reduced model parameters and better model performance. In this way, the first model is compressed by combining parameter pruning and self-distillation to reduce the number of model parameters, thereby reducing the computational cost and storage cost of the model and reducing the cost of using the model. At the same time, the third model obtained after compression can still have good performance, improving the model compression effect. Moreover, the pruning and self-distillation of the compressed model, combined with the hard label information and historical output information of the model, alleviates the degradation problem of the pruned model in self-distillation on the complete data set, and solves the problem that the pruned model has difficulty in quickly learning knowledge from the "hard labels", reducing the model training time and improving the speed and efficiency of model compression.

[0037] In the embodiment of the present application, optionally, the above step 104 may specifically include the following steps 104a to 104d:

[0038] Step 104a, iteratively updating the model parameters of the first model according to the first iteration number;

[0039] Step 104b, during each iterative update process, determining third annotation information based on the first annotation information of the first model and historical output information;

[0040] Step 104c, determining a target loss value based on the current output information of the first model and the third annotation information;

[0041] Step 104d, updating the model parameters of the first model according to the target loss value;

[0042] The third annotation information is the predicted annotation information of the first model.

[0043] In the above embodiment, based on the hard label information and historical output information of the first model and the principle of self-distillation, parameter pruning is performed on the first model. Specifically, the model compression device iteratively updates the model parameters of the first model according to a set first number of iterations to prune the model parameters of the first model, thereby reducing the number of model parameters of the first model and obtaining a second model with reduced model parameters.

[0044] In the process of each iterative update of the first model, the model compression device randomly samples from the sample data set of the first model for calculation. Specifically, the model compression device determines the third labeling information of the first model based on the hard label information of the first model, that is, the first labeling information, and the historical output information of the first model in the last iterative update. The third labeling information is the predicted labeling information of the first model, that is, the third labeling information is the soft label information of the first model. On this basis, the model compression device determines the target loss value for iterative update of the first model based on the current output information of the first model in this iterative update and the soft label information of the first model determined above, that is, the third labeling information, and then updates the model parameters of the first model based on the target loss value.

[0045] Thus, during the parameter pruning process of the first model, the sample data set of the first model is randomly sampled, and the soft label information of the first model is obtained by combining the hard label information of the first model and the historical output information during the iterative update process. The target loss value of the first model is then determined based on the current output information and the soft label information of the first model, and the model parameters of the first model are then updated based on the target loss value. In this way, during the process of pruning the model parameters of the first model, the correlation information between the first model data is taken into account, thereby alleviating the gradient deviation problem caused by randomly sampling small batches of data, ensuring the accuracy of pruning the first model, and improving the degradation problem of the pruned model on the complete data set.

[0046] In the actual application process, the soft label information of the first model, that is, the third annotation information, can be determined by the following formula (1): Soft_target=(1-α)y gt+αy e-1 , (1)

[0047] Among them, Soft_target represents the soft label information of the first model, that is, the third annotation information, gt Represents the hard label information of the first model, i.e., the first annotation information, y e-1 represents the historical output information of the first model in the last iterative update, and α is a scaling factor, which is used to indicate the proportion of the historical output information to the third labeled information.

[0048] Furthermore, in actual application, the above target loss value can be determined by the following formula (2): L sd (x) = L cls ((1-α)y gt +αy e-1 ,y e ), (2)

[0049] Among them, L sd (x) represents the target loss value, L cls represents the cross entropy loss function of the first model, (1-α)y gt +αy e-1 is the soft label information of the first model, i.e. the third annotation information, y gt Represents the hard label information of the first model, i.e., the first annotation information, y e-1 Represents the historical output information of the first model in the last iterative update, y e Represents the current output information of the first model in this iterative update, and α is the scaling factor.

[0050] In addition, those skilled in the art may set the specific value of the first iteration number according to actual conditions, and no specific limitation is imposed here.

[0051] In the embodiment of the present application, optionally, the above step 104d may specifically include the following steps 104d1 to 104d3:

[0052] Step 104d1, determining the gradient value of each model parameter in the first model according to the target loss value;

[0053] Step 104d2, determining the significance score of each model parameter based on the gradient value of each model parameter;

[0054] Step 104d3: Update the model parameters of the first model according to the significance score.

[0055] In the above embodiment, the model compression device determines the gradient value of each model parameter in the first model based on the determined target loss value, and then determines the significance score of each model parameter based on the gradient value of each model parameter in the first model. The significance score of the model parameter is used to indicate the importance of the corresponding model parameter. On this basis, the model compression device further prunes the model parameters of the first model based on the significance scores of each model parameter in the first model. Specifically, the model compression device prunes the model parameters of the first model based on the significance scores of each model parameter in the first model to complete the update of the model parameters of the first model. In this way, the correlation information between the first model data is taken into account during the process of obtaining the above gradient values, thereby alleviating the gradient deviation problem caused by randomly sampling small batches of data. Based on this, when the significance score of each model parameter in the first model is further obtained using the above gradient values, and the model parameters of the first model are pruned based on the significance scores, the model parameters that can be used for subsequent training can be accurately found, thereby ensuring the accuracy of the pruning of the first model and improving the degradation problem of the pruned model on the complete dataset.

[0056] In actual application, the above gradient value can be determined by the following formula (3):

[0057] Among them, ω l represents the lth model parameter of the first model, Δ(ω l , T i ) represents the model parameter ω l The soft gradient value, L sd represents the target loss value of the first model, i represents the round of iterative update, W i represents the parameter weight updated in the i-th iteration, T i Indicates the random data sampled for the i-th iteration update, m l represents the pruning mask, is the symbol of partial derivative.

[0058] Furthermore, in actual application, the above significance score can be determined by the following formula (4): S ωl =(|Δ(ω l , T i )|) / (∑ q |Δ(ω q , T i )|), (4)

[0059] Among them, S ωl represents the model parameter ω l The significance score of |Δ(ω l , T i)| represents the model parameter ω l The absolute value of the soft gradient value, |Δ(ω q , T i )| represents the model parameter ω q The absolute value of the soft gradient value, ∑ q |Δ(ω q , T i )| represents the sum of the absolute values ​​of the soft gradient values ​​of all model parameters in the first model, and q represents the order of the model parameters in the first model.

[0060] In the embodiment of the present application, optionally, the above step 104d3 may specifically include the following steps 104d31 and 104d32:

[0061] Step 104d31, sorting the model parameters of the first model according to the significance scores to obtain a target order;

[0062] Step 104d32: prune the model parameters of the first model according to the target order and the target compression ratio.

[0063] In the above embodiment, the model compression device prunes the model parameters of the first model based on the significance scores of the model parameters in the first model to update the model parameters of the first model. Specifically, the model compression device sorts the model parameters in the first model based on the significance scores of the model parameters in the first model, either in descending order or in descending order, to obtain a target order for the model parameters in the first model. It will be understood that the significance scores of the model parameters indicate the importance of the corresponding model parameters; the higher the significance score, the more important the corresponding model parameter. Based on this, the model compression device then prunes the model parameters of lower importance in the first model according to the determined target order. Specifically, the model compression device deletes a portion of the model parameters of the first model with lower significance scores based on the determined target order and a set target compression ratio to complete the update of the model parameters of the first model. In this way, pruning the model parameters of the first model based on the significance scores accurately identifies model parameters that can be used for subsequent training, ensuring the accuracy of the pruning of the first model.

[0064] In actual application, those skilled in the art may set the specific value of the target compression rate according to actual conditions, and no specific limitation is imposed here.

[0065] In the embodiment of the present application, optionally, the above step 104d3 may specifically include the following step 104d33:

[0066] Step 104d33: When the target score is greater than the significance score, the model parameters corresponding to the significance score are deleted.

[0067] In the above embodiment, the model parameters of the first model are pruned based on the significance scores of each model parameter in the first model to update the model parameters of the first model. Specifically, the model compression device compares the significance scores of each model parameter in the first model with a set target score. It can be understood that the significance score of a model parameter is used to indicate the importance of the corresponding model parameter. The higher the significance score, the higher the importance of the corresponding model parameter. On this basis, if the set target score is greater than the significance score of a model parameter in the first model, it means that the significance score of the model parameter is low, that is, the importance of the model parameter is low. In this case, the model compression device will delete the model parameter. In this way, the model compression device deletes the model parameters in the first model whose significance scores are less than the set target score to achieve the update of the model parameters of the first model. In this way, by pruning the model parameters of the first model based on the significance score, model parameters that can be used for subsequent training can be accurately found, ensuring the accuracy of the pruning of the first model.

[0068] In the embodiment of the present application, optionally, the above step 106 may specifically include the following steps 106a to 106c:

[0069] Step 106a, iteratively training the second model according to the second number of iterations;

[0070] Step 106b: During each iterative training process, determine fourth labeling information based on the second labeling information of the second model and historical output information;

[0071] Step 106c: training the second model based on the fourth labeled information;

[0072] The fourth annotation information is the predicted annotation information of the second model.

[0073] In the above embodiment, the second model is trained based on the hard label information and historical output information of the second model according to the principle of self-distillation. Specifically, the model compression device iteratively trains the second model according to a set second number of iterations to improve the model performance of the second model obtained after pruning without changing the number of model parameters of the second model, thereby obtaining a third model with reduced model parameters and better model performance.

[0074] During each iterative training of the second model, the model compression device determines the fourth label information of the second model based on the hard label information of the second model, i.e., the second label information, and the historical output information of the second model in the previous round of iterative training. This fourth label information is the predicted label information of the second model, i.e., the fourth label information is the soft label information of the second model. On this basis, the model compression device continues to iteratively train the second model based on the soft label information of the second model, i.e., the fourth label information.

[0075] Specifically, the model compression device determines the loss value of the iterative training of the second model based on the current output information of the second model in this iterative training, and the soft label information of the second model determined above, i.e., the fourth annotation information, and continues to iteratively train the second model based on the loss value. In this way, during the self-distillation training of the second model, the second model traverses the entire training set, and then combines the hard label information of the second model and the historical output information during the iterative training process to obtain the soft label information of the second model. The loss value of the second model is then determined based on the current output information and soft label information of the second model, and the iterative training of the second model is continued based on the loss value. In this way, the second model can quickly learn the historical output information generated by itself from the "soft labels", solving the problem that the second model has difficulty in quickly learning knowledge from the "hard labels". While improving the performance of the second model, it reduces the model training time and improves the speed and efficiency of model compression.

[0076] In the actual application process, the loss value of the second model during the training process can be determined by the following formula (5): L sd (z) = L cls ((1-α)y gt +αy e-1 ,y e ), (5)

[0077] Among them, L sd (z) represents the loss value of the second model during training, L cls represents the cross entropy loss function of the second model, (1-α)y gt +αy e-1 is the soft label information of the second model, i.e. the fourth annotation information, y gt Represents the hard label information of the second model, i.e., the second annotation information, y e-1 Represents the historical output information of the second model in the last iterative training, y e Represents the current output information of the second model in this iterative training, and α is the scaling factor.

[0078] In addition, the specific value of the second iteration number can be set by those skilled in the art according to actual conditions and is not specifically limited here.

[0079] In the embodiment of the present application, optionally, the step of determining the fourth annotation information based on the second annotation information of the second model and the historical output information may specifically include the following steps 108 and 110:

[0080] Step 108, during each iterative training process, traverse the sample data set through the second model to obtain model output information corresponding to each sample data;

[0081] Step 110: Obtain fourth labeling information corresponding to each sample data according to the second labeling information corresponding to each sample data and the model output information of the last iterative training.

[0082] In the above embodiment, the fourth labeling information is determined based on the historical output information and the second labeling information of the second model. Specifically, during each iterative training process, the second model traverses the entire sample data set, and records the model output information corresponding to each sample data during each iterative training process. On this basis, when iteratively training the model parameters of the second model, during each iterative training process, the hard label information corresponding to each sample data, i.e., the second labeling information, and the model output information corresponding to each sample data during the previous iterative training process, i.e., the historical output information, are fused to obtain the soft label information corresponding to each sample data during the current round of iterative training, i.e., the fourth labeling information. In this way, during the training of the model parameters of the second model, the second model can quickly learn the historical output information generated by itself from the "soft labels", solving the problem that the second model is difficult to quickly learn knowledge from the "hard labels". While improving the performance of the second model, the model training time is reduced, and the speed and efficiency of model compression are improved.

[0083] In the embodiment of the present application, optionally, the fourth annotation information is determined by the following formula: Soft_target=(1-α)y gt +αy e-1 , where Soft_target represents the fourth annotation information corresponding to each sample data, y gt Represents the second annotation information corresponding to each sample data, y e-1 It represents the model output information corresponding to each sample data in the last iterative training process. α is the scale factor, which is used to indicate the proportion of the model output information to the fourth annotation information.

[0084] In the above embodiment, in each iterative training process, the soft label information corresponding to each sample data, that is, the fourth annotation information, can be determined by the following formula (6): Soft_target=(1-α)y gt +αy e-1 , (6)

[0085] Among them, Soft_target represents the fourth annotation information corresponding to each sample data, y gt Represents the second annotation information corresponding to each sample data, y e-1 represents the model output information corresponding to each sample data point during the previous iterative training process. α is a scaling factor, indicating the proportion of the model output information to the fourth labeled information. This allows the second model to quickly learn its historical output information from the "soft labels" during parameter training. This improves the performance of the second model while reducing model training time and increasing the speed and efficiency of model compression.

[0086] In an embodiment of the present application, optionally, during each iterative training process, the target ratio of the historical output information of the second model to the fourth labeled information is proportional to the number of iterations.

[0087] In the above embodiment, during each iterative training of the second model, the target proportion of the historical output information of the second model in the previous round of iterative training to the fourth labeled information of the current iterative training is proportional to the iterative round of the current iterative training, i.e., the number of iterations. In this way, as the number of iterative rounds increases, the target proportion of the historical output information of the second model in the previous round of iterative training to the fourth labeled information of the current round of iterative training gradually increases. That is, during the iterative training of the second model, the target proportion of the historical output information of the second model to the soft label information gradually increases. This alleviates the overfitting problem that may occur in the training process of the second model, ensures the accuracy of the training of the second model, and thus ensures the model performance of the second model.

[0088] In an embodiment of the present application, optionally, in each iterative training process, the ratio of the preset ratio value to the target ratio value is the same as the ratio of the second iteration number to the current iteration number.

[0089] In the above embodiment, during each iterative training of the second model, the ratio of the target proportion of the historical output information of the second model in the previous round of iterative training in the fourth annotation information of this iterative training to the set preset proportion value, and the ratio of the iterative round of the current iterative training of the second model, i.e., the current number of iterations, to the set total number of iterative training, i.e., the above second number of iterations, are the same. It can be understood that the set preset proportion value and the set total number of iterative training, i.e., the above second number of iterations, are fixed values. In this way, during the iterative training of the second model, the target proportion of the historical output information of the second model in the previous round of iterative training in the fourth annotation information of the current round of iterative training can increase with the increase of the number of iterations, which alleviates the overfitting problem that may occur in the training process of the second model, ensures the accuracy of the training of the second model, and thus ensures the model performance of the second model.

[0090] Among them, those skilled in the art can set the above-mentioned preset ratio value and the specific values ​​of the second iteration number according to actual conditions, and no specific limitation is made here.

[0091] In the embodiment of the present application, optionally, the relationship between the target ratio value and the number of iterations is: t =α T ×(t / T); where α t Indicates the target ratio of the historical output information of the second model to the fourth labeled information in the current round of iterative training, α T represents the preset ratio value, T represents the second iteration number, and t represents the current iteration number.

[0092] In the above embodiment, during each iterative training of the second model, the target proportion of the historical output information of the second model in the previous iterative training to the fourth annotation information of the current iterative training can be determined by the following formula (7): t =α T ×(t / T), (7)

[0093] Among them, α t Indicates the target ratio of the historical output information of the second model in the previous round of iterative training to the fourth annotation information of this iterative training in the current round of iterative training. t represents the current number of iterations, T represents the total number of iterative training, i.e., the second number of iterations mentioned above. α T Indicates the preset ratio value, and t / T indicates the ratio of t to T.

[0094] In this way, during the iterative training of the second model, the target proportion value of the historical output information of the second model in the previous round of iterative training in the fourth annotation information of the current round of iterative training gradually increases in a dynamic linear manner, which alleviates the overfitting problem that may occur in the training process of the second model, ensures the accuracy of the training of the second model, and thus ensures the model performance of the second model.

[0095] In the embodiment of the present application, the self-distillation process and the pruning process are optionally a one-time compression process. After the above step 102, the model compression method may specifically include the following step 112 or step 114:

[0096] Step 112, performing a compression process on the first model according to the target compression ratio;

[0097] Step 114 : cyclically perform compression processing on the first model for a target number of times according to a preset pruning rate.

[0098] In the above embodiment, the self-distillation and pruning processes can be used as a single compression process. During the compression process of the model to be compressed, after obtaining the first model to be compressed, the first model can be compressed once according to a set target compression ratio. That is, the first model can be compressed once according to the target compression ratio. In other words, after the first model is subjected to a single pruning process and a single self-distillation process, the compression operation on the first model is completed.

[0099] In the above embodiment, during the compression of the model to be compressed, after obtaining the first model to be compressed, the first model may be iteratively compressed according to a preset pruning rate. That is, the compression process is cyclically performed on the first model a target number of times according to the preset pruning rate. Specifically, the first model is cyclically processed in the order of "pruning process - self-distillation process - pruning process - self-distillation process" until the pruning rate of the first model reaches the preset pruning rate, thereby completing the compression operation on the first model.

[0100] Among them, those skilled in the art can set the specific values ​​of the above-mentioned target compression rate and preset pruning rate according to actual conditions, and no specific limitation is made here.

[0101] Specifically, as shown in FIG2 and FIG3 , the model compression method proposed in this application may include two processing flows: self-distillation-based pruning and self-distillation-based training.

[0102] Among them, the processing flow based on self-distillation pruning is the above-mentioned pruning process. Specifically, as shown in Figures 2 and 3, in the processing flow based on self-distillation pruning, the sample data set T of the initial model to be compressed is randomly sampled, and after the model parameters are initialized, based on the self-knowledge of the model to be compressed, according to the self-distillation principle, the model parameters of the model to be compressed are iteratively updated, and then the mask and the gradient value for evaluating the importance of the model parameters are obtained through reverse calculation, and the significance score of each model parameter in the model to be compressed is further obtained based on the gradient value. On this basis, according to the significance score of each model parameter in the model to be compressed, the generated mask is applied to prune the model parameters of the model to be compressed to obtain the pruned model parameters. Among them, in this application, "θ1⊙m" in Figure 3 represents the point multiplication operation of the initialized model parameter θ1 and the mask m.

[0103] Furthermore, the processing flow based on self-distillation training is the self-distillation process described above. Specifically, as shown in Figures 2 and 3, in the processing flow based on self-distillation training, the model parameters of the model to be compressed after pruning are obtained and the obtained model parameters are initialized. Furthermore, the model to be compressed is made to traverse the complete sample data set T, and the model to be compressed is iteratively trained. After the iterative training process of the model to be compressed, the output information of the model to be compressed in each iterative training is saved, and the historical output information of the model to be compressed in the previous iterative training process is applied to the iterative training process of the model to be compressed in the current round.

[0104] Specifically, as shown in Figure 2, during each iterative training of the model to be compressed, the soft label information of the model to be compressed is determined based on the hard label information of the model to be compressed and the historical output information of the model to be compressed in the previous round of iterative training. Then, based on the current output information of the model to be compressed in this iterative training and the soft label information of the model to be compressed, the loss value of the iterative training of the model to be compressed is determined, and the iterative training of the second model is continued based on the loss value.

[0105] On this basis, in the process of compressing the model to be compressed using the model compression method proposed in this application, a process based on self-distillation pruning and a process based on self-distillation training can be regarded as a single compression process. Based on this, when compressing the model to be compressed, a single compression process can be performed on the model to be compressed according to the set target compression rate. That is, after the model to be compressed is subjected to a pruning process and a self-distillation process, the compression operation of the model to be compressed can be completed.

[0106] Alternatively, as shown in Figures 2 and 3, the trained model parameters are reinitialized according to a preset pruning rate, and the compression process is then cyclically performed on the model to be compressed. That is, according to the "pruning-self-distillation-pruning-self-distillation" cycle, the pruned model parameters are trained, pruned, and then trained again, and this cycle repeats until the pruning rate of the model to be compressed reaches the preset pruning rate, completing the compression operation on the model to be compressed.

[0107] In one embodiment of the present application, a data processing method is further proposed. As shown in FIG4 , the data processing method may specifically include the following steps 202 to 208:

[0108] Step 202: Obtain a data processing model;

[0109] Step 204: compress the data processing model according to the model compression method;

[0110] Step 206: input the target data to be processed into the compressed data processing model;

[0111] Step 208: Process the target data using the compressed data processing model to obtain processed target data.

[0112] The data processing method provided by the present application obtains a data processing model for processing data during the process of processing data, and compresses the data processing model according to the model compression method in any embodiment of the first aspect mentioned above, so as to reduce the amount of model parameters of the data processing model and obtain a compressed data processing model. On this basis, the target data to be processed is input into the compressed data processing model, and the input target data is processed by the compressed data processing model to obtain the processed target data. The data processing method provided by the present application includes the model compression method in any embodiment of the first aspect mentioned above. Therefore, the data processing method proposed in the second aspect of the present application has all the beneficial effects of the model compression method in any embodiment of the first aspect mentioned above, which will not be repeated here.

[0113] In an embodiment of the present application, optionally, the data processing model includes at least one of the following: an image processing model, a speech noise reduction model, a speech recognition model, and a deep learning model.

[0114] In the above embodiments, the above data processing models may specifically include image processing models, speech noise reduction models, speech recognition models, and deep learning models. Specifically, in the process of processing image data, speech data, and waiting-for-processing data using the data processing method proposed in this application, the corresponding image processing model, speech noise reduction model, speech recognition model, and deep learning model and other data processing models may be compressed based on the model compression method in any of the above embodiments, and then the image data, speech data, and waiting-for-processing data may be processed using the compressed data processing model. In this way, while ensuring the accuracy of the data processing results, the model inference speed can also be improved, thereby improving the data processing efficiency.

[0115] In some embodiments, the present application further provides a method for processing voice data, comprising: obtaining voice data and a network model; and processing the voice data using the network model to obtain a processing result of the voice data. The processing result of the voice data may be a voice recognition result, a voice wake-up result, a voice noise reduction result, etc., and the network model may be a network model obtained by compressing the model using the model compression method provided in any of the above embodiments.

[0116] In some embodiments, the present application further provides an image processing method, comprising: obtaining image data and a network model; and processing the image data using the network model to obtain a processing result of the image data. The processing result of the image data may be an image classification result, an image enhancement result, etc., and the network model may be a network model obtained by compressing the model using the model compression method provided in any of the above embodiments.

[0117] In some embodiments, the present application also provides a classification task processing method, including: obtaining classification data to be processed and a network model; processing the classification data to be processed through the network model to obtain a classification result, wherein the network model can be a network model obtained after model compression by the model compression method provided in any of the above embodiments.

[0118] In one embodiment of the present application, a model compression device is also proposed. As shown in FIG5 , FIG5 shows a structural block diagram of the model compression device 300 of the embodiment of the present application. Specifically, the model compression device 300 may include the following acquisition unit 302 and processing unit 304:

[0119] An acquiring unit 302 is configured to acquire a first model;

[0120] A processing unit 304 is configured to prune the first model according to the first annotation information and the historical output information of the first model to obtain a second model;

[0121] The processing unit 304 is further configured to perform self-distillation on the second model based on the second annotation information and the historical output information of the second model to obtain a third model;

[0122] The first annotation information is the real annotation information of the first model, and the second annotation information is the real annotation information of the second model.

[0123] The model compression device 300 provided in an embodiment of the present application is used to reduce the number of model parameters of a network model to compress the network model. When compressing a network model using the model compression device provided by the present application, it is possible to achieve a good balance between reducing the number of parameters and ensuring model performance. In other words, the model compression device provided by the present application can reduce the number of model parameters of the network model while ensuring that the network model has good performance, thereby achieving compression of the network model.

[0124] Specifically, the model compression device 300 provided in the present application includes an acquisition unit 302 and a processing unit 304. During the model compression process, the acquisition unit 302 first acquires the first model to be compressed. The processing unit 304 then performs pruning processing on the first model based on the first annotation information and historical output information of the first model to be compressed to prune the model parameters of the first model, thereby reducing the model parameter amount of the first model and obtaining a second model with reduced model parameters. The first annotation information is the real annotation information of the first model, that is, the first annotation information is the hard label information of the first model. On this basis, the processing unit 304 continues to perform self-distillation processing on the second model obtained after the pruning process based on the second annotation information and historical output information of the second model to train the second model. The self-distillation training process does not change the model parameter amount of the second model, but can improve the model performance of the second model obtained after the pruning process. After the second model obtained after the pruning process is self-distilled, a third model with reduced model parameters and better model performance can be obtained. The second annotation information is the real annotation information of the second model, that is, the second annotation information is the hard label information of the second model.

[0125] Thus, in the process of compressing the first model to be compressed, the first model is first pruned based on the hard label information and historical output information of the first model, combined with the principle of self-distillation, to obtain a second model with reduced model parameters. Then, based on the hard label information and historical output information of the second model, the second model is trained based on the principle of self-distillation to obtain a third model with reduced model parameters and better model performance. In this way, the first model is compressed by combining parameter pruning and self-distillation. While reducing the model parameters and thus the model computational and storage costs, it ensures that the third model obtained after compression has better performance and improves the model compression effect. Furthermore, the model is pruned and self-distilled based on the hard label information and historical output information of the model, which alleviates the degradation problem of the pruned model in self-distillation on the complete data set, and solves the problem that the pruned model has difficulty in quickly learning knowledge from the "hard labels", thereby reducing the model training time and improving the speed and efficiency of model compression.

[0126] In an embodiment of the present application, optionally, the processing unit 304 is specifically used to: iteratively update the model parameters of the first model according to the first number of iterations; in each iterative update process, determine the third labeling information according to the first labeling information and historical output information of the first model; determine the target loss value according to the current output information of the first model and the third labeling information; based on the target loss value, update the model parameters of the first model; wherein the third labeling information is the predicted labeling information of the first model.

[0127] In an embodiment of the present application, optionally, the processing unit 304 is specifically used to: determine the gradient value of each model parameter in the first model based on the target loss value; determine the significance score of each model parameter based on the gradient value of each model parameter; and update the model parameters of the first model based on the significance score.

[0128] In the embodiment of the present application, optionally, the processing unit 304 is specifically configured to: sort the model parameters of the first model according to the significance score to obtain a target order; and delete the model parameters of the first model according to the target order and the target compression rate.

[0129] In the embodiment of the present application, optionally, the processing unit 304 is specifically configured to: when the target score is greater than the significance score, delete the model parameters corresponding to the significance score.

[0130] In an embodiment of the present application, optionally, the processing unit 304 is specifically used to: iteratively train the second model according to the second number of iterations; during each iterative training process, determine the fourth labeling information according to the second labeling information and historical output information of the second model; train the second model according to the fourth labeling information; wherein the fourth labeling information is the predicted labeling information of the second model.

[0131] In an embodiment of the present application, optionally, the processing unit 304 is specifically used to: traverse the sample data set through the second model during each iterative training process to obtain model output information corresponding to each sample data; and obtain fourth annotation information corresponding to each sample data based on the second annotation information corresponding to each sample data and the model output information of the previous iterative training.

[0132] In the embodiment of the present application, optionally, the fourth annotation information is determined by the following formula: Soft_target=(1-α)y gt +αy e-1 , where Soft_target represents the fourth annotation information corresponding to each sample data, y gt Represents the second annotation information corresponding to each sample data, y e-1 It represents the model output information corresponding to each sample data in the last iterative training process. α is the scale factor, which is used to indicate the proportion of the model output information to the fourth annotation information.

[0133] In an embodiment of the present application, optionally, during each iterative training process, the target ratio of the historical output information of the second model to the fourth labeled information is proportional to the number of iterations.

[0134] In an embodiment of the present application, optionally, in each iterative training process, the ratio of the preset ratio value to the target ratio value is the same as the ratio of the second iteration number to the current iteration number.

[0135] In the embodiment of the present application, optionally, the relationship between the target ratio value and the number of iterations is: t =α T ×(t / T); where α t Indicates the target ratio of the historical output information of the second model to the fourth labeled information in the current round of iterative training, α T represents the preset ratio value, T represents the second iteration number, and t represents the current iteration number.

[0136] In an embodiment of the present application, optionally, the self-distillation process and the pruning process are a one-time compression process. After obtaining the first model, the processing unit 304 is specifically used to: perform a compression process on the first model according to a target compression rate; or perform a target number of compression processes on the first model according to a preset pruning rate.

[0137] In one embodiment of the present application, another model compression device is also proposed. As shown in FIG6 , FIG6 shows a structural block diagram of the model compression device 400 provided in an embodiment of the present application. The model compression device 400 includes:

[0138] Memory 402 , where programs or instructions are stored;

[0139] Processor 404, when executing the above program or instruction, implements the steps of the model compression method in any of the above embodiments.

[0140] The model compression device 400 provided in this embodiment includes a memory 402 and a processor 404. When the program or instructions in the memory 402 are executed by the processor 404, the steps of the model compression method in any of the above embodiments are implemented. Therefore, the model compression device 400 has all the beneficial effects of the model compression method in any of the above embodiments, which will not be repeated here.

[0141] This application addresses the long training time and slow cycle of existing model compression algorithms, and proposes an efficient model compression solution. Specifically, this application addresses the problem that the gradient standard used during pruning is difficult to effectively evaluate the model parameters applicable to the complete dataset, resulting in performance degradation of the pruned model on the complete dataset, and the problem that the pruned model is difficult to quickly learn knowledge from "hard labels". This application proposes an efficient model compression solution based on pre-training pruning and self-distillation, which can be used in a variety of application scenarios.

[0142] In the technical solution provided by the present application, when pruning the model, the model parameters will be updated a small amount and then the soft label information will be obtained by combining the historical output information and hard label information generated during the update process. After the target loss value is calculated by the current output information of the model and the soft label information, the soft gradient value for evaluating the importance of the model parameters is obtained. Compared with the gradient in the prior art, the soft gradient value includes the correlation information between the sample data, thereby alleviating the gradient deviation problem caused by random sampling of small batches of data, and then improving the degradation problem of the pruned model on the complete data set. In addition, in the technical solution provided by the present application, during the training process of the pruned model, the output information of the model in the previous training stage is recorded to soften the true label of the model in the current training stage, that is, the hard label information, so that the pruned model can quickly learn the historical information generated by itself from the "soft label", thereby improving the performance of the pruned model.

[0143] In some embodiments, the technical solution provided by this application addresses the problem of model compression in actual production processes and mainly includes two parts. The first part is a model pruning technology based on soft gradients, and the second part is a training technology for pruned models based on self-distillation. The complete process of this technical solution can be seen in Figure 2, which includes the corresponding pruning technology and self-distillation training technology. First, in order to obtain soft gradients to evaluate parameter importance, the model parameters are updated in a small number of iterations during pruning. During the update process, the model output results corresponding to the input data are saved at each iteration and used to construct soft labels in the next iteration. At each iteration, the loss value is calculated using the soft labels and the current output information of the model. The calculated loss value is used to further calculate the soft gradient. In this way, the process of obtaining the soft gradient takes into account the correlation information between sample data, thereby alleviating the gradient deviation problem caused by randomly sampling small batches of data. Therefore, after the obtained soft gradients are used to further calculate the significance score to evaluate the importance of model parameters, trainable parameters can be found more accurately, improving the degradation problem of the pruned model on the complete dataset.

[0144] Furthermore, during the training phase of the pruned model, the model's output information from each iteration is saved and used in the next iteration of the model's training. Specifically, as the model traverses the entire training dataset, the model output corresponding to each data point is saved in the computer memory and read the next time the model traverses the entire training dataset. This data is then fused with the hard label of the corresponding data point to obtain the soft label for the current training phase. During the actual training process, to mitigate the potential overfitting problem of the model, the proportion of historical information output by the model in the soft label is gradually increased using a dynamic linear increase method.

[0145] In practical applications, the pruning and self-distillation training in this technical solution can be performed in two ways: one is single compression, where pruning and self-distillation training are performed once after a given compression rate to complete the compression process of the model; the other is iterative compression, that is, iteratively executing the "pruning-self-distillation training-pruning" cycle until the preset pruning rate is reached.

[0146] It is understandable that model pruning and knowledge distillation often rely on a lot of training costs. The technical solution provided by this application combines the two technologies of pre-training pruning and self-distillation in a novel way to achieve more efficient model compression. In response to the problem that the gradient standard used in pruning is difficult to effectively evaluate the model parameters applicable to the complete data set, which causes the model performance degradation, the technical solution proposed in this application generates soft label information by fusing the historical output information and hard label information of the model, and then generates soft gradients to screen out the trainable parameters in the model, thereby alleviating the degradation problem of the pruned model on the complete data set. In response to the problem that the pruned model is difficult to learn knowledge from the "hard label" quickly, the technical solution proposed in this application uses self-distillation technology to learn from the historical information output by the model on the basis of reducing redundant parameters in the model, thereby improving the performance of the pruned model.

[0147] In order to address the problem of model performance degradation caused by pruning and the problem that knowledge distillation cannot be flexibly applied to actual model compression, this application proposes a model compression solution that can effectively combine pre-training pruning and self-distillation. By using the technical solution provided by this application, the performance of the pruned model can be further improved, and the knowledge distillation technology can be flexibly applied to the model compression process.

[0148] The technical solution provided in this application can be applied to different edge systems such as Linux / RTOS / Android / iOS, and provide instruction-level acceleration for different edge platforms such as ARMv7 / V8 and DSP. The technical solution of this application exhibits the characteristics of lightweight deployment, strong versatility, ease of use, and high-performance inference. It comprehensively solves the problem of low resource bottlenecks in smart devices, greatly shortens the AI ​​model deployment cycle, and reaches the industry-leading level in the field of edge AI deployment. In addition, the technical solution provided in this application can be applied to self-developed chips, for example, it can be applied to the industry's first three-in-one chip FL119 that supports voice, connection, and display. The relevant results have fully enabled the mass production of smart home appliances such as voice refrigerators, air conditioners, and robots, improving the intelligence of smart home appliances and increasing the working efficiency of smart home appliances.

[0149] Specifically, the memory 402 and the processor 404 may be connected via a bus or other means. The processor 404 may include one or more processing units, and the processor 404 may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other chips.

[0150] In one embodiment of the present application, an electronic device is also provided. As shown in FIG7 , FIG7 shows a block diagram of the structure of an electronic device 500 provided in an embodiment of the present application. The electronic device 500 includes the model compression device 400 described in the above embodiment. Therefore, the electronic device 500 has all the technical effects of the model compression device 400 described in the above embodiment, and will not be further described here.

[0151] In one embodiment of the present application, a data processing device is also proposed. As shown in FIG8 , FIG8 shows a structural block diagram of a data processing device 600 of an embodiment of the present application. Specifically, the data processing device 600 may include the following acquisition unit 602 and processing unit 604:

[0152] An acquisition unit 602 is used to acquire a data processing model;

[0153] A processing unit 604 is configured to compress the data processing model according to the model compression method in any one of the embodiments of the first aspect above;

[0154] The processing unit 604 is further configured to input the target data to be processed into the compressed data processing model;

[0155] The processing unit 604 is further configured to process the target data using the compressed data processing model to obtain processed target data.

[0156] The data processing device 600 provided in the present application, in the process of processing data, obtains the data processing model for processing data through the acquisition unit 602, and compresses the data processing model through the processing unit 604 according to the model compression method in any embodiment of the first aspect above, so as to reduce the amount of model parameters of the data processing model and obtain the compressed data processing model. On this basis, the processing unit 604 inputs the target data to be processed into the compressed data processing model, and processes the input target data through the compressed data processing model to obtain the processed target data. The data processing device 600 provided in the present application can implement the model compression method in any embodiment of the first aspect above. Therefore, the data processing device 600 proposed in the fourth aspect of the present application has all the beneficial effects of the model compression method in any embodiment of the first aspect above, which will not be repeated here.

[0157] In an embodiment of the present application, optionally, the data processing model includes at least one of the following: an image processing model, a speech noise reduction model, a speech recognition model, and a deep learning model.

[0158] In one embodiment of the present application, a readable storage medium is further provided. A program or instruction is stored on the medium. When the program or instruction is executed by a processor, the steps of the model compression method in any of the above embodiments are implemented, or when the program or instruction is executed by a processor, the steps of the data processing method in any of the above embodiments are implemented.

[0159] The readable storage medium provided in the embodiments of the present application, when executed by a processor, stores a program or instruction that implements the steps of the model compression method or data processing method described in any of the aforementioned embodiments. Therefore, the readable storage medium possesses all the beneficial effects of the model compression method or data processing method described in any of the aforementioned embodiments, and no further description is given here.

[0160] Specifically, the above-mentioned readable storage medium may include any medium capable of storing or transmitting information. Examples of readable storage media include electronic circuits, semiconductor memory devices, read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), flash memory, erasable ROM (EROM), magnetic tape, floppy disk, optical disc, hard disk, optical fiber medium, radio frequency (RF) link, optical data storage device, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0161] In one embodiment of the present application, a computer program product is further provided, comprising a computer program that, when executed by a processor, implements the model compression method or data processing method described in any of the above-described embodiments. Therefore, the computer program product provided in this application possesses all the beneficial effects of the model compression method described in any of the above-described embodiments of the first aspect, or possesses all the beneficial effects of the data processing method described in any of the above-described embodiments of the second aspect, which will not be further elaborated herein.

[0162] In one embodiment of the present application, a chip is further provided. The chip includes a program or instructions. When the chip is running, it is used to implement the steps of the model compression method in any of the above embodiments, or when the chip is running, it is used to implement the steps of the data processing method in any of the above embodiments. Therefore, the chip provided in this application has all the beneficial effects of the model compression method in any of the embodiments of the first aspect, or the chip provided in this application has all the beneficial effects of the data processing method in any of the embodiments of the second aspect, which will not be described in detail here.

[0163] In this specification, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance, unless otherwise expressly specified or limited. Terms such as "connect," "install," and "fix" should be interpreted broadly. For example, "connect" can refer to a fixed connection, a detachable connection, or an integral connection; it can refer to a direct connection or an indirect connection through an intermediary. Those skilled in the art will understand the specific meanings of these terms in this application based on specific circumstances.

[0164] Throughout this specification, terms such as "one embodiment," "some embodiments," and "specific embodiments" mean that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0165] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0166] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A model compression method, wherein: include: Get the first model; Pruning the first model according to the historical output information of the first model and the first annotation information to obtain a second model, wherein the first annotation information is the real annotation information of the first model; According to the historical output information and the second annotation information of the second model, the second model is self-distilled to obtain a third model, and the second annotation information is the real annotation information of the second model.

2. The model compression method according to claim 1, wherein: The pruning of the first model according to the historical output information and the first annotation information of the first model includes: Iteratively updating the model parameters of the first model according to a first number of iterations; In each iterative updating process, third annotation information is determined according to the historical output information of the first model and the first annotation information, where the third annotation information is the predicted annotation information of the first model; Determining a target loss value according to the third annotation information and current output information of the first model; Based on the target loss value, model parameters of the first model are updated.

3. The model compression method according to claim 2, wherein: The updating of the model parameters of the first model based on the target loss value includes: Determining a gradient value of each model parameter in the first model according to the target loss value; Determine the significance score of each model parameter based on the gradient value of each model parameter; According to the significance score, model parameters of the first model are updated.

4. The model compression method according to claim 3, wherein: The updating of the model parameters of the first model according to the significance score comprises: According to the significance score, the model parameters of the first model are sorted to obtain a target order; In accordance with the target order, model parameters of the first model are pruned according to a target compression ratio.

5. The model compression method according to claim 3 or 4, wherein: The updating of the model parameters of the first model according to the significance score comprises: When the significance score is less than the target score, the model parameter corresponding to the significance score is deleted.

6. The model compression method according to any one of claims 1 to 5, wherein: The performing self-distillation processing on the second model according to the historical output information and the second annotation information of the second model includes: Iteratively training the second model according to a second iteration number; In each iterative training process, fourth labeling information is determined according to the historical output information of the second model and the second labeling information, where the fourth labeling information is the predicted labeling information of the second model; The second model is trained according to the fourth labeling information.

7. The model compression method according to claim 6, wherein: The determining, according to the historical output information of the second model and the second annotation information, fourth annotation information includes: In each iterative training process, the sample data set is traversed through the second model to obtain the corresponding Model output information for this data; The fourth labeling information corresponding to each sample data is obtained according to the second labeling information corresponding to each sample data and the model output information of the last iterative training.

8. The model compression method according to claim 7, wherein: The fourth labeling information is determined by the following formula: Soft_target = (1-α)y gt +αy e-1 , Among them, Soft_target represents the fourth labeling information corresponding to each sample data, y gt represents the second labeling information corresponding to each sample data, y e-1 It represents the model output information corresponding to each sample data in the last iterative training process. α is the scale factor, which is used to indicate the proportion of the model output information to the fourth annotation information.

9. The model compression method according to any one of claims 6 to 8, wherein: During each iterative training process, the target ratio of the historical output information of the second model to the fourth labeled information is proportional to the number of iterations.

10. The model compression method according to claim 9, wherein: During each iterative training process, the ratio of the target ratio value to the preset ratio value is the same as the ratio of the current iteration number to the second iteration number.

11. The model compression method according to claim 10, wherein: The relationship between the target ratio value and the number of iterations is: α t =α T ×(t / T); Among them, t represents the current iteration number, T represents the second iteration number, α T Indicates the preset ratio value, α t It represents the target ratio value of the historical output information of the second model to the fourth labeled information in the current round of iterative training.

12. The model compression method according to any one of claims 1 to 11, wherein: The pruning process and the self-distillation process are a compression process. After obtaining the first model, the model compression method includes: According to a target compression ratio, a compression process is performed on the first model; or According to a preset pruning rate, the compression process is performed on the first model for a target number of cycles.

13. A data processing method, wherein: include: Get the data processing model; According to the model compression method according to any one of claims 1 to 12, compressing the data processing model; Inputting the target data to be processed into the compressed data processing model; The target data is processed by the compressed data processing model to obtain processed target data.

14. The data processing method according to claim 13, wherein: The data processing model includes at least one of the following: an image processing model, a speech noise reduction model, a speech recognition model, and a deep learning model.

15. A model compression device, wherein: include: An acquisition unit, configured to acquire a first model; a processing unit, configured to process the first model according to the historical output information of the first model and the first annotation information; The model is pruned to obtain a second model, wherein the first annotation information is the real annotation information of the first model; The processing unit is further used to perform self-distillation processing on the second model according to the historical output information of the second model and the second annotation information to obtain a third model, wherein the second annotation information is the real annotation information of the second model.

16. A data processing device, wherein: include: An acquisition unit, used for acquiring a data processing model; A processing unit, configured to compress the data processing model according to the model compression method according to any one of claims 1 to 12; The processing unit is further used to input the target data to be processed into the data processing model after compression processing; The processing unit is further used to process the target data through the compressed data processing model to obtain processed target data.

17. A readable storage medium, wherein: The readable storage medium stores a program or instruction, which, when executed by a processor, implements the steps of the model compression method as described in any one of claims 1 to 12, or, when executed by a processor, implements the steps of the data processing method as described in claim 13 or 14.

18. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the steps of the model compression method according to any one of claims 1 to 12 are implemented, or when the computer program is executed by a processor, the steps of the data processing method according to claim 13 or 14 are implemented.

19. A chip, wherein: The chip includes a program or instructions, which, when the chip is running, is used to implement the steps of the model compression method as described in any one of claims 1 to 12, or, when the chip is running, is used to implement the steps of the data processing method as described in claim 13 or 14.

Images