Model sparseness adjusting method and device

Abstract

The invention discloses a model sparseness adjusting method and device, and the method comprises the steps: sequentially obtaining at least two groups of sparseness, and each group of sparseness in the at least two groups of sparseness comprises a preset value of the sparseness of each layer in a target model; performing crossover variation on the at least two groups of sparseness through a genetic algorithm to determine a target sparseness group meeting conditions, and determining data in the target sparseness group as a determined value of the sparseness of each layer in the target model; and adjusting the sparseness of each layer in the target model based on the data in the target sparseness group.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for adjusting model sparsity. Background Art

[0002] Large language models (LLMs) pose significant deployment challenges due to their large number of parameters and heavy computational effort during inference. Model sparsification can set redundant parameters in the model to zero, reducing the computational and data handling pressure during inference.

[0003] Currently, model sparsification usually adopts the uniform pruning method, which thins out each layer in the model according to a preset fixed ratio sparsity. However, if this method is used to ensure that the model has a certain pruning rate, the accuracy of the model will be reduced. Summary of the Invention

[0004] In view of this, the present application provides a method and device for adjusting model sparsity, the specific scheme of which is as follows:

[0005] A model sparsity adjustment method, comprising:

[0006] Obtaining at least two sets of sparsities in sequence, each of the at least two sets of sparsities including a preset value of the sparsity of each layer in the target model;

[0007] Performing crossover mutation on the at least two sparsities by a genetic algorithm to determine a target sparsity group that meets a condition, and determining the data in the target sparsity group as a determined value of the sparsity of each layer in the target model;

[0008] The sparsity of each layer in the target model is adjusted based on the data in the target sparsity group.

[0009] Furthermore, the cross-mutation of the at least two sets of sparsities by a genetic algorithm to determine a target data set that meets the conditions includes:

[0010] determining a first prediction accuracy of the target model, the first prediction accuracy being a prediction accuracy of the target model when each of the at least two sets of sparsities is applied to the target model;

[0011] Performing crossover mutation on the at least two sets of sparsities by a genetic algorithm to obtain at least one set of mutated sparsities;

[0012] determining a second prediction accuracy of the target model, the second prediction accuracy being a prediction accuracy of the target model when each set of variation sparsity in the at least one set of variation sparsity is applied to the target model;

[0013] A target sparsity group that meets a condition is determined from the at least two sparsity groups and the at least one set of variation sparsity based on the first prediction accuracy and the second prediction accuracy.

[0014] Furthermore, determining the first prediction accuracy of the target model includes:

[0015] Each of the at least two groups of sparsities is input into the sparsity accuracy prediction model to obtain the prediction accuracy of the target model corresponding to each group of sparsities output by the sparsity accuracy prediction model.

[0016] Furthermore, the sparsity accuracy prediction model is obtained by the following training method, including:

[0017] Obtaining a training sparsity group and using the training sparsity group as a training sample for the sparsity of each layer in the target model;

[0018] Based on the training sparsity and a predetermined sparsification method, performing sparsification processing on the target model;

[0019] Determine the sparse prediction accuracy of the target model after the sparsification process;

[0020] The prediction model is trained based on each sparse prediction accuracy among the multiple sparse prediction accuracies and the training samples to obtain a sparsity accuracy prediction model.

[0021] Furthermore, it also includes:

[0022] Determining the outlier distribution value of each layer in the target model;

[0023] Based on the outlier distribution data of each layer, the preset value of the sparsity of the corresponding layer in each of the at least two groups of sparsity is adjusted so that the preset value of the sparsity of each layer in each group of sparsity conforms to a specific relationship with the outlier distribution data of the corresponding layer.

[0024] Furthermore, adjusting the preset value of the sparsity of the corresponding layer in each of the at least two sparsities based on the outlier distribution data of each layer, so that the preset value of the sparsity of each layer in each of the sparsities conforms to a specific relationship with the outlier distribution data of the corresponding layer, includes:

[0025] Determine a target value corresponding to each layer in the target model, wherein the target value corresponding to each layer is the difference between a specific value and a value of an outlier distribution of each layer;

[0026] The preset value of the sparsity of each layer in each set of sparsity is adjusted so that the preset value of the sparsity of each layer in each set of sparsity after adjustment is proportional to the target value of the corresponding layer.

[0027] Furthermore, it also includes:

[0028] Determine the gradient similarity between the gradient information of each target layer in the target model after sparsity adjustment and the gradient information of the corresponding layer in the target model before sparsity adjustment by knowledge distillation, wherein the target layer represents the layer for low-rank decomposition in the target model after sparsity adjustment;

[0029] The loss function of the target model after adjusting the sparsity is adjusted based on the gradient similarity of each target layer in the target model, so as to update the parameters of each target layer in the target model after adjusting the sparsity based on the loss function.

[0030] Furthermore, adjusting the loss function of the target model after adjusting the sparsity based on the gradient similarity of each target layer in the target model includes:

[0031] Determine the weighting coefficient;

[0032] Performing weighted summation on the gradient similarity of each target layer in the target model based on the weighting coefficient to determine loss data for adjusting sparsity of the target model;

[0033] The loss function of the target model after adjusting the sparsity is adjusted based on the loss data of the target model.

[0034] Furthermore, determining the weighting coefficient includes:

[0035] Determining a first feature output vector of each target layer in the target model before adjusting the sparsity, and a second feature output vector of each target layer in the target model after adjusting the sparsity;

[0036] Determining a feature similarity between a first feature output vector and a second feature output vector of each target layer in the target model;

[0037] If it is determined that the feature similarity of the first target layer in the target model is within the target range, determining a weighting coefficient of the first target layer based on the feature similarity of the first target layer;

[0038] If it is determined that the feature similarity of the second target layer in the target model is not within the target range, the weighting coefficient of the second target layer is determined to be 0.

[0039] A model sparsity adjustment device, comprising:

[0040] An obtaining unit, configured to sequentially obtain at least two sets of sparsities, wherein each set of sparsities in the at least two sets of sparsities includes a preset value of the sparsity of each layer in the target model;

[0041] a determining unit, configured to perform crossover mutation on the at least two sparsities by a genetic algorithm to determine a target sparsity group that meets a condition, and determine the data in the target sparsity group as a determined value of the sparsity of each layer in the target model;

[0042] An adjustment unit is configured to adjust the sparsity of each layer in the target model based on the data in the target sparsity group. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 This is a flow chart of a model sparsity adjustment method disclosed in an embodiment of the present application;

[0045] Figure 2 This is a flow chart of a model sparsity adjustment method disclosed in an embodiment of the present application;

[0046] Figure 3 This is a flow chart of a model sparsity adjustment method disclosed in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of a training process of a sparsity accuracy prediction model disclosed in an embodiment of the present application;

[0048] Figure 5 This is a flow chart of a model sparsity adjustment method disclosed in an embodiment of the present application;

[0049] Figure 6 This is a flow chart of a model sparsity adjustment method disclosed in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of determining gradient similarity between a target layer of a target model after sparsity adjustment and a corresponding layer of the target model before sparsity adjustment, disclosed in an embodiment of the present application;

[0051] Figure 8 This is a structural diagram of a model sparsity adjustment device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0053] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0054] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0055] This application discloses a method for adjusting model sparsity, the flow chart of which is as follows: Figure 1 Shown, including:

[0056] Step S11, sequentially obtaining at least two sets of sparsities, wherein each of the at least two sets of sparsities includes a preset value of the sparsity of each layer in the target model;

[0057] Step S12: performing crossover mutation on at least two sparsities by a genetic algorithm to determine a target sparsity group that meets the conditions, and determining the data in the target sparsity group as the determined value of the sparsity of each layer in the target model;

[0058] Step S13: Adjust the sparsity of each layer in the target model based on the data in the target sparsity group.

[0059] Large language models (LLMs) pose significant deployment challenges due to their large number of parameters and heavy computational effort during inference. Model sparsification can set redundant parameters in the model to zero, reducing the computational and data handling pressure during inference.

[0060] Currently, model sparsification usually adopts the uniform pruning method, which thins out each layer in the model according to a preset fixed ratio sparsity. However, if this method is used to ensure that the model has a certain pruning rate, the accuracy of the model will be reduced.

[0061] The pruning rate is the ratio of the number of parameters remaining after pruning to the original number of parameters. A higher pruning rate means more redundant parameters are removed. Model accuracy can be expressed using perplexity, a metric that measures the quality of a language model's predictions. Lower perplexity indicates more accurate model predictions and higher model accuracy.

[0062] Taking the LLaMA-7b model as an example, if the uniform pruning method is used, when the pruning rate is 50%, the perplexity of the model is 7.26. As the pruning rate of the model increases, when the pruning rate reaches 60%, the perplexity of the model will increase to 10.70. When the pruning rate reaches 70%, the perplexity of the model is 85.77. That is, as the pruning rate of the model increases, the perplexity becomes higher and higher, that is, the model accuracy becomes lower and lower. It can be determined that when the model has a certain pruning rate, the model accuracy will be lower.

[0063] Of course, non-uniform pruning can also be used to achieve model sparsification. This involves first setting a base sparsity, then calculating an upper and lower range based on the pre-calculated percentage of outliers per layer, thereby determining the sparsity of each layer in the model. However, when using non-uniform pruning to achieve model sparsification, model accuracy can be significantly reduced when the model has a certain pruning rate.

[0064] Taking the LLaMA-2-7b model as an example, if the non-uniform pruning method is used, when the pruning rate is 50%, the perplexity of the model is 7.23. As the pruning rate of the model increases, when the pruning rate reaches 60%, the perplexity of the model will increase to 10.46. When the pruning rate reaches 70%, the perplexity of the model is 48.81. That is, as the pruning rate of the model increases, the perplexity becomes higher and higher, that is, the model accuracy becomes lower and lower. It can be determined that when the model has a certain pruning rate, the accuracy of the model will be lower, and the accuracy of the model after sparseness using the non-uniform pruning method is not significantly improved compared to the accuracy of the model after sparseness using the uniform pruning method.

[0065] Based on this, in this solution, the optimal sparsity of each layer in the target model is automatically determined by a genetic algorithm, and the target model is sparsified based on this. This ensures that no matter what the pruning rate of the target model is, the sparsity of each layer in the target model needs to be determined according to the genetic algorithm, so that the determined sparsity is more in line with the current situation of the target model, thereby improving the model accuracy of the target model after sparsification, and avoiding the problem of reduced model accuracy when the model has a certain pruning rate.

[0066] Specifically, at least two groups of sparsity are obtained in sequence, each group of sparsity includes multiple sparsity values, and each sparsity value serves as a preset value of the sparsity of a layer in the target model. For example, if the target model has a 30-layer fully connected network, then each group of sparsity has 30 sparsity values, and these 30 sparsity values ​​correspond to a layer of fully connected network in the target model.

[0067] Among them, the preset value of sparsity is the initially determined sparsity value, not the final value. The initially determined sparsity value needs to be adjusted by the genetic algorithm to obtain the final sparsity value, that is, the determined value of the sparsity of each layer in the target model.

[0068] When at least two groups of sparsity are obtained, it can be determined whether each group of sparsity meets the conditions. If each group of sparsity in the at least two groups of sparsity initially obtained does not meet the conditions, other sparsity groups can be determined based on the at least two groups of sparsity through a genetic algorithm, and finally a target sparsity group that meets the conditions is determined; if it is determined that a certain group of sparsity meets the conditions, then this group of sparsity can be directly determined as the target sparsity group.

[0069] In the process of determining a new sparsity group through a genetic algorithm, at least part of the sparsity groups can be selected from at least two sparsity groups to perform population crossover and population mutation to obtain at least one new sparsity group, and determine whether each new sparsity group meets the conditions, thereby determining the target sparsity group. The target sparsity group includes the determined value of the sparsity of each layer in the target model.

[0070] The process of performing population crossover on at least two groups of sparsity by genetic algorithm can be: selecting two groups of sparsity, and determining partial sparsity values ​​from each of the two groups of sparsity, and combining the determined partial sparsity values ​​as a new sparsity group, such as: for the first sparsity group and the second sparsity group, determining the first data corresponding to the first fully connected network layer of the target model in the first sparsity group, and using the first data as the sparsity value corresponding to the first fully connected layer of the target model in the third sparsity group; determining the second data corresponding to the second fully connected layer of the target model in the second sparsity group, and using the second data as the sparsity value corresponding to the second fully connected layer of the target model in the third sparsity group; determining the third data corresponding to the third fully connected layer of the target model in the second sparsity group, and using the third data as the sparsity value corresponding to the third fully connected layer of the target model in the third sparsity group, until every sparsity value in the third sparsity group is determined, and the third sparsity group is the new sparsity group;

[0071] Alternatively, multiple groups of sparsity are selected, and partial sparsity values ​​are determined from the multiple groups of sparsity respectively, and the determined partial sparsity value combinations are used as new sparsity groups, such as: determining the first data corresponding to the first fully connected network layer of the target model in the fifth sparsity group, and using the first data as the sparsity value corresponding to the first fully connected layer of the target model in the sixth sparsity group; determining the second data corresponding to the second fully connected layer of the target model in the first sparsity group, and using the second data as the sparsity value corresponding to the second fully connected layer of the target model in the sixth sparsity group; determining the third data corresponding to the third fully connected layer of the target model in the fourth sparsity group, and using the third data as the sparsity value corresponding to the third fully connected layer of the target model in the third sparsity group, until each sparsity value in the sixth sparsity group is determined, and the sixth sparsity group is the new sparsity group.

[0072] When performing crossover of data in sparsity groups through genetic algorithms, the crossover rate, that is, the proportion of sparsity values ​​selected from each sparsity group, can be set to avoid the situation where excellent data may be destroyed when the crossover rate is too high, and at the same time, avoid the situation where the diversity of sparsity values ​​is reduced when the crossover rate is too low.

[0073] When performing population mutation on the data in the sparsity group through a genetic algorithm, the sparsity value in the sparsity group obtained after the population crossover can be randomly changed, or each sparsity value in the sparsity group obtained after the population crossover can be changed in sequence, or, when each sparsity value in the sparsity group obtained after the population crossover is changed in sequence, each sparsity value is adjusted to a different size in sequence.

[0074] Among them, during the population mutation process, the mutation rate can be set to avoid the problem of insufficient diversity of sparsity values ​​when the mutation rate is too low, and at the same time, avoid the situation of degenerating into random search when the mutation rate is too high.

[0075] After determining the target sparsity group, the target model needs to be sparsified according to the data included in the target sparsity group, so that the sparsity of each layer in the target model obtained after the sparsification process conforms to the target sparsity group.

[0076] The model sparsity adjustment method disclosed in this embodiment first obtains at least two groups of sparsities, each group of sparsity includes a preset value of the sparsity of each layer in the target model, and then processes the at least two groups of sparsity through a genetic algorithm to obtain a target sparsity group that meets the conditions, and determines the data in the target sparsity group as the determined value of the sparsity of each layer in the target model, and adjusts the sparsity of each layer in the target model based on this, so as to automatically determine the sparsity of each layer in the target model through a genetic algorithm, ensuring that the sparsity of each layer in the target model is different under different circumstances, so as to improve the accuracy of the target model and avoid the problem of low accuracy of the target model when the sparsity of each layer in the target model is fixed.

[0077] This embodiment discloses a method for adjusting model sparsity, and its flow chart is as follows: Figure 2 Shown, including:

[0078] Step S21: sequentially obtain at least two sets of sparsities, wherein each of the at least two sets of sparsities includes a preset value of the sparsity of each layer in the target model;

[0079] Step S22: determining a first prediction accuracy of the target model, where the first prediction accuracy is the prediction accuracy of the target model when each of the at least two sparsities is applied to the target model;

[0080] Step S23: cross-mutate at least two sets of sparsity using a genetic algorithm to obtain at least one set of mutated sparsity;

[0081] Step S24: determining a second prediction accuracy of the target model, where the second prediction accuracy is the prediction accuracy of the target model when each set of variation sparsity in the at least one set of variation sparsity is applied to the target model;

[0082] Step S25: determining a target sparsity group that meets the conditions from the at least two sparsity groups and the at least one set of variant sparsity based on the first prediction accuracy and the second prediction accuracy, and determining the data in the target sparsity group as the determined value of the sparsity of each layer in the target model;

[0083] Step S26: Adjust the sparsity of each layer in the target model based on the data in the target sparsity group.

[0084] After obtaining at least two groups of sparsity, each group of sparsity can be used as a parent population. After determining the prediction accuracy corresponding to each group of sparsity, the parent population is subjected to population crossover and population mutation through a genetic algorithm to obtain a new population after mutation, that is, at least one group of mutation sparsity. The prediction accuracy corresponding to each group of mutation sparsity is then determined. Based on the prediction accuracy corresponding to each group of sparsity and the prediction accuracy corresponding to each group of mutation sparsity, a target sparsity group is selected from at least two groups of sparsity and at least one group of mutation sparsity, so as to control the target model to perform sparsity processing according to the target sparsity group.

[0085] Among them, determining the prediction accuracy corresponding to each set of sparsity is actually determining the prediction accuracy of the target model after applying each set of sparsity, that is, the first prediction accuracy, and each set of sparsity corresponds to a first prediction accuracy.

[0086] Afterwards, when performing crossover mutation on at least two sparsities using a genetic algorithm, the crossover mutation may be performed based on the first prediction accuracy, that is, the sparsity group requiring crossover mutation is selected from the at least two sparsities based on the first prediction accuracy.

[0087] Specifically, after determining the first prediction accuracy corresponding to each group of sparsity, multiple first prediction accuracies can be sorted to obtain a first sorting result, and a preset number of first prediction accuracies with higher prediction accuracy can be selected from the first sorting result, and the sparsity groups corresponding to the selected preset number of first prediction accuracies can be determined. These sparsity groups are determined as sparsity groups that need to undergo cross-mutation to ensure that the prediction accuracy corresponding to the sparsity group undergoing cross-mutation is higher, thereby avoiding the situation where the prediction accuracy corresponding to the mutated sparsity group obtained after cross-mutation is lower.

[0088] After obtaining at least one set of variation sparsity through a genetic algorithm, the prediction accuracy corresponding to each set of variation sparsity is determined. This is actually to determine the prediction accuracy of the target model after applying each set of variation sparsity, that is, the second prediction accuracy. Each set of variation sparsity corresponds to a second prediction accuracy.

[0089] After obtaining the second prediction accuracy, a target sparsity group can be determined directly from at least two sparsities and at least one variation sparsity group based on the multiple first prediction accuracies and the multiple second prediction accuracies. Specifically, the multiple first prediction accuracies and the multiple second prediction accuracies are sorted to obtain a second sorting result, the target prediction accuracy is determined from the second sorting result, and the sparsity group or variation sparsity group corresponding to the target prediction accuracy is determined as the target sparsity group. For example, the second sorting result is to sort the multiple first prediction accuracies and the second prediction accuracies in descending order of prediction accuracy, and select the prediction accuracy with the highest ranking from the second sorting result as the target prediction accuracy.

[0090] Alternatively, after obtaining the second prediction accuracy, multiple first prediction accuracies and multiple second prediction accuracies are sorted to obtain a second sorting result, and a preset number of first prediction accuracies and / or second prediction accuracies with higher prediction accuracy are selected from the second sorting result, and the sparsity groups and / or variant sparsity groups corresponding to the selected prediction accuracy continue to be cross-mutated to obtain a new variant sparsity group, and the third prediction accuracy corresponding to the new variant sparsity group is determined. Thereafter, the first prediction accuracy, the second prediction accuracy and the third prediction accuracy are sorted to obtain a third sorting result, and the target prediction accuracy is determined from the third sorting result, and the sparsity group or variant sparsity group or the new variant sparsity group corresponding to the target prediction accuracy is determined as the target sparsity group.

[0091] Alternatively, it can also be: after obtaining the first sorting result, select the prediction accuracy with the highest accuracy from the first sorting result, and then determine whether the prediction accuracy with the highest accuracy reaches the target threshold. If so, the prediction accuracy is determined as the target prediction accuracy, and the sparsity group corresponding to the target prediction accuracy is determined as the target sparsity group; if the target threshold is not reached, it is necessary to perform cross-mutation on the parent population through a genetic algorithm to obtain a variant sparsity group, determine the prediction accuracy corresponding to each variant sparsity group, and re-sort the prediction accuracies in the first sorting result and the prediction accuracy corresponding to each variant sparsity group to obtain a second sorting result, select the prediction accuracy with the highest accuracy from the second sorting result, and then determine whether the prediction accuracy with the highest accuracy reaches the target threshold. If so, the prediction accuracy is determined as the target prediction accuracy, and the target The sparsity group corresponding to the prediction accuracy is determined as the target sparsity group; if the target threshold is not reached, it is necessary to use a genetic algorithm to perform cross-mutation on each sparsity group and the mutation sparsity group to obtain a new mutation sparsity group, determine the prediction accuracy corresponding to each new mutation sparsity group, and re-sort the prediction accuracies in the first sorting result, the prediction accuracies in the second sorting result, and the prediction accuracy corresponding to each new mutation sparsity group to obtain a third sorting result, select the prediction accuracy with the highest accuracy from the third sorting result, and then determine whether the prediction accuracy with the highest accuracy reaches the target threshold. If so, the prediction accuracy is determined as the target prediction accuracy, and the sparsity group corresponding to the target prediction accuracy is determined as the target sparsity group. If not, it is necessary to continue to perform cross-mutation through the genetic algorithm until the prediction accuracy that reaches the target threshold is determined.

[0092] The model sparsity adjustment method disclosed in this embodiment, after obtaining at least two groups of sparsities, determines the prediction accuracy of the target model when applying each group of sparsities, and determines it as the first prediction accuracy, and cross-mutates the at least two groups of sparsities through a genetic algorithm to obtain at least one group of variant sparsities, determines the prediction accuracy of the target model when applying each group of variant sparsities, and determines it as the second prediction accuracy, and determines a target sparsity group that meets the conditions from at least two groups of sparsities and at least one group of variant sparsities based on the first prediction accuracy and the second prediction accuracy, so as to perform sparsification processing on the target model based on the target sparsity group, thereby ensuring that the target sparsity group finally determined is determined based on the prediction accuracy corresponding to different sparsity groups and variant sparsity groups, and ensuring the prediction accuracy of the target model after the sparsification processing based on the target sparsity group.

[0093] This embodiment discloses a method for adjusting model sparsity, and its flow chart is as follows: Figure 3 Shown, including:

[0094] Step S31, sequentially obtaining at least two sets of sparsities, wherein each of the at least two sets of sparsities includes a preset value of the sparsity of each layer in the target model;

[0095] Step S32: input each of the at least two sparsities into the sparsity accuracy prediction model to obtain a first prediction accuracy of the target model corresponding to each sparsity output by the sparsity accuracy prediction model;

[0096] Step S33: cross-mutate at least two sets of sparsity using a genetic algorithm to obtain at least one set of mutated sparsity;

[0097] Step S34: input each set of variation sparsity in the at least one set of variation sparsity into the sparsity accuracy prediction model, and obtain a second prediction accuracy of the target model corresponding to each set of variation sparsity output by the sparsity accuracy prediction model;

[0098] Step S35: determining a target sparsity group that meets the conditions from the at least two sparsity groups and the at least one variation sparsity group based on the first prediction accuracy and the second prediction accuracy, and determining the data in the target sparsity group as the determined value of the sparsity of each layer in the target model;

[0099] Step S36: Adjust the sparsity of each layer in the target model based on the data in the target sparsity group.

[0100] In the model sparsity adjustment method disclosed in this embodiment, the first prediction accuracy of the target model is determined, which can be achieved through the sparsity accuracy prediction model, that is, each group of sparsity in at least two groups of sparsity is input into the sparsity accuracy prediction model respectively, so as to obtain the prediction accuracy of the target model corresponding to each group of sparsity output by the sparsity accuracy prediction model; correspondingly, the second prediction accuracy of the target model can also be determined: through the sparsity accuracy prediction model, that is, each group of variant sparsity in at least one group of variant sparsity is input into the sparsity accuracy prediction model respectively, so as to obtain the prediction accuracy of the target model corresponding to each group of variant sparsity output by the sparsity accuracy prediction model respectively.

[0101] That is, a sparsity accuracy prediction model is pre-trained, and when determining each sparsity group, the sparsity accuracy prediction model is used to determine the prediction accuracy corresponding to each sparsity group, that is, the prediction accuracy of the target model after the target model is sparsely processed using each sparsity group.

[0102] Among them, the sparsity accuracy prediction model can be obtained through the following training methods:

[0103] A training sparsity group is obtained and a training sample is used when the training sparsity group is used as the sparsity of each layer in the target model; the target model is sparsified based on the training sparsity and a predetermined sparsity method; the sparse prediction accuracy of the target model after the sparsification is determined; a model is trained based on each sparse prediction accuracy and the training sample among the multiple sparse prediction accuracy to obtain a sparsity accuracy prediction model.

[0104] A sparsity search space, i.e., a data set, can be predefined, and the sparsity search space includes multiple sparsity values, such as 20%, 30%, 40%, 50%, 60%, 70%, etc. When the sparsity accuracy prediction model needs to be trained, a certain number of sparsity values ​​can be determined from the sparsity search space, and the determined certain number of sparsity values ​​can be directly determined as a training sparsity group. The certain number is related to the number of fully connected network layers included in the target model. For example, if the target model includes 30 fully connected network layers, 30 sparsity values ​​need to be determined from the sparsity search space, and each sparsity value corresponds to a fully connected network layer in the target model. The sparsity values ​​corresponding to different layers in the target model can be the same or different.

[0105] In addition, it is also necessary to determine the training samples. The training samples are the training data required for training the sparsity accuracy prediction model. That is, no matter which group of sparsity groups are trained, the training samples need to be used to perform the training process when the sparsity accuracy prediction model is trained.

[0106] Secondly, the sparsification method needs to be determined. The sparsification method for the target model can be determined in advance, such as whether to use uniform pruning method or non-uniform pruning method for sparsification.

[0107] Afterwards, it is necessary to perform sparsification processing on the target model based on the training sparsity and a predetermined sparsity method. Since the training sparsity may include at least one set of sparsity values ​​for each layer in the target model, the target model is sparsified according to a set of sparsity values ​​in the training sparsity according to a predetermined sparsity method to obtain the target model after sparsification processing. At this time, it is necessary to determine the sparse prediction accuracy of the target model after the current sparsification processing, and the corresponding relationship between a set of sparsity values ​​and the sparse prediction accuracy can be obtained; when the training sparsity includes multiple sets of sparsity values, the target model can be sparsified according to a predetermined sparsity method for each set of sparsity values ​​in the multiple sets of sparsity values ​​to obtain the target model after sparsification processing, thereby determining the corresponding relationship between each set of sparsity values ​​and the sparse prediction accuracy, that is, obtaining the corresponding relationship between multiple sparsity value groups and sparse prediction accuracy.

[0108] Among them, in the process of sparsifying the target model according to a predetermined sparsity method based on each set of sparsity values, a calibration dataset (Calibration) needs to be introduced to evaluate the importance of the target model weights in the sparsification process of the target model, guide the sparsification decision, and provide an optimization basis for subsequent recovery, so as to maintain the performance of the target model while reducing computing resources.

[0109] In addition, when determining the sparse prediction accuracy of the current target model after sparsification, an evaluation data set can be introduced, and the prediction accuracy of the target model after sparsification can be evaluated through the evaluation data set, or the prediction accuracy of the target model after sparsification and accuracy restoration can be evaluated.

[0110] The prediction model is trained using the correspondence between multiple sparsity value groups and sparse prediction accuracy and training samples, thereby obtaining a trained sparsity accuracy prediction model. The prediction model is pre-built and can be a model including a 3-5 layer fully connected network.

[0111] During the training process of the sparsity precision prediction model, select some samples from the training sample as the training set, and the remaining samples from the training sample as the validation set. The trained sparsity precision prediction model is then validated to ensure the accuracy of the sparsity precision prediction model in precision prediction. For example, 70% of the data in the training sample is used as the training set, and the remaining 30% of the data in the training sample excluding the training set is used as the validation set.

[0112] The training process of the above sparsity accuracy prediction model can be as follows Figure 4 As shown, it includes: determining a training sparsity group; determining a training sample; performing sparsification processing on a target model based on the training sparsity and a predetermined sparsity method; introducing a calibration data set during the sparsification process; determining the sparse prediction accuracy of the target model after the sparsification process, and obtaining a correspondence between multiple sparsity value groups and sparse prediction accuracy; introducing an evaluation data set during the process of determining the sparse prediction accuracy; constructing a prediction model; using the correspondence between multiple sparsity value groups and sparse prediction accuracy and the training samples to train the prediction model, and obtain a sparsity accuracy prediction model.

[0113] The model sparsity adjustment method disclosed in this embodiment, after obtaining at least two groups of sparsities, uses a pre-trained sparsity accuracy prediction model to determine the prediction accuracy of the target model when applying each group of sparsities, and determines it as the first prediction accuracy, and cross-mutates the at least two groups of sparsities through a genetic algorithm to obtain at least one group of variant sparsities, and uses the sparsity accuracy prediction model to determine the prediction accuracy of the target model when applying each group of variant sparsities, and determines it as the second prediction accuracy, and based on the first prediction accuracy and the second prediction accuracy, determines a target sparsity group that meets the conditions from the at least two groups of sparsities and at least one group of variant sparsities, so as to perform sparsification processing on the target model based on the target sparsity group, so as to ensure that the target sparsity group finally determined is determined based on the prediction accuracy corresponding to different sparsity groups and variant sparsity groups, and ensures the prediction accuracy of the target model after the sparsification processing based on the target sparsity group; in addition, this embodiment determines the prediction accuracy corresponding to each group of sparsities through the sparsity accuracy prediction model, thereby ensuring the accuracy of the prediction accuracy determination.

[0114] This embodiment discloses a method for adjusting model sparsity, and its flow chart is as follows: Figure 5 Shown, including:

[0115] Step S51: sequentially obtain at least two sets of sparsities, wherein each of the at least two sets of sparsities includes a preset value of the sparsity of each layer in the target model;

[0116] Step S52: determining the outlier distribution data of each layer in the target model;

[0117] Step S53: adjusting the preset value of the sparsity of the corresponding layer in each of the at least two sparsity groups based on the outlier distribution data of each layer, so that the preset value of the sparsity of each layer in each sparsity group conforms to a specific relationship with the outlier distribution data of the corresponding layer;

[0118] Step S54: cross-mutate at least two sparsities using a genetic algorithm to determine a target sparsity group that meets the conditions, and determine the data in the target sparsity group as the determined value of the sparsity of each layer in the target model;

[0119] Step S55: Adjust the sparsity of each layer in the target model based on the data in the target sparsity group.

[0120] The outlier distribution data of each layer in the target model, that is, the LOD (Layerwise Outlier Distribution) of each layer in the target model, is determined. Its purpose is to identify outliers in the weights or activation values ​​of each layer in the target model, that is, outliers, to provide a decision basis for subsequent sparsification and avoid blind pruning that affects the prediction accuracy of the target model.

[0121] The abnormal distribution of each layer in the target model is mainly caused by the interaction between training dynamics and architectural characteristics, such as: unbalanced gradient dynamics, sparse activation preference of the attention mechanism, nonlinear amplification of the activation function, data defect propagation, etc.

[0122] By analyzing the distribution characteristics of the weights of each layer in the target model (such as mean, standard deviation, skewness, etc.), we can determine the outlier weights that have a significant impact on the output of the target model (such as parameters whose absolute values ​​are significantly larger than other weights), that is, the outlier distribution data.

[0123] After determining the outlier distribution data of each layer of the target model, the preset value of the sparsity of the corresponding layer in each group of at least two groups of sparsity is adjusted based on the outlier distribution data of each layer, that is, each group of sparsity in the at least two groups of sparsity is initialized so that the sensitivity of each layer in the target model to pruning is quantified through the outlier distribution data LOD, so that each group of sparsity after initialization is more consistent with the distribution of each layer within the target model.

[0124] The LOD (Outlier Detection) data quantifies the proportion of outliers in the weights of each layer in the target model. A higher LOD indicates that the layer in the target model is more sensitive to pruning. For example, a layer with a higher LOD (i.e., a layer with an LOD value close to 1) has more outlier weights. In this case, more parameters need to be retained, which means the sparsity needs to be reduced. In other words, the target model should have a low sparsity for this layer, such as one that can be controlled to approach 0. A layer with a lower LOD (i.e., a layer with an LOD value close to 0) has fewer outlier weights and can withstand higher pruning. This means that fewer parameters do not need to be retained, and the sparsity can be increased. In other words, the target model should have a high sparsity for this layer, such as one that can be controlled to approach 1.

[0125] In this embodiment, the preset value of the sparsity of the corresponding layer in each group of sparsity is adjusted so that the preset value of the sparsity of each layer in each group of sparsity conforms to a specific relationship with the abnormal distribution data of the corresponding layer, that is, the higher the LOD of the abnormal distribution data, the lower the preset value of the sparsity, and the lower the LOD of the abnormal distribution data, the higher the preset value of the sparsity, that is, the LOD of the abnormal distribution data is inversely proportional to the preset value of the sparsity.

[0126] Specifically, it can be: determining the target value corresponding to each layer in the target model, where the target value corresponding to each layer is the difference between a specific value and the outlier distribution value of each layer; adjusting the preset value of the sparsity of each layer in each group of sparsity so that the preset value of the sparsity of each layer in each group of sparsity after adjustment is proportional to the target value of the corresponding layer.

[0127] Among them, in the model sparsity adjustment method disclosed in this embodiment, the specific value can be specifically set to 1. After determining the difference between the target value of each layer being 1 and the abnormal distribution data LOD of the layer, the preset value of the sparsity of each layer in each set of sparsity is adjusted based on the difference, so that the preset value of the sparsity of each layer in each set of sparsity after adjustment is proportional to the target value of the corresponding layer. Since the target value is the difference between 1 and the abnormal distribution data LOD, the preset value of the sparsity is proportional to the target value, that is, the abnormal distribution data LOD is inversely proportional to the preset value of the sparsity.

[0128] In this embodiment, each group of sparsity is initialized so that the abnormal distribution data LOD is inversely proportional to the preset value of the sparsity, thereby ensuring that the value of the sparsity group after initialization conforms to the distribution of the abnormal distribution data, thereby ensuring the quality of the sparsity group after initialization.

[0129] The model sparsity adjustment method disclosed in this embodiment, after obtaining at least two sets of sparsity, needs to determine the outlier distribution data of each layer in the target model, and adjust the preset value of the sparsity of the corresponding layer in each set of at least two sets of sparsity based on the outlier distribution data of each layer, so that the preset value of the sparsity of each layer in each set of sparsity conforms to a specific relationship with the outlier distribution data of the corresponding layer, thereby ensuring the quality of the preset value of the sparsity of the corresponding layer in each set of at least two sets of sparsity after adjustment, and further improving the prediction accuracy of the target model after sparsification.

[0130] This embodiment discloses a method for adjusting model sparsity, and its flow chart is as follows: Figure 6 Shown, including:

[0131] Step S61: sequentially obtain at least two sets of sparsities, wherein each of the at least two sets of sparsities includes a preset value of the sparsity of each layer in the target model;

[0132] Step S62: cross-mutate at least two sparsities using a genetic algorithm to determine a target sparsity group that meets the conditions, and determine the data in the target sparsity group as the determined value of the sparsity of each layer in the target model;

[0133] Step S63: adjusting the sparsity of each layer in the target model based on the data in the target sparsity group;

[0134] Step S64: Determine, by means of knowledge distillation, the gradient information of each target layer in the target model after sparsity adjustment and the gradient similarity between the gradient information of the corresponding layer in the target model before sparsity adjustment, wherein the target layer represents the layer for low-rank decomposition in the target model after sparsity adjustment;

[0135] Step S65: Adjust the loss function of the target model after adjusting the sparsity based on the gradient similarity of each target layer in the target model, so as to update the parameters of each target layer in the target model after adjusting the sparsity based on the loss function.

[0136] Knowledge Distillation (KD) is a machine learning technique that transfers the knowledge of a large, complex model (the teacher model) to a small, lightweight model (the student model) to achieve model compression and acceleration while preserving the performance of the original model as much as possible. The core idea is to enable the student model to gain stronger generalization capabilities by mimicking the output distribution or intermediate features of the teacher model.

[0137] In this embodiment, the teacher model corresponds to the target model before sparsification, and the student model corresponds to the target model after sparsification. The target model before sparsification has a large number of parameters and high precision, but also has a correspondingly high inference cost. The target model after sparsification has a streamlined structure and a small number of parameters. Its goal is to achieve performance close to that of the teacher model under the guidance of the teacher model, while significantly reducing the computing resource requirements.

[0138] The target model includes a target layer, which is a layer in the target model that performs low-rank decomposition, namely, a LoRA (Low-Rank Adaptation) layer. In this embodiment, not only is the target model sparsified, but low-rank decomposition is also performed on the target model. It can be determined that the target model before sparsification is a model that has not undergone sparsification and low-rank decomposition, while the target model after sparsification is a model that has undergone sparsification and low-rank decomposition. Therefore, the target model after sparsification includes the target layer.

[0139] The LoRA layer is the layer in the target model after being processed by the LoRA method. The LoRA method is a method for efficient parameter fine-tuning. Its core is to use low-rank matrix decomposition technology to update only a small number of new parameters during the fine-tuning process instead of adjusting the original weights of the entire model, thereby significantly reducing computing resource requirements and training costs.

[0140] In practical applications, the target model can be first processed by the LoRA method to determine the target layer, and then the processed target model can be sparsely processed to obtain the sparsely processed target model; or, in practical applications, the target layer in the target model can be first determined, and then the target layer can be processed by the LoRA method synchronously during the process of sparse processing of the target model.

[0141] After the target model is subjected to sparse processing and LoRA processing, the target model before adjusting the sparsity and the target model after adjusting the sparsity are obtained. The target model before adjusting the sparsity is the target model that has not been subjected to sparse processing and LoRA processing, and the target model after adjusting the sparsity is the target model that has been subjected to sparse processing and LoRA processing.

[0142] After performing sparsification and LoRA processing on the target model, the gradient information of each target layer in the target model after sparsification and the gradient information of the corresponding layer in the target model before sparsification are determined respectively. For example: if the target layer is determined to be the third and fifth layers of the target model based on the target model after sparsification, then when determining the gradient information, only the gradient information of the third and fifth layers of the target model after sparsification and the gradient information of the third and fifth layers of the target model before sparsification are determined. For other layers in the target model except the target layer, whether it is the target model before or after sparsification, it is not necessary to determine the gradient information of other layers to avoid resource loss.

[0143] Among them, the gradient information of the target layer in the target model refers to the vector or matrix composed of the partial derivatives of the loss function with respect to all trainable parameters (such as weights and biases) of the target layer, which quantifies the rate of change and direction of the loss function when the parameters of the target layer change slightly.

[0144] After obtaining the gradient information of each target layer in the target model after sparsification and the gradient information of the corresponding layer in the target model before sparsification, it is necessary to calculate the cosine similarity of the two gradient information, that is, the gradient similarity, that is, each target layer obtains a gradient similarity, and based on the gradient similarity of each target layer, the loss function of the target model after adjusting the sparsity is adjusted to achieve the update of the parameters of each target layer in the target model after adjusting the sparsity based on the adjusted loss function. Through the calculation of the gradient similarity, the real-time measurement of the degree of gradient alignment between the target model before sparsification and the target model after sparsification is achieved to prevent the target model after sparsification from deviating from the optimization trajectory of the target model before sparsification, thereby achieving the purpose of restoring the prediction accuracy of the target model after adjusting the sparsity.

[0145] When the gradient similarity of a certain target layer approaches 1, it indicates that the gradient direction of the target layer in the target model after sparsification is consistent with the gradient direction of the target layer in the target model before sparsification. At this time, the loss of knowledge distillation approaches 0; when the gradient similarity of a certain target layer approaches -1, it indicates that the gradient direction of the target layer in the target model after sparsification is inconsistent with the gradient direction of the target layer in the target model before sparsification. At this time, the loss of knowledge distillation is large, and the learning direction of the target model after sparsification needs to be corrected to avoid the deviation of the optimization path caused by sparsification.

[0146] like Figure 7 As shown, it is a schematic diagram of determining gradient similarity based on the target layer of the target model after adjusting the sparsity and the corresponding layer of the target model before adjusting the sparsity, including: the target model before adjusting the sparsity (teacher model) and the target model after adjusting the sparsity (student model), performing knowledge distillation on the teacher model, transferring the minable knowledge to the student model, determining the layer for low-rank decomposition in the student model, that is, the target layer, determining all target layers in the student model (such as: the first target layer, the second target layer, the i-th target layer, the i+1-th target layer, the L-th target layer), Figure 7 There are L target layers in the student model shown in , where i is a positive integer between 1 and L, and L is a positive integer greater than 1. Determine the target layer in the teacher model corresponding to each target layer of the student model, and calculate the gradient information of each target layer in the teacher model and the gradient information of each target layer in the student model. Calculate the cosine similarity of the gradient information of each target layer in the teacher model and the gradient information of the corresponding target layer in the student model, that is, the gradient similarity, where Figure 7 middle is the knowledge distillation loss gradient, is the loss gradient of the downstream task, that is, the loss of the student model on the actual task, such as the loss of classification cross entropy and regression mean square error, which is independent of the teacher model. Figure 7 middle and The angle between them represents the optimization direction coordination between the knowledge distillation loss gradient and the downstream task loss gradient. The smaller the angle, the higher the synergy, and the larger the angle, the lower the synergy.

[0147] Furthermore, in the model sparsity adjustment method disclosed in this embodiment, the loss function of the target model after the sparsity adjustment is adjusted based on the gradient similarity of each target layer in the target model, which can be specifically:

[0148] Determine a weighting coefficient; perform weighted summation of the gradient similarities of each target layer in the target model based on the weighting coefficient to determine the loss data of the target model for adjusting the sparsity; and adjust the loss function of the target model after adjusting the sparsity based on the loss data of the target model for adjusting the sparsity.

[0149] When adjusting the loss function of the target model after adjusting the sparsity based on the gradient similarity of each target layer in the target model, it is necessary to perform a weighted summation of the gradient similarity of each target layer to obtain the loss data caused by adjusting the sparsity of the target model, that is, the loss value of the target model after knowledge distillation, which can be specifically:

[0150]

[0151] in, is the loss of the target model after knowledge distillation, L is the total number of target layers in the target model, is the weight coefficient of the i-th target layer, which is used to control the i-th target layer in The contribution of is the distillation loss of the i-th target layer (i.e., the gradient similarity of the i-th target layer), is the first feature output vector of the i-th target layer in the target model before sparsification, is the second feature output vector of the i-th target layer in the sparsified target model.

[0152] That is, a corresponding weighting coefficient is set for the gradient similarity of each target layer, and the gradient similarity of each target layer is multiplied by its corresponding weighting coefficient to obtain the product corresponding to each target layer. The products corresponding to each target layer are added together to obtain the total loss of knowledge distillation for the target model, that is, .

[0153] The total loss of knowledge distillation of the target model is calculated After that, based on the Adjust the loss function of the target model after adjusting the sparsity, that is, determine the loss of the target model when performing LoRA processing, that is, obtain the loss of the target model when only LoRA processing is performed without sparsification, and then increase the total loss of the target model for knowledge distillation based on this loss. , the sum obtained is the loss function of the target model after adjusting the sparsity, thereby achieving the recovery of the prediction accuracy of the target model.

[0154] The weighting coefficient can be determined as follows:

[0155] Determine the first feature output vector of each target layer in the target model before adjusting the sparsity, and the second feature output vector of each target layer in the target model after adjusting the sparsity; determine the feature similarity between the first feature output vector and the second feature output vector of each target layer in the target model; if it is determined that the feature similarity of the first target layer in the target model is within the target range, determine the weighting coefficient of the first target layer based on the feature similarity of the first target layer; if it is determined that the feature similarity of the second target layer in the target model is not within the target range, determine the weighting coefficient of the second target layer to be 0.

[0156] Being within the target range may be specifically defined as being greater than 0; not being within the target range may be specifically defined as being less than or equal to 0.

[0157] The weighting coefficient can be determined as follows:

[0158]

[0159] Among them, the first feature output vector of each target layer in the target model before adjusting the sparsity is , that is, the first feature output vector of the i-th target layer in the target model before sparsification; the second feature output vector of each target layer in the target model after adjusting the sparsity is , that is, the second feature output vector of the i-th target layer in the sparsified target model; is the feature similarity between the first feature output vector of the i-th target layer and the second feature output vector of the i-th target layer.

[0160] That is, when the feature similarity between the first feature output vector of the i-th target layer and the second feature output vector of the i-th target layer is greater than 0, the feature similarity is directly determined as the weighting coefficient of the i-th target layer. At this time, the angle between the first feature output vector of each target layer in the target model before adjusting the sparsity and the second feature output vector of each target layer in the target model after adjusting the sparsity is less than 90 degrees, and the weighting coefficient of the current target layer decreases as the angle increases;

[0161] When the feature similarity between the first feature output vector of the i-th target layer and the second feature output vector of the i-th target layer is less than or equal to 0, the weighting coefficient of the i-th target layer is directly determined to be 0. That is, at this time, the i-th target layer does not contribute to the distillation loss. At this time, the angle between the first feature output vector of each target layer in the target model before adjusting the sparsity and the second feature output vector of each target layer in the target model after adjusting the sparsity is greater than 90 degrees.

[0162] The model sparsity adjustment method disclosed in this embodiment, after cross-mutating at least two groups of sparsities through a genetic algorithm to obtain a target sparsity group, and adjusting the sparsity of each layer in the target model based on the target sparsity group, can determine the gradient information of each target layer in the target model after adjusting the sparsity and the gradient similarity between the gradient information of the corresponding layer in the target model before adjustment through knowledge distillation, where the target layer represents the layer that undergoes low-rank decomposition in the target model after the sparsity adjustment; and adjust the loss function of the target model after the sparsity adjustment based on the gradient similarity, so as to update the parameters of each target layer in the target model after the sparsity adjustment based on the loss function, so as to restore the accuracy of the target model after the sparsification processing, so as to ensure the prediction accuracy of the target model.

[0163] This embodiment discloses a model sparsity adjustment device, the structural diagram of which is shown in FIG. Figure 8 Shown, including:

[0164] An obtaining unit 81 , a determining unit 82 and an adjusting unit 83 .

[0165] The obtaining unit 81 is used to sequentially obtain at least two sets of sparsities, wherein each set of sparsities in the at least two sets of sparsities includes a preset value of the sparsity of each layer in the target model;

[0166] The determining unit 82 is configured to perform crossover mutation on at least two sparsities by a genetic algorithm to determine a target sparsity group that meets the conditions, and determine the data in the target sparsity group as the determined value of the sparsity of each layer in the target model;

[0167] The adjustment unit 83 is used to adjust the sparsity of each layer in the target model based on the data in the target sparsity group.

[0168] Furthermore, the determination unit is used to:

[0169] Determine a first prediction accuracy of the target model, where the first prediction accuracy is the prediction accuracy of the target model when each of at least two sparsities in the target model is applied; cross-mutate the at least two sparsities through a genetic algorithm to obtain at least one set of variant sparsities; determine a second prediction accuracy of the target model, where the second prediction accuracy is the prediction accuracy of the target model when each of at least one set of variant sparsities in the target model is applied; and determine a target sparsity group that meets the conditions from the at least two sparsities and the at least one set of variant sparsities based on the first prediction accuracy and the second prediction accuracy.

[0170] Furthermore, the determination unit is used to:

[0171] Each of the at least two sparsities is input into the sparsity accuracy prediction model to obtain the prediction accuracy of the target model corresponding to each sparsity output by the sparsity accuracy prediction model.

[0172] Furthermore, the sparsity accuracy prediction model is obtained through the following training methods, including:

[0173] A training sparsity group is obtained and a training sample is used when the training sparsity group is used as the sparsity of each layer in the target model; the target model is sparsified based on the training sparsity and a predetermined sparsity method; the sparse prediction accuracy of the target model after the sparsification is determined; and a prediction model is trained based on each sparse prediction accuracy and the training sample in a plurality of sparse prediction accuracies to obtain a sparsity accuracy prediction model.

[0174] Furthermore, the model sparsity adjustment device disclosed in this embodiment may further include:

[0175] An initialization unit is used to determine the outlier distribution value of each layer in the target model; based on the outlier distribution data of each layer, a preset value of the sparsity of the corresponding layer in each group of sparsity in at least two groups of sparsity is adjusted so that the preset value of the sparsity of each layer in each group of sparsity conforms to a specific relationship with the outlier distribution data of the corresponding layer.

[0176] Furthermore, the initialization unit is used to:

[0177] Determine a target value corresponding to each layer in the target model, where the target value corresponding to each layer is the difference between a specific value and a numerical value of an outlier distribution of each layer; and adjust a preset value of the sparsity of each layer in each set of sparsity so that the preset value of the sparsity of each layer in each set of sparsity after adjustment is proportional to the target value of the corresponding layer.

[0178] Furthermore, the model sparsity adjustment device disclosed in this embodiment may further include:

[0179] The precision recovery unit is used to determine the gradient information of each target layer in the target model after sparsity adjustment and the gradient similarity between the gradient information of the corresponding layer in the target model before sparsity adjustment by means of knowledge distillation, wherein the target layer represents the layer that undergoes low-rank decomposition in the target model after sparsity adjustment; based on the gradient similarity of each target layer in the target model, the loss function of the target model after sparsity adjustment is adjusted, so as to update the parameters of each target layer in the target model after sparsity adjustment based on the loss function.

[0180] Furthermore, the precision recovery unit is used to:

[0181] Determine the weighting coefficient;

[0182] Based on the weighting coefficient, the gradient similarity of each target layer in the target model is weighted and summed to determine the loss data of the target model for adjusting the sparsity; based on the loss data of the target model for adjusting the sparsity, the loss function of the target model after adjusting the sparsity is adjusted.

[0183] Furthermore, the precision recovery unit is used to:

[0184] Determine the first feature output vector of each target layer in the target model before adjusting the sparsity, and the second feature output vector of each target layer in the target model after adjusting the sparsity; determine the feature similarity between the first feature output vector and the second feature output vector of each target layer in the target model; if it is determined that the feature similarity of the first target layer in the target model is within the target range, determine the weighting coefficient of the first target layer based on the feature similarity of the first target layer; if it is determined that the feature similarity of the second target layer in the target model is not within the target range, determine the weighting coefficient of the second target layer to be 0.

[0185] The model sparsity adjustment device disclosed in this embodiment is implemented based on the model sparsity adjustment method disclosed in the above embodiment, and will not be described in detail here.

[0186] The model sparsity adjustment device disclosed in this embodiment first obtains at least two groups of sparsities, each group of sparsity includes a preset value of the sparsity of each layer in the target model, and then processes the at least two groups of sparsity through a genetic algorithm to obtain a target sparsity group that meets the conditions, and determines the data in the target sparsity group as the determined value of the sparsity of each layer in the target model, and adjusts the sparsity of each layer in the target model based on this, so as to automatically determine the sparsity of each layer in the target model through a genetic algorithm, ensuring that the sparsity of each layer in the target model is different under different circumstances, so as to improve the accuracy of the target model and avoid the problem of low accuracy of the target model when the sparsity of each layer in the target model is fixed.

[0187] This embodiment discloses an electronic device, comprising at least a processor and a memory, wherein:

[0188] The processor is configured to: sequentially obtain at least two sets of sparsities, wherein each of the at least two sets of sparsities includes a preset value of the sparsity of each layer in the target model; perform crossover mutation on the at least two sets of sparsities through a genetic algorithm to determine a target sparsity group that meets a condition, and determine the data in the target sparsity group as a determined value of the sparsity of each layer in the target model; and adjust the sparsity of each layer in the target model based on the data in the target sparsity group;

[0189] The memory is used to store the programs required by the processor to execute the above processing procedures.

[0190] The electronic device disclosed in this embodiment is implemented based on the model sparsity adjustment method disclosed in the above embodiment, which will not be described in detail here.

[0191] The electronic device disclosed in this embodiment first obtains at least two groups of sparsities, each group of sparsities includes a preset value of the sparsity of each layer in the target model, and then processes the at least two groups of sparsities through a genetic algorithm to obtain a target sparsity group that meets the conditions, and determines the data in the target sparsity group as the determined value of the sparsity of each layer in the target model, and adjusts the sparsity of each layer in the target model accordingly to automatically determine the sparsity of each layer in the target model through a genetic algorithm, ensuring that the sparsity of each layer in the target model is different under different circumstances, so as to improve the accuracy of the target model and avoid the problem of low accuracy of the target model when the sparsity of each layer in the target model is fixed.

[0192] An embodiment of the present application also provides a readable storage medium on which a computer program is stored. The computer program is loaded and executed by a processor to implement the various steps of the above-mentioned model sparsity adjustment method. The specific implementation process can refer to the description of the corresponding part of the above-mentioned embodiment, which will not be repeated in this embodiment.

[0193] The present application also proposes a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in various optional implementations of the above-mentioned model sparsity adjustment method or model sparsity adjustment device. The specific implementation process can be referred to the description of the corresponding embodiment above and is not repeated here.

[0194] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0195] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0196] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0197] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A method for adjusting model sparsity, comprising: Obtaining at least two sets of sparsities in sequence, each of the at least two sets of sparsities including a preset value of the sparsity of each layer in the target model; Performing crossover mutation on the at least two sparsities by a genetic algorithm to determine a target sparsity group that meets a condition, and determining the data in the target sparsity group as a determined value of the sparsity of each layer in the target model; The sparsity of each layer in the target model is adjusted based on the data in the target sparsity group.

2. The method according to claim 1, wherein the performing crossover mutation on the at least two sets of sparsities by a genetic algorithm to determine a target data set that meets a condition comprises: determining a first prediction accuracy of the target model, the first prediction accuracy being a prediction accuracy of the target model when each of the at least two sets of sparsities is applied to the target model; Performing crossover mutation on the at least two sets of sparsities by a genetic algorithm to obtain at least one set of mutated sparsities; determining a second prediction accuracy of the target model, the second prediction accuracy being a prediction accuracy of the target model when each set of variation sparsity in the at least one set of variation sparsity is applied to the target model; A target sparsity group that meets a condition is determined from the at least two sparsity groups and the at least one set of variation sparsity based on the first prediction accuracy and the second prediction accuracy.

3. The method according to claim 2, wherein determining the first prediction accuracy of the target model comprises: Each of the at least two groups of sparsities is input into the sparsity accuracy prediction model to obtain the prediction accuracy of the target model corresponding to each group of sparsities output by the sparsity accuracy prediction model.

4. The method according to claim 3, wherein the sparsity accuracy prediction model is obtained by the following training method, including: Obtaining a training sparsity group and using the training sparsity group as a training sample for the sparsity of each layer in the target model; Based on the training sparsity and a predetermined sparsification method, performing sparsification processing on the target model; Determine the sparse prediction accuracy of the target model after the sparsification process; The prediction model is trained based on each sparse prediction accuracy among the multiple sparse prediction accuracies and the training samples to obtain a sparsity accuracy prediction model.

5. The method according to claim 1, further comprising: Determining the outlier distribution value of each layer in the target model; Based on the outlier distribution data of each layer, the preset value of the sparsity of the corresponding layer in each of the at least two groups of sparsity is adjusted so that the preset value of the sparsity of each layer in each group of sparsity conforms to a specific relationship with the outlier distribution data of the corresponding layer.

6. The method according to claim 5, wherein adjusting the preset value of the sparsity of the corresponding layer in each of the at least two sparsities based on the outlier distribution data of each layer so that the preset value of the sparsity of each layer in each of the sparsities conforms to a specific relationship with the outlier distribution data of the corresponding layer comprises: Determine a target value corresponding to each layer in the target model, wherein the target value corresponding to each layer is the difference between a specific value and a value of an outlier distribution of each layer; The preset value of the sparsity of each layer in each set of sparsity is adjusted so that the preset value of the sparsity of each layer in each set of sparsity after adjustment is proportional to the target value of the corresponding layer.

7. The method according to claim 1, further comprising: Determine the gradient similarity between the gradient information of each target layer in the target model after sparsity adjustment and the gradient information of the corresponding layer in the target model before sparsity adjustment by knowledge distillation, wherein the target layer represents the layer for low-rank decomposition in the target model after sparsity adjustment; The loss function of the target model after adjusting the sparsity is adjusted based on the gradient similarity of each target layer in the target model, so as to update the parameters of each target layer in the target model after adjusting the sparsity based on the loss function.

8. The method according to claim 7, wherein adjusting the loss function of the target model after adjusting the sparsity based on the gradient similarity of each target layer in the target model comprises: Determine the weighting coefficient; Performing weighted summation on the gradient similarity of each target layer in the target model based on the weighting coefficient to determine loss data for adjusting sparsity of the target model; The loss function of the target model after adjusting the sparsity is adjusted based on the loss data of the target model.

9. The method according to claim 8, wherein determining the weighting coefficient comprises: Determining a first feature output vector of each target layer in the target model before adjusting the sparsity, and a second feature output vector of each target layer in the target model after adjusting the sparsity; Determining a feature similarity between a first feature output vector and a second feature output vector of each target layer in the target model; If it is determined that the feature similarity of the first target layer in the target model is within the target range, determining a weighting coefficient of the first target layer based on the feature similarity of the first target layer; If it is determined that the feature similarity of the second target layer in the target model is not within the target range, the weighting coefficient of the second target layer is determined to be 0.

10. A model sparsity adjustment device, comprising: An obtaining unit, configured to sequentially obtain at least two sets of sparsities, wherein each set of sparsities in the at least two sets of sparsities includes a preset value of the sparsity of each layer in the target model; a determining unit, configured to perform crossover mutation on the at least two sparsities by a genetic algorithm to determine a target sparsity group that meets a condition, and determine the data in the target sparsity group as a determined value of the sparsity of each layer in the target model; An adjustment unit is configured to adjust the sparsity of each layer in the target model based on the data in the target sparsity group.

Images