Neural network model cutting method and device, electronic equipment and storage medium

Abstract

The invention provides a neural network model cutting method and device, electronic equipment and a storage medium, and the method comprises the steps: dividing a to-be-cut neural network model into a plurality of structure groups according to the structural characteristics of the to-be-cut neural network model; according to the change of a loss function before and after the to-be-clipped neural network model is clipped, determining a target clipping rate for clipping each structure group in the plurality of structure groups; wherein the change of the loss function is determined according to a first factor and a second factor, the first factor is used for representing the importance of each weight in each structure group, and the second factor is used for representing the interaction among different structure groups; and according to the target cutting rate, cutting each structure group to obtain a neural network model after cutting. The method can accelerate the compression of the neural network model under the condition of small performance loss.

Description

Technical Field

[0001] The present application relates to the field of neural network compression, and in particular to a neural network model clipping method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, the application of performing intelligent tasks based on neural network models, such as intelligent detection (such as vehicle and license plate detection, etc.), is becoming more and more popular. Since the computing resources of terminal devices are usually limited, if the structure of the neural network model is too complex, the demand for computing resources will be too large, resulting in the terminal device performing intelligent tasks based on the neural network model at a relatively slow speed and poor real-time performance. Therefore, in order to improve the real-time performance of terminal devices in performing intelligent tasks, the neural network model applied on the terminal device can be reasonably tailored.

[0003] How to efficiently and reasonably trim the neural network model has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the present application provides a neural network model pruning method, device, electronic device and storage medium to accelerate the compression of the neural network model while reducing performance loss.

[0005] Specifically, the present application is implemented through the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a neural network model pruning method is provided, comprising:

[0007] Dividing the neural network model to be pruned into a plurality of structural groups according to the structural characteristics of the neural network model to be pruned;

[0008] Determining a target clipping rate for clipping each of the multiple structure groups according to a change in the loss function before and after clipping the neural network model to be clipped; wherein the change in the loss function is determined according to a first factor and a second factor, the first factor is used to characterize the importance of each weight in each structure group, and the second factor is used to characterize the interaction between different structure groups;

[0009] According to the target cropping rate, each structure group is cropped respectively to obtain a cropped neural network model.

[0010] According to a second aspect of an embodiment of the present application, a neural network model clipping device is provided, comprising:

[0011] A division unit, used for dividing the neural network model to be pruned into a plurality of structural groups according to the structural characteristics of the neural network model to be pruned;

[0012] A determination unit, configured to determine a target clipping rate for clipping each of the multiple structure groups according to a change in the loss function before and after clipping the neural network model to be clipped; wherein the change in the loss function is determined according to a first factor and a second factor, the first factor being used to characterize the importance of each weight in each structure group, and the second factor being used to characterize the interaction between different structure groups;

[0013] The cropping unit is used to crop each structure group according to the target cropping rate to obtain a cropped neural network model.

[0014] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method provided in the first aspect.

[0015] According to a fourth aspect of an embodiment of the present application, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0016] The technical solution provided by this application can at least bring the following beneficial effects:

[0017] The neural network model to be pruned is divided into multiple structural groups according to its structural characteristics, and the target cropping rate for cropping each structural group is determined according to the change of the loss function before and after the pruned neural network model. Then, each structural group is pruned according to the target cropping rate to obtain the pruned neural network model. On the one hand, the neural network model to be pruned is divided into structural groups and the cropping rate is searched based on the structural groups; on the other hand, when determining the change of the loss function before and after the pruned neural network model, the importance of each weight in each structural group and the interaction between different structural groups are considered, thereby improving the rationality of the cropping rate of each structural group, and accelerating the compression of the neural network model with reduced performance loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of a neural network model pruning method shown in an exemplary embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a visual Transform model for image classification shown in an exemplary embodiment of the present application;

[0020] Figure 3is a schematic diagram of a process of neural network model pruning shown in an exemplary embodiment of the present application;

[0021] Figure 4 is a schematic diagram of dividing the Hessian matrix into blocks according to different structural objects shown in an exemplary embodiment of the present application;

[0022] Figure 5 is a structural schematic diagram of a neural network model cutting device shown in an exemplary embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of the hardware structure of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0024] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0025] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0026] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0027] See also Figure 1 , is a flow chart of a neural network model pruning method provided in an embodiment of the present application, such as Figure 1 As shown, the neural network model pruning method may include the following steps:

[0028] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0029] Step S100: Divide the neural network model to be pruned into a plurality of structural groups according to the structural characteristics of the neural network model to be pruned.

[0030] In the embodiments of the present application, considering that different structures in a neural network model generally have different functions, if the entire neural network model is pruned, at the same pruned rate, the performance loss of the pruned neural network model may be relatively large, and the performance of the pruned neural network model cannot be guaranteed.

[0031] Therefore, in order to optimize the performance of the pruned neural network model, the neural network model to be pruned can be divided into multiple units, and each unit is taken as a pruned object to pruned the neural network model to be pruned.

[0032] Accordingly, for any neural network model to be pruned, the neural network model to be pruned can be divided into multiple structural groups according to the structural characteristics of the neural network model to be pruned.

[0033] In one example, the neural network model to be pruned includes a Transform model, and the multiple structure groups may include: a head in a self-attention structure, a fully connected layer in a feedforward neural network, and a fully connected layer constrained by a residual connection.

[0034] In another example, the neural network model to be trimmed includes a convolutional neural network model. When the convolutional neural network model is structurally grouped, different network layers can be divided into different structural groups, or different stages of the convolutional neural network can be divided into different structural groups.

[0035] Step S110, determining a target clipping rate for clipping each of the multiple structure groups based on a change in the loss function before and after clipping the neural network model to be clipped; wherein the change in the loss function is determined based on a first factor and a second factor, the first factor is used to characterize the importance of each weight in each structure group, and the second factor is used to characterize the interaction between different structure groups.

[0036] In an embodiment of the present application, in order to ensure the performance of the pruned neural network model, in the process of determining the pruned rate of the neural network model, the performance loss of the pruned neural network model relative to the neural network model before pruned can be determined based on the change in the loss function of the neural network model before and after pruned.

[0037] Exemplarily, the greater the change in the loss function, the greater the performance loss of the pruned neural network model relative to the unpruned neural network model.

[0038] In addition, considering that when pruning each structural grouping, the pruning of any structural grouping will not only affect the processing performance of the structural grouping itself, but also affect the processing performance of other structural groupings, therefore, when pruning the structural groupings, it is necessary to consider not only the importance of the single structure itself, but also the interaction between different structural groupings, so as to avoid a serious decline in the performance of the neural network model after pruning.

[0039] Accordingly, when determining the change in the loss function of the neural network model before and after pruning, it is necessary to consider the influence and interaction of each structure in the above-mentioned multiple structural groups, model the interaction between different structural groups, and obtain a first factor for characterizing the importance of each weight in a single structural group, and a second factor for characterizing the interaction between different structural groups. For example, the importance of a single weight is used to characterize the effect of each weight on the network (i.e., the effect of pruning a single weight in a structural group on the performance loss of the model), and the interaction between different structural groups is used to characterize the effect of the correlation between structural groups on the network (i.e., the effect of pruning multiple structural groups on the performance loss of the model), and the change in the loss function of the neural network model before and after pruning is determined by combining the importance of a single weight and the interaction between different structural groups.

[0040] In the process of pruning the neural network model to be pruned, the pruned rate (which may be referred to as the target pruned rate) for pruned each of the multiple structural groupings may be determined based on the change in the loss function of the neural network model to be pruned before and after pruned.

[0041] Exemplarily, the determined target cropping ratio includes a plurality of cropping ratios for cropping the plurality of structure groups respectively.

[0042] For example, assuming that the neural network model to be pruned is divided into three structural groups, the target pruned rate may include three pruned rates, which are respectively used to pruned the three structural groups.

[0043] Exemplarily, for any structural grouping, the pruning rate of the structural grouping may be the ratio of the number of weights that need to be pruned to the total number of weights of the structural grouping.

[0044] Exemplarily, after pruning the above-mentioned multiple structure groups according to the target pruning rate, the computational amount of the pruned neural network model can be consistent with the given computational amount target, for example, the same as the given computational amount target, or the deviation from the given computational amount target is within a preset range.

[0045] Step S120: Crop each structure group separately according to the target cropping rate to obtain a cropped neural network model.

[0046] In an embodiment of the present application, when the target cropping rate for cropping each structural grouping is determined in the above manner, each structural grouping can be cropped separately according to the determined target cropping rate for cropping each structural grouping to obtain a cropped neural network model (which can be called a sub-model).

[0047] For example, for any structural grouping, the number of weights that need to be trimmed can be determined based on the target trimming rate of the structural grouping, and trimming can be started from the least important weight according to the importance of each weight in the structural grouping until the number of trimmed weights meets the requirement.

[0048] It can be seen that in Figure 1 In the method flow shown, the neural network model to be pruned is divided into multiple structural groups according to the structural characteristics of the neural network model to be pruned, and the target cropping rate for cropping each structural group is determined according to the change of the loss function before and after the pruned neural network model. Then, according to the target cropping rate, each structural group is pruned respectively to obtain the pruned neural network model. On the one hand, the neural network model to be pruned is divided into structural groups and the cropping rate is searched based on the structural groups; on the other hand, when determining the change of the loss function before and after the pruned neural network model, the importance of each weight in each structural group and the interaction between different structural groups are considered, thereby improving the rationality of the determined cropping rate of each structural group, and accelerating the compression of the neural network model with less performance loss.

[0049] In some embodiments, determining the target cropping rate for cropping each of the multiple structure groups based on the change in the loss function before and after cropping the neural network model to be cropped may include:

[0050] Initializing a cropping rate population; wherein the cropping rate population includes a plurality of seeds, and any seed includes a cropping rate for cropping each of the plurality of structure groups;

[0051] According to the changes in the loss functions corresponding to various seeds, the cropping rate population is iteratively updated until the number of iterations reaches a specified number of iterations, and the target cropping rate for cropping each of the multiple structure groups is determined according to the iteratively updated cropping rate population.

[0052] Exemplarily, in order to improve the rationality of the determined target cropping rate and further optimize the performance of the pruned neural network model, a population-based iterative optimization method can be used to search for the target cropping rate for pruned each structure grouping.

[0053] Exemplarily, when the neural network to be pruned is divided into a plurality of structural groups in the above manner, the pruned rate population can be initialized.

[0054] Exemplarily, the cropping rate population may include multiple seeds, and one seed may include cropping rates for cropping each structure grouping.

[0055] For any seed in the cropping rate population, each structure group can be cropped according to the cropping rate corresponding to the seed, and the change in the loss function of the neural network model before and after cropping (which can be called the loss function change corresponding to the seed) is determined.

[0056] When the loss function changes corresponding to various seeds in the cropping rate population are determined, the cropping rate population can be iteratively updated according to the loss function changes corresponding to various seeds until the number of iterations reaches the specified number of iterations, and the target cropping rate for cropping each structural grouping is determined according to the iteratively updated cropping rate population.

[0057] It should be noted that, for any seed in the cropping rate population, after cropping the above-mentioned multiple structure groups according to the cropping rate corresponding to the seed, the computational amount of the pruned neural network model can be consistent with the given computational amount target, for example, the same as the given computational amount target, or the deviation from the given computational amount target is within a preset range.

[0058] In an example, the iterative updating of the crop rate population according to the change of the loss function corresponding to each seed may include:

[0059] For any iteration, sort the seeds in the current crop rate population according to the change of the loss function corresponding to the seeds in the current crop rate population;

[0060] Based on the M seeds with the smallest change in the loss function in the current cropping rate population, a new cropping rate population is generated using an evolutionary algorithm; where M ≥ 1.

[0061] Exemplarily, in the process of iteratively updating the cropping rate population, for any iteration, the various seeds in the current cropping rate population may be sorted according to the change in the loss function corresponding to the various seeds in the current cropping rate population.

[0062] For example, various seeds in the current cropping rate population are sorted in the order of corresponding loss function changes from low to high; or, various seeds in the current cropping rate population are sorted in the order of corresponding loss function changes from high to low.

[0063] Exemplarily, a new crop rate population may be generated by using a crossover mutation method based on the M seeds with the smallest corresponding loss function changes in the current crop rate population.

[0064] Taking the example of sorting the seeds in the current cropping rate population according to the order of the corresponding loss function changes from low to high, a new cropping rate population can be generated based on the first M seeds in the sorted cropping rate population using a crossover mutation method.

[0065] In some embodiments, the first factor is a single-weight perturbation factor, and the second factor is a weight-associated perturbation factor;

[0066] The change in the loss function before and after pruning the neural network model to be pruned can be determined in the following way:

[0067] Determining the change of the loss function before and after pruning the neural network model to be pruned according to the single weight perturbation factors of the plurality of structural groups and the weight association perturbation factors between the plurality of structural groups;

[0068] Among them, the single weight perturbation factor is used to characterize the performance loss of the neural network model before and after a single weight is pruned, and the weight association perturbation factor is used to characterize the performance loss of the neural network model before and after the associated weights are pruned.

[0069] For example, considering that there may be correlations between different weights in a neural network model, for example, for two weights in a neural network model, when either of the two weights is pruned separately, the performance loss of the neural network model before and after the pruning may be relatively small, but when both weights are pruned, the performance loss of the neural network model before and after the pruning may be relatively large.

[0070] Accordingly, when determining the change in the loss function before and after pruning the neural network model, for the pruned weights, it is necessary to consider not only the impact of the pruning of each pruned weight on the performance of the neural network model, but also the correlation between the weights, that is, to consider the impact of pruning multiple associated weights on the neural network model.

[0071] Exemplarily, in the process of searching for the optimal cropping rate for each structural grouping, the change in loss function before and after cropping the neural network model to be cropped can be determined based on the single weight perturbation factors of multiple structural groupings and the weight association perturbation factors between multiple structural groupings. Furthermore, based on the change in loss function before and after cropping the neural network model to be cropped, the optimal cropping rate for cropping each structural grouping can be determined.

[0072] Exemplarily, a single weight interference factor of a weight may be determined based on a weight gradient.

[0073] It should be noted that when performing weight pruning, there may be multiple weights that need to be pruned together.

[0074] For example, for the head in the self-attention structure of the Transform model, when performing weight pruning, pruning needs to be performed in units of heads, but a head usually includes multiple weights, that is, the multiple weights need to be pruned together.

[0075] Accordingly, for multiple weights that need to be pruned together, the single weight interference factors corresponding to the multiple weights can be determined based on the impact of the pruning of the multiple weights on the performance of the neural network model.

[0076] Exemplarily, in the process of determining the weight interference factor between the structural groups, the Hessian matrix may be calculated and determined first, and the weight interference factor may be determined based on the Hessian matrix.

[0077] In one example, the weight association disturbance factors between the above-mentioned multiple structure groups are determined by:

[0078] For the first structural grouping and the second structural grouping among the above-mentioned multiple structural groupings, determine the weight association disturbance factor between the first structural grouping and the second structural grouping based on the number of weights in the first structural grouping, the number of weights in the second structural grouping, the clipping rate of the first structural grouping, the clipping rate of the second structural grouping, the first matrix block and the second matrix block; wherein the first matrix block is the matrix block corresponding to the first structural grouping in the Hessian matrix, and the second matrix block is the matrix block corresponding to the second structural grouping in the Hessian matrix.

[0079] For example, considering that each time the weight-related perturbation factor between structural groups is calculated, the Hessian matrix needs to be calculated, which will take up a lot of computing resources and affect the computing efficiency, the entire Hessian matrix can be divided into blocks according to different structural groups, and each structural group corresponds to a block in the Hessian matrix, and different structural objects interact to form cross terms.

[0080] Exemplarily, for any two structural groupings among the above-mentioned multiple structural groupings (referred to as the first structural grouping and the second structural grouping respectively), the weight association disturbance factor between the first structural grouping and the second structural grouping can be determined based on the number of weights in the first structural grouping, the number of weights in the second structural grouping, the clipping rate of the first structural grouping, the clipping rate of the second structural grouping, the first matrix block and the second matrix block. The specific implementation can be explained in conjunction with specific examples below.

[0081] In order to enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application, the technical solutions provided in the embodiments of the present application are described below with reference to specific examples.

[0082] In this embodiment, the neural network model to be pruned is taken as an example as a Transform model.

[0083] like Figure 2 As shown in, it is a schematic diagram of a visual Transform model for image classification, such as Figure 2 As shown in the figure, the input image is divided into multiple different image blocks through the patchify stem layer, and then these image blocks are sent to multiple Transformer layers (i.e. Figure 2 The backbone model composed of the Transformer block in the figure extracts image feature information, and finally inputs the feature information into the fully connected layer (also called the linear layer) for classification. In the embodiment, each Transformer layer contains a variety of heterogeneous structures, such as Figure 2 As shown in the figure, it includes two large substructures: one is the multi-head attention layer (MSA) and the other is the feedforward neural network layer (FFN). The multi-head attention layer includes a normalization layer, a multi-head attention operation, a fully connected layer, and an add layer, while the feedforward neural network layer includes a normalization layer, a linear layer, and a summation layer.

[0084] For example, the entire network can be divided into three structural groups according to the structure: the first structural group includes the head of the multi-head attention layer, such as Figure 2 As shown in the figure, each head contains three linear layers and dot-product attention. Each head can extract different feature information, but if the number of heads is too large, the same information will be extracted.

[0085] Taking the dog image to be classified as an example, head 1 and head 2 both focus on the dog's hair texture information, so one head can be cropped off.

[0086] The second structure is grouped into fully connected layers in a feed-forward neural network structure, such as Figure 2 As shown, it can fuse the information of different channels.

[0087] Taking the dog to be classified as an example, the multi-head attention layer has learned some feature information, which can be represented as a feature tensor with multiple channels. Each channel contains different information, such as the dog's tail feature and the dog's eye feature. Each neuron in the fully connected layer can fuse these channel features. If there are too many neurons, different neurons will fuse the same features. In this case, the redundant neurons can be pruned.

[0088] The third structure is grouped into fully connected layers in two substructures, such as Figure 2As shown in Figure 2, since these two structures need to maintain the same structure through residual connection constraints, they can be divided into the same structure group.

[0089] It can be seen that through the above-mentioned structural grouping method, the redundant structures of different structural groups can be automatically identified and pruned and compressed, which effectively reduces the number of parameters and calculations with a small performance loss.

[0090] Exemplarily, the amount of computation of the pruned Transform model can be specified by the user, and then the structure can be grouped and pruned according to the above method. According to the amount of computation specified by the user, a compressed model of the corresponding size can be obtained. The entire technical solution is implemented quickly and efficiently, and the compressed model has high accuracy and good performance.

[0091] like Figure 3 As shown, the implementation process of the clipping solution of the Transform model provided in this embodiment may include:

[0092] 1. Structural grouping

[0093] For example, Figure 2 Taking the visual Tranformer model for image classification shown in FIG. 1 as an example, it can be divided into different structural groups according to its structure (a structural group can be called a structural object), and each structural object has its own cropping rate.

[0094] 2. Clipping rate evolution search

[0095] Exemplarily, an evolutionary search algorithm may be used to determine the most appropriate cropping rate (ie, the above-mentioned target cropping rate) for each structural object.

[0096] Exemplarily, when configuring the initial parameters of the evolutionary search, the cropping rate population may be randomly initialized, and each seed in the population represents the cropping rate of each object.

[0097] In each round of iteration, each seed in the population is pruned according to its pruned rate, and the obtained sub-model calculates the loss function change of the seed (which can be called the approximate loss function change). All seeds are sorted according to the approximate loss function change, and the M seeds with the smallest approximate loss function change are selected as the next generation, and a new population is generated by crossover mutation. Iteration continues until the set number of iterations is reached, and the best sub-model is searched.

[0098] Exemplarily, for any structural object, when the structural object is pruned, all weights in the structural object can be sorted according to the importance of each weight, and then pruned starting from the weight with the least importance according to a given pruned ratio to obtain a simpler structural object.

[0099] For example, the change in the approximate loss function before and after the Transform model is pruned can be determined by:

[0100] Assuming the model loss function is L and the weight is W, after pruning it becomes W+ΔW accordingly, and the corresponding loss function changes to:

[0101] ΔL=L(W+ΔW)-L(W) (1)

[0102] The change of the loss function can reflect the impact on the model performance. The loss function is expanded using Taylor to obtain a calculation formula that can be analyzed and quantified:

[0103]

[0104] The first term on the right side of Formula 2 represents the disturbance effect of each weight itself (i.e., the single weight disturbance factor).

[0105] Exemplarily, assuming that S is the minimum unit of weight clipping, for example, for a weight that can be clipped individually, S can be the single weight, and for weights that need to be clipped together, for example, multiple weights in a head, S can be a collection of the multiple weights, then the single weight perturbation factor can be:

[0106]

[0107] Among them, g represents the gradient, is the weight w i gradient.

[0108] The second term on the right side of Formula 2 represents the mutual influence between different structures (ie, the above-mentioned weight-related disturbance factor).

[0109] For example, the determination of the weight-related perturbation factor requires the calculation of the Hessian matrix. If the Hessian matrix is ​​calculated each time the weight-related perturbation factor is determined, a large amount of computing resources will be required. Since different structures play different roles, the entire Hessian matrix can be calculated according to the structural blocks, and the entire Hessian matrix can be divided into blocks according to different structural objects, such as Figure 4 That is, each structure object corresponds to a matrix block in the Hessian matrix, and different structure objects interact to form cross terms.

[0110] For example, H {kl} Represents the association matrix between structure object k and structure object l:

[0111]

[0112] Where, ΔW{k} represents the overall weight change of the structure object k. For example, assuming that the structure object k has N neurons, then ΔW {k} It is an N-dimensional vector, and the value of each dimension represents the weight change value of each neuron (for example, from 0 to 1).

[0113] ΔW {k} H {kl} ΔW {l} Represents the interaction between structure object k and structure object l. The value ranges of k and l are the same, which are the number of structure objects after grouping. For example, if the neural network to be pruned is divided into 3 structure groups (that is, the number of structure objects is 3), then the value ranges of k and l are both (1, 2, 3).

[0114] For each matrix block, the following calculations can be performed:

[0115] ΔW {k} H {kl} ΔW {l} =N {k} N {l} u {kl} ρ {k} ρ {l} (5)

[0116] Among them, N {k} Represents the number of all weights of the structure object k, which can be directly obtained from the model parameters.

[0117] The matrix block consisting of the structure object k and the structure object l is calculated and only needs to be calculated once in advance.

[0118] in, W {k} H {kl} W {l} Find the average, W {k} H {kl} W {l} is a matrix containing multiple elements. For W {k} H {kl} W {l} Find the average value of each element in .

[0119] ρ {k} Indicates the cropping rate of the structure object k.

[0120] It should be noted that the third term on the right side of the Taylor expansion formula of the above loss function has a very small effect on △L and can be ignored.

[0121] Therefore, the calculation of the entire approximate loss function change can be as follows:

[0122]

[0123] The method provided by the present application is described above. The device provided by the present application is described below:

[0124] See also Figure 5 , is a schematic diagram of the structure of a neural network model cutting device provided in an embodiment of the present application, such as Figure 5 As shown, the neural network model cutting device may include:

[0125] A division unit 510, configured to divide the neural network model to be pruned into a plurality of structural groups according to the structural characteristics of the neural network model to be pruned;

[0126] A determination unit 520 is used to determine a target clipping rate for clipping each of the multiple structure groups according to a change in the loss function before and after clipping the neural network model to be clipped; wherein the change in the loss function is determined according to a first factor and a second factor, the first factor is used to characterize the importance of each weight in each structure group, and the second factor is used to characterize the interaction between different structure groups;

[0127] The cropping unit 530 is used to crop each structure group according to the target cropping rate to obtain a cropped neural network model.

[0128] In some embodiments, the neural network model to be pruned includes a Transform model;

[0129] The multiple structure groups include: a head in a self-attention structure, a fully connected layer in a feedforward neural network, and a fully connected layer constrained by a residual connection;

[0130] or,

[0131] The neural network model to be trimmed includes a convolutional neural network model;

[0132] The division unit 510 divides the neural network model to be pruned into a plurality of structural groups, including:

[0133] Different network layers of the convolutional neural network model are divided into different structural groups; or different stages of the convolutional neural network model are divided into different structural groups.

[0134] In some embodiments, the determining unit 520 determines the target clipping rate for clipping each of the multiple structure groups according to the change of the loss function before and after clipping the neural network model to be clipped, including:

[0135] Initializing a cropping rate population; wherein the cropping rate population includes a plurality of seeds, and any seed includes a cropping rate for cropping each of the plurality of structure groups;

[0136] According to the change of the loss function corresponding to each seed, the cropping rate population is iteratively updated until the number of iterations reaches a specified number of iterations, and the target cropping rate for cropping each of the multiple structure groups is determined according to the iteratively updated cropping rate population.

[0137] In some embodiments, the determining unit 520 iteratively updates the crop rate population according to the change of the loss function corresponding to each seed, including:

[0138] For any iteration, sort the seeds in the current crop rate population according to the change of the loss function corresponding to the seeds in the current crop rate population;

[0139] Based on the M seeds with the smallest change in the loss function in the current cropping rate population, a new cropping rate population is generated using crossover mutation; where M ≥ 1.

[0140] In some embodiments, the first factor is a single-weight perturbation factor, and the second factor is a weight-associated perturbation factor;

[0141] The change in the loss function before and after pruning the neural network model to be pruned is determined by:

[0142] Determining the change of the loss function before and after pruning the neural network model to be pruned according to the single weight perturbation factors in the multiple structural groups and the weight association perturbation factors between the multiple structural groups;

[0143] Among them, the single weight perturbation factor is used to characterize the performance loss of the neural network model before and after a single weight is pruned, and the weight association perturbation factor is used to characterize the performance loss of the neural network model before and after the associated weights are pruned.

[0144] In some embodiments, the weight association perturbation factors between the plurality of structure groups are determined by:

[0145] For the first structural grouping and the second structural grouping among the multiple structural groupings, determine the weight association disturbance factor between the first structural grouping and the second structural grouping based on the number of weights in the first structural grouping, the number of weights in the second structural grouping, the cropping rate of the first structural grouping, the cropping rate of the second structural grouping, the first matrix block and the second matrix block; wherein the first matrix block is the matrix block corresponding to the first structural grouping in the Hessian matrix, and the second matrix block is the matrix block corresponding to the second structural grouping in the Hessian matrix.

[0146] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the neural network model pruning method described above.

[0147] See also Figure 6 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 601 and a memory 602 storing machine executable instructions. The processor 601 and the memory 602 may communicate via a system bus 603. In addition, by reading and executing the machine executable instructions corresponding to the neural network model pruning logic in the memory 602, the processor 601 may execute the neural network model pruning method described above.

[0148] The memory 602 mentioned herein may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (RadomAccess Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0149] In some embodiments, a machine-readable storage medium is also provided. Figure 6 The memory 602 in the machine-readable storage medium stores machine executable instructions, and when the machine executable instructions are executed by the processor, the neural network model pruning method described above is implemented. For example, the storage medium can be ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0150] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0151] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A neural network model pruning method, characterized in that: include: In the case where the neural network model to be pruned has multiple structural groups, a target pruned rate for pruned each of the multiple structural groups is determined according to a change in the loss function before and after the pruned neural network model; wherein the neural network model to be pruned includes a Transform model; the multiple structural groups include: a head in a self-attention structure, a fully connected layer in a feedforward neural network, and a fully connected layer constrained by residual connections; the mutual importance between different structural groups is characterized by a Hessian matrix; According to the target cropping rate, each structure group is cropped respectively to obtain a cropped neural network model.

2. The method according to claim 1, characterized in that The change of the loss function is determined based on a first factor and a second factor, wherein the first factor is used to characterize the importance of each weight in each structural grouping, and the second factor is used to characterize the interaction between different structural groupings.

3. The method according to claim 1, characterized in that The step of determining a target clipping rate for clipping each of the plurality of structure groups based on a change in a loss function before and after clipping the neural network model to be clipped comprises: Initializing a cropping rate population; wherein the cropping rate population includes a plurality of seeds, and any seed includes a cropping rate for cropping each of the plurality of structure groups; According to the change of the loss function corresponding to each seed, the cropping rate population is iteratively updated until the number of iterations reaches a specified number of iterations, and the target cropping rate for cropping each of the multiple structure groups is determined according to the iteratively updated cropping rate population.

4. The method according to claim 3, characterized in that The iterative updating of the cropping rate population according to the change of the loss function corresponding to each seed includes: For any iteration, sort the seeds in the current crop rate population according to the change of the loss function corresponding to the seeds in the current crop rate population; Based on the M seeds with the smallest change in the loss function in the current cropping rate population, a new cropping rate population is generated using crossover mutation; where M ≥ 1.

5. The method according to claim 2, characterized in that: The step of cutting each structure group separately according to the target cutting rate includes: For any structure group, when the structure object is pruned, all weights in the structure object are sorted according to the importance of each weight, and pruned starting from the weight with the least importance according to the given pruned ratio.

6. The method according to claim 1, characterized in that The change in the loss function of the Transform model before and after cropping is determined by the following formula: Among them, S is the pruned weight set, g represents the gradient, is the weight w i The gradient of ΔW {k} represents the overall weight change of structure group k, H {kl} represents the association matrix between structure group k and structure group l, ΔW {k} H {kl} ΔW {l} It represents the interaction between structure group k and structure group l, and the value range of k and l is the same; For each matrix block, the following formula is satisfied: ΔW {k} H {kl} ΔW {l} =N {k} N {l} you {kl} ρ {k} ρ {l} Among them, N {k} Represents the total number of weights of the structure group k, It is calculated by the matrix block composed of structure group k and structure group l, W {k} H {kl} W {l} Find the average, W {k} H {kl} W {l} is a matrix containing multiple elements. For W {k} H {kl} W {l} The average value of each element in ρ {k} represents the pruning rate of structure group k.

7. A neural network model cutting device, characterized in that: include: A determination unit is used to determine a target cropping rate for cropping each of the multiple structural groups according to a change in the loss function before and after the cropping of the neural network model to be cropped, when the neural network model to be cropped has multiple structural groups; wherein the change in the loss function is determined according to a first factor and a second factor, the first factor is used to characterize the importance of each weight in each structural group, and the second factor is used to characterize the interaction between different structural groups; wherein the neural network model to be cropped includes a Transform model; the multiple structural groups include: a head in a self-attention structure, a fully connected layer in a feedforward neural network, and a fully connected layer constrained by a residual connection; the mutual importance between different structural groups is characterized by a Hessian matrix; The cropping unit is used to crop each structure group according to the target cropping rate to obtain a cropped neural network model.

8. The device according to claim 7, characterized in that The change of the loss function is determined based on a first factor and a second factor, wherein the first factor is used to characterize the importance of each weight in each structural grouping, and the second factor is used to characterize the interaction between different structural groupings; The determining unit determines the target clipping rate for clipping each of the multiple structure groups according to the change of the loss function before and after clipping the neural network model to be clipped, including: Initializing a cropping rate population; wherein the cropping rate population includes a plurality of seeds, and any seed includes a cropping rate for cropping each of the plurality of structure groups; Iteratively updating the cropping rate population according to the change of the loss function corresponding to each seed until the number of iterations reaches a specified number of iterations, and determining the target cropping rate for cropping each of the multiple structure groups according to the iteratively updated cropping rate population; The determining unit iteratively updates the cropping rate population according to the change of the loss function corresponding to each seed, including: For any iteration, sort the seeds in the current crop rate population according to the change of the loss function corresponding to the seeds in the current crop rate population; Based on the M seeds with the smallest change in the loss function in the current crop rate population, a new crop rate population is generated by crossover mutation; where M ≥ 1; The cropping unit crops each structure group according to the target cropping rate, including: For any structure group, when pruning the structure object, all weights in the structure object are sorted according to the importance of each weight, and pruning starts from the weight with the least importance according to the given pruning rate; and / or, The change in the loss function of the Transform model before and after cropping is determined by the following formula: Among them, S is the pruned weight set, g represents the gradient, is the weight w i The gradient of ΔW {k} represents the overall weight change of structure group k, H {kl} represents the association matrix between structure group k and structure group l, ΔW {k} H {kl} ΔW {l} It represents the interaction between structure group k and structure group l, and the value range of k and l is the same; For each matrix block, the following formula is satisfied: ΔW {k} H {kl} ΔW {l} =N {k} N {l} you {kl} ρ {k} ρ {l} Among them, N {k} Represents the total number of weights of the structure group k, It is calculated by the matrix block composed of structure group k and structure group l, W {k} H {kl} W {l} Find the average, W {k} H {kl} W {l} is a matrix containing multiple elements. For W {k} H {kl} W {l} The average value of each element in ρ {k} represents the pruning rate of structure group k.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method according to any one of claims 1 to 6.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images